SAGE2 (Scalable Amplified Group Environment) is the de-facto platform to support group work on wide-band display environments. Unlike most web applications, the SAGE environment, due to the nature of its collaborative model, needs a nuanced handling of security aspects. This paper details the security requirements of SAGE2, the Identity and Access Control model that was developed to address those requirements, and the details of the Identity and Access Management system that the SAGE team implemented based on this new model. Further, we present a comparison of this new system with some of the popular collaboration platforms to highlight the uniqueness of SAGE2 integrated with this new Identity and Access Management system. 
Keywords: security, access control, collaborative environment, wide-band displays

Examples of SAGE2 wide-band displays

Securing Collaborative Work in Wide-band Display Environments

Figures within biomedical articles present essential evidence of the relevance of a publication in a curation workflow. In particular, visual cues of the image modality or experimental methods can help expert curators identify relevant papers from an increasing number of publications. Automating the identification of these content-bearing images can thus be helpful in computer-assisted curation. However, the paucity of labeled datasets and the specialized training required to label such images hinder the development of such tools. To address this problem, we present the design of ANIMO, a labeling system that integrates extraction and segmentation tools to ease the annotation burden. We first introduce two taxonomies of image modalities and experimental methods, derived in collaboration with curators. On the back-end of the system, we process batches of documents and create a labeling task per document. At the front-end, expert curators can access these tasks through a web interface and access the article of interest. We describe the evaluation of this system by a group of biocurators, and the human factor lessons learned from this interdisciplinary experience. 
doi: 10.1109/BIBM52615.2021.9669898

Front-end interface for labeling a document showing a taxonomy adapted for the CORD-19 dataset.

ANIMO: Annotation of Biomed Image Modalities

The spread of the SARS-CoV-2 virus and its contagious disease COVID-19 has impacted countries to an extent not seen since the
1918 flu pandemic. In the absence of an effective vaccine and as cases surge worldwide, governments were forced to adopt measures to inhibit the spread of the disease. To reduce its impact and to guide policy planning and resource allocation, researchers have been developing models to forecast the infectious disease. Ensemble models, by aggregating forecasts from multiple individual models, have been shown to be a useful forecasting method. However, these models can still provide less-than-adequate forecasts at higher spatial resolutions. In this paper, we built COVID-19 EnsembleVis, a web-based interactive visual interface that allows the assessment of the errors of ensembles and individual models by enabling users to effortlessly navigate through and compare the outputs of models considering their space and time dimensions. COVID-19 EnsembleVis enables a more detailed understanding of uncertainty and the range of forecasts generated by individual models. 
Index Terms: Human-centered computing; Visualization; Visualization application domains; Visual analytics

COVID-19 EnsembleVis: Visual Analysis of County-level Ensemble Forecast Models

There is a steadily growing interest in leveraging ecosystems of digital devices that go beyond a single desktop for collaborative visual data analysis and exploration. This new thrust of multi-device interfaces supports new models for complex collaboration scenarios, and have great potential to support analysts in their data analysis by utilizing each device’s capabilities. However, there are some challenges inherently associated with visual data analysis in multi-device environments (MDE). This dissertation investigates how the analytical process occurs in multi-user multi-device environments to provide a theoretical understanding of collaborative exploratory visual data analysis and better inform the design of visualization tools. First, I touched on the challenges of designing cross-device visualization tools by introducing the design and implementation of a multi-device system for collaborative visual data analysis that enables cross-device visualization sharing and simultaneous interaction. Then, through an exploratory user study, I evaluated strategies of exploratory visual data analysis in a collaborative multi-user multi-device environment. I synthesized a two-level characterization of the analysis structure from observed analysis behaviors. I observed that subjects navigate the data space in three identified exploration patterns and the analysis was primarily depth-oriented. In addition, the cost of deciding what to explore next &ldquo;Gulf of Goal Formation&rdquo; is higher in collaborative settings due to short-term memory and the recency effect. I hypothesized that visualizing the dimensions search space would increase the breadth of the analysis and reduce the decision cost. Using a between-groups study, I evaluated the effect of revealing information about what dimension&rsquo;s data space coverage(s) were investigated and what were left. The results indicate that visualizing dimensions search space increases the breadth of the analysis and reduces the decision cost by positively affecting the rate of views generation.

This research investigates the design of visualization tools for collaborative exploratory visual data analysis in multi-device environments.

Supporting Collaborative Exploratory Visual Data Analysis in Multi-device Environments

Data exploration stands to benefit from environments that permit users to examine and juxtapose many views of data, particularly views that present diverse selections of data values and attributes. Large, high-resolution environments are capable of showing many related views of data, but efficiently creating and displaying visualizations in these environments presents significant challenges. In this dissertation, I will present my research on &ldquo;multi-view data exploration interactions&rdquo; that enable users to create sets of views with coherent data value and attribute variations, through multi-modal speech and mid-air pointing gestures in large display
environments. This work enables users to rapidly and efficiently generate sets of views in support of multi-view data exploration tasks, organize these views in coherent collections, and operate on sets of views collectively, rather than individually, to efficiently reach large portions of the &lsquo;data and attribute space&rsquo;. I will present three contributions: 1) an observational study of data exploration in a large display environment with speech and mid-air gestures, 2) &lsquo;Traverse&rsquo;, an interaction technique for data exploration, based on this study, which uses natural language to create and pivot sets of views, and 3) &lsquo;Ditto&rsquo;, a multi-modal speech and mid-air pointing gesture interactive environment, which utilizes the multi-view data exploration technique, in
large display environments.

Study setting with the remote mediator in an adjacent room generating views using Tableau and dropping them to the user&rsquo;s display environment using SAGE2.

Interaction with Multiple Data Visualizations Through Natural Language Commands

Aircraft engines emit particulates that alter the chemical composition of the atmosphere and perturb the Earth’s radiation budget by creating additional ice clouds in the form of condensation trails called contrails. We propose a multi-scale visual computing system that will assist in defining contrail features with parameter analysis for computer-generated aircraft engine simulations. These simulations are computationally intensive and rely on high-performance computing (HPC) solutions. Our multi-linked visual system seeks to help in the identification of the formation and evolution of contrails and in the identification of contrail-related spatial features from the simulation workflow.

Visual interface of aircraft engine simulation analysis.

Parameter Analysis and Contrail Detection of Aircraft Engine Simulations

The graph convolutional network (GCN) has recently achieved promising performance of 3D human pose estimation (HPE) by modeling the relationship among body parts. However, most prior GCN approaches suffer from two main drawbacks. First, they share a feature transformation for each node within a graph convolution layer. This prevents them from learning different relations between different body joints. Second, the graph is usually defined according to the human skeleton and is suboptimal because human activities often exhibit motion patterns beyond the natural connections of body joints. To address these limitations, we introduce a novel Modulated GCN for 3D HPE. It consists of two main components: weight modulation and affinity modulation. Weight modulation learns different modulation vectors for different nodes so that the feature transformations of different nodes are disentangled while retaining a small model size. Affinity modulation adjusts the graph structure in a GCN so that it can model additional edges beyond the human skeleton. We investigate several affinity modulation methods as well as the impact of regularizations. Rigorous ablation study indicates both types of modulation improve performance with negligible overhead. Compared with state-of-the-art GCNs for 3D HPE, our approach either significantly reduces the estimation errors, e.g., by around 10%, while retaining a small model size or drastically reduces the model size, e.g., from 4.22M to 0.29M (a 14.5 X reduction), while achieving comparable performance. Results on two benchmarks show our Modulated GCN outperforms some recent states of the art. 
<a href="https://github.com/ZhimingZo/Modulated-GCN">Download Code</a> 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Modulated Graph Convolutional Network for 3D Human Pose Estimation

The latest developments of machine learning (ML) and deep learning (DL) algorithms have paved the way to effectively analyze the atomic structure of chemically-complex materials. In this work, we present a DL model built upon a fully convolutional neural network (FCN) to resolve the random elements distribution of complex PtNiPdCoFe high-entropy alloys (HEAs) represented in the scanning transmission electron microscopy (STEM) images at atomic resolution. The objective of the proposed neural network is to learn through semantic seg-mentation the non-linear correlations between the pixels&rsquo; intensities of STEM images and the number of atoms of different constituent elements in the atomic columns (i.e., column heights) in the HEA&rsquo;s structure. We demonstrate that our DL model is capable of correctly estimating the column heights or with an error up to 1 atom for the majority of the columns in the HEA structures represented in the simulated STEM images used to train and test the network. This establishes a sufficiently high level of confidence in the estimation of the element distribution in experimental images. The predicted distributions in different STEM images of nanoparticles reveal inhomogeneous fluctuations with local aggregations in the elemental atomic fractions within the columns. The most pronounced aggregation is displayed by Pt, which is the largest and most electronegative element in the synthesized HEA material. The proposed DL method is beneficial for an in-depth characterization of the structural properties of HEAs and multielement 3D materials in general. 
Keywords: Deep learning, High entropy alloys, STEM images, Column heights, Elements distribution

Example of atomic model of HEA characterized by a nearly equiatomic composition of Pt, Ni, Pd, Co and Fe.

Deep learning for mapping element distribution of high-entropy alloys in scanning transmission electron microscopy images

Support vector regression (SVR) is particularly beneficial when the outcome and predictors are nonlinearly related. However, when many covariates are available, the method&rsquo;s flexibility can lead to overfitting and an overall loss in predictive accuracy. To overcome this drawback, we develop a feature selection method for SVR based on a genetic algorithm that iteratively searches across potential subsets of covariates to find those that yield the best performance according to a user-defined fitness function. We evaluate the performance of our feature selection method for SVR, comparing it to alternate methods including LASSO and random forest, in a simulation study. We find that our method yields higher predictive accuracy than SVR without feature selection. Our method outperforms LASSO when the relationship between covariates and outcome is nonlinear. Random forest performs equivalently to our method in some scenarios, but more poorly when covariates are correlated. We apply our method to predict donor kidney function 1 year after transplant using data from the United Network for Organ Sharing national registry.

Results of simulation 1 setting with 300 total covariates X, 15 of them truly associated with the outcome Y.

Feature Selection for Support Vector Regression Using a Genetic Algorithm

Exploring large virtual environments, such as cities, is a central task in several domains, such as gaming and urban planning. VR systems can greatly help this task by providing an immersive experience; however, a common issue with viewing and navigating a city in the traditional sense is that users can either obtain a local or a global view, but not both at the same time, requiring them to continuously switch between perspectives, losing context and distracting them from their analysis. In this paper, our goal is to allow users to navigate to points of interest without changing perspectives. To accomplish this, we design an intuitive navigation interface that takes advantage of the strong sense of spatial presence provided by VR. We supplement this interface with a perspective that warps the environment, called UrbanRama, based on a cylindrical projection, providing a mix of local and global views. The design of this interface was performed as an interactive process in collaboration with architects and urban planners. We conducted a qualitative and a quantitative pilot user study to evaluate UrbanRama and the results indicate the effectiveness of our system in reducing perspective changes, while ensuring that the warping doesn&rsquo;t affect distance and orientation perception. 
Index Terms: Virtual Reality, VR Navigation, Cylindrical deformation. 
<a href="https://youtu.be/M8BFZnxq-Qg">Watch the video on YouTube.</a>

The UrbanRama mode in our urban VR system deforms the city allowing the user to obtain both a local as well as a global perspective of the urban environment.

UrbanRama: Navigating Cities in Virtual Reality

Motivation: Biomedical research findings are typically disseminated through publications. To simplify access to domain-specific knowledge while supporting the research community, several biomedical databases devote significant effort to manual curation of the literature - a labor intensive process. The first step toward biocuration requires identifying articles relevant to the specific area on which the database focuses. Thus, automatically identifying publications relevant to a specific topic within a large volume of publications is an important task toward expediting the biocuration process and, in turn, biomedical research. Current methods focus on textual contents, typically extracted from the title-and-abstract. Notably, images and captions are often used in publications to convey pivotal evidence about processes, experiments and results. 
Results: We present a new document classification scheme, using both image and caption information, in addition to titles-and-abstracts. To use the image information, we introduce a new image representation, namely Figure-word, based on class labels of subfigures. We use word embeddings for representing captions and titles-and-abstracts. To utilize all three types of information, we introduce two information integration methods. The first combines Figure-words and textual features obtained from captions and titles-and-abstracts into a single larger vector for document representation; the second employs a meta-classification scheme. Our experiments and results demonstrate the usefulness of the newly proposed Figure-words for representing images. Moreover, the results showcase the value of Figure-words, captions and titles-and-abstracts in providing complementary information for document classification; these three sources of information when combined, lead to an overall improved classification performance. 
Availability and implementation: Source code and the list of PMIDs of the publications in our datasets are available upon request. 
<a href="
https://doi.org/10.1093/bioinformatics/btab331">https://doi.org/10.1093/bioinformatics/btab331</a>

The process for converting figures into their corresponding Figure-words. The set of figures extracted from the biomedical document (left). The corresponding Figure-word with their vector representations indicating the types of comprising panel (right).

Utilizing Image and Caption Information for Biomedical Document Classification

Collaborative Virtual Environments have been increasingly used in supporting group work. Several collaborative systems have come up in recent years as a way of meeting the ever changing needs of group work, with asymmetric setups being the latest addition to a diverse collection of systems. In recent years, we see attempts at multi-domain collaboration. Research on interdisciplinary collaboration reveals a need for supporting a hybrid of tools and representations that are familiar to users of different domains. 
In this dissertation, I present a prototype implementation of a Hybrid Collaborative Virtual Environment that combines multiple representations to facilitate multi-domain collaboration. Preliminary evaluation of the prototype showed that hybridization of different representations can lead to role formation due to dependency on representations, when users are limited to interacting with a single representation. Integrating shared views and shared interaction into this Hybrid Collaborative Virtual Environment with the help of a large shared display environment can help solve the representational dependency issue. Shared views and shared interactions
enable users to freely interact with any of the different representations and also eases communication amongst them. I evaluate this integration by means of a formal study having a within group design: A group of three participants perform collaborative tasks using the Hybrid Collaborative Virtual Environment with and without the integration of shared views and shared interactions. I discuss the results of the study and show how shared views and shared interaction improves the collaborative experience.

Hybrid Collaborative Virtual Environments

Patient-Reported Outcome (PRO) surveys are used to monitor patients&rsquo; symptoms during and after cancer treatment. Late symptoms refer to those experienced after treatment. While most patients experience severe symptoms during treatment, these usually subside in the late stage. However, for some patients, late toxicities persist negatively affecting the patient&rsquo;s quality of life (QoL). In the case of head and neck cancer patients, PRO surveys are recorded every week during the patient&rsquo;s visit to the clinic and at different follow-up times after the treatment has concluded. In this paper, we model the PRO data as a time-series and apply Long-Short Term Memory (LSTM) neural networks for predicting symptom severity in the late stage. The PRO data used in this project corresponds to MD Anderson Symptom Inventory (MDASI) questionnaires collected from head and neck cancer patients treated at the MD Anderson Cancer Center. We show that the LSTM model is effective in predicting symptom ratings under the RMSE and NRMSE metrics. Our experiments show that the LSTM model also outperforms other machine learning models and time-series prediction models for these data. 
CCS Concepts: Computing methodologies; Neural networks 
<stong>Keywords: Long Short-Term Memory (LSTM), Patient Reported Outcomes (PRO), Late Toxicity, Symptom Severity Prediction

Validation RMSE (a) and NRMSE (b) scores for the prediction of individual symptoms 6Wk and 12Mo after treatment.

Predicting late symptoms of head and neck cancer treatment using LSTM and patient reported outcomes

To improve risk prediction for oropharyngeal cancer (OPC) patients using cluster analysis on the radiomic features extracted from pre‑treatment Computed Tomography (CT) scans. 553 OPC Patients randomly split into training (80%) and validation (20%), were classified into 2 or 3 risk groups by applying hierarchical clustering over the co‑occurrence matrix obtained from a random survival forest (RSF) trained over 301 radiomic features. The cluster label was included together with other clinical data to train an ensemble model using five predictive models (Cox, random forest, RSF, logistic regression, and logistic‑elastic net). Ensemble performance was evaluated over the independent test set for both recurrence-free survival (RFS) and overall survival (OS). The Kaplan–Meier curves for OS stratified by cluster label show significant differences for both training and testing (p val < 0.0001). When compared to the models trained using clinical data only, the inclusion of the cluster label improves AUC test performance from .62 to .79 and from .66 to .80 for OS and RFS, respectively. The extraction of a single feature, namely a cluster label, to represent the high‑dimensional radiomic feature space reduces the dimensionality and sparsity of the data. Moreover, inclusion of the cluster label improves model performance compared to clinical data only and offers comparable performance to the models including raw radiomic features.

Processing pipeline overview. The data is split into disjoint training and validation (test) sets.

Oropharyngeal cancer patient stratification using random forest based‑learning over high‑dimensional radiomic features

Dependency relations among visual entities are ubiquitous because both objects and scenes are highly structured. They provide prior knowledge about the real world that can help improve the generalization ability of deep learning approaches. Different from contextual reasoning which focuses on feature aggregation in the spatial domain, visual dependency reasoning explicitly models the dependency relations among visual entities. In this paper, we introduce a novel network architecture, termed the dependency network or DependencyNet, for semantic segmentation. It unifies dependency reasoning at three semantic levels. Intra-class reasoning decouples the representations of different object categories and updates them separately based on the internal object structures. Inter-class reasoning then performs spatial and semantic reasoning based on the dependency relations among different object categories. We will have an in-depth investigation on how to discover the dependency graph from the training annotations. Global dependency reasoning further refines the representations of each object category based on the global scene information. Extensive ablative studies with a controlled model size and the same network depth show that each individual dependency reasoning component benefits semantic segmentation and they together significantly improve the base network. Experimental results on two benchmark datasets show the DependencyNet achieves comparable performance to the recent states of the art. 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Visual dependence relations are ubiquity since both ob-jects and scenes are highly structured. They provide prior knowl-edge about the real world that can be used to improve the gener-alization ability of a learning model.

Exploit Visual Dependency Relations for Semantic Segmentation

Each year, over 60,000 people in the United States are diagnosed with head and neck cancers. Treatment strategies are dynamic and complex and can involve multiple courses of therapy with different potential outcomes. With head and neck cancer numbers increasing every year, partly due to viral infections, clinicians aim to develop customized treatment strategies tailored to each individual patient, under a healthcare model termed &ldquo;precision medicine&rdquo;. Precision medicine proposes the customization of healthcare, with medical decisions, practices, and products being tailored to the individual patient. The customization is based on individual factors collected from cohorts of patients who are similar to the patient under consideration. These factors may include disease markers, treatment options, demographics, and genetic, lifestyle, environmental, laboratory, or quality of life data. These data are heterogeneous, large-scale, and locally sparse. Making similarity-based precise recommendations for a specific patient requires novel approaches which blend biomedicine with complex quantitative methodology. We describe a first application of deep-Q-learning (DQL) and simulation as a digital twin dyad to optimize multi-step treatment of head and neck cancer patients, and to predict patient survival and toxicity outcomes. The treatment decision DQL digital twin and the patient’s digital twin were created, trained and evaluated using Google Colab for the HPC solution on a dataset of 536 oropharyngeal squamous cell carcinoma (OPC) patients with the goal of, respectively, determining the optimal treatment decisions with respect to survival and toxicity metrics, and predicting the outcomes of the optimal treatment on the patient. The models were trained on a subset of 402 patients (split randomly) and evaluated on a separate set of 134 patients. The main outcomes and measures include accuracy in predicting the final outcomes of a treatment sequence, and improvement in predicted outcomes rates of the DQL when compared to the observed physician outcomes. On the validation set, we obtained 87.09% mean and 90.85%
median accuracy in treatment outcome prediction. Given the prediction accuracy and predicted improvement on medically relevant outcomes yielded by this approach, this digital twin dyad of the patient-physician dynamic treatment problem has the potential of aiding physicians in determining the optimal course of treatment and in assessing its outcomes. 
Keywords: digital twin dyad, reinforcement learning, head and neck cancer

Model performance for the combined outcome (OS+DP) models without (left) and with radiomics (right).

A digital twin dyad with deep Q-learning in head and neck cancer treatment

Purpose: To determine whether patient similarity in terms of head and neck cancer spread through lymph nodes correlates significantly with radiation-associated toxicity. 
Materials and methods: 582 head and neck cancer patients received radiotherapy for oropharyngeal cancer (OPC) and had non-metastatic affected lymph nodes in the head and neck. Affected lymph nodes were segmented from pretreatment contrast-enhanced tomography scans and categorized according to consensus guidelines. Similar patients were clustered into 4 groups according to a graph-based representation of disease spread through affected lymph nodes. Correlation between dysphagia-associated symptoms and patient groups was calculated. 
Results: Out of 582 patients, 26% (152) experienced toxicity during a follow-up evaluation 6 months after completion of radiotherapy treatment. Patient groups identified by our approach were significantly correlated with dysphagia, feeding tube, and aspiration toxicity (p < .0005). 
Discussion: Our results suggest that structural geometry-aware characterization of affected lymph nodes can be used to better predict radiation-associated dysphagia at time of diagnosis, and better inform treatment guidelines. 
Conclusion: Our work successfully stratified a patient cohort into similar groups using a structural geometry, graph-encoding of affected lymph nodes in oropharyngeal cancer patients, that were predictive of late radiation-associated dysphagia and toxicity.

Visual representation of each cluster.

Precision association of lymphatic disease spread with radiation-associated toxicity in oropharyngeal squamous carcinomas

Cancer patients experience many symptoms throughout their cancer treatment and sometimes suffer from lasting effects post treatment. Patient-Reported Outcome (PRO) surveys provide a means for monitoring the patient’s symptoms during and after treatment. Symptom cluster (SC) research seeks to understand these symptoms and their relationships to define new treatment and disease management methods to improve patient&rsquo;s quality of life. This paper introduces association rule mining (ARM) as a novel alternative for identifying symptom clusters. We compare the results to prior research and find that while some of the SCs are similar, ARM uncovers more nuanced relationships between symptoms such as anchor symptoms that serve as connections between interference and cancer-specific symptoms. 
Keywords: Association rule mining; Symptom clusters; PRO

Symptom Severity in the (a) acute and (b) late stages.

Identifying Symptom Clusters Through Association Rule Mining

In recent years deep meta-reinforcement learning has extended the applicability of reinforcement learning (RL) algorithms: by integrating recurrent networks, trained models have the ability to quickly adapt to new unseen environments without the need for further backpropagation. These models, however, cannot adapt without having information on past rewards, and are therefore not directly applicable to a sequential decision-making setting in which multiple steps are required before observing the final reward. 
One of the main applications affected by this limitation are dynamic treatment regimes, i.e. the problem of selecting the optimal medical treatment sequence for a patient at each step, keeping into account the complete past treatment history. By expanding deep meta-reinforcement learning to handle sequential decisions, a model would be able to prescribe the optimal treatment for each patient even if the patient&rsquo;s (or physician&rsquo;s) preferences on the outcome were never encountered by the model in training. 
We propose a recursive deep meta-reinforcement learning approach which enables the model of each decision of the sequential process to learn from and adapt to unseen circumstances by recursively integrating the feedback of the models of other decisions in the process. We evaluate our approach on synthetic two-step processes with fixed transition probabilities but varying reward functions, to test the models' ability to propagate environment information from the final reward to intermediate steps. Finally, we train our model on a dataset of three-step chemo-radiotherapeutic and surgical treatment of oropharyngeal squamous cell carcinoma patients, proving our approach&rsquo;s ability to optimally handle previously unseen patient's preferences on survival and toxicity outcomes.

Main features, outcomes, and decisions of the OPC treatment sequence

Recursive meta-Reinforcement Learning for Personalized Sequential Dynamic Treatment Policies

Hearts and Minds: the Interrogations Project is an interactive virtual reality narrative artwork developed by an interdisciplinary team including humanists, social scientists, artists, and computer scientists from four different universities. The project, originally made in the CAVE2&trade; virtual reality theatre environment at the Electronic Visualization Laboratory at the University of Illinois at Chicago, attempts to extend and make accessible difficult narratives of war and torture based on actual accounts from soldiers involved. Hearts and Minds uses VR as a narrative platform
to represent a complex contemporary issue and to provide a platform for discussion and debate of military interrogation methods and their effects on detainees, soldiers, and society. We have published on this project previously in computer science and technical venues, as well as digital arts venues. Our contribution to this volume focuses on the work from an artistic and narrative perspective and on how the work functions as a digital humanities project: one which brings important documentary material addressing an important contemporary problem to contemporary new media environments for critical engagement.