Microscopy images are pervasive in biomedical research publications, where images obtained through various microscopy modalities (light, fluorescence, scanning, transmission) are often used to describe and summarize experiments and contributions. Hence, there is growing interest in automatically identifying these microscopy images’ modality and utilizing this knowledge in automated search tools. However, identifying microscopy images poses challenges due to a lack of extensive collections of labeled images. We describe and evaluate two alternative approaches to microscopy image classification. In the first approach, we progressively fine-tuned layers of ResNet models. The second approach uses shallow variants of ResNet networks, where we leverage the outputs from previous convolutional blocks. We compare these results against a Support Vector Machine (SVM)-based baseline. Our results show that fine-tuning specific layers yields better results than fine-tuning the whole model. Furthermore, shallower variants produce competitive results when compared to the entire fine-tuned model. 
Index Terms: microscopy, image classification, deep learning

Four microscopy images from the ImageCLEF dataset showcasing different modalities: (a) Electron microscopy (DMEL), (b) Transmission microscopy (DMTR), (c) Light microscopy (DMLI), and (d) Fluorescence microscopy (DMFL).

Modality-Classification of Microscopy Images Using Shallow Variants of Deep Networks

The visualization theory foundations draw on several domains, from signal processing to software design and perception. This chapter describes the landscape of visualization foundations along three aspects: a Humans aspect, a Systems aspect, and a Formal aspect, along with the domains the visualization foundations are rooted in. This chapter further provides definitions for the visualization, theory foundation, theory, model, and concept terms, and a discussion of theory granularity, from grand theories to middle-range theories and to practice theories. The chapter further discusses several
challenges related to the theory fabric of visualization that result from the diversity of our roots. The chapter ends with a discussion of possible evaluation criteria for theory components, with respect to the range of theories and models, from mathematical frameworks to guidelines and best practice advice presented in this book. 
ISBN 978-3-030-34444-3

The Fabric of Visualization

As the human body skeleton can be represented as a sparse graph, it is natural to exploit graph convolutional networks (GCNs) to model the articulated body structure for 3D human pose estimation (HPE). However, a vanilla graph convolutional layer, the building block of a GCN, only models the local relationships between each body joint and their neighbors on the skeleton graph. Some global attributes, e.g., the action of the person, can be critical to 3D HPE, especially in the case of occlusion or depth ambiguity. To address this issue, this paper introduces a new 3D HPE framework by learning global pose features in GCNs. Specifically, we add a global node to the graph and connect it to all the body joint nodes. On one hand, global features are updated by aggregating all body joint features to model the global attributes. On the other hand, the feature update of each body joint depends on not only their neighbors but also the global node. Furthermore, we propose a heterogeneous multi-task learning framework to learn the local and global features. While each local node regresses the 3D coordinate of the corresponding body joint, we force the global node to classify an action category or learn a low-dimensional pose embedding. Experimental results demonstrate the effectiveness of the proposed approach. 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Illustration of a graph convolutional network (GCN) with a global nodeufor 3D human pose estimation

Learning Global Pose Features in Graph Convolutional Networks for 3D Human Pose Estimation

Approximately 100,000 cases of Head and Neck Cancer (HNC) are diagnosed in the US annually.  Patients are increasingly likely to survive but often experience acute and long-term side effects.  Hence, great importance has been placed by clinicians on improved patient&rsquo; quality of life (QoL) and reducing symptom burden during treatment.  We introduce an interactive system which enables clinical and computational experts to visualize and assess medical data.  However, HNC patient cohort data are ofter large, multi-variate, and incomplete.  Also, anatomical and dynamic temporal components influence the outcome of therapy, and the resulting patients&rsquo; QoL.  Using novel combinations of visual encodings, our system could help clinicians to create better treatment plans.

Visualizing Symptom Development During Head and Neck Cancer Treatment

Advances in data collection in radiation therapy have led to an abundance of opportunities for applying data mining and machine learning techniques to promote new data-driven insights. In light of these advances, supporting collaboration between machine learning experts and clinicians is important for facilitating better development and adoption of these models. Although many medical use-cases rely on spatial data, where understanding and visualizing the underlying structure of the data is important, little is known about the interpretability of spatial clustering results by clinical audiences. In this work, we reflect on the design of visualizations for explaining novel approaches to clustering complex anatomical data from head and neck cancer patients. These visualizations were developed, through participatory design, for clinical audiences during a multi-year collaboration with radiation oncologists and statisticians. We distill this collaboration into a set of lessons learned for creating visual and explainable spatial clustering for clinical users. 
Keywords: Data Clustering and Aggregation, Life Sciences, Collaboration, Mixed Initiative Human-Machine Analysis, Guidelines 
YouTube Video: <a href="https://youtu.be/UM8-_MN3a2Y">https://youtu.be/UM8-_MN3a2Y</a>

(A) Lymph nodes overlaid over a diagram of the neck. (B) Example graphs of diseased nodes for 2 individual patients (datapoint representation). (C) Example consensus graph for 1 cluster (cluster representation).

Explainable Spatial Clustering: Leveraging Spatial Data in Radiation Oncology

We developed a collaborative virtual environment for architectural planning that facilitates groups of people to work together using three different interfaces. These interfaces enable users to interact with the scene with varying levels of immersion and different interaction modalities. We conducted a user study to gauge the general usability of the system and to understand how the different interfaces affect the group work. In this paper we present the architecture of the system along with its different interfaces. We also present the user study results and the insights we gained from the study. 
Keywords: group work, collaborative environments

Two users interact with the 3D interface

A Hybrid Collaborative Virtual Environment with Heterogeneous Representations for Architectural Planning

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and have demonstrated promising performance. However, since the vanilla graph convolution is performed on the one-hop neighbors of each node, it is unable to capture the long-range dependencies between body joints. They can help reduce the uncertainty caused by occlusion or depth ambiguity. To resolve this issue, we propose a high-order GCN for 3D HPE. Its core building block, termed a high-order graph convolution, aggregates features of nodes at various distances. As a result, the network can model a wide range of interactions among body joints. Furthermore, we investigate different methods to fuse those multi-order features and compare how they affect the performance. Experimental results demonstrate the effectiveness of the proposed approach.

Comparison between the vanilla graph convolution and the high-order graph convolution performed on a skeleton graph.

High-order Graph Convolutional Networks for 3D Human Pose Estimation

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and
have shown encouraging performance. One limitation of the vanilla graph convolution is that it models the relationships between neighboring nodes via a shared weight matrix. This is suboptimal for articulated body modeling as the relations between different body joints are different. The objective of this paper is to have a comprehensive and systematic study of weight sharing in GCNs for 3D HPE. We first show there are two different ways to interpret a GCN depending on whether feature transformation occurs before or after feature aggregation. These two interpretations lead to five different weight sharing methods and three more variants can be derived by decoupling the self-connections with other edges. We conduct extensive ablation study on these weight sharing methods under controlled settings and obtain new conclusions that will benefit the community.

Qualitative results of our approach on Human 3.6M

A Comprehensive Study of Weight Sharing in Graph Networks for 3D Human Pose Estimation

The VisSnippets system is designed to facilitate effective collaborative data exploration. VisSnippets leverages SAGE2 middleware that enables users to manage the display of digital media content on large displays, thereby providing collaborators with a high-resolution common workspace. Based in JavaScript, VisSnippets provides users with the flexibility to implement and/or select visualization packages and to quickly access data in the cloud. By simplifying the development process, VisSnippets removes the need to scaffold and integrate interactive visualization applications by hand. Users write reusable blocks of code called &ldquo;snippets&rdquo; for data retrieval, transformation, and visualization. By composing dataflows from the group&rsquo;s collective snippet pool, users can quickly execute and explore complementary or contrasting analyses. By giving users the ability to explore alternative scenarios, VisSnippets facilitates parallel work for collaborative data exploration leveraging large-scale displays. We describe the system, its design and implementation, and showcase its flexibility through two example applications. 
CCS Concepts: Human-centered computing, Visualization systems and tools; Collaborative and social computing systems and tools. 
Keywords: information visualization, visual data science, collaborative visual analytics

An interactive transportation dashboard for SF Muni bus data.

VisSnippets: A Web-Based System for Impromptu Collaborative Data Exploration on Large Displays

Abstract 
Purpose: Using a 200 Head and Neck cancer (HNC) patient cohort, we employ patient similarity based on tumor location, volume, and proximity to organs at risk to predict radiation-associated dysphagia (RAD) in a new patient receiving intensity-modulated
radiation therapy (IMRT). 
Material and Methods: All patients were treated using curative-intent IMRT. Anatomical features were extracted from contrast-enhanced tomography scans acquired pre-treatment. Patient similarity was computed using a topological similarity measure, which allowed for the prediction of normal tissues’ mean doses. We performed feature selection and clustering, and used the resulting groups of patients to forecast RAD. We used Logistic Regression (LG) cross-validation to assess the potential toxicity risk of these groupings. 
Results: Out of 200 patients, 34 patients were recorded as having RAD. Patient clusters were significantly correlated with RAD (p < .0001). The area under the receiver-operator curve (AUC) using pre-established, baseline features gave a predictive accuracy of .79, while the addition of our cluster labels improved accuracy to .84. 
Conclusion: Our results show that spatial information available pre-treatment can be used to robustly identify groups of RAD high-risk patients. We identify feature sets that considerably improve toxicity risk prediction beyond what is possible using baseline features. Our results also suggest that similarity-based predicted mean doses to organs can be used as valid predictors of risk to organs. 
Funding: NIH NCI-R01CA214825, NIH NCI-R01CA2251

Precision toxicity correlates of tumor spatial proximity to organs at risk in cancer patients receiving intensity-modulated radiotherapy

This paper describes results from an observational, exploratory study of visual data exploration in a large, multi-view, flexible canvas environment. Participants were provided with a set of data exploration sub-tasks associated with a local crime dataset and were instructed to pose questions to a remote mediator who would respond by generating and organizing visualizations on the large display. We observed that participants frequently posed requests to cast a net around one or several subsets of the data or a set of data attributes. They accomplished this directly and by utilizing existing views in unique ways, including by requesting to copy and pivot a group of views collectively and posing a set of parallel requests on target views expressed in one command. These observed actions depart from multi-view flexible canvas environments that typically provide interfaces in support of generating one view at a time or actions that operate on one view at a time. We describe how participants used these &ldquo;cast-a-net&rdquo; requests for tasks that spanned more than one view and describe design considerations for multi-view environments that would support the observed multi-view generation actions. 
Index Terms: Human-centered computing - Empirical studies in visualization 
This work was supported initially by NSF award IIS 1445751 and currently by NSF award CNS 1625941. 
<a href="https://doi.org/10.1111/cgf.13976">https://doi.org/10.1111/cgf.13976</a>

A participant generates 15 views of her data. In the final request, the participant references a set of three views and poses a request to copy and pivot these views collectively to four new subsets of the data. The final result is a grid of views.

Many At Once: Capturing Intentions to Create And Use Many Views At Once In Large Display Environments

Supporting exploratory visual data analysis is essential when multiple analysts collaborate using multiple devices. Yet, we still have no full understanding of how the iterative process of analysis unfolds in complex settings. In this paper, we present the results from an exploratory study where six groups of three participants performed a collaborative visual data analysis task in a complex multi-user multi-device environment. We found that the course of the analysis happens at two levels. Within each level, we observed a set of exploration patterns. We present a categorization of the analysis structure in such a complex environment and discuss the implications of device affordances on this categorization. We also discuss this categorization in relation to the current structural assumptions of exploratory visual analysis. 
Index Terms: Human-centered Computing, Empirical studies in visualization

Evaluating Strategies of Exploratory Visual Data Analysis in Multi Device Environments

Visualization, while an effective tool for identifying patterns and insights, requires expert knowledge due to challenges faced when translating user queries to visual encodings. Research has shown that using a natural language interface (NLI) is effective for these challenges because the user can simply talk to a computer capable of producing the graphs directly. In this paper, we discuss our intelligent assistant which processes speech and hand pointing gestures while also dealing with any number of visualizations on a large screen display. Evaluation of the system shows that it is capable of quickly producing visualizations. It also particularly effective
at responding to less ambiguous queries, while in certain cases can handle ambiguous or complex queries.

The System Pipeline

Intelligent Assistant for Exploring Data Visualizations

Our goal is to develop an intelligent assistant to support users explore data via visualizations. We have collected a new corpus of conversations, CHICAGO-CRIME-VIS, geared towards supporting data visualization exploration, and we have annotated it for a variety of features, including contextualized dialogue acts. In this paper, we describe our strategies and their evaluation for dialogue act classification. We highlight how thinking aloud affects interpretation of dialogue acts in our setting and how to best capture that information. A key component of our strategy is data augmentation as applied to the training data, since our corpus is inherently small. We ran experiments with the Balanced Bagging Classifier (BAGC), Condiontal Random Field (CRF), and several Long Short Term Memory (LSTM) networks, and found that all of them improved compared to the baseline (e.g., without the data augmentation pipeline). CRF outperformed the other classification algorithms, with the LSTM networks showing modest improvement, even after obtaining a performance boost from domain-trained word embeddings. This result is of note because training a CRF is far less resource-intensive than training deep learning models, hence given a similar if not better performance, traditional methods may still be preferable in order to lower resource consumption. 
Keywords: Dialogue, Corpus, Statistical and Machine Learning Methods

Contextualized actionable requests segment relevant user utterances in each turn of the conversation.

Augmenting Small Data to Classify Contextualized Dialogue Acts for Exploratory Visualization

SAGE2 is a widely known open-source environment for collaborative immersive analytics on large display walls. This paper briefly outlines our group&rsquo;s approach to evolving the SAGE2 software to support existing and future collaborative scientific exploration workflows through the application of Artificial Intelligence in Immersive Analytics. 
Author Keywords: Immersive Analytics, Large Display Walls, SAGE

Diagram of the scientific research, development and training enterprise

The Smart Amplified Group Environment

Led by EVL Director Maxine Brown, a large team of research faculty and staff from UIC&rsquo;s College of
Engineering applied for and received a Major Research Instrumentation (MRI) grant from the National Science Foundation (NSF) for approximately $1-million to acquire a deep learning and visualization infrastructure. 
The design utilizes composable infrastructure architecture from <a href="https://www.liqid.com/">Liqid</a>. The computer&rsquo;s components (traditional processor, GPU, storage, and networking) are pooled so that different applications with different workflows can be run simultaneously, with each configuring the resources it requires almost instantaneously, at any time. 
&ldquo;EVL resources are traditionally very tailored to meet the requirements of specific applications. For example, on any given day, EVL may address everything from computational fluid dynamics, ecology and neuroimaging to natural language processing, urban engineering, robotics, visualization and visual analytics,&rdquo; said Lance Long, EVL Senior Research Programmer. &ldquo;Liqid&rsquo;s composable infrastructure architecture is interesting because it allows us the flexibility to redefine cyberinfrastructure programmatically based on the applications&rsquo; requirements. This enables the EVL team to create a data science platform that presents a full-stack hardware and software solution to the challenges faced by our researchers trying to deploy and scale reproducible experiments.&rdquo;

EVL&rsquo;s COMPaaS DLV system

Case Study: UIC Utilizes Liqid Composable Infrastructure for Uneven Applications in Scientific Research

The data entry task is one of the common office tasks performed in the seated position that can be assisted by augmented reality technology. The paper introduces an ongoing project to develop an AR-based data presentation interface to assist office workers&rsquo; data entry tasks. The interface developed in this project was designed to facilitate the transition between the two steps including reading data and typing the data into a computer system using a keyboard. Usability evaluation is conducted in a controlled experiment with human subjects to examine whether the AR-based data presentation interface improves data entry performance and mitigates office workers&rsquo; physical and mental workload. 
Keywords: Office work, augmented reality, data entry, human factors. 
Index Terms: Human-centered concepts [Human computer interaction (HCI)]: Interaction paradigms-Virtual reality; Human-centered concepts [Human computer interaction (HCI)]: Visualization-Empirical studies in visualization

Interface development process.

Using Augmented Reality to Assist Seated Office Workers&rsquo; Data Entry Tasks

Clustering is the task of identifying groups of similar subjects according to certain criteria. The AJCC staging system can be thought as a clustering mechanism that groups patients based on their disease stage. This grouping drives prognosis and influences treatment. The goal of this work is to evaluate the efficacy of machine learning algorithms to cluster the patients into discriminative groups to improve prognosis for overall survival (oS) and relapse free survival (RfS) outcomes. We apply clustering over a retrospectively collected data from 644 head and neck cancer patients including both clinical and radiomic features. In order to incorporate outcome information into the clustering process and deal with the large proportion of censored samples, the feature space was scaled using the regression coefficients fitted using a proxy dependent variable, martingale residuals, instead of follow-up time. Two clusters were identified and evaluated using cross validation. The Kaplan Meier (KM) curves between the two clusters differ significantly for OS and RFS (p-value < 0.0001). Moreover, there was a relative predictive improvement when using the cluster label in addition to the clinical features compared to using only clinical features where AUC increased by 5.7% and 13.0% for OS and RFS, respectively. 
Funding: NIH NCI-R01CA214825, NIH NCI-R01CA2251

Supervised Scaling Approach. The clustering implementation here shown is consensus clustering over 1k runs of the k-median (k = 2) clustering method using different initial seeds and Manhattan distance as the dissimilarity measure.

Clustering of Largely Right-Censored Oropharyngeal Head and Neck Cancer Patients for Discriminative Groupings to Improve Outcome Prediction

Precision medicine seeks to tailor therapy to the individual patient, based on statistical correlates from patients who are similar to the one under consideration. These correlates can and should go beyond genetics, and in general, beyond tabular or array data that can be easily represented computationally and compared. For example, in many types of cancer, cancer treatment and toxicity depend in large measure on the spatial disease spread - e.g., metastasizes to regional lymph nodes in head and neck cancer. However, there is currently a lack of methodology for integrating spatial information when considering patient similarity. We present a novel modeling methodology for the comparison of cancer patients within a cohort, based on the spatial spread of the lymph nodes affected in each patient. The method uses a topological map, bigrams, and hierarchical clustering to group patients based on their similarity. We compare this approach against a nonspatial (categorical) similarity approach where patients are binned solely by their affected nodes. We present similarity results on a 582 head and neck cancer patient cohort, along with two visual abstractions for analysis of the results, and we present clinician feedback. Our novel methodology partitions a patient cohort into clinically meaningful groups more susceptible to treatment side-effects. Such spatially-aware similarity approaches can help maximize the effectiveness of each patient&rsquo;s treatment. 
Funding: NIH NCI-R01CA214825, NIH NCI-R01CA2251, and NIH NLM-R01LM012527

Topological map and graph representation.

A Spatial Neighborhood Methodology for Computing and Analyzing Lymph Node Carcinoma Similarity in Precision Medicine

SAGE2 is a user-centered platform for small groups or large distributed teams to access digital media datasets from various sources and juxtapose, share and investigate content on tiled display walls. SAGE2 builds on SAGE (Scalable Adaptive Graphics Environment), developed with prior NSF funding. SAGE2 is having a transformative effect on data exploration, visualization and collaboration, making cyberinfrastructure more accessible to end-users, both in the laboratory and in the classroom. 
<a href="https://doi.org/10.6084/m9.figshare.12837638">2020 Poster: https://doi.org/10.6084/m9.figshare.12837638</a> 
<a href="https://doi.org/10.6084/m9.figshare.12837638">2020 PPT: https://doi.org/10.6084/m9.figshare.12837695</a>

SAGE2 Scalable Amplified Group Environment (poster 2020)

COMPaaS DLV, COMposable Platform as a Service Instrument for Deep Learning & Visualization, was purchased from Liqid, Inc., with NSF funds. Its 4th-generation computer architecture, composable infrastructure, treats computer components - computer processing units (CPUs), graphical processing units (GPUs), networking, memory and storage - as pools of resources. It is: 
<ul><li>Highly flexible. Each application defines the resources it needs, and the operating system composes, or combines, resources on the fly.</li> 
<li>Scalable. As more infrastructure is added, it’s auto-integrated with the existing infrastructure and becomes part of the pool of capacity.</li> 
<li>High-throughput. System components are interconnected with a high-speed internal fabric, enabling Big Data to quickly move among CPU, GPU, memory and storage at optimum speeds with little to no bottlenecks.</ul></li> 
Funded in part by NSF award #CNS-1828265.