This collaborative research project led by the colleagues at the University of Delaware aims to develop and advance tools for using image-data appearing in scientific publications, in addition to text, in order to support beneficial, targeted access to the biomedical literature. 
UIC's principal investigator, Computer Science associate professor, G. Elisabeta Marai will be responsible for the image analysis algorithms to distinguish between gel and plate images, the image analysis algorithms to distinguish between Electronic Microscopy vs Fluorescent Microscopy vs Light Microscopy and X-Rays, and the design of the visual interface to present results to end users. She will lead the design, development and implementation of the feature extraction and selection tools and techniques related to this research project.

Incorporating Image-based Features into Biomedical Document Classification

Authors: Hesham Elhalawani, Timothy A Lin, Stefania Volpe, Abdallah S.R. Mohamed, Aubrey L. White, James Zafereo, Andrew Wong, Joel E. Berends, Shady Abohashem, Bowman Williams, Jeremy M. Aymard, Aasheesh Kanwar, Subha Perni, Crosby D. Rock, Luke Cooksey, Shauna Campbell, Pei Yang, Khanh Nguyen, Rachel Ger, Carlos Eduardo Cardenas, Xenia Fave, Carlo Sansone, Gabriele Piantadosi, Stefano Marrone, Rongjie Liu, Chao Huang, Kaixian Yu, Tenfei Li, Yang Yu, Youyi Zhang, Hongtu Zhu, Jeffrey S. Morris, Veerabhadran Baladandayuthapani, John W. Shumway, Alokananda Ghosh, Andrei Pöhlmann, Hady Ahmady Phoulady, Vibhas Goyal, Guadalupe Canahuate, G. Elisabeta Marai, David Vock, Stephen Y. Lai, Dennis S. Mackin, Laurence E. Court, John Freymann, Keyvan Farahani, Jayashree Kalpathy-Cramer, Clifton D Fuller 
Abstract: Radiomics leverages existing image datasets to provide non-visible data extraction via image post-processing, with the aim of identifying prognostic, and predictive imaging features at a sub-region of interest level. However, the application of radiomics is hampered by several challenges such as lack of image acquisition/analysis method standardization, impeding generalizability. As of yet, radiomics remains intriguing, but not clinically validated. We aimed to test the feasibility of a non-custom-constructed platform for disseminating existing large, standardized databases across institutions for promoting radiomics studies. Hence, University of Texas MD Anderson Cancer Center organized two public radiomics challenges in head and neck radiation oncology domain. This was done in conjunction with MICCAI 2016 satellite symposium using Kaggle-in-Class, a machine-learning and predictive analytics platform. We drew on clinical data matched to radiomics data derived from diagnostic contrast-enhanced computed tomography (CECT) images in a dataset of 315 patients with oropharyngeal cancer. Contestants
were tasked to develop models for (i) classifying patients according to their human papillomavirus status, or (ii) predicting local tumor recurrence, following radiotherapy. Data were split into training, and test sets. Seventeen teams from various professional domains participated in one or both of the challenges. This review paper was based on the contestants’ feedback; provided by 8 contestants only (47%). Six contestants (75%) incorporated extracted radiomics features into their predictive model building, either alone (n = 5; 62.5%), as was the case with the winner of the “HPV” challenge, or in conjunction with matched clinical attributes (n = 2; 25%). Only 23% of contestants, notably, including the winner of the “local recurrence” challenge, built their model relying solely on clinical data. In addition to the value of the integration of machine learning into clinical decision-making, our experience sheds light on challenges in sharing and directing existing datasets toward clinical applications of radiomics, including hyperdimensionality of the clinical/imaging data attributes. Our experience may help guide researchers to create a framework for sharing and reuse of already published data that we believe will ultimately accelerate the pace of clinical applications of radiomics; both in challenge or clinical settings. 
Keywords: machine learning, radiomics challenge, radiation oncology, head and neck, big data

Workflow of radiomics challenges.

Machine Learning Applications in Head and Neck Radiation Oncology: Lessons from Open-Source Radiomics Challenges

MS Graduate, 2019

Ganesh Jagadeesan

BS Graduate, 2023 Research Experience Undergraduate (REU) 2019-2023

Zhongyi Chen

In today&rsquo;s Big Data era, data scientists require new computational instruments in order to quickly analyze large-scale datasets using complex codes and quicken the rate of scientific progress. While Federally-funded computer resources, from supercomputers to clouds, are beneficial, they are often limiting - particularly for deep learning and visualization - as they have few Graphics Processing Units (GPUs). GPUs are at the center of modern high-performance computing and artificial intelligence, efficiently performing mathematical operations that can be massively parallelized, speeding up codes used for deep learning, visualization and image processing, more so than general-purpose microprocessors, or Central Processing Units (CPUs). The University of Illinois at Chicago is acquiring a much-in-demand GPU-based instrument, COMPaaS DLV - COMposable Platform as a Service Instrument for Deep Learning &amp; Visualization, based on composable infrastructure, an advanced architecture that disaggregates the underlying compute, storage, and network resources for scaling needs, but operates as a single cohesive infrastructure for management and workload purposes. We are experimenting with a small system and learning a great deal about composability, and we believe COMPaaS DLV users will benefit from the varied workflow that composable infrastructure allows. 
Keywords: distributed systems, testbed implementation &amp; deployment, composable infrastructure, deep learning, visualization

COMPaaS DLV: Composable Infrastructure for Deep Learning in an Academic Research Environment

Maxine Brown (UIC PI), Andrew Johnson (UIC co-PI), Luc Renambot (UIC co-PI) 
Jason Leigh (University of Hawaii Manoa PI) Christopher North (Virginia Polytechnic PI) 
NSF Award Number: 2003800 
Award Period of Performance: 05/15/2020 - 04/30/2025 
Total Amount Obligated: $2,250,000 
UIC Computer Science Electronic Visualization Lab (EVL) Director Maxine Brown, PI, co-PIs UIC CS Professor Andy Johnson and UIC CS Research Associate Professor Luc Renambot, and Senior Personnel EVL Senior Research Programmer Lance Long are awarded a $2.25 million dollar National Science Foundation grant entitled &ldquo;Collaborative Research: CSSI Frameworks: SAGE3: Smart Amplified Group Environment for Harnessing the Data Revolution.&rdquo; The grant runs from May 2020 to April 2025. It&rsquo;s part of an overall $5 million project, that also includes a team from University of Hawaii Manoa led by Jason Leigh and a team from Virginia Polytechnic led by Christopher North. 
ABSTRACT 
The Big Data revolution necessitates the use of sophisticated tools such as Artificial Intelligence (AI) and Data Visualization to harness the sheer volume, velocity and variety of datasets that are becoming the norm. However, it is the research community that must make sense of the data being amassed, so cyberinfrastructure must extend to people. SAGE3 (Smart Amplified Group Environment) puts the &ldquo;human in the loop&rdquo; by providing scientists with an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help them access, share, explore and analyze their data, come to conclusions, and make decisions with greater speed, accuracy, comprehension and confidence. 
SAGE3 builds upon the NSF-funded first-generation SAGE (Scalable Adaptive Graphics Environment) and second-generation SAGE2 (Scalable Amplified Group Environment), which are today&rsquo;s de facto operating systems to manage Big Data on tiled display walls. Its development focuses on two fundamental components: AI-enhanced &ldquo;smart&rdquo; services and advanced computing resource orchestration to support reproducible work models for secure collaborative work. SAGE3 amplifies user productivity, providing them with commercially available and open-source AI solutions, which autonomously and transparently analyze data while continually learning and improving through user interactions. SAGE3 makes AI technologies broadly accessible, not just a privilege for the technically savvy. SAGE3 further democratizes AI by using Data Visualization to help interpret and explain AI models so users better understand how AI came to its decisions, which engenders user trust and can help identify potentially prejudiced or biased models. 
SAGE3 augments every step of the scientific discovery enterprise - from quickly summarizing large data, to finding trends and similarities or anomalies among one or more linked datasets, to communicating findings to scientists, public policy and government officials, and the general public, to educating the next-generation workforce. Ultimately, it is the scientists and future scientists who must Harness the Big Data revolution to solve the nation&rsquo;s grand challenge problems that will benefit society as a whole - from studying the diversity of life on Earth, to understanding the Earth and its systems from satellite imagery of its poles, to developing response scenarios for natural disasters such as landslides and pandemics that impact the citizens and economies of the world.

UIC Computer Science/EVL awarded $2.25 Million from NSF for SAGE3

UIC PI: Luc Renambot 
University of Hawaii Manoa PI: Jason Leigh 
Virginia Polytechnic PI: Chirstopher North 
NSF Award Number: 2003800 
Award Period of Performance: 05/15/2020 - 04/30/2026 
Total Amount Obligated: $2,250,000 
The Big Data revolution necessitates the use of sophisticated tools such as Artificial Intelligence (AI) and Data Visualization to harness the sheer volume, velocity and variety of datasets that are becoming the norm. However, it is the research community that must make sense of the data being amassed, so cyberinfrastructure must extend to people. SAGE3 (Smart Amplified Group Environment) puts the &ldquo;human in the loop&rdquo; by providing scientists with an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help them access, share, explore and analyze their data, come to conclusions, and make decisions with greater speed, accuracy, comprehension and confidence. 
SAGE3 builds upon the NSF-funded first-generation SAGE (Scalable Adaptive Graphics Environment) and second-generation SAGE2 (Scalable Amplified Group Environment), which are today&rsquo;s de facto operating systems to manage Big Data on tiled display walls. Its development focuses on two fundamental components: AI-enhanced &ldquo;smart&rdquo; services and advanced computing resource orchestration to support reproducible work models for secure collaborative work. SAGE3 amplifies user productivity, providing them with commercially available and open-source AI solutions, which autonomously and transparently analyze data while continually learning and improving through user interactions. SAGE3 makes AI technologies broadly accessible, not just a privilege for the technically savvy. SAGE3 further democratizes AI by using Data Visualization to help interpret and explain AI models so users better understand how AI came to its decisions, which engenders user trust and can help identify potentially prejudiced or biased models. 
SAGE3 augments every step of the scientific discovery enterprise - from quickly summarizing large data, to finding trends and similarities or anomalies among one or more linked datasets, to communicating findings to scientists, public policy and government officials, and the general public, to educating the next-generation workforce. Ultimately, it is the scientists and future scientists who must Harness the Big Data revolution to solve the nation&rsquo;s grand challenge problems that will benefit society as a whole - from studying the diversity of life on Earth, to understanding the Earth and its systems from satellite imagery of its poles, to developing response scenarios for natural disasters such as landslides and pandemics that impact the citizens and economies of the world. 
<a href="https://youtu.be/kvNSLGcs5KE">View a video of SAGE3 being demonstrated</a> on EVL&rsquo;s YouTube channel.

SAGE3: Smart Amplified Group Environment for Harnessing the Data Revolution

Clustering is the task of identifying groups of similar subjects according to certain criteria. The AJCC staging system can be thought as a clustering mechanism that groups patients based on their disease stage. This grouping drives prognosis and influences treatment. The goal of this work is to evaluate the efficacy of machine learning algorithms to cluster the patients into discriminative groups to improve prognosis for overall survival (oS) and relapse free survival (RfS) outcomes. We apply clustering over a retrospectively collected data from 644 head and neck cancer patients including both clinical and radiomic features. In order to incorporate outcome information into the clustering process and deal with the large proportion of censored samples, the feature space was scaled using the regression coefficients fitted using a proxy dependent variable, martingale residuals, instead of follow-up time. Two clusters were identified and evaluated using cross validation. The Kaplan Meier (KM) curves between the two clusters differ significantly for OS and RFS (p-value < 0.0001). Moreover, there was a relative predictive improvement when using the cluster label in addition to the clinical features compared to using only clinical features where AUC increased by 5.7% and 13.0% for OS and RFS, respectively. 
Funding: NIH NCI-R01CA214825, NIH NCI-R01CA2251

Supervised Scaling Approach. The clustering implementation here shown is consensus clustering over 1k runs of the k-median (k = 2) clustering method using different initial seeds and Manhattan distance as the dissimilarity measure.

Clustering of Largely Right-Censored Oropharyngeal Head and Neck Cancer Patients for Discriminative Groupings to Improve Outcome Prediction

PhD Graduate Student

Mir Masood Ali

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and
have shown encouraging performance. One limitation of the vanilla graph convolution is that it models the relationships between neighboring nodes via a shared weight matrix. This is suboptimal for articulated body modeling as the relations between different body joints are different. The objective of this paper is to have a comprehensive and systematic study of weight sharing in GCNs for 3D HPE. We first show there are two different ways to interpret a GCN depending on whether feature transformation occurs before or after feature aggregation. These two interpretations lead to five different weight sharing methods and three more variants can be derived by decoupling the self-connections with other edges. We conduct extensive ablation study on these weight sharing methods under controlled settings and obtain new conclusions that will benefit the community.

Qualitative results of our approach on Human 3.6M

A Comprehensive Study of Weight Sharing in Graph Networks for 3D Human Pose Estimation

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and have demonstrated promising performance. However, since the vanilla graph convolution is performed on the one-hop neighbors of each node, it is unable to capture the long-range dependencies between body joints. They can help reduce the uncertainty caused by occlusion or depth ambiguity. To resolve this issue, we propose a high-order GCN for 3D HPE. Its core building block, termed a high-order graph convolution, aggregates features of nodes at various distances. As a result, the network can model a wide range of interactions among body joints. Furthermore, we investigate different methods to fuse those multi-order features and compare how they affect the performance. Experimental results demonstrate the effectiveness of the proposed approach.

Comparison between the vanilla graph convolution and the high-order graph convolution performed on a skeleton graph.

High-order Graph Convolutional Networks for 3D Human Pose Estimation

October 19, 2020 - - Fabio Miranda joins the EVL faculty as an assistant professor with the Department of Computer Science. Miranda interests include developing techniques that allow for the interactive visual analysis of large-scale data, combining methods from visualization, data management, machine learning, and computer graphics. Of particular concern is how visual data analytics can help address different problems cities face by integrating data on different resolutions and from different sources. 
Miranda&rsquo;s research in data science, human-computer interaction and visualization include: visual data analysis, data structures and indexing techniques for big data visualization, urban analytics, harnessing data sets of different types (images, land use information, 3D building geometry, etc.) to address concrete city problems, and machine learning. He has worked closely with domain experts from different fields, from urban planning to occupational therapy, and the outcome of these collaborations include not only research published in leading visualization, database, HCI, and AI venues, but also systems that were made available to experts in academia, industry, and government agencies. 
Miranda received his Ph.D. from New York University (2018).

Fabio Miranda Joins EVL Faculty

October 1 - - EVL Director of Research Andrew Johnson has been named Interim Director of EVL following Maxine Brown&rsquo;s retirement from UIC. Johnson now assumes the responsibility of heading EVL - overseeing EVL&rsquo;s overall mission in education, research, development, and technology transfer. 
For over 23 years, Johnson has been a valued EVL faculty member (Professor of Computer Science) and researcher who has focused upon interaction and collaboration using advanced visualization displays and the application of those displays to enhance discovery and learning. 
His areas of expertise include visualization, virtual and augmented reality, natural language processing, visual analytics, and high-performance networking. 
Over the years, Johnson has been instrumental in the development of EVL instrumentation and systems, including the Continuum smart environment, CAVE2&trade;, LambdaVision high-resolution tiled displays, LambdaTable tabletop touch system, GeoWall portable geoscience display environment, PARIS augmented reality and ImmersaDesk virtual reality systems. He has overseen the development of numerous visualization and virtual reality applications across diverse domain disciplines from science and engineering to the humanities and art. Johnson is a highly regarded mentor and educator to EVL&rsquo;s graduate students and undergraduates alike. 
Johnson retains his position as Director of Research while leading EVL&rsquo;s future initiatives and innovations as Interim Director.

Andrew Johnson Named Interim Director, EVL

For the Supercomputing (SC20) Conference this year, a research community partnership was formed to design an international networking testbed using existing facilities, with several international extensions. Demonstrations and experiments were conducted prior to the conference with results discussed during a SC20 virtual forum. &ldquo;The Forum For Experiments and Demonstrations Showcasing Innovations In Large-Scale, Data-Intensive-Science Networking&rdquo; was organized by Research Consortium for Data-Intensive Science Networking to communicate: a) the motivation behind these experiments and demonstrations, b) their design, c) technologies used, d) results and e) future research in these areas. 
EVL senior research programmer, Lance Long participated to discuss EVL&rsquo;s research in Composable Infrastructure, specifically MRI: Acquisition of a Composable Platform as a Service Instrument for Deep Learning & Visualization (COMPaaS DLV) funded under NSF Award #CNS-1828265. Items discussed included the benefits of composable infrastructure, implementation for deep learning in an academic research environment, operating systems, dynamic workflows, orchestration middleware, current use cases and future work. 
Other participants included: Ciena Research Lab, NASA Goddard Space Flight Center (GSFC), Naval Research Laboratory, iCAIR/Northwestern, StarLight, UCSD, FNAL, CalTech, SURFnet.

Composable Infrastructure for Data Science

Lance Long Presents EVL&rsquo;s Composable Infrastructure Research at SC20

Advances in data collection in radiation therapy have led to an abundance of opportunities for applying data mining and machine learning techniques to promote new data-driven insights. In light of these advances, supporting collaboration between machine learning experts and clinicians is important for facilitating better development and adoption of these models. Although many medical use-cases rely on spatial data, where understanding and visualizing the underlying structure of the data is important, little is known about the interpretability of spatial clustering results by clinical audiences. In this work, we reflect on the design of visualizations for explaining novel approaches to clustering complex anatomical data from head and neck cancer patients. These visualizations were developed, through participatory design, for clinical audiences during a multi-year collaboration with radiation oncologists and statisticians. We distill this collaboration into a set of lessons learned for creating visual and explainable spatial clustering for clinical users. 
Keywords: Data Clustering and Aggregation, Life Sciences, Collaboration, Mixed Initiative Human-Machine Analysis, Guidelines 
YouTube Video: <a href="https://youtu.be/UM8-_MN3a2Y">https://youtu.be/UM8-_MN3a2Y</a>

(A) Lymph nodes overlaid over a diagram of the neck. (B) Example graphs of diseased nodes for 2 individual patients (datapoint representation). (C) Example consensus graph for 1 cluster (cluster representation).

Explainable Spatial Clustering: Leveraging Spatial Data in Radiation Oncology

EVL faculty and Computer Science associate professor G. Elisabeta Marai (MPI) along with H. Shatkay (MPI) at the University of Delaware have been awarded supplemental funding for COVID-19 related research to their NIH research project entitled &ldquo;Uncovering Clinical Evidence in COVID-19 Publications: An Integrated Search via Text &amp; Images&rdquo;. This supplement expands the work done under current R01 funding &ldquo;Incorporating Image-based Features into Biomedical Document Classification&rdquo;, to support discovery within the CORD-19 collection -- a large, continually updated collection of more than 60,000 scientific articles pertaining to COVID-19. This supplement will create a comprehensive platform for retrieving and displaying well-focused, need-targeted medical information within COVID-19 biomedical publications. Our approach is unique compared to other text-mining methods because it leverages the image content in the dataset. 
Funding: NIH R01 NLM Covid-19 supplement to Parent Grant: 1R01LM012527 (&ldquo;Incorporating Image-based Features into Biomedical Document Classification&rdquo;)

EVL Faculty G.E. Marai Awarded NIH COVID-19 Supplement

National Institutes of Health: National Cancer Institute has awarded UIC funding to continue research in head and neck cancer therapy. UIC PI, EVL faculty and Computer Science associate professor Liz Marai in conjunction with co-PI Xinhua Zhang are joined by collaborators from the University of Texas MD Anderson Cancer Center (Dave Fuller, PI) and the University of Iowa (Guadalupe Canahuate, PI) to develop a longitudinal decision support tool to assist clinicians in selecting treatments that balance survival and side-effects responses to cancer therapy. The project will lead to better insights into the biological and therapeutic factors related to cancer survival and toxicity, allowing personalization of care at the individual level. 
Title: Longitudinal Spatial-Nonspatial Decision Support for Competing Outcomes in Head and Neck Cancer Therapy 
PI: G. Elisabeta Marai, Co-PI: Xinhua Zhang 
Sponsor: National Institutes of Health: National Cancer Institute 
Dates: March 1, 2021 - Feb 28, 2026 
Amount: $2,797,868 
Grant Number: 1 R01 CA258827-01

NIH Awards &ldquo;Longitudinal Spatial-Nonspatial Decision Support for Competing Outcomes in Head and Neck Cancer Therapy&rdquo;

Cancers that depend on the spatial location of the disease affect all ethnicities, and are diagnosed each year in large numbers in the United States, leading to large, rich repositories of patient data. This project develops a longitudinal decision support tool to assist clinicians in selecting treatments that balance survival and side-effects responses to cancer therapy. The project will lead to better insights into the biological and therapeutic factors related to cancer survival and toxicity, allowing personalization of care at the individual level. 
This research is sponsored by the National Institutes of Health: National Cancer Institute, and led by EVL faculty and Computer Science associate professor Liz Marai (PI) and co-PI Xinhua Zhang, along with collaborators Guadalupe Canahuate at University of Iowa, and Dave Fuller at University of Texas, MD Anderson Cancer Center.

Longitudinal Spatial-Nonspatial Decision Support for Competing Outcomes in Head and Neck Cancer Therapy

EVL faculty and Computer Science assistant professor Wei Tang has been working with Motorola to use computer vision to improve cell phone photos taken in low-light conditions. Dr. Tang&rsquo;s research focuses on the integration of deep neural networks with conventional image processing techniques to address complex computer vision problems. Specifically, Tang and his graduate student researchers are working on methods to deblur low-light cell phone images, to return a crisp, clean version of the picture. The process includes providing the neural network a series of images - both blurry and clean - to create a loss function, or learning objective, to identify the preferred images. The resulting deblurred images are then provided to the Motorola team to brighten the low-light photos for highly improved results. 
Tang&rsquo;s research interests in computer vision, pattern recognition and machine learning can also be applied to augmented reality and intelligent robotics applications. Stay tuned for updates in Tang's research and collaborations. 
Read the related <a href="https://cs.uic.edu/news-stories/wei-tang-using-computer-vision-to-improve-cell-phone-photos-taken-in-low-light-conditions/">CS e-newsletter</a> article by Andrea Poet for more detail.

Wei Tang&rsquo;s Machine Learning Researh Featured in UIC Computer Science e-News

EVL faculty and computer science (CS) associate professor Liz Marai in collaboration with co-investigator, CS assistant professor Xinhua Zhang are developing a computational artificial intelligence system to serve as a decision support tool for oncologists prescribing treatment for head and neck cancer. The research entitled &ldquo;Longitudinal Spatial-Nonspatial Decision Support for Competing Outcomes in Head and Neck Cancer Therapy&rdquo; is supported by a $2.8 million grant from the National Institutes of Health NIH grant #R01CA258827 with co-investigators at the University of Texas MD Anderson Cancer Center and the University of Iowa. The goal is to develop a scalable system based on real cancer patient data that can help guide physicians in how to treat specific patients. 
Marai and Zhang are using the computational resources at Electronic Visualization Laboratory to develop their cancer treatment AI. Guadalupe Canahuate of the University of Iowa and Dr. David Fuller of the University of Texas are co-investigators on the NIH grant. 
See the full press release on <a href="https://today.uic.edu/maximizing-cancer-survival-minimizing-treatment-side-effects-with-ai">UIC Today</a> by Jacqueline Carey.

Collaborators examine a large dataset in EVL&rsquo;s CAVE2

Maximizing cancer survival, minimizing treatment side effects with AI

Visualization, while an effective tool for identifying patterns and insights, requires expert knowledge due to challenges faced when translating user queries to visual encodings. Research has shown that using a natural language interface (NLI) is effective for these challenges because the user can simply talk to a computer capable of producing the graphs directly. In this paper, we discuss our intelligent assistant which processes speech and hand pointing gestures while also dealing with any number of visualizations on a large screen display. Evaluation of the system shows that it is capable of quickly producing visualizations. It also particularly effective
at responding to less ambiguous queries, while in certain cases can handle ambiguous or complex queries.

The System Pipeline

Intelligent Assistant for Exploring Data Visualizations

Our goal is to develop an intelligent assistant to support users explore data via visualizations. We have collected a new corpus of conversations, CHICAGO-CRIME-VIS, geared towards supporting data visualization exploration, and we have annotated it for a variety of features, including contextualized dialogue acts. In this paper, we describe our strategies and their evaluation for dialogue act classification. We highlight how thinking aloud affects interpretation of dialogue acts in our setting and how to best capture that information. A key component of our strategy is data augmentation as applied to the training data, since our corpus is inherently small. We ran experiments with the Balanced Bagging Classifier (BAGC), Condiontal Random Field (CRF), and several Long Short Term Memory (LSTM) networks, and found that all of them improved compared to the baseline (e.g., without the data augmentation pipeline). CRF outperformed the other classification algorithms, with the LSTM networks showing modest improvement, even after obtaining a performance boost from domain-trained word embeddings. This result is of note because training a CRF is far less resource-intensive than training deep learning models, hence given a similar if not better performance, traditional methods may still be preferable in order to lower resource consumption. 
Keywords: Dialogue, Corpus, Statistical and Machine Learning Methods

Contextualized actionable requests segment relevant user utterances in each turn of the conversation.

Augmenting Small Data to Classify Contextualized Dialogue Acts for Exploratory Visualization

As the human body skeleton can be represented as a sparse graph, it is natural to exploit graph convolutional networks (GCNs) to model the articulated body structure for 3D human pose estimation (HPE). However, a vanilla graph convolutional layer, the building block of a GCN, only models the local relationships between each body joint and their neighbors on the skeleton graph. Some global attributes, e.g., the action of the person, can be critical to 3D HPE, especially in the case of occlusion or depth ambiguity. To address this issue, this paper introduces a new 3D HPE framework by learning global pose features in GCNs. Specifically, we add a global node to the graph and connect it to all the body joint nodes. On one hand, global features are updated by aggregating all body joint features to model the global attributes. On the other hand, the feature update of each body joint depends on not only their neighbors but also the global node. Furthermore, we propose a heterogeneous multi-task learning framework to learn the local and global features. While each local node regresses the 3D coordinate of the corresponding body joint, we force the global node to classify an action category or learn a low-dimensional pose embedding. Experimental results demonstrate the effectiveness of the proposed approach. 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Illustration of a graph convolutional network (GCN) with a global nodeufor 3D human pose estimation

Learning Global Pose Features in Graph Convolutional Networks for 3D Human Pose Estimation

Dependency relations among visual entities are ubiquitous because both objects and scenes are highly structured. They provide prior knowledge about the real world that can help improve the generalization ability of deep learning approaches. Different from contextual reasoning which focuses on feature aggregation in the spatial domain, visual dependency reasoning explicitly models the dependency relations among visual entities. In this paper, we introduce a novel network architecture, termed the dependency network or DependencyNet, for semantic segmentation. It unifies dependency reasoning at three semantic levels. Intra-class reasoning decouples the representations of different object categories and updates them separately based on the internal object structures. Inter-class reasoning then performs spatial and semantic reasoning based on the dependency relations among different object categories. We will have an in-depth investigation on how to discover the dependency graph from the training annotations. Global dependency reasoning further refines the representations of each object category based on the global scene information. Extensive ablative studies with a controlled model size and the same network depth show that each individual dependency reasoning component benefits semantic segmentation and they together significantly improve the base network. Experimental results on two benchmark datasets show the DependencyNet achieves comparable performance to the recent states of the art. 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Visual dependence relations are ubiquity since both ob-jects and scenes are highly structured. They provide prior knowl-edge about the real world that can be used to improve the gener-alization ability of a learning model.

Exploit Visual Dependency Relations for Semantic Segmentation

PhD Student

Kazi Shahrukh Omar

Construction activities for the new $117.8 million, 135,000-square-foot Computer Design Research and Learning Center to be located at 900 W. Taylor St. is underway. The CDRLC will support the Department of Computer Science&rsquo;s growing student enrollment with 16,000 square feet of classroom space for collaborative teaching and learning environments, as well as 35 labs for faculty research activities in the computer science and engineering disciplines. 
The CDRLC will be the new home for the Electronic Visualization Laboratory featuring the latest technological and computing innovations to support research in advanced visualization, networking and collaborative environments. 
Read more about the CDRLC groundbreaking in the <a href="https://today.uic.edu/uic-breaks-ground-on-new-computer-design-research-and-learning-center">July 15th issue of UIC Today.</a> 
See a live webcam of the building&rsquo;s progress via the <a href="https://webcampub.multivista.com/index.cfm?fuseaction=aPublicWebcam.embed&WebcamPublicEmbedUID=5D4DCF05-AB2B-4770-A090-D79BE467F958">Multivista Construction Documentation webcam.</a>

New Computer Design Research and Learning Center Breaks Ground

The latest developments of machine learning (ML) and deep learning (DL) algorithms have paved the way to effectively analyze the atomic structure of chemically-complex materials. In this work, we present a DL model built upon a fully convolutional neural network (FCN) to resolve the random elements distribution of complex PtNiPdCoFe high-entropy alloys (HEAs) represented in the scanning transmission electron microscopy (STEM) images at atomic resolution. The objective of the proposed neural network is to learn through semantic seg-mentation the non-linear correlations between the pixels&rsquo; intensities of STEM images and the number of atoms of different constituent elements in the atomic columns (i.e., column heights) in the HEA&rsquo;s structure. We demonstrate that our DL model is capable of correctly estimating the column heights or with an error up to 1 atom for the majority of the columns in the HEA structures represented in the simulated STEM images used to train and test the network. This establishes a sufficiently high level of confidence in the estimation of the element distribution in experimental images. The predicted distributions in different STEM images of nanoparticles reveal inhomogeneous fluctuations with local aggregations in the elemental atomic fractions within the columns. The most pronounced aggregation is displayed by Pt, which is the largest and most electronegative element in the synthesized HEA material. The proposed DL method is beneficial for an in-depth characterization of the structural properties of HEAs and multielement 3D materials in general. 
Keywords: Deep learning, High entropy alloys, STEM images, Column heights, Elements distribution

Example of atomic model of HEA characterized by a nearly equiatomic composition of Pt, Ni, Pd, Co and Fe.

Deep learning for mapping element distribution of high-entropy alloys in scanning transmission electron microscopy images

The National Institutes of Health: National Cancer Institute has awarded Liz Marai and collaborators at the University of Texas M.D. Anderson Cancer Center a Research Supplement to Promote Diversity in Health-Related Research Programs, for federal award R01CA258827, &ldquo;Longitudinal Spatial-Nonspatial Decision Support for Competing Outcomes in Head and Neck Cancer Therapy&rdquo; to run 2021 -2024. The award will support the interdisciplinary training of Dr. Rachel Glenn, an individual from a disadvantaged background (as defined by PA-21-071), under the direction of PIs Marai (UIC) and Fuller (U of Texas).

G. Elisabeta Marai Awarded NIH Diversity Supplement

In recent years deep meta-reinforcement learning has extended the applicability of reinforcement learning (RL) algorithms: by integrating recurrent networks, trained models have the ability to quickly adapt to new unseen environments without the need for further backpropagation. These models, however, cannot adapt without having information on past rewards, and are therefore not directly applicable to a sequential decision-making setting in which multiple steps are required before observing the final reward. 
One of the main applications affected by this limitation are dynamic treatment regimes, i.e. the problem of selecting the optimal medical treatment sequence for a patient at each step, keeping into account the complete past treatment history. By expanding deep meta-reinforcement learning to handle sequential decisions, a model would be able to prescribe the optimal treatment for each patient even if the patient&rsquo;s (or physician&rsquo;s) preferences on the outcome were never encountered by the model in training. 
We propose a recursive deep meta-reinforcement learning approach which enables the model of each decision of the sequential process to learn from and adapt to unseen circumstances by recursively integrating the feedback of the models of other decisions in the process. We evaluate our approach on synthetic two-step processes with fixed transition probabilities but varying reward functions, to test the models' ability to propagate environment information from the final reward to intermediate steps. Finally, we train our model on a dataset of three-step chemo-radiotherapeutic and surgical treatment of oropharyngeal squamous cell carcinoma patients, proving our approach&rsquo;s ability to optimally handle previously unseen patient's preferences on survival and toxicity outcomes.

Main features, outcomes, and decisions of the OPC treatment sequence

Recursive meta-Reinforcement Learning for Personalized Sequential Dynamic Treatment Policies

Former EVL student and graduate research assistant Elisa Tardini is recognized as a recipient of the Outstanding Thesis/Dissertation Award from the Graduate College at the University of Illinois at Chicago for her thesis entitled &ldquo;Recursive deep meta-reinforcement learning for personalized sequential dynamic treatment policies.&rdquo; This research focuses on a recursive deep meta-reinforcement learning approach to develop a model to prescribe an optimal treatment for patients.

2021 Outstanding Thesis/Dissertation Award to Elisa Tardini

Pose guided person image generation means to generate a photo-realistic person image conditioned on an input person image and a desired pose. This task requires spatial manipulation of the source image according to the target pose. However, convolutional neural networks (CNNs) are inherently limited to geometric transformations due to the fixed geometric structures in their building modules, i.e., convolution, pooling and unpooling, which cannot handle large motion and occlusions caused by large pose transform. This paper introduces a novel two-stream context-aware appearance transfer network to address these challenges. It is a three-stage architecture consisting of a source stream and a target stream. Each stage features an appearance transfer module, a multi-scale context module and two-stream feature fusion modules. The appearance transfer module handles large motion by finding the dense correspondence between the two-stream feature maps and then transferring the appearance information from the source stream to the target stream. The multi-scale context module handles occlusion via contextual modeling, which is achieved by atrous convolutions of different sampling rates. Both quantitative and qualitative results indicate the proposed network can effectively handle challenging cases of large pose transform while retaining the appearance details. Compared with state-of-the-art approaches, it achieves comparable or superior performance using much fewer parameters while being significantly faster. 
Keywords: Person image generation, Appearance transfer, Multi-scale context, Efficient image generation 
Relevant Funding: The COMPaaS DLV project (NSF award CNS-1828265) 
<a href="https://doi.org/10.1016/j.patcog.2021.108451">https://doi.org/10.1016/j.patcog.2021.108451</a>

Exploiting appearance transfer and multi-scale context for efficient person image generation

MS Student

Veronica Grosso

Figures within biomedical articles present essential evidence of the relevance of a publication in a curation workflow. In particular, visual cues of the image modality or experimental methods can help expert curators identify relevant papers from an increasing number of publications. Automating the identification of these content-bearing images can thus be helpful in computer-assisted curation. However, the paucity of labeled datasets and the specialized training required to label such images hinder the development of such tools. To address this problem, we present the design of ANIMO, a labeling system that integrates extraction and segmentation tools to ease the annotation burden. We first introduce two taxonomies of image modalities and experimental methods, derived in collaboration with curators. On the back-end of the system, we process batches of documents and create a labeling task per document. At the front-end, expert curators can access these tasks through a web interface and access the article of interest. We describe the evaluation of this system by a group of biocurators, and the human factor lessons learned from this interdisciplinary experience. 
doi: 10.1109/BIBM52615.2021.9669898

Front-end interface for labeling a document showing a taxonomy adapted for the CORD-19 dataset.

ANIMO: Annotation of Biomed Image Modalities

Support vector regression (SVR) is particularly beneficial when the outcome and predictors are nonlinearly related. However, when many covariates are available, the method&rsquo;s flexibility can lead to overfitting and an overall loss in predictive accuracy. To overcome this drawback, we develop a feature selection method for SVR based on a genetic algorithm that iteratively searches across potential subsets of covariates to find those that yield the best performance according to a user-defined fitness function. We evaluate the performance of our feature selection method for SVR, comparing it to alternate methods including LASSO and random forest, in a simulation study. We find that our method yields higher predictive accuracy than SVR without feature selection. Our method outperforms LASSO when the relationship between covariates and outcome is nonlinear. Random forest performs equivalently to our method in some scenarios, but more poorly when covariates are correlated. We apply our method to predict donor kidney function 1 year after transplant using data from the United Network for Organ Sharing national registry.

Results of simulation 1 setting with 300 total covariates X, 15 of them truly associated with the outcome Y.

Feature Selection for Support Vector Regression Using a Genetic Algorithm

Candidate: Juan Trelles Trabucco 
Date: Friday, April 29, 2022 
Time: 11 am CT 
Committee Members: Dr. G. Elisabeta Marai, Dr. Andy Johnson, Dr. Wei Tang, Dr. Steven Drucker, Dr. Cecilia Arighi 
Abstract: 
Biomedical researchers need to search over an increasing number of publications to find knowledge. Among these researchers, biocurators specialize in identifying relevant publications and extracting information to populate structured databases. Efforts from biocuration groups benefit a much larger community who access this curated information daily. To effectively deal with the extensive collection of documents, the biocuration workflow uses text-mining approaches to automate tasks, such as predicting the relevance of a publication. While exploiting patterns in text data has yielded significant breakthroughs for biocuration, images in publications contain important cues to determine the document's relevance. Still, integrating these image content with textual features is underexplored due to difficulties collecting images, defining classification schemes, and having access to domain experts for data labeling. 
Rooted in a years-long collaboration with text-mining researchers and biocurators, I propose strategies for harvesting labels from biomedical publications and modeling image classifiers to support biocuration. First, I present two approaches for training image classifiers with deep learning for small and unbalanced datasets. Then, I characterize the labeling task in biocuration and introduce a labeling system tailored for domain experts. Next, I build on lessons from this labeling system to develop a labeling approach for model builders by combining visual analytics and machine learning techniques. These works provide the building blocks for the future design of search interfaces that leverage text- and image-based data for biomedical experts. In addition, it supports the development of novel multi-modal approaches for detecting publication's relevance and interpreting these predictions.

Biomedical Image Classification Application User-Interface

PhD Prelim Announcement: Integrating Image Features to Support the Biocuration Workflow

The COMPaaS DLV: Composable Platform as a Service Instrument for Deep Learning & Visualization user community consists of faculty from Physics and four College of Engineering departments (Computer Science, Civil, Materials & Environmental Engineering, Mechanical and Industrial Engineering, and Electrical and Computer Engineering) - primarily run applications that are GPU-centric for compute, with significant variability in storage and networking around their data requirements. 
Computer Science applications primarily focus on security, data science, computer vision and Machine Learning (ML). Security projects explore the complexity of modern web applications and the intricacies of security mechanisms that often result in flaws that expose users to significant security and privacy threats. These projects try to develop methods and tools that enable users to understand and more effectively manage retrospective privacy in the context of modern, long-lived, online archives. Composable resources were used to develop Natural Language Processing (NLP)-based domain-specific classifiers that identified data practices stated in privacy policies. Adherence of corresponding applications were then adjusted based on this ground truth. 
Data-science applications have an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help users access, share, explore and analyze their data, whether local or remote, come to conclusions, and make decisions with greater speed, accuracy, comprehensiveness and confidence. One such project is developing and advancing tools that identify image data in biomedical literature to locate beneficial, targeted publications. This work involves training image classifiers, integrating classifiers into labeling pipelines, designing retrieval user interfaces, and identifying related visual representations. 
Computer vision projects include semantic segregation and 3D human pose estimation. Researchers are developing a novel network architecture, termed DependencyNet (dependency network), for semantic segmentation. They also achieved experimental results that demonstrate an effective approach for 3D human pose estimation. Over the past two years, they found COMPaaS to be consistently stable and efficient, with result output as expected, and the group&rsquo;s models achieved state-of-the-art performance on their respective benchmarks. 
ML applications include frameworks for many complex real-world reinforcement learning problems, such as the coordination of autonomous vehicles, network packet delivery, and distributed logistics. 
Civil, Materials & Environmental Engineering applications focus on simulation and modeling. Researchers run data-driven models on high-performance computers to develop an accurate and general neural network ML model that uses crystallographic data to study patterns of synthesizability. They also perform simulations of mass transport in alloys and ceramics. COMPaaS has been performing 1.5-10 times faster than comparable infrastructures they are familiar with. Additionally, they found our use of Jupyter Notebooks to be a significant asset. 
Mechanical and Industrial Engineering researchers use COMPaaS for three research projects: feature extraction in fluid flow using a Convolutional Neural Network (CNN); column height detection in metallic nanoparticles using a CNN; and, electric vehicle battery state-of-charge estimation using different ML methods. 
Electrical and Computer Engineering researchers recently started running mathematical models of ML algorithms and training language models using long-short term memory (LSTMs). 
In 2020, we began implementing a GPUoE (GPU over Ethernet) prototype. Using a GPU expansion chassis connected to compute nodes over Ethernet, we were able to compose remote GPUs into our existing composable infrastructure. Our APIs developed for composable infrastructure have been extended to support these remote GPUs. 
In 2021, we enhanced COMPaaS with a public-facing JupyterHub server providing secure web-based access to experiments, a modern PCIe fabric and a supporting expansion chassis, thereby making composable co-location available to our users. As familiarity with composable infrastructure grows, the idea of using composable co-location instead of traditional servers co-located in racks is possible. By composable co-location, we mean the integration of emerging hardware needs without changing the existing core infrastructure. Researchers can now co-locate their own accelerators or compute nodes and leverage existing COMPaaS resources natively.

COMPaaS DLV User Community

In today&rsquo;s Big Data era, data scientists require modern workflows to quickly analyze large-scale datasets using complex codes to maintain the rate of scientific progress. These scientists often rely on available campus resources or off-the-shelf computational systems for their applications. Unified infrastructure or over-provisioned servers can quickly become
bottlenecks for specific tasks, wasting time and resources. Composable infrastructure helps solve these problems by providing users with new ways to increase resource utilization. Composable infrastructure disaggregates a computer&rsquo;s components – CPU, GPU (accelerators), storage and networking - into fluid pools of resources, but typically relies upon infrastructure engineers to
architect individual machines. Infrastructure is either managed with specialized command-line utilities, user interfaces or specification files. These management models are cumbersome and difficult to incorporate into data-science workflows. We
developed a high-level software API, Composastructure, which, when integrated into modern workflows, can be used by infrastructure engineers as well as data scientists to reorganize composable resources on demand. Composastructure enables infrastructures to be programmable, secure, persistent and reproducible. Our API composes machines, frees resources, supports multi-rack operations, and includes a Python module for Jupyter Notebooks. 
Keywords - distributed systems, testbed implementation and deployment, composable infrastructure, deep learning, visualization, infrastructure as code

Composastructure System Architecture

Moving from Composable to Programmable

Composable infrastructure holds the promise of accelerating the pace of academic research and discovery by enabling researchers to tailor the resources of a machine (e.g., GPUs, storage, NICs), on-demand, to address application needs. We were first introduced to composable infrastructure in 2018, and at the same time, there was growing demand among our College of Engineering faculty for GPU systems for data science, artificial intelligence / machine learning / deep learning, and visualization. Many purchased their own individual desktop or deskside systems, a few pursued more costly cloud and HPC solutions, and others looked to the College or campus computer center for GPU resources which, at the time, were scarce. After surveying the diverse needs of our faculty and studying product offerings by a few nascent startups in the composable infrastructure sector, we applied for and received a grant from the National Science Foundation in November 2019 to purchase a mid-scale system, configured to our specifications, for use by faculty and students for research and research training. 
This paper describes our composable infrastructure solution and implementation for our academic community. Given how modern workflows are progressively moving to containers and cloud frameworks (using Kubernetes) and to programming notebooks (primarily Jupyter), both for ease of use and for ensuring reproducible experiments, we initially adapted these tools for our system. We have since made it simpler to use our system, and now provide our users with a public facing JupyterHub server. We also added an expansion chassis to our system to enable composable co-location, which is a shared central architecture in which our researchers can insert and integrate specialized resources (GPUs, accelerators, networking cards, etc.) needed for their research. 
In February 2020, installation of our system was finalized and made operational and we began providing access to faculty in the College of Engineering. Now, two years later, it is used by over 40 faculty and students plus some external collaborators for research and research training. Their use cases and experiences are briefly described in this paper. Composable infrastructure has proven to be a useful computational system for workload variability, uneven applications, and modern workflows in academic environments. 
Keywords - composable infrastructure, deep learning, visualization, resource management, workload management, user workflow, composable co-location, infrastructure as code

COMPaaS Hardware Racks

Composable Infrastructures for an Academic Research Environment: Lessons Learned

A collaboration of the UIC, Columbia College Chicago and the Chicago Design Archive (CDA), the &ldquo;Chicago Design Through the Decades&rdquo; project is a swift exciting journey through the history of Chicago design over the last one hundred years with content based on resources from the collection of the CDA. Following engaging characters and textual tidbits found in design works held in the archive, the journey begins in the 1920s with the era&rsquo;s painterly and illustrative techniques, evolves through the decades, and ends in the 2010s with digital rendering using AI and machine learning (ML)-technologies becoming increasingly prevalent in the contemporary landscape. At once nostalgic and whimsical, both humorous and sobering moments abound through the decades. 
The project will be showcased as part of <a href="https://artonthemart.com/">Art onTheMart</a> with previews this summer and an official opening on November 17th. 
The Chicago Design Archive is an exclusive and permanent online record of Chicago design currently holding over 3200 examples of work by over 1100 designers. 
As designers and educators, Daria Tsoupikova, Sharon Oiga and Guy Villa Jr lead a team of creators and archivists whose overarching goal is to increase the understanding and appreciation of Chicago design history and to contribute to the recognition of Chicago as an international modern design center through their use of innovative technologies. 
The project team consists of the following individuals: 
Daria Tsoupikova, UIC School of Design & EVL 
Sharon Oiga, UIC School of Design & CDA 
Guy Villa Jr, Columbia College Chicago 
Krystofer Kim, NASA 
Fabio Miranda, UIC Computer Science & EVL 
Jack Weiss, CDA 
Cheri McIntyre, CDA 
Lauren Meranda, CDA 
Louis Schwadron, Sky White Sound

Work by Choral, Marlena Novak & Jay Yim as part of Art onTheMart

Chicago Design Through the Decades

Save the date for June 11th for the preview of the &ldquo;Chicago Design Through the Decades&rdquo; project on the <a href="https://artonthemart.com/">Art ontheMART</a> at the Merchandise Mart Chicago. A collaboration between the Chicago Design Archive and EVL and School of Design professor Daria Tsoupikova and EVL assistant professor Fabio Miranda, who created the AI/ML visualization for the last contemporary decade - 2010s. 
The exhibition will showcase works from the Chicago Design Archive taking viewers on a journey that begins in the 1920s with the era&rsquo;s painterly and illustrative techniques, evolves through the decades, and ends in the 2010s with digital rendering using AI and machine learning (ML)-technologies becoming increasingly prevalent in the contemporary landscape. 
This preview will be held for one day only to align with the opening of the NeoCon Design Conference to be held at the Merchandise Mart June 13-15. This is a FREE OUTDOOR public event to be held at the corner of West Wacker Drive and North Orleans Streets on the Riverwalk.

Art ontheMart Features &ldquo;Chicago Design Through the Decades&rdquo;

In today&rsquo;s Big Data era, data scientists require modern workflows to quickly analyze large-scale datasets using complex codes to maintain the rate of scientific progress. These scientists often rely on available campus resources or off-the-shelf
computational systems for their applications. Unified infrastructure or over-provisioned servers can quickly become bottlenecks for specific tasks, wasting time and resources. Composable infrastructure helps solve these problems by providing users with new ways to increase resource utilization. Composable infrastructure disaggregates a computer&rsquo;s components - CPU, GPU (accelerators), storage and networking - into fluid pools of resources, but typically relies upon infrastructure engineers to architect individual machines. Infrastructure is either managed with specialized command-line utilities, user interfaces or specification files. These management models are cumbersome and difficult to incorporate into data-science workflows. We developed a high-level software API, Composastructure, which, when integrated into modern workflows, can be used by infrastructure engineers as well as data scientists to reorganize composable resources on demand. Composastructure enables infrastructures to be programmable, secure, persistent and reproducible. Our API composes machines, frees resources, supports multi-rack operations, and includes a Python module for Jupyter Notebooks. 
Keywords: distributed systems, testbed implementation and deployment, composable infrastructure, deep learning, visualization, infrastructure as code

Composable infrastructure holds the promise of accelerating the pace of academic research and discovery by enabling researchers to tailor the resources of a machine (e.g., GPUs, storage, NICs), on-demand, to address application needs. We were first introduced to composable infrastructure in 2018, and at the same time, there was growing demand among our College of Engineering faculty for GPU systems for data science, artificial intelligence / machine learning / deep learning, and visualization. Many purchased their own individual desktop or deskside systems, a few pursued more costly cloud and HPC solutions, and others looked to the College or campus computer center for GPU resources which, at the time, were scarce. After surveying the diverse needs of our faculty and studying product offerings by a few nascent startups in the composable infrastructure sector, we applied for and received a grant from the National Science Foundation in November 2019 to purchase a mid-scale system, configured to our specifications, for use by faculty and students for research and research training. 
This paper describes our composable infrastructure solution and implementation for our academic community. Given how modern workflows are progressively moving to containers and cloud frameworks (using Kubernetes) and to programming notebooks (primarily Jupyter), both for ease of use and for ensuring reproducible experiments, we initially adapted these tools for our system. We have since made it simpler to use our system, and now provide our users with a public facing JupyterHub server. We also added an expansion chassis to our system to enable composable co-location, which is a shared central architecture in which our researchers can insert and integrate specialized resources (GPUs, accelerators, networking cards, etc.) needed for their research. 
In February 2020, installation of our system was finalized and made operational and we began providing access to faculty in the College of Engineering. Now, two years later, it is used by over 40 faculty and students plus some external collaborators for research and research training. Their use cases and experiences are briefly described in this paper. Composable infrastructure has proven to be a useful computational system for workload variability, uneven applications, and modern workflows in academic environments. 
Keywords: composable infrastructure, deep learning, visualization, resource management, workload management, user workflow, composable co-location, infrastructure as code

This academic year 2023 EVL alum Mike Papka has joined UIC&rsquo;s Department of Computer Science and EVL faculty as a professor, bringing years of expertise in high performance computing to campus. For nearly three decades, Papka has held the position of Research Scientist at Argonne National Laboratory contributing to research in high performance computing, scientific visualization of large and complex datasets, information visualization related to high-performance computing, virtual reality for science and the integration of edge services. 
Since 2006, he’s served as Deputy Associate Laboratory Director, Computing, Environment and Life Sciences (CELS) Directorate, providing strategic planning and management of the Laboratory Directorate comprising Biosciences, Computational Science, Data Science and Learning, Environmental Sciences, Mathematics and Computer Science, a joint institute with Northwestern University, and the Argonne Leadership Computing Facility. In his role, Papka aids in executing a research program that aims to couple computing-related activities with several science domains whose future is tied to progress in computing.
 For the last 12 years, Papka has served and continues to serve as Division Director at Argonne Leadership Computing Facility (ALCF). He is responsible for planning and executing facility upgrades and integrated data strategies, risk assessment and management, operations reporting and reviews, staff promotion and retention, and safety. The ALCF is home to some of the world&rsquo;s fastest supercomputers: Theta; a Cray XC40 supercomputer based on Intel’s Xeon Phi KNL processor and NVIDIA DGX-3, Polaris; a HPE Apollo 6500 Gen10+ based on AMD Milan processor and NVIDIA A100s, and Aurora; a Intel CPU/GPU system currently being installed that support scientific and engineering open science research. 
We welcome Mike&rsquo;s return to UIC and EVL! He undoubtedly will be a valued asset to our research and educational enterprise.

Alum Michael E. Papka Joins EVL Faculty

The SAGE3 Team has been quiet far too long, but we&rsquo;ve been very busy! Here&rsquo;s what to look forward to in the coming months: 
* SAGE3 beta version software with documentation will be released the week of the IEEE/ACM Supercomputing (SC22) conference, to be held in Dallas, Texas, USA, November 13-18, 2022. More information about the release will be forthcoming. For information on SC22, see: <a href="https://sc22.supercomputing.org/">https://sc22.supercomputing.org/</a> 
* A 1-hour SAGE3 Birds of a Feather (BoF) session and ongoing demonstrations and presentations will take place during SC22. At the BoF (Tuesday, Nov 15 from 12:15-13:15 in Room D168), the SAGE3 team will introduce the latest software release. The team will also share a booth in the research exhibition (StarLight booth #2847 as well as partner booths) and will conduct demos and presentations. More information will be forthcoming. 
* Educational SAGE3 webinars will be held, starting next year. Team members will present SAGE3&rsquo;s features and capabilities, provide use cases, answer questions, and solicit user feedback and use cases. More information will be forthcoming. 
Have you tried the SAGE3 alpha version for Linux, Windows and MacOS, which was released in December 2021? If not, go to the SAGE3 website, download the software and display client, and watch the &ldquo;How To&rdquo; video. 
<a href="https://sage3.sagecommons.org/">https://sage3.sagecommons.org/</a> 
REMINDER: SAGE3 / SAGE2 COMMUNITY USER FORUMS are now hosted on Slack. To join, go to the SAGE3 website and click on the Support menu: 
<a href="https://sage3.sagecommons.org/?page_id=60">https://sage3.sagecommons.org/?page_id=60</a> 
For SAGE3, subscribe to the sage3-support channel. 
For SAGE2, subscribe to the sage2-support channel. 
Thank you for your longstanding support of SAGE3 and we look forward to continued collaborations. 
Maxine Brown 
SAGE team member 
<a href="mailto:maxine@uic.edu">maxine@uic.edu</a> 
SAGE3 is being developed by the Laboratory for Advanced Visualization and Applications (LAVA) and the Hawaii Data Science Institute (HI-DSI) at University of Hawai&rsquo;i at Manoa (UHM), the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC), and the InfoVis Lab at Virginia Tech (VT). SAGE3 receives major funding from the U.S. National Science Foundation (NSF), awards #2004014 (UHM), #2003800 (UIC) and #2003387 (VT). 
SAGE2 received major NSF support, award #ACI-1441963, to UHM and UIC. 
SAGE, SAGE2 and SAGE3 are trademarks of the University of Illinois Board of Trustees.

Announcing SAGE3 Upcoming Events !!

The Electronic Visualization Laboratory at the University of Illinois Chicago acquired a 24 compute node, 64 GPU composable infrastructure compute cluster, COMPaaS DLV - Composable Platform as a Service: Instrument for Deep Learning & Visualization in 2019.  Since then 40+ users have designed and run computations on COMPaaS.  To track cluster utilization, evaluate hardware
performance, and simply analyze application resource utilization of this experimental composable infrastructure system we set out to
collect metrics on the hardware and software infrastructure.  Users deploy their applications on the cluster using Kubernetes, a container orchestration platform that enables users to deploy applications on the cluster without the need to have complete knowledge of the underlying hardware and with minimal assistance from system administrators.  Following this, we utilized tooling already adapted for Kubernetes-based clusters to collect hardware and software metrics.  The metrics provided by this tooling is useful for system administrators in evaluating cluster performance and utilization, as well as useful for users in determining their application&rsquo;s performance and discovering bottlenecks.

GPU Power Consumption

Monitoring COMPaaS

UIC Department of Computer Science 
Doctoral Admissions Webinar 
November 2nd 2022 @7 am-8:30am Chicago time (CDT) 
Webinar: <a href="https://tinyurl.com/uicwebinar2022">https://tinyurl.com/uicwebinar2022</a> 
See what the Department of Computer Science at the University of Illinois Chicago has to offer! Located in the beautiful city of Chicago, UIC is among the most culturally diverse universities in the United States, and home to one of the most innovative computer science departments in the United States. UIC is Chicago&rsquo;s top public research university, and was recently ranked in the top 20 of the best 100 universities worldwide that are under 50 years of age. Doctoral students in their first five years of enrollment are guaranteed a fully funded appointment that includes a tuition waiver, and a stipend. Students are also eligible for summer support. Our students get jobs in academia, industry, and entertainment upon graduation, including positions at top research labs and companies such as Google, Microsoft, Apple, Disney, and Amazon. 
During this webinar, UIC Computer Science faculty will talk about our research agenda, application fee waivers, the application process, application expectations in terms of GPA/GRE etc, and will take questions. A recording of the webinar will be made available on our Graduate Admissions page. 
Apply: <a href="https://cs.uic.edu/graduate/admissions/">https://cs.uic.edu/graduate/admissions/</a> 
Deadline: December 15th 2022 
See our flyer at: <a href="https://tinyurl.com/uic2022flyer">https://tinyurl.com/uic2022flyer</a> 
 <iframe width="420" height="315"src="https://www.youtube.com/embed/eNoBbCm1skM">
</iframe>

Interested in pursuing a PhD in Computer Science in the United States?

Background. Personalized radiotherapy can improve treatment outcomes of head and neck cancer (HNC) patients, where currently a &rsquo;one-dose-fits-all&rsquo; approach is the standard. The aim was to establish individualized outcome prediction based on multi-institutional international &ldquo;big data&rdquo; to facilitate risk-based stratification of HNC patients. 
Methods. The data of 4611 HNC radiotherapy patients from three academic cancer centers was split into 4 cohorts: a training (n=2241), independent test (n=786), and external validation cohorts 1 (n=1087) and 2 (n=497). Tumor- and patient-related clinical variables were considered in a machine learning pipeline to predict overall survival (primary endpoint) and local and regional tumor control (secondary endpoints); serially, imaging features were considered for optional model improvement. Finally, patients were stratified into high, intermediate, and low risk groups. 
Results. Performance score, AJCC8th stage, pack-years, and Age were identified as predictors for overall survival, demonstrating good performance in both the training cohort (c-index=0.72 [95% CI, 0.66-0.77]) and in all three validation cohorts (c-indices: 0.76 [0.69-0.83], 0.73 [0.68- 0.77], and 0.75 [0.68-0.80]). Excellent stratification of HNC patients into high, intermediate, and low mortality risk was achieved; with 5-year overall survival rates of 17-46% for the high-risk group compared to 92-98% for the low-risk group. The addition of morphological image feature further improved the performance (c-index=0.73 [0.64-0.81]). These models are integrated in a clinic-ready interactive web-interface: <a href="https://uic-evl.github.io/hnc-predictor/">https://uic-evl.github.io/hnc-predictor/</a>. 
Conclusions. Robust model-based prediction was able to stratify HNC patients in distinct high, intermediate and low mortality risk groups. This can effectively be capitalized for personalized radiotherapy, e.g., for tumor radiation dose escalation/de-escalation. 
See <a href+"https://www.sciencedirect.com/science/article/pii/S095980492200805X">Science Direct</a> for more information.

Head and Neck Cancer Predictive Risk Estimator to Determine Control and Therapeutic Outcomes of Radiotherapy (HNCPREDICTOR)

Cluster schedulers are crucial in high-performance computing (HPC). They determine when and which user jobs should be allocated to available system resources. Existing cluster scheduling heuristics are developed by human experts based on their experience with speciﬁc HPC systems and workloads. However, the increasing complexity of computing systems and the highly dynamic nature of application workloads have placed tremendous burden on manually designed and tuned scheduling heuristics. More aggressive optimization and automation are needed for cluster scheduling in HPC. In this work, we present an automated HPC scheduling agent named DRAS (Deep Reinforcement Agent for Scheduling) by leveraging deep reinforcement learning. DRAS is built on a hierarchical neural network incorporating special HPC scheduling features such as resource reservation and backﬁlling. An efﬁcient training strategy is presented to enable DRAS to rapidly learn the target environment. Once being provided a speciﬁc scheduling objective given by the system manager, DRAS automatically learns to improve its policy through interaction with the scheduling environment and dynamically adjusts its policy as workload changes. We implement DRAS into an HPC scheduling platform called CQGym. CQGym provides a common platform allowing users to ﬂexibly evaluate DRAS and other scheduling methods such as heuristic and optimization methods. The experiments using CQGym with different production workloads demonstrate that DRAS outperforms the existing heuristic and optimization approaches by up to 50%. 
Index Terms - High-performance computing, cluster scheduling, deep reinforcement learning, job starvation, backﬁlling, resource reservation, OpenAI Gym 
<a href="https://ieeexplore.ieee.org/abstract/document/9894371">https://ieeexplore.ieee.org/abstract/document/9894371</a>

DRAS overview. DRAS agent (at the bottom) represents the scheduling agent; the environment (at the top) comprises the rest of the system, including job wait queue and HPC cluster.

DRAS: Deep Reinforcement Learning for Cluster Scheduling in High Performance Computing

Powerful detectors at modern experimental facilities routinely collect data at multiple GB/s. Online analysis methods are needed to enable the collection of only interesting subsets of such massive data streams, such as by explicitly discarding some data elements or by directing instruments to relevant areas of experimental space. Thus, methods are required for configuring and running distributed computing pipelines-what we call flows-that link instruments, computers (e.g., for analysis, simulation, artificial intelligence [AI] model
training), edge computing (e.g., for analysis), data stores, metadata catalogs, and high-speed networks. We review common patterns associated with such flows and describe methods for instantiating these patterns. 
We present experiences with the application of these methods to the processing of data from five different scientific instruments, each of which engages powerful computers for data inversion, model training, or other purposes. We also discuss implications of such methods for operators and users of scientific facilities. 
<a href="https://www.cell.com/patterns/fulltext/S2666-3899(22)00231-8">https://www.cell.com/patterns/fulltext/S2666-3899(22)00231-8</a>

Linking scientific instruments and computation: Patterns, technologies, and experiences

SAGE3 beta version software and documentation will be introduced next week at the IEEE/ACM Supercomputing (SC22) conference in Dallas, Texas, USA, November 13-18, 2022. The SAGE User Community is invited to attend the SC22 Birds of a Feather (BOF) meeting and participate in ongoing SAGE3 demonstrations in the SC22 Exhibition. 
Attend the &ldquo;SAGE 12th SC BOF: Smart Amplified Group Environment Enhanced with Artificial Intelligence for Global Collaboration&rdquo;: 
 * Tuesday, November 15, 2022 
 *12:15PM - 01:15PM CST 
 * Room: D168, Kay Bailey Hutchison Convention Center Dallas 
 * <a href="https://sc22.supercomputing.org/presentation/?id=bof141&sess=sess312">https://sc22.supercomputing.org/presentation/?id=bof141&sess=sess312</a> 
Join us for SAGE3 demonstrations in the StarLight research booth # 2847 on the exhibition floor, Tuesday-Thursday. 
SAGE3, the Smart Amplified Group Environment, is the next-generation, &ldquo;human-in-the-loop&rdquo; collaboration platform, providing HPC users with the tools to access, explore, discover, publish, share, integrate, and reuse complex datasets. SAGE3 supports today&rsquo;s hybrid work/home environments (laptops, single monitors and display walls), and interfaces with a variety of computational infrastructures, workflows, notebooks, and analytics software through Artificial Intelligence-enabled services and orchestration services. It lowers the barrier of entry into AI for non-expert users, democratizing access to disruptive technologies for those with varying skills. 
The BOF will not be streamed. SAGE3 software and documentation will be available for download on the SAGE3 website. The SAGE3 team plans to hold SAGE3 educational webinars, starting next year. 
REMINDER: SAGE3 / SAGE2 COMMUNITY USER FORUMS are now hosted on Slack. To join, go to the <a href="https://sage3.sagecommons.org/">SAGE3 website</a> and click on the Support menu. 
 * For SAGE3, subscribe to the sage3-support channel. 
 * For SAGE2, subscribe to the sage2-support channel. 
Thank you for your longstanding support of SAGE and we look forward to continued collaborations. 
Maxine Brown 
SAGE team member 
<a href="mailto:maxine@uic.edu">maxine@uic.edu</a> 
SAGE3 is being developed by the Laboratory for Advanced Visualization and Applications (LAVA) and the Hawaii Data Science Institute (HI-DSI) at University of Hawaiʻi at Mānoa (UHM), the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC), and the InfoVis Lab at Virginia Tech (VT). SAGE3 receives major funding from the U.S. National Science Foundation (NSF), awards #2004014 (UHM), #2003800 (UIC) and #2003387 (VT). 
SAGE2 received major NSF support, award #ACI-1441963, to UHM and UIC. 
SAGE, SAGE2 and SAGE3 are trademarks of the University of Illinois Board of Trustees.

INTRODUCING SAGE3 at the SC22 - Tuesday, November 15, 12:15pm - 1:15pm

Three UIC faculty members, including Professor Daria Tsoupikova and Assistant Professor Fabio Miranda from UIC&rsquo;s Electronic Visualization Laboratory (EVL), and the School of Design&rsquo;s Professor Sharon Oiga, can see their work on display nightly, as part of a larger-than-life public art installation downtown - <a href="https://artonthemart.com/chicago-design-through-the-decades/">Art on theMART</a>. 
The display referred to as Chicago Design Through the Decades is a journey through the last hundred years of the history of Chicago design and is based on the collection of the <a href="https://chicagodesignarchive.org/">Chicago Design Archive (CDA)</a>, a permanent online record of design in the city. The historical imagery is assembled using machine learning algorithms provided by Miranda, which is a tribute to Chicago as the home of neural networks. The final imagery is projected across the Chicago River onto the Merchandise Mart for a spectacular digital art experience. The display runs through December 30th nightly at 7:30 p.m. and 8 p.m. 
Stay tuned for the possibility of student collaboration on another Art on theMART display or similar public digital art project given the new undergraduate <a href="https://cs.uic.edu/undergraduate/cs-and-design/">CS + Design</a> major, which debuted Fall 2022. UIC is the only public university that currently offers this degree, and prospects are bright for students to participate in innovative digital art projects such as Chicago Design Through the Decades in the future. 
<a href="https://cs.uic.edu/news-stories/larger-than-life-cs-faculty-participate-in-public-art-project-in-downtown-chicago/">Read Andrea Poet&rsquo;s full article</a>.

UIC CS News Features &ldquo;Larger than life: CS faculty participate in public art project in downtown Chicago&rdquo;

MS Candidate

Hal Brynteson

Nikita Thakur

MS Student Undergraduate Researcher 2023

Sreten Kljaic

Undergraduate Researcher

Fredric Massa

Research Experience Undergraduate

Aakash Lilaramani

While cities around the world are increasingly promoting streets and public spaces that prioritize pedestrians over vehicles, significant data gaps have made pedestrian mapping, analysis, and modeling challenging to carry out. Most cities, even in industrialized economies, still lack information about the location and connectivity of their sidewalks, making it difficult to implement research on pedestrian infrastructure and holding the technology industry back from developing accurate, location-based Apps for pedestrians, wheelchair users, street vendors, and other sidewalk users. To address this gap, we have designed and implemented an end-to-end open-source tool - TILE2NET - for extracting sidewalk, crosswalk, and footpath polygons from orthorectified aerial imagery using semantic segmentation. The segmentation model, trained on aerial imagery from Cambridge, MA, Washington DC, and New York City, offers the first open-source scene classification model for pedestrian infrastructure from sub-meter resolution aerial tiles, which can be used to generate planimetric sidewalk data in North American cities. TILE2NET also generates pedestrian networks from the resulting polygons, which can be used to prepare datasets for pedestrian routing applications. The work offers a low-cost and scalable data collection methodology for systematically generating sidewalk network datasets, where orthorectified aerial imagery is available, contributing to over-due efforts to equalize data opportunities for pedestrians, particularly in cities that lack the resources necessary to collect such data using more conventional methods. 
Keywords: Pedestrian network extraction, Semantic segmentation, Automated network generation, Pedestrian infrastructure

Model results showing detected sidewalk, crosswalk and footpath centerlines in a) Boston and Cambridge, b) Manhattan and parts of Brooklyn, c) Wash- ington, DC. The maps are shown at the same scale for comparison.

Mapping the walk: A scalable computer vision approach for generating sidewalk network datasets from aerial imagery

&ldquo;Where the sidewalk ends&rdquo; in <a href="https://news.mit.edu/2023/open-source-tool-mapping-sidewalks-0315">MIT News</a> discusses how Miranda along with researchers from MIT, University of Sao Paulo, and New York University have built an open-source tool that uses aerial imagery and image-recognition to create complete maps of sidewalks and crosswalks. The tool, called TILE2NET can help planners, policymakers, and urbanists who want to expand pedestrian infrastructure. 
The research team used computerized image-recognition techniques to build a tool that will visually recognize sidewalks, crosswalks, and footpaths. To do that, the researchers first used 20,000 aerial images from Boston, Cambridge, New York City, and Washington - places where comprehensive pedestrian maps already existed. By training the image-recognition model on such clearly defined objects and using portions of those cities as a starting point, they were able to see how well TILE2NET would work elsewhere in those cities. 
<a href="https://news.mit.edu/2023/open-source-tool-mapping-sidewalks-0315">Read the full MIT News article.</a> 
For additional information regarding this research see the Computers, Environment and Urban Systems publication <a href="https://www2.evl.uic.edu/entry.php?id=2703">&ldquo;Mapping the walk: A scalable computer vision approach for generating sidewalk network datasets from aerial imagery&rdquo;</a>.

Boston Commons: a) Aerial image, b) Detected sidewalk and footpath polygons (in orange) and detected crosswalks (in red), c) Fitted sidewalk, crosswalk, and footpath centerlines superimposed on the aerial image.

EVL Faculty Member Fabio Miranda&rsquo;s TILE2NET Collaborative Research Featured in MIT News

MS Graduate, 2023

Matt Ziminski

Nguyen Hoa Pham

Dian Jia

Developing applicable clinical machine learning models is a difficult task when the data includes spatial information, for example, radiation dose distributions across adjacent organs at risk. We describe the co-design of a modeling system, DASS,to support the hybrid human-machine development and validation of predictive models for estimating long-term toxicities related to radiotherapy doses in head and neck cancer patients. Developed in collaboration with domain experts in oncology and data mining, DASS incorporates human-in-the-loop visual steering, spatial data, and explainable AI to augment domain knowledge with automatic data mining. We demonstrate DASS with the development of two practical clinical stratification models and report feedback from domain experts. Finally, we describe the design lessons learned from this collaborative experience. 
CCS Concepts 
Human-centered computing→Scientific visualization; Computing methodologies→Machine learning; Applied computing→Life and medical sciences

DASS interactive model building for head/neck cancer.

DASS Good: Explainable Data Mining of Oncology Imaging and Toxicity Data

DASS is a visual computing system designed to help scientists build machine learning models that classify head and neck cancer patients into groups, based on their risk of developing permanent side effects from radiation therapy. Its visual interface helps data scientists to build a model for each side-effect by visualizing the patients&rsquo anatomical and other clinical data, and to figure out what parameters the model should consider, such as which organs may be damaged when side effects occur, what amount of radiation dose causes the damage to the organs, and how many risk groups to create. DASS also has features for explaining to clinicians the finished models and examining whether their predictions perform well, and the reasons behind why a given patient belongs in a certain risk group. DASS was created by visual computing researchers at EVL, in collaboration with clinicians at the MD Anderson Cancer Center and data mining specialists at University of Iowa, and is supported by funding from the US National Institutes of Health (NIH awards NCI-R01-CA258827 and NLM-R01-LM012527), and US National Science Foundation (NSF awards CDSE-1854815 and CNS-1828265).

Stylized scatterplot. Patients are represented by a custom glyph that encodes the outcome of interest as marks extending radially. Markers are colored by cluster membership, and a contour is shown around the currently selected cluster.

DASS

EVL professor Mike Papka and his research students have been awarded an Honorable Mention in the EuroVis Best Paper Category for &ldquo;ChemoGraph: Interactive Visual Exploration of the Chemical Space&rdquo;. Co-authored by collaborators Bharat Kale, Austin Clyde, Maoyuan Sun, Arvind Ramanathan, Rick Stevens, and Michael E. Papka, ChemoGraph is a novel visual analytics technique for interactively exploring related chemicals. A chemical space is formalized as a hypergraph with novel machine-learning models to compute related chemical compounds applied. It uses a database to find related compounds from an available space and a machine learning model to generate new ones, which helps enlarge the known space. Moreover, ChemoGraph highlights interactive features that support users in viewing, comparing, and organizing computationally identified related chemicals. 
<a href="https://youtu.be/kVpz6TDjQMw ">See a video teaser on EVLTube.</a>

&ldquo;ChemoGraph: Interactive Visual Exploration of the Chemical Space&rdquo; Awarded with Honorable Mention in the EuroVis Best Paper Category

The 2023-2024 University Scholar Program has selected EVL professor G. Elisabeta Marai among its awarded and honored candidates. Dr. Marai is among two other designees within the College of Engineering - Professor Natasha Devroye with the Department of Electrical and Computer Engineering Department, and Professor Karl Rockne with Civil, Materials, and Environmental Engineering. Sponsored by the Office of the President, the program rewards outstanding faculty who have been recognized for their superior performance in research, teaching, and scholarly activities. 
Join us in congratulating professor Liz! 
<a href="https://emails.uofi.uic.edu/newsletter/25/464513625.html">See the full article.</a>

EVL Professor G. Elisabeta Marai Awarded 2023-2024 University Scholar

Personalized head and neck cancer therapeutics have greatly improved survival rates for patients, but are often leading to understudied long-lasting symptoms which affect quality of life. Sequential rule mining (SRM) is a promising unsupervised machine
learning method for predicting longitudinal patterns in temporal data which, however, can output many repetitive patterns that are difficult to interpret without the assistance of visual analytics. We present a data-driven, human-machine analysis visual system
developed in collaboration with SRM model builders in cancer symptom research, which facilitates mechanistic knowledge discovery in large-scale, multivariate cohort symptom data. Our system supports multivariate predictive modeling of post-treatment symptoms
based on during-treatment symptoms. It supports this goal through an SRM, clustering, aggregation back end, and a custom front
end to help develop and tune the predictive models. The system also explains the resulting predictions in the context of therapeutic
decisions typical in personalized care delivery. We evaluate the resulting models and system with an interdisciplinary group of modelers and head and neck oncology researchers. The results demonstrate that our system effectively supports clinical and symptom research. 
Index Terms: Temporal Data; Life Sciences; Mixed Initiative Human-Machine Analysis; Data Clustering and Aggregation

Longitudinal symptom analysis and prediction for head and neck patients (ICC treatment).

Roses Have Thorns: Understanding the Downside of Oncological Care Delivery Through Visual Analytics and Sequential Rule Mining

We present an interactive analysis system to support sequential rule mining (SRM) model builders who work in cancer symptom research. SRM is a promising unsupervised machine learning method for predicting longitudinal patterns in temporal data which, however, can output many repetitive patterns that are difficult to interpret without the assistance of visual analytics. Our system facilitates mechanistic knowledge discovery of treatment-related toxicities (i.e., "roses have thorns") in large-scale cohort data. 
This work was developed by visual computing researchers at EVL in collaboration with data mining specialists at the University of Iowa, and clinicians at the MD Anderson Cancer Center in Texas, and it is supported by funding from the US National Institutes of Health (NIH awards NCI-R01-CA258827 and NLM-R01-LM012527), and US National Science Foundation (NSF awards CDSE-1854815 and CNS-1828265).

Symptom clustering for treatment ICC.

Motivation: Figures in biomedical papers communicate essential information with the potential to identify relevant documents in biomedical and clinical settings. However, academic search interfaces mainly search over text fields. 
Results: We describe a search system for biomedical documents that leverages image modalities and an existing index server. We integrate a problem-specific taxonomy of image modalities and image-based data into a custom search system. Our solution features a front-end interface to enhance classical document search results with image-related data, including page thumbnails, figures, captions, and image-modality information. We demonstrate the system on a subset of the CORD-19 document collection. A quantitative evaluation demonstrates higher precision and recall for biomedical document retrieval. A qualitative evaluation with domain experts further highlights our solution’s benefits to biomedical search. 
Availability and implementation: A demonstration is available at <a href="https://runachay.evl.uic.edu/scholar">https://runachay.evl.uic.edu/scholar</a>. Our code and image models can be accessed via <a href="https://github.com/uic-evl/bio-search">https://github.com/uic-evl/bio-search</a>. The dataset is continuously expanded. 
Image caption: Search interface displaying the results of a boolean query. (A) The color-coded legend displays the image modalities in the first level of the taxonomy. (B) The search bar, with a keyword box showing a boolean query, a time filter, and an image modalities filter limiting results to documents containing \textit{radiology} or \textit{fluorescence microscopy} figures. (C) The metadata of a result includes the document title, year, venue, authors, and the hits on the abstract and full text. Below is the count of figures and modalities in the document. (D) The enhanced document surrogate shows the image content, including the page thumbnail, extracted figure, subfigures and modalities, and caption. 
<a href="https://doi.org/10.1093/bioadv/vbad095">https://doi.org/10.1093/bioadv/vbad095</a>

Search interface displaying the results of a boolean query.

Enhancing biomedical search interfaces with images

Searching for scientific documents is a pervasive task among researchers. While most academic search engines, such as Google Scholar, search over text data, image data provides relevant information to complement text-based queries. For instance, in biomedical and biology research, images show experimental results and provide cues about the methods followed. This work shows the importance of combining text and image-based features to aid researchers in finding relevant documents in a COVID-19 collection. We derived a taxonomy to represent image content based on their acquisition method (e.g., captured in a microscope), integrated this information by leveraging deep learning image classifiers, and presented the result in hybrid image+text surrogates. Our design improved the user experience during document retrieval.

The system architecture. The online components include: web interface, application server for data retrieval, and Apache Lucene search engine. The offline components are responsible for labeling, model training and document preprocessing.

Candidate: Juan Trelles Trabucco 
Committee members: 
- Dr. Liz Marai 
- Dr. Andy Johnson 
- Dr. Wei Tang 
- Dr. Cecilia Arighi (University of Delaware) 
- Dr. Steven Drucker (Microsoft Research) 
Date/time: Aug 24th, 11 am CT 
Location: ERF 3036 and Zoom 
https://uic.zoom.us/j/84392472211?pwd=RTVwQjZaSlVBcERHcUloc1N6THQ0QT09
Meeting ID: 843 9247 2211 Passcode: UA6mfGMR 
Abstract: 
Images in scientific publications communicate essential information, such as experiments performed and results obtained, that researchers can use as a proxy for the publication'&rsquo;s relevance to their research interests. Domains like biocuration, where biomedical experts analyze the literature to extract and organize information for future retrieval and analysis, can benefit from computationally leveraging these images in their workflows. Prior work in biomedical document classification for biocuration has shown that combining features from images (e.g., acquisition modality) with text features can improve precision and recall. However, the considerations and benefits obtained from computationally integrating such image-based features in biocuration workflows are less clear. 
Based on a multi-year collaboration with text-mining researchers and biocurators, this dissertation identifies and tackles several challenges to computationally using image-data in the biocuration process. These challenges include the scarcity of labeled datasets for training image classifiers, the lack of characterization for the needed modality classes in biocuration, the lack of available labeling systems to label extensive collections of images, and the lack of support for image-based features in academic search engines. This dissertation addresses these issues by proposing two taxonomies for image modalities found in biomedical publications; introducing training strategies using shallower neural networks and hierarchical organization of classifiers; describing two labeling systems, one for domain experts and one for model builders who require support for data understanding; and concludes by merging several building blocks into document search systems that successfully leverage image-based data.

PhD Dissertation Announcement: &ldquo;Integrating Image Features to Support the Biocuration Workflow&rdquo;

Shanghao Li

Starting Academic Year 23/24 visiting professor Zhiling Lan joins as faculty in the Department of Computer Science and EVL. Dr. Lan comes with significant expertise in high-performance computing and parallel/distributed systems. Prior to her tenure at UIC, she was a professor of Computer Science, at the Illinois Institute of Technology between 2002 and 2023. She is also a guest research faculty at Argonne National Laboratory. 
Dr. Lan also heads the <a href="https://spear.lab.uic.edu/">SPEAR (Systems for Performance, Energy, And Resiliency) lab</a>, which conducts research spanning various areas of parallel and distributed systems including cluster management, interconnection networking, performance modeling and simulation, power and energy efficiency, and fault tolerance. SPEAR&rsquo;smission is to unify systems and AI, where we are committed to creating software systems to expedite AI applications and exploring advanced AI technologies to tackle critical research challenges in computer systems.

EVL Welcomes Professor Zhiling Lan to High Performance Computing Faculty Team

Hyeonjeong Park

Teddy Zaremba

EVL Faculty Assistant Professor, Computer Science

Sidharth Kumar

The National Science Foundation has awarded EVL a Major Research Instrumentation (MRI) grant to build the Data Observation and Computation Collaboratory (DOCC) in the new Computer Design Research and Learning Center (CDRBC) building. The Data Observation and Computation Collaboratory (DOCC, &rsquo;the dock&rsquo;) is a much-needed visual data science instrument for UIC science and engineering research and research training. DOCC provides high-performance computing, advanced networking, high-speed data storage, interactive high-resolution displays and software and services for Artificial Intelligence / Machine Learning (AI/ML), data analytics, data simulations, visualization, and virtual-reality requirements. Given how varied modern data-science application requirements can be, DOCC is designed as a configurable, heterogeneous computing environment that prioritizes real-time, interactive, collaborative applications and remote data streaming, but can be scheduled off-hours for large, batch-mode processing. 
DOCC will empower users to work with their data and their colleagues in real-time, test codes, view a &rsquo;window in the oven&rsquo;to watch large simulations run, and simultaneously view results along with related data and ensemble runs on large display walls. DOCC has a visual front end that enables users to easily access familiar collaboration tools, scientific notebooks, AI/ML workflows, and popular visualization software. 
Alternatively, users can add new AI/ML models or create them from scratch. And because DOCC will be connected to the campus research network, data and/or computer codes can be ported to remote computer and cloud resources located on campus, regionally, nationally, or internationally. 
A room-sized instrument, DOCC will be located in the high-trafficked atrium of the new CDRLC, home to the UIC Computer Science Department, part of the College of Engineering, situated across from an auditorium and multi-purpose event space. This location will promote UIC data-science research initiatives and the instrument&rsquo;s collaboration potential. DOCC will also be used to conduct campus research meetings, programs and training initiatives. 
Award Number (FAIN): 2320261 
Period of Performance: 10/01/2023 - 09/30/2026 
Project Title: Research Infrastructure: MRI: Track 2 Acquisition of Data Observation and Computation Collaboratory (DOCC)

EVL Awarded NSF MRI Support to Build the Data Observation and Computation Collaboratory (DOCC)

Biocuration is the process of analyzing biological or biomedical articles to organize biological data into data repositories using taxonomies and ontologies. Due to the expanding number of articles and the relatively small number of biocurators, automation is desired to improve the workflow of assessing articles worth curating. As figures convey essential information, automatically integrating images may improve curation. In this work, we instantiate and evaluate a first-in-kind, hybrid image+text document search system for biocuration. The system, MouseScholar, leverages an image modality taxonomy derived in collaboration with biocurators, in addition to figure segmentation, and classifiers components as a back-end and a streamlined front-end interface to search and present document results. We formally evaluated the system with ten biocurators on a mouse genome informatics biocuration dataset and collected feedback. The results demonstrate the benefits of blending text and image information when presenting scientific articles for biocuration. 
Index Terms - document search, biocuration

MouseScholar interface.

MouseScholar: Evaluating an Image+Text Search System for Biocuration

&ldquo;MouseScholar: Evaluating an Image+Text Search System for Biocuration&rdquo;, Juan Trelles Trabucco, Carla Floricel, Cecilia Arighi, Hagit Shatkay, Daniela Raciti, Martin Ringwald, and G. Elisabeta Marai is an accepted paper at 2023 IEEE International Conference on Bioinformatics and Biomedicine (IEEE BIBM 2023). Of the 1084 paper submissions submitted to BIBM this year, only 211 regular papers were accepted (acceptance rate: 19.5%). 
Project Description: 
In this work, we instantiate and evaluate a first-in-kind, hybrid image+text document search system for biocuration (the process of analyzing biological or biomedical articles to organize biological data into data repositories using taxonomies and ontologies). The system, MouseScholar, leverages an image modality taxonomy derived in collaboration with biocurators, in addition to figure segmentation, and classifiers components as a back-end and a streamlined front-end interface to search and present document results. We formally evaluated the system with ten biocurators on a mouse genome informatics biocuration dataset and collected feedback. The results demonstrate the benefits of blending text and image information when presenting scientific articles for biocuration. 
Congratulations EVL research team on being accepted for publication with this prestigious Conference!

Biocuration taxonomy of figures in scientific publications, with examples.

EVLers Juan Trelles Trabucco, Carla Floricel and Professor Elisabeta Marai Biocuration Research Recognized by IEEE BIBM 2023

PhD Student: Andrew Wentzel 
Committee members: 
Dr G.Elisabeta Marai (chair) 
Dr. Xinhua Zhang 
Dr. Fabio Miranda 
Dr. Guadalupe Canahuate (University of Iowa) 
Dr. Renata Raidou (Vienna University of Technology) 
Date Time: Wednesday, Dec 6th at 10 am CST 
Location: ERF 2068 
Zoom Details: 
<a href="https://uic.zoom.us/j/83499037419?pwd=MFhvVHg2aXNZWnZwZHBhM0RKUE5ZQT09">Join Zoom Meeting</a> 
Meeting ID: 834 9903 7419 
Passcode: L1fCBhu9 
One tap mobile 
+13126266799,,83499037419#,,,,*43232065# US (Chicago) 
+13092053325,,83499037419#,,,,*43232065# US 
Abstract: 
Research in visual computing (VC) focuses on integrating visual representations of datasets with computing systems, thus leveraging the complementary abilities of humans and machines. The resulting VC systems may include machine learning (ML) techniques, which have remarkable abilities in terms of processing vast amounts of data. However, creating VC+ML systems that can operate on spatio-temporal data is a largely unexplored, difficult topic. Challenges in this area include: capturing domain characterizations for spatial VC+ML problems, in particular when serving both ML model builders and ML model clients; creating explainable VC+ML models that can operate on spatial data, including measures that can capture spatial structure similarity; designing visual encodings for ML models that use spatial information and that enhance the modeler and client understanding of the model; and measuring the effect of deploying such VC+ML systems in practice. 
Based on several multi-year collaborations, I document several instances where spatial and explainable VC+ML solutions are required, and what role spatiality plays in these instances. Throughout this process, I extract and describe design requirements for explainable spatial VC+ML systems, and I identify necessary VC advances, including spatially-aware similarity measures and spatial visual explainability. I construct spatially-aware similarity measures to support ML problems in precision oncology, patient risk stratification in data mining, and explainable interactive regression modeling for hypothesis testing in social science. I introduce novel encodings for 3-dimensional and geospatial data and explore how to integrate such encodings with ML model explanations. I implement and evaluate several VC+ML systems and several resulting models that are explicitly designed to integrate domain-specific structural spatial information and unsupervised ML. Throughout this process, I also articulate design lessons related to model actionability. In my proposed work, I expand my focus to exploring the issue of spatial VC+ML explainability, this time in a supervised deep-learning setting.

Presentation Slides

PhD Prelim Announcement: Visual Computing Design for Explainable, Spatially-aware Machine Learning

Leonardo Ferreira

Sunlight and shadow play critical roles in how urban spaces are utilized, thrive, and grow. While access to sunlight is essential to the success of urban environments, shadows can provide shaded places to stay during the hot seasons, mitigate heat island effect, and increase pedestrian comfort levels. Properly quantifying sunlight access and shadows in large urban environments is key in tackling some of the important challenges facing cities today. In this paper, we propose Dee Umbra, a novel computational framework that enables the quantification of sunlight access and shadows at a global scale. Our framework is based on a conditional generative adversarial network that considers the physical form of cities to compute high-resolution spatial information of accumulated sunlight access for the different seasons of the year. We use data from seven different cities to train our model, and show, through an extensive set of experiments, its low overall RMSE (below 0.1) as well as its extensibility to cities that were not part of the training set. Additionally, we contribute a set of case studies and a comprehensive dataset with sunlight access information for more than 100 cities across six continents of the world. Deep Umbra is available at <a href="https://urbantk.org/shadows">https://urbantk.org/shadows</a>. 
Index Terms - Urban computing, Urban analytics, Sunlight access, Shadow, Generative adversarial networks.

Example where shadows are cast across tiles.

Deep Umbra: A Generative Approach for Sunlight Access Computation in Urban Spaces

Unsupervised semantic segmentation (USS) aims at partitioning an image into semantically meaningful segments by learning from a collection of unlabeled images. The effectiveness of current approaches is plagued by difficulties in coordinating representation learning and pixel clustering, modeling the varying feature distributions of different classes, handling outliers and noise, and addressing the pixel class imbalance problem. This paper introduces a novel approach, termed Imbalance-Aware Dense Discriminative Clustering (IDDC), for USS, which addresses all these difficulties in a unified framework. Different from existing approaches, which learn USS in two stages (i.e., generating and updating pseudo masks, or refining and clustering embeddings), IDDC learns pixel-wise feature representation and dense discriminative clustering in an end-to-end and self-supervised manner, through a novel objective function that transfers the manifold structure of pixels in the embedding space of a vision Transformer (ViT) to the label space while tolerating the noise in pixel affinities. During inference, the trained model directly outputs the classification probability of each pixel conditioned on the image. In addition, this paper proposes a new regularizer, based on the Weibull function, to handle pixel class imbalance and cluster degeneration in a single shot. Experimental results demonstrate that IDDC significantly outperforms all previous USS methods on three real-world datasets, COCO-Stuff-27, COCO-Stuff-171, and Cityscapes. Extensive ablation studies validate the effectiveness of each design. Our code is available at https://github.com/MY-LIU100101/IDDC. Our code is available at https://github.com/MY-LIU100101/IDDC. 
Keywords: Unsupervised semantic segmentation · Imbalance-Aware Dense Discriminative Clustering · End-to-end training ·
Deep clustering 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Imbalance-Aware Discriminative Clustering for Unsupervised Semantic Segmentation

Though the object detection performance on standard benchmarks has been improved drastically in the last decade, current object detectors are often vulnerable to domain shift between the training data and testing images. Domain adaptation techniques have been developed to adapt an object detector trained in a source domain to a target domain. However, they assume that the target domain is known and fixed and that a target dataset is available for training, which cannot be satisfied in many real-world applications. To close this gap, this paper investigates fully test-time adaptation for object detection. It means to update a trained object detector on a single testing image before making a prediction, without access to the training data. Through a diagnostic study of a baseline self-training framework, we show that a great challenge of this task is the unreliability of pseudo labels caused by domain shift. We then propose a simple yet effective method, termed the IoU Filter, to address this challenge. It consists of two new IoU-based indicators, both of which are complementary to the detection confidence. Experimental results on five datasets demonstrate that our approach could effectively adapt a trained detector to various kinds of domain shifts at test time and bring substantial performance gains. Code is available at https://github.com/XiaoqianRuan1/IoU-filter. Code is available at https://github.com/ XiaoqianRuan1/IoU-filter. 
Funding:: The COMPaaS DLV project (NSF award CNS-1828265)

Fully Test-time Adaptation for Object Detection

Developing sensing circuits and systems capable of perceiving the physical environment is an essential step toward next-generation cyber-physical systems that leverage computation, communication, and sensing integrated with the physical domain. The recent introduction of a class of artificial intelligence (AI) methods known as deep learning has drastically advanced vision-based environment perception. However, the applicability of these methods is significantly limited by their reliance on vast amounts of 3D annotations for training, which are expensive, unnatural, and often unobtainable. Acquiring 3D annotations necessitates specialized data captured by 3D sensors rather than utilizing abundant and readily available images or videos. Moreover, it requires a controlled environment or compromises with inaccurate subjective labeling. This project aims to address real-time environment perception in a very challenging, largely unexplored, yet highly practical setting, where only 2D annotations are available during training, by modeling and learning rich space-time structures of the environment. The outcomes of this project will significantly impact cyber-physical systems and facilitate a wide range of applications, from robots in manufacturing and personal services to autonomous vehicles that enhance people’s mobility and safety. Furthermore, this project will tightly integrate research and education at the University of Illinois Chicago (UIC), which is a Minority Serving Institution (MSI), through curriculum development, research training for high school, undergraduate, and graduate students, broadening the participation of female and minority students, and community outreach. 
This project will address real-time environment perception when only 2D annotations are available during training by systematically pursuing a novel approach that decomposes the complex physical environment into geometry and motion substructures in 3D space and models their rich space-time interactions. From streaming video, the trained system will not only recognize objects and scene layouts in the environment but also estimate their 3D geometry and 3D motion in real-time. This project is focused on (1) establishing a self-supervised framework for lifting 2D objects to 3D, by bridging geometry and motion, brightness constancy, and differentiable depth rendering, (2) scaling 2D-to-3D lifting of individual objects to perceive the entire environment, by modeling and learning short-term and long-term dynamics of the environment, and (3) achieving flexible and generalizable environment perception by handling articulated motion and out-of-distribution environments. The new approach is expected to be efficient in terms of the amount of annotations required for training, scalable to a wide range of objects, and robust to the complexity and diversity of real-world environments. This project will advance and enrich the fundamental research of visual sensing, perception, and learning. Moreover, this project will demonstrate the usability and robustness of the proposed approach in real cyber-physical systems.

CCSS: Modeling and Learning Space-time Structures for Real-time Environment Perception

EVL affiliated faculty and assistant professor of Computer Science Wei Tang has been appointed by Science Direct as associate editor of the journal &ldquo;Pattern Recognition&rdquo;.  The journal publishes developments in the fields of computer vision, image processing, text and document analysis, and neural networks. Dr. Tang&rsquo;s expertise in these fields makes him the perfect candidate for this position.  We congratulate Dr. Tang on his appointment.

Wei Tang Appointed Associate Editor for &ldquo;Pattern Recognition&rdquo;

Shrut Patel

Piotr Brozek

Andrew Wentzel presents his research entitled &ldquo;Digital Twins Visualization&rdquo; at the IEEE VIS2024 Conference at 3:30 (4:30 EDT) October 17, 2024. 
Abstract: 
Digital twin models are of high interest to Head and Neck Cancer (HNC) oncologists, who have to navigate a series of complex treatment decisions that weigh the efficacy of tumor control against toxicity and mortality risks. Evaluating individual risk profiles necessitates a deeper understanding of the interplay between different factors such as patient health, spatial tumor location and spread, and risk of subsequent toxicities that can not be adequately captured through simple heuristics. To support clinicians in better understanding tradeoffs when deciding on treatment courses, we developed DITTO, a digital-twin and visual computing system that allows clinicians to analyze detailed risk profiles for each patient, and decide on a treatment plan. DITTO relies on a sequential Deep Reinforcement Learning digital twin (DT) to deliver personalized risk of both long-term and short-term disease outcome and toxicity risk for HNC patients. Based on a participatory collaborative design alongside oncologists, we also implement several visual explainability methods to promote clinical trust and encourage healthy skepticism when using our system. We evaluate the efficacy of DITTO through quantitative evaluation of performance and case studies with qualitative feedback. Finally, we discuss design lessons for developing clinical visual XAI applications for clinical end users.

EVL PhD Candidate Andrew Wentzel Presents his Research at IEEE VIS2024

Seyfal Sultanov

Hemalatha Ningappa Kondakundi

Michael E. Papka has been announced as the new Director of EVL. Mike has a long history with EVL, starting in 1994 when he graduated with his Masters degree in Computer Science - - through many long-term collaborations between Argonne National Laboratory and EVL. 
After receiving his MS at UIC, Mike completed a second MS and PhD with the University of Chicago. Mike holds a joint appointment as a full UIC CS professor and at Argonne, where he has research and senior leadership positions. 
As the Director of the Argonne Leadership Computing Facility (ACLF), Mike leads one of the U.S. Department of Energy&rsquo;s premier facilities for open science. He has spearheaded numerous efforts to build and deploy world-class computing capabilities for open science, notably advancing AI and large-scale data analysis. Under his leadership, ALCF recently joined the exascale era with the development of the Aurora supercomputer. 
We welcome Mike as EVL&rsquo;s new leader!

Michael E. Papka Named EVL Director

Congratulations to Mike Papka, one of nine speakers at <a href="https://www.tedxchicago.com/">TEDxChicago 2024: Lighting the Way</a> to be held Friday, September 27th at Harris Theater, Millennium Park fro 3pm - 7pm. 
Mike holds a joint appointment as a UIC Computer Science faculty member and serves as the Argonne Leadership Computing Facility director at Argonne National Laboratory. As director of ALCF, Mike oversees the development and deployment of <a href="https://www.anl.gov/aurora">Aurora</a>, the <a href="https://top500.org/">world&rsquo;s second exascale supercomputer</a> and the fastest <a href="https://www.intel.com/content/www/us/en/newsroom/news/intel-powered-aurora-supercomputer-breaks-exascale-barrier.html">AI supercomputer</a>. 
Mike is one of the inaugural recipients of the William S. McCulloch Collegiate Professorship of Computer Science. Warren S. McCulloch was a UIC professor of psychiatry who with Walter Pitts co-invented <a href="https://link.springer.com/article/10.1007/BF02478259">artificial neural networks in 1943</a>. Today, neural nets form the foundation of much AI technology, including deep learning and large language models. 
Mike is a professor of Computer Science, director of EVL, and co-director of the <a href="https://crabtree.uic.edu/">George Crabtree Institute for Discovery and Sustainability</a>.

Machine Learning

Bibliography

pubs (34)

Related Topics

art (1)

people (25)

events (34)

research (10)

Contact