This collaborative research project led by the colleagues at the University of Delaware aims to develop and advance tools for using image-data appearing in scientific publications, in addition to text, in order to support beneficial, targeted access to the biomedical literature. 
UIC's principal investigator, Computer Science associate professor, G. Elisabeta Marai will be responsible for the image analysis algorithms to distinguish between gel and plate images, the image analysis algorithms to distinguish between Electronic Microscopy vs Fluorescent Microscopy vs Light Microscopy and X-Rays, and the design of the visual interface to present results to end users. She will lead the design, development and implementation of the feature extraction and selection tools and techniques related to this research project.

Incorporating Image-based Features into Biomedical Document Classification

UIC PI: Andrew Johnson, co-PI: Luc Renambot 
University of Hawaii Manoa PI: Jason Leigh 
Virginia Polytechnic PI: Chirstopher North 
NSF Award Number: 2003800 
Award Period of Performance: 05/15/2020 - 04/30/2025 
Total Amount Obligated: $2,250,000 
The Big Data revolution necessitates the use of sophisticated tools such as Artificial Intelligence (AI) and Data Visualization to harness the sheer volume, velocity and variety of datasets that are becoming the norm. However, it is the research community that must make sense of the data being amassed, so cyberinfrastructure must extend to people. SAGE3 (Smart Amplified Group Environment) puts the &ldquo;human in the loop&rdquo; by providing scientists with an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help them access, share, explore and analyze their data, come to conclusions, and make decisions with greater speed, accuracy, comprehension and confidence. 
SAGE3 builds upon the NSF-funded first-generation SAGE (Scalable Adaptive Graphics Environment) and second-generation SAGE2 (Scalable Amplified Group Environment), which are today&rsquo;s de facto operating systems to manage Big Data on tiled display walls. Its development focuses on two fundamental components: AI-enhanced &ldquo;smart&rdquo; services and advanced computing resource orchestration to support reproducible work models for secure collaborative work. SAGE3 amplifies user productivity, providing them with commercially available and open-source AI solutions, which autonomously and transparently analyze data while continually learning and improving through user interactions. SAGE3 makes AI technologies broadly accessible, not just a privilege for the technically savvy. SAGE3 further democratizes AI by using Data Visualization to help interpret and explain AI models so users better understand how AI came to its decisions, which engenders user trust and can help identify potentially prejudiced or biased models. 
SAGE3 augments every step of the scientific discovery enterprise - from quickly summarizing large data, to finding trends and similarities or anomalies among one or more linked datasets, to communicating findings to scientists, public policy and government officials, and the general public, to educating the next-generation workforce. Ultimately, it is the scientists and future scientists who must Harness the Big Data revolution to solve the nation&rsquo;s grand challenge problems that will benefit society as a whole - from studying the diversity of life on Earth, to understanding the Earth and its systems from satellite imagery of its poles, to developing response scenarios for natural disasters such as landslides and pandemics that impact the citizens and economies of the world. 
<a href="https://youtu.be/39x-cOQJRVc">View a video of SAGE3 being demonstrated</a> on EVL&rsquo;s YouTube channel.

SAGE3: Smart Amplified Group Environment for Harnessing the Data Revolution

SAGE2 is a widely known open-source environment for collaborative immersive analytics on large display walls. This paper briefly outlines our group&rsquo;s approach to evolving the SAGE2 software to support existing and future collaborative scientific exploration workflows through the application of Artificial Intelligence in Immersive Analytics. 
Author Keywords: Immersive Analytics, Large Display Walls, SAGE

Diagram of the scientific research, development and training enterprise

The Smart Amplified Group Environment

Researchers at University of Hawai&rsquo;i at Mānoa, University of Illinois Chicago, and Virginia Tech were awarded a $5 million National Science Foundation grant to synergize two complementary technologies - large-scale data visualization and artificial intelligence - to create the Smart Amplified Group Environment (SAGE3) open-source software. 
SAGE, soon to be on its third iteration as SAGE3, is the most widely used big-data visualization and collaboration software in the world. 
SAGE and SAGE2 are software to enable data-rich collaboration on high-resolution display walls. SAGE2 moved SAGE into cloud computing and SAGE3 ushers in the inclusion of artificial intelligence. 
Principal investigator Jason Leigh is a computer and information science professor at University of Hawai&rsquo;i at Mānoa and the inventor of SAGE. SAGE is software to enable teams of collaborators to work together with data in the form of data visualizations. 
SAGE3&rsquo;s novel interactive artificial intelligence methods will learn from human analytic activity and use the knowledge gained to assist people in synthesizing relevant data connections that lead to new hypotheses and findings, explained Chris North, a professor of computer science in the College of Engineering at Virginia Tech and a co-principal investigator. 
Scientists analyzing their data in SAGE3 will collaborate with each other and with artificial intelligence through large interactive visualization spaces, such as multi-monitor workstations, tiled display walls, and virtual reality headsets. 
Leigh said the applications for the SAGE3 technology are endless. &ldquo;It will be tremendously useful to enable evidence-based response during natural disasters and events like the current COVID-19 pandemic,&rdquo; Leigh said. 
He created SAGE in collaboration with researchers from the University of Illinois Chicago 16 years ago. They&rsquo;ve since further developed SAGE and SAGE2 with 15 years of research funding by the National Science Foundation. 
&ldquo;This new funding provides us with an opportunity to take SAGE to the next level - providing smart end-user services to amplify users&rsquo; productivity, making commercial and open-source artificial intelligence solutions available that can autonomously and transparently analyze data while continually learning and improving through user interactions,&rdquo; said Maxine Brown, director of the Electronic Visualization Laboratory at the University of Illinois Chicago. Brown has been involved with SAGE since its inception. 
&ldquo;SAGE3 also supports resource orchestration services that integrates these smart services with applications, workflows, and visualizations and collaboration services, and makes it easy for users to connect to local and remote display walls and computational systems and to securely run reproducible work models,&rdquo; explained Luc Renambot, Electronic Visualization Laboratory associate research professor and co-architect with Leigh of the SAGE software since its inception. 
&ldquo;As a &rsquo;co-pilot&rsquo; to individuals or teams who are trying to examine large datasets, SAGE3 will provide a discovery pathway from data sources to human insights and knowledge, which is currently a gap in the e-Science pipeline that the National Science Foundation has called the &rsquo;&ldquo;missing middle,&rsquo;&rdquo; said Mahdi Belcaid, assistant professor at University of Hawai&rsquo;i at Mānoa and the artificial intelligence co-principal investigator on the project. 
According to the researchers, SAGE3 will make artificial intelligence technologies broadly accessible, not just a privilege for the technically savvy. North said the project will conduct user-centered design research studies to ensure usability of this novel combination of data visualization and artificial intelligence in SAGE3. 
Through seed funds provided by Virginia Tech&rsquo;s Institute for Creativity, Arts, and Technology and the Hawai&rsquo;i Data Science Institute, North had the opportunity to work side-by-side with Leigh in the Laboratory for Advanced Visualization & Applications at the University of Hawai&rsquo;i during the spring 2020 semester. This in-person exchange, and numerous video-teleconferences with the Electronic Visualization Laboratory, was essential to helping plan and initiate the five-year SAGE3 collaboration, noted North. SAGE3 will serve the Institute&rsquo;s missions of bringing together computer scientists, domain scientists, engineers, artists, and designers to tackle some of the world&rsquo;s most complex challenges. 
The user base goes well beyond scientists. Leigh said he sees SAGE3 helping teachers use and share artificial intelligence concepts to future generations of students. And for those who work in creative media, such as filmmakers and video game designers, Leigh envisions it being a highly visual tool to brainstorm new ideas and analyze all forms of digital media to see how they relate to each other, which ones were most popular and why. 
Written by Jenise L. Jacques, Virginia Tech 
For further information or queries, contact Maxine Brown <a href="mailto:maxine@uic.edu">maxine@uic.edu</a>. 
Also, see <a href="https://cs.uic.edu/news-stories/data-visualization-gets-artificial-intelligence-boost-with-5-million-nsf-grant/">UIC&rsquo;s Computer Science News Story</a>.

An example of SAGE2 at work, using cloud-based and web-browser technologies in order to enhance data intensive co-located and remote collaboration.

Announcing SAGE3: Data visualization gets artificial intelligence boost with $5 million NSF grant

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and
have shown encouraging performance. One limitation of the vanilla graph convolution is that it models the relationships between neighboring nodes via a shared weight matrix. This is suboptimal for articulated body modeling as the relations between different body joints are different. The objective of this paper is to have a comprehensive and systematic study of weight sharing in GCNs for 3D HPE. We first show there are two different ways to interpret a GCN depending on whether feature transformation occurs before or after feature aggregation. These two interpretations lead to five different weight sharing methods and three more variants can be derived by decoupling the self-connections with other edges. We conduct extensive ablation study on these weight sharing methods under controlled settings and obtain new conclusions that will benefit the community.

Qualitative results of our approach on Human 3.6M

A Comprehensive Study of Weight Sharing in Graph Networks for 3D Human Pose Estimation

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and have demonstrated promising performance. However, since the vanilla graph convolution is performed on the one-hop neighbors of each node, it is unable to capture the long-range dependencies between body joints. They can help reduce the uncertainty caused by occlusion or depth ambiguity. To resolve this issue, we propose a high-order GCN for 3D HPE. Its core building block, termed a high-order graph convolution, aggregates features of nodes at various distances. As a result, the network can model a wide range of interactions among body joints. Furthermore, we investigate different methods to fuse those multi-order features and compare how they affect the performance. Experimental results demonstrate the effectiveness of the proposed approach.

Comparison between the vanilla graph convolution and the high-order graph convolution performed on a skeleton graph.

High-order Graph Convolutional Networks for 3D Human Pose Estimation

October 19, 2020 - - Fabio Miranda joins the EVL faculty as an assistant professor with the Department of Computer Science. Miranda interests include developing techniques that allow for the interactive visual analysis of large-scale data, combining methods from visualization, data management, machine learning, and computer graphics. Of particular concern is how visual data analytics can help address different problems cities face by integrating data on different resolutions and from different sources. 
Miranda&rsquo;s research in data science, human-computer interaction and visualization include: visual data analysis, data structures and indexing techniques for big data visualization, urban analytics, harnessing data sets of different types (images, land use information, 3D building geometry, etc.) to address concrete city problems, and machine learning. He has worked closely with domain experts from different fields, from urban planning to occupational therapy, and the outcome of these collaborations include not only research published in leading visualization, database, HCI, and AI venues, but also systems that were made available to experts in academia, industry, and government agencies. 
Miranda received his Ph.D. from New York University (2018).

Fabio Miranda Joins EVL Faculty

October 1 - - EVL Director of Research Andrew Johnson has been named Interim Director of EVL following Maxine Brown&rsquo;s retirement from UIC. Johnson now assumes the responsibility of heading EVL - overseeing EVL&rsquo;s overall mission in education, research, development, and technology transfer. 
For over 23 years, Johnson has been a valued EVL faculty member (Professor of Computer Science) and researcher who has focused upon interaction and collaboration using advanced visualization displays and the application of those displays to enhance discovery and learning. 
His areas of expertise include visualization, virtual and augmented reality, natural language processing, visual analytics, and high-performance networking. 
Over the years, Johnson has been instrumental in the development of EVL instrumentation and systems, including the Continuum smart environment, CAVE2&trade;, LambdaVision high-resolution tiled displays, LambdaTable tabletop touch system, GeoWall portable geoscience display environment, PARIS augmented reality and ImmersaDesk virtual reality systems. He has overseen the development of numerous visualization and virtual reality applications across diverse domain disciplines from science and engineering to the humanities and art. Johnson is a highly regarded mentor and educator to EVL&rsquo;s graduate students and undergraduates alike. 
Johnson retains his position as Director of Research while leading EVL&rsquo;s future initiatives and innovations as Interim Director.

Andrew Johnson Named Interim Director, EVL

Microscopy images are pervasive in biomedical research publications, where images obtained through various microscopy modalities (light, fluorescence, scanning, transmission) are often used to describe and summarize experiments and contributions. Hence, there is growing interest in automatically identifying these microscopy images’ modality and utilizing this knowledge in automated search tools. However, identifying microscopy images poses challenges due to a lack of extensive collections of labeled images. We describe and evaluate two alternative approaches to microscopy image classification. In the first approach, we progressively fine-tuned layers of ResNet models. The second approach uses shallow variants of ResNet networks, where we leverage the outputs from previous convolutional blocks. We compare these results against a Support Vector Machine (SVM)-based baseline. Our results show that fine-tuning specific layers yields better results than fine-tuning the whole model. Furthermore, shallower variants produce competitive results when compared to the entire fine-tuned model. 
Index Terms: microscopy, image classification, deep learning

Four microscopy images from the ImageCLEF dataset showcasing different modalities: (a) Electron microscopy (DMEL), (b) Transmission microscopy (DMTR), (c) Light microscopy (DMLI), and (d) Fluorescence microscopy (DMFL).

Modality-Classification of Microscopy Images Using Shallow Variants of Deep Networks

EVL faculty and Computer Science associate professor G. Elisabeta Marai (MPI) along with H. Shatkay (MPI) at the University of Delaware have been awarded supplemental funding for COVID-19 related research to their NIH research project entitled &ldquo;Uncovering Clinical Evidence in COVID-19 Publications: An Integrated Search via Text &amp; Images&rdquo;. This supplement expands the work done under current R01 funding &ldquo;Incorporating Image-based Features into Biomedical Document Classification&rdquo;, to support discovery within the CORD-19 collection -- a large, continually updated collection of more than 60,000 scientific articles pertaining to COVID-19. This supplement will create a comprehensive platform for retrieving and displaying well-focused, need-targeted medical information within COVID-19 biomedical publications. Our approach is unique compared to other text-mining methods because it leverages the image content in the dataset. 
Funding: NIH R01 NLM Covid-19 supplement to Parent Grant: 1R01LM012527 (&ldquo;Incorporating Image-based Features into Biomedical Document Classification&rdquo;)

EVL Faculty G.E. Marai Awarded NIH COVID-19 Supplement

EVL faculty and computer science (CS) associate professor Liz Marai in collaboration with co-investigator, CS assistant professor Xinhua Zhang are developing a computational artificial intelligence system to serve as a decision support tool for oncologists prescribing treatment for head and neck cancer. The research entitled &ldquo;Longitudinal Spatial-Nonspatial Decision Support for Competing Outcomes in Head and Neck Cancer Therapy&rdquo; is supported by a $2.8 million grant from the National Institutes of Health NIH grant #R01CA258827 with co-investigators at the University of Texas MD Anderson Cancer Center and the University of Iowa. The goal is to develop a scalable system based on real cancer patient data that can help guide physicians in how to treat specific patients. 
Marai and Zhang are using the computational resources at Electronic Visualization Laboratory to develop their cancer treatment AI. Guadalupe Canahuate of the University of Iowa and Dr. David Fuller of the University of Texas are co-investigators on the NIH grant. 
See the full press release on <a href="https://today.uic.edu/maximizing-cancer-survival-minimizing-treatment-side-effects-with-ai">UIC Today</a> by Jacqueline Carey.

Collaborators examine a large dataset in EVL&rsquo;s CAVE2

Maximizing cancer survival, minimizing treatment side effects with AI

Since receiving an <a href="http://sage3.sagecommons.org/">NSF award for SAGE3</a>, the Smart Amplified Group Environment, in May 2020, the SAGE3 Team has been hard at work designing and developing SAGE3. An alpha version of the SAGE3 remote collaboration platform is being developed, and the Team will give a Zoom-based SAGE3 presentation and demonstration as part of the CineGrid organization&rsquo;s Summer Community Forum series. All are welcome to attend! 
Jason Leigh and Luc Renambot will be presenting next-generation SAGE3 along with the rest of the Team. Jason and Luc are CineGrid founding members and SAGE/SAGE2/SAGE3 inventors and architects. The CineGrid organization previously focused on the use of advanced networks for digital media processing, analysis, transformation and preservation, but more recently it is focusing on Artificial Intelligence (AI) research and its relevance to digital media. 
SAGE3 is being designed to integrate AI technologies with applications, visualization workflows and collaboration services. SAGE3 takes advantage of major technological and sociological changes that have occurred since SAGE2 was introduced in 2013. Notably, web browsers and languages are maturing; scientific communities are relying on notebooks; science workloads incorporate AI programs and programmable infrastructure; and, given the pandemic, the workplace is changing to hybrid office/home environments that require not just video-teleconferencing but data-driven tools for largely distributed groups. 
All are welcome to attend the CineGrid Community Forum. There will be two SAGE3 sessions to accommodate international audiences; both sessions will present the same content, though the audience and questions may vary. 
* Tuesday, June 22 at 4:30pm - 6:00pm PDT 
* Wednesday, June 23 at 10am - 11:30am PDT 
Registration is required for each session, but there is no cost. To register, <a href="https://www.eventbrite.com/e/cinegrid-community-forum-june-tickets-156005222821">click here</a>. 
This event has been recorded and available on the <a href="https://www.youtube.com/watch?v=PyE09TxwhPE">CineGrid Community Forum archive website</a>. 
SAGE3 is being developed by the Laboratory for Advanced Visualization and Applications (LAVA) and the Hawaii Data Science Institute (HI-DSI) at the University of Hawaiʻi at Mānoa (UHM), the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC), and the Sanghani Center for AI and Data Analytics (Sanghani) at Virginia Tech (VT). SAGE3 receives major funding from the U.S. National Science Foundation, awards #2004014 (UHM), #2003800 (UIC) and #2003387 (VT). SAGE, SAGE2 and SAGE3 are trademarks of the University of Illinois Board of Trustees.

Jason Leigh, UHM, is the primary principal investigator and inventor of SAGE/SAGE2/SAGE3.

Introducing SAGE3 - Zoom event hosted by the CineGrid Community Forum

Visualization, while an effective tool for identifying patterns and insights, requires expert knowledge due to challenges faced when translating user queries to visual encodings. Research has shown that using a natural language interface (NLI) is effective for these challenges because the user can simply talk to a computer capable of producing the graphs directly. In this paper, we discuss our intelligent assistant which processes speech and hand pointing gestures while also dealing with any number of visualizations on a large screen display. Evaluation of the system shows that it is capable of quickly producing visualizations. It also particularly effective
at responding to less ambiguous queries, while in certain cases can handle ambiguous or complex queries.

The System Pipeline

Intelligent Assistant for Exploring Data Visualizations

Our goal is to develop an intelligent assistant to support users explore data via visualizations. We have collected a new corpus of conversations, CHICAGO-CRIME-VIS, geared towards supporting data visualization exploration, and we have annotated it for a variety of features, including contextualized dialogue acts. In this paper, we describe our strategies and their evaluation for dialogue act classification. We highlight how thinking aloud affects interpretation of dialogue acts in our setting and how to best capture that information. A key component of our strategy is data augmentation as applied to the training data, since our corpus is inherently small. We ran experiments with the Balanced Bagging Classifier (BAGC), Condiontal Random Field (CRF), and several Long Short Term Memory (LSTM) networks, and found that all of them improved compared to the baseline (e.g., without the data augmentation pipeline). CRF outperformed the other classification algorithms, with the LSTM networks showing modest improvement, even after obtaining a performance boost from domain-trained word embeddings. This result is of note because training a CRF is far less resource-intensive than training deep learning models, hence given a similar if not better performance, traditional methods may still be preferable in order to lower resource consumption. 
Keywords: Dialogue, Corpus, Statistical and Machine Learning Methods

Contextualized actionable requests segment relevant user utterances in each turn of the conversation.

Augmenting Small Data to Classify Contextualized Dialogue Acts for Exploratory Visualization

As the human body skeleton can be represented as a sparse graph, it is natural to exploit graph convolutional networks (GCNs) to model the articulated body structure for 3D human pose estimation (HPE). However, a vanilla graph convolutional layer, the building block of a GCN, only models the local relationships between each body joint and their neighbors on the skeleton graph. Some global attributes, e.g., the action of the person, can be critical to 3D HPE, especially in the case of occlusion or depth ambiguity. To address this issue, this paper introduces a new 3D HPE framework by learning global pose features in GCNs. Specifically, we add a global node to the graph and connect it to all the body joint nodes. On one hand, global features are updated by aggregating all body joint features to model the global attributes. On the other hand, the feature update of each body joint depends on not only their neighbors but also the global node. Furthermore, we propose a heterogeneous multi-task learning framework to learn the local and global features. While each local node regresses the 3D coordinate of the corresponding body joint, we force the global node to classify an action category or learn a low-dimensional pose embedding. Experimental results demonstrate the effectiveness of the proposed approach. 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Illustration of a graph convolutional network (GCN) with a global nodeufor 3D human pose estimation

Learning Global Pose Features in Graph Convolutional Networks for 3D Human Pose Estimation

Dependency relations among visual entities are ubiquitous because both objects and scenes are highly structured. They provide prior knowledge about the real world that can help improve the generalization ability of deep learning approaches. Different from contextual reasoning which focuses on feature aggregation in the spatial domain, visual dependency reasoning explicitly models the dependency relations among visual entities. In this paper, we introduce a novel network architecture, termed the dependency network or DependencyNet, for semantic segmentation. It unifies dependency reasoning at three semantic levels. Intra-class reasoning decouples the representations of different object categories and updates them separately based on the internal object structures. Inter-class reasoning then performs spatial and semantic reasoning based on the dependency relations among different object categories. We will have an in-depth investigation on how to discover the dependency graph from the training annotations. Global dependency reasoning further refines the representations of each object category based on the global scene information. Extensive ablative studies with a controlled model size and the same network depth show that each individual dependency reasoning component benefits semantic segmentation and they together significantly improve the base network. Experimental results on two benchmark datasets show the DependencyNet achieves comparable performance to the recent states of the art. 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Visual dependence relations are ubiquity since both ob-jects and scenes are highly structured. They provide prior knowl-edge about the real world that can be used to improve the gener-alization ability of a learning model.

Exploit Visual Dependency Relations for Semantic Segmentation

PhD Student

Kazi Shahrukh Omar

PRAGMA is hosting a Zoom-based SAGE3 presentation and live demonstration by SAGE3 Principal Investigator (PI) Jason Leigh and co-PI Luc Renambot as well as other SAGE3 Team members. In addition, SAGE2 users Prapaporn (Nan) Rattanatamrong of Thammasat University in Thailand and Jason Haga of the National Institute of Advanced Industrial Science and Technology (AIST) in Japan start the presentation with several ongoing SAGE2 use cases. 
This presentation is part of the 2021 Re-Connect Webinar series organized by the PRAGMA (Pacific Rim Application and Grid Middleware Assembly) organization. Many PRAGMA members are long-time SAGE users. 
Since receiving an NSF award for SAGE3, the Smart Amplified Group Environment, in May 2020, the SAGE3 Team has been hard at work designing and developing SAGE3. An alpha version of the SAGE3 remote collaboration platform is being presented. 
SAGE3 is being designed to integrate AI technologies with applications, visualization workflows and collaboration services. SAGE3 takes advantage of major technological and sociological changes that have occurred since SAGE2 was introduced in 2013. Notably, web browsers and languages are maturing; scientific communities are relying on notebooks; science workloads incorporate AI programs and programmable infrastructure; and, given the pandemic, the workplace is changing to hybrid office/home environments that require not just video-teleconferencing but data-driven tools for largely distributed groups. 
All are welcome to attend! No pre-registration is required. 
Asia: Aug 27 (Friday), 8am(WIB/ICT) / 9am(MYT/TST/PST) / 10am(JST/KST) 
US: Aug 26 (Thursday), 6pm(PDT) / 8pm(CDT) / 9pm(EDT) / 3pm(HST) 
Zoom: 
<a HREF="https://hawaii.zoom.us/j/96474855666?pwd=UTBNYU4wQkJnc1c2cGwxdkxCTFBCZz09
">https://hawaii.zoom.us/j/96474855666?pwd=UTBNYU4wQkJnc1c2cGwxdkxCTFBCZz09</a> 
Meeting ID: 964 7485 5666 
Passcode: sage3 
The SAGE3 presentation and demonstration were first done for the CineGrid organization in June 2021. However, if you were unable to see it then, we urge you to tune into the PRAGMA webinar and then ask questions of SAGE3 Team members. This presentation will be recorded and available at a later date. 
SAGE3 is being developed by the Laboratory for Advanced Visualization and Applications (LAVA) and the Hawaii Data Science Institute (HI-DSI) at University of Hawaiʻi at Mānoa (UHM), the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC), and the Sanghani Center for AI and Data Analytics (Sanghani) at Virginia Tech (VT). SAGE3 receives major funding from the U.S. National Science Foundation, awards #2004014 (UHM), #2003800 (UIC) and #2003387 (VT). SAGE, SAGE2 and SAGE3 are trademarks of the University of Illinois Board of Trustees.

Introducing SAGE3 - Zoom event hosted by PRAGMA

For those unable to attend the SAGE3 presentations and live demonstrations by SAGE3 Principal Investigator (PI) Jason Leigh and co-PI Luc Renambot as well as other SAGE3 Team members to PRAGMA and CineGrid earlier this summer, the sessions are now available via YouTube. 
<a href="https://youtu.be/UprsquVI6yo">Access the PRAGMA recording.</a> 
The same SAGE3 virtual presentation and demonstration was done for the CineGrid community; however, the Questions & Answers following the demo were very different. <a href="https://www.youtube.com/watch?v=PyE09TxwhPE">Access the CineGrid recording.</a> 
SAGE3 is being developed by the Laboratory for Advanced Visualization and Applications (LAVA) and the Hawaii Data Science Institute (HI-DSI) at University of Hawaiʻi at Mānoa (UHM), the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC), and the Sanghani Center for AI and Data Analytics (Sanghani) at Virginia Tech (VT). SAGE3 receives major funding from the U.S. National Science Foundation (NSF), awards #2004014 (UHM), #2003800 (UIC) and #2003387 (VT). 
SAGE, SAGE2 and SAGE3 are trademarks of the University of Illinois Board of Trustees.

SAGE3 Principal Investigator (PI) Jason Leigh, co-PI Luc Renambot and other SAGE3 Team members introduce and demo SAGE3 to the PRAGMA community as part of its 2021 Re-Connect Webinar series.

SAGE3 Introduction & Demo to PRAGMA and CineGrid - RECORDINGS AVAILABLE

Construction activities for the new $117.8 million, 135,000-square-foot Computer Design Research and Learning Center to be located at 900 W. Taylor St. is underway. The CDRLC will support the Department of Computer Science&rsquo;s growing student enrollment with 16,000 square feet of classroom space for collaborative teaching and learning environments, as well as 35 labs for faculty research activities in the computer science and engineering disciplines. 
The CDRLC will be the new home for the Electronic Visualization Laboratory featuring the latest technological and computing innovations to support research in advanced visualization, networking and collaborative environments. 
Read more about the CDRLC groundbreaking in the <a href="https://today.uic.edu/uic-breaks-ground-on-new-computer-design-research-and-learning-center">July 15th issue of UIC Today.</a> 
See a live webcam of the building&rsquo;s progress via the <a href="https://webcampub.multivista.com/index.cfm?fuseaction=aPublicWebcam.embed&WebcamPublicEmbedUID=5D4DCF05-AB2B-4770-A090-D79BE467F958">Multivista Construction Documentation webcam.</a>

New Computer Design Research and Learning Center Breaks Ground

The latest developments of machine learning (ML) and deep learning (DL) algorithms have paved the way to effectively analyze the atomic structure of chemically-complex materials. In this work, we present a DL model built upon a fully convolutional neural network (FCN) to resolve the random elements distribution of complex PtNiPdCoFe high-entropy alloys (HEAs) represented in the scanning transmission electron microscopy (STEM) images at atomic resolution. The objective of the proposed neural network is to learn through semantic seg-mentation the non-linear correlations between the pixels&rsquo; intensities of STEM images and the number of atoms of different constituent elements in the atomic columns (i.e., column heights) in the HEA&rsquo;s structure. We demonstrate that our DL model is capable of correctly estimating the column heights or with an error up to 1 atom for the majority of the columns in the HEA structures represented in the simulated STEM images used to train and test the network. This establishes a sufficiently high level of confidence in the estimation of the element distribution in experimental images. The predicted distributions in different STEM images of nanoparticles reveal inhomogeneous fluctuations with local aggregations in the elemental atomic fractions within the columns. The most pronounced aggregation is displayed by Pt, which is the largest and most electronegative element in the synthesized HEA material. The proposed DL method is beneficial for an in-depth characterization of the structural properties of HEAs and multielement 3D materials in general. 
Keywords: Deep learning, High entropy alloys, STEM images, Column heights, Elements distribution

Example of atomic model of HEA characterized by a nearly equiatomic composition of Pt, Ni, Pd, Co and Fe.

Deep learning for mapping element distribution of high-entropy alloys in scanning transmission electron microscopy images

For the last several years EVL faculty member and CS associate professor Liz Marai has been collaborating with the University of Texas M.D. Anderson Cancer Center and the University of Iowa to develop tools to help oncologists make better assessments for treating cancer patients. More recently, Liz Marai and co-investigator Xinhua Zhang also from UIC have been developing a scalable artificial intelligence system based on real patient data to serve as a decision support tool for physicians treating head and neck cancer patients. This research is funded by the National Institutes of Health, National Cancer Institute Award #3R01CA258827-01S1, entitled &ldquo;Longitudinal Spatial-Nonspatial Decision Support for Competing Outcomes in Head and Neck Cancer Therapy&rdquo; with principal investigators from UIC, the University of Texas, and the University of Iowa. 
See the full article in the <a href="https://engineering.uic.edu/about/coe-news/rise/fall-2021/better-ai-for-better-decisions/">College of Engineering&rsquo;s CEO News.</a>

EVL Artificial Intelligence Research Featured in COE News

The National Institutes of Health: National Cancer Institute has awarded Liz Marai and collaborators at the University of Texas M.D. Anderson Cancer Center a Research Supplement to Promote Diversity in Health-Related Research Programs, for federal award R01CA258827, &ldquo;Longitudinal Spatial-Nonspatial Decision Support for Competing Outcomes in Head and Neck Cancer Therapy&rdquo; to run 2021 -2024. The award will support the interdisciplinary training of Dr. Rachel Glenn, an individual from a disadvantaged background (as defined by PA-21-071), under the direction of PIs Marai (UIC) and Fuller (U of Texas).

G. Elisabeta Marai Awarded NIH Diversity Supplement

In recent years deep meta-reinforcement learning has extended the applicability of reinforcement learning (RL) algorithms: by integrating recurrent networks, trained models have the ability to quickly adapt to new unseen environments without the need for further backpropagation. These models, however, cannot adapt without having information on past rewards, and are therefore not directly applicable to a sequential decision-making setting in which multiple steps are required before observing the final reward. 
One of the main applications affected by this limitation are dynamic treatment regimes, i.e. the problem of selecting the optimal medical treatment sequence for a patient at each step, keeping into account the complete past treatment history. By expanding deep meta-reinforcement learning to handle sequential decisions, a model would be able to prescribe the optimal treatment for each patient even if the patient&rsquo;s (or physician&rsquo;s) preferences on the outcome were never encountered by the model in training. 
We propose a recursive deep meta-reinforcement learning approach which enables the model of each decision of the sequential process to learn from and adapt to unseen circumstances by recursively integrating the feedback of the models of other decisions in the process. We evaluate our approach on synthetic two-step processes with fixed transition probabilities but varying reward functions, to test the models' ability to propagate environment information from the final reward to intermediate steps. Finally, we train our model on a dataset of three-step chemo-radiotherapeutic and surgical treatment of oropharyngeal squamous cell carcinoma patients, proving our approach&rsquo;s ability to optimally handle previously unseen patient's preferences on survival and toxicity outcomes.

Main features, outcomes, and decisions of the OPC treatment sequence

Recursive meta-Reinforcement Learning for Personalized Sequential Dynamic Treatment Policies

Former EVL student and graduate research assistant Elisa Tardini is recognized as a recipient of the Outstanding Thesis/Dissertation Award from the Graduate College at the University of Illinois at Chicago for her thesis entitled &ldquo;Recursive deep meta-reinforcement learning for personalized sequential dynamic treatment policies.&rdquo; This research focuses on a recursive deep meta-reinforcement learning approach to develop a model to prescribe an optimal treatment for patients.

2021 Outstanding Thesis/Dissertation Award to Elisa Tardini

Pose guided person image generation means to generate a photo-realistic person image conditioned on an input person image and a desired pose. This task requires spatial manipulation of the source image according to the target pose. However, convolutional neural networks (CNNs) are inherently limited to geometric transformations due to the fixed geometric structures in their building modules, i.e., convolution, pooling and unpooling, which cannot handle large motion and occlusions caused by large pose transform. This paper introduces a novel two-stream context-aware appearance transfer network to address these challenges. It is a three-stage architecture consisting of a source stream and a target stream. Each stage features an appearance transfer module, a multi-scale context module and two-stream feature fusion modules. The appearance transfer module handles large motion by finding the dense correspondence between the two-stream feature maps and then transferring the appearance information from the source stream to the target stream. The multi-scale context module handles occlusion via contextual modeling, which is achieved by atrous convolutions of different sampling rates. Both quantitative and qualitative results indicate the proposed network can effectively handle challenging cases of large pose transform while retaining the appearance details. Compared with state-of-the-art approaches, it achieves comparable or superior performance using much fewer parameters while being significantly faster. 
Keywords: Person image generation, Appearance transfer, Multi-scale context, Efficient image generation 
Relevant Funding: The COMPaaS DLV project (NSF award CNS-1828265) 
<a href="https://doi.org/10.1016/j.patcog.2021.108451">https://doi.org/10.1016/j.patcog.2021.108451</a>

Exploiting appearance transfer and multi-scale context for efficient person image generation

The SAGE team has been hard at work designing and developing SAGE3, the NSF-funded Smart Amplified Group Environment, which is a total rewrite of SAGE2. 
SAGE3 ALPHA RELEASE AND WEBSITE 
<a href="https://sage3.sagecommons.org/">SAGE3 alpha version for Linux, Windows and MacOS</a> is now available for download. For those who wish to test SAGE3 and provide our team with constructive feedback, go to the new SAGE3 website: (1) download the software and display client and (2) watch the &ldquo;How To&rdquo; video. Note: Users who tested earlier SAGE3 versions should delete old versions and use the latest one on the website. 
SAGE2 
For users who rely on SAGE2 for production (collaborative research, teaching, meetings), continue to use. We will answer support questions for the foreseeable future, but will not introduce new releases. The final release and documentation can be found on the <a href="https://sage2.sagecommons.org/">SAGE2 website</a>. 
SAGE3 / SAGE2 COMMUNITY USER FORUMS 
New SAGE3/SAGE2 Community user forums are now hosted on Slack; to join: 
<a href="https://tinyurl.com/sagecommunity">tinyurl.com/sagecommunity</a>. 
<ul>
<li>For SAGE2, subscribe to the sage2-support channel.</li>
<li>For SAGE3, subscribe to the sage3-support channel.</li>
</ul>

Screen shot of SAGE3 session.

SAGE3 Alpha Version Available - Users: Please Help Test and Provide Feedback

MS Student

Veronica Grosso

Figures within biomedical articles present essential evidence of the relevance of a publication in a curation workflow. In particular, visual cues of the image modality or experimental methods can help expert curators identify relevant papers from an increasing number of publications. Automating the identification of these content-bearing images can thus be helpful in computer-assisted curation. However, the paucity of labeled datasets and the specialized training required to label such images hinder the development of such tools. To address this problem, we present the design of ANIMO, a labeling system that integrates extraction and segmentation tools to ease the annotation burden. We first introduce two taxonomies of image modalities and experimental methods, derived in collaboration with curators. On the back-end of the system, we process batches of documents and create a labeling task per document. At the front-end, expert curators can access these tasks through a web interface and access the article of interest. We describe the evaluation of this system by a group of biocurators, and the human factor lessons learned from this interdisciplinary experience. 
doi: 10.1109/BIBM52615.2021.9669898

Front-end interface for labeling a document showing a taxonomy adapted for the CORD-19 dataset.

ANIMO: Annotation of Biomed Image Modalities

Each year, over 60,000 people in the United States are diagnosed with head and neck cancers. Treatment strategies are dynamic and complex and can involve multiple courses of therapy with different potential outcomes. With head and neck cancer numbers increasing every year, partly due to viral infections, clinicians aim to develop customized treatment strategies tailored to each individual patient, under a healthcare model termed &ldquo;precision medicine&rdquo;. Precision medicine proposes the customization of healthcare, with medical decisions, practices, and products being tailored to the individual patient. The customization is based on individual factors collected from cohorts of patients who are similar to the patient under consideration. These factors may include disease markers, treatment options, demographics, and genetic, lifestyle, environmental, laboratory, or quality of life data. These data are heterogeneous, large-scale, and locally sparse. Making similarity-based precise recommendations for a specific patient requires novel approaches which blend biomedicine with complex quantitative methodology. We describe a first application of deep-Q-learning (DQL) and simulation as a digital twin dyad to optimize multi-step treatment of head and neck cancer patients, and to predict patient survival and toxicity outcomes. The treatment decision DQL digital twin and the patient’s digital twin were created, trained and evaluated using Google Colab for the HPC solution on a dataset of 536 oropharyngeal squamous cell carcinoma (OPC) patients with the goal of, respectively, determining the optimal treatment decisions with respect to survival and toxicity metrics, and predicting the outcomes of the optimal treatment on the patient. The models were trained on a subset of 402 patients (split randomly) and evaluated on a separate set of 134 patients. The main outcomes and measures include accuracy in predicting the final outcomes of a treatment sequence, and improvement in predicted outcomes rates of the DQL when compared to the observed physician outcomes. On the validation set, we obtained 87.09% mean and 90.85%
median accuracy in treatment outcome prediction. Given the prediction accuracy and predicted improvement on medically relevant outcomes yielded by this approach, this digital twin dyad of the patient-physician dynamic treatment problem has the potential of aiding physicians in determining the optimal course of treatment and in assessing its outcomes. 
Keywords: digital twin dyad, reinforcement learning, head and neck cancer

Model performance for the combined outcome (OS+DP) models without (left) and with radiomics (right).

A digital twin dyad with deep Q-learning in head and neck cancer treatment

Support vector regression (SVR) is particularly beneficial when the outcome and predictors are nonlinearly related. However, when many covariates are available, the method&rsquo;s flexibility can lead to overfitting and an overall loss in predictive accuracy. To overcome this drawback, we develop a feature selection method for SVR based on a genetic algorithm that iteratively searches across potential subsets of covariates to find those that yield the best performance according to a user-defined fitness function. We evaluate the performance of our feature selection method for SVR, comparing it to alternate methods including LASSO and random forest, in a simulation study. We find that our method yields higher predictive accuracy than SVR without feature selection. Our method outperforms LASSO when the relationship between covariates and outcome is nonlinear. Random forest performs equivalently to our method in some scenarios, but more poorly when covariates are correlated. We apply our method to predict donor kidney function 1 year after transplant using data from the United Network for Organ Sharing national registry.

Results of simulation 1 setting with 300 total covariates X, 15 of them truly associated with the outcome Y.

Feature Selection for Support Vector Regression Using a Genetic Algorithm

The COMPaaS DLV: Composable Platform as a Service Instrument for Deep Learning & Visualization user community consists of faculty from Physics and four College of Engineering departments (Computer Science, Civil, Materials & Environmental Engineering, Mechanical and Industrial Engineering, and Electrical and Computer Engineering) - primarily run applications that are GPU-centric for compute, with significant variability in storage and networking around their data requirements. 
Computer Science applications primarily focus on security, data science, computer vision and Machine Learning (ML). Security projects explore the complexity of modern web applications and the intricacies of security mechanisms that often result in flaws that expose users to significant security and privacy threats. These projects try to develop methods and tools that enable users to understand and more effectively manage retrospective privacy in the context of modern, long-lived, online archives. Composable resources were used to develop Natural Language Processing (NLP)-based domain-specific classifiers that identified data practices stated in privacy policies. Adherence of corresponding applications were then adjusted based on this ground truth. 
Data-science applications have an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help users access, share, explore and analyze their data, whether local or remote, come to conclusions, and make decisions with greater speed, accuracy, comprehensiveness and confidence. One such project is developing and advancing tools that identify image data in biomedical literature to locate beneficial, targeted publications. This work involves training image classifiers, integrating classifiers into labeling pipelines, designing retrieval user interfaces, and identifying related visual representations. 
Computer vision projects include semantic segregation and 3D human pose estimation. Researchers are developing a novel network architecture, termed DependencyNet (dependency network), for semantic segmentation. They also achieved experimental results that demonstrate an effective approach for 3D human pose estimation. Over the past two years, they found COMPaaS to be consistently stable and efficient, with result output as expected, and the group&rsquo;s models achieved state-of-the-art performance on their respective benchmarks. 
ML applications include frameworks for many complex real-world reinforcement learning problems, such as the coordination of autonomous vehicles, network packet delivery, and distributed logistics. 
Civil, Materials & Environmental Engineering applications focus on simulation and modeling. Researchers run data-driven models on high-performance computers to develop an accurate and general neural network ML model that uses crystallographic data to study patterns of synthesizability. They also perform simulations of mass transport in alloys and ceramics. COMPaaS has been performing 1.5-10 times faster than comparable infrastructures they are familiar with. Additionally, they found our use of Jupyter Notebooks to be a significant asset. 
Mechanical and Industrial Engineering researchers use COMPaaS for three research projects: feature extraction in fluid flow using a Convolutional Neural Network (CNN); column height detection in metallic nanoparticles using a CNN; and, electric vehicle battery state-of-charge estimation using different ML methods. 
Electrical and Computer Engineering researchers recently started running mathematical models of ML algorithms and training language models using long-short term memory (LSTMs). 
In 2020, we began implementing a GPUoE (GPU over Ethernet) prototype. Using a GPU expansion chassis connected to compute nodes over Ethernet, we were able to compose remote GPUs into our existing composable infrastructure. Our APIs developed for composable infrastructure have been extended to support these remote GPUs. 
In 2021, we enhanced COMPaaS with a public-facing JupyterHub server providing secure web-based access to experiments, a modern PCIe fabric and a supporting expansion chassis, thereby making composable co-location available to our users. As familiarity with composable infrastructure grows, the idea of using composable co-location instead of traditional servers co-located in racks is possible. By composable co-location, we mean the integration of emerging hardware needs without changing the existing core infrastructure. Researchers can now co-locate their own accelerators or compute nodes and leverage existing COMPaaS resources natively.

COMPaaS DLV User Community

In today&rsquo;s Big Data era, data scientists require modern workflows to quickly analyze large-scale datasets using complex codes to maintain the rate of scientific progress. These scientists often rely on available campus resources or off-the-shelf computational systems for their applications. Unified infrastructure or over-provisioned servers can quickly become
bottlenecks for specific tasks, wasting time and resources. Composable infrastructure helps solve these problems by providing users with new ways to increase resource utilization. Composable infrastructure disaggregates a computer&rsquo;s components – CPU, GPU (accelerators), storage and networking - into fluid pools of resources, but typically relies upon infrastructure engineers to
architect individual machines. Infrastructure is either managed with specialized command-line utilities, user interfaces or specification files. These management models are cumbersome and difficult to incorporate into data-science workflows. We
developed a high-level software API, Composastructure, which, when integrated into modern workflows, can be used by infrastructure engineers as well as data scientists to reorganize composable resources on demand. Composastructure enables infrastructures to be programmable, secure, persistent and reproducible. Our API composes machines, frees resources, supports multi-rack operations, and includes a Python module for Jupyter Notebooks. 
Keywords - distributed systems, testbed implementation and deployment, composable infrastructure, deep learning, visualization, infrastructure as code

Composastructure System Architecture

Moving from Composable to Programmable

Composable infrastructure holds the promise of accelerating the pace of academic research and discovery by enabling researchers to tailor the resources of a machine (e.g., GPUs, storage, NICs), on-demand, to address application needs. We were first introduced to composable infrastructure in 2018, and at the same time, there was growing demand among our College of Engineering faculty for GPU systems for data science, artificial intelligence / machine learning / deep learning, and visualization. Many purchased their own individual desktop or deskside systems, a few pursued more costly cloud and HPC solutions, and others looked to the College or campus computer center for GPU resources which, at the time, were scarce. After surveying the diverse needs of our faculty and studying product offerings by a few nascent startups in the composable infrastructure sector, we applied for and received a grant from the National Science Foundation in November 2019 to purchase a mid-scale system, configured to our specifications, for use by faculty and students for research and research training. 
This paper describes our composable infrastructure solution and implementation for our academic community. Given how modern workflows are progressively moving to containers and cloud frameworks (using Kubernetes) and to programming notebooks (primarily Jupyter), both for ease of use and for ensuring reproducible experiments, we initially adapted these tools for our system. We have since made it simpler to use our system, and now provide our users with a public facing JupyterHub server. We also added an expansion chassis to our system to enable composable co-location, which is a shared central architecture in which our researchers can insert and integrate specialized resources (GPUs, accelerators, networking cards, etc.) needed for their research. 
In February 2020, installation of our system was finalized and made operational and we began providing access to faculty in the College of Engineering. Now, two years later, it is used by over 40 faculty and students plus some external collaborators for research and research training. Their use cases and experiences are briefly described in this paper. Composable infrastructure has proven to be a useful computational system for workload variability, uneven applications, and modern workflows in academic environments. 
Keywords - composable infrastructure, deep learning, visualization, resource management, workload management, user workflow, composable co-location, infrastructure as code

COMPaaS Hardware Racks

Composable Infrastructures for an Academic Research Environment: Lessons Learned

EVL is a technology Innovator, building industry and cross-institutional connections and partnerships. In 2022, EVL deployed two GPU FIONAs in partnership with the Pacific Research Platform (PRP) and the University of California San Diego. The PRP is a partnership of more than 50 institutions that establishes a science-driven high-capacity data-centric &lsquo;freeway system&rsquo;. PRP has support from the National Science Foundation, and includes the US Department of Energy&rsquo;s Energy Science Network (ESnet), as well as scores of research universities in the nation and around the world. PRP leverages Nautilus, a HyperCluster for running containerized Big Data Applications using Kubernetes, and FIONA, flash input/output network applications. A GPU FIONA is a rack-mounted PCs designed by the PRP team that is optimized for 100Gbps large data storage with Mellanox ConnectX-6 SmartNICs, a 24 core Epic Rome CPU, 256GB of memory and 8 NVIDIA RTX-3090 GPUs. Nautilus provides over 7500 CPU cores, over 500 GPUs and 4 PBs of storage. The addition of two Nautilus GPU FIONAs at EVL builds upon our 2018 DTN (Data Transfer Node) that has provided over 300GB of Ceph Storage to Nautilus and our CAVE2 GPU Nodes that are contributing 5 NVIDIA GPUS. EVL provides Nautilus with 300GB of Ceph Storage, 128 CPU cores, and 21 NVIDIA GPUs. The platform is utilized for a wide variety of applications, including applying advanced astronomy AI to microscopy workflow, simulations for natural disasters, climate modeling, virtual reality and even accelerating NSF&rsquo;s IceCube Photon Propagation. UIC is enabled to participate in these cross-institutional connections using UIC&rsquo;s HPRNet, a network environment tailored to the needs of high-performance science applications, built on a Science DMZ architecture. 
<a href="https://pacificresearchplatform.org/media/">Read more about PRP.</a> 
<a href="https://pacificresearchplatform.org/media/sc21-larry-smarr-on-the-rise-of-supernetwork-data-intensive-computing/">See media coverage from SC21 on PRP and Nautilus.</a>

FIONA

Pacific Research Platform (PRP) - Nautilus

A collaboration of the UIC, Columbia College Chicago and the Chicago Design Archive (CDA), the &ldquo;Chicago Design Through the Decades&rdquo; project is a swift exciting journey through the history of Chicago design over the last one hundred years with content based on resources from the collection of the CDA. Following engaging characters and textual tidbits found in design works held in the archive, the journey begins in the 1920s with the era&rsquo;s painterly and illustrative techniques, evolves through the decades, and ends in the 2010s with digital rendering using AI and machine learning (ML)-technologies becoming increasingly prevalent in the contemporary landscape. At once nostalgic and whimsical, both humorous and sobering moments abound through the decades. 
The project will be showcased as part of <a href="https://artonthemart.com/">Art onTheMart</a> with previews this summer and an official opening on November 17th. 
The Chicago Design Archive is an exclusive and permanent online record of Chicago design currently holding over 3200 examples of work by over 1100 designers. 
As designers and educators, Daria Tsoupikova, Sharon Oiga and Guy Villa Jr lead a team of creators and archivists whose overarching goal is to increase the understanding and appreciation of Chicago design history and to contribute to the recognition of Chicago as an international modern design center through their use of innovative technologies. 
The project team consists of the following individuals: 
Daria Tsoupikova, UIC School of Design & EVL 
Sharon Oiga, UIC School of Design & CDA 
Guy Villa Jr, Columbia College Chicago 
Krystofer Kim, NASA 
Fabio Miranda, UIC Computer Science & EVL 
Jack Weiss, CDA 
Cheri McIntyre, CDA 
Lauren Meranda, CDA 
Louis Schwadron, Sky White Sound

Work by Choral, Marlena Novak & Jay Yim as part of Art onTheMart

Chicago Design Through the Decades

Save the date for June 11th for the preview of the &ldquo;Chicago Design Through the Decades&rdquo; project on the <a href="https://artonthemart.com/">Art ontheMART</a> at the Merchandise Mart Chicago. A collaboration between the Chicago Design Archive and EVL and School of Design professor Daria Tsoupikova and EVL assistant professor Fabio Miranda, who created the AI/ML visualization for the last contemporary decade - 2010s. 
The exhibition will showcase works from the Chicago Design Archive taking viewers on a journey that begins in the 1920s with the era&rsquo;s painterly and illustrative techniques, evolves through the decades, and ends in the 2010s with digital rendering using AI and machine learning (ML)-technologies becoming increasingly prevalent in the contemporary landscape. 
This preview will be held for one day only to align with the opening of the NeoCon Design Conference to be held at the Merchandise Mart June 13-15. This is a FREE OUTDOOR public event to be held at the corner of West Wacker Drive and North Orleans Streets on the Riverwalk.

Art ontheMart Features &ldquo;Chicago Design Through the Decades&rdquo;

In today&rsquo;s Big Data era, data scientists require modern workflows to quickly analyze large-scale datasets using complex codes to maintain the rate of scientific progress. These scientists often rely on available campus resources or off-the-shelf
computational systems for their applications. Unified infrastructure or over-provisioned servers can quickly become bottlenecks for specific tasks, wasting time and resources. Composable infrastructure helps solve these problems by providing users with new ways to increase resource utilization. Composable infrastructure disaggregates a computer&rsquo;s components - CPU, GPU (accelerators), storage and networking - into fluid pools of resources, but typically relies upon infrastructure engineers to architect individual machines. Infrastructure is either managed with specialized command-line utilities, user interfaces or specification files. These management models are cumbersome and difficult to incorporate into data-science workflows. We developed a high-level software API, Composastructure, which, when integrated into modern workflows, can be used by infrastructure engineers as well as data scientists to reorganize composable resources on demand. Composastructure enables infrastructures to be programmable, secure, persistent and reproducible. Our API composes machines, frees resources, supports multi-rack operations, and includes a Python module for Jupyter Notebooks. 
Keywords: distributed systems, testbed implementation and deployment, composable infrastructure, deep learning, visualization, infrastructure as code

Composable infrastructure holds the promise of accelerating the pace of academic research and discovery by enabling researchers to tailor the resources of a machine (e.g., GPUs, storage, NICs), on-demand, to address application needs. We were first introduced to composable infrastructure in 2018, and at the same time, there was growing demand among our College of Engineering faculty for GPU systems for data science, artificial intelligence / machine learning / deep learning, and visualization. Many purchased their own individual desktop or deskside systems, a few pursued more costly cloud and HPC solutions, and others looked to the College or campus computer center for GPU resources which, at the time, were scarce. After surveying the diverse needs of our faculty and studying product offerings by a few nascent startups in the composable infrastructure sector, we applied for and received a grant from the National Science Foundation in November 2019 to purchase a mid-scale system, configured to our specifications, for use by faculty and students for research and research training. 
This paper describes our composable infrastructure solution and implementation for our academic community. Given how modern workflows are progressively moving to containers and cloud frameworks (using Kubernetes) and to programming notebooks (primarily Jupyter), both for ease of use and for ensuring reproducible experiments, we initially adapted these tools for our system. We have since made it simpler to use our system, and now provide our users with a public facing JupyterHub server. We also added an expansion chassis to our system to enable composable co-location, which is a shared central architecture in which our researchers can insert and integrate specialized resources (GPUs, accelerators, networking cards, etc.) needed for their research. 
In February 2020, installation of our system was finalized and made operational and we began providing access to faculty in the College of Engineering. Now, two years later, it is used by over 40 faculty and students plus some external collaborators for research and research training. Their use cases and experiences are briefly described in this paper. Composable infrastructure has proven to be a useful computational system for workload variability, uneven applications, and modern workflows in academic environments. 
Keywords: composable infrastructure, deep learning, visualization, resource management, workload management, user workflow, composable co-location, infrastructure as code

APLU&rsquo;s Council on Academic Affairs (CAA) provosts, chief academic officers, and senior vice presidents for academic affairs of APLU member institutions and systems visited EVL for presentations and CAVE2 virtual reality experiences during their 2022 Summer Meeting, July 18th. The annual meeting was hosted by the University of Illinois at Chicago July 17 - 19. This event provides a unique opportunity for senior leaders to meet in a private setting to network and discuss important topics of current interest. This year, the meeting focused on the pandemic&rsquo;s impact on faculty and student performance and engagement as well as their mental health issues, preparing students for the current and future workforce needs, and emerging uses of AI in higher education. 
See the detailed program <a href="https://web.cvent.com/event/fa3fc6ed-582b-413b-a6b0-0ca84442b020/websitePage:16f39142-8fd2-4e65-9685-e11be7fd120c">Agenda.</a>

APLU Visitors in EVL&rsquo;s CAVE2

APLU&rsquo;s Council on Academic Affairs (CAA) Visit EVL

July 19th - 22nd, EVL hosted the SAGE3 research and development team to discuss the past year&rsquo;s major activities, significant results, key outcomes, other achievements, and future developments. Meeting topics include current SAGE3 usage patterns, user interface assessment and refinement, documentation improvements, future development requirements, as well as the dissemination of research results to the SAGE user community. 
&ldquo;SAGE3: Smart Amplified Group Environment for Harnessing the Data Revolution&rdquo; has been funded by the NSF CSI program to continue and advance its efforts on the SAGE collaboration and visualization framework. SAGE3 - Smart Amplified Group Environment - is the next-generation, &ldquo;human-in-the-loop&rdquo; collaboration platform with artificial intelligence (AI) enabled services that serve as a user’s &ldquo;co-pilot&rdquo; to autonomously manage and link disparate datasets and orchestrate services. Its predecessors, SAGE (Scalable Adaptive Graphics Environment) in 2004 and second-generation SAGE2 (Scalable Amplified Group Environment) in 2013, are today&rsquo;s de facto operating systems to manage Big Data on tiled display walls, enabling users to collaborate locally and remotely and access, display, share and manipulate documents and visualizations. 
SAGE3 is being developed by renowned cyberinfrastructure experts from the Laboratory for Advanced Visualization and Applications (LAVA) and the Hawaii Data Science Institute (HI-DSI) at University of Hawaiʻi at Manoa (UHM), the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC), and the Sanghani Center for AI and Data Analytics (Sanghani) at Virginia Tech (VT).

SAGE3 running on EVL Continuum Smart Space Tiled Display

EVL Hosts SAGE3 2022 All Hands Meeting

This academic year 2023 EVL alum Mike Papka has joined UIC&rsquo;s Department of Computer Science and EVL faculty as a professor, bringing years of expertise in high performance computing to campus. For nearly three decades, Papka has held the position of Research Scientist at Argonne National Laboratory contributing to research in high performance computing, scientific visualization of large and complex datasets, information visualization related to high-performance computing, virtual reality for science and the integration of edge services. 
Since 2006, he’s served as Deputy Associate Laboratory Director, Computing, Environment and Life Sciences (CELS) Directorate, providing strategic planning and management of the Laboratory Directorate comprising Biosciences, Computational Science, Data Science and Learning, Environmental Sciences, Mathematics and Computer Science, a joint institute with Northwestern University, and the Argonne Leadership Computing Facility. In his role, Papka aids in executing a research program that aims to couple computing-related activities with several science domains whose future is tied to progress in computing.
 For the last 12 years, Papka has served and continues to serve as Division Director at Argonne Leadership Computing Facility (ALCF). He is responsible for planning and executing facility upgrades and integrated data strategies, risk assessment and management, operations reporting and reviews, staff promotion and retention, and safety. The ALCF is home to some of the world&rsquo;s fastest supercomputers: Theta; a Cray XC40 supercomputer based on Intel’s Xeon Phi KNL processor and NVIDIA DGX-3, Polaris; a HPE Apollo 6500 Gen10+ based on AMD Milan processor and NVIDIA A100s, and Aurora; a Intel CPU/GPU system currently being installed that support scientific and engineering open science research. 
We welcome Mike&rsquo;s return to UIC and EVL! He undoubtedly will be a valued asset to our research and educational enterprise.

Alum Michael E. Papka Joins EVL Faculty

The SAGE3 Team has been quiet far too long, but we&rsquo;ve been very busy! Here&rsquo;s what to look forward to in the coming months: 
* SAGE3 beta version software with documentation will be released the week of the IEEE/ACM Supercomputing (SC22) conference, to be held in Dallas, Texas, USA, November 13-18, 2022. More information about the release will be forthcoming. For information on SC22, see: <a href="https://sc22.supercomputing.org/">https://sc22.supercomputing.org/</a> 
* A 1-hour SAGE3 Birds of a Feather (BoF) session and ongoing demonstrations and presentations will take place during SC22. At the BoF (Tuesday, Nov 15 from 12:15-13:15 in Room D168), the SAGE3 team will introduce the latest software release. The team will also share a booth in the research exhibition (StarLight booth #2847 as well as partner booths) and will conduct demos and presentations. More information will be forthcoming. 
* Educational SAGE3 webinars will be held, starting next year. Team members will present SAGE3&rsquo;s features and capabilities, provide use cases, answer questions, and solicit user feedback and use cases. More information will be forthcoming. 
Have you tried the SAGE3 alpha version for Linux, Windows and MacOS, which was released in December 2021? If not, go to the SAGE3 website, download the software and display client, and watch the &ldquo;How To&rdquo; video. 
<a href="https://sage3.sagecommons.org/">https://sage3.sagecommons.org/</a> 
REMINDER: SAGE3 / SAGE2 COMMUNITY USER FORUMS are now hosted on Slack. To join, go to the SAGE3 website and click on the Support menu: 
<a href="https://sage3.sagecommons.org/?page_id=60">https://sage3.sagecommons.org/?page_id=60</a> 
For SAGE3, subscribe to the sage3-support channel. 
For SAGE2, subscribe to the sage2-support channel. 
Thank you for your longstanding support of SAGE3 and we look forward to continued collaborations. 
Maxine Brown 
SAGE team member 
<a href="mailto:maxine@uic.edu">maxine@uic.edu</a> 
SAGE3 is being developed by the Laboratory for Advanced Visualization and Applications (LAVA) and the Hawaii Data Science Institute (HI-DSI) at University of Hawai&rsquo;i at Manoa (UHM), the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC), and the InfoVis Lab at Virginia Tech (VT). SAGE3 receives major funding from the U.S. National Science Foundation (NSF), awards #2004014 (UHM), #2003800 (UIC) and #2003387 (VT). 
SAGE2 received major NSF support, award #ACI-1441963, to UHM and UIC. 
SAGE, SAGE2 and SAGE3 are trademarks of the University of Illinois Board of Trustees.

Announcing SAGE3 Upcoming Events !!

Research Experience Undergraduate

Oli Kosiacki

UIC Department of Computer Science 
Doctoral Admissions Webinar 
November 2nd 2022 @7 am-8:30am Chicago time (CDT) 
Webinar: <a href="https://tinyurl.com/uicwebinar2022">https://tinyurl.com/uicwebinar2022</a> 
See what the Department of Computer Science at the University of Illinois Chicago has to offer! Located in the beautiful city of Chicago, UIC is among the most culturally diverse universities in the United States, and home to one of the most innovative computer science departments in the United States. UIC is Chicago&rsquo;s top public research university, and was recently ranked in the top 20 of the best 100 universities worldwide that are under 50 years of age. Doctoral students in their first five years of enrollment are guaranteed a fully funded appointment that includes a tuition waiver, and a stipend. Students are also eligible for summer support. Our students get jobs in academia, industry, and entertainment upon graduation, including positions at top research labs and companies such as Google, Microsoft, Apple, Disney, and Amazon. 
During this webinar, UIC Computer Science faculty will talk about our research agenda, application fee waivers, the application process, application expectations in terms of GPA/GRE etc, and will take questions. A recording of the webinar will be made available on our Graduate Admissions page. 
Apply: <a href="https://cs.uic.edu/graduate/admissions/">https://cs.uic.edu/graduate/admissions/</a> 
Deadline: December 15th 2022 
See our flyer at: <a href="https://tinyurl.com/uic2022flyer">https://tinyurl.com/uic2022flyer</a> 
 <iframe width="420" height="315"src="https://www.youtube.com/embed/eNoBbCm1skM">
</iframe>

Interested in pursuing a PhD in Computer Science in the United States?

UIC <a href="https://cs.uic.edu/undergraduate/cs-and-design/">Engineering&rsquo;s News</a> promotes the new and very popular computer science + design undergraduate degree. UIC is the only public university offering this interdisciplinary degree, which addresses the growing trend and demand for creative technologists in the job market. The CSX program, as the computer science + design degree is also known will teach topics from augmented and virtual reality to 3D fabrication. 
To see more of the Engineering&rsquo; newsletter and activities featured as the Dean&rsquo;s list see: 
<a href="https://emails.uofi.uic.edu/newsletter/669921009.html">https://emails.uofi.uic.edu/newsletter/669921009.html</a>

Computer Science + Design Major

AI competency will be an essential workforce skill. A group of computer scientists and learning science experts are considering what a foundational introduction to AI might look like for middle school and high school students. EVL affiliated faculty, Michael E. Papka, professor of computer science at UIC and deputy associate laboratory director and supercomputing facility director at the U.S. Department of Energy’s (DOE) Argonne National Laboratory, along with a like-minded group of STEM (science, technology, engineering and mathematics) educators, and computer scientists are researching the gap in future workforce skills in addressing AI problems. They began to brainstorm about a new tool, or perhaps a teaching module, that would introduce AI concepts to the young researchers of tomorrow. 
&ldquo;AI research is making major strides,&rdquo; said Papka. &ldquo;We are seeing progress in many areas of AI, made not only by new techniques, but especially by new hardware for running computationally intensive AI models.&rdquo; 
Read Laura Wolf&rsquo;s featured article <a href="https://www.anl.gov/article/science-beyond-siri-a-team-of-educators-and-computer-scientists-take-on-ai">Science beyond Siri: A team of educators and computer scientists take on AI</a> in the Argonne National Laboratory news.

Argonne National Laboratory and Northern Illinois University collaborated to host a camp that introduced middle school and high school students to artificial intelligence.

Science beyond Siri: A team of educators and computer scientists take on AI

Cluster schedulers are crucial in high-performance computing (HPC). They determine when and which user jobs should be allocated to available system resources. Existing cluster scheduling heuristics are developed by human experts based on their experience with speciﬁc HPC systems and workloads. However, the increasing complexity of computing systems and the highly dynamic nature of application workloads have placed tremendous burden on manually designed and tuned scheduling heuristics. More aggressive optimization and automation are needed for cluster scheduling in HPC. In this work, we present an automated HPC scheduling agent named DRAS (Deep Reinforcement Agent for Scheduling) by leveraging deep reinforcement learning. DRAS is built on a hierarchical neural network incorporating special HPC scheduling features such as resource reservation and backﬁlling. An efﬁcient training strategy is presented to enable DRAS to rapidly learn the target environment. Once being provided a speciﬁc scheduling objective given by the system manager, DRAS automatically learns to improve its policy through interaction with the scheduling environment and dynamically adjusts its policy as workload changes. We implement DRAS into an HPC scheduling platform called CQGym. CQGym provides a common platform allowing users to ﬂexibly evaluate DRAS and other scheduling methods such as heuristic and optimization methods. The experiments using CQGym with different production workloads demonstrate that DRAS outperforms the existing heuristic and optimization approaches by up to 50%. 
Index Terms - High-performance computing, cluster scheduling, deep reinforcement learning, job starvation, backﬁlling, resource reservation, OpenAI Gym 
<a href="https://ieeexplore.ieee.org/abstract/document/9894371">https://ieeexplore.ieee.org/abstract/document/9894371</a>

DRAS overview. DRAS agent (at the bottom) represents the scheduling agent; the environment (at the top) comprises the rest of the system, including job wait queue and HPC cluster.

DRAS: Deep Reinforcement Learning for Cluster Scheduling in High Performance Computing

Powerful detectors at modern experimental facilities routinely collect data at multiple GB/s. Online analysis methods are needed to enable the collection of only interesting subsets of such massive data streams, such as by explicitly discarding some data elements or by directing instruments to relevant areas of experimental space. Thus, methods are required for configuring and running distributed computing pipelines-what we call flows-that link instruments, computers (e.g., for analysis, simulation, artificial intelligence [AI] model
training), edge computing (e.g., for analysis), data stores, metadata catalogs, and high-speed networks. We review common patterns associated with such flows and describe methods for instantiating these patterns. 
We present experiences with the application of these methods to the processing of data from five different scientific instruments, each of which engages powerful computers for data inversion, model training, or other purposes. We also discuss implications of such methods for operators and users of scientific facilities. 
<a href="https://www.cell.com/patterns/fulltext/S2666-3899(22)00231-8">https://www.cell.com/patterns/fulltext/S2666-3899(22)00231-8</a>

Linking scientific instruments and computation: Patterns, technologies, and experiences

SAGE3 beta version software and documentation will be introduced next week at the IEEE/ACM Supercomputing (SC22) conference in Dallas, Texas, USA, November 13-18, 2022. The SAGE User Community is invited to attend the SC22 Birds of a Feather (BOF) meeting and participate in ongoing SAGE3 demonstrations in the SC22 Exhibition. 
Attend the &ldquo;SAGE 12th SC BOF: Smart Amplified Group Environment Enhanced with Artificial Intelligence for Global Collaboration&rdquo;: 
 * Tuesday, November 15, 2022 
 *12:15PM - 01:15PM CST 
 * Room: D168, Kay Bailey Hutchison Convention Center Dallas 
 * <a href="https://sc22.supercomputing.org/presentation/?id=bof141&sess=sess312">https://sc22.supercomputing.org/presentation/?id=bof141&sess=sess312</a> 
Join us for SAGE3 demonstrations in the StarLight research booth # 2847 on the exhibition floor, Tuesday-Thursday. 
SAGE3, the Smart Amplified Group Environment, is the next-generation, &ldquo;human-in-the-loop&rdquo; collaboration platform, providing HPC users with the tools to access, explore, discover, publish, share, integrate, and reuse complex datasets. SAGE3 supports today&rsquo;s hybrid work/home environments (laptops, single monitors and display walls), and interfaces with a variety of computational infrastructures, workflows, notebooks, and analytics software through Artificial Intelligence-enabled services and orchestration services. It lowers the barrier of entry into AI for non-expert users, democratizing access to disruptive technologies for those with varying skills. 
The BOF will not be streamed. SAGE3 software and documentation will be available for download on the SAGE3 website. The SAGE3 team plans to hold SAGE3 educational webinars, starting next year. 
REMINDER: SAGE3 / SAGE2 COMMUNITY USER FORUMS are now hosted on Slack. To join, go to the <a href="https://sage3.sagecommons.org/">SAGE3 website</a> and click on the Support menu. 
 * For SAGE3, subscribe to the sage3-support channel. 
 * For SAGE2, subscribe to the sage2-support channel. 
Thank you for your longstanding support of SAGE and we look forward to continued collaborations. 
Maxine Brown 
SAGE team member 
<a href="mailto:maxine@uic.edu">maxine@uic.edu</a> 
SAGE3 is being developed by the Laboratory for Advanced Visualization and Applications (LAVA) and the Hawaii Data Science Institute (HI-DSI) at University of Hawaiʻi at Mānoa (UHM), the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC), and the InfoVis Lab at Virginia Tech (VT). SAGE3 receives major funding from the U.S. National Science Foundation (NSF), awards #2004014 (UHM), #2003800 (UIC) and #2003387 (VT). 
SAGE2 received major NSF support, award #ACI-1441963, to UHM and UIC. 
SAGE, SAGE2 and SAGE3 are trademarks of the University of Illinois Board of Trustees.

INTRODUCING SAGE3 at the SC22 - Tuesday, November 15, 12:15pm - 1:15pm

Art on theMART&rsquo;s newest projection, Chicago Design Through the Decades, which debuted November 18 and can be viewed nightly through December 30, isn&rsquo;t the work of a single artist but a century-spanning celebration-slash-compendium of Chicago design, beginning with the city&rsquo;s art deco heyday in the 1920s and ending in contemporary digital portraiture produced using neural networks, a subset of deep learning technology with roots at the University of Illinois Chicago (UIC) and the University of Chicago. 
Chicago Design Through the Decades is a cross-institutional collaborative effort led by Daria Tsoupikova, UIC School of Design professor and EVL faculty; Sharon Oiga, UIC School of Design professor; and Guy Villa, Jr., Columbia College Chicago assistant professor. Additional contributors to the project are CDA directors Jack Weiss, Cheri McIntyre, and Lauren Meranda, along with NASA science animator Krystofer Kim, and the EVL&rsquo;s Fabio Miranda.
The projection is based on the 3,250-work collection of the <a href="https://chicagodesignarchive.org/">Chicago Design Archive (CDA)</a> and is on view nightly through December 30 at 7:30 and 8:00 p.m. 
Read the full article <a href="https://www.archpaper.com/2022/11/century-chicago-design-history-art-deco-facade-merchandise-mart/">&ldquo;A century of Chicago design history comes to life on the sprawling art deco facade of the iconic Merchandise Mart&rdquo;</a> by Matt Hickman. 
See the <a href="https://designobserver.com/">Design Observer</a> December posting.

The Architect&rsquo;s Newspaper and The Design Observer Feature A Century of Chicago Design History

NOVEMBER 14-17, 2022 -- SAGE3 Team members announced the release of the SAGE3 alpha version at the IEEE/ACM Supercomputing 2022 (SC22) conference, held at the Kay Bailey Hutchison Convention Center in Dallas, Texas, USA, November 14-18, 2022. Software and documentation are available on the SAGE3 website: 
<a href="https://sage3.sagecommons.org/">https://sage3.sagecommons.org/</a> 
The Team conducted ongoing SAGE3 demonstrations in the SC22 Exhibition and held its 12th SC Birds of a Feather (BOF) meeting to introduce current and potential users to SAGE3 features. You can learn about our SC22 activities and link to the BOF presentation, videos and photographs from the SAGE3 NEWS website: 
<a href="https://sage3.sagecommons.org/?p=807#more-807">https://sage3.sagecommons.org/?p=807#more-807</a> 
Note: Several long-time users have asked when server and client code will be released. The SAGE3 roadmap (see BOF presentation) targets February 1, 2023 for the SAGE3 Server release and March 1, 2023 for source code release. More information will be forthcoming. 
We hope you enjoy using SAGE3! Your feedback is always welcomed. 
REMINDER: SAGE3 / SAGE2 COMMUNITY USER FORUMS are now hosted on Slack. To join, go to the SAGE3 website and click on the Support menu. 
 * For SAGE3, subscribe to the sage3-support channel. 
 * For SAGE2, subscribe to the sage2-support channel. 
Maxine Brown 
SAGE team member 
<a href="mailto:maxine@uic.edu">maxine@uic.edu</a> 
SAGE3 is being developed by the Laboratory for Advanced Visualization and Applications (LAVA) and the Hawaii Data Science Institute (HI-DSI) at University of Hawaiʻi at Mānoa (UHM), the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC), and the InfoVis Lab at Virginia Tech (VT). SAGE3 receives major funding from the U.S. National Science Foundation (NSF), awards #2004014 (UHM), #2003800 (UIC) and #2003387 (VT). 
SAGE2 received major NSF support, award #ACI-1441963, to UHM and UIC. 
SAGE, SAGE2 and SAGE3 are trademarks of the University of Illinois Board of Trustees.

Artificial Intelligence

Bibliography

pubs (20)

Related Topics

art (1)

people (4)

events (24)

research (4)

Contact