Deep learning is often referred to as a &lsquo;black box&rsquo;, because, aside from selecting model and the model hyperparameters (like the learning rate, activation function parameters), users don&rsquo;t have access to what happens inside the model. We can know that the model works by measuring its accuracy on some testing data set, suggesting that the model must have &lsquo;learned&rsquo; something meaningful from the training data. But how can we peek inside and &lsquo;see&rsquo; what the model &lsquo;sees&rsquo;? 
Biologists have recently begun to explore how deep learning approaches could be used to address questions involving diverse biological datasets. For this community, producing an accurate model is important, but also insufficient, since the larger goal is to contribute to our understanding of complex biological systems and phenomena. Visualization could help biology researchers &lsquo;peak&rsquo; inside the black box and relate deep learning mechanics to existing biological knowledge. However, most visualization techniques for deep learning have been developed for images and text, so many of the encodings and layouts are designed for input data with fundamentally different perceptual properties than biological data sets, such as gene sequence data or gene expression data. 
This research explores current approaches in visualization for deep learning and how these approaches may (or may not) be applied to biological data. This work is conducted with researchers at Argonne National Labs studying deep learning for cancer research and genomics.

Argonne Leadership Computing Facility

Vis for Deep Learning for Biology

This collaborative research project led by the colleagues at the University of Delaware aims to develop and advance tools for using image-data appearing in scientific publications, in addition to text, in order to support beneficial, targeted access to the biomedical literature. 
UIC's principal investigator, Computer Science associate professor, G. Elisabeta Marai will be responsible for the image analysis algorithms to distinguish between gel and plate images, the image analysis algorithms to distinguish between Electronic Microscopy vs Fluorescent Microscopy vs Light Microscopy and X-Rays, and the design of the visual interface to present results to end users. She will lead the design, development and implementation of the feature extraction and selection tools and techniques related to this research project.

Incorporating Image-based Features into Biomedical Document Classification

Data &Time: Friday, May 4th, 2018, 11 am - 1pm 
Location: Room 3036 ERF 
Defense committee: 
Angus G. Forbes (Chair and Advisor) 
Andrew Johnson 
G. Elisabeta Marai 
Abstract: 
Simulating real-life light in a virtual world is computationally very expensive even in a limited capacity. The quality of images synthesized using computer graphics algorithms for offline rendering used in movies has increased significantly over the years. Real-time applications such as games still rely on non-physics based approximation methods for computing light to meet the time budget to render a frame. The demand for higher quality graphics for more realism in games is high which requires a better approximation technique. Acceleration data structures are used for speed up the rendering process by skipping the unwanted parts of the virtual scene for light calculations. Even with such structures rendering can take a very long time depending on the size of the 3D scene. Recently, deep learning approaches have shown great success in image processing problems. Although neural networks are extensively used for training AI agents and other parts of a game, only a handful of research exists in the 3D rendering space. 
In this work, I explore deep learning based approaches to 1) approximate indirect illumination and soft shadow from image space buffers, and 2) build acceleration structure for faster rendering. I also show evaluations of these networks to get insights about their performance and limitations.

MS Thesis Defense Announcement: &ldquo;Exploring Deep Learning Techniques for Real-time Graphics&rdquo;

We introduce a deep learning approach for the identification of shock locations in large-scale tensor field datasets. Such datasets are typically generated by turbulent combustion simulations. In this proof of concept approach, we use deep learning to learn mappings from strain tensors to Schlieren images which serve as labels. The use of neural networks allows for the Schlieren values to be approximated more efficiently than calculating the values from the density gradient. In addition, we show that this approach can be used to predict the Schlieren values for both two-dimensional and three-dimensional tensor fields, potentially allowing for anomaly detection in tensor flows. Results on two shock example datasets show that this approach can assist in the extraction of features from reacting flow tensor fields. 
Support: 
NSF CAREER IIS-1541277

Networks for 3D Sod dataset (left) and the 2D Blast dataset (right).

A Deep Learning Approach to Identifying Shock Locations in Turbulent Combustion Tensor Fields

Research Experience Undergraduate (REU)

Zhongyi Chen

Animation studios render 3D scenes using a technique called path tracing which enables them to create high quality photorealistic frames. Path tracing involves shooting 1000&rsquo;s of rays into a pixel randomly (Monte Carlo) which will then hit the objects in the scene and, based on the reflective property of the object, these rays reflect or refract or get absorbed. The colors returned by these rays are averaged to determine the color of the pixel. This process is repeated for all the pixels. Due to the computational complexity it might take 8-16 hours to render a single frame. We implemented a neural network-based solution to reduce the time it takes to render a frame to less than a second using a generative adversarial network (GAN), once the network is trained. The main idea behind this proposed method is to render the image using a much smaller number of samples per pixel than is normal for path tracing (e.g., 1, 4, or 8 samples instead of, say, 32,000 samples) and then pass the noisy, incompletely rendered image to our network, which is capable of generating a high-quality photorealistic image. 
Keywords: image denoising, rendering, deep learning

Gaussian noise image (left), denoised image (middle) and ground truth photorealistic image (right).

Image Denoising Using A Generative Adversarial Network

Maxine Brown (UIC PI), Andrew Johnson (UIC co-PI), Luc Renambot (UIC co-PI) 
Jason Leigh (University of Hawaii Manoa PI) Christopher North (Virginia Polytechnic PI) 
NSF Award Number: 2003800 
Award Period of Performance: 05/15/2020 - 04/30/2025 
Total Amount Obligated: $2,250,000 
UIC Computer Science Electronic Visualization Lab (EVL) Director Maxine Brown, PI, co-PIs UIC CS Professor Andy Johnson and UIC CS Research Associate Professor Luc Renambot, and Senior Personnel EVL Senior Research Programmer Lance Long are awarded a $2.25 million dollar National Science Foundation grant entitled &ldquo;Collaborative Research: CSSI Frameworks: SAGE3: Smart Amplified Group Environment for Harnessing the Data Revolution.&rdquo; The grant runs from May 2020 to April 2025. It&rsquo;s part of an overall $5 million project, that also includes a team from University of Hawaii Manoa led by Jason Leigh and a team from Virginia Polytechnic led by Christopher North. 
ABSTRACT 
The Big Data revolution necessitates the use of sophisticated tools such as Artificial Intelligence (AI) and Data Visualization to harness the sheer volume, velocity and variety of datasets that are becoming the norm. However, it is the research community that must make sense of the data being amassed, so cyberinfrastructure must extend to people. SAGE3 (Smart Amplified Group Environment) puts the &ldquo;human in the loop&rdquo; by providing scientists with an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help them access, share, explore and analyze their data, come to conclusions, and make decisions with greater speed, accuracy, comprehension and confidence. 
SAGE3 builds upon the NSF-funded first-generation SAGE (Scalable Adaptive Graphics Environment) and second-generation SAGE2 (Scalable Amplified Group Environment), which are today&rsquo;s de facto operating systems to manage Big Data on tiled display walls. Its development focuses on two fundamental components: AI-enhanced &ldquo;smart&rdquo; services and advanced computing resource orchestration to support reproducible work models for secure collaborative work. SAGE3 amplifies user productivity, providing them with commercially available and open-source AI solutions, which autonomously and transparently analyze data while continually learning and improving through user interactions. SAGE3 makes AI technologies broadly accessible, not just a privilege for the technically savvy. SAGE3 further democratizes AI by using Data Visualization to help interpret and explain AI models so users better understand how AI came to its decisions, which engenders user trust and can help identify potentially prejudiced or biased models. 
SAGE3 augments every step of the scientific discovery enterprise - from quickly summarizing large data, to finding trends and similarities or anomalies among one or more linked datasets, to communicating findings to scientists, public policy and government officials, and the general public, to educating the next-generation workforce. Ultimately, it is the scientists and future scientists who must Harness the Big Data revolution to solve the nation&rsquo;s grand challenge problems that will benefit society as a whole - from studying the diversity of life on Earth, to understanding the Earth and its systems from satellite imagery of its poles, to developing response scenarios for natural disasters such as landslides and pandemics that impact the citizens and economies of the world.

UIC Computer Science/EVL awarded $2.25 Million from NSF for SAGE3

UIC PI: Maxine Brown, co-PIs: Andrew Johnson, Luc Renambot 
University of Hawaii Manoa PI: Jason Leigh 
Virginia Polytechnic PI: Chirstopher North 
NSF Award Number: 2003800 
Award Period of Performance: 05/15/2020 - 04/30/2025 
Total Amount Obligated: $2,250,000 
The Big Data revolution necessitates the use of sophisticated tools such as Artificial Intelligence (AI) and Data Visualization to harness the sheer volume, velocity and variety of datasets that are becoming the norm. However, it is the research community that must make sense of the data being amassed, so cyberinfrastructure must extend to people. SAGE3 (Smart Amplified Group Environment) puts the &ldquo;human in the loop&rdquo; by providing scientists with an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help them access, share, explore and analyze their data, come to conclusions, and make decisions with greater speed, accuracy, comprehension and confidence. 
SAGE3 builds upon the NSF-funded first-generation SAGE (Scalable Adaptive Graphics Environment) and second-generation SAGE2 (Scalable Amplified Group Environment), which are today&rsquo;s de facto operating systems to manage Big Data on tiled display walls. Its development focuses on two fundamental components: AI-enhanced &ldquo;smart&rdquo; services and advanced computing resource orchestration to support reproducible work models for secure collaborative work. SAGE3 amplifies user productivity, providing them with commercially available and open-source AI solutions, which autonomously and transparently analyze data while continually learning and improving through user interactions. SAGE3 makes AI technologies broadly accessible, not just a privilege for the technically savvy. SAGE3 further democratizes AI by using Data Visualization to help interpret and explain AI models so users better understand how AI came to its decisions, which engenders user trust and can help identify potentially prejudiced or biased models. 
SAGE3 augments every step of the scientific discovery enterprise - from quickly summarizing large data, to finding trends and similarities or anomalies among one or more linked datasets, to communicating findings to scientists, public policy and government officials, and the general public, to educating the next-generation workforce. Ultimately, it is the scientists and future scientists who must Harness the Big Data revolution to solve the nation&rsquo;s grand challenge problems that will benefit society as a whole - from studying the diversity of life on Earth, to understanding the Earth and its systems from satellite imagery of its poles, to developing response scenarios for natural disasters such as landslides and pandemics that impact the citizens and economies of the world.

SAGE3: Smart Amplified Group Environment for Harnessing the Data Revolution

PhD Graduate Student

Mir Masood Ali

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and
have shown encouraging performance. One limitation of the vanilla graph convolution is that it models the relationships between neighboring nodes via a shared weight matrix. This is suboptimal for articulated body modeling as the relations between different body joints are different. The objective of this paper is to have a comprehensive and systematic study of weight sharing in GCNs for 3D HPE. We first show there are two different ways to interpret a GCN depending on whether feature transformation occurs before or after feature aggregation. These two interpretations lead to five different weight sharing methods and three more variants can be derived by decoupling the self-connections with other edges. We conduct extensive ablation study on these weight sharing methods under controlled settings and obtain new conclusions that will benefit the community.

Qualitative results of our approach on Human 3.6M

A Comprehensive Study of Weight Sharing in Graph Networks for 3D Human Pose Estimation

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and have demonstrated promising performance. However, since the vanilla graph convolution is performed on the one-hop neighbors of each node, it is unable to capture the long-range dependencies between body joints. They can help reduce the uncertainty caused by occlusion or depth ambiguity. To resolve this issue, we propose a high-order GCN for 3D HPE. Its core building block, termed a high-order graph convolution, aggregates features of nodes at various distances. As a result, the network can model a wide range of interactions among body joints. Furthermore, we investigate different methods to fuse those multi-order features and compare how they affect the performance. Experimental results demonstrate the effectiveness of the proposed approach.

Comparison between the vanilla graph convolution and the high-order graph convolution performed on a skeleton graph.

High-order Graph Convolutional Networks for 3D Human Pose Estimation

October 19, 2020 - - Fabio Miranda joins the EVL faculty as an assistant professor with the Department of Computer Science. Miranda interests include developing techniques that allow for the interactive visual analysis of large-scale data, combining methods from visualization, data management, machine learning, and computer graphics. Of particular concern is how visual data analytics can help address different problems cities face by integrating data on different resolutions and from different sources. 
Miranda&rsquo;s research in data science, human-computer interaction and visualization include: visual data analysis, data structures and indexing techniques for big data visualization, urban analytics, harnessing data sets of different types (images, land use information, 3D building geometry, etc.) to address concrete city problems, and machine learning. He has worked closely with domain experts from different fields, from urban planning to occupational therapy, and the outcome of these collaborations include not only research published in leading visualization, database, HCI, and AI venues, but also systems that were made available to experts in academia, industry, and government agencies. 
Miranda received his Ph.D. from New York University (2018).

Fabio Miranda Joins EVL Faculty

October 1 - - EVL Director of Research Andrew Johnson has been named Interim Director of EVL following Maxine Brown&rsquo;s retirement from UIC. Johnson now assumes the responsibility of heading EVL - overseeing EVL&rsquo;s overall mission in education, research, development, and technology transfer. 
For over 23 years, Johnson has been a valued EVL faculty member (Professor of Computer Science) and researcher who has focused upon interaction and collaboration using advanced visualization displays and the application of those displays to enhance discovery and learning. 
His areas of expertise include visualization, virtual and augmented reality, natural language processing, visual analytics, and high-performance networking. 
Over the years, Johnson has been instrumental in the development of EVL instrumentation and systems, including the Continuum smart environment, CAVE2&trade;, LambdaVision high-resolution tiled displays, LambdaTable tabletop touch system, GeoWall portable geoscience display environment, PARIS augmented reality and ImmersaDesk virtual reality systems. He has overseen the development of numerous visualization and virtual reality applications across diverse domain disciplines from science and engineering to the humanities and art. Johnson is a highly regarded mentor and educator to EVL&rsquo;s graduate students and undergraduates alike. 
Johnson retains his position as Director of Research while leading EVL&rsquo;s future initiatives and innovations as Interim Director.

Andrew Johnson Named Interim Director, EVL

For the Supercomputing (SC20) Conference this year, a research community partnership was formed to design an international networking testbed using existing facilities, with several international extensions. Demonstrations and experiments were conducted prior to the conference with results discussed during a SC20 virtual forum. &ldquo;The Forum For Experiments and Demonstrations Showcasing Innovations In Large-Scale, Data-Intensive-Science Networking&rdquo; was organized by Research Consortium for Data-Intensive Science Networking to communicate: a) the motivation behind these experiments and demonstrations, b) their design, c) technologies used, d) results and e) future research in these areas. 
EVL senior research programmer, Lance Long participated to discuss EVL&rsquo;s research in Composable Infrastructure, specifically MRI: Acquisition of a Composable Platform as a Service Instrument for Deep Learning & Visualization (COMPaaS DLV) funded under NSF Award #CNS-1828265. Items discussed included the benefits of composable infrastructure, implementation for deep learning in an academic research environment, operating systems, dynamic workflows, orchestration middleware, current use cases and future work. 
Other participants included: Ciena Research Lab, NASA Goddard Space Flight Center (GSFC), Naval Research Laboratory, iCAIR/Northwestern, StarLight, UCSD, FNAL, CalTech, SURFnet.

Composable Infrastructure for Data Science

Lance Long Presents EVL&rsquo;s Composable Infrastructure Research at SC20

Microscopy images are pervasive in biomedical research publications, where images obtained through various microscopy modalities (light, fluorescence, scanning, transmission) are often used to describe and summarize experiments and contributions. Hence, there is growing interest in automatically identifying these microscopy images’ modality and utilizing this knowledge in automated search tools. However, identifying microscopy images poses challenges due to a lack of extensive collections of labeled images. We describe and evaluate two alternative approaches to microscopy image classification. In the first approach, we progressively fine-tuned layers of ResNet models. The second approach uses shallow variants of ResNet networks, where we leverage the outputs from previous convolutional blocks. We compare these results against a Support Vector Machine (SVM)-based baseline. Our results show that fine-tuning specific layers yields better results than fine-tuning the whole model. Furthermore, shallower variants produce competitive results when compared to the entire fine-tuned model. 
Index Terms: microscopy, image classification, deep learning

Four microscopy images from the ImageCLEF dataset showcasing different modalities: (a) Electron microscopy (DMEL), (b) Transmission microscopy (DMTR), (c) Light microscopy (DMLI), and (d) Fluorescence microscopy (DMFL).

Modality-Classification of Microscopy Images Using Shallow Variants of Deep Networks

EVL faculty and Computer Science associate professor G. Elisabeta Marai (MPI) along with H. Shatkay (MPI) at the University of Delaware have been awarded supplemental funding for COVID-19 related research to their NIH research project entitled &ldquo;Uncovering Clinical Evidence in COVID-19 Publications: An Integrated Search via Text &amp; Images&rdquo;. This supplement expands the work done under current R01 funding &ldquo;Incorporating Image-based Features into Biomedical Document Classification&rdquo;, to support discovery within the CORD-19 collection -- a large, continually updated collection of more than 60,000 scientific articles pertaining to COVID-19. This supplement will create a comprehensive platform for retrieving and displaying well-focused, need-targeted medical information within COVID-19 biomedical publications. Our approach is unique compared to other text-mining methods because it leverages the image content in the dataset. 
Funding: NIH R01 NLM Covid-19 supplement to Parent Grant: 1R01LM012527 (&ldquo;Incorporating Image-based Features into Biomedical Document Classification&rdquo;)

Deep Learning

Bibliography

pubs (5)

Related Topics

people (2)

events (6)

research (3)

Contact