This collaborative research project led by the colleagues at the University of Delaware aims to develop and advance tools for using image-data appearing in scientific publications, in addition to text, in order to support beneficial, targeted access to the biomedical literature. 
UIC's principal investigator, Computer Science associate professor, G. Elisabeta Marai will be responsible for the image analysis algorithms to distinguish between gel and plate images, the image analysis algorithms to distinguish between Electronic Microscopy vs Fluorescent Microscopy vs Light Microscopy and X-Rays, and the design of the visual interface to present results to end users. She will lead the design, development and implementation of the feature extraction and selection tools and techniques related to this research project.

Incorporating Image-based Features into Biomedical Document Classification

UIC PI: Maxine Brown, co-PIs: Andrew Johnson, Luc Renambot 
University of Hawaii Manoa PI: Jason Leigh 
Virginia Polytechnic PI: Chirstopher North 
NSF Award Number: 2003800 
Award Period of Performance: 05/15/2020 - 04/30/2025 
Total Amount Obligated: $2,250,000 
The Big Data revolution necessitates the use of sophisticated tools such as Artificial Intelligence (AI) and Data Visualization to harness the sheer volume, velocity and variety of datasets that are becoming the norm. However, it is the research community that must make sense of the data being amassed, so cyberinfrastructure must extend to people. SAGE3 (Smart Amplified Group Environment) puts the &ldquo;human in the loop&rdquo; by providing scientists with an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help them access, share, explore and analyze their data, come to conclusions, and make decisions with greater speed, accuracy, comprehension and confidence. 
SAGE3 builds upon the NSF-funded first-generation SAGE (Scalable Adaptive Graphics Environment) and second-generation SAGE2 (Scalable Amplified Group Environment), which are today&rsquo;s de facto operating systems to manage Big Data on tiled display walls. Its development focuses on two fundamental components: AI-enhanced &ldquo;smart&rdquo; services and advanced computing resource orchestration to support reproducible work models for secure collaborative work. SAGE3 amplifies user productivity, providing them with commercially available and open-source AI solutions, which autonomously and transparently analyze data while continually learning and improving through user interactions. SAGE3 makes AI technologies broadly accessible, not just a privilege for the technically savvy. SAGE3 further democratizes AI by using Data Visualization to help interpret and explain AI models so users better understand how AI came to its decisions, which engenders user trust and can help identify potentially prejudiced or biased models. 
SAGE3 augments every step of the scientific discovery enterprise - from quickly summarizing large data, to finding trends and similarities or anomalies among one or more linked datasets, to communicating findings to scientists, public policy and government officials, and the general public, to educating the next-generation workforce. Ultimately, it is the scientists and future scientists who must Harness the Big Data revolution to solve the nation&rsquo;s grand challenge problems that will benefit society as a whole - from studying the diversity of life on Earth, to understanding the Earth and its systems from satellite imagery of its poles, to developing response scenarios for natural disasters such as landslides and pandemics that impact the citizens and economies of the world.

SAGE3: Smart Amplified Group Environment for Harnessing the Data Revolution

SAGE2 is a widely known open-source environment for collaborative immersive analytics on large display walls. This paper briefly outlines our group&rsquo;s approach to evolving the SAGE2 software to support existing and future collaborative scientific exploration workflows through the application of Artificial Intelligence in Immersive Analytics. 
Author Keywords: Immersive Analytics, Large Display Walls, SAGE

Diagram of the scientific research, development and training enterprise

The Smart Amplified Group Environment

Researchers at University of Hawai&rsquo;i at Mānoa, University of Illinois Chicago, and Virginia Tech were awarded a $5 million National Science Foundation grant to synergize two complementary technologies - large-scale data visualization and artificial intelligence - to create the Smart Amplified Group Environment (SAGE3) open-source software. 
SAGE, soon to be on its third iteration as SAGE3, is the most widely used big-data visualization and collaboration software in the world. 
SAGE and SAGE2 are software to enable data-rich collaboration on high-resolution display walls. SAGE2 moved SAGE into cloud computing and SAGE3 ushers in the inclusion of artificial intelligence. 
Principal investigator Jason Leigh is a computer and information science professor at University of Hawai&rsquo;i at Mānoa and the inventor of SAGE. SAGE is software to enable teams of collaborators to work together with data in the form of data visualizations. 
SAGE3&rsquo;s novel interactive artificial intelligence methods will learn from human analytic activity and use the knowledge gained to assist people in synthesizing relevant data connections that lead to new hypotheses and findings, explained Chris North, a professor of computer science in the College of Engineering at Virginia Tech and a co-principal investigator. 
Scientists analyzing their data in SAGE3 will collaborate with each other and with artificial intelligence through large interactive visualization spaces, such as multi-monitor workstations, tiled display walls, and virtual reality headsets. 
Leigh said the applications for the SAGE3 technology are endless. &ldquo;It will be tremendously useful to enable evidence-based response during natural disasters and events like the current COVID-19 pandemic,&rdquo; Leigh said. 
He created SAGE in collaboration with researchers from the University of Illinois Chicago 16 years ago. They&rsquo;ve since further developed SAGE and SAGE2 with 15 years of research funding by the National Science Foundation. 
&ldquo;This new funding provides us with an opportunity to take SAGE to the next level - providing smart end-user services to amplify users&rsquo; productivity, making commercial and open-source artificial intelligence solutions available that can autonomously and transparently analyze data while continually learning and improving through user interactions,&rdquo; said Maxine Brown, director of the Electronic Visualization Laboratory at the University of Illinois Chicago. Brown has been involved with SAGE since its inception. 
&ldquo;SAGE3 also supports resource orchestration services that integrates these smart services with applications, workflows, and visualizations and collaboration services, and makes it easy for users to connect to local and remote display walls and computational systems and to securely run reproducible work models,&rdquo; explained Luc Renambot, Electronic Visualization Laboratory associate research professor and co-architect with Leigh of the SAGE software since its inception. 
&ldquo;As a &rsquo;co-pilot&rsquo; to individuals or teams who are trying to examine large datasets, SAGE3 will provide a discovery pathway from data sources to human insights and knowledge, which is currently a gap in the e-Science pipeline that the National Science Foundation has called the &rsquo;&ldquo;missing middle,&rsquo;&rdquo; said Mahdi Belcaid, assistant professor at University of Hawai&rsquo;i at Mānoa and the artificial intelligence co-principal investigator on the project. 
According to the researchers, SAGE3 will make artificial intelligence technologies broadly accessible, not just a privilege for the technically savvy. North said the project will conduct user-centered design research studies to ensure usability of this novel combination of data visualization and artificial intelligence in SAGE3. 
Through seed funds provided by Virginia Tech&rsquo;s Institute for Creativity, Arts, and Technology and the Hawai&rsquo;i Data Science Institute, North had the opportunity to work side-by-side with Leigh in the Laboratory for Advanced Visualization & Applications at the University of Hawai&rsquo;i during the spring 2020 semester. This in-person exchange, and numerous video-teleconferences with the Electronic Visualization Laboratory, was essential to helping plan and initiate the five-year SAGE3 collaboration, noted North. SAGE3 will serve the Institute&rsquo;s missions of bringing together computer scientists, domain scientists, engineers, artists, and designers to tackle some of the world&rsquo;s most complex challenges. 
The user base goes well beyond scientists. Leigh said he sees SAGE3 helping teachers use and share artificial intelligence concepts to future generations of students. And for those who work in creative media, such as filmmakers and video game designers, Leigh envisions it being a highly visual tool to brainstorm new ideas and analyze all forms of digital media to see how they relate to each other, which ones were most popular and why. 
Written by Jenise L. Jacques, Virginia Tech 
For further information or queries, contact Maxine Brown <a href="mailto:maxine@uic.edu">maxine@uic.edu</a>. 
Also, see <a href="https://cs.uic.edu/news-stories/data-visualization-gets-artificial-intelligence-boost-with-5-million-nsf-grant/">UIC&rsquo;s Computer Science News Story</a>.

An example of SAGE2 at work, using cloud-based and web-browser technologies in order to enhance data intensive co-located and remote collaboration.

Announcing SAGE3: Data visualization gets artificial intelligence boost with $5 million NSF grant

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and
have shown encouraging performance. One limitation of the vanilla graph convolution is that it models the relationships between neighboring nodes via a shared weight matrix. This is suboptimal for articulated body modeling as the relations between different body joints are different. The objective of this paper is to have a comprehensive and systematic study of weight sharing in GCNs for 3D HPE. We first show there are two different ways to interpret a GCN depending on whether feature transformation occurs before or after feature aggregation. These two interpretations lead to five different weight sharing methods and three more variants can be derived by decoupling the self-connections with other edges. We conduct extensive ablation study on these weight sharing methods under controlled settings and obtain new conclusions that will benefit the community.

Qualitative results of our approach on Human 3.6M

A Comprehensive Study of Weight Sharing in Graph Networks for 3D Human Pose Estimation

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and have demonstrated promising performance. However, since the vanilla graph convolution is performed on the one-hop neighbors of each node, it is unable to capture the long-range dependencies between body joints. They can help reduce the uncertainty caused by occlusion or depth ambiguity. To resolve this issue, we propose a high-order GCN for 3D HPE. Its core building block, termed a high-order graph convolution, aggregates features of nodes at various distances. As a result, the network can model a wide range of interactions among body joints. Furthermore, we investigate different methods to fuse those multi-order features and compare how they affect the performance. Experimental results demonstrate the effectiveness of the proposed approach.

Comparison between the vanilla graph convolution and the high-order graph convolution performed on a skeleton graph.

High-order Graph Convolutional Networks for 3D Human Pose Estimation

October 19, 2020 - - Fabio Miranda joins the EVL faculty as an assistant professor with the Department of Computer Science. Miranda interests include developing techniques that allow for the interactive visual analysis of large-scale data, combining methods from visualization, data management, machine learning, and computer graphics. Of particular concern is how visual data analytics can help address different problems cities face by integrating data on different resolutions and from different sources. 
Miranda&rsquo;s research in data science, human-computer interaction and visualization include: visual data analysis, data structures and indexing techniques for big data visualization, urban analytics, harnessing data sets of different types (images, land use information, 3D building geometry, etc.) to address concrete city problems, and machine learning. He has worked closely with domain experts from different fields, from urban planning to occupational therapy, and the outcome of these collaborations include not only research published in leading visualization, database, HCI, and AI venues, but also systems that were made available to experts in academia, industry, and government agencies. 
Miranda received his Ph.D. from New York University (2018).

Fabio Miranda Joins EVL Faculty

October 1 - - EVL Director of Research Andrew Johnson has been named Interim Director of EVL following Maxine Brown&rsquo;s retirement from UIC. Johnson now assumes the responsibility of heading EVL - overseeing EVL&rsquo;s overall mission in education, research, development, and technology transfer. 
For over 23 years, Johnson has been a valued EVL faculty member (Professor of Computer Science) and researcher who has focused upon interaction and collaboration using advanced visualization displays and the application of those displays to enhance discovery and learning. 
His areas of expertise include visualization, virtual and augmented reality, natural language processing, visual analytics, and high-performance networking. 
Over the years, Johnson has been instrumental in the development of EVL instrumentation and systems, including the Continuum smart environment, CAVE2&trade;, LambdaVision high-resolution tiled displays, LambdaTable tabletop touch system, GeoWall portable geoscience display environment, PARIS augmented reality and ImmersaDesk virtual reality systems. He has overseen the development of numerous visualization and virtual reality applications across diverse domain disciplines from science and engineering to the humanities and art. Johnson is a highly regarded mentor and educator to EVL&rsquo;s graduate students and undergraduates alike. 
Johnson retains his position as Director of Research while leading EVL&rsquo;s future initiatives and innovations as Interim Director.

Andrew Johnson Named Interim Director, EVL

Microscopy images are pervasive in biomedical research publications, where images obtained through various microscopy modalities (light, fluorescence, scanning, transmission) are often used to describe and summarize experiments and contributions. Hence, there is growing interest in automatically identifying these microscopy images’ modality and utilizing this knowledge in automated search tools. However, identifying microscopy images poses challenges due to a lack of extensive collections of labeled images. We describe and evaluate two alternative approaches to microscopy image classification. In the first approach, we progressively fine-tuned layers of ResNet models. The second approach uses shallow variants of ResNet networks, where we leverage the outputs from previous convolutional blocks. We compare these results against a Support Vector Machine (SVM)-based baseline. Our results show that fine-tuning specific layers yields better results than fine-tuning the whole model. Furthermore, shallower variants produce competitive results when compared to the entire fine-tuned model. 
Index Terms: microscopy, image classification, deep learning

Four microscopy images from the ImageCLEF dataset showcasing different modalities: (a) Electron microscopy (DMEL), (b) Transmission microscopy (DMTR), (c) Light microscopy (DMLI), and (d) Fluorescence microscopy (DMFL).

Modality-Classification of Microscopy Images Using Shallow Variants of Deep Networks

EVL faculty and Computer Science associate professor G. Elisabeta Marai (MPI) along with H. Shatkay (MPI) at the University of Delaware have been awarded supplemental funding for COVID-19 related research to their NIH research project entitled &ldquo;Uncovering Clinical Evidence in COVID-19 Publications: An Integrated Search via Text &amp; Images&rdquo;. This supplement expands the work done under current R01 funding &ldquo;Incorporating Image-based Features into Biomedical Document Classification&rdquo;, to support discovery within the CORD-19 collection -- a large, continually updated collection of more than 60,000 scientific articles pertaining to COVID-19. This supplement will create a comprehensive platform for retrieving and displaying well-focused, need-targeted medical information within COVID-19 biomedical publications. Our approach is unique compared to other text-mining methods because it leverages the image content in the dataset. 
Funding: NIH R01 NLM Covid-19 supplement to Parent Grant: 1R01LM012527 (&ldquo;Incorporating Image-based Features into Biomedical Document Classification&rdquo;)

Artificial Intelligence

Bibliography

pubs (4)

Related Topics

events (4)

research (2)

Contact