PI: Andrew Johnson (UIC/EVL); co-PIs: Barbara DiEugenio (UIC/CS), Moira Zellner (Norheastern University) 
Collaborator: Jason Leigh (University of Hawaii Manoa) 
The Internet gives the public, including scientists, access to vast amounts of data. Visualization is a powerful tool to take that data and turn it into a form that is easier to understand, helping people make better decisions. However, creating good visualizations from data is not easy. The Articulate+ Project will work to generate new knowledge about how to empower people with information visualization, by using speech and gesture as inputs, in new ways. A goal is to enable users to benefit from imprecise statements about what they want, rather than needing to give specific commands telling the computer what to do. This project will make it easier for citizens to engage in conversations with the computer about economic, medical data, demographic, transportation data, climate, and sports data; to answers that are more meaningful, and so to democratize data literacy. 
This research will use speech, gesture, and log data as new input interaction modalities for visual analytics. It will develop new understandings of how the imprecise and vague nature of natural language queries and gestures, as contextualized by work on a specific visual analysis problem, can be modeled as an opportunity for an intelligent software system to learn more about the underlying intent of the users. This model of intent, in turn, will be used to provide contextualized visualizations, which are expected to help those users gain valuable insights from their data. The audio speech data will be used to computationally model overhearing, to help infer users&rsquo; current contextualized goals. The gesture data will be used to disambiguate expressions that refer to visual elements of a visualization (e.g., &ldquo;that pie chart about electricity consumption&rdquo;). The project will develop new understandings of how to combine the log of visualization state, the computational model of overhearing, and the gesture data in order to generate new visualizations in support of users&rsquo; work on tasks. Evaluations will be performed in both controlled laboratory and naturalistic study settings to determine whether effective semantic parsers can be derived for these specific visual analysis domains, and how this contextual interface affects users&rsquo; experience of visualization and discovery. The integrated annotated datasets from the studies will be made available to the research community satisfying a need for ecologically valid situated language interaction corpora.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Articulate+ - A Conversational Interface for Democratizing Visual Analysis

UIC PI: Maxine Brown, co-PIs: Andrew Johnson, Luc Renambot 
University of Hawaii Manoa PI: Jason Leigh 
Virginia Polytechnic PI: Chirstopher North 
NSF Award Number: 2003800 
Award Period of Performance: 05/15/2020 - 04/30/2025 
Total Amount Obligated: $2,250,000 
The Big Data revolution necessitates the use of sophisticated tools such as Artificial Intelligence (AI) and Data Visualization to harness the sheer volume, velocity and variety of datasets that are becoming the norm. However, it is the research community that must make sense of the data being amassed, so cyberinfrastructure must extend to people. SAGE3 (Smart Amplified Group Environment) puts the &ldquo;human in the loop&rdquo; by providing scientists with an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help them access, share, explore and analyze their data, come to conclusions, and make decisions with greater speed, accuracy, comprehension and confidence. 
SAGE3 builds upon the NSF-funded first-generation SAGE (Scalable Adaptive Graphics Environment) and second-generation SAGE2 (Scalable Amplified Group Environment), which are today&rsquo;s de facto operating systems to manage Big Data on tiled display walls. Its development focuses on two fundamental components: AI-enhanced &ldquo;smart&rdquo; services and advanced computing resource orchestration to support reproducible work models for secure collaborative work. SAGE3 amplifies user productivity, providing them with commercially available and open-source AI solutions, which autonomously and transparently analyze data while continually learning and improving through user interactions. SAGE3 makes AI technologies broadly accessible, not just a privilege for the technically savvy. SAGE3 further democratizes AI by using Data Visualization to help interpret and explain AI models so users better understand how AI came to its decisions, which engenders user trust and can help identify potentially prejudiced or biased models. 
SAGE3 augments every step of the scientific discovery enterprise - from quickly summarizing large data, to finding trends and similarities or anomalies among one or more linked datasets, to communicating findings to scientists, public policy and government officials, and the general public, to educating the next-generation workforce. Ultimately, it is the scientists and future scientists who must Harness the Big Data revolution to solve the nation&rsquo;s grand challenge problems that will benefit society as a whole - from studying the diversity of life on Earth, to understanding the Earth and its systems from satellite imagery of its poles, to developing response scenarios for natural disasters such as landslides and pandemics that impact the citizens and economies of the world.

SAGE3: Smart Amplified Group Environment for Harnessing the Data Revolution

PI: Maureen Dunne, Disability and Human Development, UIC 
co-PI: Tamar Heller, Disability and Human Development, UIC 
co-PI: Daria Tsoupikova, New Media Design; Electronic Visualization Laboratory, UIC 
In recent years, the shortage of technology talent in Illinois and the outmigration of engineers to other states has been identified as a major concern. It is also estimated that approximately 85 percent of people with autism spectrum disorder (ASD) remain unemployed or underemployed. ASD is the fastest growing developmental disability and is diagnosed more frequently than AIDS, cancer and diabetes combined, with 1 in 59 children now being diagnosed (CDC, 2018). The mission of the Autism and Innovation Initiative is to strengthen the technology and innovation talent pipeline in Illinois by including people with autism spectrum disorders (ASD) and taking a strength-based approach to training, entrepreneurship and workforce development. 
In pioneering an inclusive model to training, attracting and retaining talent in Illinois, the program will offer opportunities for engagement with innovative technology projects to inspire enthusiasm for technology, innovation and entrepreneurship. The Autism and Innovation Initiative, in collaboration with industry and community partners, will spearhead the foundation of a statewide Innovation Hub that takes a strength-based approach to including individuals with autism in the Illinois STEM workforce. 
The initial milestones this initiative will accomplish include 1) Coding Bootcamps and Hackathons in different regions of Illinois; 2) STEM Faculty Training Workshop on ASD; 3) Community College Bridge to UIC Program to Support STEM Transfer Students with Autism; and 4) Autism and Technology Working Research Group and Seminar Courses to drive interdisciplinary collaborations, innovative technology solutions and entrepreneurship. While a core focus is on integrating persons with autism into the Illinois workforce, the statewide Innovation Hub seeks to invigorate the overall technology pipeline and retain engineering talent. 
Industry Partners: Microsoft Technology and Civic Engagement Group, Yolobe.com, Chamber 630, Code DuPage, Venture 1818 Partners, Autism 911, LearnStep, Inklusion Works, Inklusion Capital, Social Skills Hero, TMT Partners 
Community Partners: Autism Community Ventures, Autism Society of Southern Illinois, Mental Health Association of Greater Chicago, The Transiiton2Success Project, MakerSpace Lab, Community High School District 99, University Centers for Excellence in Developmental Disabilities (University Partner), Leadership Education in Neurodevelopmental Disabilities (University Partner), Grupo Salto, The Autism Program of Illinois (TAP) 
Funding: Discover Partners Institute

Discovery Partners Institute: The Autism and Innovation Initiative

In this project, a novel computational method to model the evolution of aircraft contrail (ice particles forming in aircraft engine&rsquo;s exhaust) using high-performance computing is being developed. The specific goals of the research are: (1) to carry out the first fully three-dimensional spatial large-eddy simulations of contrail formation that include the aircraft geometry and to develop accurate data-set of contrail evolution in the early phases of the aircraft wake; (2) to identify the 3D contrail features and fit parameters by journaling the simulation workflow using advanced visualization techniques; and (3): to apply reduced-order modeling, such as Polynomial Chaos reconstructions, and machine-learning techniques, such as Artificial Neural Networks, to estimate model parameters and to reduce the large dimensionality of the problem to ease integration into global atmospheric models. 
EVL&rsquo; visualization work aims to create a multi-scale visual computing system that can identify the spatial features and parameters of the 3D contrails to analyze and explore multiple simulations of the ensemble data. These simulations can be performed for many different cases with specific parameter combinations. A guided summary and comparison of all of the simulations, aim to identify areas of interest across multiple outcomes. Additionally, the visual interface of the visualization system developed will provide users the ability to manipulate input or output parameters to find ways to minimize the formation of the contrails. An anticipated outcome, will be to enable climate researchers to select and examine specific simulations under given parameter values and gain more insight.

Visualization System Layout Prototype

High-performance computing and data-driven modeling of aircraft contrails

Cancer has a major impact on society all around the world. It affects people of all population groups though certain groups bear more burden of cancer. Some of the contributing factors for cancer disparities are socioeconomic status, geographic location, race/ethnicity. The objective of this project is to provide an interface to show cancer incidents and address the effect of different socioeconomic factors on cancer incidents in the areas of Chicago. 
The web-based application was developed by Computer Science Master degree-seeking graduate student Yashika Goyal, under the advisement of EVL and Computer Science associate professor, G. Elisabeta Marai.

Screenshot of Cancer Disparities application

Cancer Disparities in Chicago

Project Period: October 1, 2018 - September 30, 2021 
September 9, 2018 - The National Science Foundation (NSF) awarded a three-year grant to the UIC Computer Science (CS) Department&rsquo;s Electronic Visualization Laboratory (EVL) to acquire the &ldquo;Composable Platform as a Service Instrument for Deep Learning &amp; Visualization (COMPaaS DLV)&rdquo;. The total project cost is $1,424,806, with NSF providing $997,363 for hardware and the balance provided by UIC cost share to cover staff and graduate student support to design, operate and maintain the system. Maxine Brown, EVL Director, is Principal Investigator (PI) and co-PIs are CS faculty and EVL affiliated faculty members Andrew Johnson, Robert Kenyon and G. Elisabeta Marai. 
COMPaaS DLV will be acquired and made available for Deep Learning and Visualization research and training. Initial users will be 17 faculty from 5 departments in the UIC College of Engineering (COE) who provided compelling use cases for this system, and then expanded to other COE faculty as availability and demand permits. Users include the above-mentioned PI and co-PIs, plus COE Senior Personnel: Tanya Berger-Wolf (CS), Sybil Derrible (Civil &amp; Materials Eng.), Barbara Di Eugenio (CS), Chris Kanich (CS), Alex Leow (BioEng/Psychiatry, and by courtesy CS), Bing Liu (CS), Farzad Mashayek (Mechanical &amp; Industrial Eng.), Roberto Paoli (Mechanical &amp; Industrial Eng.), Luc Renambot (CS), Philip Yu (CS), Milos Zefran (Electrical &amp; Computer Eng.), Xinhua Zhang (CS), and Brian D. Ziebart (CS). 
COMPaaS DLV is critical infrastructure on which UIC faculty can pursue fundamental research in deep learning (data mining, data analytics, computer vision, natural language processing, artificial intelligence), visualization (simulation, rendering, visual analytics, video streaming, image processing), and a combination of deep learning and visualization (that is, when data is so large that it cannot be easily visualized, deep learning is used to extract features of interest to be visualized). UIC faculty are investigating, and contributing to, solutions for societal issues in such areas as anthropology, biology, cybersecurity, data literacy, fraud detection, healthcare, manufacturing, urban sustainability, and cyber-physical systems (e.g., autonomous vehicles). 
Though EVL specializes in visualization and virtual reality, it has a longstanding reputation as a technology enabler for the campus. EVL will design and operate COMPaaS DLV as a highly flexible, scalable, high-throughput system for Big Data. Its flexible 4th-generation computer architecture, &ldquo;composable infrastructure,&rdquo; treats computer components - computer processing units (CPUs), graphical processing units (GPUs), and storage - as pools of resources. Each application defines what resources it needs, and the operating system composes, or combines, resources on the fly and infrastructure is provisioned &lsquo;right-sized&rsquo; and resized as needed. The system is easily scalable - new infrastructure is automatically integrated with existing infrastructure and becomes part of the pool of capacity. The system&rsquo;s components are all interconnected with a high-speed internal network fabric, so that Big Data can quickly move between CPU, GPU and storage at optimum speed, i.e., high-throughput, with little to no bottlenecks. 
COMPaaS DLV will complement, but be connected via UIC&rsquo;s research network, to campus and regional clusters, data storage, supercomputing and cloud testbed facilities managed by the UIC ACCC&rsquo;s Advanced Cyberinfrastructure for Education and Research (ACER) group, the National Center for Supercomputing Applications at University of Illinois at Urbana-Champaign, Argonne National Laboratory&rsquo;s Leadership Computing Facility, and University of Chicago. 
Access to COMPaaS DLV will enable faculty, students and postdocs to execute deep learning and visualization codes faster, apply more sophisticated models to large-scale problems, gain greater insights, accelerate discovery, and open new avenues of research. The instrument will enable UIC to recruit new faculty who are attracted to institutions that can support their computational needs, and to recruit new students and produce a better trained workforce - which in turn attracts industrial partnerships. 
See <a href="https://today.uic.edu/nsf-grant-to-fund-advanced-deep-learning-and-visualization-computing-platform">UIC Today News Release</a> for more information. 
PHOTO CAPTION: One of the applications that will benefit from COMPaaS DLV is &ldquo;SENSEI Panama,&rdquo; a UIC collaborative project with evolutionary anthropologist Meg Crofoot (UC Davis) to enable anthropologists to use virtual reality to &lsquo;walk inside their data&rsquo; and study animal behavior. Specifically Crofoot and UIC computer scientists Tanya Berger-Wolf, Andy Johnson and Maxine Brown, are applying Deep Learning and Visualization to process data from LIDAR, aerial photography, terrain maps, GPS collars, and accelerometers, in order to study how coatis, kinkajous, spider monkeys, and capuchin monkeys on Barro Colorado Island (a Smithsonian Tropical Research Institute Research Center) in Panama, forage for food. This project received some funding from the NSF SENSEI $3.5M award to UIC CS Maxine Brown (principal investigator) and UIC CS co-PIs Andy Johnson, Tanya Berger-Wolf, and Robert Kenyon. (This project/photo won Honorable Mention in the <a href="https://www.evl.uic.edu/entry.php?id=2242">UIC 2016 Image of Research Competition</a>).

An application that will benefit from COMPaaS DLV is &ldquo;SENSEI Panama,&rdquo; which applies Deep Learning and Visualization to help anthropologists walk inside their data using virtual reality to study animal behavior.

COMPaaS DLV: Composable Platform as a Service Instrument for Deep Learning & Visualization

The SAGE Development Team at the Laboratory for Advanced Visualization &amp; Applications (LAVA) at the University of Hawaiʻi at Manoa and the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago recently received one-year supplemental funding from the National Science Foundation (NSF) to their SAGE2 award, in the amount of $527,000, to focus on SAGE2 cybersecurity. The diversity and distributed nature of the SAGE2 user base provides a growing set of security concerns. Identity and access management procedures, in particular, provide unique challenges, given the variety of institutions using SAGE2 to collaborate using display walls. 
In July 2018, the SAGE2 Team began working with Trusted CI, the NSF Cybersecurity Center of Excellence at Indiana University, to develop an Engagement Plan that outlines the scope of a 6-month timeline for the engagement, followed by 6 months of implementation. Trusted CI is helping outline Identity and Access Management (IAM) procedures appropriate for SAGE2&rsquo;s distributed user base. For example, SAGE2 needs to identify and provide access to local and remote sites involved in a collaboration as well as the users at those sites; such as: who is allowed to collaborate, are they allowed to share information or simply read it, and are they allowed to download information, etc. A secondary goal of the engagement, time permitting, is to determine if there are any unidentified high-priority cybersecurity needs revealed during the fact-finding stage that the SAGE2 Team should pursue; such as: securing the web server, securing documents that are shared, etc. 
<a href="http://blog.trustedci.org/search?updated-max=2018-08-02T11:59:00-04:00&max-results=7">Trusted CI Blog</a>

The SAGE2 team at UIC/EVL remotely collaborated with the University of Chicago’s Research Computing Center during its fourth annual research computing symposium and exposition, Mind Bytes.

SAGE2 Team Engaging with the Trusted CI Center on SAGE2 Cybersecurity

Ableton Live is music software designed for computer-based composition and performance. <a href="https://en.wikipedia.org/wiki/Ambisonics">Ambisonics</a> is a full-sphere surround sound technique used for playing audio on sound systems with more than two speakers, providing the illusion that sound is coming from all directions. Combining these has allowed music producers to compose music inside of Ableton, in 3D. Programmed in the visual programming language <a href="https://cycling74.com/products/max/">Max</a>, is an ambisonic decoder for the Sound Dome built by former EVL post-doc Anil Camci. 
The decoder outputs sound sources through specific speakers in the sound system based on their specified Azimuth, Elevation, and Distance parameters. To integrate Ableton Live with the decoder, all of the audio channels in Ableton were streamed into the decoder via an application called Jack which allows users to route audio from one application to another. A Max utility was also built to run inside of Ableton Live and allow the user to conveniently adjust the azimuth, elevation, and distance parameters of a particular channel. 
The azimuth elevation and distance values are sent over a network back to the Max decoder allowing producers to change the location of playback for every channel in Ableton without leaving the application. Custom audio effects are also being programmed to exploit all of the surrounding speakers and take full effect of system.

EVL Sound Dome developed by Anil Camci

Ableton Live Integration & Audio Effects for a 32 Speaker Ambisonic Sound System

Pi: Barbara Di Eugenio; Co-I: Debaleena Chattopadhyay 
A growing body of research has found that patients who actively manage their health have better outcomes, and patient engagement has become a central tenet in recent healthcare approaches. However, it has also been shown that many patients lack knowledge of their disease and medications, particularly as concerns chronic health conditions, whose exacerbation causes about two thirds of hospitalizations every year in the USA. Discharge instructions after hospitalization are a critical tool to engage patients in their self-care, but are often unsuccessful. Our hypothesis is that, to engage patients at discharge, a personalized hospitalization summary should be provided, that integrates physician and nurse documentation, and is informed by the patient's level of engagement in self-care and their socio-cultural perspective. 
Our transformative pilot work on physician and nurse documentation has identified an additional obstacle to the effective flow of information from professionals to patients: vocabulary differences between physicians and nurses. By leveraging the Unified Medical Language System Metathesaurus, we found no common synonyms between physician and nurse documentations for 75% of patients. Further, 86% of terms patients use are simple compared to only 15% for physicians. The lack of common language among patients, physicians and nurses motivates including all perspectives to make instructions meaningful for patients. Our hypothesis is that a better informed patient population who understands the connections between their illness, hospital experience, and discharge instructions will be more empowered and more engaged in their chronic disease management. Our innovation is to automatically create an easy-to-understand, personalized hospitalization summary that integrates events described in the EHR; and that is responsive to the patient's needs and preferences, and their activation framework (PAF) level. We pursue four specific aims: 1. To investigate socio-cultural patterns of patient perceptions of illness via a socio linguistic analysis of patient interviews; 2. To integrate multi-faceted information about a patient, as provided by physicians and nurses; 3a. To develop MyPHA, a novel application program, which will automatically generate the personalized summaries and deliver them to the patient on a tablet. MyPHA will be informed by aims 1 and 2, and will be grounded in Natural Language Generation technology; 3b. To evaluate MyPHA with multiple cycles of formative evaluation with all stakeholders, and a summative evaluation with patients. 
Whereas our pilot work has focused on heart failure, the techniques we will develop can easily be adapted to the management of other chronic health conditions, in particular for patients with cancer and cancer survivors. In addition, cancer and heart failure can co-occur, especially in older patients, and several cancer treatments can lead to the development of heart failure. The software program MyPHA will be released ooen source to the research communitv to enable raoid customization to other chronic conditions. 
Awarded Amount: $1,495,213

NIH: R01: SCH: INT: MyPHA: Automatically generating personalized accounts of inpatient hospitalization

The Continuum environment is a high-resolution computer-enhanced group collaboration workspace, consisting of large format 2D touch-enabled and 3D tiled displays, and instrumented with arrays of ambient sensors, enabling users to collaborate with local and remote colleagues and their data more effectively by giving the room itself the ability to anticipate their needs, improve their interaction, and focus their time on discovery rather than the technology. 
EVL&rsquo;s SAGE2 middleware enables teams of users to manage their data, documents, images and media on the Continuum display walls as if the room were one seamless canvas. SAGE2 lets users post and manipulate information, visualizations and animations across the Continuum displays. EVL REU Sean Walker developed a spatial audio system that ties audio playback from SAGE2 media content directly to the media&rsquo;s (i.e. video) location on the display wall. 
Additionally, any audio playback in the Continuum environment can be played back from any of the X number of speakers in the space. The spatial audio functionality was implemented using AVB (Audio Video Bridging) to connect all of the speakers positioned throughout the room coupled with SuperCollider (a functional programming language for audio) to pan the sound around the space.

Spatial Audio in the Continuum Environment

Caterpillar Inc. is working with the UIC Electronic Visualization Laboratory (EVL) to develop a SAGE2&trade; application to enable distributed teams to collaborate on project planning and scheduling. 
Planning and scheduling the design, development, engineering and manufacturing processes of construction equipment with worldwide teams and suppliers is an extraordinarily complex task. Caterpillar, inspired by &ldquo;StickySchedule,&rdquo; an interactive SAGE2 multi-user application for conference scheduling, asked whether EVL could help automate their project management workflow, which is currently a very manual process. 
The goal is to develop an innovative SAGE2 application to enable people from different Caterpillar units to schedule interdependent tasks over several months or years using electronic post-it notes on a large tiled display. The resulting schedule would be translated into a spreadsheet, which would then be input to project management software. SAGE2 would also enable groups to work collaboratively over distance, whereas now, everyone must be in the same room at the same time. Today, these management plans are updated only a couple times per year due to the high labor and cost of travel. This tool will enable these planning sessions to occur for significantly less labor and cost, resulting in a more regular update by the extended team. 
Additional information: 
<a href="http://sage2.sagecommons.org">SAGE2 Website</a> 
<a href="http://sage2.sagecommons.org/download/2117/">StickySchedule Paper</a> 
<a href="http://apps.sagecommons.org/">StickySchedule App</a> 
Project team: 
Lisa Creger, Lean Product Development Advisor, Innovation & Technology Development Division (ITDD), Caterpillar 
Peter E. Cooper, Product Development Manager, Medium Wheel Loaders, Caterpillar 
Luc Renambot, Computer Science Research Professor and EVL SAGE2 Project Manager, UIC 
Tommaso Massari, Computer Science MS student and EVL Research Assistant, UIC

Example of Caterpillar 'manual' project management workflow session.

SAGE2 Project Management Application

The Shirley Ryan AbilityLab of Chicago worked with the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago to use virtual reality to help people needing rehabilitation be more enthusiastic and energetic when doing their exercises. 
Using the VIVE head-mounted display, EVL researchers develop virtual reality worlds to augment the rehabilitation patients&rsquo; physical therapy spaces and sessions. The goal of this project is to increase patient interest and engagement, by supplementing physical therapy tasks and exercises with tasks incorporated into a number of different virtual worlds. Unity3D is used to build virtual environments, that are &lsquo;mapped&rsquo; to the physical therapy areas (e.g. a physical balance beam is represented by a very narrow bridge over a stream in a forest; the patient steps onto the actual beam, but are stepping onto a virtual bridge in the VR world). Data about the patient&rsquo;s movements and activities in these virtual worlds are visualized in real-time, allowing AbilityLab personnel to tailor the experience interactively. The enhanced rehabilitation experience intends to foster patient recovery and treatment success.

EVL director Maxine Brown explores a virtual world designed to enhance rehab patient treatment options.

Virtual Reality Enhanced Physical Therapy

In the past decades, science has become increasingly interdisciplinary and scientific discovery has experienced a paradigm shift from the classical scientific method to data-intensive scientific discovery. SAGE2 provides these interdisciplinary collaborators with a powerful, shared workspace. However, there remains a high cost of parallel work integration for data-intensive work. VisSnippets is a system which allows collaborators to perform expressive, impromptu data analysis and exploration within SAGE2. 
VisSnippets acts as a collaborative interface to exploratory programming, allowing users to write modular &ldquo;snippets&rdquo; of code for data retrieval, processing/aggregation, and visualization. Users may then compose modular analysis blocks from the shared snippet pool into branching data pipelines. The snippets may be edited or &ldquo;cloned&rdquo; by collaborators, and reused/applied multiple times across a collaborative session. 
To further facilitate collaborative work, the VisSnippets system provides further abstraction onto the expressive foundation of exploratory programming. VisSnippets reduces the overhead to scaffold a responsive web application and allows users to walk away from the collaborative analysis with a packaged web application containing all of the analysis.

VisSnippets: A Web-Based System for Impromptu Collaborative Data Exploration on Large Displays

This collaborative research project led by the colleagues at the University of Delaware aims to develop and advance tools for using image-data appearing in scientific publications, in addition to text, in order to support beneficial, targeted access to the biomedical literature. 
UIC's principal investigator, Computer Science associate professor, G. Elisabeta Marai will be responsible for the image analysis algorithms to distinguish between gel and plate images, the image analysis algorithms to distinguish between Electronic Microscopy vs Fluorescent Microscopy vs Light Microscopy and X-Rays, and the design of the visual interface to present results to end users. She will lead the design, development and implementation of the feature extraction and selection tools and techniques related to this research project.

Incorporating Image-based Features into Biomedical Document Classification

Cancer treatment courses which rely on imaging and spatially-dependent therapy involve making multiple treatment decisions (e.g., radiotherapy alone, radiotherapy plus chemotherapy, induction chemotherapy) over time. These decisions depend on complex factors, including the tumor location with respect to sensitive organs and its response to treatment, laboratory data, toxicity, anticipated side effects and survival probability. In this project, we design and develop novel statistical methodology for dynamic and personalized treatment decisions with specific application to head and neck cancer radiotherapy planning. We extend the state of the art by developing methods which take into account both imaging data features (such as radiation dosage and spatial distribution) and nonspatial data (such as demographics and toxicity) in the development of predictive models for anticipated outcomes and of the optimal treatment rules. We introduce novel similarity metrics over these hybrid, high-dimensional and sometimes incomplete patient data, and we extract informative, relevant and non-redundant features to identify similar patients. We further combine statistical models for outcome prediction with local models built over cohorts of patients. These ensemble models are used to further improve the reinforcement learning process to account for patient-specific preferences regarding side effects and survival outcomes. Specifically, we extend q-learning, a type of reinforcement learning, to account for competing, multiple outcomes and different preferences on the probability of short versus long-term outcomes. We validate our precision-driven model against known toxicity outcomes using a retrospective head-and-neck cancer dataset. 
Our specific aims are to: 
- Create quantitative methodology for extracting and selecting spatial relationships from medical images as well as non-imaging features, and generate a hybrid similarity metric which can identify similar cohorts of patients. 
- Create an ensemble model that combines global static models with dynamic local models over similar cohorts of patients with locally-sparse data. 
- Create dynamic reinforcement learning methodology which uses spatial and non-imaging features and metrics and accounts for patient preferences of multiple treatment outcomes. 
The project defines novel reinforcement learning techniques which account for multidimensional outcomes and patient preferences. Previous examples of reinforcement learning in the statistical literature have utilized examples with a small number of well-defined features, which is not applicable in the case of head and neck cancer. This project creates novel hybrid techniques for extracting informative and relevant features and similarity from imaging and nonimaging data. The project further introduces novel predictive modeling using a combination of global and dynamic local models over multiple outcomes. The proposed techniques leverage Big Data that is not only heterogeneous but also high-dimensional and locally sparse, in order to derive a novel precision-medicine approach. 
The empirically-derived treatment rules developed in this project have the potential to improve the standard of care (i.e., treatment plans chosen by the tumor treatment board) and the quality of life of surviving patients. The methods developed in the proposal may be used to derive optimal treatment strategies across not only a variety of spatially-dependent cancer diagnoses, but also other chronic conditions including mental health disorders, substance abuse diseases, or diabetes, that require making multiple decisions that must weigh the tradeoffs between efficacy and toxicity.

QuBBD: Precision E-Radiomics for Dynamic Big Head & Neck Cancer Data

Deep learning is often referred to as a &lsquo;black box&rsquo;, because, aside from selecting model and the model hyperparameters (like the learning rate, activation function parameters), users don&rsquo;t have access to what happens inside the model. We can know that the model works by measuring its accuracy on some testing data set, suggesting that the model must have &lsquo;learned&rsquo; something meaningful from the training data. But how can we peek inside and &lsquo;see&rsquo; what the model &lsquo;sees&rsquo;? 
Biologists have recently begun to explore how deep learning approaches could be used to address questions involving diverse biological datasets. For this community, producing an accurate model is important, but also insufficient, since the larger goal is to contribute to our understanding of complex biological systems and phenomena. Visualization could help biology researchers &lsquo;peak&rsquo; inside the black box and relate deep learning mechanics to existing biological knowledge. However, most visualization techniques for deep learning have been developed for images and text, so many of the encodings and layouts are designed for input data with fundamentally different perceptual properties than biological data sets, such as gene sequence data or gene expression data. 
This research explores current approaches in visualization for deep learning and how these approaches may (or may not) be applied to biological data. This work is conducted with researchers at Argonne National Labs studying deep learning for cancer research and genomics.

Argonne Leadership Computing Facility

Vis for Deep Learning for Biology

Exploring big cities like Chicago can be a daunting experience when the tourist has to gather information from multiple sources regarding places, directions, transportation, safety, and weather.  We propose a large display solution (kiosks) which aims to encourage people to explore the city by showing cultural attractions, special events and other points of interest (POI) on a map along with multi-modal transportation options to take them to their destination. POI&rsquo;s and events are updated frequently based on the time and safety information. Kiosks are intended to be located throughout the city especially locations with the highest density of points of interest.

Screen shot of the &ldquo;Exploring Chicago&rdquo; application prototype.

Exploring Chicago

The Englewood Data Hub (EDH) is a project that brings together both crowdsourced and public data surrounding the Englewood area into a single place. 
The <a href="https://englewooddatahub.org/">EDH Resource Directory</a> makes use of the crowdsourced service and school data to display an interactive map with their locations. Its user interface (UI) allows for an exploratory experience that offers filtering of services by names and type and displays detailed information about the services available in Englewood. 
The <a href="https://englewooddatahub.org/analytics">EDH Analytics Tool</a> combines the crowdsourced service and school data with public data sets to display visualizations; specifically, the public datasets used are 2010 Census data along with Vacant Lot and Crime data from the Chicago Data Portal. 
The Englewood Data Hub has been developed by a group of researchers at the Electronic Visualization Laboratory, University of Illinois at Chicago under the direction of Prof. G. Elisabeta Marai. 
Team members include EVL research experience undergraduate students (REUs): Andrew Burks, Joshua Castor, and Isabel Lindmae. 
This project is funded by the The Joseph and Bessie Feinberg Foundation and the University of Illinois at Chicago.

Englewood Social Service Finder and Englewood Analytics

Head and neck squamous carcinomas affect all ethnicities and age groups, accounting for significant mortality and therapy-related
side effects. Over 50,000 new cases are diagnosed each year in the United States, leading to large, rich repositories of patient data. For each of these cases, oncologists need to anticipate survival, oncologic, and side effect outcomes associated with treatment strategies in order to select a treatment which balances efficacy and toxicity. However, despite the wealth of data available, risk prediction algorithms for cancers are rudimentary and incorporate only a handful of patient characteristics, largely due to a lack of computational methodologies. 
This project will develop a computational methodology that supports the construction of a validated precision model of patient-specific outcomes for head and neck cancers, based on demographics, toxicity, and complex imaging data. The methodology will be the first in the field to include both complex imaging and non-imaging data, while taking into account large-scale biological and clinical correlates. A web-based environment will further allow practitioners to interactively explore and communicate the model mechanics and outcomes to stakeholders. The approach is innovative through its leverage of big imaging data repositories and through its unique blend of computational modeling principles from bioengineering, statistics, and computer science (medical imaging and visualization). 
From a clinical perspective, this integrative approach is novel in the field of cancer therapy. The resulting model and web-based
environment will mark a significant advance in biomedical computing because it will be able to identify, for the first time, specific subgroups who are at risk for distinct oncologic, toxicity, and survival profiles. Furthermore, the currently clinical practice paradigm for head and neck treatment is stage-driven. With validation, this novel approach can identify patients based not on clinical staging, but on precision modeling using cohort data and similar sets of patients. The methodology is applicable to clinical practice involving human subjects and has the potential to change the standard of care and outcomes of treatment in the field.

SMART-ACT: Spatial Methodologic Approaches for Risk Assessment and Therapeutic Adaptation in Cancer Treatment

The <a href="http://hci.cs.uic.edu/">Human-Computer Interaction (HCI) Laboratory</a> at the University of Illinois at Chicago (UIC) is led by assistant professor Debaleena Chattopadhyay of the Department of Computer Science. The HCI Lab designs computer technologies to transform human experiences and deliver positively impactful societal change. 
UIC HCI is an affiliate laboratory of the Electronic Visualization Laboratory (EVL) providing a human centered approach to the development of tools and technologies related to advanced visualization, virtual reality, visual analytics and networked collaboration. 
The HCI Lab&rsquo;s mission is to support individual and social activities and improve everyday life, designing, building and evaluating ubiquitous computing technologies. The design of these systems draws from research on how people work, play, and communicate in day-to-day life. 
The UIC HCI Lab is located on the UIC campus at: 
Science & Engineering Laboratory West Room 1032 
900 West Taylor Street 
Chicago, IL 60607

UIC Human-Computer Interaction (HCI) Laboratory

Just as NSF cyberinfrastructure produces greater volumes and varieties of data from data storage systems, online instrumentation and major computational resources like XSEDE and Blue Waters, advanced visualization instruments serve as the &ldquo;lenses&rdquo; of these data generators (similar to the telescopes and microscopes of yore), enabling researchers to peer into cyberspace to manage, access and analyze big data. 
Continuum is a cyberinfrastructure instrument that unifies high-resolution computer-enhanced group collaboration workspaces with arrays of ambient sensors, enabling users to collaborate with local and remote colleagues and their data more effectively by giving the room itself the ability to anticipate their needs, improve their interaction, and focus their time on discovery rather than the technology. Continuum is a digital assistant, benefiting global scientific collaboratories and providing a foundation for new computer science research. War rooms benefit collaboration because they support continuous interaction and awareness of team members, enable persistent information sharing, and encourage mobility among team members and information. A computer-enhanced war room, such as Continuum, which (1) connects distributed teams with their data, whether stored in the cloud or streamed from major computational resources over high-speed networks, (2) supports persistent digital artifacts, (3) enables ultra-high-resolution 2D and 3D stereoscopic digital media to be simultaneously viewed, and (4) supports ubiquitous and intuitive interaction, creates a powerful and easy-to-use information-rich environment in support of scientific discovery. 
Continuum is the culmination of the University of Illinois at Chicago’s (UIC) 20+ years experience developing interactive instruments, from the original CAVE in 1992, to the ultra-high-resolution LambdaVision tiled LCD wall in 2004, and the hybrid-reality CAVE2 in 2012. Each new generation of visualization instrumentation provided scientific communities with more advanced features Continuum development focuses on three areas: (1) transforming passive display walls that respond to user commands into attentive display walls that monitor the users in a room and anticipate their needs to improve their experience; (2) providing an alternative approach to constructing high-resolution visualization display walls using thin-border 4K passive stereo OLED displays that have impressive brightness, contrast, uniformity, off-axis viewing, and low reflectivity in brightly lit rooms; and (3) utilizing cloud computing to prototype future Continuum spaces that don’t require local high-end compute and display resources. Twelve funded research projects from local institutions, in Astronomy, Astrophysics, Biology,
Combustion, Geophysics, Neuroscience, Pathology, Physics, Psychiatry, and High Performance Computing, are poised to use Continuum for their data visualization needs. Continuum will open up new opportunities in computer science research at the intersection of large-scale data visualization, human-computer interaction, and high-speed networking. Continuum will also directly support 10 classes taught in the UIC Computer Science, Art and Design, and Biomedical Science departments.

Artist's rendering of Continuum space.

MRI - Development of Continuum: A Virtualized Attentive Environment for Amplified Collaboration

The visualization of aleatoric uncertainty in ensemble simulations - i.e. the study of high concentrations in mixtures - provides many challenges due to the size and dimensionality of the data sets. EVL researchers have developed a web-based application designed to locate and track areas of high concentration in salt-solution simulations. 
The goal is to learn more about the behavior of Viscous Fingers, or contiguous areas of high concentration of salt within a cylinder of pure water. The application allows viewers to explore 3-D ensemble simulation data using both 2-D vertical slices of the cylindrical data to get a more detailed view of the data in a specific time step, as well as a feature tracking merge tree to better understand the evolution of these Viscous Fingers over the course of a simulation run.

Screen Shot of Viscous Fingers Visualization Application

Interactive Exploration and Tracking of Viscous Fingers in Large-Scale Ensemble Simulations

The Chicago 0,0 Riverwal AR experience provides a novel, interactive way for users to explore historical photographs sourced from museum archives. As users walk along the Chicago River, they are instructed to use their smartphone or tablet to view these photographs alongside the current views of the city. By matching location and view orientation the Riverwalk application creates an illusion of &ldquo;then and now&rdquo; co-extant. This superimposition of the historical photographer&rsquo;s view and the user view is the basis of educational engagement for the user and a key factor in curating the images and the narrative surrounding them, facilitating a meaningful museum experience in a public, outdoor context. 
As such, creating the AR experience involves a complex back-and-forth between 3D and 2D experiences of locations: the historical
images are 2D, taken from specific locations through specific optics and views; the user, present at the real-world 3D location is
orienting their own camera and 2D phone video screen in space in order to achieve the AR experience and illusion; and the ultimate experience of superimposition is one of seeing two integrated views simultaneously: the stream of data taken from a smartphone&rsquo; camera and the historic image. The site-specific nature of the publication makes it necessary to utilize a virtual 3D environment in which to place 2D augmented content that enhances the experience. This content needs to be placed in such a way so that the designer can accurately visualize what the user will see on screen from a particular location. 
The first episode of the Riverwalk AR experience focuses on a single block between N. LaSalle and Clark Streets; the site of
the Eastland Disaster in 1915. The site was selected because of the importance of this historical event - the sinking of the Eastland
cruise ship 100 years ago was the largest single loss of life in Chicago’s history - and because of the abundant media available
in the archive including extensive photographic documentation, newspapers and film reels. Moreover, the site potentially offers
natural way, finding characteristics that are amenable to developing our project, including a newly built pedestrian walkway along
the river with viewing platforms, views from pedestrian walks along the bridges, and historical markers. Additionally, the urban environment offers many features providing sufficient quality for robustly tracking user location based on their cameras from a variety of views and positions. This is a typical scenario in the creation of public outdoor AR experiences; the quality of content and narrative are the primary motivations for their creation.

Chicago 0,0 is a mobile application aimed at presenting 2D media content from archives of historical media through the use of augmented reality.

Riverwalk: Incorporating Historical Photographs in Public Outdoor Augmented Reality Experiences

Patients come to Centers for Physical Therapy and Rehabilitation from all over the globe, &ldquo;often with catastrophic or unusual conditions and always with great hope for an improved life ahead&rdquo;. In the quest to assist these patients, Virtual Rehabilitation is becoming popular in clinical research thanks to the development and availability of new immersive technologies. For example, the rehabilitation of post-stroke patients frequently uses virtual reality, and sometimes employs a first-person Serious Games approach to make therapy more enjoyable for patients. In such approaches, the patient experiences the virtual environment in a first-person view. 
Unlike the existing body of work, we present and evaluate an immersive virtual rehabilitation application which uses a third person perspective: the patient controls with her body a 3D avatar. The application was commissioned by biomechanics researchers at a major center for rehabilitation research, and could be used to compare the learning-curve and time needed for rehabilitation across different therapies and environments. One of the main challenges in this work is the design of simple, non-distracting avatars and virtual world, which allows the patient to focus on performing correctly the rehabilitation exercises. An additional challenge comes from the trade-offs typically associated with virtual reality, for example, loss of depth perception. 
Our solution, RehabJim, is a third-person serious game for the rehabilitation of upper limbs in post-stroke patients, with particular emphasis on reaching movements. Our solution uses a state-of-the-art virtual reality environment, Unity 3D, and a Kinect controller in the CAVE2&trade; Hybrid Reality Environment.

External view of CAVE2 environment with a running instance of our application. The screens cover 320 degrees of visibility and a monitor at the entrance is used for managing the hardware resources.

RehabJim: A Third Person Approach To Virtual Reality Biomechanical Rehabilitation

Google Street View can be considered as an online browsable dataset consisting of billions of street-level panoramic images acquired all around the world. Since 2007, the company has continuously improved and updated its global image database, now collecting on average one panorama each 5-10m in urban environments. Google provides public APIs for requesting virtual camera views of a given panorama from its GPS position or unique identifier. These views are rectilinear projection of the spherical panorama with a user selected field of view, orientation and elevation angle. These views can be considered as undistorted images taken from pinhole cameras, free of distortion, and need to be partially overlapped in order to reconstruct the 360 degree panorama. 
Additionally, the speed with which Google&rsquo;s laser scanners get reflected by surfaces allows the company to calculate the distance from the camera to objects or buildings, allowing the creation of 3D models to be used in other applications such as Google Earth. Google API indeed provides a way to obtain the depth map corresponding to a specific panorama, information that we will exploit in order to reconstruct the 3D scene. Once decoded and visualized in space, this depth data and the models generated from it may turn out to be useful for many different applications like navigation and augmented reality, but also virtual reality oriented videogames.

Sample point cloud reconstruction based on the output of merging 6 different panoramic images along a street.

3D City Reconstruction From Google Street View

SENSEI: Sensor Environment Imaging (SENSEI) Instrument is a National Science Foundation award to construct a reconfigurable, ultra-high-resolution, spherical (4π steradian), photometric, radiometric and photogrammetric real-time data-acquisition, sensor-based, camera system capable of capturing 3D stereo and still images for viewing in collaboration-enabled, nationally networked,
virtual-reality systems. 
As SENSEI researchers develop the SENSEI camera system, collaborators are exploring and refining techniques for displaying data captured via a variety of sensors, as well as investigating human computer interaction modalities for stereoscopic, virtual reality applications and environments incorporating complex layers of sensor data. 
SENSEI-Panama (SENSor Environment Imaging-Panama) is a environmental visualization approach developed at the Electronic Visualization Laboratory (EVL) to investigate the movement patterns on newly isolated species in the anthropological sciences. Researchers who have left the field are unable to visually determine what animals see in their environment; however, they produce large amounts of data through the use of GPS collars. accelerometers, LIDAR, aerial photography, and terrain maps. SENSEI-Panama considers how to visualize this kind of data in an environment like CAVE2&trade; and combine it with the animal movement data collected. The data is transformed to create a virtual map to enable researchers at the University of California Davis to easily track the movements of animals through their habitat on the island, and study their behavior.

EVL researchers Jillian Aurisano (lead) and James Hwang explore SENSEI-Panama in EVL's CAVE2&trade;

SENSEI-Panama

EVL PhD candidate Arthur Nishimoto has developed a virtual reality representation of the proposed new College of Engineering (COE) building for display in the CAVE2. Participants can navigate through and explore the virtual COE building and surrounding UIC campus. This rendition consists of models converted to Unity3D from the AutoDesk Revit model provided by the architecture firm Lothan Van Hook DeStefano Architecture LLC (LVDA). Additional Chicago city buildings were added to show how the downtown Chicago and other areas around UIC would look from the perspective of the new building. 
Using EVL&rsquo;s CAVE2 Unity simulator and Mechdyne&rsquo;s getReal3D plugin, the Unity scene was exported for use in the CAVE2 Hybrid-Reality Environment. Within the virtual COE building users can explore rooms within the proposed building including collaborative classrooms, the winter garden, and laboratories. Other 2D elements and renderings provided by LVDA are integrated into the virtual walk-thru. 
Model Credits: 
Chicago city model by Andrew Lock 
<a href="http://arthurnishimoto.blogspot.com/2015/10/uic-science-and-engineering-offices.html">UIC Science and Engineering Offices by Arthur Nishimoto</a>

EVL PhD candidate Arthur Nishimoto navigates through the proposed virtual COE building in the CAVE2.

Proposed UIC College of Engineering Building Rendition in the CAVE2&trade; Hybrid Reality Environment

SAGEBoard is an electronic whiteboard for tiled touch displays that can be used locally or shared between remote collaborators. An electronic whiteboard can combine the advantages of an electronic device with the fast paced interaction provided by a regular whiteboard, enhancing both solutions with new functions. Users can perform classic whiteboard actions such as drawing or writing free-handed on the screen using their finger or a stylus. They can use brushes to paint, and they can erase content. However, unlike on traditional whiteboards, users can also relocate content on the screen, insert figures and drawings, and save or load previous sessions. Moreover, everything can be shared between all the users, and more people can interact simultaneously, improving the collaborative task. 
SAGEBoard runs on multiple displays, and is integrated with other display-enabled applications; the application manages multiple simultaneous inputs, as well as the rendering of graphic elements on different sized displays. SAGEBoard is a lightweight networked solution that address any latency issues that might impact the interactive experience. 
This research explores human computer interaction issues - how people behave when collaborating on a shared touch enabled display, or deal with remotely controlled collaboration, since the whole application can be shared among both co-located and remote screens. A small scale user study was designed to analyze the advantages of using multi-touch displays over conventional keyboard and mouse configurations, as well as remote interaction using mobile touch displays.

User drawing on EVL&rsquo;s Cyber-Commons tiled display using SAGEBoard

SAGEBoard: a Whiteboard for Large Multitouch Displays

SENSEI will be a reconfigurable, ultra-high-resolution, spherical (4π steradian), photometric, radiometric and photogrammetric real-time data-acquisition, sensor-based, camera system capable of capturing 3D stereo and still images for viewing in collaboration-enabled, nationally networked, virtual-reality systems. 
The ultimate goal of SENSEI will be to capture ultra-high-resolution still and motion 3D stereoscopic video and images of real-world
scenes, that can be viewed in 3D high-performance virtual-reality systems, and can be used to extract objects&rsquo; sizes, shapes and distances within a scene. 
SENSEI, given its ~terapixel/minute flood of imagery, presents big data challenges as well. Specifically, SENSEI will be an integrated, distributed cyber-collaboration instrument with big attributes: 
- &ldquo;Big Resolution,&rdquo; configurable, portable, sensor-based camera systems: Sensor-based, dynamic 3D stereoscopic motion/video and/or multi-spectral data camera systems will be developed. They will be flexibly configurable to capture geometries such as: outside-in, inside-out, light field arrays, photogrammetry, macroscopy and microscopy. SENSEI will have an optional weatherproof transport container for field use and underwater cases developed by partners. SENSEI will create images that are ~9 times IMAX resolution. Open source software and wikis will encourage adoption and modification by other domain scientists, as with our OptIPortals and CAVEs from prior MRIs. 
- &ldquo;Big data&rdquo; computational and storage systems: Graphics processing unit (GPU) computing with RDMA, fast storage systems, and software development are needed to perform the necessary image processing calibration, stitching, synchronization, partitioning, and video serving/streaming. 
- &ldquo;Big displays&rdquo; for 3D stereoscopic viewing: Virtual-reality displays, such as the CAVE2, WAVE,
and similar technologies, are ideal for end-user viewing and multi-user, multi-site collaboration. The next generation of 4K 3D OLED TVs and LCD TVs, singly or in small arrays, will also be supported. 
- &ldquo;Big networks&rdquo; for an end-to-end, tightly coupled, distributed system: The distributed system -
cameras, computational and storage systems, and display technologies - will use 10/40/100Gbps optical networks at/between partner institutions to share and correlate data in useful time frames. 
SENSEI can advance computer science and electrical engineering disciplines in the development of image capture systems, display technologies, image processing, visualization, and image creation, and advance scientific domain disciplines such as archaeology, architecture, astronomy, biology, cultural heritage, digital arts, earth science, emergency preparedness, engineering, geographic information systems, manufacturing, oceanography, planetary science, and science education. 
SENSEI partner institutions are: 
- University of Illinois at Chicago (UIC), Electronic Visualization Laboratory (EVL) 
- Jackson State University (JSU), College of Science, Engineering and Technology 
- Louisiana State University (LSU), Computer Science Department (Beginning in Year 2)
- Scripps Institution of Oceanography (SIO), Jaffe Laboratory for Underwater Imaging 
- University of California, San Diego (UCSD), Calit2-Qualcomm Institute (Calit2-QI) 
- University of Hawaii at Manoa (UHM), Lab for Advanced Visualization &amp; Applications (LAVA) 
UIC, UCSD, LSU and SIO are the primary technology developers of SENSEI. JSU and UHM, as well as the developer sites, have application teams who will be early adopters of SENSEI and provide feedback on the workflow, operation, and use.

Close-up rendition of SENSEI camera prototype

SENSEI: Sensor Environment Imaging (SENSEI) Instrument

Our interdisciplinary team of computer science and health care researchers will develop augmented communication tools (ACTs) within a prototype Electronic Health Record (EHR) visual interface and test its usability. The focus of this project is Nurse-Nurse (RN-RN) communication of six high risk clinical events (CEs): fever, pain, changes in respiratory status, changes in level of consciousness, changes in output, and bleeding. Using combined EHR and hand-off report data, we will develop a predictive computational model for detecting these clinical events within the EHR, as well as their predicted severity. The ACTs system will then use light-weight automated inference to detect these clinical events, alert the nurse, and quickly provide highly relevant contextual data to improve quality care, patient safety, and reduce patient deaths.

Enhancing Nurse Effectiveness via Augmented Communication Tools

The REACH system extracts (i.e., reads and assembles) causal mechanisms from research abstracts and papers. The fundamental research proposed has three guiding principles. The mechanisms extracted must: (a) be runnable, i.e., they must predict the data; (b) model current uncertainty of causal relations being active given context information; and (c) be holistic, that is, they explain multiple layers of abstraction, ranging from cellular to organism phenotypes. Thus, running our model produces quantifiable probabilities for the modeled variables (e.g., presence of pancreatic cancer) given customizable input values (e.g., overexpression of the VEGF protein). We assume that the model is deterministic given a specified context, but, in practice, variance in results exists due to difference in context - either reported or completely unknown - in the acquired mechanisms. The REACH system will provide a platform for communicating explanation with novel visualization tools for the extracted mechanisms. We will evaluate our approach in the domain of cancer biology, by extracting signaling pathways, but our work is largely domain independent.

Reading and Assembling Contextual and Holistic Big Mechanisms

Developed by EVL PhD Candidate Arthur Nishimoto the CAVE2/Unity simulator is a Unity3D package for testing and development of Unity3D applications for the CAVE2&trade; Hybrid-Reality Environment. Using only standard Unity features (no platform-dependent libraries or Unity Professional license required), this package acts as a wrapper around the CAVE2 API to simulate the CAVE2 viewport and 3D spatial wand interactions on a desktop/laptop development environment. 
The associated <a href="https://github.com/arthurnishimoto/module-omicron">module-omicron package</a> also serves as the Unity/C# interface of accessing the omicron input abstraction library.

CAVE2/Unity Simulator being presented by EVL PhD candidate Arthur Nishimoto.

CAVE2/Unity Simulator

The International Space Station (ISS) and Space Shuttle Discovery is a CAVE2&trade; Hybrid Reality Environment simulation where participants can explore the Space Station in full scale, orbiting the Earth at the current location and altitude of the ISS. Users experience the current field of view from ISS and the territory over which it is orbiting. The ISS orbits the Earth roughly once every 90 minutes at an altitude of 420 km. Participants can fly around the space station and the shuttle to get a better sense of their size in relationship to the shuttle. The model of the ISS and position data come from the National Aeronautics and Space Administration (NASA). The model of the shuttle comes from the google sketchup 3D Warehouse. The earth is rendered using <a href='http://osgearth.org'>osgearth</a>. EVL&rsquo;s OmegaLib software is used to display the 3D models in the CAVE2 System.

ISS 3D Model as displayed in EVL&rsquo;s CAVE2 System

International Space Station (ISS) and Space Shuttle Discovery

SAGE2&trade; Scalable Amplified Group Environment

A connectome is a map of neural connections in the brain. In this CAVE2&trade; Hybrid Reality Environment application, participants walk inside streamlines depicting the white matter fiber tracts in the brain, derived from diffusion tensor images (DTI) obtained in an MRI scan. The fiber tracts are color-coded by their primary direction (green: front-back, red: left-right, blue: up-down). This research is being done by the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago in collaboration with Olusola Ajilore, UIC Department of Psychiatry. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC) and the UIC Department of Psychiatry.

Connectome application displayed in EVL&rsquo;s CAVE2

Connectome

Mechanistic models that describe the dynamical behaviors of biochemical systems are common in computational systems biology, especially in the realm of cellular signaling. The development of families of such network models, either by a single research group or from multiple sources, presents signiﬁcant challenges that range from identifying functional structural diﬀerences to keeping track of the model history. We present the development and features of an interactive model exploration system, MOSBIE, which tracks the features and development history of a family of models and provides utilities for identifying similarities and diﬀerences between models. Models within a collection can be clustered using a custom similarity metric. We also provide a visual interface for exploring collections of models, allowing a researcher to interactively compare the similarities and diﬀerences between pairs of models, search for models which contain a given structural motif, and explore the parameters and simulations of models. We evaluate the usefulness of this system with a case study in the cell signaling domain. We present the feedback provided by domain experts, and discuss the beneﬁts of this approach.

MOSBIE: A Tool for Comparison and Analysis of Rule-Based Biochemical Models

Funded by NASA, the SIMPLE project (Sub-ice Investigation of Marine and Planetary-analog Ecosystems) is to deploy an autonomous underwater vehicle (AUV) whose sensors collect biogeochemical data of the water&rsquo;s composition (salinity, pressure, conductivity, and oxygen concentrations) and photograph the underside the ice surface, and use sonar data to map the underside of the ice surface. The goal is to compare measurements of the ice shelf taken from under the ice with measurements taken from above the ice, to improve future mapping missions to Jupiter’s ice-covered moon Europa, and eventually send an AUV under that ice to search for life. SIMPLE is a collaboration of UIC, Stone Aerospace, Georgia Tech, Louisiana State University, University of Nebraska-Lincoln, San Jose State, University of Texas at Austin, and Montana State University. Prior to SIMPLE, NASA funded the project <a href="https://www.evl.uic.edu/entry.php?id=2027">ENDURANCE (Environmentally Non-Disturbing Under-ice Robotic ANtarctiC Explorer).</a>

Arthur Nishimoto in EVL&rsquo;s CAVE2 demonstrating the SIMPLE application.

SIMPLE project (Sub-ice Investigation of Marine and Planetary-analog Ecosystems)

We present the design and implementation of a visual mining and analysis tool to help identify protein mutations across family structural models, and to help discover the effect of these mutations on protein function. We follow a client-server approach in which distributed data sources for 3D structure and non-spatial sequence information are seamlessly integrated into a common visual interface. Multiple linked views and a computational backbone allow comparison at the molecular and atomic levels, while a trend-image visual abstraction allows for the sorting and mining of large collections of sequences and of their residues. We evaluate our tool on the triosephosphate isomerase (TIM) family structural models and sequence data, and show that our tool provides an effective, scalable way to navigate a family of proteins, as well as a means to inspect the structure and sequence of individual proteins.

FixingTIM: Interactive Exploration of Sequence and Structural Data to Identify Functional Mutations in Protein Families

This CAVE2&trade; Hybrid Reality Environment simulation depicts the burning of three aluminum nanoparticles at very high temperatures (approximately 2000 C) in order to study their potential use in fuel cells. Visualization by Michael Papka of the Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory and Khairi Reda of the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. Simulation and data provided by Kenichi Nomura and Priya Vashishta of the University of Southern California. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2. 

Note: This dataset cannot be visualized in real time, but EVL has a pre-rendered movie of it. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC), Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory, and the University of Southern California.

Burning Aluminum Nanoparticles for use in Fuel Cells

Professor Peter Doran of the Earth and Environmental Sciences Department at University of Illinois at Chicago (UIC) employs virtual-reality to further biological research in terrestrial environments analogous to those found on other planets, and to enable remote searches for, and identification of, life in extreme environments. Doran and EVL are part of the NASA-funded ENDURANCE project (Environmentally Non-Disturbing Under-ice Robotic ANtarctiC Explorer), a collaboration of UIC, Stone Aerospace, NASA Ames Research Center and Montana State University. Project members are analyzing 3D biogeochemical information captured by an autonomous underwater vehicle (AUV) in the extreme environment of West Lake Bonney, a perennially ice-covered Antarctic dry valley lake in the McMurdo Dry Valleys of Antarctica. They hope to apply what they learn to planetary science, such as capturing and analyzing information about the water ice surface of Jupiter&rsquo;s moon, Europa. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2&trade; System. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC), UIC Earth and Environmental Sciences Department, Stone Aerospace, NASA Ames Research Center and Montana State University.

Users exploring data collected from the botton of West Lake Bonney

ENDURANCE (Environmentally Non-Disturbing Under-ice Robotic ANtarctiC Explorer)

In this application, participants view a number of 2D high-resolution images captured by the Mars Curiosity Rover, along side a 3-dimensional model of Mars in the CAVE2&trade;. EVL&rsquo;s SAGE&trade; (Scalable Adaptive Graphics Environment) software is used to display the images in the CAVE2. 

As the CAVE2 System is a hybrid environment, it enables users to display both 2D and stereo images, as well as 3D models using SAGE software. This functionality is in particularly useful for information analysis and insight. 

The data used in this application is from the NASA Jet Propulsion Laboratory and California Institute of Technology.

Mars Curiosity imagery displayed in EVL&rsquo;s CAVE2

Mars Curiosity

Stroke is the fifth most common cause of mortality in the United States as reported by the Center for Disease Control and Prevention.  Medical treatment and physical therapy are imperative for the management of stroke, the reduction in risks of recurrent strokes and recovery of lost motor controls.  The success of a medical treatment is determined largely by adherence to therapy.  However, research has shown that adherence to therapy in patients who have had a stroke is often suboptimal.  This project focuses on the end goal of being able to test adherence, aims at creating and setting up an immersive 3D environment with an interactive game through the use of modern motion detection sensors.

MARS3 - Machines Assisting Recovery from Stroke

In this application, participants view several 2-dimensional high-resolution images captured by the Mars Curiosity Rover in EVL&rsquo;s CAVE2&trade; Hybrid Reality Environment.  Using EVL&rsquo;s SAGE&trade; (Scalable Adaptive Graphics Environment) middleware a broad range of digital media, including 2D images, files and documents, as well stereo imagery, 3D models, can be displayed for contrast, comparison and discussion. The 2D data displayed in this application is from the NASA Jet Propulsion Laboratory and California Institute of Technology.

Mars Curiosity imagery displayed in EVL&rsquo;s CAVE2 using SAGE

Mars Curiosity (SAGE in CAVE2&trade;)

Participants can explore the self-assembly of a ligated gold nanoparticle and proteins inside an ionic solution in the CAVE2&trade; Hybrid Reality Environment. Data and visualization provided by the Petr Kral Research Group at the Department of Chemistry at the University of Illinois at Chicago (UIC) and the Vincent M. Rotello Research Group at University of Massachusetts, Amherst. OmegaLib software by the UIC Electronic Visualization Laboratory (EVL) is used to display the 3D model in the CAVE2. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC), UIC Department of Chemistry&rsquo;s Petr Kral Research Group, and University of Massachusetts, Amherst&rsquo;s Vincent M. Rotello Research Group.

Gold Nanoparticle displayed in EVL&rsquo;s CAVE2 system

Gold Nanoparticle

Participants stand inside virtual train tracks in the CAVE2&trade; System to observe distortion as high-speed trains roar past. This visualization of fundamental dynamics phenomena related to high-speed rail operations was done in collaboration with the National University Rail (NURail) Center - which consists of researchers at University of Illinois Urbana-Champaign, University of Illinois at Chicago (UIC), Massachusetts Institute of Technology, Michigan Tech University, University of Kentucky, University of Tennessee, Knoxville, and the Rose-Hulman Institute of Technology - and is funded by the U.S. Department of Transportation. Understanding track structure movements, liquid sloshing, and wheel / rail contact locations are among several transportation research areas that require sophisticated virtual-reality tools such as the CAVE2 System. Visualization by the UIC Electronic Visualization Laboratory (EVL). EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago and the National University Rail (NURail) Center.

Virtual train tracks as seen in EVL&rsquo;s CAVE2

High-Speed Trains

To show medical illustrators and educators the value of virtual reality for medical training, the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago mocked up a 3D skeleton, complete with body organs, including the heart, lungs, and intestines, that students could manipulate on the walls of the CAVE2&trade; System. Datasets used are publicly available on the web. An EVL student lies on a bench in front of the walls for added effect. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2 System.

Medical training application in EVL&rsquo;s CAVE2

Virtual Reality for Medical Training

Participants fly through a close-up of a glass fissure computed in a 5-Million atom molecular dynamics nanoscale simulation. Visualization by Aaron Knoll and Michael Papka of the Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory and Khairi Reda of the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. Simulation and data provided by Kenichi Nomura and Priya Vashishta of University of Southern California. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2&trade; System. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago, the Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory, and the University of Southern California.

Molecular Dynamics displayed in EVL&rsquo;s CAVE2

Molecular Dynamics Simulation of a Glass Fissure

On-line, real-time audio dilation is a technique that time stretches, or slows down the tempo of audio signals as they are generated. This research focuses upon the control over incoming live audio tempo for increased intelligibility and comfort. 

The system features networked, full duplex communication, independent variable dilation rates, variable voice activation detection and a length (visual) audio storage buffer. Anticipated application areas include public safety, pilot / flight controllers, language translation, international communication, hearing loss, to name a few.

On-line Audio Dilation with Voice Activation Detection

Audio Dilation

SpiderSense is a wearable device that projects the wearer&rsquo;s near environment on the skin and allows for directional awareness of objects around him. The millions of sensory receptors that cover the skin presents opportunities for conveying alerts and messages. 

Drawing inspiration from Spiderman&rsquo;s SpiderSense, the Human Augmentics wearable tactile display utilizes the skin&rsquo;s pressure receptors to communicate an awareness of the surrounding environment by conveying information about the wearer&rsquo;s distance from surrounding objects. While previous tactile vests communicate directional information, SpiderSense creates an actual feeling of the environment by conveying distance information from objects by applying pressure on the skin. SpiderSense consists of a series of Sensor Modules that are positioned on the body of the user. The Sensor Modules scan the environment using ultrasound to alert the user of objects that are closer than 60 feet. Worn at strategic points over the body, the user can potentially gain a sense of all the obstacles that surround him. 

One can envision two categories of broad applications of SpiderSense: compensating for a dysfunctional or missing sense (i.e. visually or hearing impaired); or supplementing the existing senses (spatial awarness in a spacesuit or seeing behind one&rsquo;s head).

SpiderSense on a blindfolded user

SpiderSense

Participants in the CAVE2&trade; System stand inside a balls-and-sticks model of a molecular chemistry dataset. The clouds represent the density of electrons around atoms. Chemistry models and simulation data are provided by Aslihan Sumer, Juliuis Jellineck and Michael Papka of the Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory. Visualization was created by the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago and Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory.

Electron Density displayed in EVL&rsquo;s CAVE2

Electron Density of a Molecular Dataset

BactoGeNIE, the Bacterial Gene Neighborhood Investigation Environment, is a genome visualization approach developed at Electronic Visualization Laboratory (EVL) to address growing volumes of genome sequence data, specifically microbial genomes, in the biological sciences. Decreases in sequencing costs have driven a proliferation in the volume of complete sequenced genomes. Existing visualization approaches are not designed to enable the comparison of hundreds to thousands of sequenced genomes in one view. At the same time, advances in display hardware have enabled growth in display resolution and the development of large, high-resolution display environments, presenting an opportunity for high-resolution visualization design. 

This novel high-resolution genomics visualization approach transforms genomic data into high-resolution digital maps that enable rapid comparison of hundreds of microbial genomes in one view. The image shown right shows segments of 600 draft E. coli genomes from the PubMed database displayed on the Cyber-Commons display wall at the University of Illinois at Chicago, EVL. Alignment of similar genes from related bacterial strains and coloring by gene identity on a gradient around a target gene, enables researchers to quickly spot insertion, inversion and deletion events across hundreds of bacterial strains. BactoGeNIE demonstrates the potential of high-resolution display environments to facilitate large-scale genomic data analysis. 

See a <a href='http://www.youtube.com/watch?v=yrSyi1RWcUw&amp;feature=share&amp;list=UU3GTGwIcQfqqMS-Wt9uMwyQ&amp;index=2'>BactoGeNIE video</a> on the EVL YouTube channel.

BactoGeNIE displayed on EVL Cyber-Commons

BactoGeNIE, the Bacterial Gene Neighborhood Investigation Environment

omegalib is middleware designed to ease the development of applications for virtual reality (VR) and immersive display environments. A C++ and Python API. Applications can be developed as standalone executables in C++ or as scripts in Python. omegalib also supports mixed native / script applications with user-defined C++ modules that can be exposed to Python through a simple declarative interface. omelib&rsquo;s main features are: 
<ul>
<li>Support for hybrid systems, presenting high definition 2D and 3D content on the same display</li>
<li>Display system scalability: omegalib runs on desktop machines, multi-GPU workstations driving tiled displays, and cluster systems like the 36-machine, 72-display CAVE2</li>
<li>Runtime application switching: applications can be reloaded or swapped without restarting the display system</li>
<li>Web interface control and pixel streaming to HTML5 clients</li>
<li>Experimental SAGE integration</li>
<li>A customizable 2D / 3D widgets library</li>
<li>Support for a wide range of input peripherals (controllers, motion capture systems, touch surfaces)</li>
<li>Integration with third party higher level toolkits like Unity, VTK, Open Scene Graph, Qt, etc.</li>
</ul> 

Although omegalib is designed mostly to run on VR systems, it can be used to develop applications on standard desktop systems as well, leveraging the power of multiple GPUs when available.

omegalib in action

omegalib

Spaced Out uses a NeuroSky MindWave and Unity3d to generate a model of the Solar System affected by brain waves. Solar flares are triggered by high attention, time slows down with high meditation, alpha waves change the size of Saturn&rsquo;s rings, delta waves set the cloud density of Venus, and theta wave spikes sends comets toward the sun. 

This project uses a Hyundai 3D tiled display, Vicon tracking system, the Nintendo Wii for interaction, and surround audio.

Spaced Out

omicron sdk is an input abstraction and utility library for virtual reality systems, providing access to a variety of input devices, mostly used in immersive installations and stereo display systems. 

Some of the devices currently supported by omicron are: 
<ul>
<li>All motion capture systems supporting the VRPN protocol</li>
<li>NaturalPoint trackers (TrackIR, Optitrack)</li>
<li>Wii controllers</li>
<li>Xbox 360 controllers</li>
<li>Microsoft Kinect (multiple Kinects supported. omicron also offers some functions for Multi-kinect transform calibration)</li>
<li>SAGE pointer connections</li>
<li>PQLabs multitouch overlays</li>
<li>iPad touch interfaces and dynamic GUIs through the custom Porthole protocol</li>
<li>Thinkgear brainwave interfaces</li>
</ul> 

omicron abstracts input using modular event services, each providing access to a specific input device. Events can be easily streamed over the network using the omicron connector API. Also, Event services can be chained together to provide advanced functionality. 

In addition to event services, omicron provides additional utility APIs to simplify the developement of the non-graphics part of VR applications. These APIs are mostly lightweight wrappers on top of well established open source libraries. omicron includes: 
<ul>
<li>A configuration file reading system (libconfig)</li>
<li>An xml reading / writing API (<a href='http://www.grinninglizard.com/tinyxml2/index.html'>tinyxml-2</a>)</li>
<li>Multithreading support (<a href='http://tinythread.sourceforge.net'>tinythread++</a>)</li>
<li>A simple tcp client / server API (<a href='http://think-async.com'>asio</a>)</li>
<li>A mathematical / geometry library (<a href='http://eigen.tuxfamily.org/index.php?title=Main_Page'>eigen</a>)</li>
</ul> 

omicron input support can be integrated into C++ applications as a static library, or it can be run as a standalone input server, streaming input data to multiple applications. Omicron comes with client-side interfaces for Unity and Processing. 

C++, using a single header. Only a few lines of code needed. For more information see the omicronConnector section on the <a href='http://code.google.com/p/omicron-sdk/wiki/GettingStarted'>Getting started</a> page.

omicron sdk

Researchers from the UIC Departments of Computer Science and BioEngineering propose to develop graphical stereoscopic visual applications for a standard single monitor as well as an ultra-high resolution multi-paneled platform. The visualization will focus on viewing 4D-dimensional (space and time) medical images from angiography, MRI, CT, and computer simulations for an individual patient. The graphical applications developed as part of &ldquo;Artificially created cortical functional blood unit&rdquo; will allow neurosurgeons to explore patient-specific data by efficiently organizing and simultaneously displaying all relevant information. Software is being developed to extract quantitative information from patient-specific images such as contrast transit times, as well as sizes and dimensions of anatomic regions of interest. Interfaces are also being created to overlay hemodynamic simulation results into the graphical environment. The visualization tools will enhance insight about the pathology of patients suffering from neurovascular disease, especially stroke. Improved preparation for surgical intervention is expected to empower surgeons to choose better medical treatment paths. 

A semi-cylindrical multi-paneled mini-CAVE prototype is being built to render the graphical software applications mentioned above. This virtual reality environment will afford surgeons the ability to incorporate stereo depth perception and immerse themselves fully in a high-resolution environment created by the medical image. User interaction with touch inputs, tablet control, and body motion will provide a novel means of data exploration. This mini-CAVE will offer physicians a means to explore physiologically accurate patient-specific medical images with links to quantitative computer analysis. This research aims to create a three-dimensional virtual reality interface that allows physicians to interact with existing advanced scientific tools in an intuitive fashion. Tools are being created for surgical planning of medical interventions for cerebrovascular diseases that will address spatial constraints such as trajectories through bone and brain tissue, as well as avoidance of white matter fiber tracts. To assess the benefits for 4D surgical planning, a user study is proposed comparing surgeons provided with conventional standard images versus the mini-CAVE exploration environment.

Major arteries reconstructed from a medical image

Reconstructing the Cerebral Vascular Network

Modern biological experiments generate large amounts of data describing intracellular dynamics. Rule-based languages for describing intracellular biochemistry allow for the construction and simulation of detailed models with unprecedented scope and precision. Rule-Based Models (RBM) can be used to suggest new hypotheses and new ideas for future experimentation. 

Rule-based models can be difficult to understand and time consuming to debug. RuleBender is a free tool for constructing, debugging, simulating and analyzing rule-based biological models in the BioNetGen language. The MOSBIE extension further provides a tool for the comparison and analysis of biochemical model families.

RuleBender: Rule Based Modeling and Simulation with RuleBender

The growing onslaught of astronomical data available presents a great challenge. Astronomy lacks an easy-to-use and scalable way to collect and distribute expert information about objects from datasets of tens of thousands to billions of individual events and objects. Over the next decade, the amount of information available to the typical astronomer will grow by two orders of magnitude both in raw data size and in the number of objects. This project pursues three research directions, each of which has the potential to transform how astronomers interface with large datasets: 
1) a visual scalable annotation framework to enable linking of observations to specific experiments, models, or other observations; 
(2) a continuous workflow enactment system that would support automated reasoning in the presence of uncertainty; and > 
(3) a computational framework for interactively analyzing astronomical datasets, allowing the construction and testing of hypotheses. 
Project plans include the design and development of a prototype system (AstroShelf) and its evaluation using two timely science programs: 
(a) Using multi-wavelength data from the DEEP3 and AEGIS surveys, develop methods to incorporate images and catalogs from disparate datasets, allowing us to study how the demographics of galaxies have changed over the last 8 billion years. 
(b) Using properties of time-variable events found by the Pan-STARRS survey, develop techniques for rapid classification of transient phenomena, dissemination of their properties, and incorporation of feedback from follow-up observations. 
AstroShelf will include a publicly accessible, flexible annotation system for public datasets, which will also lend itself to outreach efforts involving annotations by the general public. This project\'s significant impact is the ability to share information and expert opinions quickly and widely, about each new observation or event, fundamentally changing our ability to learn about the Universe; such functionality can also be applied in support of other scientific domains.

Astronomical Data Visualization

Astroshelf: Panning and Zooming the Observable Universe

We developed a framework for integrating measured data - such as medical images, tracked motion, and anatomy-book knowledge - into the physically-based simulation of anatomical joints. The resulting models and simulations can be used to analyze medical data collected from a specific individual.

Predictive Computational Models for Orthopaedics

Simulation and modeling of turbulent flow, and of turbulent reacting flowin particular, involves solving for and analyzing time-dependent and spatially dense tensor quantities, such as turbulent stress tensors. Visual exploration of these tensor quantities can effectively steer the computational modeling of combustion systems. The project builds a visual analysis tool to be utilized in debugging, benchmarking, and verification of models and solutions in turbulent combustion.

Visual Analysis for Tensors in Computational Turbulent Combustion

Virtual Reality Environment Assisting Recovery from Stroke is a research collaboration between the Rehabilitation Institute of Chicago, UIC / EVL, and the School of Art Institute of Chicago (SAIC). Led by RIC, Bioengineering Nikolay Stoykov project participants Daria Tsoupikova (EVL) and Randy Vick (Art Therapy, SAIC) are aiming to develop technology to facilitate therapeutic training of individuals with hand impairment following stroke. 

Investigation of the kinds of techniques and devices best suited to address varying degrees of impairment experienced by stroke survivors is explored. The current focus, following the first phase of Machines Assisting Recovery from Stroke (MARS) Development Project D1 is to evaluate a full reach-to-grasp training system. One approach will be to coordinate the pneumatic system with the arm robot and VRROOM, a unique, 3D large-workspace system, which uses a haptics (robotic forces) and graphics (visual display) interface, and developed by Drs. Patton (RIC) and Kenyon (UIC / EVL). 

EVL&rsquo;s Tsoupikova is contributing her expertise in virtual reality and interactive techniques by creating a novel environment inspired by the tale Alice in Wonderland, whereby the virtual host / therapist (March Hare) directs the guest / patient to perform tasks requiring repetitive grasp-and-release and individuated finger movements in the course of interacting with the numerous animated objects in the scene.

Subject using instrumentation developed to aid stroke impairment

Virtual Reality Environment Assisting Recovery from Stroke

STCI: OptIPlanet Cyber-Mashup: Persistent Visualization and Collaboration Services for Global Cyber Infrastructure 

The goal of this project is to accelerate the growth of a burgeoning community of international institutions that have adopted OptIPortals - ultra-resolution visualization display instruments interconnected by high-speed networks. 

OptIPortals have been identified in recent DOE and NSF reports as a crucial cyberinfrastructure technology for facilitating large-scale data visualization and collaboration in science and engineering. This project will develop a collection of new community-requested capabilities for the Scalable Adaptive Graphics Environment (SAGE) - the fundamental software that drives OptIPortals. Furthermore this project will establish a persistent and expandable collaboration service to facilitate distance collaborations between organizations that have adopted OptIPortals. 

Transformative Intellectual Merit 
Coping with complexity and scale in data is a problem that spans all of e-science. The Internet now provides unprecedented access to data, and that will continue to grow exponentially. NSF&rsquo;s many cyberinfrastructure initiatives are amassing terabytes to petabytes of data, and will soon reach exabytes. 

Over the past five years, we have witnessed a trend toward the use of ultra-resolution displays and environments as &ldquo;lenses&rdquo; for observing data amassed in these cyberinfrastructure repositories. To the scientists that have built them, these displays are their new &ldquo;microscopes&rdquo; and &ldquo;telescopes&rdquo;. Our work in OptIPortals and SAGE enables scientific users to connect visualization pipelines running on supercomputers, data storage systems and/or instruments (e.g., high-definition cameras) to form a giant Cyber-Mashup on these ultra-high-resolution environments. Cyber-Mashups created on OptIPortals are a future-thinking, transformative approach to making large-scale data visualization, exploration and collaboration more scalable, and therefore affordable, for scientists and engineers to use NSF cyberinfrastructure. 

Transformative Broader Impact 
SAGE and OptIPortals are transformative technologies that demonstrate the United States&rsquo; leadership in cyberinfrastructure research. Already over 40 top institutions around the world ranging from national laboratories and universities to private industry have invested millions of dollars to install OptIPortals and high speed networking to enable them to become part of the OptIPlanet Collaboratory. 

The first adopters of OptIPortals have used them in disciplines such as: geoscience, homeland security, bioscience, cosmology, atmospheric science, chemistry, computer science, medical education and art. We anticipate that the robust and persistent technologies that will be developed and deployed in this project will enable this technology to reach out to an even larger community of scientists and students, and transform the way they make discoveries using large-scale data using cyberinfrastructure.

PIs Andy Johnson &amp; Jason Leigh Using One of EVL&rsquo;s OptiPortals

OptIPlanet Cyber-Mashup

The Next-Generation CAVE Virtual Environment, or NG-CAVE, is a scientific instrument that enables researchers to visualize data in a fully immersive 3D stereoscopic environment; it serves as the lense of a &ldquo;telescope&rdquo; or &ldquo;microscope,&rdquo; enabling them to see their e-science datasets that reside in cyberspace. Its development is being funded by the National Science Foundation&rsquo;s Major Research Instrumentation (MRI) program, award # CNS-0959053, for the period May 1, 2010 - April 30, 2013. 

Upon completion, the NG-CAVE will be the world&rsquo;s first flat-panel-based, high-resolution CAVE - i.e., Cave Automatic Virtual Environment, which the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC) built and successfully commercialized in the 1990s. The original CAVE was a 10x10x10-foot room within a room, with a resolution of 4 Megapixels and a cost of about $1M. The NG-CAVE provides potentially 10 times the stereo resolution at half the cost. This is achieved by using the latest near-seamless, flat-panel, passive-stereo, LCD technology to create a multi-person collaborative space. 

The NG-CAVE benefits large-scale collaboratories of discipline scientists in need of advanced cyberinfrastructure. Today, all science is e-science: computers capture, filter, analyze and visualize data from instruments and simulations. EVL has longstanding partnerships with a number of leading scientific research and education organizations - computer-science communities, domain-science communities, government laboratories, museums, and corporate research - who have built many of its prior visualization systems, and have provided EVL with feedback on new requirements based on their evolving scientific needs. EVL truly believes it can transform their workflows by providing them with new and more intuitive ways of interacting with their data. 

In addition, NG-CAVE benefits computer scientists, enabling them to study a wide range of new problems at the intersection of human-computer interaction, virtual reality, computer graphics, high-performance computing, high-speed networking, and computer-supported cooperative work, with the goal of providing better ways to help knowledge workers manage scale and complexity in their research. 

This MRI enables EVL to continue to provide state-of-the-art visualization instrumentation to research scientists and educational opportunities that enhance UIC graduate and undergraduate education. EVL&rsquo;s MRI-funded instruments are developed by its faculty, staff and students, and used by approximately 11 EVL-affiliated UIC departments in Computer Science, Geoscience, Medicine, Communications, Education, and Art. These instruments enable graduate students to conduct research toward their degrees, and attract and inspire undergraduates to pursue graduate degrees. Additionally, EVL has engaged scores of children through its outreach efforts with the public schools and museums.

Prototype Design of the NG-CAVE

Next-Generation CAVE

CAVE2&trade;, the next-generation large-scale virtual-reality environment, is a hybrid system that combines the benefits of both scalable-resolution display walls and virtual-reality systems to create a seamless 2D / 3D environment that supports both information-rich analysis as well as virtual-reality simulation exploration at a resolution matching human visual acuity. 

CAVE2 creates new opportunities for users to collaborate using both 2D and 3D. Today, all science is e-science: computers capture, filter, analyze and visualize data from instruments and simulations. CAVE2 serves as the lens of a virtual &ldquo;telescope&rdquo; or &ldquo;microscope&rdquo; enabling users to simultaneously see and analyze one or more e-science datasets that reside in cyberspace. 

CAVE2 benefits large-scale collaboratories of discipline scientists in need of advanced cyberinfrastructure, providing them with new and more intuitive ways of interacting with their data. In addition, CAVE2 benefits computer scientists interested in providing knowledge workers with better ways to manage the scale and complexity of their data, by enabling them to study a wide range of new problems at the intersection of human-computer interaction, virtual reality, computer graphics, high-performance computing, high-speed networking, and computer-supported cooperative work. 

CAVE2 was developed by the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC), a renowned interdisciplinary research laboratory that pioneered the development of the <a href='http://www.evl.uic.edu/core.php?mod=4&amp;type=1&amp;indi=161'>CAVE</a> virtual-reality system (1992) and the <a href='http://www.evl.uic.edu/core.php?mod=4&amp;type=1&amp;indi=273'>OptIPortal</a> scalable-resolution display wall (2004-present). These displays, deployed to hundreds of research, industrial and educational researchers and developers worldwide, enable users to see and navigate through complex datasets. 

Classic CAVEs are virtual-reality rooms with stereoscopic 3D computer graphics rear-projected onto the walls and down-projected onto the floor. OptIPortals are tiled display walls. Today, both are powered by computer clusters connected to high-speed networks to enable users to better cope with information-intensive tasks. The CAVE immerses people into worlds too large, too small, too dangerous, too remote, or too complex to be viewed otherwise. The OptIPortal creates a virtual &ldquo;project room&rdquo; in which people can display very-large images and/or simultaneously juxtapose more information, and better spatially organize, see and infer relationships among the data. The CAVE2 merges the major benefits of the CAVE and OptIPortal into a single unified environment. 

CAVE2 is programmable using a number of application programming interfaces. While some are virtual-reality only, it is EVL&rsquo;s <a href='http://www.sagecommons.org'>SAGE (Scalable Adaptive Graphics Environment)</a>, that enables CAVE2&rsquo;s huge wall to be partitioned into &ldquo;windows&rdquo; - enabling one or many 2D and 3D windows of information to simultaneously be displayed. 

CAVE2 is approximately 24 feet in diameter and 8 feet tall, and consists of 72 near-seamless passive stereo off-axis-optimized 3D LCD panels, a 36-node high-performance computer cluster, a 20-speaker surround audio system, a 10-camera optical tracking system and a 100-Gigabit/second connection to the outside world. CAVE2 provides users with a 320-degree panoramic environment for displaying information at 37 Megapixels in 3D or 74 Megapixels in 2D with a horizontal visual acuity of 20/20 - almost 10 times the 3D resolution of the original CAVE. 

The development of CAVE2, also referred to as the Next-Generation CAVE (NG-CAVE), has been funded by the National Science Foundation&rsquo;s Major Research Instrumentation (MRI) program, award # CNS-0959053 and the Department of Energy, agreement # DE-SC005067. Grants such as these enable EVL to continue to provide state-of-the-art visualization instrumentation to research scientists and educational opportunities to UIC students. EVL&rsquo;s visualization and virtual-reality instruments are developed by its faculty, staff and students, and used by approximately 11 EVL-affiliated UIC departments in Computer Science, Geoscience, Medicine, Communications, Education, and Art. These instruments enable graduate students to conduct research toward their degrees, and attract and inspire undergraduates to pursue graduate degrees. Additionally, EVL has engaged scores of children through its outreach efforts with the public schools and museums. 

CAVE2 debuted October 2012. <a href="https://www.youtube.com/user/evltube/search?query=cave2">Videos showcasing CAVE2 applications.</a> 

CAVE2 is a trademark of the University of Illinois Board of Trustees.

Exploring a model of the Starship Enterprise in EVL&rsquo;s CAVE2

CAVE2: Next-Generation Virtual-Reality and Visualization Hybrid Environment for Immersive Simulation and Information Analysis

While many visualization tools exist that offer sophisticated functions for charting complex data, they still expect users to possess a high degree of expertise in wielding the tools to create an effective visualization. Articulate is a semi-automated visual analytic system that is guided by a conversational user interface to allow users to verbally describe and then manipulate what they want to see. Natural language processing and machine learning methods are used to translate the imprecise sentences into explicit expressions, and then a heuristic graph generation algorithm is applied to create a suitable visualization. The goal is to relieve the user of the burden of having to learn a complex user-interface in order to craft a visualization.

Screenshot of Articulate system

Articulate: a Semi-automated System for Translating Natural Language Queries into Meaningful Visualizations

Understanding the social interaction of wild animals in the context of their ecology is imperative to the conservation of endangered species. Our ability to observe individuals in the wild has increased exponentially with the availability of GPS enabled sensor networks. Yet, our ability to qualitatively analyze these data is still lagging. 

The goal of this research is to develop a series of new interactive visualization methods that will allow scientists to better analyze the behavior of wild animals in order to understand how these populations interact in the wild, make use of habitat, minimize predation risks, and assess the effect of human activity on their survivability. 

EVL MS graduate candidate, Khairi Reda is collaborating with the <a href='http://compbio.cs.uic.edu/'>Computational Population Biology Lab</a>, University of Illinois at Chicago, and <a href='http://www.princeton.edu/~equids/index.html'>Equid Research and Conservation</a>, Princeton University to develop ZebrasVis, an interactive visualization tool that will assist scientists in their analysis.

Screen Shot of ZebraVis Application

Visual Analysis of Social Behavior of Wild Animals

The 4 Megapixel multi-touch LCD table prototype developed by EVL researchers focuses upon seamless and intuitive user interaction with the device. The current prototype supports simultaneous multi-user interaction, whereby participants simply reach-out and touch the images being displayed with their hands. This advancement in user interaction and instrumentation development research is a natural progression from EVL&rsquo;s development and deployment of the Lambda Table just under a year ago. 

The multi-touch LCD table prototype is the first step towards a larger system that better supports group collaboration. Also, though the prototype system only supports monoscopic information, future instrumentation research and development promises to incorporate stereoscopic imagery without the need for specialized 3D glasses.

The multi-touch LCD table prototype, showing a rain simulation

4 Megapixel Multi-Touch LCD Table Prototype

Data-intensive scientific applications require rapid access to local and geographically distributed data, however, there are significant I/O latency bottlenecks associated with storage systems and wide-area networking. In the case of climate modeling, reducing I/O latency would allow employing more complex models, which will then result in faster and more accurate weather prediction and forecasts. LambdaRAM is a high-performance, multi-dimensional, distributed cache, developed by the Electronic Visualization Laboratory (EVL) that takes advantage of memory from multiple clusters interconnected by ultra-high-speed networking, to provide applications with rapid access to both local and remote data. We are currently working with NASA Goddard to use LambdaRAM to enable climate models rapidly stride and access the Modern Era Retrospective Analysis for Research and Application data (MERRA) - a 700TB  data repository. Retrospective-analyses (or reanalyses) have been a critical tool in studying weather and climate variability. In the current experiments, a climate simulation at NASA Goddard investigating the variations in ozone levels of the atmosphere used LambdaRAM to access and stride through a 1TB MERRA data located in Chicago over a 10 Gbps optical network, and achieved a 30% performance improvement over local data access. Thus, real-time remote data access and computation can be achieve by using high-bandwidth network and high-performance data-access middleware including LambdaRAM.

Accelerating surface temperature computation using LambdaRAM

MERRA Computation Using LambdaRAM

Planetary-scale Terrain Composition is a GPU-centric real-time approach to generating and rendering planetary bodies composed of arbitrary quantities and types of height-map data, textured and illuminated using arbitrary quantities and types of surface-map data. At the core of this approach lies the assertion that modern graphics hardware need not draw a distinction between colors and vectors. Massively parallel vector stream processors allow color computation to be performed with 32-bit floating point precision, merging the processing of color with that of geometry. With the distinction between color and vector blurred, a variety of highly efficient image processing operations become applicable to both terrain geometry and terrain surface maps. 

This work demonstrates a mechanism for the real-time manipulation and display of very large scale terrain height and surface data. Beyond simply rendering terrain, this mechanism affords opportunities to combine data in powerful ways, bringing together disparate planetary-scale data sets smoothly and efficiently, and adapting to produce a uniform composite visualization of them. 

Many inter-related planetary height map and surface image map data sets exist, and more data are collected each day. Broad communities of scientists require tools to compose these data interactively and explore them via real-time visualization. While related, these data sets are often unregistered with one another, having different projection, resolution, format, and type. This GPU-centric approach to the real-time composition and display of unregistered-but related planetary-scale data, employs a GPGPU process to tessellate spherical height fields. It uses a render-to-vertex-buffer technique to operate upon polygonal surface meshes in image space, allowing geometry processes to be expressed in terms of image processing. With height and surface map data processing unified in this fashion, a number of powerful composition operations may be universally applied to both. Examples include adaptation to non-uniform sampling due to projection, seamless blending of data of disparate resolution or transformation regardless of boundary, and the smooth interpolation of levels of detail in both geometry and imagery. Issues of scalability and precision are addressed, giving out-of-core access to giga-pixel data sources, and correct rendering at sub-meter scales.

A review of the Planetary-scale Terrain Composition process

Planetary-scale Terrain Composition

NASA&rsquo;s Astrobiology Science and Technology for Exploring Planets (ASTEP) Program is supporting ENDURANCE - the Environmentally Non-Disturbing Under-ice Robotic ANtarctic Explorer - to demonstrate concepts for exploring the ice-covered ocean on Jupiter&rsquo;s moon Europa. ENDURANCE is designed to swim untethered under ice, in order to create three-dimensional maps of the underwater environment, and to collect data and samples of microbial life. 

In preparation for a future mission to Europa, a series of field tests in Lake Mendota on the campus of the University of Wisconsin at Madison, and Lake Bonney, Antarctica have been conducted to prove that ENDURANCE is suited to operating in a cold environment. One successful mission to Antarctica was conducted in last fall 2008 with another scheduled fall 2009. Additional missions are anticipated with significant planning before a Europa mission will take place likely a couple of decades from now. 

As part of the first field test in Antarctica, collected data was relayed back to EVL to generate three-dimensional images, maps and data renderings of the lake for analysis. Among EVL&rsquo;s contributions to this project is the development of a visualization application that allows researchers to visualize and compare geo-referenced images of the ice sheet covering the lake. At present, the total dataset consists of more than 100,000 images, well suited for display on EVL&rsquo;s high-resolution tiled display - LambdaVision. The tiled display allows scientists to simultaneously display hundreds of images from the acquired dataset in full resolution. The analysis of these images will help in understanding the distribution of sediments trapped in the ice surface. The goal of the ENDURANCE project at this stage, is to develop the required tools and techniques for use during the future mission to Europa. 

This project is a collaboration between UIC&rsquo;s departments of Earth and Environmental Science and Computer Science, NASA Ames Research Center, Montana State University and Stone Aerospace.

ENDURANCE Images of Ice Surface, Lake Bonney, Antarctica

NASA Environmentally Non-Disturbing Under-ice Robotic ANtarctiC Explorer (ENDURANCE)

Tellurion is a real-time planetary-scale data visualization. It uses a GPU-based algorithm to adaptively tessellate the sphere and apply terrain displacement, composing height field images of arbitrary resolution and projection. A deferred rendering approach combines per-pixel illumination and atmospheric effects for added realism. 

Data sources: NASA&rsquo;s Blue Marble Next Generation and Shuttle Radar Topography Mission. 

Funding provided by the Adler Planetarium &amp; Astronomy Museum and by the National Science Foundation cooperative agreement OCI-0225642 to University of California, San Diego for the OptIPuter project.

NASA&rsquo;s Blue Marble Next Generation &amp; Shuttle Radar Topography

Tellurion

RTK is an open source API that enables applications to co-schedule CPU cores, GPUs, Memory and Network resources within multi- and many- core computer systems. We define a &ldquo;Rail&rdquo; as the co-scheduling of two or more of these resources. Using RTK, applications will be able to create one or more rails over which their data-intensive computation and data retrieval can be accelerated without interference from other rails or applications - hence dramatically increasing performance. RTK can be used to improve the performance of high performance computing applications, high-speed data delivery applications, as well as high-resolution graphics and video streaming.

RTK: The Rails Toolkit

Individuals standing in the CAVE2&trade; System interactively fly through Earth&rsquo;s galaxy to observe constellations from different viewpoints. Data is from the European Space Agency&rsquo;s Tycho-2 catalog, which contains the positions and magnitudes of more than 2.5 million of the brightest stars collected by the star mapper of the Hipparcos satellite. EVL graduate Robert Kooima created this application; his Electro software is being used to display it in the CAVE2.

Constellations in CAVE2

Constellations

Participants interactively fly around a 3D representation of Mars in the CAVE2&trade; System. The 3D model was originally created by EVL graduate Robert Kooima using planetary data from NASA&rsquo;s Mars Global Surveyor and Viking missions and star data from the European Space Agency&rsquo;s Hipparcos mission. Kooima&rsquo;s Electro software is used to display the 3D model in the CAVE2.

Professor Andy Johnson explores Mars in EVL&rsquo;s CAVE2

Mars

The OptIPuter is a powerful distributed cyber-infrastructure to support data-intensive scientific research and collaboration. The OptiStore Project aims to develop a data management system that bridges the gap between the very large data sets (usually three dimensional volume data sets and time-varying spatial data sets) and highly intensive computing applications in the context of OptIPuter. In the OptIPuter concepts, the distributed components, such as rendering clusters, data storage clusters and computation clusters are inter-connected by wide area optical network. Thus the data that the computation cluster needs at one location may exist on different remote data storage clusters at other locations. It is the goal of OptiStore to access large amount of data (from terabytes to petabytes) on remote locations, query them on the distributed servers, transfer them among OptIPuter components, fast and reliably, and transform them from one data model to another in real-time.

OptiStore

SAGE Display Controller is a web based user-interface application designed to access and control the SAGE environment. 

SAGE, the Scalable Adaptive Graphics Environment, is a graphics streaming architecture for supporting collaborative scientific visualization environments with potentially hundreds of mega-pixels of contiguous display resolution. 

The application allows the user to interact with the environment from a standard web browser such as Firefox or Internet explorer. 

Users can move, resize or close the application windows running on the SAGE environment. The application also allows users to share their desktop on the SAGE display. 

Please, visit the <a href='http://www.evl.uic.edu/cavern/sage/sagewebui/documentation.php'>project documentation page</a> on the SAGE website to learn more about Desktop sharing.

A screen shot of the SAGE Display Controller GUI

SAGE Display Controller

Autostereoscopic displays enable unencumbered immersive virtual reality, but at a significant computational expense. This expense impacts the feasibility of autostereo displays in highperformance real-time interactive applications. A new autostereo rendering algorithm named Autostereo Combiner addresses this problem using the programmable vertex and fragment pipelines of modern graphics processing units (GPUs). 

This algorithm is applied to the Varrier, a large-scale, head-tracked, parallax barrier autostereo virtual reality platform. In this capacity, the Combiner algorithm has shown performance gains of 4x over traditional parallax barrier rendering algorithms. It has enabled highperformance rendering at sub-pixel scales, affording a 2x increase in resolution and showing a 1.4x improvement in visual acuity.