Project Overview 
Cities are the loci of resource consumption, economic activity, and innovation; they are where our looming sustainability problems were born and where those problems must be solved. Given our increasing ability to collect, transmit, store, and analyze data, we have the opportunity to go beyond today’s imperfect and often anecdotal understanding of cities to enable better operations, better planning, and better policy. To understand cities, we must analyze the data exhaust from their components - infrastructure, environment, and people - and how they interact over space and time. While there are already troves of open data about cities, their potential remains largely untapped due to unique challenges related to diversity, scale, and complexity of both urban data and computations required to obtain insights from these data. In this project, we aim to democratize data-driven and data-intensive approaches in urban computing to empower a broad range of stakeholders to analyze urban data at scale and tackle problems from different domains, including transportation, accessibility, and disaster management. We will address two critical obstacles to greater progress in urban computing: the lack of documented, robust, well-engineered tools and open computing platforms, and a dispersed community of researchers and developers. At the core of our proposal is the development of a cyberinfrastructure that integrates methods and tools for the exploration of urban data that are scalable, reusable, interoperable, and offer solutions to common challenges, including data cleaning, management, and visualization. We will deploy an integrated suite of tools in a collaborative cloud environment, enabling users from different domains to easily reuse existing methods across domains to ensure sustainability, facilitate collaboration between researchers and developers, and develop a cohesive community. 
Keywords: urban data, urban science, big data 
Intellectual Merit 
We will develop a software infrastructure to address a wide range of urban computing challenges by integrating and extending existing open-source tools as well as translating academic research products. We will also deploy a cloud-based, open collaborative environment that will support the use of this infrastructure over large and diverse urban data sets (e.g., spatiotemporal, geometry, image). The intellectual merit of the proposed effort lies within the novelty of the tools and techniques it will produce, as well as in the software engineering challenges involved in developing, maintaining and supporting code that will be deployed and widely adopted. 
Broader Impacts 
This project will contribute to the creation of a sustainable environment of urban software and tools, readily available for researchers to use and extend. The proposed infrastructure aims to propel urban computing into the open-source and open-science movement, making urban computing research more tractable, replicable/reproducible, and impactful, and delivering immediate benefit to a broad range of disciplines, including urban planning, urban design, transportation, GIS, data science, and human behavior studies. It will reduce barriers to entry for newcomers to urban computing, through intuitive interfaces, easy access to data, and participation in an active community. This project will contribute to education in computer science and urban computing-related areas through new graduate-level curricula to educate the next generation of urban scientists about computing, and expose computer and data scientists to urban-related problems. We will lead collaborative projects, seminars, and workshops to incentivize the creation of deeper connections across urban computing-related areas. We will release all software produced using a permissive open-source license to afford dissemination, commercialization, and incorporation into other software packages. We will actively contribute to our open-source projects to build community and ensure sustainability. The PIs of this project have a proven record of releasing data, widely used tools, and frameworks that are distributed as open source.

Urbane&rsquo;s map view. The number of NYC taxi pickups (b) is visualized for a chosen time range (a,c).

Collaborative Research: Frameworks: Cyberinfrastructure to Catalyze and Sustain the Urban Computing Community

Developing sensing circuits and systems capable of perceiving the physical environment is an essential step toward next-generation cyber-physical systems that leverage computation, communication, and sensing integrated with the physical domain. The recent introduction of a class of artificial intelligence (AI) methods known as deep learning has drastically advanced vision-based environment perception. However, the applicability of these methods is significantly limited by their reliance on vast amounts of 3D annotations for training, which are expensive, unnatural, and often unobtainable. Acquiring 3D annotations necessitates specialized data captured by 3D sensors rather than utilizing abundant and readily available images or videos. Moreover, it requires a controlled environment or compromises with inaccurate subjective labeling. This project aims to address real-time environment perception in a very challenging, largely unexplored, yet highly practical setting, where only 2D annotations are available during training, by modeling and learning rich space-time structures of the environment. The outcomes of this project will significantly impact cyber-physical systems and facilitate a wide range of applications, from robots in manufacturing and personal services to autonomous vehicles that enhance people’s mobility and safety. Furthermore, this project will tightly integrate research and education at the University of Illinois Chicago (UIC), which is a Minority Serving Institution (MSI), through curriculum development, research training for high school, undergraduate, and graduate students, broadening the participation of female and minority students, and community outreach. 
This project will address real-time environment perception when only 2D annotations are available during training by systematically pursuing a novel approach that decomposes the complex physical environment into geometry and motion substructures in 3D space and models their rich space-time interactions. From streaming video, the trained system will not only recognize objects and scene layouts in the environment but also estimate their 3D geometry and 3D motion in real-time. This project is focused on (1) establishing a self-supervised framework for lifting 2D objects to 3D, by bridging geometry and motion, brightness constancy, and differentiable depth rendering, (2) scaling 2D-to-3D lifting of individual objects to perceive the entire environment, by modeling and learning short-term and long-term dynamics of the environment, and (3) achieving flexible and generalizable environment perception by handling articulated motion and out-of-distribution environments. The new approach is expected to be efficient in terms of the amount of annotations required for training, scalable to a wide range of objects, and robust to the complexity and diversity of real-world environments. This project will advance and enrich the fundamental research of visual sensing, perception, and learning. Moreover, this project will demonstrate the usability and robustness of the proposed approach in real cyber-physical systems.

CCSS: Modeling and Learning Space-time Structures for Real-time Environment Perception

EVL graduate research assistants are working with Argonne National Laboratory scientists on the project entitled &ldquo;Tele-autonomous Dexterous Robotics&rdquo;, contributing expertise in (i) building digital models of the key components (robot arms, manipulator, some processing equipment) and optimization of models for real-time, high spacial resolution interaction; (ii) explore Virtual and Augmented Reality (VR and AR) technologies/methodologies/architectures; and (iii) design an immersive display with high spatial resolution and low latency. 
The tasks include: 
* Explore Virtual and Augmented Reality (VR and AR) technologies/methodologies/architectures. 
* Build and optimize an immersive display for real-time, low latency, high spatial resolution tele-autonomous control of the remote processes.

Tele-robotics to Tele-autonomous Robotics for Isotope Production

EVL graduate research assistants will work with Argonne National Laboratory scientists to develop enhancements to SciStream, a platform that enables memory-to-memory data streaming between federated scientific instruments residing in different security domains with no direct external network connectivity. We will explore utilizing advanced packet processing technologies such as DPDK, programmable processors such as DPU and programmable networking to improve the performance of SciStream data plane. We will also benchmark SciStream performance for different deployment scenarios including deploying SciStream alongside Globus Connect Servers on existing DTNs and deploying SciStream on dedicated streaming DTNs. We will also benchmark the performance of SciStream between Argonne&rsquo;s Advanced Photon Source and the Argonne Leadership Computing Facility. 
Research activities include: 
1) Research and integrate advanced packet processing and programmable data processing in SciStream and characterize performance in testbed environments. 
2) Evaluate the performance of SciStream in testbed and production environments for various deployment scenarios. 
3) Create reports and contribute to the development of scholarly manuscripts.

Enhancements to SciStream, a platform to enable memory-to-memory data streaming between federated instruments

Curio is a framework for collaborative urban visual analytics that uses a dataflow model with multiple abstraction levels (code, grammar, GUI elements) to facilitate collaboration across the design and implementation of visual analytics components. Curio allows users to build and iterate on dataflows with reusable modules, supporting collaborative design and tracking changes. We evaluated Curio with domain experts on urban accessibility, climate, and sunlight access. 
Video overview, source code, examples, and tutorials are available here: <a=href"https://urbantk.org/curio/">https://urbantk.org/curio/</a>

Curio: A Dataflow-Based Framework for Collaborative Urban Visual Analytics

Searching for scientific documents is a pervasive task among researchers. While most academic search engines, such as Google Scholar, search over text data, image data provides relevant information to complement text-based queries. For instance, in biomedical and biology research, images show experimental results and provide cues about the methods followed. This work shows the importance of combining text and image-based features to aid researchers in finding relevant documents in a COVID-19 collection. We derived a taxonomy to represent image content based on their acquisition method (e.g., captured in a microscope), integrated this information by leveraging deep learning image classifiers, and presented the result in hybrid image+text surrogates. Our design improved the user experience during document retrieval.

The system architecture. The online components include: web interface, application server for data retrieval, and Apache Lucene search engine. The offline components are responsible for labeling, model training and document preprocessing.

Enhancing biomedical search interfaces with images

The development and construction of facilities that allow for efficient data sharing while also providing computational resources for data analysis is becoming a crucial activity across all fields of science. In many research areas, the generation of large-scale, sophisticated simulations on HPC platforms is now as important a part of science as are the associated experimental and theoretical studies. The simulations can be computationally expensive, and often only a limited number of codes exist that can take full advantage of large-scale HPC systems. Therefore, sharing simulation results within the community to enable comparisons to experimental data and further data exploration in search of new insights is essential to push science forward. 
EVL graduate research assistants work with Argonne National Laboratory scientists on the following tasks: 
• Active development of the data portal to support releasing cosmological simulation data to the community. 
• Test data portal and develop tests to increase the portal&rsquo;s robustness. 
• Develop of documentation and tutorials on the use of data portal.

Data Sharing and Analysis Across Leadership Computing Facilities

Advanced radiotherapy treatments for Oropharyngeal cancer have improved patient survival rates, although they lead to understudied long-term symptoms. By analyzing three-dimensional dose distributions to vital organs via custom unsupervised machine learning, we identify correlations with long-term patient-reported symptoms. Our innovative approach surpasses existing clinical guidelines, offering actionable insights for optimizing treatment plans and minimizing post-treatment side effects. 
This work by visual computing researchers at EVL is in collaboration with clinicians at MD Anderson Cancer Center from Texas and data mining specialists at University of Iowa, and it is supported by funding from the US National Institutes of Health (NIH awards NCI-R01-CA258827 and NLM-R01-LM012527), and US National Science Foundation (NSF awards CDSE-1854815 and CNS-1828265).

Overview of the methods used for each symptom of interest.

Multi-Organ Spatial Stratification of 3-D Dose Distributions Improves Risk Prediction of Long-Term Self-Reported Severe Symptoms in Oropharyngeal Cancer Patients Receiving Radiotherapy: Development of a Pre-Treatment Decision Support Tool

We present an interactive analysis system to support sequential rule mining (SRM) model builders who work in cancer symptom research. SRM is a promising unsupervised machine learning method for predicting longitudinal patterns in temporal data which, however, can output many repetitive patterns that are difficult to interpret without the assistance of visual analytics. Our system facilitates mechanistic knowledge discovery of treatment-related toxicities (i.e., "roses have thorns") in large-scale cohort data. 
This work was developed by visual computing researchers at EVL in collaboration with data mining specialists at the University of Iowa, and clinicians at the MD Anderson Cancer Center in Texas, and it is supported by funding from the US National Institutes of Health (NIH awards NCI-R01-CA258827 and NLM-R01-LM012527), and US National Science Foundation (NSF awards CDSE-1854815 and CNS-1828265).

Symptom clustering for treatment ICC.

Roses Have Thorns: Understanding the Downside of Oncological Care Delivery Through Visual Analytics and Sequential Rule Mining

EVL student researchers work with Argonne Leadership Computing Facility (ALCF) group on high-performance computing applications. Research and development include: 
• Visualization and analysis of abstract datasets based on ALCF operational logs. 
• Vsualization of workflow execution and data transfers. 
• Dashboard development for monitoring of systems. 
• Development of code and algorithms that demonstrate the use of Intel oneAPI for the ALCF supercomputer Aurora. 
• Integration of visualizations using Unity and Unreal gaming engines into virtual reality environments from data generated using Argonne supercomputers. 
• Production of documentation and sample code as needed.

High-Performance Computing Research

Sidewalks impart a safe walkway for pedestrians, links public transportation facilities, and equip people with limited abilities with routing and navigation services.  There is a scarcity of open sidewalk data worldwide.  As one of the most famous crowdsourced cartographic data repositories and the only globally available open geospatial data, OpenStreetMap (OSM) could aid the lack of open sidewalk data to a large extent.  However, completeness and quality of OSM data have long been a major issue.  In this research, we analyze city-scale availability and trustworthiness of sidewalk networks.  As part of this work, we compare the coverage of sidewalks in OpenStreetMap across 50 cities in the US by computing the ratio of highways to sidewalk geometry and the percentage of highways containing sidewalk information.  Additionally, we plot the availability of OpenStreetMap sidewalk attributes in a number of cities, and compare the distribution of trustworthiness index of sidewalks, and highways with sidewalk information in these cities.

Sidewalk coverage of (a) Seattle, (b) a sportion of downtown Seattle, & (c) View Ridge neighborhood. Overall Seattle OSM has comparably higher sidewalk geometries mapped than other cities. However, smaller neighborhoods like View Ridge have poor coverage.

Crowdsourcing and Sidewalk Data: A Preliminary Study on the Trustworthiness of OpenStreetMap Data in the US

Researchers at UIC from the Departments of Computer Science and the Department of Earth and Environmental Sciences are studying how to design technology-enabled group learning experiences for Earth Science classrooms. The purpose of the GeoDataOnSphere project is to investigate how future teachers collaboratively explore and make meaning from large-scale geoscience data visualizations. Our team hopes to use the lessons learned from this research to design engaging collaborative learning software applications for Earth Science classrooms. 
This research is sponsored by <a href="https://isgc.aerospace.illinois.edu/ ">the Illinois Space Grant Consortium</a> via a subaward to UIC from UIUC. Nikita Soni, Ph.D., a Computer Science Assistant Professor is the lead principal investigator on the project working with co-PI Stefany Sit Ph.D., a Clinical Associate Professor in the Department of Earth and Environmental Sciences. Feel free to contact Nikita Soni, Ph.D. at nnsoni@uic.edu for any queries.

The data visualization application running on the multi-touch spherical display was developed by <a href="https://pufferfishdisplays.com/">Pufferfish.Ltd.</a>

GeoDataOnSphere: Collaborative Exploration of GeoData on Multi-touch Spherical Displays for Future Teachers

EVL is a technology Innovator, building industry and cross-institutional connections and partnerships. In 2022, EVL deployed two GPU FIONAs in partnership with the Pacific Research Platform (PRP) and the University of California San Diego. The PRP is a partnership of more than 50 institutions that establishes a science-driven high-capacity data-centric &lsquo;freeway system&rsquo;. PRP has support from the National Science Foundation, and includes the US Department of Energy&rsquo;s Energy Science Network (ESnet), as well as scores of research universities in the nation and around the world. PRP leverages Nautilus, a HyperCluster for running containerized Big Data Applications using Kubernetes, and FIONA, flash input/output network applications. A GPU FIONA is a rack-mounted PCs designed by the PRP team that is optimized for 100Gbps large data storage with Mellanox ConnectX-6 SmartNICs, a 24 core Epic Rome CPU, 256GB of memory and 8 NVIDIA RTX-3090 GPUs. Nautilus provides over 7500 CPU cores, over 500 GPUs and 4 PBs of storage. The addition of two Nautilus GPU FIONAs at EVL builds upon our 2018 DTN (Data Transfer Node) that has provided over 300GB of Ceph Storage to Nautilus and our CAVE2 GPU Nodes that are contributing 5 NVIDIA GPUS. EVL provides Nautilus with 300GB of Ceph Storage, 128 CPU cores, and 21 NVIDIA GPUs. The platform is utilized for a wide variety of applications, including applying advanced astronomy AI to microscopy workflow, simulations for natural disasters, climate modeling, virtual reality and even accelerating NSF&rsquo;s IceCube Photon Propagation. UIC is enabled to participate in these cross-institutional connections using UIC&rsquo;s HPRNet, a network environment tailored to the needs of high-performance science applications, built on a Science DMZ architecture. 
<a href="https://pacificresearchplatform.org/media/">Read more about PRP.</a> 
<a href="https://pacificresearchplatform.org/media/sc21-larry-smarr-on-the-rise-of-supernetwork-data-intensive-computing/">See media coverage from SC21 on PRP and Nautilus.</a>

FIONA

Pacific Research Platform (PRP) - Nautilus

Deep Umbra is a novel computational framework that enables the quantification of sunlight access and shadows at a global scale. Our framework is based on a conditional generative adversarial network that considers the physical form of cities to compute high-resolution spatial information of accumulated shadow cast for the different seasons of the year. 
Deep Umbra&rsquo;s primary motivation is the impact that shadow management can have on people&rsquo;s quality of life, since it can affect levels of comfort, heat distribution, public parks, etc. We also present the Global Shadow Dataset, a comprehensive dataset with the accumulated shadow information for over 100 cities on 6 continents. 
A <a href="https://www.evl.uic.edu/shadows/map">webviewer</a> has been developed to visualize accumulated shadow cast over a specific geographic area. The map shows the accumulated shadow for three different days of the year - Summer, Spring/Fall, and Winter.

Deep Umbra: A Global-Scale Generative Adversarial Approach for Sunlight Access and Shadow Accumulation in Urban Spaces

The COVID-19 pandemic has upended the world over the past year, in a health crisis not seen since the 1918 flu pandemic. As of July 2021, the disease has caused the death of more than 4 million people worldwide, 600,000 in the US alone. While waiting for an
effective vaccine amidst the surging cases worldwide, governments adopted measures to inhibit the spread of the disease. Despite these efforts, waves of COVID-19 infection with varying characteristics, continued to ravage communities [24], highlighting the necessity to understand the spatio-temporal complexity of the problem. Forecast models are among the tools available to public health experts and policymakers to predict likely pandemic outcomes (i.e., number of cases, deaths) and prepare for different scenarios. In the past year, several models have been proposed by a myriad of experts, and using vastly different assumptions, methodologies, parameters and data sources. As a result, models produce a range of sometimes radically different forecasts, especially at higher spatial resolutions, limiting their use by decision makers and the public in general. In order to address this problem, more recently, ensemble models have been proposed for COVID-19 forecasts, with promising results. Our research in this area has resulted in COVID-19 EnsembleVis, a web-based interactive visual interface that allows the assessment of the errors of ensembles and individual models by enabling users to effortlessly navigate through and compare the outputs of models considering their space and time dimensions. COVID-19 EnsembleVis enables a more detailed understanding of uncertainty and the range of forecasts generated by individual models.

Enabling the visual analysis of COVID-19 ensemble forecast models with COVID-19 EnsembleVis.

COVID-19 EnsembleVis: Visual Analysis of County-level Ensemble Forecast Models

DASS is a visual computing system designed to help scientists build machine learning models that classify head and neck cancer patients into groups, based on their risk of developing permanent side effects from radiation therapy. Its visual interface helps data scientists to build a model for each side-effect by visualizing the patients&rsquo anatomical and other clinical data, and to figure out what parameters the model should consider, such as which organs may be damaged when side effects occur, what amount of radiation dose causes the damage to the organs, and how many risk groups to create. DASS also has features for explaining to clinicians the finished models and examining whether their predictions perform well, and the reasons behind why a given patient belongs in a certain risk group. DASS was created by visual computing researchers at EVL, in collaboration with clinicians at the MD Anderson Cancer Center and data mining specialists at University of Iowa, and is supported by funding from the US National Institutes of Health (NIH awards NCI-R01-CA258827 and NLM-R01-LM012527), and US National Science Foundation (NSF awards CDSE-1854815 and CNS-1828265).

Stylized scatterplot. Patients are represented by a custom glyph that encodes the outcome of interest as marks extending radially. Markers are colored by cluster membership, and a contour is shown around the currently selected cluster.

DASS

Cancers that depend on the spatial location of the disease affect all ethnicities, and are diagnosed each year in large numbers in the United States, leading to large, rich repositories of patient data. This project develops a longitudinal decision support tool to assist clinicians in selecting treatments that balance survival and side-effects responses to cancer therapy. The project will lead to better insights into the biological and therapeutic factors related to cancer survival and toxicity, allowing personalization of care at the individual level. 
This research is sponsored by the National Institutes of Health: National Cancer Institute, and led by EVL faculty and Computer Science associate professor Liz Marai (PI) and co-PI Xinhua Zhang, along with collaborators Guadalupe Canahuate at University of Iowa, and Dave Fuller at University of Texas, MD Anderson Cancer Center.

Longitudinal Spatial-Nonspatial Decision Support for Competing Outcomes in Head and Neck Cancer Therapy

The UIC School of Public Health (UIC-SPH) Emergency Management and Resiliency Planning (EMRP) program, in collaboration with the Electronic Visualization Laboratory (EVL) and content experts, is presenting a series of Midwest Comprehensive Visualization Dashboard(s) (MCVD) focusing on environmental health and justice issues in this region. A MCVD has two components: the visualization interface(s), i.e., link to interactive maps, and the end-user documentation (i.e., background information, data sources, methodology, findings, etc.). 
<a href="https://univofillinois.maps.arcgis.com/apps/instant/nearby/index.html?appid=486f6e438ecf4b048deebc8dafd9f2c1&sliderDistance=1">Link to the interactive map on the MCVD dashboard.</a> 
MCVD has been created by Joel Flax-Hatch, Michael D. Cailas (School of Public Health), Fabio Miranda (EVL), and Apostolis Sambanis (Division of Environmental and Occupational Health Sciences - EOHS)) with the participation of Michael Siciliano (College of Urban Planning and Public Affairs), Sybil Derrible (Civil, Materials, and Environmental Engineering), Yuan Shao (EOHS), and Greg Arling (Purdue University). 
<a href="https://ago-item-storage.s3.us-east-1.amazonaws.com/0e1922e480134854918ed0f3f97319fe/MCVD_EJ.3.V1.1.pdf?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEKb%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCID3FXdMyOfYIb%2Fafu1eOW49fXssLx10%2Bs%2FOh29tg7m7yAiEA1YqdmEA8DmO%2FPfoparcE3Mr%2BOmRA89eoTEPh8ZoxOPcqgwQI3v%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgw2MDQ3NTgxMDI2NjUiDITVsh%2FbXDCTjerOWyrXA9skMbqdx8pG2RMMt4k5ms%2BFG30KSO3pLnufYGx%2B7nvR%2FVMNwS4lJZoAEhMED%2B6y22UVHgV1N3lUcEhAuc6X8i2ZiFHReKQDLODjfuL9ZoU3ubq%2FhOXmxaqZTw%2BSZ852w%2FeZK77wD0TsrUcabCDNLManoYWOMxkwqiPnyHChuwCON8529O6KlCMp%2BoXYl%2BMk7AXVdS3Fd1onYEy10MpPnhVN%2Bp0cdixCuaxfcmN3EPxfbY4q667z5fyg1fcDBfKgW8tk%2B5kzXMxpk0V%2Bq3Q3pII%2B9i3872DDEZFicjSLaLTt1w%2BHGg22mn9wBH7tFDTzmDF6UJWLpuRXnw3Ek8MWmDqdB8BWgxAbFmJbmQBGEcB9Bb%2FxAKIReMXFtrM2nThAMcOfx%2BwuiPlqVuHAdbjN6gtUrd3tX8Kd7WI3Q2mRzok5nf8XZ96EmYr3dA3ck7fBiC2ywkCzhn6U%2B4919EFAy3gJkOEbSNileqOKg1L%2Fv%2FOn4MRXwsQbAXJsb6Fq4XO%2BzYfUUvHHviqYMDdyIZU1Za9mPX9UER9XdrBJ9D6%2BDiJmOnbEz6MKAdjzAjDwmHqR0VHWj%2B8zVk98xa0x%2FLxol1CHliq5YjWT%2FfPBl3GoVRXFtorDyjkJzDCil4aQBjqlATd4Im%2B%2F5Hnm3WhZD3xNPje0qXVNEpEczPEUg77lN747bEEo47cI25tpDgVFxS%2BTkWv4Bqv2ylplFaPThOnwvUW1ms4f7Br3Z%2BI64bISJNaBXGv%2F7hb68bFvHsCc8yqBo1TcZcw6CaOyLyLfA5dAz1BKQNpdoI0K5m9DLHlBtZaAIn2uXUp75ttP0UqMXjhTDsHJWwkuM%2FQNDPGMGtbHJMynIEjZ7w%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20220207T221949Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=ASIAYZTTEKKE2RWCPQVC%2F20220207%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=3d12d4d450d382bd441b4d23e229780f45b867eae9a23cec52b1794716f8d1cc">Read the press release outlining this research project.</a>

MCVD: Environmental Justice Landscape Map Interface

Midwest Comprehensive Visualization Dashboards: A New Environmental Justice Tool for Chicago Communities

PI: Andrew Johnson (UIC/EVL); co-PIs: Barbara DiEugenio (UIC/CS), Moira Zellner (Norheastern University) 
Collaborator: Jason Leigh (University of Hawaii Manoa) 
The Internet gives the public, including scientists, access to vast amounts of data. Visualization is a powerful tool to take that data and turn it into a form that is easier to understand, helping people make better decisions. However, creating good visualizations from data is not easy. The Articulate+ Project will work to generate new knowledge about how to empower people with information visualization, by using speech and gesture as inputs, in new ways. A goal is to enable users to benefit from imprecise statements about what they want, rather than needing to give specific commands telling the computer what to do. This project will make it easier for citizens to engage in conversations with the computer about economic, medical data, demographic, transportation data, climate, and sports data; to answers that are more meaningful, and so to democratize data literacy. 
This research will use speech, gesture, and log data as new input interaction modalities for visual analytics. It will develop new understandings of how the imprecise and vague nature of natural language queries and gestures, as contextualized by work on a specific visual analysis problem, can be modeled as an opportunity for an intelligent software system to learn more about the underlying intent of the users. This model of intent, in turn, will be used to provide contextualized visualizations, which are expected to help those users gain valuable insights from their data. The audio speech data will be used to computationally model overhearing, to help infer users&rsquo; current contextualized goals. The gesture data will be used to disambiguate expressions that refer to visual elements of a visualization (e.g., &ldquo;that pie chart about electricity consumption&rdquo;). The project will develop new understandings of how to combine the log of visualization state, the computational model of overhearing, and the gesture data in order to generate new visualizations in support of users&rsquo; work on tasks. Evaluations will be performed in both controlled laboratory and naturalistic study settings to determine whether effective semantic parsers can be derived for these specific visual analysis domains, and how this contextual interface affects users&rsquo; experience of visualization and discovery. The integrated annotated datasets from the studies will be made available to the research community satisfying a need for ecologically valid situated language interaction corpora.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Articulate+ - A Conversational Interface for Democratizing Visual Analysis

UIC PI: Andrew Johnson, co-PI: Luc Renambot 
University of Hawaii Manoa PI: Jason Leigh 
Virginia Polytechnic PI: Chirstopher North 
NSF Award Number: 2003800 
Award Period of Performance: 05/15/2020 - 04/30/2025 
Total Amount Obligated: $2,250,000 
The Big Data revolution necessitates the use of sophisticated tools such as Artificial Intelligence (AI) and Data Visualization to harness the sheer volume, velocity and variety of datasets that are becoming the norm. However, it is the research community that must make sense of the data being amassed, so cyberinfrastructure must extend to people. SAGE3 (Smart Amplified Group Environment) puts the &ldquo;human in the loop&rdquo; by providing scientists with an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help them access, share, explore and analyze their data, come to conclusions, and make decisions with greater speed, accuracy, comprehension and confidence. 
SAGE3 builds upon the NSF-funded first-generation SAGE (Scalable Adaptive Graphics Environment) and second-generation SAGE2 (Scalable Amplified Group Environment), which are today&rsquo;s de facto operating systems to manage Big Data on tiled display walls. Its development focuses on two fundamental components: AI-enhanced &ldquo;smart&rdquo; services and advanced computing resource orchestration to support reproducible work models for secure collaborative work. SAGE3 amplifies user productivity, providing them with commercially available and open-source AI solutions, which autonomously and transparently analyze data while continually learning and improving through user interactions. SAGE3 makes AI technologies broadly accessible, not just a privilege for the technically savvy. SAGE3 further democratizes AI by using Data Visualization to help interpret and explain AI models so users better understand how AI came to its decisions, which engenders user trust and can help identify potentially prejudiced or biased models. 
SAGE3 augments every step of the scientific discovery enterprise - from quickly summarizing large data, to finding trends and similarities or anomalies among one or more linked datasets, to communicating findings to scientists, public policy and government officials, and the general public, to educating the next-generation workforce. Ultimately, it is the scientists and future scientists who must Harness the Big Data revolution to solve the nation&rsquo;s grand challenge problems that will benefit society as a whole - from studying the diversity of life on Earth, to understanding the Earth and its systems from satellite imagery of its poles, to developing response scenarios for natural disasters such as landslides and pandemics that impact the citizens and economies of the world. 
<a href="https://youtu.be/kvNSLGcs5KE">View a video of SAGE3 being demonstrated</a> on EVL&rsquo;s YouTube channel.

SAGE3: Smart Amplified Group Environment for Harnessing the Data Revolution

PI: Maureen Dunne, Disability and Human Development, UIC 
co-PI: Tamar Heller, Disability and Human Development, UIC 
co-PI: Daria Tsoupikova, New Media Design; Electronic Visualization Laboratory, UIC 
In recent years, the shortage of technology talent in Illinois and the outmigration of engineers to other states has been identified as a major concern. It is also estimated that approximately 85 percent of people with autism spectrum disorder (ASD) remain unemployed or underemployed. ASD is the fastest growing developmental disability and is diagnosed more frequently than AIDS, cancer and diabetes combined, with 1 in 59 children now being diagnosed (CDC, 2018). The mission of the Autism and Innovation Initiative is to strengthen the technology and innovation talent pipeline in Illinois by including people with autism spectrum disorders (ASD) and taking a strength-based approach to training, entrepreneurship and workforce development. 
In pioneering an inclusive model to training, attracting and retaining talent in Illinois, the program will offer opportunities for engagement with innovative technology projects to inspire enthusiasm for technology, innovation and entrepreneurship. The Autism and Innovation Initiative, in collaboration with industry and community partners, will spearhead the foundation of a statewide Innovation Hub that takes a strength-based approach to including individuals with autism in the Illinois STEM workforce. 
The initial milestones this initiative will accomplish include 1) Coding Bootcamps and Hackathons in different regions of Illinois; 2) STEM Faculty Training Workshop on ASD; 3) Community College Bridge to UIC Program to Support STEM Transfer Students with Autism; and 4) Autism and Technology Working Research Group and Seminar Courses to drive interdisciplinary collaborations, innovative technology solutions and entrepreneurship. While a core focus is on integrating persons with autism into the Illinois workforce, the statewide Innovation Hub seeks to invigorate the overall technology pipeline and retain engineering talent. 
Industry Partners: Microsoft Technology and Civic Engagement Group, Yolobe.com, Chamber 630, Code DuPage, Venture 1818 Partners, Autism 911, LearnStep, Inklusion Works, Inklusion Capital, Social Skills Hero, TMT Partners 
Community Partners: Autism Community Ventures, Autism Society of Southern Illinois, Mental Health Association of Greater Chicago, The Transiiton2Success Project, MakerSpace Lab, Community High School District 99, University Centers for Excellence in Developmental Disabilities (University Partner), Leadership Education in Neurodevelopmental Disabilities (University Partner), Grupo Salto, The Autism Program of Illinois (TAP) 
Funding: Discover Partners Institute

Discovery Partners Institute: The Autism and Innovation Initiative

In this project, a novel computational method to model the evolution of aircraft contrail (ice particles forming in aircraft engine&rsquo;s exhaust) using high-performance computing is being developed. The specific goals of the research are: (1) to carry out the first fully three-dimensional spatial large-eddy simulations of contrail formation that include the aircraft geometry and to develop accurate data-set of contrail evolution in the early phases of the aircraft wake; (2) to identify the 3D contrail features and fit parameters by journaling the simulation workflow using advanced visualization techniques; and (3): to apply reduced-order modeling, such as Polynomial Chaos reconstructions, and machine-learning techniques, such as Artificial Neural Networks, to estimate model parameters and to reduce the large dimensionality of the problem to ease integration into global atmospheric models. 
EVL&rsquo; visualization work aims to create a multi-scale visual computing system that can identify the spatial features and parameters of the 3D contrails to analyze and explore multiple simulations of the ensemble data. These simulations can be performed for many different cases with specific parameter combinations. A guided summary and comparison of all of the simulations, aim to identify areas of interest across multiple outcomes. Additionally, the visual interface of the visualization system developed will provide users the ability to manipulate input or output parameters to find ways to minimize the formation of the contrails. An anticipated outcome, will be to enable climate researchers to select and examine specific simulations under given parameter values and gain more insight.

Visualization System Layout Prototype

High-performance computing and data-driven modeling of aircraft contrails

Cancer has a major impact on society all around the world. It affects people of all population groups though certain groups bear more burden of cancer. Some of the contributing factors for cancer disparities are socioeconomic status, geographic location, race/ethnicity. The objective of this project is to provide an interface to show cancer incidents and address the effect of different socioeconomic factors on cancer incidents in the areas of Chicago. 
The web-based application was developed by Computer Science Master degree-seeking graduate student Yashika Goyal, under the advisement of EVL and Computer Science associate professor, G. Elisabeta Marai.

Screenshot of Cancer Disparities application

Cancer Disparities in Chicago

The COMPaaS DLV: Composable Platform as a Service Instrument for Deep Learning & Visualization user community consists of faculty from Physics and four College of Engineering departments (Computer Science, Civil, Materials & Environmental Engineering, Mechanical and Industrial Engineering, and Electrical and Computer Engineering) - primarily run applications that are GPU-centric for compute, with significant variability in storage and networking around their data requirements. 
Computer Science applications primarily focus on security, data science, computer vision and Machine Learning (ML). Security projects explore the complexity of modern web applications and the intricacies of security mechanisms that often result in flaws that expose users to significant security and privacy threats. These projects try to develop methods and tools that enable users to understand and more effectively manage retrospective privacy in the context of modern, long-lived, online archives. Composable resources were used to develop Natural Language Processing (NLP)-based domain-specific classifiers that identified data practices stated in privacy policies. Adherence of corresponding applications were then adjusted based on this ground truth. 
Data-science applications have an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help users access, share, explore and analyze their data, whether local or remote, come to conclusions, and make decisions with greater speed, accuracy, comprehensiveness and confidence. One such project is developing and advancing tools that identify image data in biomedical literature to locate beneficial, targeted publications. This work involves training image classifiers, integrating classifiers into labeling pipelines, designing retrieval user interfaces, and identifying related visual representations. 
Computer vision projects include semantic segregation and 3D human pose estimation. Researchers are developing a novel network architecture, termed DependencyNet (dependency network), for semantic segmentation. They also achieved experimental results that demonstrate an effective approach for 3D human pose estimation. Over the past two years, they found COMPaaS to be consistently stable and efficient, with result output as expected, and the group&rsquo;s models achieved state-of-the-art performance on their respective benchmarks. 
ML applications include frameworks for many complex real-world reinforcement learning problems, such as the coordination of autonomous vehicles, network packet delivery, and distributed logistics. 
Civil, Materials & Environmental Engineering applications focus on simulation and modeling. Researchers run data-driven models on high-performance computers to develop an accurate and general neural network ML model that uses crystallographic data to study patterns of synthesizability. They also perform simulations of mass transport in alloys and ceramics. COMPaaS has been performing 1.5-10 times faster than comparable infrastructures they are familiar with. Additionally, they found our use of Jupyter Notebooks to be a significant asset. 
Mechanical and Industrial Engineering researchers use COMPaaS for three research projects: feature extraction in fluid flow using a Convolutional Neural Network (CNN); column height detection in metallic nanoparticles using a CNN; and, electric vehicle battery state-of-charge estimation using different ML methods. 
Electrical and Computer Engineering researchers recently started running mathematical models of ML algorithms and training language models using long-short term memory (LSTMs). 
In 2020, we began implementing a GPUoE (GPU over Ethernet) prototype. Using a GPU expansion chassis connected to compute nodes over Ethernet, we were able to compose remote GPUs into our existing composable infrastructure. Our APIs developed for composable infrastructure have been extended to support these remote GPUs. 
In 2021, we enhanced COMPaaS with a public-facing JupyterHub server providing secure web-based access to experiments, a modern PCIe fabric and a supporting expansion chassis, thereby making composable co-location available to our users. As familiarity with composable infrastructure grows, the idea of using composable co-location instead of traditional servers co-located in racks is possible. By composable co-location, we mean the integration of emerging hardware needs without changing the existing core infrastructure. Researchers can now co-locate their own accelerators or compute nodes and leverage existing COMPaaS resources natively.

COMPaaS DLV User Community

Project Period: October 1, 2018 - September 30, 2021 
September 9, 2018 - The National Science Foundation (NSF) awarded a three-year grant to the UIC Computer Science (CS) Department&rsquo;s Electronic Visualization Laboratory (EVL) to acquire the &ldquo;Composable Platform as a Service Instrument for Deep Learning &amp; Visualization (COMPaaS DLV)&rdquo;. The total project cost is $1,424,806, with NSF providing $997,363 for hardware and the balance provided by UIC cost share to cover staff and graduate student support to design, operate and maintain the system. Maxine Brown, EVL Director, is Principal Investigator (PI) and co-PIs are CS faculty and EVL affiliated faculty members Andrew Johnson, Robert Kenyon and G. Elisabeta Marai. 
COMPaaS DLV will be acquired and made available for Deep Learning and Visualization research and training. Initial users will be 17 faculty from 5 departments in the UIC College of Engineering (COE) who provided compelling use cases for this system, and then expanded to other COE faculty as availability and demand permits. Users include the above-mentioned PI and co-PIs, plus COE Senior Personnel: Tanya Berger-Wolf (CS), Sybil Derrible (Civil &amp; Materials Eng.), Barbara Di Eugenio (CS), Chris Kanich (CS), Alex Leow (BioEng/Psychiatry, and by courtesy CS), Bing Liu (CS), Farzad Mashayek (Mechanical &amp; Industrial Eng.), Roberto Paoli (Mechanical &amp; Industrial Eng.), Luc Renambot (CS), Philip Yu (CS), Milos Zefran (Electrical &amp; Computer Eng.), Xinhua Zhang (CS), and Brian D. Ziebart (CS). 
COMPaaS DLV is critical infrastructure on which UIC faculty can pursue fundamental research in deep learning (data mining, data analytics, computer vision, natural language processing, artificial intelligence), visualization (simulation, rendering, visual analytics, video streaming, image processing), and a combination of deep learning and visualization (that is, when data is so large that it cannot be easily visualized, deep learning is used to extract features of interest to be visualized). UIC faculty are investigating, and contributing to, solutions for societal issues in such areas as anthropology, biology, cybersecurity, data literacy, fraud detection, healthcare, manufacturing, urban sustainability, and cyber-physical systems (e.g., autonomous vehicles). 
Though EVL specializes in visualization and virtual reality, it has a longstanding reputation as a technology enabler for the campus. EVL will design and operate COMPaaS DLV as a highly flexible, scalable, high-throughput system for Big Data. Its flexible 4th-generation computer architecture, &ldquo;composable infrastructure,&rdquo; treats computer components - computer processing units (CPUs), graphical processing units (GPUs), and storage - as pools of resources. Each application defines what resources it needs, and the operating system composes, or combines, resources on the fly and infrastructure is provisioned &lsquo;right-sized&rsquo; and resized as needed. The system is easily scalable - new infrastructure is automatically integrated with existing infrastructure and becomes part of the pool of capacity. The system&rsquo;s components are all interconnected with a high-speed internal network fabric, so that Big Data can quickly move between CPU, GPU and storage at optimum speed, i.e., high-throughput, with little to no bottlenecks. 
COMPaaS DLV will complement, but be connected via UIC&rsquo;s research network, to campus and regional clusters, data storage, supercomputing and cloud testbed facilities managed by the UIC ACCC&rsquo;s Advanced Cyberinfrastructure for Education and Research (ACER) group, the National Center for Supercomputing Applications at University of Illinois at Urbana-Champaign, Argonne National Laboratory&rsquo;s Leadership Computing Facility, and University of Chicago. 
Access to COMPaaS DLV will enable faculty, students and postdocs to execute deep learning and visualization codes faster, apply more sophisticated models to large-scale problems, gain greater insights, accelerate discovery, and open new avenues of research. The instrument will enable UIC to recruit new faculty who are attracted to institutions that can support their computational needs, and to recruit new students and produce a better trained workforce - which in turn attracts industrial partnerships. 
See <a href="https://today.uic.edu/nsf-grant-to-fund-advanced-deep-learning-and-visualization-computing-platform">UIC Today News Release</a> for more information. 
PHOTO CAPTION: One of the applications that will benefit from COMPaaS DLV is &ldquo;SENSEI Panama,&rdquo; a UIC collaborative project with evolutionary anthropologist Meg Crofoot (UC Davis) to enable anthropologists to use virtual reality to &lsquo;walk inside their data&rsquo; and study animal behavior. Specifically Crofoot and UIC computer scientists Tanya Berger-Wolf, Andy Johnson and Maxine Brown, are applying Deep Learning and Visualization to process data from LIDAR, aerial photography, terrain maps, GPS collars, and accelerometers, in order to study how coatis, kinkajous, spider monkeys, and capuchin monkeys on Barro Colorado Island (a Smithsonian Tropical Research Institute Research Center) in Panama, forage for food. This project received some funding from the NSF SENSEI $3.5M award to UIC CS Maxine Brown (principal investigator) and UIC CS co-PIs Andy Johnson, Tanya Berger-Wolf, and Robert Kenyon. (This project/photo won Honorable Mention in the <a href="https://www.evl.uic.edu/entry.php?id=2242">UIC 2016 Image of Research Competition</a>).

An application that will benefit from COMPaaS DLV is &ldquo;SENSEI Panama,&rdquo; which applies Deep Learning and Visualization to help anthropologists walk inside their data using virtual reality to study animal behavior.

COMPaaS DLV: Composable Platform as a Service Instrument for Deep Learning & Visualization

The SAGE Development Team at the Laboratory for Advanced Visualization &amp; Applications (LAVA) at the University of Hawaiʻi at Manoa and the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago recently received one-year supplemental funding from the National Science Foundation (NSF) to their SAGE2 award, in the amount of $527,000, to focus on SAGE2 cybersecurity. The diversity and distributed nature of the SAGE2 user base provides a growing set of security concerns. Identity and access management procedures, in particular, provide unique challenges, given the variety of institutions using SAGE2 to collaborate using display walls. 
In July 2018, the SAGE2 Team began working with Trusted CI, the NSF Cybersecurity Center of Excellence at Indiana University, to develop an Engagement Plan that outlines the scope of a 6-month timeline for the engagement, followed by 6 months of implementation. Trusted CI is helping outline Identity and Access Management (IAM) procedures appropriate for SAGE2&rsquo;s distributed user base. For example, SAGE2 needs to identify and provide access to local and remote sites involved in a collaboration as well as the users at those sites; such as: who is allowed to collaborate, are they allowed to share information or simply read it, and are they allowed to download information, etc. A secondary goal of the engagement, time permitting, is to determine if there are any unidentified high-priority cybersecurity needs revealed during the fact-finding stage that the SAGE2 Team should pursue; such as: securing the web server, securing documents that are shared, etc. 
<a href="http://blog.trustedci.org/search?updated-max=2018-08-02T11:59:00-04:00&max-results=7">Trusted CI Blog</a>

The SAGE2 team at UIC/EVL remotely collaborated with the University of Chicago’s Research Computing Center during its fourth annual research computing symposium and exposition, Mind Bytes.

SAGE2 Team Engaging with the Trusted CI Center on SAGE2 Cybersecurity

Ableton Live is music software designed for computer-based composition and performance. <a href="https://en.wikipedia.org/wiki/Ambisonics">Ambisonics</a> is a full-sphere surround sound technique used for playing audio on sound systems with more than two speakers, providing the illusion that sound is coming from all directions. Combining these has allowed music producers to compose music inside of Ableton, in 3D. Programmed in the visual programming language <a href="https://cycling74.com/products/max/">Max</a>, is an ambisonic decoder for the Sound Dome built by former EVL post-doc Anil Camci. 
The decoder outputs sound sources through specific speakers in the sound system based on their specified Azimuth, Elevation, and Distance parameters. To integrate Ableton Live with the decoder, all of the audio channels in Ableton were streamed into the decoder via an application called Jack which allows users to route audio from one application to another. A Max utility was also built to run inside of Ableton Live and allow the user to conveniently adjust the azimuth, elevation, and distance parameters of a particular channel. 
The azimuth elevation and distance values are sent over a network back to the Max decoder allowing producers to change the location of playback for every channel in Ableton without leaving the application. Custom audio effects are also being programmed to exploit all of the surrounding speakers and take full effect of system.

EVL Sound Dome developed by Anil Camci

Ableton Live Integration & Audio Effects for a 32 Speaker Ambisonic Sound System

Pi: Barbara Di Eugenio; Co-I: Debaleena Chattopadhyay 
A growing body of research has found that patients who actively manage their health have better outcomes, and patient engagement has become a central tenet in recent healthcare approaches. However, it has also been shown that many patients lack knowledge of their disease and medications, particularly as concerns chronic health conditions, whose exacerbation causes about two thirds of hospitalizations every year in the USA. Discharge instructions after hospitalization are a critical tool to engage patients in their self-care, but are often unsuccessful. Our hypothesis is that, to engage patients at discharge, a personalized hospitalization summary should be provided, that integrates physician and nurse documentation, and is informed by the patient's level of engagement in self-care and their socio-cultural perspective. 
Our transformative pilot work on physician and nurse documentation has identified an additional obstacle to the effective flow of information from professionals to patients: vocabulary differences between physicians and nurses. By leveraging the Unified Medical Language System Metathesaurus, we found no common synonyms between physician and nurse documentations for 75% of patients. Further, 86% of terms patients use are simple compared to only 15% for physicians. The lack of common language among patients, physicians and nurses motivates including all perspectives to make instructions meaningful for patients. Our hypothesis is that a better informed patient population who understands the connections between their illness, hospital experience, and discharge instructions will be more empowered and more engaged in their chronic disease management. Our innovation is to automatically create an easy-to-understand, personalized hospitalization summary that integrates events described in the EHR; and that is responsive to the patient's needs and preferences, and their activation framework (PAF) level. We pursue four specific aims: 1. To investigate socio-cultural patterns of patient perceptions of illness via a socio linguistic analysis of patient interviews; 2. To integrate multi-faceted information about a patient, as provided by physicians and nurses; 3a. To develop MyPHA, a novel application program, which will automatically generate the personalized summaries and deliver them to the patient on a tablet. MyPHA will be informed by aims 1 and 2, and will be grounded in Natural Language Generation technology; 3b. To evaluate MyPHA with multiple cycles of formative evaluation with all stakeholders, and a summative evaluation with patients. 
Whereas our pilot work has focused on heart failure, the techniques we will develop can easily be adapted to the management of other chronic health conditions, in particular for patients with cancer and cancer survivors. In addition, cancer and heart failure can co-occur, especially in older patients, and several cancer treatments can lead to the development of heart failure. The software program MyPHA will be released ooen source to the research communitv to enable raoid customization to other chronic conditions. 
Awarded Amount: $1,495,213

NIH: R01: SCH: INT: MyPHA: Automatically generating personalized accounts of inpatient hospitalization

The Continuum environment is a high-resolution computer-enhanced group collaboration workspace, consisting of large format 2D touch-enabled and 3D tiled displays, and instrumented with arrays of ambient sensors, enabling users to collaborate with local and remote colleagues and their data more effectively by giving the room itself the ability to anticipate their needs, improve their interaction, and focus their time on discovery rather than the technology. 
EVL&rsquo;s SAGE2 middleware enables teams of users to manage their data, documents, images and media on the Continuum display walls as if the room were one seamless canvas. SAGE2 lets users post and manipulate information, visualizations and animations across the Continuum displays. EVL REU Sean Walker developed a spatial audio system that ties audio playback from SAGE2 media content directly to the media&rsquo;s (i.e. video) location on the display wall. 
Additionally, any audio playback in the Continuum environment can be played back from any of the X number of speakers in the space. The spatial audio functionality was implemented using AVB (Audio Video Bridging) to connect all of the speakers positioned throughout the room coupled with SuperCollider (a functional programming language for audio) to pan the sound around the space.

Spatial Audio in the Continuum Environment

Caterpillar Inc. is working with the UIC Electronic Visualization Laboratory (EVL) to develop a SAGE2&trade; application to enable distributed teams to collaborate on project planning and scheduling. 
Planning and scheduling the design, development, engineering and manufacturing processes of construction equipment with worldwide teams and suppliers is an extraordinarily complex task. Caterpillar, inspired by &ldquo;StickySchedule,&rdquo; an interactive SAGE2 multi-user application for conference scheduling, asked whether EVL could help automate their project management workflow, which is currently a very manual process. 
The goal is to develop an innovative SAGE2 application to enable people from different Caterpillar units to schedule interdependent tasks over several months or years using electronic post-it notes on a large tiled display. The resulting schedule would be translated into a spreadsheet, which would then be input to project management software. SAGE2 would also enable groups to work collaboratively over distance, whereas now, everyone must be in the same room at the same time. Today, these management plans are updated only a couple times per year due to the high labor and cost of travel. This tool will enable these planning sessions to occur for significantly less labor and cost, resulting in a more regular update by the extended team. 
Additional information: 
<a href="http://sage2.sagecommons.org">SAGE2 Website</a> 
<a href="http://sage2.sagecommons.org/download/2117/">StickySchedule Paper</a> 
<a href="http://apps.sagecommons.org/">StickySchedule App</a> 
Project team: 
Lisa Creger, Lean Product Development Advisor, Innovation & Technology Development Division (ITDD), Caterpillar 
Peter E. Cooper, Product Development Manager, Medium Wheel Loaders, Caterpillar 
Luc Renambot, Computer Science Research Professor and EVL SAGE2 Project Manager, UIC 
Tommaso Massari, Computer Science MS student and EVL Research Assistant, UIC

Example of Caterpillar 'manual' project management workflow session.

SAGE2 Project Management Application

The Shirley Ryan AbilityLab of Chicago worked with the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago to use virtual reality to help people needing rehabilitation be more enthusiastic and energetic when doing their exercises. 
Using the VIVE head-mounted display, EVL researchers develop virtual reality worlds to augment the rehabilitation patients&rsquo; physical therapy spaces and sessions. The goal of this project is to increase patient interest and engagement, by supplementing physical therapy tasks and exercises with tasks incorporated into a number of different virtual worlds. Unity3D is used to build virtual environments, that are &lsquo;mapped&rsquo; to the physical therapy areas (e.g. a physical balance beam is represented by a very narrow bridge over a stream in a forest; the patient steps onto the actual beam, but are stepping onto a virtual bridge in the VR world). Data about the patient&rsquo;s movements and activities in these virtual worlds are visualized in real-time, allowing AbilityLab personnel to tailor the experience interactively. The enhanced rehabilitation experience intends to foster patient recovery and treatment success.

EVL director Maxine Brown explores a virtual world designed to enhance rehab patient treatment options.

Virtual Reality Enhanced Physical Therapy

In the past decades, science has become increasingly interdisciplinary and scientific discovery has experienced a paradigm shift from the classical scientific method to data-intensive scientific discovery. SAGE2 provides these interdisciplinary collaborators with a powerful, shared workspace. However, there remains a high cost of parallel work integration for data-intensive work. VisSnippets is a system which allows collaborators to perform expressive, impromptu data analysis and exploration within SAGE2. 
VisSnippets acts as a collaborative interface to exploratory programming, allowing users to write modular &ldquo;snippets&rdquo; of code for data retrieval, processing/aggregation, and visualization. Users may then compose modular analysis blocks from the shared snippet pool into branching data pipelines. The snippets may be edited or &ldquo;cloned&rdquo; by collaborators, and reused/applied multiple times across a collaborative session. 
To further facilitate collaborative work, the VisSnippets system provides further abstraction onto the expressive foundation of exploratory programming. VisSnippets reduces the overhead to scaffold a responsive web application and allows users to walk away from the collaborative analysis with a packaged web application containing all of the analysis.

VisSnippets: A Web-Based System for Impromptu Collaborative Data Exploration on Large Displays

This collaborative research project led by the colleagues at the University of Delaware aims to develop and advance tools for using image-data appearing in scientific publications, in addition to text, in order to support beneficial, targeted access to the biomedical literature. 
UIC's principal investigator, Computer Science associate professor, G. Elisabeta Marai will be responsible for the image analysis algorithms to distinguish between gel and plate images, the image analysis algorithms to distinguish between Electronic Microscopy vs Fluorescent Microscopy vs Light Microscopy and X-Rays, and the design of the visual interface to present results to end users. She will lead the design, development and implementation of the feature extraction and selection tools and techniques related to this research project.

Incorporating Image-based Features into Biomedical Document Classification

Cancer treatment courses which rely on imaging and spatially-dependent therapy involve making multiple treatment decisions (e.g., radiotherapy alone, radiotherapy plus chemotherapy, induction chemotherapy) over time. These decisions depend on complex factors, including the tumor location with respect to sensitive organs and its response to treatment, laboratory data, toxicity, anticipated side effects and survival probability. In this project, we design and develop novel statistical methodology for dynamic and personalized treatment decisions with specific application to head and neck cancer radiotherapy planning. We extend the state of the art by developing methods which take into account both imaging data features (such as radiation dosage and spatial distribution) and nonspatial data (such as demographics and toxicity) in the development of predictive models for anticipated outcomes and of the optimal treatment rules. We introduce novel similarity metrics over these hybrid, high-dimensional and sometimes incomplete patient data, and we extract informative, relevant and non-redundant features to identify similar patients. We further combine statistical models for outcome prediction with local models built over cohorts of patients. These ensemble models are used to further improve the reinforcement learning process to account for patient-specific preferences regarding side effects and survival outcomes. Specifically, we extend q-learning, a type of reinforcement learning, to account for competing, multiple outcomes and different preferences on the probability of short versus long-term outcomes. We validate our precision-driven model against known toxicity outcomes using a retrospective head-and-neck cancer dataset. 
Our specific aims are to: 
- Create quantitative methodology for extracting and selecting spatial relationships from medical images as well as non-imaging features, and generate a hybrid similarity metric which can identify similar cohorts of patients. 
- Create an ensemble model that combines global static models with dynamic local models over similar cohorts of patients with locally-sparse data. 
- Create dynamic reinforcement learning methodology which uses spatial and non-imaging features and metrics and accounts for patient preferences of multiple treatment outcomes. 
The project defines novel reinforcement learning techniques which account for multidimensional outcomes and patient preferences. Previous examples of reinforcement learning in the statistical literature have utilized examples with a small number of well-defined features, which is not applicable in the case of head and neck cancer. This project creates novel hybrid techniques for extracting informative and relevant features and similarity from imaging and nonimaging data. The project further introduces novel predictive modeling using a combination of global and dynamic local models over multiple outcomes. The proposed techniques leverage Big Data that is not only heterogeneous but also high-dimensional and locally sparse, in order to derive a novel precision-medicine approach. 
The empirically-derived treatment rules developed in this project have the potential to improve the standard of care (i.e., treatment plans chosen by the tumor treatment board) and the quality of life of surviving patients. The methods developed in the proposal may be used to derive optimal treatment strategies across not only a variety of spatially-dependent cancer diagnoses, but also other chronic conditions including mental health disorders, substance abuse diseases, or diabetes, that require making multiple decisions that must weigh the tradeoffs between efficacy and toxicity.

QuBBD: Precision E-Radiomics for Dynamic Big Head & Neck Cancer Data

Deep learning is often referred to as a &lsquo;black box&rsquo;, because, aside from selecting model and the model hyperparameters (like the learning rate, activation function parameters), users don&rsquo;t have access to what happens inside the model. We can know that the model works by measuring its accuracy on some testing data set, suggesting that the model must have &lsquo;learned&rsquo; something meaningful from the training data. But how can we peek inside and &lsquo;see&rsquo; what the model &lsquo;sees&rsquo;? 
Biologists have recently begun to explore how deep learning approaches could be used to address questions involving diverse biological datasets. For this community, producing an accurate model is important, but also insufficient, since the larger goal is to contribute to our understanding of complex biological systems and phenomena. Visualization could help biology researchers &lsquo;peak&rsquo; inside the black box and relate deep learning mechanics to existing biological knowledge. However, most visualization techniques for deep learning have been developed for images and text, so many of the encodings and layouts are designed for input data with fundamentally different perceptual properties than biological data sets, such as gene sequence data or gene expression data. 
This research explores current approaches in visualization for deep learning and how these approaches may (or may not) be applied to biological data. This work is conducted with researchers at Argonne National Labs studying deep learning for cancer research and genomics.

Argonne Leadership Computing Facility

Vis for Deep Learning for Biology

Exploring big cities like Chicago can be a daunting experience when the tourist has to gather information from multiple sources regarding places, directions, transportation, safety, and weather.  We propose a large display solution (kiosks) which aims to encourage people to explore the city by showing cultural attractions, special events and other points of interest (POI) on a map along with multi-modal transportation options to take them to their destination. POI&rsquo;s and events are updated frequently based on the time and safety information. Kiosks are intended to be located throughout the city especially locations with the highest density of points of interest.

Screen shot of the &ldquo;Exploring Chicago&rdquo; application prototype.

Exploring Chicago

The Englewood Data Hub (EDH) is a project that brings together both crowdsourced and public data surrounding the Englewood area into a single place. 
The <a href="https://englewooddatahub.org/">EDH Resource Directory</a> makes use of the crowdsourced service and school data to display an interactive map with their locations. Its user interface (UI) allows for an exploratory experience that offers filtering of services by names and type and displays detailed information about the services available in Englewood. 
The <a href="https://englewooddatahub.org/analytics">EDH Analytics Tool</a> combines the crowdsourced service and school data with public data sets to display visualizations; specifically, the public datasets used are 2010 Census data along with Vacant Lot and Crime data from the Chicago Data Portal. 
The Englewood Data Hub has been developed by a group of researchers at the Electronic Visualization Laboratory, University of Illinois at Chicago under the direction of Prof. G. Elisabeta Marai. 
Team members include EVL research experience undergraduate students (REUs): Andrew Burks, Joshua Castor, and Isabel Lindmae. 
This project is funded by the The Joseph and Bessie Feinberg Foundation and the University of Illinois at Chicago.

Englewood Social Service Finder and Englewood Analytics

Head and neck squamous carcinomas affect all ethnicities and age groups, accounting for significant mortality and therapy-related
side effects. Over 50,000 new cases are diagnosed each year in the United States, leading to large, rich repositories of patient data. For each of these cases, oncologists need to anticipate survival, oncologic, and side effect outcomes associated with treatment strategies in order to select a treatment which balances efficacy and toxicity. However, despite the wealth of data available, risk prediction algorithms for cancers are rudimentary and incorporate only a handful of patient characteristics, largely due to a lack of computational methodologies. 
This project will develop a computational methodology that supports the construction of a validated precision model of patient-specific outcomes for head and neck cancers, based on demographics, toxicity, and complex imaging data. The methodology will be the first in the field to include both complex imaging and non-imaging data, while taking into account large-scale biological and clinical correlates. A web-based environment will further allow practitioners to interactively explore and communicate the model mechanics and outcomes to stakeholders. The approach is innovative through its leverage of big imaging data repositories and through its unique blend of computational modeling principles from bioengineering, statistics, and computer science (medical imaging and visualization). 
From a clinical perspective, this integrative approach is novel in the field of cancer therapy. The resulting model and web-based
environment will mark a significant advance in biomedical computing because it will be able to identify, for the first time, specific subgroups who are at risk for distinct oncologic, toxicity, and survival profiles. Furthermore, the currently clinical practice paradigm for head and neck treatment is stage-driven. With validation, this novel approach can identify patients based not on clinical staging, but on precision modeling using cohort data and similar sets of patients. The methodology is applicable to clinical practice involving human subjects and has the potential to change the standard of care and outcomes of treatment in the field.

SMART-ACT: Spatial Methodologic Approaches for Risk Assessment and Therapeutic Adaptation in Cancer Treatment

The <a href="http://hci.cs.uic.edu/">Human-Computer Interaction (HCI) Laboratory</a> at the University of Illinois at Chicago (UIC) is led by assistant professor Debaleena Chattopadhyay of the Department of Computer Science. The HCI Lab designs computer technologies to transform human experiences and deliver positively impactful societal change. 
UIC HCI is an affiliate laboratory of the Electronic Visualization Laboratory (EVL) providing a human centered approach to the development of tools and technologies related to advanced visualization, virtual reality, visual analytics and networked collaboration. 
The HCI Lab&rsquo;s mission is to support individual and social activities and improve everyday life, designing, building and evaluating ubiquitous computing technologies. The design of these systems draws from research on how people work, play, and communicate in day-to-day life. 
The UIC HCI Lab is located on the UIC campus at: 
Science & Engineering Laboratory West Room 1032 
900 West Taylor Street 
Chicago, IL 60607

UIC Human-Computer Interaction (HCI) Laboratory

Just as NSF cyberinfrastructure produces greater volumes and varieties of data from data storage systems, online instrumentation and major computational resources like XSEDE and Blue Waters, advanced visualization instruments serve as the &ldquo;lenses&rdquo; of these data generators (similar to the telescopes and microscopes of yore), enabling researchers to peer into cyberspace to manage, access and analyze big data. 
Continuum is a cyberinfrastructure instrument that unifies high-resolution computer-enhanced group collaboration workspaces with arrays of ambient sensors, enabling users to collaborate with local and remote colleagues and their data more effectively by giving the room itself the ability to anticipate their needs, improve their interaction, and focus their time on discovery rather than the technology. Continuum is a digital assistant, benefiting global scientific collaboratories and providing a foundation for new computer science research. War rooms benefit collaboration because they support continuous interaction and awareness of team members, enable persistent information sharing, and encourage mobility among team members and information. A computer-enhanced war room, such as Continuum, which (1) connects distributed teams with their data, whether stored in the cloud or streamed from major computational resources over high-speed networks, (2) supports persistent digital artifacts, (3) enables ultra-high-resolution 2D and 3D stereoscopic digital media to be simultaneously viewed, and (4) supports ubiquitous and intuitive interaction, creates a powerful and easy-to-use information-rich environment in support of scientific discovery. 
Continuum is the culmination of the University of Illinois at Chicago’s (UIC) 20+ years experience developing interactive instruments, from the original CAVE in 1992, to the ultra-high-resolution LambdaVision tiled LCD wall in 2004, and the hybrid-reality CAVE2 in 2012. Each new generation of visualization instrumentation provided scientific communities with more advanced features Continuum development focuses on three areas: (1) transforming passive display walls that respond to user commands into attentive display walls that monitor the users in a room and anticipate their needs to improve their experience; (2) providing an alternative approach to constructing high-resolution visualization display walls using thin-border 4K passive stereo OLED displays that have impressive brightness, contrast, uniformity, off-axis viewing, and low reflectivity in brightly lit rooms; and (3) utilizing cloud computing to prototype future Continuum spaces that don’t require local high-end compute and display resources. Twelve funded research projects from local institutions, in Astronomy, Astrophysics, Biology,
Combustion, Geophysics, Neuroscience, Pathology, Physics, Psychiatry, and High Performance Computing, are poised to use Continuum for their data visualization needs. Continuum will open up new opportunities in computer science research at the intersection of large-scale data visualization, human-computer interaction, and high-speed networking. Continuum will also directly support 10 classes taught in the UIC Computer Science, Art and Design, and Biomedical Science departments.

Artist's rendering of Continuum space.

MRI - Development of Continuum: A Virtualized Attentive Environment for Amplified Collaboration

The visualization of aleatoric uncertainty in ensemble simulations - i.e. the study of high concentrations in mixtures - provides many challenges due to the size and dimensionality of the data sets. EVL researchers have developed a web-based application designed to locate and track areas of high concentration in salt-solution simulations. 
The goal is to learn more about the behavior of Viscous Fingers, or contiguous areas of high concentration of salt within a cylinder of pure water. The application allows viewers to explore 3-D ensemble simulation data using both 2-D vertical slices of the cylindrical data to get a more detailed view of the data in a specific time step, as well as a feature tracking merge tree to better understand the evolution of these Viscous Fingers over the course of a simulation run.

Screen Shot of Viscous Fingers Visualization Application

Interactive Exploration and Tracking of Viscous Fingers in Large-Scale Ensemble Simulations

The Chicago 0,0 Riverwal AR experience provides a novel, interactive way for users to explore historical photographs sourced from museum archives. As users walk along the Chicago River, they are instructed to use their smartphone or tablet to view these photographs alongside the current views of the city. By matching location and view orientation the Riverwalk application creates an illusion of &ldquo;then and now&rdquo; co-extant. This superimposition of the historical photographer&rsquo;s view and the user view is the basis of educational engagement for the user and a key factor in curating the images and the narrative surrounding them, facilitating a meaningful museum experience in a public, outdoor context. 
As such, creating the AR experience involves a complex back-and-forth between 3D and 2D experiences of locations: the historical
images are 2D, taken from specific locations through specific optics and views; the user, present at the real-world 3D location is
orienting their own camera and 2D phone video screen in space in order to achieve the AR experience and illusion; and the ultimate experience of superimposition is one of seeing two integrated views simultaneously: the stream of data taken from a smartphone&rsquo; camera and the historic image. The site-specific nature of the publication makes it necessary to utilize a virtual 3D environment in which to place 2D augmented content that enhances the experience. This content needs to be placed in such a way so that the designer can accurately visualize what the user will see on screen from a particular location. 
The first episode of the Riverwalk AR experience focuses on a single block between N. LaSalle and Clark Streets; the site of
the Eastland Disaster in 1915. The site was selected because of the importance of this historical event - the sinking of the Eastland
cruise ship 100 years ago was the largest single loss of life in Chicago’s history - and because of the abundant media available
in the archive including extensive photographic documentation, newspapers and film reels. Moreover, the site potentially offers
natural way, finding characteristics that are amenable to developing our project, including a newly built pedestrian walkway along
the river with viewing platforms, views from pedestrian walks along the bridges, and historical markers. Additionally, the urban environment offers many features providing sufficient quality for robustly tracking user location based on their cameras from a variety of views and positions. This is a typical scenario in the creation of public outdoor AR experiences; the quality of content and narrative are the primary motivations for their creation.

Chicago 0,0 is a mobile application aimed at presenting 2D media content from archives of historical media through the use of augmented reality.

Riverwalk: Incorporating Historical Photographs in Public Outdoor Augmented Reality Experiences

Patients come to Centers for Physical Therapy and Rehabilitation from all over the globe, &ldquo;often with catastrophic or unusual conditions and always with great hope for an improved life ahead&rdquo;. In the quest to assist these patients, Virtual Rehabilitation is becoming popular in clinical research thanks to the development and availability of new immersive technologies. For example, the rehabilitation of post-stroke patients frequently uses virtual reality, and sometimes employs a first-person Serious Games approach to make therapy more enjoyable for patients. In such approaches, the patient experiences the virtual environment in a first-person view. 
Unlike the existing body of work, we present and evaluate an immersive virtual rehabilitation application which uses a third person perspective: the patient controls with her body a 3D avatar. The application was commissioned by biomechanics researchers at a major center for rehabilitation research, and could be used to compare the learning-curve and time needed for rehabilitation across different therapies and environments. One of the main challenges in this work is the design of simple, non-distracting avatars and virtual world, which allows the patient to focus on performing correctly the rehabilitation exercises. An additional challenge comes from the trade-offs typically associated with virtual reality, for example, loss of depth perception. 
Our solution, RehabJim, is a third-person serious game for the rehabilitation of upper limbs in post-stroke patients, with particular emphasis on reaching movements. Our solution uses a state-of-the-art virtual reality environment, Unity 3D, and a Kinect controller in the CAVE2&trade; Hybrid Reality Environment.

External view of CAVE2 environment with a running instance of our application. The screens cover 320 degrees of visibility and a monitor at the entrance is used for managing the hardware resources.

RehabJim: A Third Person Approach To Virtual Reality Biomechanical Rehabilitation

Google Street View can be considered as an online browsable dataset consisting of billions of street-level panoramic images acquired all around the world. Since 2007, the company has continuously improved and updated its global image database, now collecting on average one panorama each 5-10m in urban environments. Google provides public APIs for requesting virtual camera views of a given panorama from its GPS position or unique identifier. These views are rectilinear projection of the spherical panorama with a user selected field of view, orientation and elevation angle. These views can be considered as undistorted images taken from pinhole cameras, free of distortion, and need to be partially overlapped in order to reconstruct the 360 degree panorama. 
Additionally, the speed with which Google&rsquo;s laser scanners get reflected by surfaces allows the company to calculate the distance from the camera to objects or buildings, allowing the creation of 3D models to be used in other applications such as Google Earth. Google API indeed provides a way to obtain the depth map corresponding to a specific panorama, information that we will exploit in order to reconstruct the 3D scene. Once decoded and visualized in space, this depth data and the models generated from it may turn out to be useful for many different applications like navigation and augmented reality, but also virtual reality oriented videogames.

Sample point cloud reconstruction based on the output of merging 6 different panoramic images along a street.

3D City Reconstruction From Google Street View

SENSEI: Sensor Environment Imaging (SENSEI) Instrument is a National Science Foundation award to construct a reconfigurable, ultra-high-resolution, spherical (4π steradian), photometric, radiometric and photogrammetric real-time data-acquisition, sensor-based, camera system capable of capturing 3D stereo and still images for viewing in collaboration-enabled, nationally networked,
virtual-reality systems. 
As SENSEI researchers develop the SENSEI camera system, collaborators are exploring and refining techniques for displaying data captured via a variety of sensors, as well as investigating human computer interaction modalities for stereoscopic, virtual reality applications and environments incorporating complex layers of sensor data. 
SENSEI-Panama (SENSor Environment Imaging-Panama) is a environmental visualization approach developed at the Electronic Visualization Laboratory (EVL) to investigate the movement patterns on newly isolated species in the anthropological sciences. Researchers who have left the field are unable to visually determine what animals see in their environment; however, they produce large amounts of data through the use of GPS collars. accelerometers, LIDAR, aerial photography, and terrain maps. SENSEI-Panama considers how to visualize this kind of data in an environment like CAVE2&trade; and combine it with the animal movement data collected. The data is transformed to create a virtual map to enable researchers at the University of California Davis to easily track the movements of animals through their habitat on the island, and study their behavior.

EVL researchers Jillian Aurisano (lead) and James Hwang explore SENSEI-Panama in EVL's CAVE2&trade;

SENSEI-Panama

EVL PhD candidate Arthur Nishimoto has developed a virtual reality representation of the proposed new College of Engineering (COE) building for display in the CAVE2. Participants can navigate through and explore the virtual COE building and surrounding UIC campus. This rendition consists of models converted to Unity3D from the AutoDesk Revit model provided by the architecture firm Lothan Van Hook DeStefano Architecture LLC (LVDA). Additional Chicago city buildings were added to show how the downtown Chicago and other areas around UIC would look from the perspective of the new building. 
Using EVL&rsquo;s CAVE2 Unity simulator and Mechdyne&rsquo;s getReal3D plugin, the Unity scene was exported for use in the CAVE2 Hybrid-Reality Environment. Within the virtual COE building users can explore rooms within the proposed building including collaborative classrooms, the winter garden, and laboratories. Other 2D elements and renderings provided by LVDA are integrated into the virtual walk-thru. 
Model Credits: 
Chicago city model by Andrew Lock 
<a href="http://arthurnishimoto.blogspot.com/2015/10/uic-science-and-engineering-offices.html">UIC Science and Engineering Offices by Arthur Nishimoto</a>

EVL PhD candidate Arthur Nishimoto navigates through the proposed virtual COE building in the CAVE2.

Proposed UIC College of Engineering Building Rendition in the CAVE2&trade; Hybrid Reality Environment

SAGEBoard is an electronic whiteboard for tiled touch displays that can be used locally or shared between remote collaborators. An electronic whiteboard can combine the advantages of an electronic device with the fast paced interaction provided by a regular whiteboard, enhancing both solutions with new functions. Users can perform classic whiteboard actions such as drawing or writing free-handed on the screen using their finger or a stylus. They can use brushes to paint, and they can erase content. However, unlike on traditional whiteboards, users can also relocate content on the screen, insert figures and drawings, and save or load previous sessions. Moreover, everything can be shared between all the users, and more people can interact simultaneously, improving the collaborative task. 
SAGEBoard runs on multiple displays, and is integrated with other display-enabled applications; the application manages multiple simultaneous inputs, as well as the rendering of graphic elements on different sized displays. SAGEBoard is a lightweight networked solution that address any latency issues that might impact the interactive experience. 
This research explores human computer interaction issues - how people behave when collaborating on a shared touch enabled display, or deal with remotely controlled collaboration, since the whole application can be shared among both co-located and remote screens. A small scale user study was designed to analyze the advantages of using multi-touch displays over conventional keyboard and mouse configurations, as well as remote interaction using mobile touch displays.

User drawing on EVL&rsquo;s Cyber-Commons tiled display using SAGEBoard

SAGEBoard: a Whiteboard for Large Multitouch Displays

SENSEI will be a reconfigurable, ultra-high-resolution, spherical (4π steradian), photometric, radiometric and photogrammetric real-time data-acquisition, sensor-based, camera system capable of capturing 3D stereo and still images for viewing in collaboration-enabled, nationally networked, virtual-reality systems. 
The ultimate goal of SENSEI will be to capture ultra-high-resolution still and motion 3D stereoscopic video and images of real-world
scenes, that can be viewed in 3D high-performance virtual-reality systems, and can be used to extract objects&rsquo; sizes, shapes and distances within a scene. 
SENSEI, given its ~terapixel/minute flood of imagery, presents big data challenges as well. Specifically, SENSEI will be an integrated, distributed cyber-collaboration instrument with big attributes: 
- &ldquo;Big Resolution,&rdquo; configurable, portable, sensor-based camera systems: Sensor-based, dynamic 3D stereoscopic motion/video and/or multi-spectral data camera systems will be developed. They will be flexibly configurable to capture geometries such as: outside-in, inside-out, light field arrays, photogrammetry, macroscopy and microscopy. SENSEI will have an optional weatherproof transport container for field use and underwater cases developed by partners. SENSEI will create images that are ~9 times IMAX resolution. Open source software and wikis will encourage adoption and modification by other domain scientists, as with our OptIPortals and CAVEs from prior MRIs. 
- &ldquo;Big data&rdquo; computational and storage systems: Graphics processing unit (GPU) computing with RDMA, fast storage systems, and software development are needed to perform the necessary image processing calibration, stitching, synchronization, partitioning, and video serving/streaming. 
- &ldquo;Big displays&rdquo; for 3D stereoscopic viewing: Virtual-reality displays, such as the CAVE2, WAVE,
and similar technologies, are ideal for end-user viewing and multi-user, multi-site collaboration. The next generation of 4K 3D OLED TVs and LCD TVs, singly or in small arrays, will also be supported. 
- &ldquo;Big networks&rdquo; for an end-to-end, tightly coupled, distributed system: The distributed system -
cameras, computational and storage systems, and display technologies - will use 10/40/100Gbps optical networks at/between partner institutions to share and correlate data in useful time frames. 
SENSEI can advance computer science and electrical engineering disciplines in the development of image capture systems, display technologies, image processing, visualization, and image creation, and advance scientific domain disciplines such as archaeology, architecture, astronomy, biology, cultural heritage, digital arts, earth science, emergency preparedness, engineering, geographic information systems, manufacturing, oceanography, planetary science, and science education. 
SENSEI partner institutions are: 
- University of Illinois at Chicago (UIC), Electronic Visualization Laboratory (EVL) 
- Jackson State University (JSU), College of Science, Engineering and Technology 
- Louisiana State University (LSU), Computer Science Department (Beginning in Year 2)
- Scripps Institution of Oceanography (SIO), Jaffe Laboratory for Underwater Imaging 
- University of California, San Diego (UCSD), Calit2-Qualcomm Institute (Calit2-QI) 
- University of Hawaii at Manoa (UHM), Lab for Advanced Visualization &amp; Applications (LAVA) 
UIC, UCSD, LSU and SIO are the primary technology developers of SENSEI. JSU and UHM, as well as the developer sites, have application teams who will be early adopters of SENSEI and provide feedback on the workflow, operation, and use.

Close-up rendition of SENSEI camera prototype

SENSEI: Sensor Environment Imaging (SENSEI) Instrument

Our interdisciplinary team of computer science and health care researchers will develop augmented communication tools (ACTs) within a prototype Electronic Health Record (EHR) visual interface and test its usability. The focus of this project is Nurse-Nurse (RN-RN) communication of six high risk clinical events (CEs): fever, pain, changes in respiratory status, changes in level of consciousness, changes in output, and bleeding. Using combined EHR and hand-off report data, we will develop a predictive computational model for detecting these clinical events within the EHR, as well as their predicted severity. The ACTs system will then use light-weight automated inference to detect these clinical events, alert the nurse, and quickly provide highly relevant contextual data to improve quality care, patient safety, and reduce patient deaths.

Enhancing Nurse Effectiveness via Augmented Communication Tools

The REACH system extracts (i.e., reads and assembles) causal mechanisms from research abstracts and papers. The fundamental research proposed has three guiding principles. The mechanisms extracted must: (a) be runnable, i.e., they must predict the data; (b) model current uncertainty of causal relations being active given context information; and (c) be holistic, that is, they explain multiple layers of abstraction, ranging from cellular to organism phenotypes. Thus, running our model produces quantifiable probabilities for the modeled variables (e.g., presence of pancreatic cancer) given customizable input values (e.g., overexpression of the VEGF protein). We assume that the model is deterministic given a specified context, but, in practice, variance in results exists due to difference in context - either reported or completely unknown - in the acquired mechanisms. The REACH system will provide a platform for communicating explanation with novel visualization tools for the extracted mechanisms. We will evaluate our approach in the domain of cancer biology, by extracting signaling pathways, but our work is largely domain independent.

Reading and Assembling Contextual and Holistic Big Mechanisms

Developed by EVL PhD Candidate Arthur Nishimoto the CAVE2/Unity simulator is a Unity3D package for testing and development of Unity3D applications for the CAVE2&trade; Hybrid-Reality Environment. Using only standard Unity features (no platform-dependent libraries or Unity Professional license required), this package acts as a wrapper around the CAVE2 API to simulate the CAVE2 viewport and 3D spatial wand interactions on a desktop/laptop development environment. 
The associated <a href="https://github.com/arthurnishimoto/module-omicron">module-omicron package</a> also serves as the Unity/C# interface of accessing the omicron input abstraction library.

CAVE2/Unity Simulator being presented by EVL PhD candidate Arthur Nishimoto.

CAVE2/Unity Simulator

The International Space Station (ISS) and Space Shuttle Discovery is a CAVE2&trade; Hybrid Reality Environment simulation where participants can explore the Space Station in full scale, orbiting the Earth at the current location and altitude of the ISS. Users experience the current field of view from ISS and the territory over which it is orbiting. The ISS orbits the Earth roughly once every 90 minutes at an altitude of 420 km. Participants can fly around the space station and the shuttle to get a better sense of their size in relationship to the shuttle. The model of the ISS and position data come from the National Aeronautics and Space Administration (NASA). The model of the shuttle comes from the google sketchup 3D Warehouse. The earth is rendered using <a href='http://osgearth.org'>osgearth</a>. EVL&rsquo;s OmegaLib software is used to display the 3D models in the CAVE2 System.

ISS 3D Model as displayed in EVL&rsquo;s CAVE2 System

International Space Station (ISS) and Space Shuttle Discovery

SAGE&trade; software enables teams of users to manage their data in a room covered with displays, as if the room were one seamless canvas. SAGE lets users post and manipulate information, visualizations and animations. The goal of SAGE is to make it possible for people to look at large amounts of information and come to conclusions with greater speed, accuracy, comprehensiveness and confidence. 

SAGE was originally conceived in 2004. With National Science Foundation (NSF) funding in 2009, the SAGE team was able to continue to develop and harden SAGE, thereby helping over 100 major institutions worldwide better utilize content on their display walls with SAGE. 

With NSF funding in 2013, the SAGE team had an opportunity to develop SAGE2, the next-generation SAGE. SAGE is being rewritten from the ground up, using new and emerging technologies. 

SAGE2 is browser-based, and everything is done from the Chrome web browser. Special software is no longer needed. By simply dragging and dropping content from one&rsquo;s laptop to a visual representation of the wall in the browser, one can transfer the content to the wall. 

The user interface has been completely redesigned to make it easier and faster for users to work on really large display walls. The SAGE2 interface emphasizes transportable interfaces rather than stationary interfaces so a user can control SAGE2 from anywhere in a room (or even away from the room). Entire laptop screens can also be pushed onto windows in SAGE2, again directly leveraging browser technology. SAGE2 also enables users to connect to collaborating SAGE2 walls and send documents among them by simply dragging and dropping the content in specially marked &ldquo;collaboration folders&rdquo;. 

Custom applications are easily built in SAGE2. While SAGE is written in C++, SAGE2 is written entirely in Javascript. SAGE2 includes a small software toolkit that enables users to develop applications or modify existing applications to work seamlessly across multiple displays in a synchronized fashion. This unleashes the possibility of creating a new generation of applications that can take advantage of the special qualities of ultra-high-resolution display walls. 

SAGE is being developed by the Laboratory for Advanced Visualization &amp; Applications at University of Hawai&#8216;i at M&#257;noa and the Electronic Visualization Laboratory at University of Illinois at Chicago. 

SAGE / SAGE2 receives major support from the National Science Foundation award # ACI-1441963. 

SAGE and SAGE2 are trademarks of the University of Illinois Board of Trustees.

SAGE2&trade; Scalable Amplified Group Environment

A connectome is a map of neural connections in the brain. In this CAVE2&trade; Hybrid Reality Environment application, participants walk inside streamlines depicting the white matter fiber tracts in the brain, derived from diffusion tensor images (DTI) obtained in an MRI scan. The fiber tracts are color-coded by their primary direction (green: front-back, red: left-right, blue: up-down). This research is being done by the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago in collaboration with Olusola Ajilore, UIC Department of Psychiatry. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC) and the UIC Department of Psychiatry.

Connectome application displayed in EVL&rsquo;s CAVE2

Connectome

Mechanistic models that describe the dynamical behaviors of biochemical systems are common in computational systems biology, especially in the realm of cellular signaling. The development of families of such network models, either by a single research group or from multiple sources, presents signiﬁcant challenges that range from identifying functional structural diﬀerences to keeping track of the model history. We present the development and features of an interactive model exploration system, MOSBIE, which tracks the features and development history of a family of models and provides utilities for identifying similarities and diﬀerences between models. Models within a collection can be clustered using a custom similarity metric. We also provide a visual interface for exploring collections of models, allowing a researcher to interactively compare the similarities and diﬀerences between pairs of models, search for models which contain a given structural motif, and explore the parameters and simulations of models. We evaluate the usefulness of this system with a case study in the cell signaling domain. We present the feedback provided by domain experts, and discuss the beneﬁts of this approach.

MOSBIE: A Tool for Comparison and Analysis of Rule-Based Biochemical Models

Funded by NASA, the SIMPLE project (Sub-ice Investigation of Marine and Planetary-analog Ecosystems) is to deploy an autonomous underwater vehicle (AUV) whose sensors collect biogeochemical data of the water&rsquo;s composition (salinity, pressure, conductivity, and oxygen concentrations) and photograph the underside the ice surface, and use sonar data to map the underside of the ice surface. The goal is to compare measurements of the ice shelf taken from under the ice with measurements taken from above the ice, to improve future mapping missions to Jupiter’s ice-covered moon Europa, and eventually send an AUV under that ice to search for life. SIMPLE is a collaboration of UIC, Stone Aerospace, Georgia Tech, Louisiana State University, University of Nebraska-Lincoln, San Jose State, University of Texas at Austin, and Montana State University. Prior to SIMPLE, NASA funded the project <a href="https://www.evl.uic.edu/entry.php?id=2027">ENDURANCE (Environmentally Non-Disturbing Under-ice Robotic ANtarctiC Explorer).</a>

Arthur Nishimoto in EVL&rsquo;s CAVE2 demonstrating the SIMPLE application.

SIMPLE project (Sub-ice Investigation of Marine and Planetary-analog Ecosystems)

We present the design and implementation of a visual mining and analysis tool to help identify protein mutations across family structural models, and to help discover the effect of these mutations on protein function. We follow a client-server approach in which distributed data sources for 3D structure and non-spatial sequence information are seamlessly integrated into a common visual interface. Multiple linked views and a computational backbone allow comparison at the molecular and atomic levels, while a trend-image visual abstraction allows for the sorting and mining of large collections of sequences and of their residues. We evaluate our tool on the triosephosphate isomerase (TIM) family structural models and sequence data, and show that our tool provides an effective, scalable way to navigate a family of proteins, as well as a means to inspect the structure and sequence of individual proteins.

FixingTIM: Interactive Exploration of Sequence and Structural Data to Identify Functional Mutations in Protein Families

This CAVE2&trade; Hybrid Reality Environment simulation depicts the burning of three aluminum nanoparticles at very high temperatures (approximately 2000 C) in order to study their potential use in fuel cells. Visualization by Michael Papka of the Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory and Khairi Reda of the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. Simulation and data provided by Kenichi Nomura and Priya Vashishta of the University of Southern California. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2. 

Note: This dataset cannot be visualized in real time, but EVL has a pre-rendered movie of it. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC), Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory, and the University of Southern California.

Burning Aluminum Nanoparticles for use in Fuel Cells

Professor Peter Doran of the Earth and Environmental Sciences Department at University of Illinois at Chicago (UIC) employs virtual-reality to further biological research in terrestrial environments analogous to those found on other planets, and to enable remote searches for, and identification of, life in extreme environments. Doran and EVL are part of the NASA-funded ENDURANCE project (Environmentally Non-Disturbing Under-ice Robotic ANtarctiC Explorer), a collaboration of UIC, Stone Aerospace, NASA Ames Research Center and Montana State University. Project members are analyzing 3D biogeochemical information captured by an autonomous underwater vehicle (AUV) in the extreme environment of West Lake Bonney, a perennially ice-covered Antarctic dry valley lake in the McMurdo Dry Valleys of Antarctica. They hope to apply what they learn to planetary science, such as capturing and analyzing information about the water ice surface of Jupiter&rsquo;s moon, Europa. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2&trade; System. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC), UIC Earth and Environmental Sciences Department, Stone Aerospace, NASA Ames Research Center and Montana State University.

Users exploring data collected from the botton of West Lake Bonney

ENDURANCE (Environmentally Non-Disturbing Under-ice Robotic ANtarctiC Explorer)

In this application, participants view a number of 2D high-resolution images captured by the Mars Curiosity Rover, along side a 3-dimensional model of Mars in the CAVE2&trade;. EVL&rsquo;s SAGE&trade; (Scalable Adaptive Graphics Environment) software is used to display the images in the CAVE2. 

As the CAVE2 System is a hybrid environment, it enables users to display both 2D and stereo images, as well as 3D models using SAGE software. This functionality is in particularly useful for information analysis and insight. 

The data used in this application is from the NASA Jet Propulsion Laboratory and California Institute of Technology.

Mars Curiosity imagery displayed in EVL&rsquo;s CAVE2

Mars Curiosity

Stroke is the fifth most common cause of mortality in the United States as reported by the Center for Disease Control and Prevention.  Medical treatment and physical therapy are imperative for the management of stroke, the reduction in risks of recurrent strokes and recovery of lost motor controls.  The success of a medical treatment is determined largely by adherence to therapy.  However, research has shown that adherence to therapy in patients who have had a stroke is often suboptimal.  This project focuses on the end goal of being able to test adherence, aims at creating and setting up an immersive 3D environment with an interactive game through the use of modern motion detection sensors.

MARS3 - Machines Assisting Recovery from Stroke

In this application, participants view several 2-dimensional high-resolution images captured by the Mars Curiosity Rover in EVL&rsquo;s CAVE2&trade; Hybrid Reality Environment.  Using EVL&rsquo;s SAGE&trade; (Scalable Adaptive Graphics Environment) middleware a broad range of digital media, including 2D images, files and documents, as well stereo imagery, 3D models, can be displayed for contrast, comparison and discussion. The 2D data displayed in this application is from the NASA Jet Propulsion Laboratory and California Institute of Technology.

Mars Curiosity imagery displayed in EVL&rsquo;s CAVE2 using SAGE

Mars Curiosity (SAGE in CAVE2&trade;)

Participants can explore the self-assembly of a ligated gold nanoparticle and proteins inside an ionic solution in the CAVE2&trade; Hybrid Reality Environment. Data and visualization provided by the Petr Kral Research Group at the Department of Chemistry at the University of Illinois at Chicago (UIC) and the Vincent M. Rotello Research Group at University of Massachusetts, Amherst. OmegaLib software by the UIC Electronic Visualization Laboratory (EVL) is used to display the 3D model in the CAVE2. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC), UIC Department of Chemistry&rsquo;s Petr Kral Research Group, and University of Massachusetts, Amherst&rsquo;s Vincent M. Rotello Research Group.

Gold Nanoparticle displayed in EVL&rsquo;s CAVE2 system

Gold Nanoparticle

Participants stand inside virtual train tracks in the CAVE2&trade; System to observe distortion as high-speed trains roar past. This visualization of fundamental dynamics phenomena related to high-speed rail operations was done in collaboration with the National University Rail (NURail) Center - which consists of researchers at University of Illinois Urbana-Champaign, University of Illinois at Chicago (UIC), Massachusetts Institute of Technology, Michigan Tech University, University of Kentucky, University of Tennessee, Knoxville, and the Rose-Hulman Institute of Technology - and is funded by the U.S. Department of Transportation. Understanding track structure movements, liquid sloshing, and wheel / rail contact locations are among several transportation research areas that require sophisticated virtual-reality tools such as the CAVE2 System. Visualization by the UIC Electronic Visualization Laboratory (EVL). EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago and the National University Rail (NURail) Center.

Virtual train tracks as seen in EVL&rsquo;s CAVE2

High-Speed Trains

To show medical illustrators and educators the value of virtual reality for medical training, the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago mocked up a 3D skeleton, complete with body organs, including the heart, lungs, and intestines, that students could manipulate on the walls of the CAVE2&trade; System. Datasets used are publicly available on the web. An EVL student lies on a bench in front of the walls for added effect. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2 System.

Medical training application in EVL&rsquo;s CAVE2

Virtual Reality for Medical Training

Participants fly through a close-up of a glass fissure computed in a 5-Million atom molecular dynamics nanoscale simulation. Visualization by Aaron Knoll and Michael Papka of the Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory and Khairi Reda of the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. Simulation and data provided by Kenichi Nomura and Priya Vashishta of University of Southern California. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2&trade; System. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago, the Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory, and the University of Southern California.

Molecular Dynamics displayed in EVL&rsquo;s CAVE2

Molecular Dynamics Simulation of a Glass Fissure

On-line, real-time audio dilation is a technique that time stretches, or slows down the tempo of audio signals as they are generated. This research focuses upon the control over incoming live audio tempo for increased intelligibility and comfort. 

The system features networked, full duplex communication, independent variable dilation rates, variable voice activation detection and a length (visual) audio storage buffer. Anticipated application areas include public safety, pilot / flight controllers, language translation, international communication, hearing loss, to name a few.

On-line Audio Dilation with Voice Activation Detection

Audio Dilation

SpiderSense is a wearable device that projects the wearer&rsquo;s near environment on the skin and allows for directional awareness of objects around him. The millions of sensory receptors that cover the skin presents opportunities for conveying alerts and messages. 

Drawing inspiration from Spiderman&rsquo;s SpiderSense, the Human Augmentics wearable tactile display utilizes the skin&rsquo;s pressure receptors to communicate an awareness of the surrounding environment by conveying information about the wearer&rsquo;s distance from surrounding objects. While previous tactile vests communicate directional information, SpiderSense creates an actual feeling of the environment by conveying distance information from objects by applying pressure on the skin. SpiderSense consists of a series of Sensor Modules that are positioned on the body of the user. The Sensor Modules scan the environment using ultrasound to alert the user of objects that are closer than 60 feet. Worn at strategic points over the body, the user can potentially gain a sense of all the obstacles that surround him. 

One can envision two categories of broad applications of SpiderSense: compensating for a dysfunctional or missing sense (i.e. visually or hearing impaired); or supplementing the existing senses (spatial awarness in a spacesuit or seeing behind one&rsquo;s head).

SpiderSense on a blindfolded user

SpiderSense

Participants in the CAVE2&trade; System stand inside a balls-and-sticks model of a molecular chemistry dataset. The clouds represent the density of electrons around atoms. Chemistry models and simulation data are provided by Aslihan Sumer, Juliuis Jellineck and Michael Papka of the Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory. Visualization was created by the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. EVL&rsquo;s OmegaLib software is used to display the 3D model in the CAVE2. 

Credits 
Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago and Argonne Leadership Computing Facility (ALCF) at Argonne National Laboratory.

Electron Density displayed in EVL&rsquo;s CAVE2

Electron Density of a Molecular Dataset

BactoGeNIE, the Bacterial Gene Neighborhood Investigation Environment, is a genome visualization approach developed at Electronic Visualization Laboratory (EVL) to address growing volumes of genome sequence data, specifically microbial genomes, in the biological sciences. Decreases in sequencing costs have driven a proliferation in the volume of complete sequenced genomes. Existing visualization approaches are not designed to enable the comparison of hundreds to thousands of sequenced genomes in one view. At the same time, advances in display hardware have enabled growth in display resolution and the development of large, high-resolution display environments, presenting an opportunity for high-resolution visualization design. 

This novel high-resolution genomics visualization approach transforms genomic data into high-resolution digital maps that enable rapid comparison of hundreds of microbial genomes in one view. The image shown right shows segments of 600 draft E. coli genomes from the PubMed database displayed on the Cyber-Commons display wall at the University of Illinois at Chicago, EVL. Alignment of similar genes from related bacterial strains and coloring by gene identity on a gradient around a target gene, enables researchers to quickly spot insertion, inversion and deletion events across hundreds of bacterial strains. BactoGeNIE demonstrates the potential of high-resolution display environments to facilitate large-scale genomic data analysis. 

See a <a href='http://www.youtube.com/watch?v=yrSyi1RWcUw&amp;feature=share&amp;list=UU3GTGwIcQfqqMS-Wt9uMwyQ&amp;index=2'>BactoGeNIE video</a> on the EVL YouTube channel.

BactoGeNIE displayed on EVL Cyber-Commons

BactoGeNIE, the Bacterial Gene Neighborhood Investigation Environment

omegalib is middleware designed to ease the development of applications for virtual reality (VR) and immersive display environments. A C++ and Python API. Applications can be developed as standalone executables in C++ or as scripts in Python. omegalib also supports mixed native / script applications with user-defined C++ modules that can be exposed to Python through a simple declarative interface. omelib&rsquo;s main features are: 
<ul>
<li>Support for hybrid systems, presenting high definition 2D and 3D content on the same display</li>
<li>Display system scalability: omegalib runs on desktop machines, multi-GPU workstations driving tiled displays, and cluster systems like the 36-machine, 72-display CAVE2</li>
<li>Runtime application switching: applications can be reloaded or swapped without restarting the display system</li>
<li>Web interface control and pixel streaming to HTML5 clients</li>
<li>Experimental SAGE integration</li>
<li>A customizable 2D / 3D widgets library</li>
<li>Support for a wide range of input peripherals (controllers, motion capture systems, touch surfaces)</li>
<li>Integration with third party higher level toolkits like Unity, VTK, Open Scene Graph, Qt, etc.</li>
</ul> 

Although omegalib is designed mostly to run on VR systems, it can be used to develop applications on standard desktop systems as well, leveraging the power of multiple GPUs when available.

omegalib in action

omegalib

Spaced Out uses a NeuroSky MindWave and Unity3d to generate a model of the Solar System affected by brain waves. Solar flares are triggered by high attention, time slows down with high meditation, alpha waves change the size of Saturn&rsquo;s rings, delta waves set the cloud density of Venus, and theta wave spikes sends comets toward the sun. 

This project uses a Hyundai 3D tiled display, Vicon tracking system, the Nintendo Wii for interaction, and surround audio.

Spaced Out

omicron sdk is an input abstraction and utility library for virtual reality systems, providing access to a variety of input devices, mostly used in immersive installations and stereo display systems. 

Some of the devices currently supported by omicron are: 
<ul>
<li>All motion capture systems supporting the VRPN protocol</li>
<li>NaturalPoint trackers (TrackIR, Optitrack)</li>
<li>Wii controllers</li>
<li>Xbox 360 controllers</li>
<li>Microsoft Kinect (multiple Kinects supported. omicron also offers some functions for Multi-kinect transform calibration)</li>
<li>SAGE pointer connections</li>
<li>PQLabs multitouch overlays</li>
<li>iPad touch interfaces and dynamic GUIs through the custom Porthole protocol</li>
<li>Thinkgear brainwave interfaces</li>
</ul> 

omicron abstracts input using modular event services, each providing access to a specific input device. Events can be easily streamed over the network using the omicron connector API. Also, Event services can be chained together to provide advanced functionality. 

In addition to event services, omicron provides additional utility APIs to simplify the developement of the non-graphics part of VR applications. These APIs are mostly lightweight wrappers on top of well established open source libraries. omicron includes: 
<ul>
<li>A configuration file reading system (libconfig)</li>
<li>An xml reading / writing API (<a href='http://www.grinninglizard.com/tinyxml2/index.html'>tinyxml-2</a>)</li>
<li>Multithreading support (<a href='http://tinythread.sourceforge.net'>tinythread++</a>)</li>
<li>A simple tcp client / server API (<a href='http://think-async.com'>asio</a>)</li>
<li>A mathematical / geometry library (<a href='http://eigen.tuxfamily.org/index.php?title=Main_Page'>eigen</a>)</li>
</ul> 

omicron input support can be integrated into C++ applications as a static library, or it can be run as a standalone input server, streaming input data to multiple applications. Omicron comes with client-side interfaces for Unity and Processing. 

C++, using a single header. Only a few lines of code needed. For more information see the omicronConnector section on the <a href='http://code.google.com/p/omicron-sdk/wiki/GettingStarted'>Getting started</a> page.

omicron sdk

Researchers from the UIC Departments of Computer Science and BioEngineering propose to develop graphical stereoscopic visual applications for a standard single monitor as well as an ultra-high resolution multi-paneled platform. The visualization will focus on viewing 4D-dimensional (space and time) medical images from angiography, MRI, CT, and computer simulations for an individual patient. The graphical applications developed as part of &ldquo;Artificially created cortical functional blood unit&rdquo; will allow neurosurgeons to explore patient-specific data by efficiently organizing and simultaneously displaying all relevant information. Software is being developed to extract quantitative information from patient-specific images such as contrast transit times, as well as sizes and dimensions of anatomic regions of interest. Interfaces are also being created to overlay hemodynamic simulation results into the graphical environment. The visualization tools will enhance insight about the pathology of patients suffering from neurovascular disease, especially stroke. Improved preparation for surgical intervention is expected to empower surgeons to choose better medical treatment paths. 

A semi-cylindrical multi-paneled mini-CAVE prototype is being built to render the graphical software applications mentioned above. This virtual reality environment will afford surgeons the ability to incorporate stereo depth perception and immerse themselves fully in a high-resolution environment created by the medical image. User interaction with touch inputs, tablet control, and body motion will provide a novel means of data exploration. This mini-CAVE will offer physicians a means to explore physiologically accurate patient-specific medical images with links to quantitative computer analysis. This research aims to create a three-dimensional virtual reality interface that allows physicians to interact with existing advanced scientific tools in an intuitive fashion. Tools are being created for surgical planning of medical interventions for cerebrovascular diseases that will address spatial constraints such as trajectories through bone and brain tissue, as well as avoidance of white matter fiber tracts. To assess the benefits for 4D surgical planning, a user study is proposed comparing surgeons provided with conventional standard images versus the mini-CAVE exploration environment.

Major arteries reconstructed from a medical image

Reconstructing the Cerebral Vascular Network

Modern biological experiments generate large amounts of data describing intracellular dynamics. Rule-based languages for describing intracellular biochemistry allow for the construction and simulation of detailed models with unprecedented scope and precision. Rule-Based Models (RBM) can be used to suggest new hypotheses and new ideas for future experimentation. 

Rule-based models can be difficult to understand and time consuming to debug. RuleBender is a free tool for constructing, debugging, simulating and analyzing rule-based biological models in the BioNetGen language. The MOSBIE extension further provides a tool for the comparison and analysis of biochemical model families.

RuleBender: Rule Based Modeling and Simulation with RuleBender

The growing onslaught of astronomical data available presents a great challenge. Astronomy lacks an easy-to-use and scalable way to collect and distribute expert information about objects from datasets of tens of thousands to billions of individual events and objects. Over the next decade, the amount of information available to the typical astronomer will grow by two orders of magnitude both in raw data size and in the number of objects. This project pursues three research directions, each of which has the potential to transform how astronomers interface with large datasets: 
1) a visual scalable annotation framework to enable linking of observations to specific experiments, models, or other observations; 
(2) a continuous workflow enactment system that would support automated reasoning in the presence of uncertainty; and > 
(3) a computational framework for interactively analyzing astronomical datasets, allowing the construction and testing of hypotheses. 
Project plans include the design and development of a prototype system (AstroShelf) and its evaluation using two timely science programs: 
(a) Using multi-wavelength data from the DEEP3 and AEGIS surveys, develop methods to incorporate images and catalogs from disparate datasets, allowing us to study how the demographics of galaxies have changed over the last 8 billion years. 
(b) Using properties of time-variable events found by the Pan-STARRS survey, develop techniques for rapid classification of transient phenomena, dissemination of their properties, and incorporation of feedback from follow-up observations. 
AstroShelf will include a publicly accessible, flexible annotation system for public datasets, which will also lend itself to outreach efforts involving annotations by the general public. This project\'s significant impact is the ability to share information and expert opinions quickly and widely, about each new observation or event, fundamentally changing our ability to learn about the Universe; such functionality can also be applied in support of other scientific domains.

Astronomical Data Visualization

Astroshelf: Panning and Zooming the Observable Universe

We developed a framework for integrating measured data - such as medical images, tracked motion, and anatomy-book knowledge - into the physically-based simulation of anatomical joints. The resulting models and simulations can be used to analyze medical data collected from a specific individual.

Predictive Computational Models for Orthopaedics

Simulation and modeling of turbulent flow, and of turbulent reacting flowin particular, involves solving for and analyzing time-dependent and spatially dense tensor quantities, such as turbulent stress tensors. Visual exploration of these tensor quantities can effectively steer the computational modeling of combustion systems. The project builds a visual analysis tool to be utilized in debugging, benchmarking, and verification of models and solutions in turbulent combustion.