EDITOR&rsquo;S NOTE: The following news release was posted to UIC NEWS on December 17, 2014. It describes NSF award #IIS-1445796 for &ldquo;Articulate: Augmenting Data Visualization With Natural Language Interaction,&rdquo; for a one year period (August 2014 - July 2015) in the amount of $300,000. This is a collaborative project between UIC and University of Hawaii at Manoa (UHM). Those involved are UIC computer science faculty members Barbara Di Eugenio (PI), Andy Johnson (co-PI) and Adjunct CS Professor Leland Wilkinson (co-PI), and from UHM, computer science faculty member Jason Leigh (PI). 

&lsquo;Draw me a picture,&rsquo; say scientists; computer may respond 
Jeanne Galatzer-Levy, UIC NEWS 
December 17, 2014 

Like the rest of us, scientists wish they could just ask a computer a question and have it respond with an answer presented in an easy-to-understand picture. Today’s visualization tools can translate huge raw data sets into graphs and maps - but most scientists lack the time and training to use the tools effectively. 

The National Science Foundation has awarded a $300,000 grant to the University of Illinois at Chicago and the University of Hawaii to take the first steps towards developing a computer that can take data and produce meaningful visualizations based on natural language requests, accompanied by common gestures like pointing. 

With nearly a third of the human brain devoted to processing visual stimuli, vision has always been our dominant way of acquiring information. Visualization is the most effective means of converting raw data into insight that can support scientific discovery, says Andrew Johnson, director of research at UIC&rsquo;s Electronic Visualization Laboratory.

&ldquo;Today, with big data, you really need to be using visualizations to help you figure out what it is you’re looking at,&rdquo; said Johnson, who is a co-principal investigator on the NSF grant. &ldquo;Visualization should be interactive; a dynamic process. We want scientists to be able get ideas out there quickly.&rdquo; 

&ldquo;The object is to make it more like a conversation with a person in the room,&rdquo; said Barbara Di Eugenio, UIC professor of computer science and principal investigator on the grant. &ldquo;&lsquo;Can we recolor that? What if we moved it this way, or inverted that axis?&lsquo;&rdquo; 

&ldquo;We also don’t want to require explicit questions,&rdquo; she said. &ldquo;The goal is for the computer to be able to interpret even indirect questions, like &lsquo;It would be better to show salinity only at 10 meters&rsquo;, or even statements that hint at something, and put together the visualization.&rdquo; 

<a href="http://news.uic.edu/draw-me-a-picture-say-scientists-computer-may-respond">Read the article</a>. 

NOTE: This work is based on UIC computer science PhD student Yiwen Sun&rsquo;s dissertation, <a href="http://www.evl.uic.edu/core.php?mod=4&type=3&indi=457">&ldquo;Articulate: Creating Meaningful Visualizations from Natural Language&rdquo;</a>. 

<a href="http://youtu.be/V1CFRv4Q8bk?list=PL4071582316A3714C">View a video describing this earlier work.</a>

Jillian Aurisano, Ph.D. student in computer science, works with the Articulate prototype in EVL cyber-commons classroom.

&lsquo;Draw me a picture,&rsquo; say scientists; computer may respond

EVL PhD candidate, Jillian Aurisano present her research in data visualization and exploration using a conversational interface. 
Visual data exploration poses challenges for &rsquo;InfoVis Novices&rsquo;. A &rsquo;conversational interface&rsquo; which would enable users to generate and interact with visualizations through natural language and gestures, while maintaining a history of the data exploration, has the potential to ameliorate many of these challenges. We performed an exploratory, observational study designed to examine the role of such a conversational interface in visual data exploration. We simulated a conversational interface, using a remote human mediator, with multiple cycles of visualization construction. We believe analysis of this data will yield concrete design goals for conversational interfaces in information visualization. Research results and future work will be presented.

&ldquo;Show Me Data.&rdquo; Observational Study of a Conversational Interface in Visual Data Exploration

A natural language interface to visual data exploration would allow a user to directly specify questions through speech, allowing the user to focus on higher-order tasks, such as hypothesis generation and question formulation. 
However, visual data exploration involves repeated cycles of visualization construction and interaction, as well as reasoning across many visualizations generated over the course of an exploratory session. A &rsquo;conversational interface&rsquo;, which maintains a dialog with the user through natural language and gestures, could support these complex tasks. 
We conducted an observational, exploratory study to observe the interaction between a subject and a remote data analysis expert (DAE) who assists the subject in an exploratory data analysis task.

Research subject involved in an exploratory data analysis task using the EVL Cyber-Commons tiled display wall.

&ldquo;Show me data&rdquo; Observational study of a conversational interface in visual data exploration

Newcity magazine is Chicago&rsquo;s only locally owned and operated cultural weekly, and has been covering Chicago&rsquo;s local scene for more than 25 years. It&rsquo;s current &ldquo;Best of Chicago&rdquo; issue just came out (Volume 31, No. 1359). The section on City Life lists many best things about Chicago, including &ldquo;Best place to get your geek on.&rdquo; Um, that would be the Electronic Visualization Laboratory at the University of Illinois at Chicago! The article is reproduced below. 
* * * * * * * * * * 
Best place to get your geek on 
The Electronic Visualization Laboratory (EVL) at UIC 
Cutting-edge research meets a massive wall made out of small screens that would make any hardcore videogame fan scream and a literal cave that opens up a highly immersive, surround-screen, surround-sound, virtual reality environment at the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. The lab brings together academics, designers, engineers, artists and mad scientists using technologies that bend reality off into unimaginable directions. EVL is the place where visualization, virtual reality and real-time interactive computer graphics are born. As if that were not enough, it was here that the computer graphics for the first &ldquo;Star Wars&rdquo; film were created in 1976. How cool is that? 
UIC College of Engineering, Department of Computer Science, 842 W. Taylor 
Best of Chicago 2016

Best of Chicago 2016

Newcity magazine rates UIC/EVL as &ldquo;Best place to get your geek on&rdquo;

InfoVis novices struggle with visualization construction. Even with the aid of visualization software, such users may face challenges when translating their questions into appropriate visual encodings, or interactively refining the representation to achieve a desired result. 
A &lsquo;conversational interface&rsquo; which maintains a dialog with the user through natural language and gestures, could allow users to engage in repeated cycles of visualization generation and modification, asking questions directly through speech. 
In this poster we present a prototype conversational visual data analysis system. Our prototype was developed from a corpus consisting in 15-subjects engaging in exploratory data visualization with a simulated conversational interface. It features 1) speech to visualization pipeline, 2) classification system to divide utterances into major types, 3) history manager and knowledge-base.

Articulate2: Toward a Conversational Interface for Visual Data Exploration

The goal of our research is to support full-fledged dialogue between a user and a system that transforms the user queries into visualizations. So far, we have collected a corpus where users explore data via visualizations; we have annotated the corpus for user intentions; and we have developed the core NL-to-visualization pipeline.

User examining crime data on tiled display.

Towards a Dialogue System that Supports Rich Visualizations of Data

Just as NSF cyberinfrastructure produces greater volumes and varieties of data from data storage systems, online instrumentation and major computational resources like XSEDE and Blue Waters, advanced visualization instruments serve as the &ldquo;lenses&rdquo; of these data generators (similar to the telescopes and microscopes of yore), enabling researchers to peer into cyberspace to manage, access and analyze big data. 
Continuum is a cyberinfrastructure instrument that unifies high-resolution computer-enhanced group collaboration workspaces with arrays of ambient sensors, enabling users to collaborate with local and remote colleagues and their data more effectively by giving the room itself the ability to anticipate their needs, improve their interaction, and focus their time on discovery rather than the technology. Continuum is a digital assistant, benefiting global scientific collaboratories and providing a foundation for new computer science research. War rooms benefit collaboration because they support continuous interaction and awareness of team members, enable persistent information sharing, and encourage mobility among team members and information. A computer-enhanced war room, such as Continuum, which (1) connects distributed teams with their data, whether stored in the cloud or streamed from major computational resources over high-speed networks, (2) supports persistent digital artifacts, (3) enables ultra-high-resolution 2D and 3D stereoscopic digital media to be simultaneously viewed, and (4) supports ubiquitous and intuitive interaction, creates a powerful and easy-to-use information-rich environment in support of scientific discovery. 
Continuum is the culmination of the University of Illinois at Chicago’s (UIC) 20+ years experience developing interactive instruments, from the original CAVE in 1992, to the ultra-high-resolution LambdaVision tiled LCD wall in 2004, and the hybrid-reality CAVE2 in 2012. Each new generation of visualization instrumentation provided scientific communities with more advanced features Continuum development focuses on three areas: (1) transforming passive display walls that respond to user commands into attentive display walls that monitor the users in a room and anticipate their needs to improve their experience; (2) providing an alternative approach to constructing high-resolution visualization display walls using thin-border 4K passive stereo OLED displays that have impressive brightness, contrast, uniformity, off-axis viewing, and low reflectivity in brightly lit rooms; and (3) utilizing cloud computing to prototype future Continuum spaces that don’t require local high-end compute and display resources. Twelve funded research projects from local institutions, in Astronomy, Astrophysics, Biology,
Combustion, Geophysics, Neuroscience, Pathology, Physics, Psychiatry, and High Performance Computing, are poised to use Continuum for their data visualization needs. Continuum will open up new opportunities in computer science research at the intersection of large-scale data visualization, human-computer interaction, and high-speed networking. Continuum will also directly support 10 classes taught in the UIC Computer Science, Art and Design, and Biomedical Science departments.

Artist's rendering of Continuum space.

MRI - Development of Continuum: A Virtualized Attentive Environment for Amplified Collaboration

Research Experience Undergraduate, BS 2018

Joe Borowicz

BS Graduate, 2022 Research Experience Undergraduate (2018-2021)

Kenan Arica

Undergraduate Researcher (REU), BS 2018

Kayla Condrella

Pi: Barbara Di Eugenio; Co-I: Debaleena Chattopadhyay 
A growing body of research has found that patients who actively manage their health have better outcomes, and patient engagement has become a central tenet in recent healthcare approaches. However, it has also been shown that many patients lack knowledge of their disease and medications, particularly as concerns chronic health conditions, whose exacerbation causes about two thirds of hospitalizations every year in the USA. Discharge instructions after hospitalization are a critical tool to engage patients in their self-care, but are often unsuccessful. Our hypothesis is that, to engage patients at discharge, a personalized hospitalization summary should be provided, that integrates physician and nurse documentation, and is informed by the patient's level of engagement in self-care and their socio-cultural perspective. 
Our transformative pilot work on physician and nurse documentation has identified an additional obstacle to the effective flow of information from professionals to patients: vocabulary differences between physicians and nurses. By leveraging the Unified Medical Language System Metathesaurus, we found no common synonyms between physician and nurse documentations for 75% of patients. Further, 86% of terms patients use are simple compared to only 15% for physicians. The lack of common language among patients, physicians and nurses motivates including all perspectives to make instructions meaningful for patients. Our hypothesis is that a better informed patient population who understands the connections between their illness, hospital experience, and discharge instructions will be more empowered and more engaged in their chronic disease management. Our innovation is to automatically create an easy-to-understand, personalized hospitalization summary that integrates events described in the EHR; and that is responsive to the patient's needs and preferences, and their activation framework (PAF) level. We pursue four specific aims: 1. To investigate socio-cultural patterns of patient perceptions of illness via a socio linguistic analysis of patient interviews; 2. To integrate multi-faceted information about a patient, as provided by physicians and nurses; 3a. To develop MyPHA, a novel application program, which will automatically generate the personalized summaries and deliver them to the patient on a tablet. MyPHA will be informed by aims 1 and 2, and will be grounded in Natural Language Generation technology; 3b. To evaluate MyPHA with multiple cycles of formative evaluation with all stakeholders, and a summative evaluation with patients. 
Whereas our pilot work has focused on heart failure, the techniques we will develop can easily be adapted to the management of other chronic health conditions, in particular for patients with cancer and cancer survivors. In addition, cancer and heart failure can co-occur, especially in older patients, and several cancer treatments can lead to the development of heart failure. The software program MyPHA will be released ooen source to the research communitv to enable raoid customization to other chronic conditions. 
Awarded Amount: $1,495,213

NIH: R01: SCH: INT: MyPHA: Automatically generating personalized accounts of inpatient hospitalization

Ph.D. Student: Jillian Aurisano 
Date: Tuesday, September 17, 2019 
Time: 11:00 am 
Location: Room 3036, Engineering Research Facility 
Committee: 
Andrew Johnson (Chair) 
Debaleena Chattopadhyay 
Barbara Di Eugenio 
G. Elisabeta Marai 
Rick Stevens (Argonne National Laboratory, University of Chicago) 
Abstract: 
Analysis of large, complex datasets stands to benefit from environments that permit users to view and juxtapose many views of data. Large, high-resolution environments are capable of showing many related views of data, but interaction with these views poses significant challenges in visual and interaction design. In this talk, I will present work toward &ldquo;multi-view interactions&rdqo; that enable users to create, organize and act on many views at once, through multi-modal speech+gesture queries in large flexible canvas environments. The goal is the enable users to rapidly and efficiently generate sets of views in support of multi-view analysis tasks, organize these views to reflect changing analysis goals, and operate on sets of views collectively, rather than individually, to efficiently reach large portions of the &lsquo;data+attribute space&rsquo;. I will implement and evaluate this approach within two domains and two user communities: City of Chicago data for informed citizens and deep learning predictions data for precision medicine in cancer.

Preliminary Examination Announcement: &ldquo;Multi-view, multi-modal speech+gesture interaction in large display environments&rdquo;

PhD Student: Abhinav Kumar 
Date: September 27, 2019 
Time: 1:00 PM to 3:00 PM 
Location: Room 2068, Engineering Research Facility 
Committee: 
Barbara Di Eugenio (Chair and Advisor) 
Andrew Johnson 
Natalie Parde 
Cornelia Caragea 
Kallirroi Georgila 
Abstract: 
When exploring data visualizations, users that are novices to visualization, oftentimes struggle with translating their particular queries into an effective visual representation, which this research aims to address by automatically constructing and managing visualizations appropriately. I am particularly focusing on the language processing, which is challenging because spoken dialogue is oftentimes under-specified, leading to open interpretation and in our case potentially the wrong visualization. My main contributions are as follows. First, I am developing language processing in the multimodal setting. I worked with others in collecting and annotating a corpus of dialogue data, comprising of speech text and hand gestures, that came from data collected via human experiments. Second, I used a local context for building a module for interpreting the underlying intent of what the user says. Third, given the small size of our dataset, I augmented it using paraphrasing and showed that the intent interpretation module improved on the augmented data. As part of my proposed work, I will work on text segmentation to identify the boundaries of the local context that my current intent interpretation module operates on. I also plan to develop a model that captures references made through language and hand pointing gestures and determines the appropriate target visualizations for them.

Preliminary Examination Announcement: &ldquo;Towards a Natural Language Interface for Exploring Data Visualization&rdquo;

PhD Candidate

Abhinav Kumar

SAGE2 is a user-centered platform for small groups or large distributed teams to access digital media datasets from various sources and juxtapose, share and investigate content on tiled display walls. SAGE2 builds on SAGE (Scalable Adaptive Graphics Environment), developed with prior NSF funding. SAGE2 is having a transformative effect on data exploration, visualization and collaboration, making cyberinfrastructure more accessible to end-users, both in the laboratory and in the classroom. 
<a href="https://doi.org/10.6084/m9.figshare.12837638">2020 Poster: https://doi.org/10.6084/m9.figshare.12837638</a> 
<a href="https://doi.org/10.6084/m9.figshare.12837638">2020 PPT: https://doi.org/10.6084/m9.figshare.12837695</a>

SAGE2 Scalable Amplified Group Environment (poster 2020)

This paper describes results from an observational, exploratory study of visual data exploration in a large, multi-view, flexible canvas environment. Participants were provided with a set of data exploration sub-tasks associated with a local crime dataset and were instructed to pose questions to a remote mediator who would respond by generating and organizing visualizations on the large display. We observed that participants frequently posed requests to cast a net around one or several subsets of the data or a set of data attributes. They accomplished this directly and by utilizing existing views in unique ways, including by requesting to copy and pivot a group of views collectively and posing a set of parallel requests on target views expressed in one command. These observed actions depart from multi-view flexible canvas environments that typically provide interfaces in support of generating one view at a time or actions that operate on one view at a time. We describe how participants used these &ldquo;cast-a-net&rdquo; requests for tasks that spanned more than one view and describe design considerations for multi-view environments that would support the observed multi-view generation actions. 
Index Terms: Human-centered computing - Empirical studies in visualization 
This work was supported initially by NSF award IIS 1445751 and currently by NSF award CNS 1625941. 
<a href="https://doi.org/10.1111/cgf.13976">https://doi.org/10.1111/cgf.13976</a>

A participant generates 15 views of her data. In the final request, the participant references a set of three views and poses a request to copy and pivot these views collectively to four new subsets of the data. The final result is a grid of views.

Many At Once: Capturing Intentions to Create And Use Many Views At Once In Large Display Environments

PI: Andrew Johnson (UIC/EVL); co-PIs: Barbara DiEugenio (UIC/CS), Moira Zellner (Norheastern University) 
Collaborator: Jason Leigh (University of Hawaii Manoa) 
The Internet gives the public, including scientists, access to vast amounts of data. Visualization is a powerful tool to take that data and turn it into a form that is easier to understand, helping people make better decisions. However, creating good visualizations from data is not easy. The Articulate+ Project will work to generate new knowledge about how to empower people with information visualization, by using speech and gesture as inputs, in new ways. A goal is to enable users to benefit from imprecise statements about what they want, rather than needing to give specific commands telling the computer what to do. This project will make it easier for citizens to engage in conversations with the computer about economic, medical data, demographic, transportation data, climate, and sports data; to answers that are more meaningful, and so to democratize data literacy. 
This research will use speech, gesture, and log data as new input interaction modalities for visual analytics. It will develop new understandings of how the imprecise and vague nature of natural language queries and gestures, as contextualized by work on a specific visual analysis problem, can be modeled as an opportunity for an intelligent software system to learn more about the underlying intent of the users. This model of intent, in turn, will be used to provide contextualized visualizations, which are expected to help those users gain valuable insights from their data. The audio speech data will be used to computationally model overhearing, to help infer users&rsquo; current contextualized goals. The gesture data will be used to disambiguate expressions that refer to visual elements of a visualization (e.g., &ldquo;that pie chart about electricity consumption&rdquo;). The project will develop new understandings of how to combine the log of visualization state, the computational model of overhearing, and the gesture data in order to generate new visualizations in support of users&rsquo; work on tasks. Evaluations will be performed in both controlled laboratory and naturalistic study settings to determine whether effective semantic parsers can be derived for these specific visual analysis domains, and how this contextual interface affects users&rsquo; experience of visualization and discovery. The integrated annotated datasets from the studies will be made available to the research community satisfying a need for ecologically valid situated language interaction corpora.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Articulate+ - A Conversational Interface for Democratizing Visual Analysis

UIC Computer Science Professors Andy Johnson (PI) and Barbara Di Eugenio (co-PI) have been awarded a $333,000 NSF grant entitled &ldquo;CHS: Small: Collaborative Research: Articulate+ - A Conversational Interface for Democratizing Visual Analysis&rdquo; project period October 2020 to September 2022. Dr. Moira Zellner, now of Northeastern University will also participate as co-PI on this grant. This project is part of a larger overall award of $500,000 that also includes Jason Leigh at University of Hawaii at Manoa. 
&ldquo;Articulate+&rdquo; continues EVL&rsquo; research in Natural Language Processing and Visualization also sponsored by NSF for the last 5+ years. See <a href="https://www.evl.uic.edu/entry.php?id=2522">Articulate+ - A Conversational Interface for Democratizing Visual Analysis</a> for more information on this project. 
NSF Award Number (FAIN): 2007257

NSF Grant &ldquo;CHS: Small: Collaborative Research: Articulate+&rdquo; Awarded

October 1 - - EVL Director of Research Andrew Johnson has been named Interim Director of EVL following Maxine Brown&rsquo;s retirement from UIC. Johnson now assumes the responsibility of heading EVL - overseeing EVL&rsquo;s overall mission in education, research, development, and technology transfer. 
For over 23 years, Johnson has been a valued EVL faculty member (Professor of Computer Science) and researcher who has focused upon interaction and collaboration using advanced visualization displays and the application of those displays to enhance discovery and learning. 
His areas of expertise include visualization, virtual and augmented reality, natural language processing, visual analytics, and high-performance networking. 
Over the years, Johnson has been instrumental in the development of EVL instrumentation and systems, including the Continuum smart environment, CAVE2&trade;, LambdaVision high-resolution tiled displays, LambdaTable tabletop touch system, GeoWall portable geoscience display environment, PARIS augmented reality and ImmersaDesk virtual reality systems. He has overseen the development of numerous visualization and virtual reality applications across diverse domain disciplines from science and engineering to the humanities and art. Johnson is a highly regarded mentor and educator to EVL&rsquo;s graduate students and undergraduates alike. 
Johnson retains his position as Director of Research while leading EVL&rsquo;s future initiatives and innovations as Interim Director.

Andrew Johnson Named Interim Director, EVL

Visualization, while an effective tool for identifying patterns and insights, requires expert knowledge due to challenges faced when translating user queries to visual encodings. Research has shown that using a natural language interface (NLI) is effective for these challenges because the user can simply talk to a computer capable of producing the graphs directly. In this paper, we discuss our intelligent assistant which processes speech and hand pointing gestures while also dealing with any number of visualizations on a large screen display. Evaluation of the system shows that it is capable of quickly producing visualizations. It also particularly effective
at responding to less ambiguous queries, while in certain cases can handle ambiguous or complex queries.

The System Pipeline

Intelligent Assistant for Exploring Data Visualizations

Our goal is to develop an intelligent assistant to support users explore data via visualizations. We have collected a new corpus of conversations, CHICAGO-CRIME-VIS, geared towards supporting data visualization exploration, and we have annotated it for a variety of features, including contextualized dialogue acts. In this paper, we describe our strategies and their evaluation for dialogue act classification. We highlight how thinking aloud affects interpretation of dialogue acts in our setting and how to best capture that information. A key component of our strategy is data augmentation as applied to the training data, since our corpus is inherently small. We ran experiments with the Balanced Bagging Classifier (BAGC), Condiontal Random Field (CRF), and several Long Short Term Memory (LSTM) networks, and found that all of them improved compared to the baseline (e.g., without the data augmentation pipeline). CRF outperformed the other classification algorithms, with the LSTM networks showing modest improvement, even after obtaining a performance boost from domain-trained word embeddings. This result is of note because training a CRF is far less resource-intensive than training deep learning models, hence given a similar if not better performance, traditional methods may still be preferable in order to lower resource consumption. 
Keywords: Dialogue, Corpus, Statistical and Machine Learning Methods

Contextualized actionable requests segment relevant user utterances in each turn of the conversation.

Augmenting Small Data to Classify Contextualized Dialogue Acts for Exploratory Visualization

Candidate: Jillian Aurisano 
DATE: Friday, July 9, 2021 
TIME: 1:00PM 
LOCATION: <a href="https://uic.zoom.us/j/84367747564?pwd=T3NiZUR3cjQwanQxa0JiaC92RXQzUT09">Zoom Link</a> 
Meeting ID: 843 6774 7564 
Passcode: 6DwiV1hA 
Committee Members: Andrew Johnson (chair), Barbara Di Eugenio, G. Elisabeta Marai, Debaleena Chattopadhyay, Moira Zellner 
Abstract:
Visual data exploration stands to benefit from environments that permit users to view and juxtapose many views of data, particularly views that present diverse selections of data values and attributes. Large, high-resolution environments are capable of showing many related views of data, but efficiently creating and displaying visualizations in these environments presents significant challenges. In this talk, I will present my research on &ldquo;multi-view data exploration interactions&rdquo; that enable users to create and pivot many views at once, creating sets of views with coherent data value and attribute variations, through multi-modal speech and mid-air pointing gestures in large display environments. This work enables users to rapidly and efficiently generate sets of views in support of multi-view data exploration tasks, organize these views in coherent collections, and operate on sets of views collectively, rather than individually, to efficiently reach large portions of the &rsquo;data and attribute space&rsquo;. I will present three contributions: 1) an observational study of data exploration in a large display environment with speech and mid-air gestures, 2) &rsquo;Traverse&rsquo;, an interaction technique for data exploration, based on this study, which uses natural language to create and pivot sets of views, and 3) &rsquo;Ditto&rsquo;, a multi-modal speech and mid-air pointing gesture interactive environment, which utilizes the multi-view data exploration technique, in large display environments.

PhD Dissertation Announcement: Multi-view, multi-modal speech and mid-air gesture interaction for data exploration in large display environments

Construction activities for the new $117.8 million, 135,000-square-foot Computer Design Research and Learning Center to be located at 900 W. Taylor St. is underway. The CDRLC will support the Department of Computer Science&rsquo;s growing student enrollment with 16,000 square feet of classroom space for collaborative teaching and learning environments, as well as 35 labs for faculty research activities in the computer science and engineering disciplines. 
The CDRLC will be the new home for the Electronic Visualization Laboratory featuring the latest technological and computing innovations to support research in advanced visualization, networking and collaborative environments. 
Read more about the CDRLC groundbreaking in the <a href="https://today.uic.edu/uic-breaks-ground-on-new-computer-design-research-and-learning-center">July 15th issue of UIC Today.</a> 
See a live webcam of the building&rsquo;s progress via the <a href="https://webcampub.multivista.com/index.cfm?fuseaction=aPublicWebcam.embed&WebcamPublicEmbedUID=5D4DCF05-AB2B-4770-A090-D79BE467F958">Multivista Construction Documentation webcam.</a>

New Computer Design Research and Learning Center Breaks Ground

MS Student

Veronica Grosso

Data exploration stands to benefit from environments that permit users to examine and juxtapose many views of data, particularly views that present diverse selections of data values and attributes. Large, high-resolution environments are capable of showing many related views of data, but efficiently creating and displaying visualizations in these environments presents significant challenges. In this dissertation, I will present my research on &ldquo;multi-view data exploration interactions&rdquo; that enable users to create sets of views with coherent data value and attribute variations, through multi-modal speech and mid-air pointing gestures in large display
environments. This work enables users to rapidly and efficiently generate sets of views in support of multi-view data exploration tasks, organize these views in coherent collections, and operate on sets of views collectively, rather than individually, to efficiently reach large portions of the &lsquo;data and attribute space&rsquo;. I will present three contributions: 1) an observational study of data exploration in a large display environment with speech and mid-air gestures, 2) &lsquo;Traverse&rsquo;, an interaction technique for data exploration, based on this study, which uses natural language to create and pivot sets of views, and 3) &lsquo;Ditto&rsquo;, a multi-modal speech and mid-air pointing gesture interactive environment, which utilizes the multi-view data exploration technique, in
large display environments.

Study setting with the remote mediator in an adjacent room generating views using Tableau and dropping them to the user&rsquo;s display environment using SAGE2.

Interaction with Multiple Data Visualizations Through Natural Language Commands

The COMPaaS DLV: Composable Platform as a Service Instrument for Deep Learning & Visualization user community consists of faculty from Physics and four College of Engineering departments (Computer Science, Civil, Materials & Environmental Engineering, Mechanical and Industrial Engineering, and Electrical and Computer Engineering) - primarily run applications that are GPU-centric for compute, with significant variability in storage and networking around their data requirements. 
Computer Science applications primarily focus on security, data science, computer vision and Machine Learning (ML). Security projects explore the complexity of modern web applications and the intricacies of security mechanisms that often result in flaws that expose users to significant security and privacy threats. These projects try to develop methods and tools that enable users to understand and more effectively manage retrospective privacy in the context of modern, long-lived, online archives. Composable resources were used to develop Natural Language Processing (NLP)-based domain-specific classifiers that identified data practices stated in privacy policies. Adherence of corresponding applications were then adjusted based on this ground truth. 
Data-science applications have an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help users access, share, explore and analyze their data, whether local or remote, come to conclusions, and make decisions with greater speed, accuracy, comprehensiveness and confidence. One such project is developing and advancing tools that identify image data in biomedical literature to locate beneficial, targeted publications. This work involves training image classifiers, integrating classifiers into labeling pipelines, designing retrieval user interfaces, and identifying related visual representations. 
Computer vision projects include semantic segregation and 3D human pose estimation. Researchers are developing a novel network architecture, termed DependencyNet (dependency network), for semantic segmentation. They also achieved experimental results that demonstrate an effective approach for 3D human pose estimation. Over the past two years, they found COMPaaS to be consistently stable and efficient, with result output as expected, and the group&rsquo;s models achieved state-of-the-art performance on their respective benchmarks. 
ML applications include frameworks for many complex real-world reinforcement learning problems, such as the coordination of autonomous vehicles, network packet delivery, and distributed logistics. 
Civil, Materials & Environmental Engineering applications focus on simulation and modeling. Researchers run data-driven models on high-performance computers to develop an accurate and general neural network ML model that uses crystallographic data to study patterns of synthesizability. They also perform simulations of mass transport in alloys and ceramics. COMPaaS has been performing 1.5-10 times faster than comparable infrastructures they are familiar with. Additionally, they found our use of Jupyter Notebooks to be a significant asset. 
Mechanical and Industrial Engineering researchers use COMPaaS for three research projects: feature extraction in fluid flow using a Convolutional Neural Network (CNN); column height detection in metallic nanoparticles using a CNN; and, electric vehicle battery state-of-charge estimation using different ML methods. 
Electrical and Computer Engineering researchers recently started running mathematical models of ML algorithms and training language models using long-short term memory (LSTMs). 
In 2020, we began implementing a GPUoE (GPU over Ethernet) prototype. Using a GPU expansion chassis connected to compute nodes over Ethernet, we were able to compose remote GPUs into our existing composable infrastructure. Our APIs developed for composable infrastructure have been extended to support these remote GPUs. 
In 2021, we enhanced COMPaaS with a public-facing JupyterHub server providing secure web-based access to experiments, a modern PCIe fabric and a supporting expansion chassis, thereby making composable co-location available to our users. As familiarity with composable infrastructure grows, the idea of using composable co-location instead of traditional servers co-located in racks is possible. By composable co-location, we mean the integration of emerging hardware needs without changing the existing core infrastructure. Researchers can now co-locate their own accelerators or compute nodes and leverage existing COMPaaS resources natively.

COMPaaS DLV User Community

In today&rsquo;s Big Data era, data scientists require modern workflows to quickly analyze large-scale datasets using complex codes to maintain the rate of scientific progress. These scientists often rely on available campus resources or off-the-shelf computational systems for their applications. Unified infrastructure or over-provisioned servers can quickly become
bottlenecks for specific tasks, wasting time and resources. Composable infrastructure helps solve these problems by providing users with new ways to increase resource utilization. Composable infrastructure disaggregates a computer&rsquo;s components – CPU, GPU (accelerators), storage and networking - into fluid pools of resources, but typically relies upon infrastructure engineers to
architect individual machines. Infrastructure is either managed with specialized command-line utilities, user interfaces or specification files. These management models are cumbersome and difficult to incorporate into data-science workflows. We
developed a high-level software API, Composastructure, which, when integrated into modern workflows, can be used by infrastructure engineers as well as data scientists to reorganize composable resources on demand. Composastructure enables infrastructures to be programmable, secure, persistent and reproducible. Our API composes machines, frees resources, supports multi-rack operations, and includes a Python module for Jupyter Notebooks. 
Keywords - distributed systems, testbed implementation and deployment, composable infrastructure, deep learning, visualization, infrastructure as code

Composastructure System Architecture

Moving from Composable to Programmable

Composable infrastructure holds the promise of accelerating the pace of academic research and discovery by enabling researchers to tailor the resources of a machine (e.g., GPUs, storage, NICs), on-demand, to address application needs. We were first introduced to composable infrastructure in 2018, and at the same time, there was growing demand among our College of Engineering faculty for GPU systems for data science, artificial intelligence / machine learning / deep learning, and visualization. Many purchased their own individual desktop or deskside systems, a few pursued more costly cloud and HPC solutions, and others looked to the College or campus computer center for GPU resources which, at the time, were scarce. After surveying the diverse needs of our faculty and studying product offerings by a few nascent startups in the composable infrastructure sector, we applied for and received a grant from the National Science Foundation in November 2019 to purchase a mid-scale system, configured to our specifications, for use by faculty and students for research and research training. 
This paper describes our composable infrastructure solution and implementation for our academic community. Given how modern workflows are progressively moving to containers and cloud frameworks (using Kubernetes) and to programming notebooks (primarily Jupyter), both for ease of use and for ensuring reproducible experiments, we initially adapted these tools for our system. We have since made it simpler to use our system, and now provide our users with a public facing JupyterHub server. We also added an expansion chassis to our system to enable composable co-location, which is a shared central architecture in which our researchers can insert and integrate specialized resources (GPUs, accelerators, networking cards, etc.) needed for their research. 
In February 2020, installation of our system was finalized and made operational and we began providing access to faculty in the College of Engineering. Now, two years later, it is used by over 40 faculty and students plus some external collaborators for research and research training. Their use cases and experiences are briefly described in this paper. Composable infrastructure has proven to be a useful computational system for workload variability, uneven applications, and modern workflows in academic environments. 
Keywords - composable infrastructure, deep learning, visualization, resource management, workload management, user workflow, composable co-location, infrastructure as code

COMPaaS Hardware Racks

Composable Infrastructures for an Academic Research Environment: Lessons Learned

This academic year 2023 EVL alum Mike Papka has joined UIC&rsquo;s Department of Computer Science and EVL faculty as a professor, bringing years of expertise in high performance computing to campus. For nearly three decades, Papka has held the position of Research Scientist at Argonne National Laboratory contributing to research in high performance computing, scientific visualization of large and complex datasets, information visualization related to high-performance computing, virtual reality for science and the integration of edge services. 
Since 2006, he’s served as Deputy Associate Laboratory Director, Computing, Environment and Life Sciences (CELS) Directorate, providing strategic planning and management of the Laboratory Directorate comprising Biosciences, Computational Science, Data Science and Learning, Environmental Sciences, Mathematics and Computer Science, a joint institute with Northwestern University, and the Argonne Leadership Computing Facility. In his role, Papka aids in executing a research program that aims to couple computing-related activities with several science domains whose future is tied to progress in computing.
 For the last 12 years, Papka has served and continues to serve as Division Director at Argonne Leadership Computing Facility (ALCF). He is responsible for planning and executing facility upgrades and integrated data strategies, risk assessment and management, operations reporting and reviews, staff promotion and retention, and safety. The ALCF is home to some of the world&rsquo;s fastest supercomputers: Theta; a Cray XC40 supercomputer based on Intel’s Xeon Phi KNL processor and NVIDIA DGX-3, Polaris; a HPE Apollo 6500 Gen10+ based on AMD Milan processor and NVIDIA A100s, and Aurora; a Intel CPU/GPU system currently being installed that support scientific and engineering open science research. 
We welcome Mike&rsquo;s return to UIC and EVL! He undoubtedly will be a valued asset to our research and educational enterprise.

Alum Michael E. Papka Joins EVL Faculty

UIC Department of Computer Science 
Doctoral Admissions Webinar 
November 2nd 2022 @7 am-8:30am Chicago time (CDT) 
Webinar: <a href="https://tinyurl.com/uicwebinar2022">https://tinyurl.com/uicwebinar2022</a> 
See what the Department of Computer Science at the University of Illinois Chicago has to offer! Located in the beautiful city of Chicago, UIC is among the most culturally diverse universities in the United States, and home to one of the most innovative computer science departments in the United States. UIC is Chicago&rsquo;s top public research university, and was recently ranked in the top 20 of the best 100 universities worldwide that are under 50 years of age. Doctoral students in their first five years of enrollment are guaranteed a fully funded appointment that includes a tuition waiver, and a stipend. Students are also eligible for summer support. Our students get jobs in academia, industry, and entertainment upon graduation, including positions at top research labs and companies such as Google, Microsoft, Apple, Disney, and Amazon. 
During this webinar, UIC Computer Science faculty will talk about our research agenda, application fee waivers, the application process, application expectations in terms of GPA/GRE etc, and will take questions. A recording of the webinar will be made available on our Graduate Admissions page. 
Apply: <a href="https://cs.uic.edu/graduate/admissions/">https://cs.uic.edu/graduate/admissions/</a> 
Deadline: December 15th 2022 
See our flyer at: <a href="https://tinyurl.com/uic2022flyer">https://tinyurl.com/uic2022flyer</a> 
 <iframe width="420" height="315"src="https://www.youtube.com/embed/eNoBbCm1skM">
</iframe>

Interested in pursuing a PhD in Computer Science in the United States?

MS Student Undergraduate Researcher 2023

Sreten Kljaic

This project was performed during the COVID-19 pandemic. It presents a visual computing framework (MOTIV) for analyzing moral rhetoric on social media around controversial topics such as Stay-at-Home orders. Using Moral Foundation Theory, it proposes a methodology for deconstructing and visualizing the when, where, and who behind each of these moral dimensions as expressed in microblog data. The project was developed in collaboration with experts from language processing, communications, and causal inference. Our approach integrates microblog data with multiple sources of geospatial and temporal data, and leverages unsupervised machine learning (generalized additive models) to support collaborative hypothesis discovery and testing. 
Abstract: We present a visual computing framework for analyzing moral rhetoric on social media around controversial topics. Using Moral Foundation Theory, we propose a methodology for deconstructing and visualizing the when, where, and who behind each of these moral dimensions as expressed in microblog data. We characterize the design of this framework, developed in collaboration with experts from language processing, communications, and causal inference. Our approach integrates microblog data with multiple sources of geospatial and temporal data, and leverages unsupervised machine learning (generalized additive models) to support collaborative hypothesis discovery and testing. We implement this approach in a system named MOTIV. We illustrate this approach on two problems, one related to Stay-at-home policies during the COVID-19 pandemic, and the other related to the Black Lives Matter movement. Through detailed case studies and discussions with collaborators, we identify several insights discovered regarding the different drivers of moral sentiment in social media. Our results indicate that this visual approach supports rapid, collaborative hypothesis testing, and can help give insights into the underlying moral values behind controversial political issues. 
Supplemental Material: <a href="https://osf.io/ygkzn/?view_only=6310c0886938415391d977b8aae8b749">https://osf.io/ygkzn/?view_only=6310c0886938415391d977b8aae8b749</a>

MOTIV visualization of moral framing on social media in 2020.

MOTIV: Visual Exploration of Moral Framing in Social Media

September 2023 marks the 50th anniversary of the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC). EVL&rsquo;s introduction of the CAVE Automatic Virtual Environment in 1992, the first widely replicated, projection-based, walk-in, virtual-reality (VR) system in the world, put EVL at the forefront of collaborative, immersive data exploration and analytics. However, the journey did not begin then. Since its founding in 1973, EVL has been developing tools and techniques for real-time, interactive visualizations - pillars of VR. But EVL&rsquo;s culture is also relevant to its successes, as it has always been an interdisciplinary lab that fosters teamwork, where each person&rsquo;s expertise contributes to the development of the necessary tools, hardware, system software, applications, and human interface models to solve problems. Over the years, as multidisciplinary collaborations evolved and advanced scientific instruments and data resources were distributed globally, the need to access and share data and visualizations while working with colleagues, local and remote, synchronous and asynchronous, also became important fields of study. This paper is a retrospective of EVL&rsquo;s past 50 years that surveys the many networked, immersive, collaborative visualization and VR systems and applications it developed and deployed, as well as lessons learned and future plans. 
Article appears in the journal PRESENCE: Virtual and Augmented Reality, in the Special Issue on Immersive Visualization Laboratories - Past, Present and Future, edited by S. Su and W. Sherman, <a href="https://direct.mit.edu/pvar/issue/volume/33">https://direct.mit.edu/pvar/issue/volume/33</a>. 
Article copyrighted by Massachusetts Institute of Technology (MIT). 
DOI: <a href="https://doi.org/10.1162/pres_a_00421">https://doi.org/10.1162/pres_a_00421</a>

EVL faculty, students, and collaborators 1975.

Electronic Visualization Laboratory&rsquo;s 50th Anniversary Retrospective: Look to the Future, Build on the Past

The SAGE3 development team hosts a one-hour Zoom event to introduce the SAGE3 AI-enhanced collaboration platform developed jointly by the University of Hawaii, the University of Illinois at Chicago, and Virginia Tech January 21-23, 2025. 
Funded by the U.S. National Science Foundation (NSF), SAGE3 is available to a wide range of scientific research & education communities at no charge. SAGE3 enables researchers and educators to seamlessly work together with data and visualizations, across devices - from laptops to large display walls. ﻿﻿Unleash spaces to think, collaborate, analyze, and make decisions with greater speed and confidence. See <a href="https://go.hawaii.edu/a5o">meeting details and registration information</a>. 
To learn more, please register for one of the upcoming 1-hour informational sessions. We will introduce newcomers to SAGE3 and update long-time users about its latest improvements.

SAGE3 AI-Enhanced Collaboration Platform Introduced

September 2023 marked the 50th anniversary of the Electronic Visualization Laboratory (EVL). This paper summarizes EVL&rsquo;s efforts in Visual Data Science, with a focus on the many networked, immersive, collaborative visualization and virtual reality (VR) systems and applications the Lab has developed and deployed, as
well as lessons learned and future plans. 
Index terms: Visualization, Virtual Reality, Visual Data Science, Collaboration

An EVL CAD Rendering of the DOCC Space

Immersive Analytics at the Electronic Visualization Laboratory

The Data Observation and Computation Collaboratory (DOCC, &rsquo;dock&rsquo;) is a visual data science instrument for University of Illinois Chicago (UIC) science and engineering research and research training. Many UIC researchers are part of multi-disciplinary teams investigating and contributing to solutions for societal issues, in such fields as biology, cybersecurity, healthcare, manufacturing, urban sustainability, and cyber-physical systems (e.g., autonomous cars). DOCC is a digital &rsquo;Project Room&rsquo; that enables domain scientists and computer scientists to collaborate on real-world problems by providing them with high-performance computing, high-resolution displays, and sophisticated Artificial Intelligence (AI), data analytic, data simulation, image processing, visualization, and virtual-reality software and services. 
Societal challenges can only be addressed by research teams, with each contributing expertise in complementary areas. DOCC&rsquo;s Intellectual Merit is two-fold: it provides onsite computing with adequate support, and it provides collaboration software services. With respect to the former, DOCC is a configurable computing environment that prioritizes real-time, interactive, collaborative, data-intensive applications, but can be scheduled off-hours for batch-mode processing. Users can execute codes faster, apply more sophisticated analytics to large-scale problems, gain greater insights, and open new avenues of research. With respect to the latter, DOCC&rsquo;s collaboration software lets researchers either sit around one of DOCC&rsquo;s large tables and display walls or video-teleconference in while simultaneously sharing and interacting with their data on the large displays – which remote viewers can access online. 
To encourage adoption and ensure Broader Impact, DOCC has a visual front end, enabling users to easily access familiar collaboration tools, scientific notebooks, AI workflows, and popular visualization software. Alternatively, users can add new AI models or create them from scratch. And because DOCC is connected to the UIC research network, data and/or computer codes can be ported to remote computer and cloud resources located on campus, regionally, nationally, or internationally. DOCC personnel can assist new users make their applications more robust, dynamic and configurable. DOCC can transform scientific research, data exploration and discovery, recruit and retain new faculty, make cyberinfrastructure more accessible to students, and help educate the next-generation workforce. UIC is a Minority Serving Institution, a Hispanic Serving Institution and an Asian American and Native American Pacific Islander Serving Institution. 
DOCC and its associated data will remain active for at least three years after the completion of this project, as demand dictates. Software will be stored and archived via EVL&rsquo;s storage servers, as well as code repositories.

Natural Language Processing

Bibliography

pubs (13)

Related Topics

people (6)

events (12)

research (5)

Contact