EDITOR&rsquo;S NOTE: The following news release was posted to UIC NEWS on December 17, 2014. It describes NSF award #IIS-1445796 for &ldquo;Articulate: Augmenting Data Visualization With Natural Language Interaction,&rdquo; for a one year period (August 2014 - July 2015) in the amount of $300,000. This is a collaborative project between UIC and University of Hawaii at Manoa (UHM). Those involved are UIC computer science faculty members Barbara Di Eugenio (PI), Andy Johnson (co-PI) and Adjunct CS Professor Leland Wilkinson (co-PI), and from UHM, computer science faculty member Jason Leigh (PI). 

&lsquo;Draw me a picture,&rsquo; say scientists; computer may respond 
Jeanne Galatzer-Levy, UIC NEWS 
December 17, 2014 

Like the rest of us, scientists wish they could just ask a computer a question and have it respond with an answer presented in an easy-to-understand picture. Today’s visualization tools can translate huge raw data sets into graphs and maps - but most scientists lack the time and training to use the tools effectively. 

The National Science Foundation has awarded a $300,000 grant to the University of Illinois at Chicago and the University of Hawaii to take the first steps towards developing a computer that can take data and produce meaningful visualizations based on natural language requests, accompanied by common gestures like pointing. 

With nearly a third of the human brain devoted to processing visual stimuli, vision has always been our dominant way of acquiring information. Visualization is the most effective means of converting raw data into insight that can support scientific discovery, says Andrew Johnson, director of research at UIC&rsquo;s Electronic Visualization Laboratory.

&ldquo;Today, with big data, you really need to be using visualizations to help you figure out what it is you’re looking at,&rdquo; said Johnson, who is a co-principal investigator on the NSF grant. &ldquo;Visualization should be interactive; a dynamic process. We want scientists to be able get ideas out there quickly.&rdquo; 

&ldquo;The object is to make it more like a conversation with a person in the room,&rdquo; said Barbara Di Eugenio, UIC professor of computer science and principal investigator on the grant. &ldquo;&lsquo;Can we recolor that? What if we moved it this way, or inverted that axis?&lsquo;&rdquo; 

&ldquo;We also don’t want to require explicit questions,&rdquo; she said. &ldquo;The goal is for the computer to be able to interpret even indirect questions, like &lsquo;It would be better to show salinity only at 10 meters&rsquo;, or even statements that hint at something, and put together the visualization.&rdquo; 

<a href="http://news.uic.edu/draw-me-a-picture-say-scientists-computer-may-respond">Read the article</a>. 

NOTE: This work is based on UIC computer science PhD student Yiwen Sun&rsquo;s dissertation, <a href="http://www.evl.uic.edu/core.php?mod=4&type=3&indi=457">&ldquo;Articulate: Creating Meaningful Visualizations from Natural Language&rdquo;</a>. 

<a href="http://youtu.be/V1CFRv4Q8bk?list=PL4071582316A3714C">View a video describing this earlier work.</a>

Jillian Aurisano, Ph.D. student in computer science, works with the Articulate prototype in EVL cyber-commons classroom.

&lsquo;Draw me a picture,&rsquo; say scientists; computer may respond

EVL PhD candidate, Jillian Aurisano present her research in data visualization and exploration using a conversational interface. 
Visual data exploration poses challenges for &rsquo;InfoVis Novices&rsquo;. A &rsquo;conversational interface&rsquo; which would enable users to generate and interact with visualizations through natural language and gestures, while maintaining a history of the data exploration, has the potential to ameliorate many of these challenges. We performed an exploratory, observational study designed to examine the role of such a conversational interface in visual data exploration. We simulated a conversational interface, using a remote human mediator, with multiple cycles of visualization construction. We believe analysis of this data will yield concrete design goals for conversational interfaces in information visualization. Research results and future work will be presented.

&ldquo;Show Me Data.&rdquo; Observational Study of a Conversational Interface in Visual Data Exploration

A natural language interface to visual data exploration would allow a user to directly specify questions through speech, allowing the user to focus on higher-order tasks, such as hypothesis generation and question formulation. 
However, visual data exploration involves repeated cycles of visualization construction and interaction, as well as reasoning across many visualizations generated over the course of an exploratory session. A &rsquo;conversational interface&rsquo;, which maintains a dialog with the user through natural language and gestures, could support these complex tasks. 
We conducted an observational, exploratory study to observe the interaction between a subject and a remote data analysis expert (DAE) who assists the subject in an exploratory data analysis task.

Research subject involved in an exploratory data analysis task using the EVL Cyber-Commons tiled display wall.

&ldquo;Show me data&rdquo; Observational study of a conversational interface in visual data exploration

Newcity magazine is Chicago&rsquo;s only locally owned and operated cultural weekly, and has been covering Chicago&rsquo;s local scene for more than 25 years. It&rsquo;s current &ldquo;Best of Chicago&rdquo; issue just came out (Volume 31, No. 1359). The section on City Life lists many best things about Chicago, including &ldquo;Best place to get your geek on.&rdquo; Um, that would be the Electronic Visualization Laboratory at the University of Illinois at Chicago! The article is reproduced below. 
* * * * * * * * * * 
Best place to get your geek on 
The Electronic Visualization Laboratory (EVL) at UIC 
Cutting-edge research meets a massive wall made out of small screens that would make any hardcore videogame fan scream and a literal cave that opens up a highly immersive, surround-screen, surround-sound, virtual reality environment at the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. The lab brings together academics, designers, engineers, artists and mad scientists using technologies that bend reality off into unimaginable directions. EVL is the place where visualization, virtual reality and real-time interactive computer graphics are born. As if that were not enough, it was here that the computer graphics for the first &ldquo;Star Wars&rdquo; film were created in 1976. How cool is that? 
UIC College of Engineering, Department of Computer Science, 842 W. Taylor 
Best of Chicago 2016

Best of Chicago 2016

Newcity magazine rates UIC/EVL as &ldquo;Best place to get your geek on&rdquo;

InfoVis novices struggle with visualization construction. Even with the aid of visualization software, such users may face challenges when translating their questions into appropriate visual encodings, or interactively refining the representation to achieve a desired result. 
A &lsquo;conversational interface&rsquo; which maintains a dialog with the user through natural language and gestures, could allow users to engage in repeated cycles of visualization generation and modification, asking questions directly through speech. 
In this poster we present a prototype conversational visual data analysis system. Our prototype was developed from a corpus consisting in 15-subjects engaging in exploratory data visualization with a simulated conversational interface. It features 1) speech to visualization pipeline, 2) classification system to divide utterances into major types, 3) history manager and knowledge-base.

Articulate2: Toward a Conversational Interface for Visual Data Exploration

The goal of our research is to support full-fledged dialogue between a user and a system that transforms the user queries into visualizations. So far, we have collected a corpus where users explore data via visualizations; we have annotated the corpus for user intentions; and we have developed the core NL-to-visualization pipeline.

User examining crime data on tiled display.

Towards a Dialogue System that Supports Rich Visualizations of Data

Just as NSF cyberinfrastructure produces greater volumes and varieties of data from data storage systems, online instrumentation and major computational resources like XSEDE and Blue Waters, advanced visualization instruments serve as the &ldquo;lenses&rdquo; of these data generators (similar to the telescopes and microscopes of yore), enabling researchers to peer into cyberspace to manage, access and analyze big data. 
Continuum is a cyberinfrastructure instrument that unifies high-resolution computer-enhanced group collaboration workspaces with arrays of ambient sensors, enabling users to collaborate with local and remote colleagues and their data more effectively by giving the room itself the ability to anticipate their needs, improve their interaction, and focus their time on discovery rather than the technology. Continuum is a digital assistant, benefiting global scientific collaboratories and providing a foundation for new computer science research. War rooms benefit collaboration because they support continuous interaction and awareness of team members, enable persistent information sharing, and encourage mobility among team members and information. A computer-enhanced war room, such as Continuum, which (1) connects distributed teams with their data, whether stored in the cloud or streamed from major computational resources over high-speed networks, (2) supports persistent digital artifacts, (3) enables ultra-high-resolution 2D and 3D stereoscopic digital media to be simultaneously viewed, and (4) supports ubiquitous and intuitive interaction, creates a powerful and easy-to-use information-rich environment in support of scientific discovery. 
Continuum is the culmination of the University of Illinois at Chicago’s (UIC) 20+ years experience developing interactive instruments, from the original CAVE in 1992, to the ultra-high-resolution LambdaVision tiled LCD wall in 2004, and the hybrid-reality CAVE2 in 2012. Each new generation of visualization instrumentation provided scientific communities with more advanced features Continuum development focuses on three areas: (1) transforming passive display walls that respond to user commands into attentive display walls that monitor the users in a room and anticipate their needs to improve their experience; (2) providing an alternative approach to constructing high-resolution visualization display walls using thin-border 4K passive stereo OLED displays that have impressive brightness, contrast, uniformity, off-axis viewing, and low reflectivity in brightly lit rooms; and (3) utilizing cloud computing to prototype future Continuum spaces that don’t require local high-end compute and display resources. Twelve funded research projects from local institutions, in Astronomy, Astrophysics, Biology,
Combustion, Geophysics, Neuroscience, Pathology, Physics, Psychiatry, and High Performance Computing, are poised to use Continuum for their data visualization needs. Continuum will open up new opportunities in computer science research at the intersection of large-scale data visualization, human-computer interaction, and high-speed networking. Continuum will also directly support 10 classes taught in the UIC Computer Science, Art and Design, and Biomedical Science departments.

Artist's rendering of Continuum space.

MRI - Development of Continuum: A Virtualized Attentive Environment for Amplified Collaboration

Research Experience Undergraduate, BS 2018

Joe Borowicz

Research Experience Undergraduate

Kenan Arica

Undergraduate Researcher (REU), BS 2018

Kayla Condrella

Pi: Barbara Di Eugenio; Co-I: Debaleena Chattopadhyay 
A growing body of research has found that patients who actively manage their health have better outcomes, and patient engagement has become a central tenet in recent healthcare approaches. However, it has also been shown that many patients lack knowledge of their disease and medications, particularly as concerns chronic health conditions, whose exacerbation causes about two thirds of hospitalizations every year in the USA. Discharge instructions after hospitalization are a critical tool to engage patients in their self-care, but are often unsuccessful. Our hypothesis is that, to engage patients at discharge, a personalized hospitalization summary should be provided, that integrates physician and nurse documentation, and is informed by the patient's level of engagement in self-care and their socio-cultural perspective. 
Our transformative pilot work on physician and nurse documentation has identified an additional obstacle to the effective flow of information from professionals to patients: vocabulary differences between physicians and nurses. By leveraging the Unified Medical Language System Metathesaurus, we found no common synonyms between physician and nurse documentations for 75% of patients. Further, 86% of terms patients use are simple compared to only 15% for physicians. The lack of common language among patients, physicians and nurses motivates including all perspectives to make instructions meaningful for patients. Our hypothesis is that a better informed patient population who understands the connections between their illness, hospital experience, and discharge instructions will be more empowered and more engaged in their chronic disease management. Our innovation is to automatically create an easy-to-understand, personalized hospitalization summary that integrates events described in the EHR; and that is responsive to the patient's needs and preferences, and their activation framework (PAF) level. We pursue four specific aims: 1. To investigate socio-cultural patterns of patient perceptions of illness via a socio linguistic analysis of patient interviews; 2. To integrate multi-faceted information about a patient, as provided by physicians and nurses; 3a. To develop MyPHA, a novel application program, which will automatically generate the personalized summaries and deliver them to the patient on a tablet. MyPHA will be informed by aims 1 and 2, and will be grounded in Natural Language Generation technology; 3b. To evaluate MyPHA with multiple cycles of formative evaluation with all stakeholders, and a summative evaluation with patients. 
Whereas our pilot work has focused on heart failure, the techniques we will develop can easily be adapted to the management of other chronic health conditions, in particular for patients with cancer and cancer survivors. In addition, cancer and heart failure can co-occur, especially in older patients, and several cancer treatments can lead to the development of heart failure. The software program MyPHA will be released ooen source to the research communitv to enable raoid customization to other chronic conditions. 
Awarded Amount: $1,495,213

NIH: R01: SCH: INT: MyPHA: Automatically generating personalized accounts of inpatient hospitalization

Ph.D. Student: Jillian Aurisano 
Date: Tuesday, September 17, 2019 
Time: 11:00 am 
Location: Room 3036, Engineering Research Facility 
Committee: 
Andrew Johnson (Chair) 
Debaleena Chattopadhyay 
Barbara Di Eugenio 
G. Elisabeta Marai 
Rick Stevens (Argonne National Laboratory, University of Chicago) 
Abstract: 
Analysis of large, complex datasets stands to benefit from environments that permit users to view and juxtapose many views of data. Large, high-resolution environments are capable of showing many related views of data, but interaction with these views poses significant challenges in visual and interaction design. In this talk, I will present work toward &ldquo;multi-view interactions&rdqo; that enable users to create, organize and act on many views at once, through multi-modal speech+gesture queries in large flexible canvas environments. The goal is the enable users to rapidly and efficiently generate sets of views in support of multi-view analysis tasks, organize these views to reflect changing analysis goals, and operate on sets of views collectively, rather than individually, to efficiently reach large portions of the &lsquo;data+attribute space&rsquo;. I will implement and evaluate this approach within two domains and two user communities: City of Chicago data for informed citizens and deep learning predictions data for precision medicine in cancer.

Preliminary Examination Announcement: &ldquo;Multi-view, multi-modal speech+gesture interaction in large display environments&rdquo;

PhD Student: Abhinav Kumar 
Date: September 27, 2019 
Time: 1:00 PM to 3:00 PM 
Location: Room 2068, Engineering Research Facility 
Committee: 
Barbara Di Eugenio (Chair and Advisor) 
Andrew Johnson 
Natalie Parde 
Cornelia Caragea 
Kallirroi Georgila 
Abstract: 
When exploring data visualizations, users that are novices to visualization, oftentimes struggle with translating their particular queries into an effective visual representation, which this research aims to address by automatically constructing and managing visualizations appropriately. I am particularly focusing on the language processing, which is challenging because spoken dialogue is oftentimes under-specified, leading to open interpretation and in our case potentially the wrong visualization. My main contributions are as follows. First, I am developing language processing in the multimodal setting. I worked with others in collecting and annotating a corpus of dialogue data, comprising of speech text and hand gestures, that came from data collected via human experiments. Second, I used a local context for building a module for interpreting the underlying intent of what the user says. Third, given the small size of our dataset, I augmented it using paraphrasing and showed that the intent interpretation module improved on the augmented data. As part of my proposed work, I will work on text segmentation to identify the boundaries of the local context that my current intent interpretation module operates on. I also plan to develop a model that captures references made through language and hand pointing gestures and determines the appropriate target visualizations for them.

Preliminary Examination Announcement: &ldquo;Towards a Natural Language Interface for Exploring Data Visualization&rdquo;

PhD Candidate

Abhinav Kumar

SAGE2 is a user-centered platform for small groups or large distributed teams to access digital media datasets from various sources and juxtapose, share and investigate content on tiled display walls. SAGE2 builds on SAGE (Scalable Adaptive Graphics Environment), developed with prior NSF funding. SAGE2 is having a transformative effect on data exploration, visualization and collaboration, making cyberinfrastructure more accessible to end-users, both in the laboratory and in the classroom.

SAGE2 Scalable Amplified Group Environment (poster 2020)

This paper describes results from an observational, exploratory study of visual data exploration in a large, multi-view, flexible canvas environment. Participants were provided with a set of data exploration sub-tasks associated with a local crime dataset and were instructed to pose questions to a remote mediator who would respond by generating and organizing visualizations on the large display. We observed that participants frequently posed requests to cast a net around one or several subsets of the data or a set of data attributes. They accomplished this directly and by utilizing existing views in unique ways, including by requesting to copy and pivot a group of views collectively and posing a set of parallel requests on target views expressed in one command. These observed actions depart from multi-view flexible canvas environments that typically provide interfaces in support of generating one view at a time or actions that operate on one view at a time. We describe how participants used these &ldquo;cast-a-net&rdquo; requests for tasks that spanned more than one view and describe design considerations for multi-view environments that would support the observed multi-view generation actions. 
Index Terms: Human-centered computing - Empirical studies in visualization 
This work was supported initially by NSF award IIS 1445751 and currently by NSF award CNS 1625941. 
<a href="https://doi.org/10.1111/cgf.13976">https://doi.org/10.1111/cgf.13976</a>

A participant generates 15 views of her data. In the final request, the participant references a set of three views and poses a request to copy and pivot these views collectively to four new subsets of the data. The final result is a grid of views.

Many At Once: Capturing Intentions to Create And Use Many Views At Once In Large Display Environments

PI: Andrew Johnson (UIC/EVL); co-PIs: Barbara DiEugenio (UIC/CS), Moira Zellner (Norheastern University) 
Collaborator: Jason Leigh (University of Hawaii Manoa) 
The Internet gives the public, including scientists, access to vast amounts of data. Visualization is a powerful tool to take that data and turn it into a form that is easier to understand, helping people make better decisions. However, creating good visualizations from data is not easy. The Articulate+ Project will work to generate new knowledge about how to empower people with information visualization, by using speech and gesture as inputs, in new ways. A goal is to enable users to benefit from imprecise statements about what they want, rather than needing to give specific commands telling the computer what to do. This project will make it easier for citizens to engage in conversations with the computer about economic, medical data, demographic, transportation data, climate, and sports data; to answers that are more meaningful, and so to democratize data literacy. 
This research will use speech, gesture, and log data as new input interaction modalities for visual analytics. It will develop new understandings of how the imprecise and vague nature of natural language queries and gestures, as contextualized by work on a specific visual analysis problem, can be modeled as an opportunity for an intelligent software system to learn more about the underlying intent of the users. This model of intent, in turn, will be used to provide contextualized visualizations, which are expected to help those users gain valuable insights from their data. The audio speech data will be used to computationally model overhearing, to help infer users&rsquo; current contextualized goals. The gesture data will be used to disambiguate expressions that refer to visual elements of a visualization (e.g., &ldquo;that pie chart about electricity consumption&rdquo;). The project will develop new understandings of how to combine the log of visualization state, the computational model of overhearing, and the gesture data in order to generate new visualizations in support of users&rsquo; work on tasks. Evaluations will be performed in both controlled laboratory and naturalistic study settings to determine whether effective semantic parsers can be derived for these specific visual analysis domains, and how this contextual interface affects users&rsquo; experience of visualization and discovery. The integrated annotated datasets from the studies will be made available to the research community satisfying a need for ecologically valid situated language interaction corpora.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Articulate+ - A Conversational Interface for Democratizing Visual Analysis

UIC Computer Science Professors Andy Johnson (PI) and Barbara Di Eugenio (co-PI) have been awarded a $333,000 NSF grant entitled &dquo;CHS: Small: Collaborative Research: Articulate+ - A Conversational Interface for Democratizing Visual Analysis&rdquo; project period October 2020 to September 2022. Dr. Moira Zellner, now of Northeastern University will also participate as co-PI on this grant. This project is part of a larger overall award of $500,000 that also includes Jason Leigh at University of Hawaii at Manoa. 
&ldquo;Articulate+&rdquo; continues EVL&rsquo; research in Natural Language Processing and Visualization also sponsored by NSF for the last 5+ years. See <a href="https://www.evl.uic.edu/entry.php?id=2522">Articulate+ - A Conversational Interface for Democratizing Visual Analysis</a> for more information on this project. 
NSF Award Number (FAIN): 2007257

NSF Grant &ldquo;CHS: Small: Collaborative Research: Articulate+&rdquo; Awarded

October 1 - - EVL Director of Research Andrew Johnson has been named Interim Director of EVL following Maxine Brown&rsquo;s retirement from UIC. Johnson now assumes the responsibility of heading EVL - overseeing EVL&rsquo;s overall mission in education, research, development, and technology transfer. 
For over 23 years, Johnson has been a valued EVL faculty member (Professor of Computer Science) and researcher who has focused upon interaction and collaboration using advanced visualization displays and the application of those displays to enhance discovery and learning. 
His areas of expertise include visualization, virtual and augmented reality, natural language processing, visual analytics, and high-performance networking. 
Over the years, Johnson has been instrumental in the development of EVL instrumentation and systems, including the Continuum smart environment, CAVE2&trade;, LambdaVision high-resolution tiled displays, LambdaTable tabletop touch system, GeoWall portable geoscience display environment, PARIS augmented reality and ImmersaDesk virtual reality systems. He has overseen the development of numerous visualization and virtual reality applications across diverse domain disciplines from science and engineering to the humanities and art. Johnson is a highly regarded mentor and educator to EVL&rsquo;s graduate students and undergraduates alike. 
Johnson retains his position as Director of Research while leading EVL&rsquo;s future initiatives and innovations as Interim Director.

Natural Language Processing

Bibliography

pubs (5)

Related Topics

people (4)

events (7)

research (3)

Contact