This research explores a novel approach to the design of Mixed Reality (MR) applications, which combines elements from markerless and location-based Augmented Reality in order to explore new types of interaction, leveraging a spatial representation of virtual content. In particular, a method is derived for developing both a mobile application, that can be used by people to see augmented content in the real world, and an editor application, which can be used to design and create MR experiences. Currently available mobile technologies needed to achieve these results are exploited, and the integration of sensors in order to improve tracking with a fully location-based approach, in which every content has a defined position in real space is explored. The advantages and the new interaction possibilities enabled by this method are weighed, in order to define the requirements for a custom MR authoring tool and its implementation. Particular emphasis is placed on the mapping between real-world and virtual coordinates and on the possibilities enabled by real-time editing, which allows the designer to preview the MR experience and also to interact with users.

Case Study - Overlaying Historical Photos

Merging Worlds: A Location-based Approach to Mixed Reality

UIC Computer Science and EVL exchange student from the Politecnico di Milano in Italy, Marco Cavallo presents his thesis research on Mixed Reality. 
&ldquo;Merging Worlds: A Location-based Approach to Mixed Reality&rdquo; presents a novel approach to the design of Mixed Reality (MR) applications, which combines elements from markerless and location-based Augmented Reality in order to explore new types of interaction, leveraging a spatial representation of virtual content. In particular, a method for developing both a mobile application, that can be used by people to see augmented content in the real world, and an editor application, which can be used to design and create MR experiences is proposed.

&ldquo;Merging Worlds&rdquo; Presentation

Navigation And Information Browsing Case Study

Marco Cavallo Thesis Presentation &ldquo;Merging Worlds: A Location-based Approach to Mixed Reality&rdquo;

EVL summer intern, Shreyas Mohan from Metea Valley High School, Aurora, IL presents the results of his efforts towards developing a robust music instruction application for use in virtual and augmented reality environments - the CAVE2 and Microsoft HoloLens systems. 
EVL Cyber-Commons, 2068 ERF 
Wednesday, August 10, 2016, 1:00 PM 
Shreyas developed an application for the CAVE2 and Microsoft HoloLens to investigate whether these technologies could be used effectively to teach music to a novice. Both applications taught the user &lsquo;Mary Had a Little Lamb&rsquo; on the viola, graphically representing the correct bow direction, tempo, fingering, and sheet music. Additional application features included adjustable tempo and metronome functionality. Combining graphical visual cues while seeing the physical instrument in a mixed reality environment proved to be a powerful method for independent music instruction.

Music Instructiton in VR/AR Presentation

HoloLens Interface

Music Instruction in a Virtual/Augmented Reality Environment (CAVE2&trade; and HoloLens)

This project&rsquo;s purpose was to investigate the ability of virtual reality to teach music. An application was developed for the CAVE2&trade; and Microsoft HoloLens, and their ability to teach music effectively was analyzed. Both the applications teach the user how to play &lsquo;Mary Had a Little Lamb&rsquo; on the viola, and each virtual reality platform had it&rsquo;s own benefits and drawbacks. An user interface was created for both platforms that shows the user the correct bow direction, tempo, fingering, and sheet music. Additional features included adjustable tempo and a metronome that can be turned on and off. The application was helpful since the user could get visual cues from the graphics while still keeping an eye on his/her instrument. Future goals for
the application include songs that can be uploaded by the user, microphone input, more instruments, and better graphics. The results of this research show that music instruction is a viable possibility in a virtual reality environment and will spark developers and musicians to work together to further explore this field.

Trackers placed on viola for CAVE2 version of the music instruction application.

Music Instruction in a Virtual/Augmented Reality Environment Using the CAVE2 and Microsoft HoloLens

Newcity magazine is Chicago&rsquo;s only locally owned and operated cultural weekly, and has been covering Chicago&rsquo;s local scene for more than 25 years. It&rsquo;s current &ldquo;Best of Chicago&rdquo; issue just came out (Volume 31, No. 1359). The section on City Life lists many best things about Chicago, including &ldquo;Best place to get your geek on.&rdquo; Um, that would be the Electronic Visualization Laboratory at the University of Illinois at Chicago! The article is reproduced below. 
* * * * * * * * * * 
Best place to get your geek on 
The Electronic Visualization Laboratory (EVL) at UIC 
Cutting-edge research meets a massive wall made out of small screens that would make any hardcore videogame fan scream and a literal cave that opens up a highly immersive, surround-screen, surround-sound, virtual reality environment at the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. The lab brings together academics, designers, engineers, artists and mad scientists using technologies that bend reality off into unimaginable directions. EVL is the place where visualization, virtual reality and real-time interactive computer graphics are born. As if that were not enough, it was here that the computer graphics for the first &ldquo;Star Wars&rdquo; film were created in 1976. How cool is that? 
UIC College of Engineering, Department of Computer Science, 842 W. Taylor 
Best of Chicago 2016

Best of Chicago 2016

Newcity magazine rates UIC/EVL as &ldquo;Best place to get your geek on&rdquo;

Node Kara is an audiovisual mixed reality installation by the artist Anıl Camci. It offers a body-based interaction using a 3D imaging of the exhibition space. With a blurred audiovisual scene in its steady-state, it invites visitors to interact with the multiplicity of layers it holds within. By moving in the exhibition space, the visitors can deblur both the sound and the visuals as they get immersed in the piece. Rather than a detection of discrete events or gestures, Node Kara relies on fluid interactions with continuous visual layers and stochastic audio. This way, the piece reacts to each visitor in a unique way. A video preview of Node Kara can be seen at <a href="https://vimeo.com/anilcamci/nodekara">https://vimeo.com/anilcamci/nodekara</a> (password: nodekara). 
Index Terms: H.5.1 [Information interfaces and presentation (e.g., HCI)]: Multimedia Information Systems-Artificial, augmented, and virtual realities; H.5.2 [Information Interfaces and Presentation]: User Interfaces-Auditory (non-speech) feedback.

Four layers of Node Kara superimposed onto each other with a representation of the visitor’s depth image highlighted.

Node Kara: An Audiovisual Mixed Reality Installation

October 2016 - Augmented reality (AR) is being used in a very novel and compelling way to engage Chicago residents and tourists into learning more about the city&rsquo;s history. John Russick of the Chicago History Museum teamed with Geoffrey Alan Rhodes at the School of the Art Institute (SAIC) to create Chicago 0,0 Riverwalk, a mobile AR application that superimposes historical imagery onto matching views of Chicago&rsquo;s downtown riverwalk. The first project focuses on a single block between N. LaSalle and Clark Streets, the site of the Eastland Disaster in 1915, in which 844 people lost their lives on a pleasure boat when it rolled over in the Chicago River. On October 19, 2016, journalist Jay Shefsky told the story of how AR to being used to chronicle the tragedy of the <a href="http://chicagotonight.wttw.com/2016/10/19/augmented-reality-app-offers-new-look-eastland-disaster">S.S. Eastland on an episode of WTTW Chicago Tonight television</a>. 
During the development of this application, UIC Computer Science (CS) professor Angus Forbes and his MS student Marco Cavallo, both affiliated with the Electronic Visualization Laboratory (EVL), worked with the SAIC faculty and students on technical considerations that have since been incorporated into the second version of the application software. EVL&rsquo;s expertise enabled better tracking and feature detection of real-world objects; better registration and orientation of the real-world environment with historical photographs; the use of a smart, intuitive interface to help users navigate in the real world and locate historical images; and, better image contrast, even on overcast days with adverse lighting conditions. Together, these technical efforts have contributed to better user experiences. 
In the WTTW story, John Russick of the Chicago History Museum notes the value of this AR application: &ldquo;One of the biggest challenges in communicating history is finding human emotional connections to very old stories … that draw us in a deeper, more meaningful way.&rdquo; Rather than passively view photographs or documentary films, AR enables users to experience history firsthand, in its relevant surroundings. 
<a href="http://chicago00.org/">Download more information and the Chicago 0,0 app for smartphones or tablets</a>. 
Marco Cavallo (UIC), Geoffrey Alan Rhodes (SAIC) and Angus Forbes (UIC) co-authored a poster paper <a href="https://www.evl.uic.edu/entry.php?id=2255">&ldquo;Riverwalk: Incorporating Historical Photographs in Public Outdoor Augmented Reality Experiences,&rdquo;</a> that was presented at the 15th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), September 15-23, 2016, in Merida, Mexico.

A sample photo from the historical archive superimposed on the live camera stream. In order to create an appealing AR effect, the photo needs to appear aligned with specific environmental features.

Augmented Reality App Offers New Look at Eastland Disaster

Undergraduate Researcher (REU), BS 2018

Kayla Condrella

OCTOBER 18, 2018 - &ldquo;Herstory: Digital Innovations Symposium and Book Signing,&rdquo; celebrates the book &ldquo;New Media Futures (NMF): The Rise of Women in the Digital Arts.&rdquo; The Symposium will take place at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign (UIUC), 1205 West Clark, Room 1030, Urbana, IL, on October 26, 2018, from 8:30am - 2:00pm. 
The NMF chronicles 22 women who went to school or worked at UIUC, University of Illinois at Chicago (UIC), and the School of the Art Institute of Chicago, in the 1980s and beyond, during a pivotal time for the digital arts media in the Midwest. Nine of the 22 women profiled in the book are PhD/MS/MFA/BFA graduates and/or employees of the UIC Electronic Visualization Laboratory (EVL), which, at the time, was a formal collaboration between the UIC Computer Science department and UIC School of Art and Design. 
Thirteen of the book&rsquo;s contributors will attend the Symposium - including the woman who designed the first graphical web browser, the first female gaming company founder, and the woman who helped build the first room-sized virtual reality room environment - and will give TED-inspired Lightning Talks, and meet with attendees. They will discuss their passions, digital innovations and their herstories - both formally in short presentations and informally in small group breakouts - to inspire and mentor a young and diverse audience. Seven of the speakers have an affiliation with EVL: Maxine Brown, Carolina Cruz-Neira, Margaret Dolinsky, Copper Giloth, Dana Plepys, Mary Rasmussen and Barbara Sykes. 
For more information about the Symposium, see the flyer below and <a href="http://avl.ncsa.illinois.edu/events/herstory-digital-innovations-symposium">go to the website.</a> 
<a href="https://www.evl.uic.edu/entry.php?id=2369">More information about the NMF book.</a>

UIC Electronic Visualization Laboratory Graduates and Staff Participate in Symposium &ldquo;Herstory: Digital Innovations&rdquo;

Umwelt, the German word for environment, is a psychology term referring to the individual&rsquo;s mental image of the surrounding world. This subjective universe is shaped by an organism&rsquo;s unique perception of the environment around them and, therefore, differs from individual to individual. In other words, organisms experience the world through their senses, construct the mental model of their vicinity, and react upon it. Hence, this image never remains static - instead it is constantly refined, as a result of the dynamic nature of the world. Umwelt is closely related to the concept of Situation Awareness, that is the perception of environmental elements, comprehension of their nature, and projection of their near future status with respect to time and/or space. 
Sensory perception (exteroception) of the surrounding environment is, as a matter of fact, restricted to the limitations of biological senses. For example, although we consider humans with five fully developed senses (sight, hearing, touch, smell and taste) as able-bodied individuals, they are essentially blind to a multitude of information that constantly surrounds them. To put things into perspective, both human sight and hearing, can only perceive a fraction of information- part of the electromagnetic spectrum (light waves), and of the sound frequencies accordingly. However, as sensory systems differ among species, other creatures have an entirely different perceptual experience, as they are able to perceive a distinct spectrum of information. 
Despite the fact that our limited human senses restrict our Umwelt and, therefore, our Situation Awareness, Human Augmentics, referring to technologies that expand the capabilities and characteristics of humans, allows us to peek into an invisible &dlquo;other&drquo; world. Thus, some important questions arise: What would the consequences of an expanded perception of the
surrounding environment be, and how would that affect an individual&rsquo;s Situation Awareness? Since our Umwelt is limited by genetics, how would individuals attune with characteristics and capacities that lie beyond human range? Could Human Augmentics possibly expand an individual&rsquo;s Umwelt, leading to improved Situation Awareness? 
Jakob von Uexkull (1864 - 1944), a German biologist who worked on animal behavior studies, cybernetics of life, and muscular physiology, is attributed for coining the term Umwelt. Uexkull&rsquo;s Umwelt is the phenomenal world of a living animal, in which it exists and acts, and as a result is shaped according to the stimuli received from its senses. Due to the fact that individuals, even across the same species, sense the world differently, each animal creates its own unique Umwelt. 
Perception and, as a result, interaction with the world begins with the raw input of data from the environment through the sensory system. Anatomically, the sensory system is part of the nervous system and includes the sensory receptors that act as probes of the environment, parts of the brain that are used for sensory perception, and the neural pathways that connect the brain with the receptors. Notably, humans&rsquo; sensory system consists of five traditional senses: Sight, hearing, taste, smell and touch. The stimulation type and bandwidth of each sensory channel are limited to its characteristics, which in turn constrain the size of the Umwelt.
However not all living organisms possess the same qualitative or quantitative senses. Depending on the environment in which they have evolved, their sensory system may be entirely different than ours; some have more while others have less sensitive senses. Others have quite different ones. Bats and dolphins, for instance, have the ability to calculate orientation and position of other animals and objects through reflected beams of sound, an ability known as echolocation. Sharks and rays, on the other hand, can detect changes in the electric field of their vicinity, an ability known as electroreception. 
Situation Awareness 
It is a well-known fact that humans are capable of sensing only a small fraction of the data that constantly surrounds us. Magnetic fields, infrared light, ultrasound, cosmic rays, radiation are just a few of the things that are completely invisible to us. However, recent technological advances in specialized sensors, enable the detection of some of this information. For instance, geiger counters are used by hazmat personnel to assess radiation levels in nuclear accidents. These technologies provide feedback using one of our existing senses, usually visual or audio, in order to help workers safely perform daily tasks. 
Situation Awareness (SA) is the perception of environmental elements, comprehension of their nature, and projection of their near future status with respect to time and/or space. In other words, it is the awareness of the immediate vicinity, comprehension of the current state of the environment and ability to predict what may happen. Inputs from the sensory system are used by the individual in order to create and update their Situation Awareness state, and as a result, decisions depend on the accuracy of the perceived model. Situation Awareness is tightly associated with experience; somebody less experienced may miss information that a more experienced user will not. To put things into perspective, the hazmat personnel&rsquo;s (that we discussed above) Situation Awareness includes the geiger counter readings, comprehension of their meaning and ability to predict what will happen if they raise above a certain level. 
Despite the fact that sensory systems have been honed by millions of years of evolution, they are very limited to a multitude of data. We are blind to a plethora of information that exists around us, and fortunately most of it is harmless. We depend on our sensory system to interpret the world, and we use our Situation Awareness to comprehend the environment around us and act upon it. As a result, lack of information for a certain situation may lead to misjudgements and accidents. In fact 76% of aviation errors, happen because the pilot did not perceive one piece of the airplane&rsquo;s crucial information. To that end, another important question arises: Given that today&rsquo;s technology can sense some of this invisible information, can we expand the Umwelt and improve Situation Awareness? 
Research Questions 
Genetic defects, diseases or accidents can lead to sense disabilities or complete sense lose, which has been observed to both animals and humans. To that end, individuals learn over time to use the remaining of their senses in order to overcome a disability. Recent scientific discoveries have shown that while this new ability results in anatomical changes of the brain, the responsible regions remain the same. In other words, even though the sensory input channel has changed, the brain will still process the information in the same region|a phenomenon known as Sensory Substitution. 
As a result, Sensory Substitution devices use one or more of the remaining senses to increase an individual&rsquo;s Situation Awareness through a different sensory pathway. More specifically, tactile displays are able to translate external environmental information onto one&rsquo;s skin, through the use of haptics. While these systems are primarily designed for people with disabilities, their concepts can be extended to other sensory deprived individuals, who find themselves in no or limited visibility environments due to external, environmental reasons. For example, firefighters operating within smoky, low visibility hazardous environments or soldiers fighting in pitch black darkness, are more likely to have no Situation Awareness of their surrounding environment. 
The ever-dynamic nature of the world around us, however, makes navigating with a wearable tactile display, a quite challenging task. From feeling the vibrations onto one&rsquo;s skin, comprehending their meaning and correctly acting upon them, to getting inundated with tactile sensory overload, tactile displays have great potential, when communicating information in an eloquent
way. 
In this dissertation we will explore different environment-to-vibration mappings while assessing one&rsquo;s Situation Awareness. Furthermore, we seek to understand whether one's experience with a tactile display would have an effect on their Situation Awareness. Finally, we hypothesize that a one-size-fits-all wearable tactile display would not perform equally well in all environmental scenarios and therefore we are going to explore how different vibration configurations, patterns and environment-to-vibration mappings would affect one&rsquo;s navigation. All of the above are distilled into the following research questions: 
- Could tactile displays be used to increase Situation Awareness of visually deprived individuals? 
- To what extent would Situation Awareness be associated with experience when using a tactile display?

Positioning of Sensor Modules and Controller Box

Supporting Navigation with a Torso Wearable Tactile Display

JANUARY 24, 2019 - Daria Tsoupikova, associate professor in the UIC School of Design and an affiliated faculty member of the UIC Electronic Visualization Laboratory (EVL) which is part of the Computer Science (CS) department, recently received a $150,000 grant from the University of Illinois (UI) System&rsquo;s Presidential Initiative to Celebrate the Impact of the Arts and the Humanities. Given how important the arts and humanities are to education, research, scholarship, and public engagement, this newly-created initiative supports the visibility and excellence of the arts and humanities at the UI System&rsquo;s sister campuses. 
Tsoupikova&rsquo;s &ldquo;Transforming Storytelling: Multi-User Virtual Reality Theater for Collaborative Tele-Immersive Exploration&rdquo; is a collaboration with EVL Director of Research and CS associate professor Andrew Johnson, CS/EVL Senior Research Programmer Lance Long, CS/EVL PhD student Arthur Nishimoto, and Chicago-based theater director Jo Cattell, who recently received the Goodman Theatre&rsquo;s 2018 Michael Maggio Directing Fellowship. They will partner with the Goodman Theatre&rsquo;s Alice Rapoport Center for Education and Engagement, whose youth program works with Chicago&rsquo;s diverse demographics of middle-grade and teenage kids (ages 11-18). 
&ldquo;Transforming Storytelling&rdquo; will demonstrate how virtual-reality technology can be both an artistic medium to further immerse participants in live theater experiences and a storytelling medium. Project collaborators will develop multi-user theatrical performances that merge real and virtual worlds with the goal of encouraging stronger theater engagement with tech-savvy younger audiences. Chicago youth will attend performances at the Goodman Theatre and at UIC/EVL&rsquo;s CAVE2&trade;, allowing for immersive participation in the plot lines through avatars, hand-held devices and 3D platforms. 
While non-profit theater groups have great interest in incorporating new technologies in live performances, they don&rsquo;t necessarily have the financial support to do so. UIC&rsquo;s collaboration with the Goodman Theatre can demonstrate that live performance combined with virtual reality is a viable option for immersive theater. 
The UI System&rsquo;s Presidential Initiative selected 14 faculty projects from 53 proposals, which will share nearly $2-Million in funding over the next two years. Of the 14 proposals, all but one was allocated $100,000 or more and eight will receive $150,000 or more. By comparison, the average grant award in 2016 by the National Endowment for the Arts (NEA) was $25,000 and fewer than 40 NEA grants totaled more than $100,000, according to Grantmakers in the Arts. 
<a href="https://news.uillinois.edu/view/7815/738301">University of Illinois System News, January 24, 2019</a> 
<a href="https://cs.uic.edu/news-stories/uics-electronic-visiualization-lab-is-helping-reimagine-live-stage-performance/">UIC Computer Science Department News, April 25, 2019 
Chicago Tribune, January 25, 2019 
<a href="https://www.pressreader.com/usa/chicago-tribune/20190125/281638191417696">&ldquo;U. of I. grants $2M to bolster arts, humanities&rdquo;</a>

Young adults explore &ldquo;GenWorld,&rdquo; a virtual world previously developed by UIC CS/Design students Peter Hanula, Kyle Dalton, Morgan Osborn, and Christel Tiu as part of the Spring 2017 CS/Design Creative Coding class co-taught by Daria Tsoupikova

Tsoupikova&rsquo;s &ldquo;Transforming Storytelling: Multi-User Virtual Reality Theater for Collaborative Tele-Immersive Exploration&rdquo; Awarded

Thanks to technological advancement and lower priced head-mounted displays, Virtual Reality is becoming popular in the consumer market. One of the main aspects on which researchers are focusing on these days, besides increasing the resolution of the displays, is finding a way to interact in a way that is as similar as possible to the real world. In this document, we will first analyze some previous work done in the field. Then, we will show our implementation of a passive feedback system which uses real objects to augment virtual reality experiences. The system will be used on two different sets of real objects. The first is composed of the exact replica of the objects present in the virtual environment, the second is composed of objects with several levels of mismatched physical characteristics. The document will then explain in detail the integration of the system in VR applications using Unity. 
In the second part of this document, we will describe a user study to compare the execution of object manipulation tasks with the system that we developed to two other state-of-art input devices: Leap Motion and VR Controllers. We will analyze the factors of the task execution in virtual reality, such as the accuracy, velocity, time to execution, level of realism and immersion, and we will compare the results with the execution of the same task in the real world. Finally, we will give suggestions on how to enhance our system in future work.

User wearing trackers on the hands for object manipulation tasks.

Passive Haptic Feedback for Object Manipulation in Virtual Reality

Research Experience Undergraduate (REU), 2019

Kyra Jones

Research Experience Undergraduate (REU)

Szymon Popowski

Since 2017, Leonardo commenced a quarterly recognition of exceptional peer reviewers noted for their in-depth and deeply constructive feedback on papers under consideration for publication. Among those recognized in the August 2019 edition of <a href="https://www.leonardo.info/blog/2019/07/22/recognition-of-leonardos-outstanding-peer-reviewers?utm_source=LNN+subscribers&utm_campaign=054f45e4b7-EMAIL_CAMPAIGN_2019_07_31_05_49&utm_medium=email&utm_term=0_59c5038641-054f45e4b7-116710661&mc_cid=054f45e4b7&mc_eid=c32370193e">Leonardo 52:4</a> is Daria Tsoupikova, associate professor in the UIC School of Design and an affiliated faculty member of the Electronic Visualization Laboratory. 
Tsoupikova's research and artwork focus on the development of virtual reality (VR) art projects and networked multi-user exhibitions for VR projection systems, such as the Cave Automatic Virtual Environment (CAVE and CAVE2). Her work lies at the crossroads of artistic and technological innovation and investigates computational creativity to design real-life applications that advance healthcare, education and social change.

Daria Tsoupikova Recognized Among Leonardo's Outstanding Peer Reviewers

Transforming Storytelling: Multiuser Virtual Reality Theater for Collaborative Tele-Immersive Interaction (aka &ldquo;Hummignbird&rdquo;) is a collaborative project between the Electronic Visualization Laboratory (EVL) of the University of Illinois at Chicago (UIC) and the <a href="https://www.goodmantheatre.org/">Goodman Theatre</a>. Led by Daria Tsoupikova, professor of the UIC School of Design in conjunction with Goodman&rsquo;s Jo Cattell, &ldquo;Hummingbird&rdquo; is a multiuser theatrical performance that merges real and virtual worlds with the goal of encouraging stronger theatrical engagement with a tech-savvy younger generation audience. 
The project is being designed for the Goodman Theater New Stages, which showcases innovative and experimental theater works. Chicago youth will experience &ldquo;Hummingbird&rdquo; by attending tele-immersive real-time performances running concurrently at the Goodman Theatre&rsquo;s Alice Rapport Center and EVL&rsquo;s Continuum smart space. 
&ldquo;Hummingbird&rdquo; performers will take theatergoers on a tele-immersive journey donning virtual reality head-mounted displays and interacting with the virtual environment to tell a story via this new medium. 
Performances are anticipated for mid-2021 pending pandemic regulations and clearances.
See a video excerpt of <a href="https://www.youtube.com/watch?v=yorUSe_2-Kc&t=7s">&ldquo;Hummingbird&rdquo; on YouTube</a>.

Performers Interact with a &ldquo;Hummingbird&rdquo; Virtual Scene

Hummingbird: Collaborative Virtual Reality for Tele-Immersive Interactive Theater

October 1 - - EVL Director of Research Andrew Johnson has been named Interim Director of EVL following Maxine Brown&rsquo;s retirement from UIC. Johnson now assumes the responsibility of heading EVL - overseeing EVL&rsquo;s overall mission in education, research, development, and technology transfer. 
For over 23 years, Johnson has been a valued EVL faculty member (Professor of Computer Science) and researcher who has focused upon interaction and collaboration using advanced visualization displays and the application of those displays to enhance discovery and learning. 
His areas of expertise include visualization, virtual and augmented reality, natural language processing, visual analytics, and high-performance networking. 
Over the years, Johnson has been instrumental in the development of EVL instrumentation and systems, including the Continuum smart environment, CAVE2&trade;, LambdaVision high-resolution tiled displays, LambdaTable tabletop touch system, GeoWall portable geoscience display environment, PARIS augmented reality and ImmersaDesk virtual reality systems. He has overseen the development of numerous visualization and virtual reality applications across diverse domain disciplines from science and engineering to the humanities and art. Johnson is a highly regarded mentor and educator to EVL&rsquo;s graduate students and undergraduates alike. 
Johnson retains his position as Director of Research while leading EVL&rsquo;s future initiatives and innovations as Interim Director.

Andrew Johnson Named Interim Director, EVL

EVL hosts the pilot performances of &ldquo;HUMMINGBIRD&rdquo; the multiuser theatrical work that merges real and virtual worlds, engaging tech-savvy youth with a highly interactive virtual reality experience. This innovative project combines elements of traditional theater with virtual storytelling where participants collaborate to traverse virtual worlds and accomplish goals while being led by actors throughout their experience. 
EVL, the Department of Computer Science, and the College of Architecture, Design, and the Arts faculty and students in collaboration with award-winning Immersive Theatre Director Jo Cattell are organizing the pilot performance on June 18-20. This ambitious project involves a large team of faculty, graduate and undergraduate students, Goodman Theater Directors, actors, producers, a professional music composer, sound engineer, and many other contributors. This project is supported by the University of Illinois Presidential Award for Art and Humanities, the Office of the Vice Chancellor for Research Creative Activity Award, and the National Endowment for the Arts. 
The project team consists of School of Design professor, Daria Tsoupikova, award-winning theater director Jo Cattell, application developers and PhD candidates Arthur Nishimoto and Sai Priy Jyouthua, professor and EVL interim director Andrew Johnson, and project network engineering lead Lance Long. The theater team and actors include Denise Savas (stage manager), Devonte Washington (actor/assistant stage manager), Stephanie Shum (lead actor), Liviu Passare (videographer). 
Pilot performances are by invitation only. Stay tuned for the upcoming premiere this fall hosted by EVL and Goodman Theatre&rsquo;s Alice Rapport Center.

Pilot Performances of &ldquo;HUMMINGBIRD&rdquo; Collaborative Virtual Reality for Tele-Immersive Interactive Theater

Two major form factors for virtual reality are head-mounted displays and large display environments such as CAVE&trade; and the LCD-based successor CAVE2&trade;. Each of these has distinct advantages and limitations based on how they&rsquo;re used. This work explores preserving the high resolution and sense of presence of CAVE2 environments in full stereoscopic mode by using a see-through augmented reality HMD to expand the user&rsquo;s field of regard beyond the physical display walls. In our explorative study, we found that in a visual search task in a stereoscopic CAVE2, the addition of the HoloLens to expand the field of regard did not hinder the performance or accuracy of the participant, but promoted more physical navigation which in post-study interviews participants felt aided in their spatial awareness of the virtual environment. 
CCS CONCEPTS: Human-centered computing, Human computer interaction (HCI); Mixed / augmented reality; Virtual reality. 
KEYWORDS; Augmented Reality; Virtual Reality; Display Wall; Immersive Displays; Presence

HoloLens view inside CAVE2

Extending Virtual Reality Display Wall Environments Using Augmented Reality

Congratulations to Daria Tsoupikova in collaboration with award winning theater director Jo Cattell and their team, including application developers - EVL PhD candidates Arthur Nishimoto and Sai Priya, Computer Science professor / EVL interim director Andrew Johnson, and project network engineering lead Lance Long! The theater team and actors include Denise Savas (stage manager), Devonte Washington (actor/assistant stage manager), Stephanie Shum (lead actor), Liviu Passare (videographer). 
The following story appears in the <a href="http://digitaledition.chicagotribune.com/infinity/article_share.aspx?guid=2f768537-b6cd-42bd-af70-f5fa419357fe">Chicago Tribune, June 22, 2021</a>. 
Erratum: The article lists funding sources but incorrectly identifies the amounts of funding received. Hummingbird received the UI Presidential Award for Art and Humanities in 2019 for $150,000, the UIC Office of the Vice Chancellor for Research Creative Activity award in in 2020 for $10,000, and the National Endowment for the Arts in 2021 for $20,000. 
* * * * * * * * * * * * * * * 
Virtual-reality play: 
UIC debuts &rsquo;Hummingbird,&rsquo; allowing audience to participate in the story 
BY DARCEL ROCKETT CHICAGO TRIBUNE 
June 22, 2021 
Live theaters have yet to reopen as much of the state has, but when they do, take comfort in the fact that there are people working on taking the traditional theater-going experience to the next level. 
A group of designers and developers from the University of Illinois at Chicago&rsquo;s Electronic Visualization Laboratory in collaboration with award-winning theater director Jo Cattell, are making theater more participatory with virtual reality. The goal was realized this past weekend, June 18-20, when a variety of people ranging from preteen to retirement age were invited to partake in the immersive play &ldquo;Hummingbird.&rdquo; 
&ldquo;The whole idea here is moving in space and doing things together,&rdquo; said Daria Tsoupikova, a professor in the School of Design at UIC. &ldquo;You’re not just passive observer, you&rsquo;re actively participating.&rdquo; 
The approximately half-hour event took place in the Engineering Research building with a group of willing participants, including this Chicago Tribune reporter. Upon entering, each guest dons a lab coat and fills out a questionnaire about their soul: How old is it? When&rsquo;s the last time it was fed/replenished? After name tags are given, the group is introduced to a main character in the play, Aya, the teenage daughter of Demi, a Gerard Corp. employee. Aya is a bit antsy, but speaks her own mind; she is our guide along the experience. 
Participants follow Aya into a room where everyone is outfitted with their own Oculus Quest, an untethered virtual-reality headset with embedded tracking and touch controllers that enable multi-user interaction. At this time, the group is introduced to Gerard, a more contemporary version of Max Headroom, with better makeup, blond wig and more sass. Gerard is a floating head that talks back to Aya in real time. Gerard built Gerard Corp. to create Hummingbird technology, and he captured Demi, the talented scientist and mom of Aya. 
What follows is a collaborative team effort to escape Gerard&rsquo;s world of Zen gardens, flashlight-esque stalactites, moving statues, talking fauna and rocks by finding clues to lead us out and save Aya&rsquo;s mom. There&rsquo;s underground work, underwater work and even aerial work. As avatars, we all have different colors and stick-figure faces. The only thing that doesn&rsquo;t change is the participants&rsquo; voices. We all can talk to one another to communicate, do tasks together and problem solve. The colors, the sound design and camaraderie with strangers keeps everyone engaged with the play and one another. 
The project was designed for Goodman Theater New Stages 2020, showcasing experimental and groundbreaking theater works. The goal: to have participants attend tele-immersive, real-time performances running concurrently at the Goodman Theatre&rsquo;s Alice Rapport Center and at UIC, allowing for untethered social interaction and collaborative exploration of the plot lines through avatars. 
&ldquo;It&rsquo;s supposed to be totally immersive, it&rsquo;s supposed to be in real time, and the group at the Goodman and the group in here, they meet with each other and collaboratively explore the virtual space in co-located events, but inside virtual reality,&rdquo; Tsoupikova said. 
It&rsquo;s a meeting of art, science and live theater, where professional actors from the Goodman join with UIC faculty and students from the Department of Computer Science, School of Design and the Electronic Visualization Laboratory. The lab is known around the world for having introduced CAVE VR theater to the world In 1992 and in 2012, the CAVE2 VR theater. 
Tsoupikova says &ldquo;Hummingbird&rdquo; is to be conducted in more than one location simultaneously. The production targets youth, ages 11 and older, in under-served Chicago communities who lack access to the performing arts within their schools. 
&ldquo;We want to engage underrepresented Chicago youth - a tech-savvy younger audience through the use of
technology to engage them in the theater, because the theater attendance rates among youth are dropping,&rdquo; Tsoupikova said. &ldquo;We really want to show them that this is the future of storytelling, this is how it can be, and the theaters that productions through collaborative design like this is possible.&rdquo; 
According to Tsoupikova, the multi-user theatrical performance was conceptualized in 2018 and supported by a UI Presidential Award for Art and Humanities and an Office of the Vice Chancellor for Research Creative Activity award. A 2019 six-figure grant from the National Endowment of the Arts also helped with the development of the technology, creation of virtual scenes, scripted audience tracks, stage sets and professional actors. With the Goodman being the first major nonprofit Chicago theater to return to live performances on July 30, Tsoupikova hopes &ldquo;Hummingbird&rdquo; will open at Goodman Theatre this fall. 
&ldquo;My son hasn&rsquo;t been able to play with anyone, given the pandemic for 15 months, and for him to be so excited to be in a room with four other kids and suddenly to be interacting with people was like a dream come true,&rdquo; said &ldquo;Hummingbird&rdquo; director and writer Cattell. &ldquo;That was something nice to see, and (to) be able to give him a controlled environment where he can play with people again. I hope that continues - kids being really excited to be in a situation where it’s a different type of playground, it&rsquo;s a technical playground, because that&rsquo;s what we&rsquo;re really trying to create: a playground for everyone to play in.&rdquo;

Genevieve Thiers participates Sunday in &ldquo;Hummingbird&rdquo; at the University of Illinois at Chicago.

Virtual-reality play: UIC debuts &rsquo;Hummingbird,&rsquo; allowing audience to participate in the story

<ahref="https://today.uic.edu/uics-hummingbird-play-a-union-between-design-engineering-and-theater-in-virtual-reality?utm_source=mailpoet&utm_medium=email&utm_campaign=today%2B06.29.21">UIC Today</a> covers the &ldquo;Hummingbird&rdquo; project, a collaboration between UIC and the Goodman Theatre that blends VR storytelling and theatrical performance with the goal of encouraging stronger theatrical engagement with a tech-savvy younger generation audiences. It is an immersive play that was the brainchild of Daria Tsoupikova, professor in the school of design, Jo Cattell, an award-winning theater director, and members of the Electronic Visualization Laboratory (EVL), including Andrew Johnson, interim director and director of research. 
Workshop performances of &ldquo;Hummingbird&rdquo; were held at EVL June 18th through the 20th with a theatrical premiere planned for this coming fall.

&ldquo;Hummingbird&rdquo; Among UIC Today&rsquo;s Top Stories

Construction activities for the new $117.8 million, 135,000-square-foot Computer Design Research and Learning Center to be located at 900 W. Taylor St. is underway. The CDRLC will support the Department of Computer Science&rsquo;s growing student enrollment with 16,000 square feet of classroom space for collaborative teaching and learning environments, as well as 35 labs for faculty research activities in the computer science and engineering disciplines. 
The CDRLC will be the new home for the Electronic Visualization Laboratory featuring the latest technological and computing innovations to support research in advanced visualization, networking and collaborative environments. 
Read more about the CDRLC groundbreaking in the <a href="https://today.uic.edu/uic-breaks-ground-on-new-computer-design-research-and-learning-center">July 15th issue of UIC Today.</a> 
See a live webcam of the building&rsquo;s progress via the <a href="https://webcampub.multivista.com/index.cfm?fuseaction=aPublicWebcam.embed&WebcamPublicEmbedUID=5D4DCF05-AB2B-4770-A090-D79BE467F958">Multivista Construction Documentation webcam.</a>

New Computer Design Research and Learning Center Breaks Ground

Following the premiere of &ldquo;Hummingbird&rdquo; as part of the Chicago&rsquo;s Tony Award-winning Goodman Theatre New Stages Festival in December 2021, the virtual reality theatrical experience was exhibited at the SIGGRAPH 2022 Conference Immersive Pavilion held in Vancouver, CA, August 8-11th. Team members enhanced the existing experience to allow for a stand-alone version running consecutively for 4 days to sell-out audiences/participants - augmenting interactivity, animations, videography as well as upgrades to the underlying software and VR systems. By all accounts the exhibition was extremely well received and a success. 
&ldquo;Hummingbird: Multi-User Virtual Reality Theater for Collaborative Tele-Immersive Exploration&rdquo; is a modern, innovative performance, merging live theater and interactive virtual reality by bringing a group of active participants into a shared space for a live performance. This project bridges art, science and live theater through a collaborative research effort between computer science and design faculty and students at the University of Illinois Chicago (UIC) Electronic Visualization Laboratory (EVL) and Chicago theater directors, actors, videographers and producers. &ldquo;Hummingbird&rdquo; is developed by Daria Tsoupikova (Professor, School of Design); Andrew Johnson (Professor, EVL, Computer Science); Jo Cattell (Goodman Theatre); Lance Long (Senior Research Programmer, EVL, Computer Science); Arthur Nishimoto (PhD Candidate, EVL, Computer Science); Sai Jyothula Priya (PhD Candidate, EVL, Computer Science).

Participants engaged in the &ldquo;Hummingbird&rdquo; virtual reality performance at S2022.

&ldquo;Hummingbird&rdquo; Exhibited at SIGGRAPH 2022 (S2022)

Hummingbird: is a modern, innovative performance merging live theater and interactive virtual reality by bringing a group of active participants into a shared space for a live performance. The performance premiered as part of Chicago&rsquo;s Tony Award-winning Goodman Theatre New Stages Festival showcasing innovative and ground-breaking theater works in December 2021. This project bridges art, science and live theater through a collaborative research effort between computer science and design faculty and students at the University of Illinois Chicago (UIC) Electronic Visualization Laboratory (EVL) and Chicago theater directors, actors, videographers and producers. Hummingbird&rsquo;s story celebrates courage and coming of age through the eyes of a gutsy teen who must outsmart her mother&rsquo's narcissistic boss and survive dangerous new technology in a live, immersive adventure. Hummingbird extends traditional live theater and makes virtual reality art accessible to a broader audience, demonstrating how virtual reality can transform theatrical storytelling. 
CCS CONCEPTS 
Applied computing - Arts and humanities; Performing arts; Human-centered computing - Collaborative and social computing; Collaborative and social computing theory, concepts and paradigms; Computer-supported cooperative work. 
KEYWORDS 
Virtual reality, Storytelling, Theater, Multi-user, Performance

&ldquo;Hummingbird&rdquo; being performed at EVL during pilot performances summer of 2021

Hummingbird: A Collaborative Live Theater and Virtual Reality Adventure

UIC Department of Computer Science 
Doctoral Admissions Webinar 
November 2nd 2022 @7 am-8:30am Chicago time (CDT) 
Webinar: <a href="https://tinyurl.com/uicwebinar2022">https://tinyurl.com/uicwebinar2022</a> 
See what the Department of Computer Science at the University of Illinois Chicago has to offer! Located in the beautiful city of Chicago, UIC is among the most culturally diverse universities in the United States, and home to one of the most innovative computer science departments in the United States. UIC is Chicago&rsquo;s top public research university, and was recently ranked in the top 20 of the best 100 universities worldwide that are under 50 years of age. Doctoral students in their first five years of enrollment are guaranteed a fully funded appointment that includes a tuition waiver, and a stipend. Students are also eligible for summer support. Our students get jobs in academia, industry, and entertainment upon graduation, including positions at top research labs and companies such as Google, Microsoft, Apple, Disney, and Amazon. 
During this webinar, UIC Computer Science faculty will talk about our research agenda, application fee waivers, the application process, application expectations in terms of GPA/GRE etc, and will take questions. A recording of the webinar will be made available on our Graduate Admissions page. 
Apply: <a href="https://cs.uic.edu/graduate/admissions/">https://cs.uic.edu/graduate/admissions/</a> 
Deadline: December 15th 2022 
See our flyer at: <a href="https://tinyurl.com/uic2022flyer">https://tinyurl.com/uic2022flyer</a> 
 <iframe width="420" height="315"src="https://www.youtube.com/embed/eNoBbCm1skM">
</iframe>

Interested in pursuing a PhD in Computer Science in the United States?

EVL alumna and Associate Professor in Interactive Systems at the Department of Informatics & Telecommunications, National and Kapodistrian University of Athens, Maria Roussou presents her research in immersive eXtended Reality (XR): 
Wednesday, May 3, 2023, 1:30PM 
EVL Continuum Room 2068 ERF 
Dr. Roussou&rsquo;s talk entitled &ldquo:Extending the virtual: the museum as a hybrid experience&rdquo; will discuss the use of advanced digital technologies, such as Virtual Reality, in the GLAM sector (e.g., galleries, libraries, archives, museums, etc.) to create immersive experiences for visitors has come of age. However, to date, the emphasis has been on reconstructing the real to transgress the built environment's limitations. Nowadays, the focus turns to integrating the physical and the virtual. The physical space is highlighted, complemented, and expanded through close coupling with the digital. Tangible objects are integrated and play a leading role in the digital experiences of visitors, creating a hybrid reality. Drawing from a range of high-end immersive eXtended Reality (XR) applications but also lower-end mobile tours created for cultural or informal educational purposes, this talk will explore how the future’s hybrid nature is shifting immersive presentation from the mainly didactic walkthroughs to experiences that are playful, emotive, social and shared, meaningfully interactive, and, ultimately, relevant and resonant to those they are designed for. 
Specifically, Roussou will present aspects of the BRIDGES project, a multi-user, multisensory XR experience set in informal learning contexts, and the EMOTIVE project, which aims to leverage interactive, social, and participatory capabilities. Both combine historical accuracy in representation with dramatic storytelling to enhance knowledge about the past, historical empathy, and resonance. 
--- 
The <a href="https://www.bridges-horizon.eu">BRIDGES project</a> and the <a href="https://www.emotiveproject.eu/">EMOTIVE project</a> have received funding from the European Union’s Horizon 2020 research and innovation program. 
Bio: 
Maria Roussou is an Associate Professor in Interactive Systems at the Department of Informatics & Telecommunications, National and Kapodistrian University of Athens. In 2003 she founded and co-directed <a href="http://www.makebelieve.gr/mroussou
">makebelieve</a>, an experience design and consulting company. Previously, she established (in 1998) and directed (until 2003) the Virtual Reality Department at the Foundation of the Hellenic World, where she was responsible for setting up the immersive projection-based VR exhibits and managing the research, design and development of the VR programs and related visitor experiences. She holds a PhD in Computer Science (VR/HCI) from the University of London (UCL), a Master in Fine Arts (MFA) degree in Electronic Visualization from the School of Art and Design, University of Illinois at Chicago, a Master of Science (M.Sc) in Electrical Engineering & Computer Science from the same university, and an undergraduate degree in Applied Informatics from the Athens University of Economics and Business. Maria is a Senior Member of the ACM, and Vice-chair of the Greek ACM SIGCHI Chapter and the Greek ACM-W Chapter.

Extending the virtual: the museum as a hybrid experience

Applications in immersive environments have gained popularity for training, learning, and recreational tasks. Due to the increasing availability of sensors and data-capturing devices, research is extending the use of immersive environments to support visual analytics processes, including sensemaking and strategic immersion for interaction and task completion. In this work, we propose Personal Situated Analytics (PSA) framework to embed users into recorded meetings with support for multiple degrees of immersion in the Reality-Virtuality spectrum. Our proposed framework encompasses various stages such as tracking, data capturing, data cleaning, data synchronization, prototype building, and deploying the final product to end-user hardware. We evaluate this framework on a data analysis scenario between human subjects and a conversational AI agent. Our pilot study (n=12) using this framework compares user experiences when using two different devices: Hololens2 and Quest2.

A user embedded in a conversation using Personal Situated Analytics in Virtual Reality as virtual avatars.

Using Personal Situated Analytics (PSA) to Interpret Recorded Meetings

Using immersive environments for training, education, and recreation is becoming increasingly popular among individuals. In recent years these environments have also been used to facilitate data analytics processes involving interaction and task completion. In this work, we discuss Personal Situated Analytics (PSA) a cross-platform framework to embed users into recorded conversations with support for multiple degrees of immersion in the Reality-Virtuality spectrum. Our development pipeline consisted of several stages: tracking, data capturing, data cleaning, data synchronization, prototype building, and deploying the final product to end-user hardware. We evaluate PSA by analyzing data from a recorded meeting involving human participants and a visual conversational AI agent. Our pilot study (n = 12) using this framework compares user experiences for MR and VR environments. 
Index Terms: Immersive Analytics, Situated Analytics, Visualization techniques, Human-centered computing

PSA is an immersive MR/VR framework for experiencing and analyzing recorded conversations and events. We see two participants represented as virtual avatars engaged in a conversation.

PSA: A Cross-Platform Framework for Situated Analytics in MR and VR

Candidate: Ashwini G. Naik 
Committee Members: 
Dr. Andrew Johnson (Advisor and Chair) 
Dr. Robert Kenyon (Co-Advisor) 
Dr. Debaleena Chattopadhyay 
Dr. Nikita Soni 
Dr. Steve Jones (External Member - Dept. of Communication) 
Date and Time: Wednesday, April 24th, 2024, 10 AM - 12 PM AM CST 
Location: ERF 2036 
Abstract: 
Meetings and conversations today are multimodal in nature, including the usage of diverse technologies in the form of text, video, images, voice, AI agents, and also immersive technologies. If it were possible to replay important multimodal conversations in their spatial and chronological contexts, it would benefit several audiences, such as - researchers to study and analyze multimodal conversations, novices in any field who could learn from experts, experts who have missed being part of crucial conversations. However, recreating such multimodal interactions requires integrating all of the modalities used in the conversation. Immersive environments provide a convenient setting to present data with its spatial and temporal referents through the use of situated analytics. We present MuSA (Multimodal Situated Analytics), an immersive environment prototype, as a way to recreate multimodal conversations for exploration and analysis in immersive environments. Our development pipeline consisted of various stages such as data capturing, data cleaning, data synchronization, prototype building, and deploying the final product to end-user hardware. We conducted two comprehensive user studies to evaluate the usability, user adoption, and space usage during the exploration of multimodal conversations with our application. In the first study, we explore conversations between seated participants (n=12), and in the second, we explore conversations between non-seated moving participants (n=13). We share insights gained during the development phase, as well as empirical results and feedback collected through both user studies.

(a)Data Collection: 2 participants +  visual conversational AI agent (b)User Study: Replaying (a) in VR - MuSA in Quest2. (c)Data Collection: 2 participants exploring AR (d)User Study: Replaying (c) in Mixed Reality through MuSA in HoloLens2.

PhD Dissertation Announcement: &ldquo;Multimodal Situated Analytics (MuSA) for Analyzing Conversations in Extended Reality&rdquo;

September 2023 marks the 50th anniversary of the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC). EVL&rsquo;s introduction of the CAVE Automatic Virtual Environment in 1992, the first widely replicated, projection-based, walk-in, virtual-reality (VR) system in the world, put EVL at the forefront of collaborative, immersive data exploration and analytics. However, the journey did not begin then. Since its founding in 1973, EVL has been developing tools and techniques for real-time, interactive visualizations - pillars of VR. But EVL&rsquo;s culture is also relevant to its successes, as it has always been an interdisciplinary lab that fosters teamwork, where each person&rsquo;s expertise contributes to the development of the necessary tools, hardware, system software, applications, and human interface models to solve problems. Over the years, as multidisciplinary collaborations evolved and advanced scientific instruments and data resources were distributed globally, the need to access and share data and visualizations while working with colleagues, local and remote, synchronous and asynchronous, also became important fields of study. This paper is a retrospective of EVL&rsquo;s past 50 years that surveys the many networked, immersive, collaborative visualization and VR systems and applications it developed and deployed, as well as lessons learned and future plans. 
Article appears in the journal PRESENCE: Virtual and Augmented Reality, in the Special Issue on Immersive Visualization Laboratories - Past, Present and Future, edited by S. Su and W. Sherman, <a href="https://direct.mit.edu/pvar/issue/volume/33">https://direct.mit.edu/pvar/issue/volume/33</a>. 
Article copyrighted by Massachusetts Institute of Technology (MIT). 
DOI: <a href="https://doi.org/10.1162/pres_a_00421">https://doi.org/10.1162/pres_a_00421</a>

EVL faculty, students, and collaborators 1975.

Electronic Visualization Laboratory&rsquo;s 50th Anniversary Retrospective: Look to the Future, Build on the Past

September 2023 marked the 50th anniversary of the Electronic Visualization Laboratory (EVL). This paper summarizes EVL&rsquo;s efforts in Visual Data Science, with a focus on the many networked, immersive, collaborative visualization and virtual reality (VR) systems and applications the Lab has developed and deployed, as
well as lessons learned and future plans. 
Index terms: Visualization, Virtual Reality, Visual Data Science, Collaboration

An EVL CAD Rendering of the DOCC Space

Immersive Analytics at the Electronic Visualization Laboratory

Meetings and conversations today are multimodal in nature, including the usage of diverse technologies in the form of text, video, images, voice, AI agents, and also immersive technologies. If it were possible to replay important multimodal conversations in their spatial and chronological contexts, it would benefit several audiences, such as - researchers to study and analyze multimodal conversations, novices in any field who could learn from experts, experts who have missed being part of crucial conversations. However, recreating such multimodal interactions requires integrating all of the modalities used in the conversation. Immersive environments provide a convenient setting to present data with its spatial and temporal referents through the use of situated analytics. We present MuSA (Multimodal Situated Analytics), an immersive environment prototype, as a way to recreate multimodal conversations for exploration and analysis in immersive environments. Our development pipeline consisted of various stages such as data capturing, data cleaning, data synchronization, prototype building, and deploying the final product to end-user hardware. We conducted two comprehensive user studies to evaluate the usability, user adoption, and space usage during the exploration of multimodal conversations with our application. In the first study, we explore conversations between seated participants (n=12), and in the second, we explore conversations between non-seated moving participants (n=13). We share insights gained during the development phase, as well as empirical results and feedback collected through both user studies.

Depiction of an analyst investigating a multimodal conversation that includes participants and other entities of the conversation through MuSA in Mixed Reality (HoloLens2).

Mixed Reality

Bibliography

pubs (11)

Related Topics

art (2)

people (3)

events (16)

Contact