The CAVE2 is a large-scale, 320-degree, 3D / 2D, virtual-reality environment developed by the Electronic Visualization Laboratory at the University of Illinois at Chicago. The environment has a tracking system that determines the location of the primary user, and then renders 3D images to his perspective. The CAVE2 also has a 20.2 channel sound system controlled by a SuperCollider-based audio server that is, in turn, controlled via the Omicron SoundAPI. Audio imaging for the system had operated with the assumption that the listener was stationary in the center of the CAVE2 and the individual source width of audio objects was not dynamically altered. Our team added functionality to maintain audio imaging, including source position and width, as a tracked user and sound object move relative to one another within the environment.

CAVE2 as located at the University of Chicago at Illinois&rsquo; Electronic Visualization Laboratory

Dynamic Audio Imaging in Radial Virtual Reality Environments

EDITOR&rsquo;S NOTE: Here are excerpts from a recent article by Northwestern University journalism student Michael Epstein on the audio dilation project being developed by UIC computer science PhD candidate and UIC Electronic Visualization Laboratory (EVL) student John Novak. Novak is working with EVL and computer science professor Robert Kenyon. 

Say it slowly: &lsquo;Audio dilation&rsquo; could enhance hearing comprehension 
Michael Epstein 
December 4, 2014 

John Novak thinks that we all need to slow down and listen if we want to really hear a conversation or even your favorite song. 

Novak, a Ph.D. candidate studying computer science at the University of Illinois at Chicago&rsquo;s Electronic Visualization Lab, has spent the last four years developing software he calls &ldquo;audio dilation,&rdquo; which can reduce the speed of audio streaming through a smartphone or laptop - music or a phone call - in real time with little to no effect on its clarity. 

&ldquo;If you really analyzed this [dilated musical clip] and really pulled out the mathematics sample by sample,&rdquo; Novak explained, &ldquo;yes, there would be distortions in it, but to the human ear it&rsquo;s a pretty good reproduction of what would happen if you told a bunch of musicians, &lsquo;just play this slower.&rsquo;&rdquo; 

While the average person might see slowing down the sound as a waste of precious time, Novak believes that giving listeners more time to take things in allows them to more fully comprehend information. Novak specifically theorized that his software could make it easier to translate foreign languages on the fly and could help the hearing impaired. 

Novak suggested that his software could be helpful when used in conjunction with hearing aids. For people who have partial hearing loss and can&rsquo;t hear certain frequencies, slowing down sounds could help the brain recognize what&rsquo;s missing, essentially filling in the blanks of a person&rsquo;s hearing. 

To read the complete article, see: 
<a href="http://medillreports.com/2014/12/04/say-it-slowly-audio-dilation-could-enhance-hearing-comprehension/#more-4647">medillreports.com/2014/12/04/say-it-slowly-audio-dilation-could-enhance-hearing-comprehension/#more-4647</a>

PhD candidate John Novak prepares to demonstrate his audio dilation software.

Say it slowly: &lsquo;Audio dilation&rsquo; could enhance hearing comprehension

Professor Chris Brown, director of the Mills Center of Contemporary Music will be present &ldquo;Supercollider Software for Live, Interactive Performance and Improvisation&rdquo; at the March 2015 AES@UIC meeting to be held at the Electronic Visualization Laboratory&rsquo;s Cyber-Commons. This presentation will include an overview and live demos of original software optimized for live interactive performance and improvisation. 
Chris Brown is an internationally renown composer, pianist, and electronic musician; he creates music for acoustic instruments with interactive electronics, for computer networks, and for improvising ensembles. Collaboration and improvisation are consistent themes in his work, as well as the invention and performance of new electronic instruments. Supercollider provides the underpinnings for his current work and will be the focus of his presentation to the AES@UIC community.

AES@UIC March Meeting Announcement

Chris Brown

AES@UIC March Meeting - &ldquo;Supercollider Software for Live, Interactive Performance and Improvisation&rdquo;

Algorithmically decreasing speech tempo, or &ldquo;audio dilation,&rdquo; can improve speech perception in attentionally demanding tasks [Gygi & Shafiro (2014), Hear. Res. 310, pp. 76-86]. On-line audio dilation [Novak, et al. (2013), Interspeech 1869-1871] is a recently developed technique that decreases the tempo of audio signals as they are generated. This pilot study investigated effects of on-line audio dilation on performance in interactive problem solving tasks. We used a Diapix task [Baker & Hazan (2011), Behav. Res. Methods 43(3), 761-770] to elicit and record spontaneous speech from pairs of participants under various dilation conditions: participants, seated in different rooms, were asked to find ten differences on two similar pictures, while their speech was either transmitted as spoken or dilated. Conditions tested include stretching one, both, or neither audio signal by 40%. Subsequent analysis shows that the technique, even using this substantial increase, did not interfere with interactive problem solving tasks, and did not lead to changes in speech production rate, measured as number of syllables per second. The lack of negative effects of on-line speech dilation provides a preliminary basis for further assessment of this method in speech perception tasks with high attentional and memory load.

Diapix Image Test

Audio dilation in real time speech communication

University of Illinois at Chicago&rsquo;s CAVE2 is a large-scale, 320-degree radial visualization environment with a 360-degree 20.2 channel radial speaker system. The purpose of our research is to develop solutions for spatially accurate playback of audio within a virtual reality environment, reconciling differences between the circular speaker array, the location of a user in the physical space, and the location of virtual sound objects within CAVE2&rsquo;s OmegaLib virtual reality software, all in real time. Previous research presented at AES 137 detailed our work on object geometry, dynamically mapping a virtual object’s width and distance to the speaker array with volume and delay compensation. Our recent work improves virtual width perception using dynamic decorrelation with transient fidelity, implemented via Supercollider on the CAVE2 sound server.

Decorrelated Audio Imaging in Radial Virtual Reality Environments

A range of platforms facilitate the design of virtual environments. Most commonly, game engines, such as Unity and Unreal, are used for developing and simulating virtual realities. However, such platforms are primarily oriented towards visual design and provide only limited audio functionality. Furthermore, these platforms utilize separate editing and rendering modes, and operate on desktop platforms. Making use of existing research into the development of interactive virtual soundscapes, we introduce a novel user interface that enables the rapid design of both virtual sonic environments, and the assets (i.e. sound objects and sound zones) contained within them. 
Specifically, our UI offers the following contributions: 
<ul>
<li>provides a 3D design environment specific to sonic virtual realities, with specialized components such as sound objects and sound zones;</li> 
<li>offers both interactive and parametric control over the attributes of such components, enabling a precise authoring of highly-detailed environments;</li> 
<li>introduces a multi-cone model for creating 3D sound objects with complex propagation characteristics;</li> 
<li>enables adding dynamism to objects via hand drawn motion trajectories that can be edited in 3D;</li> 
<li>makes it possible to design virtual sound fields at various scales using multiple view and attribute windows;</li> 
<li>offers a unified interface for the design and the simulation of such realities, allowing the user to modify a sound field in real-time;</li> 
<li>operates on the web-browser so that it supports mobile devices, which therefore makes it possible for the user to simultaneously explore and edit augmented sonic realities.</li>
</ul>

A screenshot of our user interface on a desktop computer displaying an object with two cones and a motion trajectory being edited.

Designing and Controlling Virtual Sonic Environments Using a Browser-based 3DUI

EVL summer intern, Shreyas Mohan from Metea Valley High School, Aurora, IL presents the results of his efforts towards developing a robust music instruction application for use in virtual and augmented reality environments - the CAVE2 and Microsoft HoloLens systems. 
EVL Cyber-Commons, 2068 ERF 
Wednesday, August 10, 2016, 1:00 PM 
Shreyas developed an application for the CAVE2 and Microsoft HoloLens to investigate whether these technologies could be used effectively to teach music to a novice. Both applications taught the user &lsquo;Mary Had a Little Lamb&rsquo; on the viola, graphically representing the correct bow direction, tempo, fingering, and sheet music. Additional application features included adjustable tempo and metronome functionality. Combining graphical visual cues while seeing the physical instrument in a mixed reality environment proved to be a powerful method for independent music instruction.

Music Instructiton in VR/AR Presentation

HoloLens Interface

Music Instruction in a Virtual/Augmented Reality Environment (CAVE2&trade; and HoloLens)

This project&rsquo;s purpose was to investigate the ability of virtual reality to teach music. An application was developed for the CAVE2&trade; and Microsoft HoloLens, and their ability to teach music effectively was analyzed. Both the applications teach the user how to play &lsquo;Mary Had a Little Lamb&rsquo; on the viola, and each virtual reality platform had it&rsquo;s own benefits and drawbacks. An user interface was created for both platforms that shows the user the correct bow direction, tempo, fingering, and sheet music. Additional features included adjustable tempo and a metronome that can be turned on and off. The application was helpful since the user could get visual cues from the graphics while still keeping an eye on his/her instrument. Future goals for
the application include songs that can be uploaded by the user, microphone input, more instruments, and better graphics. The results of this research show that music instruction is a viable possibility in a virtual reality environment and will spark developers and musicians to work together to further explore this field.

Trackers placed on viola for CAVE2 version of the music instruction application.

Music Instruction in a Virtual/Augmented Reality Environment Using the CAVE2 and Microsoft HoloLens

The Experimental Sound Studio presents eclectic performance and installation programming as part of it's OSCILLATIONS 2016 series. Among this year&rsquo;s line-up is EVL post-doctoral research associate Anil Camci, who will perform &lsquo;Temas&rsquo; - a real-time, stochastic, electronic, acoustic performance. In this performance, Camci traverses the fine line between the organic and the synthesized, analyzing patterns of sound and performing probabilities. 
Saturday, November 12, 2016 
Experimental Sound Studio 
5925 N Ravenswood 
Chicago, IL 60660 
Doors open at 7:30pm; performance begins at 8:00pm.

EVL Post-doctoral Research Associate Anil Camci Performs at the Experimental Sound Studio: OSCILLATIONS 2016

Newcity magazine is Chicago&rsquo;s only locally owned and operated cultural weekly, and has been covering Chicago&rsquo;s local scene for more than 25 years. It&rsquo;s current &ldquo;Best of Chicago&rdquo; issue just came out (Volume 31, No. 1359). The section on City Life lists many best things about Chicago, including &ldquo;Best place to get your geek on.&rdquo; Um, that would be the Electronic Visualization Laboratory at the University of Illinois at Chicago! The article is reproduced below. 
* * * * * * * * * * 
Best place to get your geek on 
The Electronic Visualization Laboratory (EVL) at UIC 
Cutting-edge research meets a massive wall made out of small screens that would make any hardcore videogame fan scream and a literal cave that opens up a highly immersive, surround-screen, surround-sound, virtual reality environment at the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago. The lab brings together academics, designers, engineers, artists and mad scientists using technologies that bend reality off into unimaginable directions. EVL is the place where visualization, virtual reality and real-time interactive computer graphics are born. As if that were not enough, it was here that the computer graphics for the first &ldquo;Star Wars&rdquo; film were created in 1976. How cool is that? 
UIC College of Engineering, Department of Computer Science, 842 W. Taylor 
Best of Chicago 2016

Best of Chicago 2016

Newcity magazine rates UIC/EVL as &ldquo;Best place to get your geek on&rdquo;

Node Kara is an audiovisual mixed reality installation by the artist Anıl Camci. It offers a body-based interaction using a 3D imaging of the exhibition space. With a blurred audiovisual scene in its steady-state, it invites visitors to interact with the multiplicity of layers it holds within. By moving in the exhibition space, the visitors can deblur both the sound and the visuals as they get immersed in the piece. Rather than a detection of discrete events or gestures, Node Kara relies on fluid interactions with continuous visual layers and stochastic audio. This way, the piece reacts to each visitor in a unique way. A video preview of Node Kara can be seen at <a href="https://vimeo.com/anilcamci/nodekara">https://vimeo.com/anilcamci/nodekara</a> (password: nodekara). 
Index Terms: H.5.1 [Information interfaces and presentation (e.g., HCI)]: Multimedia Information Systems-Artificial, augmented, and virtual realities; H.5.2 [Information Interfaces and Presentation]: User Interfaces-Auditory (non-speech) feedback.

Four layers of Node Kara superimposed onto each other with a representation of the visitor’s depth image highlighted.

Node Kara: An Audiovisual Mixed Reality Installation

In this paper, we investigate methods for improving networked musical collaboration by introducing game mechanics to musical performance. We present a novel audiovisual performance system, where multiple users contribute to an over-the-network performance that is mediated by a variety of game-like rules. User evaluations and audience feedback collected in both private performance and public concert contexts indicate the potential of game-like music environments for improving musical interaction and listener attention in networked computer music performances. 
Author Keywords: Networked music performance; game mechanics; audiovisual performance systems. 
ACM Classification Keywords: H.5.5 [Information interfaces and presentation.

A screencast from a performance with Monad v0.1 [6], which utilized a client-driven rewarding mechanism.

Using Game Mechanics to Facilitate Networked Musical Collaboration

In this paper, we describe a novel multimedia system for networked musical collaboration. Our system, called Monad, offers a 3D virtual environment that can be shared by multiple participants to collaborate remotely on a musical performance. With Monad, we explore how various features of this environment in relation to game mechanics, network architecture, and audiovisual aesthetics can be used to mitigate problems inherent to networked musical performance, such as time delays, data loss, and reduced agency of users. Finally, we describe the results of a series of qualitative user studies that illustrate the effectiveness of some of our design
decisions with two separate versions of Monad. 
Author Keywords: Networked music performance; virtual environments; game mechanics; audiovisual performance systems. 
ACM Classification: H.5.5 [Information Interfaces and Presentation] Sound and Music Computing|Systems, H.5.2 [Information Interfaces and Presentation] User Interfaces|Auditory (non-speech) feedback, H.5.2 [Information Interfaces and Presentation] Multimedia Information Systems

A public performance with Monad v0.2 with local players in Istanbul and remote players from Vancouver and Berlin.

Networked Virtual Environments as Collaborative Music Spaces

In this paper, we investigate the role affect plays in electronic music listening. By referring to a listening experiment conducted over the course of three years, we explore the relation between affect and diegetic affordances (i.e. those of the spatiotemporal universes created by electronic music). We will compare existing perspectives on affect with the psychologist James Gibson&rsquo;s model of affordances in the context of an electronic music practice. We will conclude that both the sounds themselves and the diegetic affordances of these sounds may elicit affective reactions, and that further study into the relation between diegesis, affordance, and affect may contribute to a better understanding of what we hear in electronic music.

Diegetic Affordances and Affect in Electronic Music

The highly rewarding experience of electronic music persists throughout our daily lives. Our immediate environments are replete with events that emit sounds that are extremely complex. Electronic music engages with the listening habits we take for granted in our everyday lives, and reveals how intricate they can be. Inspired by such intricacies, I have conducted a series of listening experiments with 80 participants over the course of three years to explore the cognition of electronic music. In this article, I will first present the method and the results of this experiment, including a categorical analysis of mental associations evoked by different works of electronic music. Next, I will offer a discussion of narrativity in electronic music supported by these results and diverse perspectives on narrativity from a number of disciplines. I will then construct a definition of gestures as narrative units in electronic music in relation to events in the environment. In doing so, I will bring together various theories on electronic music with not only the findings of the current study but also existing research on auditory cognition.

Categorical distribution of real-time descriptors by piece.

Imagining through Sound: An experimental analysis of narrativity in electronic music

Electromagnetic fields are formed through complex interactions between outer space, the Sun, our Earth, its atmosphere, and the built environment. Our communications technology makes use of them to enable the transmission of information at local, global, and even extraterrestrial scales. Mastery of electromagnetic fields made the information age possible, and the constant acceleration of innovation in communication technology continues to profoundly influence contemporary culture. Because wireless communication takes advantage of ranges of the electromagnetic spectrum that we are unable to sense physically, it is necessary to translate these signals into a domain that human beings can more readily perceive in order to understand them. Not only is it relevant to see the signals themselves (which is commonly done through instruments such as oscilloscopes or spectrum analyzers), it is also important to understand what kinds of information are being transmitted via these signals and the sociopolitical mechanisms that define and govern their use. 
This article introduces a series of artworks that explore new creative opportunities made possible via low-cost sensors (which can be used to home in on a small region of the spectrurm within limited ranges) and state-of-the-art receivers (which are able to capture terabytes of spectrum data per second from great distances.

Data Improvisations, the IEEE VIS Arts Program exhibition. Traversing the Spectrum Radio Streams and Aerial Modulations were first presented at this exhibition October 2015.

Spectral Landscapes: Visualizing Electromagnetic Interactions

With the advent of new hardware and software technologies, virtual reality has gained a significant momentum recently. VR design tools, such as game engines, have become much more accessible and are being used in a variety of applications ranging from physical rehabilitation to immersive art. These tools, however, offer a limited set of tools for audio processing in 3D virtual environments. Furthermore, they are platform-dependent due to performance requirements and feature separate editing and rendering modes, which can be limiting for sonic VR implementations. To address these, we introduce a novel web-based system that makes it possible to compose and control the binaural rendering of a dynamic open-space auditory scene. Developed within a framework of well-established theories on sound, our system enables a highly detailed bottom-up construction of interactive virtual soundscapes by offering tools to populate navigable sound fields at various scales (i.e. from sound cones to 3D sound objects to sound zones). Based on modern web technologies, such as WebGL and Web Audio, our system operates on both desktop computers and mobile devices. This enables our system to be used for a variety of mixed reality applications, including those where users can simultaneously manipulate and experience a virtual soundscape.

A Web-based System for Designing Interactive Virtual Soundscapes

Distractions is a kinetic sculpture that brings together sonification and visualization in a single physical artifact. It utilizes inaudible frequencies to visualize invisible electromagnetic signals, such as those from cellphone, Wi-Fi and radio communications, that surround us in daily life. This way, the piece highlights some of our routine sources of distraction in their raw forms. By picking up the electromagnetic signals in the exhibition space, the sculpture visualizes the signals to and from the mobile devices that are brought into this space by the visitors. The sub frequency vibrations, which oscillate the structure according to these intentional or unintentional communications, not only represent the distractions the artist faces on a regular basis, but also intercept the dialogue between the audience and the inert steady state of the sculpture. The piece therefore interacts with the audience through the distractions that draw them away from the experience of the artwork. 
Author Keywords: Kinetic sculpture; sonification; depth image; radio frequency data; visualization; CNC 
ACM Classification Keywords: H.5.1 [Information interfaces and presentation (e.g., HCI)]: Multimedia Information Systems

The MDF framwework of the sculpture being milled with a CNC machine.

Distractions: Interactive Visualizations on a Kinetic Sculpture Using Infrasound Vibrations

PORTAL is an interactive performance tool that uses a laser projector to visualize computer-generated audio signals. In this paper, we firrst offer an overview of earlier work on audiovisual and laser art that inspired the current project. We then discuss our own implementation, focusing not only on the technical issues related to the use of a laser projector in an artistic context, but also on the aesthetic considerations in dealing with the translation of sounds into visuals, and vice versa. We provide detailed descriptions of our hardware implementation, our software system, and its desktop and mobile interfaces, which are made available online. Finally, we offer the results of a user study we conducted in the form of an interactive online survey on audience perception of the relationship between analogous sounds and visuals, which was explored as part of our performance practice. 
Author Keywords: Audiovisual art, sound-image interaction, multimodality, laser projection
ACM Classification: H.5.5 [Information Interfaces and Presentation] Sound and Music Computing, H.5.2 [Information Interfaces and Presentation] User Interfaces

A photo from a PORTAL performance.

PORTAL: An Audiovisual Laser Performance System

Clear Speech consists of crisp, loud, and slowly spoken sentence(s) for increased intelligibility where the utterances are distinguishable and audible without much strain on the human ear. For individuals with auditory comprehension disabilities, clear speech would be easier to comprehend than casual or normal speech. In this effort, we try to automate the conversion of casual speech to clear speech using machine translation. 
In the current work we develop a Recurrent Neural Network (RNN) that uses parallel corpora with one set of corpus as input and predicts output by the parameters learned from another corpus for this translation. Basing our study from literature, we make use of Deep Bidirectional Long Short term memory (LSTM) based RNN model because of its proven effectiveness in speech translation domain. To generalize, we convert voice A to voice B by passing voice A through multiple forward-backward LSTM exploiting its capability to learn voice B from both past and future time steps in voice A trained to regress voice B via backpropagation through time. 
We performed our preliminary experiments on a male to female speech conversion task, following which we were able to extend the developed pipeline for casual to clear speech conversion.

Male(top), Female(middle), Output(bottom) of the synthesized speech of the ARCTI Ccorpus

Deep Bidirectional LSTM based RNN for Casual Speech to Clear Speech conversion

Just as NSF cyberinfrastructure produces greater volumes and varieties of data from data storage systems, online instrumentation and major computational resources like XSEDE and Blue Waters, advanced visualization instruments serve as the &ldquo;lenses&rdquo; of these data generators (similar to the telescopes and microscopes of yore), enabling researchers to peer into cyberspace to manage, access and analyze big data. 
Continuum is a cyberinfrastructure instrument that unifies high-resolution computer-enhanced group collaboration workspaces with arrays of ambient sensors, enabling users to collaborate with local and remote colleagues and their data more effectively by giving the room itself the ability to anticipate their needs, improve their interaction, and focus their time on discovery rather than the technology. Continuum is a digital assistant, benefiting global scientific collaboratories and providing a foundation for new computer science research. War rooms benefit collaboration because they support continuous interaction and awareness of team members, enable persistent information sharing, and encourage mobility among team members and information. A computer-enhanced war room, such as Continuum, which (1) connects distributed teams with their data, whether stored in the cloud or streamed from major computational resources over high-speed networks, (2) supports persistent digital artifacts, (3) enables ultra-high-resolution 2D and 3D stereoscopic digital media to be simultaneously viewed, and (4) supports ubiquitous and intuitive interaction, creates a powerful and easy-to-use information-rich environment in support of scientific discovery. 
Continuum is the culmination of the University of Illinois at Chicago’s (UIC) 20+ years experience developing interactive instruments, from the original CAVE in 1992, to the ultra-high-resolution LambdaVision tiled LCD wall in 2004, and the hybrid-reality CAVE2 in 2012. Each new generation of visualization instrumentation provided scientific communities with more advanced features Continuum development focuses on three areas: (1) transforming passive display walls that respond to user commands into attentive display walls that monitor the users in a room and anticipate their needs to improve their experience; (2) providing an alternative approach to constructing high-resolution visualization display walls using thin-border 4K passive stereo OLED displays that have impressive brightness, contrast, uniformity, off-axis viewing, and low reflectivity in brightly lit rooms; and (3) utilizing cloud computing to prototype future Continuum spaces that don’t require local high-end compute and display resources. Twelve funded research projects from local institutions, in Astronomy, Astrophysics, Biology,
Combustion, Geophysics, Neuroscience, Pathology, Physics, Psychiatry, and High Performance Computing, are poised to use Continuum for their data visualization needs. Continuum will open up new opportunities in computer science research at the intersection of large-scale data visualization, human-computer interaction, and high-speed networking. Continuum will also directly support 10 classes taught in the UIC Computer Science, Art and Design, and Biomedical Science departments.

Artist's rendering of Continuum space.

MRI - Development of Continuum: A Virtualized Attentive Environment for Amplified Collaboration

Ableton Live is music software designed for computer-based composition and performance. <a href="https://en.wikipedia.org/wiki/Ambisonics">Ambisonics</a> is a full-sphere surround sound technique used for playing audio on sound systems with more than two speakers, providing the illusion that sound is coming from all directions. Combining these has allowed music producers to compose music inside of Ableton, in 3D. Programmed in the visual programming language <a href="https://cycling74.com/products/max/">Max</a>, is an ambisonic decoder for the Sound Dome built by former EVL post-doc Anil Camci. 
The decoder outputs sound sources through specific speakers in the sound system based on their specified Azimuth, Elevation, and Distance parameters. To integrate Ableton Live with the decoder, all of the audio channels in Ableton were streamed into the decoder via an application called Jack which allows users to route audio from one application to another. A Max utility was also built to run inside of Ableton Live and allow the user to conveniently adjust the azimuth, elevation, and distance parameters of a particular channel. 
The azimuth elevation and distance values are sent over a network back to the Max decoder allowing producers to change the location of playback for every channel in Ableton without leaving the application. Custom audio effects are also being programmed to exploit all of the surrounding speakers and take full effect of system.

EVL Sound Dome developed by Anil Camci

Ableton Live Integration & Audio Effects for a 32 Speaker Ambisonic Sound System

Talkers intentionally producing high-intelligibility speech for listeners in challenging situations often reduce their speech rate. This study affords listeners fine-grained control over the playback rate of a desired speech signal in varying levels of background noise, and tests listener intelligibility with their preferred and unmodified rates of speech. We find clear listener preference for decreased rates of speech as background noise increased. However, we also found degraded performance on a speech-in-noise intelligibility test relative to unmodified speech in these same conditions. 
Index Terms: perception of prosody; adverse listening conditions; speech intelligibility

Score Difference, Expansion vs SNR

Effects of User Controlled Speech Rate on Intelligibility in Noisy Environments

PhD Candidate John Novak and Prof. Kenyon&rsquo;s paper &ldquo;Effects of User Controlled Speech Rate on Intelligibility in Noisy Environments&rdquo; Interspeech 2018, in Hyderabad, India this September! 
Their paper was one of twelve nominated out of hundreds of accepted papers and over 2,500 submissions for Interspeech 2018 being held in Hyderabad, India September 2-6. Interspeech is the world&rsquo;s largest and most comprehensive conference on the science and technology of spoken language processing. 
Novak explained their research in more detail. &ldquo;The difficult situation is a type of background noise called
&lsquo;cocktail party noise,&rsquo; which is exactly what it sounds like: multiple people talking in the background of a target voice,&rdquo; he said. &ldquo;The computer mediated tool was a method of artificially slowing speech without causing distortion, especially without dropping the pitch of the voices. This is something that talkers do naturally [in addition to speaking louder] to help their listeners, but we wanted to put some control in the hands of the listener. We believe this is very important, although very
difficult to achieve.&rdquo; 
<a href="https://www.cs.uic.edu/wp-content/uploads/2018/06/interspeech-2018.pdf">Read the full UIC Computer Science, College of Engineering release.</a>

John Novak&rsquo;s Interspeech 2018 paper nominated for best student award

When speaking to second language learners, talkers often reduce their rate of speech to assist their listeners&rsquo; understanding and comprehension. This study grants English as a Second Language subjects fine-grained, real-time control over the playback rates of lengthy audio tracks of conversational speech, and tests the subjects&rsquo; listening comprehension at their desired playback speeds and at unmodified speeds. We find evidence that slower playback rates are preferred, but no evidence that such playback rates affect listener comprehension. 
Index Terms: Perception of prosody; ESL; human-computer interaction

The Effects of Time Expansion on English as a Second Language Individuals

Construction activities for the new $117.8 million, 135,000-square-foot Computer Design Research and Learning Center to be located at 900 W. Taylor St. is underway. The CDRLC will support the Department of Computer Science&rsquo;s growing student enrollment with 16,000 square feet of classroom space for collaborative teaching and learning environments, as well as 35 labs for faculty research activities in the computer science and engineering disciplines. 
The CDRLC will be the new home for the Electronic Visualization Laboratory featuring the latest technological and computing innovations to support research in advanced visualization, networking and collaborative environments. 
Read more about the CDRLC groundbreaking in the <a href="https://today.uic.edu/uic-breaks-ground-on-new-computer-design-research-and-learning-center">July 15th issue of UIC Today.</a> 
See a live webcam of the building&rsquo;s progress via the <a href="https://webcampub.multivista.com/index.cfm?fuseaction=aPublicWebcam.embed&WebcamPublicEmbedUID=5D4DCF05-AB2B-4770-A090-D79BE467F958">Multivista Construction Documentation webcam.</a>

New Computer Design Research and Learning Center Breaks Ground

It has often been observed, informally and in academic literature, that speech directed toward individuals who are hard of hearing is naturally modified by talkers in an attempt to accommodate the listeners. These modifications take numerous forms, from the easily expressed &ldquo;louder&rdquo; and/or &ldquo;slower&rdquo; to the highly technical &ldquo;vowel space expansion&rdquo; and many others. There are also speech modifications made in numerous other circumstances, not only for hard of hearing listeners, but for listeners in noisy backgrounds, second language listeners, and many others, known under the collective (is sometimes too-broad) umbrella as clear speech. There is, however, an asymmetry inherent to all these talker-listener scenarios, which is that talkers have only the broadest guidance at best (&ldquo;Louder!&rdquo; &ldquo;Slower!&rdquo;) from their listener partners, which they may simply ignore. 
With an interest in developing new and practical interventions for the elderly and hard of hearing, this dissertation takes as its research questions: First, the feasibility of the production of a signal processing system which allows real-time modification of audio signals. Second, the tolerance and effectiveness of partially emulating clear speech, by using signal processing techniques to modify (specifically, to slow) the rate of received speech signals. And finally, the effectiveness of placing the control of speech rate directly into the hands of listeners, a capability that has never been possible prior to electronic mediation. 
This dissertation is organized as follows: 
Chapter One is a survey of both the circumstances which prompt these spontaneous speech modifications, as well as of the details of the modifications which are made. 
Chapter Two presents an overview of the various methods of changing the rate of speech including the simple but ineffective technique of re-sampling, as well as several frequency domain techniques. These systems were intended for use on pre-existing, rather than live, signals. 
Chapter Three presents the details of several incarnations of this real-time system, including a single-talker system implemented on a laptop computer, presented in [1] and used directly (with two laptops for two talkers) in the study presented in [3]; a two-talker wireless system using Android smartphones and local network connectivity presented at [2]; and an unpublished system using a central laptop base station, to which Android smart phones could
connect even when hundreds of miles apart. This chapter contains details required for the implementation of both live signals, and live control of the playback rate of those signals. Finally, this system includes the design details of systems used in the three user studies of Chapters Five, Six, and Seven. 
Chapter Four presents a user study designed to examine the effects of using real-time temporal modification software in a conversational setting, previously published in [3]. The key element of the study design was the use of a scored Diapix test [7] which was designed to elicit spontaneous conversational speech between two study participants without the involvement of or prompting by researchers. The main results of this study are (i) that at modest amounts of stretching (40% additional playback time, with lengthy silences not stretched) the speech patterns of subjects are not changed to a statistically significant degree, and (ii) under those same circumstances, there is no statistical evidence of changes in performance on the Diapix test. 
Chapter Five presents the first of three user studies focusing on the little-studied subject of the interaction of user choice and user control over received speech rates in adverse conditions. This study was based off the preexisting QuickSIN test [8], designed to test and characterize a listener&rsquo;s ability to understand short sentences of speech played against a background of noise. In the first part of this study, users were asked to listen to several short sentences each played against multiple levels of background noise. For each level of background noise, the subjects were asked to use a computer interface to set the rate of the speech to whatever rate they believed was most helpful in understanding the foreground speech. (The background babble did not change rate.) These preferred expansion rates were recorded for later use and analysis. In the second part of the study, subjects were asked to listen to and immediately repeat back sentences at those same noise conditions, both with speech modified to their preferred expansion rates, and unmodified. The subjects’ repetition of those sentences was also recorded for analysis. 
Our results included statistically significant differences in received speech rate at opposite ends of the noise condition, with subjects requesting slower speech in the presence of increase noise; as well as an overall belief that slowing speech was beneficial to subjects’ performance. However, analysis of the subjects&rsquo; ability to repeat was in fact mildly degraded at moderate to high noise conditions and was not improved at any noise condition. This work was published at Interspeech 2018 [4]. 
Chapter Six presents the second of the three user control studies. In this study, non-native students with recent TOEFL scores were recruited as test subjects. After familiarization with a graphical interface, test subjects were asked to participate in six comprehension tasks, each of which was a lengthy (several minutes long) audio track of a TOEFL test to which they had not previously been exposed. During three of these tasks, the subjects were given
the ability to change the rate of playback as they saw fit, and asked to do so in whatever fashion they believed would best help them understand the audio passage. The other three comprehension tasks were presented unmodified, without a control interface. After each task, the subjects were immediately given a short multiple choice quiz about the contents of the previous audio passage. Subjects&rsquo; behavior (i.e., the rate changes specified) and quiz responses were collected and analyzed. 
Our analysis of the data shows that a considerable majority of the subjects used speech slowing in all three of their trials (roughly 2/3) and an overwhelming majority used the technique in at least one of their trials (80%), but did not reveal a statistically significant improvement or degradation in listening comprehension. There is also a weak correlation between lower TOEFL scores and more slowing. This work was published at Interspeech 2019 [5]. In the work described above, the algorithms employed stretch speech uniformly - if set to expand an audio track by 40%, all sounds, including all parts of speech all background noises, and even silences, are expanded by the same rate. For large expansion factors, the speech so produced sounds somewhat unnatural, because this is not how talkers spontaneously produce slow speech. With this in mind, Chapter Seven returns to the topic of speech in noise, but puts even more precise control in the hands of test subjects: A relatively simple neural network was developed to classify individual phonemes into six broad phonemic categories within an audio track. A study was also designed around this tool which asked subjects to listen to short sentences in noise, and then expand or contract the audio only for particular phonemes (e.g., first modify vowels, then modify fricatives, etc.) Following this, all modifications were applied simultaneously, and subjects were asked to listen to modified and unmodified/control sentences in three levels
of background noise and asked to transcribe the sentences into a computer interface. 
Our analysis found either no statistically significant improvement or degradation (at one noise level) or statistically significant degradation (at two noise levels) of intelligibility while using this technique. 
Finally, Chapter Eight presents overall conclusions, with additional discussion of the entire work and of directions for future studies.

Unwrapped Schematic of Cochlea

Audio Expansion Techniques For Real-Time Communication And Intelligibility Enhancement

September 2023 marks the 50th anniversary of the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC). EVL&rsquo;s introduction of the CAVE Automatic Virtual Environment in 1992, the first widely replicated, projection-based, walk-in, virtual-reality (VR) system in the world, put EVL at the forefront of collaborative, immersive data exploration and analytics. However, the journey did not begin then. Since its founding in 1973, EVL has been developing tools and techniques for real-time, interactive visualizations - pillars of VR. But EVL&rsquo;s culture is also relevant to its successes, as it has always been an interdisciplinary lab that fosters teamwork, where each person&rsquo;s expertise contributes to the development of the necessary tools, hardware, system software, applications, and human interface models to solve problems. Over the years, as multidisciplinary collaborations evolved and advanced scientific instruments and data resources were distributed globally, the need to access and share data and visualizations while working with colleagues, local and remote, synchronous and asynchronous, also became important fields of study. This paper is a retrospective of EVL&rsquo;s past 50 years that surveys the many networked, immersive, collaborative visualization and VR systems and applications it developed and deployed, as well as lessons learned and future plans. 
Article appears in the journal PRESENCE: Virtual and Augmented Reality, in the Special Issue on Immersive Visualization Laboratories - Past, Present and Future, edited by S. Su and W. Sherman, <a href="https://direct.mit.edu/pvar/issue/volume/33">https://direct.mit.edu/pvar/issue/volume/33</a>. 
Article copyrighted by Massachusetts Institute of Technology (MIT). 
DOI: <a href="https://doi.org/10.1162/pres_a_00421">https://doi.org/10.1162/pres_a_00421</a>

Audio Research

Bibliography

pubs (19)

Related Topics

events (7)

research (2)

Contact