Research Experience Undergraduate

Aakash Lilaramani

PhD Graduate, 2024

Zhiming Zou

PhD Student

Dian Jia

MS Candidate

Raghuraman Venkatesh

Sarthak Agarwal

Xiaoqian Ruan

Visiting Scholar

Kun Xia

Hyeonjeong Park

The National Science Foundation has awarded EVL a Major Research Instrumentation (MRI) grant to build the Data Observation and Computation Collaboratory (DOCC) in the new Computer Design Research and Learning Center (CDRBC) building. The Data Observation and Computation Collaboratory (DOCC, &rsquo;the dock&rsquo;) is a much-needed visual data science instrument for UIC science and engineering research and research training. DOCC provides high-performance computing, advanced networking, high-speed data storage, interactive high-resolution displays and software and services for Artificial Intelligence / Machine Learning (AI/ML), data analytics, data simulations, visualization, and virtual-reality requirements. Given how varied modern data-science application requirements can be, DOCC is designed as a configurable, heterogeneous computing environment that prioritizes real-time, interactive, collaborative applications and remote data streaming, but can be scheduled off-hours for large, batch-mode processing. 
DOCC will empower users to work with their data and their colleagues in real-time, test codes, view a &rsquo;window in the oven&rsquo;to watch large simulations run, and simultaneously view results along with related data and ensemble runs on large display walls. DOCC has a visual front end that enables users to easily access familiar collaboration tools, scientific notebooks, AI/ML workflows, and popular visualization software. 
Alternatively, users can add new AI/ML models or create them from scratch. And because DOCC will be connected to the campus research network, data and/or computer codes can be ported to remote computer and cloud resources located on campus, regionally, nationally, or internationally. 
A room-sized instrument, DOCC will be located in the high-trafficked atrium of the new CDRLC, home to the UIC Computer Science Department, part of the College of Engineering, situated across from an auditorium and multi-purpose event space. This location will promote UIC data-science research initiatives and the instrument&rsquo;s collaboration potential. DOCC will also be used to conduct campus research meetings, programs and training initiatives. 
Award Number (FAIN): 2320261 
Period of Performance: 10/01/2023 - 09/30/2026 
Project Title: Research Infrastructure: MRI: Track 2 Acquisition of Data Observation and Computation Collaboratory (DOCC)

EVL Awarded NSF MRI Support to Build the Data Observation and Computation Collaboratory (DOCC)

Roberto Cesar Jr., professor/department head of Computer Science at the University of Sao Paolo and director of eScience Research Center presents his research in computer vision and image processing for urban visualization. Dr. Cesar&rsquo; invited talk will be held: 
EVL&rsquo;s Continuum 
Engineering Research Facility, Room 2068 
Friday, September 29th, noon 
&ldquo;Computer vision and physical models for urban informatics&rdquo; 
Computer vision techniques have played an important role in urban data analysis. Such techniques are useful for capturing and analyzing visual information about cities&rsquo; infrastructure, social manifestations and mobility, to name just a few. However, the adoption of computer vision techniques in urban environments faces important challenges, such as the restricted heterogeneous hardware typically involved, outdoor image variations, and other uncontrolled aspects. In this talk, we present some recent results to overcome these limitations. Specifically, we discuss computer vision techniques for rain scenes, pedestrian mobility analysis, and physical models for urban green regions and graffiti distributions. 
Bio: 
Roberto is a professor at the University of Sao Paulo (USP) since 1998 (BSc in Computer Science - UNESP - 1991; MSc in Electrical Engineering - UNICAMP - 1993; Ph.D. in Physics - USP/Brazil, IPT-UCL/Belgium - 1997). He is currently a full professor in the Department of Computer Science - IME - USP working in the Data Science Research Group. He is currently a special advisor for Physical Sciences and Engineering at the Sao Paulo Research Foundation - FAPESP. He served as the Director of the eScience Research Center at USP and as the head of the Computer Science Department. He was a member of the Image and Vision Computing and the Signal, Image and Video Processing editorial boards, chair and invited speaker of conferences and workshops (Sibgrapi 2003, CIARP 2010, Sibgrapi 2011; SHAPES 2.0 - 2012, eSon - IEEE eScience 2013, IEEE eScience 2014). He has experience in computer science, with an emphasis on computer vision, machine learning, and artificial intelligence. 
<a href="https://www.ime.usp.br/~cesar/">https://www.ime.usp.br/~cesar/</a>

Roberto Cesar Jr. Invited to Present &ldquo;Computer vision and physical models for urban informatics&rdquo;

Unsupervised semantic segmentation (USS) aims at partitioning an image into semantically meaningful segments by learning from a collection of unlabeled images. The effectiveness of current approaches is plagued by difficulties in coordinating representation learning and pixel clustering, modeling the varying feature distributions of different classes, handling outliers and noise, and addressing the pixel class imbalance problem. This paper introduces a novel approach, termed Imbalance-Aware Dense Discriminative Clustering (IDDC), for USS, which addresses all these difficulties in a unified framework. Different from existing approaches, which learn USS in two stages (i.e., generating and updating pseudo masks, or refining and clustering embeddings), IDDC learns pixel-wise feature representation and dense discriminative clustering in an end-to-end and self-supervised manner, through a novel objective function that transfers the manifold structure of pixels in the embedding space of a vision Transformer (ViT) to the label space while tolerating the noise in pixel affinities. During inference, the trained model directly outputs the classification probability of each pixel conditioned on the image. In addition, this paper proposes a new regularizer, based on the Weibull function, to handle pixel class imbalance and cluster degeneration in a single shot. Experimental results demonstrate that IDDC significantly outperforms all previous USS methods on three real-world datasets, COCO-Stuff-27, COCO-Stuff-171, and Cityscapes. Extensive ablation studies validate the effectiveness of each design. Our code is available at https://github.com/MY-LIU100101/IDDC. Our code is available at https://github.com/MY-LIU100101/IDDC. 
Keywords: Unsupervised semantic segmentation · Imbalance-Aware Dense Discriminative Clustering · End-to-end training ·
Deep clustering 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Imbalance-Aware Discriminative Clustering for Unsupervised Semantic Segmentation

Though the object detection performance on standard benchmarks has been improved drastically in the last decade, current object detectors are often vulnerable to domain shift between the training data and testing images. Domain adaptation techniques have been developed to adapt an object detector trained in a source domain to a target domain. However, they assume that the target domain is known and fixed and that a target dataset is available for training, which cannot be satisfied in many real-world applications. To close this gap, this paper investigates fully test-time adaptation for object detection. It means to update a trained object detector on a single testing image before making a prediction, without access to the training data. Through a diagnostic study of a baseline self-training framework, we show that a great challenge of this task is the unreliability of pseudo labels caused by domain shift. We then propose a simple yet effective method, termed the IoU Filter, to address this challenge. It consists of two new IoU-based indicators, both of which are complementary to the detection confidence. Experimental results on five datasets demonstrate that our approach could effectively adapt a trained detector to various kinds of domain shifts at test time and bring substantial performance gains. Code is available at https://github.com/XiaoqianRuan1/IoU-filter. Code is available at https://github.com/ XiaoqianRuan1/IoU-filter. 
Funding:: The COMPaaS DLV project (NSF award CNS-1828265)

Fully Test-time Adaptation for Object Detection

Semi-Supervised Temporal Action Localization (SS-TAL) aims to improve the generalization ability of action detectors with large-scale unlabeled videos. Albeit the recent advancement, one of the major challenges still remains: noisy pseudo labels hinder efficient learning on abundant unlabeled videos, embodied as location biases and category errors. In this paper, we dive deep into such an important but understudied dilemma. To this end, we propose a unified framework, termed Noisy Pseudo-Label Learning, to handle both location biases and category errors. Specifically, our method is featured with (1) Noisy Label Ranking to rank pseudo labels based on the semantic confidence and boundary reliability, (2) Noisy Label Filtering to address the class-imbalance problem of pseudo labels caused by category errors, (3) Noisy Label Learning to penalize inconsistent boundary predictions to achieve noise-tolerant learning for heavy location biases. As a result, our method could effectively handle the label noise problem and improve the utilization of a large amount of unlabeled videos. Extensive experiments on THUMOS14 and ActivityNet v1.3 demonstrate the effectiveness of our method. The code is available at github.com/kunnxia/NPL.

Learning from noisy pseudo labels for semi-supervised temporal action localization

Estimating the human posture from an image or a video is a fundamental task in computer vision. It not only enhances other vision tasks like action recognition, person re-identification, and virtual try-on but also facilitates many real-world applications including robotics, healthcare, sports, and retail. An effective and efficient human pose estimation system can enable robots to acquire skills from demonstrations, support physical therapists in diagnosing and rehabilitating patients, assist sports analysts and coaches in tracking and training athletes, and empower retailers to establish employee-free stores. Thanks to the advancement of deep learning and the availability of large-scale datasets, the performance of state-of-the-art human pose estimation methods has drastically improved in recent years. These approaches can accurately estimate postures in daily activities and sports. However, several challenges persist. For example, (1) it remains challenging to estimate postures that occur rarely or are entirely absent in the training data; (2) handling complex scenarios, such as crowded environments, motion blur, low-light conditions, and occlusions, is still a formidable task; (3) there is a growing need to develop efficient models that can estimate human poses in real-time or on low-power devices; (4) exploring novel applications for human pose estimation that can bring societal benefits or transform industries is an exciting avenue of research. This special issue presents a collection of high-quality papers on human pose estimation and its applications. They cover a wide range of topics, from fundamental human pose models with improved accuracy and efficiency to applications in human behavior understanding and assessment, sign language recognition, and multi-camera calibration. Through this special issue, we would like to open the path toward more discussion between practitioners and researchers in this very important yet challenging field of human pose estimation.

Guest editorial: special issue on human pose estimation and its applications

Weakly-supervised Video Anomaly Detection (WVAD) aims to detect abnormal events in videos given only video-level labels for training. Recent methods relying on multiple instance learning (MIL) and self-training achieve good performance, but they tend to focus on learning easy abnormal patterns while ignoring hard ones, e.g., unusual driving trajectory or over-speeding driving. How to detect hard anomalies is a critical but largely ignored problem in W-VAD. To tackle this challenge, we propose a novel framework, termed Abnormal Ratios guided Multi-phase Self-training (ARMS), for W-VAD. It includes a new abnormal ratio-based MIL (AR-MIL) loss and a new multi-phase self-training paradigm. The AR-MIL loss guides the learning of hard anomalies by enforcing a minimum ratio of abnormal snippets in an abnormal video and no abnormal snippets in a normal video. Our multi-phase self-training paradigm sequentially performs bootstrapping, hard anomalies mining, and adaptive self-training so as to address pseudo labeling on easy anomalies, detect hard anomalies, and setting adaptive abnormal ratios for different videos in a unified framework. Experimental results on three benchmark datasets, i.e., ShanghaiTech, UCF-Crime, and XD-Violence, show that ARMS outperforms all previous state-of-the-art methods and has a great advantage in detecting hard anomalies. 
Index Terms: Anomaly detection, weakly-supervised video anomaly detection, multiple instance learning.

Abnormal Ratios Guided Multi-Phase Self-Training for Weakly-Supervised Video Anomaly Detection

Developing sensing circuits and systems capable of perceiving the physical environment is an essential step toward next-generation cyber-physical systems that leverage computation, communication, and sensing integrated with the physical domain. The recent introduction of a class of artificial intelligence (AI) methods known as deep learning has drastically advanced vision-based environment perception. However, the applicability of these methods is significantly limited by their reliance on vast amounts of 3D annotations for training, which are expensive, unnatural, and often unobtainable. Acquiring 3D annotations necessitates specialized data captured by 3D sensors rather than utilizing abundant and readily available images or videos. Moreover, it requires a controlled environment or compromises with inaccurate subjective labeling. This project aims to address real-time environment perception in a very challenging, largely unexplored, yet highly practical setting, where only 2D annotations are available during training, by modeling and learning rich space-time structures of the environment. The outcomes of this project will significantly impact cyber-physical systems and facilitate a wide range of applications, from robots in manufacturing and personal services to autonomous vehicles that enhance people’s mobility and safety. Furthermore, this project will tightly integrate research and education at the University of Illinois Chicago (UIC), which is a Minority Serving Institution (MSI), through curriculum development, research training for high school, undergraduate, and graduate students, broadening the participation of female and minority students, and community outreach. 
This project will address real-time environment perception when only 2D annotations are available during training by systematically pursuing a novel approach that decomposes the complex physical environment into geometry and motion substructures in 3D space and models their rich space-time interactions. From streaming video, the trained system will not only recognize objects and scene layouts in the environment but also estimate their 3D geometry and 3D motion in real-time. This project is focused on (1) establishing a self-supervised framework for lifting 2D objects to 3D, by bridging geometry and motion, brightness constancy, and differentiable depth rendering, (2) scaling 2D-to-3D lifting of individual objects to perceive the entire environment, by modeling and learning short-term and long-term dynamics of the environment, and (3) achieving flexible and generalizable environment perception by handling articulated motion and out-of-distribution environments. The new approach is expected to be efficient in terms of the amount of annotations required for training, scalable to a wide range of objects, and robust to the complexity and diversity of real-world environments. This project will advance and enrich the fundamental research of visual sensing, perception, and learning. Moreover, this project will demonstrate the usability and robustness of the proposed approach in real cyber-physical systems.

CCSS: Modeling and Learning Space-time Structures for Real-time Environment Perception

EVL affiliated faculty and assistant professor of Computer Science Wei Tang has been appointed by Science Direct as associate editor of the journal &ldquo;Pattern Recognition&rdquo;.  The journal publishes developments in the fields of computer vision, image processing, text and document analysis, and neural networks. Dr. Tang&rsquo;s expertise in these fields makes him the perfect candidate for this position.  We congratulate Dr. Tang on his appointment.

Wei Tang Appointed Associate Editor for &ldquo;Pattern Recognition&rdquo;

Hossein Fathollahian

Unsupervised video anomaly detection (UVAD) aims to detect abnormal events in videos without any annotations. It remains challenging because anomalies are rare, diverse, and usually not well-defined. Existing UVAD methods are purely data-driven and perform unsupervised learning by identifying various abnormal patterns in videos. Since these methods largely rely on the feature representation and data distribution, they can only learn salient anomalies that are substantially different from normal events but ignore the less distinct ones. To address this challenge, this paper pursues a different approach that leverages data-irrelevant prior knowledge about normal and abnormal events for UVAD. We first propose a new normality prior for UVAD, suggesting that the start and end of a video are predominantly normal. We then propose normality propagation, which propagates normal knowledge based on relationships between video snippets to estimate the normal magnitudes of unlabeled snippets. Finally, unsupervised learning of abnormal detection is performed based on the propagated labels and a new loss re-weighting method. These components are complementary to normality propagation and mitigate the negative impact of incorrectly propagated labels. Extensive experiments on the ShanghaiTech and UCF-Crime benchmarks demonstrate the superior performance of our method. 
Funding: NSF award ECCS-2400900

Learning Anomalies with Normality Prior for Unsupervised Video Anomaly Detection

Deep learning approaches have made significant success in single-view 3D reconstruction, but they often rely on expensive 3D annotations for training. Recent efforts tackle this challenge by adopting an analysis-by-synthesis paradigm to learn 3D reconstruction with only 2D annotations. However, existing methods face limitations in both shape reconstruction and texture generation. This paper introduces an innovative Analysis-by-Synthesis Transformer that addresses these limitations in a unified framework by effectively modeling pixel-to-shape and pixel-to-texture relationships. It consists of a Shape Transformer and a Texture Transformer. The Shape Transformer employs learnable shape queries to fetch pixel-level features from the image, thereby achieving high-quality mesh reconstruction and recovering occluded vertices. The Texture Transformer employs texture queries for non-local gathering of texture information and thus eliminates incorrect inductive bias. Experimental results on CUB-200-2011 and ShapeNet datasets demonstrate superior performance in shape reconstruction and texture generation compared to previous methods. 
Funding: NSF award CNS-1828265, NSF award ECCS-2400900

Analysis-by-Synthesis Transformer for Single-View 3D Reconstruction

This year, students at the Electronic Visualization Laboratory (EVL) at the University of Illinois Chicago College of Engineering participated in the <a href="https://ieeevis.org/year/2024/welcome">IEEE VIS 2024 Conference</a>. 
Prof. Liz Marai&rsquo;s students had several presentations. PhD Student Siyuan Zhao presented his work &ldquo;A Part-to-Whole Circular Cell Explorer&rdquo; at the Bio+Med+Vis Workshop on Oct 13. PhD Student Andrew Wentzel presented his paper <a href="https://www.evl.uic.edu/pubs/2827">&ldquo;DITTO: A Visual Digital Twin for Interventions and Temporal Treatment Outcomes in Head and Neck Cancer&rdquo;</a> on Oct 17. 
Prof. Fabio Miranda&rsquo;s PhD Student Gustavo Moreira presented his work <a href="https://www.evl.uic.edu/pubs/2843">&ldquo;Curio: A Dataflow-Based Framework for Collaborative Urban Visual Analytics&rdquo;</a> on Oct 16.
Dr. Saeed BoorBoor presented his two papers, &ldquo;VoxAR: Adaptive Visualization of Volume Rendered Objects in Optical See-Through Augmented Reality&rdquo; and &ldquo;Submerse: Visualizing Storm Surge Flooding Simulations in Immersive Display Ecologies&rdquo;. He will join as an EVL Faculty member at the University of Illinois Chicago next term. 
UIC&rsquo;s EVL faculty Liz Marai, PhD Students Md Nafiul Alam Nipu, Siyuan Zhao, and Hossein Fathollahian also participated at the VIS Chicago Satellite Event hosted by the University of Chicago Data Science Institute. After the watch party for the plenary session, we heard from faculty and PhD students at UChicago, UIC&rsquo;s EVL, Northwestern, and DePaul.

IEEE VIS 2024: Visualization & Virtual Analytics Virtual Conference

Assistant professor Hy Kim, director of UIC&rsquo;s Human-Centered Engineering Lab will present his research: 
Friday 10/25/24, 1pm 
EVL Continuum (Room 2068 Engineering Research Facility) 
Abstract: 
Augmented reality (AR) has the potential to fundamentally change the way we interact with information. Direct perception of computer-generated graphics atop physical reality can afford hands-free access to contextual information on the fly. However, as users must interact with not only information on the display but also environmental changes in the real world, the traditional User-Centered Design (UCD) approaches that focus on human-computer interaction, may not always adequately address the dynamic nature of human-environment interaction. This seminar will introduce, Ecological Interface Design (EID), a new approach to the design and evaluation of AR interfaces to complement UCD. EID is a framework for interface design that respects dynamic, environmental constraints imposed on users&rsquo; behavior. EID addresses two questions for interface design; what information should be displayed (the content and structure of information that represent the physical reality of the work environment), and how that information should be presented (the perceptual forms of interface elements compatible with human information processing). The talk will start with the theoretical background of EID and show how this approach can benefit AR user interface design and evaluation with an example of an AR head-up display for vehicle drivers. 
Bio 
Hy Kim is an Assistant Professor of Industrial Engineering at the University of Illinois Chicago, where he is leading the <a href="https://sites.google.com/view/human-centered-engineering">Human-Centered Engineering lab</a>. His current research focuses on new methods for the requirements analysis, design, prototyping, and evaluation of extended reality (XR) applications, emphasizing human-technology partnerships at future workplaces. Sponsored by government agencies and industry partners, he has investigated basic topics on human factors in XR, such as human depth perception, visual attention, situation awareness, workload, and task performance while interacting with XR applications. Recently, his team has developed virtual environments to study how people would interact with future urban transportation services such as external human-machine interfaces for robotaxis & urban street robots (Honda), driver interfaces for SAE L3 automated vehicles (GM), and crew stations for remote operations of search & rescue vehicles (US Army). 
Dr. Kim received a Ph.D. in industrial and systems engineering specializing in human factors and ergonomics from Virginia Tech in 2017. He has more than ten years of research experience in Human-Computer Interaction from academia and industry, including Oakland University, Virginia Tech Center for HCI, and General Motors. He has served as PI or Co-PI of more than 10 research projects and published more than 30 papers in top-tier conferences and journals such as IEEE Virtual Reality, IEEE International Symposium on Mixed and Augmented Reality, ACM Automotive User Interfaces, and ACM Intelligent User Interfaces, IEEE Transactions on Visualization and Computer Graphics, and Human Factors. He actively serves the above research communities as an organizer of international conferences and workshops.

Beyond User-Centered Design: Ecological Approach to the User Interface Design and Evaluation in Augmented Reality

EVL Affiliated Faculty Assistant Professor, Human-Centered Engineering Lab

Hyungil Kim

September 2023 marks the 50th anniversary of the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC). EVL&rsquo;s introduction of the CAVE Automatic Virtual Environment in 1992, the first widely replicated, projection-based, walk-in, virtual-reality (VR) system in the world, put EVL at the forefront of collaborative, immersive data exploration and analytics. However, the journey did not begin then. Since its founding in 1973, EVL has been developing tools and techniques for real-time, interactive visualizations - pillars of VR. But EVL&rsquo;s culture is also relevant to its successes, as it has always been an interdisciplinary lab that fosters teamwork, where each person&rsquo;s expertise contributes to the development of the necessary tools, hardware, system software, applications, and human interface models to solve problems. Over the years, as multidisciplinary collaborations evolved and advanced scientific instruments and data resources were distributed globally, the need to access and share data and visualizations while working with colleagues, local and remote, synchronous and asynchronous, also became important fields of study. This paper is a retrospective of EVL&rsquo;s past 50 years that surveys the many networked, immersive, collaborative visualization and VR systems and applications it developed and deployed, as well as lessons learned and future plans. 
Article appears in the journal PRESENCE: Virtual and Augmented Reality, in the Special Issue on Immersive Visualization Laboratories - Past, Present and Future, edited by S. Su and W. Sherman, <a href="https://direct.mit.edu/pvar/issue/volume/33">https://direct.mit.edu/pvar/issue/volume/33</a>. 
Article copyrighted by Massachusetts Institute of Technology (MIT). 
DOI: <a href="https://doi.org/10.1162/pres_a_00421">https://doi.org/10.1162/pres_a_00421</a>

EVL faculty, students, and collaborators 1975.

Electronic Visualization Laboratory&rsquo;s 50th Anniversary Retrospective: Look to the Future, Build on the Past

EVL affiliated faculty and Computer Science assistant professor Wei Tang has been awarded his first NSF grant entitled &ldquo;CCSS: Modeling and Learning Space-time Structures for Real-time Environment Perception&rdquo;. 
Amount: $240,000 
Dates: August 1, 2024-July 31, 2027 
Brief abstract: 
Developing sensing circuits and systems capable of perceiving the physical environment is an essential step toward next-generation cyber-physical systems that leverage computation, communication, and sensing integrated with the physical domain. The recent introduction of a class of artificial intelligence (AI) methods known as deep learning has drastically advanced vision-based environment perception. However, the applicability of these methods is significantly limited by their reliance on vast amounts of 3D annotations for training, which are expensive, unnatural, and often unobtainable. Acquiring 3D annotations necessitates specialized data captured by 3D sensors rather than utilizing abundant and readily available images or videos. Moreover, it requires a controlled environment or compromises with inaccurate subjective labeling. This project aims to address real-time environment perception in a very challenging, largely unexplored, yet highly practical setting, where only 2D annotations are available during training, by modeling and learning rich space-time structures of the environment. The outcomes of this project will significantly impact cyber-physical systems and facilitate a wide range of applications, from robots in manufacturing and personal services to autonomous vehicles that enhance people&rsquo;s mobility and safety. Furthermore, this project will tightly integrate research and education at the University of Illinois Chicago (UIC), which is a Minority Serving Institution (MSI), through curriculum development, research training for high school, undergraduate, and graduate students, broadening the participation of female and minority students, and community outreach.

Wei Tang Awarded NSF Grant &ldquo;CCSS: Modeling and Learning Space-time Structures for Real-time Environment Perception&rdquo;

UIC CS Assistant Professor Wei Tang has received the NSF CAREER Award for his proposed research &ldquo;Compositional Learning and Understanding of the Physical World&rdquo;. $539,051 has been awarded for the five-year project period July 2025 to June 2030. 
Tan&rsquo;s research project aims to develop a computer vision framework that learns and understands the physical world in a compositional manner, offering two significant benefits. First, a compositional interpretation of objects and scenes enables intelligent systems to engage in richer physical interactions and accomplish more complex tasks. Second, by decomposing complex entities into simpler constituents and modeling their relationships, this compositional approach addresses fundamental challenges faced by purely data-driven methods, including data inefficiency, curse of dimensionality, and limited explainability. The outcomes of this project will impact a wide range of emerging applications, including robots that support manufacturing or assist with daily tasks, autonomous vehicles that enhance mobility and safety, and virtual or augmented reality interfaces that facilitate assistive workflows and remote collaboration. This project will tightly integrate research and education through curriculum development, research training for high school, undergraduate, and graduate students, and community outreach. 
This project will develop new methodologies for learning and understanding the innate compositionality of objects and scenes in the physical world. It consists of three innovative thrusts. Thrust I aims to establish a unified framework for representing, parsing, and learning the compositionality of physical objects, through disentangled modeling of large shape variations, constituent parts, and detailed deformations of each part as multi-granularity neural fields. Thrust II aims to develop a new compositional model that parses 3D dynamic scenes from streaming video into an explainable layout graph on the fly, by constructing distributed representations of low-level geometry and motion and performing explicit reasoning about high-level scene compositionality. Thrust III will extend the first two thrusts by modeling the compositionality of generic articulated objects and investigating test-time adaptation for 3D dynamic scene parsing. Distinct from purely data-driven methods, this new compositional paradigm reduces reliance on extensive 3D annotations, naturally handles the high dimensionality of geometry and motion, and enables a deeper, more explainable understanding of the physical world. This project will advance and enrich fundamental research in visual compositionality, physical object and scene understanding, and explainable parsing.

Computer Vision

Bibliography

pubs (8)

Related Topics

people (10)

events (7)

research (1)

Contact