Efficient RDF, graph-based queries are becoming more pertinent based on the increased interest in data analytics and its intersection with large, unstructured but connected data. Many commercial systems have adopted distributed RDF graph systems in order to handle increasing dataset sizes and complex queries. This paper introduces a distribute graph approach to pre-processing linked data. Instead of traversing the memory graph, our system indexes pre-processed join elements that are organized in a graph structure. We analyze the Dbpedia data-set (derived from the Wikipedia cor-pus) and compare our access method to the graph traversal access approach which we also devise. Results show from our experiments that the distributed, pre-processed graph approach to accessing linked data is faster than the traversal approach over a specific range of linked queries. 
CCS CONCEPTS: Computing methodologies, Distributed algorithms 
KEYWORDS: RDF, High Performance Computing; Distributed Algorithms

(a) Graph-store algorithm Graph construction and neighbor times over 1M, .5M triple dataset.

A Distributed Graph Approach for Pre-processing Linked RDF Data Using Supercomputers

In situ analysis has been proposed as a promising solution to glean faster insights and reduce the amount of data to storage. A critical challenge here is that the reduced dataset is typically located on a subset of the nodes and needs to be written out to storage. Data coupling in multiphysics codes also exhibits a sparse data movement pattern wherein data movement occurs among a subset of nodes. We evaluate the performance of data movement for sparse data patterns on the IBM Blue Gene/Q supercomputing system &ldquo;Mira&rdquo; and identify performance bottlenecks. We propose a multipath data movement algorithm for sparse data patterns based on an adaptation of a maximum flow algorithm together with a breadth-first search that fully exploits all the underlying datapaths and I/O nodes to improve data movement. We demonstrate the efficacy of our solutions through a set of micro benchmarks and application benchmarks on Mira scaling up to 131,072 compute cores. The results show that our approach achieves up to 5X improvement in achievable throughput compared with the default mechanisms. 
Keywords: multiple paths, sparse data movement, topology-aware aggregation, data-intensive, Blue Gene/Q

Efficacy of the approaches to move the data to /dev/null on the I/O nodes.

Improving Sparse Data Movement Performance Using Multiple Paths on the Blue Gene/Q Supercomputer

Achievable networking performance of applications in a supercomputer depends on the exact combination of the communication patterns of the applications and the routing algorithms used by the supercomputer. In order to achieve the highest networking performance for the applications, the routing algorithms need to be designed optimally for those communication patterns. However, while communication patterns usually vary from application to application and even from phase to phase in an application, routing algorithms have limited variation and usually are optimized for typical communication patterns. This results in high networking performance for some communication patterns. In this paper, we present approaches for improving communication performance by using multiple paths and re-balancing load on physical links on the Blue Gene/Q supercomputer. We realize our approaches in a framework called OPTIQ and demonstrate the efficacy of our framework via a set of benchmarks. Our results show that we can achieve 43 -- 67% higher throughput on average from 91 experiments, and can achieve higher throughput than default MPI_Alltoallv used for certain communication patterns.

Improving Communication Throughput by Multipath Load Balancing on Blue Gene/Q

Achievable networking performance of applications in a supercomputer depends on the exact combination of the communication patterns of the applications and the routing algorithms used by the supercomputer. In order to achieve the highest networking performance for the applications the routing algorithms need to be designed optimally for those communication patterns. However, while communication patterns usually have a wide variation from application to application and even from phase to phase in an application, routing algorithms have a limited variation and usually are optimized for typical communication patterns. This results in high networking performance for favored communication patterns but low networking performance for others. In this paper we present approaches for improving networking performance by rebalancing load on physical links on the Blue Gene Q supercomputer. We realize our approaches in a framework called OPTIQ and demonstrate the efficacy of our framework via a set of benchmarks. Our results show that we can achieve 30% higher throughput on experiment with data and patterns from a real application. The improvement can be up to several times higher throughput than default MPI_Alltoallv used in the Blue Gene Q supercomputer for certain communication patterns. 
Keywords: multi-path data movement, BG/Q, optimization, heuristic, interconnection network, communication patterns, network load balancing

Multipath Load Balancing for M x N Communication Patterns on the Blue Gene⁄Q Supercomputer Interconnection Network

SEPTEMBER 22, 2021 - The UIC Electronic Visualization Laboratory&rsquo;s (EVL) long-time collaborator, the University of California, San Diego (UCSD), received a National Science Foundation (NSF) grant, to start October 2021, to Enhance and Sustain their CHASE-CI project. As part of EVL&rsquo;s ongoing commitment to this research effort, EVL will host GPU resources and data servers to enhance and extend UCSD&rsquo;s Pacific Research Platform (PRP) Nautilus Hypercluster through 2021-2024. 
In 2017, UCSD received an NSF award for CHASE-CI, the Cognitive Hardware And Software Ecosystem - Community Infrastructure, which initially provided a community of Computer & Information Science and Engineering researchers doing Machine Learning and Computational Media applications with a distributed cluster containing 256 Graphics Processing Units (GPUs). This cluster - named Nautilus - was originally distributed across 10 California campuses and built on PRP. The NSF-funded PRP project focused on integrating California&rsquo;s high-speed regional research & education network (for fast data transfers among campuses) with DTNs, or Data Transfer Nodes, which are specialized storage systems located at each campus edge, optimized to quickly send and receive large data flows. 
Early on, EVL was among 50 UCSD partners to connect to PRP, thanks to UIC&rsquo;s high-bandwidth connectivity (100 Gigabits per second) to national research and education networks via the StarLight International/National Communications Exchange Facility in downtown Chicago. At that time, UCSD provided EVL with the campus&rsquo;s first DTN, which has since been integrated into the Nautilus cluster, providing 192 Terabytes (TB) of data storage for Big Data applications. More recently, as part of its ongoing PRP award, UCSD is providing EVL with an additional 216 TB of data storage. 
Over the past 5 years, Nautilus has grown. PRP interoperability with national and international research and education networks creates a high-capacity data-centric &ldquo;freeway system&rdquo; now stretching across the nation and across oceans that moves Nautilus data among DTNs located at other research institutions. UCSD&rsquo;s Nautilus cluster contains hundreds of GPUs and many Terabytes of storage. While 25% of the computational hardware and data resources are provided by the UCSD CHASE-CI award, 75% is provided by other researchers and institutions who voluntarily contribute resources, building a shared platform that they jointly own and control. EVL has contributed some of its own GPU nodes to Nautilus, with plans to federate more of its GPU resources over time. However, any Computer & Information Science and Engineering researcher can get an account on Nautilus, whether or not they contribute resources. 
This new CHASE-CI Enhance and Sustain award aims to enhance and sustain Nautilus&rsquo;s user community. EVL, as a UCSD partner institution, will receive an 8 GPU Nautilus Machine Learning node, to be integrated with Nautilus, to provide the community with additional GPUs and data services. While EVL currently has substantial computing facilities to support the needs of UIC College of Engineering faculty who require GPUs and data storage, being part of the CHASE-CI Enhance and Sustain grant and integrating the Lab&rsquo;s resources with Nautilus expands faculty and student access to national (and international) scale community computational and data resources. 
NSF award #2120019 to UCSD, ENS: CHASE-CI, 2021-2024 
NSF award #1730158 to UCSD, CHASE-CI, 2017-2020 
NSF award #1541349 to UCSD, PRP, 2015-2022

EVL Continues UCSD Collaboration to Enhance and Sustain CHASE-CI

EVL at the University of Illinois Chicago has joined Chameleon! Chameleon is a large-scale, deeply reconfigurable experimental platform built to support Computer Sciences systems research. Publicly available since 2015, Chameleon supports projects ranging from systems research developing new operating systems, virtualization methods, performance variability studies, and power management research to projects in software-defined networking, artificial intelligence, and resource management. 
Chameleon supports a bare metal reconfiguration system giving users full control of the software stack including root privileges, kernel customization, and console access. Chameleon adapted OpenStack, a mainstream open source cloud technology, to provide its capabilities. 
The EVL site brings back 8 Haswell nodes retired from <a href="https://www.tacc.utexas.edu/">Texas Advanced Computing Center (TACC)</a> with 13 more nodes coming online next month, for a total of 21. The EVL site can be accessed with an existing Chameleon Account, Projects, and Allocations, either by going directly to <a href="https://chi.evl.uic.edu/">https://chi.evl.uic.edu/</a>, the experiment dropdown on <a href="https://chameleoncloud.org">https://chameleoncloud.org</a>, or the dropdown on horizon for any other sites. 
Each node is equipped with 24 cores (48 threads), 128 GB of RAM, 2x 1G network connections, and a 250GB hard drive. Hardware details and availability can be reviewed on the hardware discovery page. 
<a href="https://github.com/ChameleonCloud/chi-in-a-box">https://github.com/ChameleonCloud/chi-in-a-box</a>

EVL Joins Chameleon

The COMPaaS DLV: Composable Platform as a Service Instrument for Deep Learning & Visualization user community consists of faculty from Physics and four College of Engineering departments (Computer Science, Civil, Materials & Environmental Engineering, Mechanical and Industrial Engineering, and Electrical and Computer Engineering) - primarily run applications that are GPU-centric for compute, with significant variability in storage and networking around their data requirements. 
Computer Science applications primarily focus on security, data science, computer vision and Machine Learning (ML). Security projects explore the complexity of modern web applications and the intricacies of security mechanisms that often result in flaws that expose users to significant security and privacy threats. These projects try to develop methods and tools that enable users to understand and more effectively manage retrospective privacy in the context of modern, long-lived, online archives. Composable resources were used to develop Natural Language Processing (NLP)-based domain-specific classifiers that identified data practices stated in privacy policies. Adherence of corresponding applications were then adjusted based on this ground truth. 
Data-science applications have an intuitive framework that integrates state-of-the-art AI technologies with applications, workflows, smart visualizations and collaboration services to help users access, share, explore and analyze their data, whether local or remote, come to conclusions, and make decisions with greater speed, accuracy, comprehensiveness and confidence. One such project is developing and advancing tools that identify image data in biomedical literature to locate beneficial, targeted publications. This work involves training image classifiers, integrating classifiers into labeling pipelines, designing retrieval user interfaces, and identifying related visual representations. 
Computer vision projects include semantic segregation and 3D human pose estimation. Researchers are developing a novel network architecture, termed DependencyNet (dependency network), for semantic segmentation. They also achieved experimental results that demonstrate an effective approach for 3D human pose estimation. Over the past two years, they found COMPaaS to be consistently stable and efficient, with result output as expected, and the group&rsquo;s models achieved state-of-the-art performance on their respective benchmarks. 
ML applications include frameworks for many complex real-world reinforcement learning problems, such as the coordination of autonomous vehicles, network packet delivery, and distributed logistics. 
Civil, Materials & Environmental Engineering applications focus on simulation and modeling. Researchers run data-driven models on high-performance computers to develop an accurate and general neural network ML model that uses crystallographic data to study patterns of synthesizability. They also perform simulations of mass transport in alloys and ceramics. COMPaaS has been performing 1.5-10 times faster than comparable infrastructures they are familiar with. Additionally, they found our use of Jupyter Notebooks to be a significant asset. 
Mechanical and Industrial Engineering researchers use COMPaaS for three research projects: feature extraction in fluid flow using a Convolutional Neural Network (CNN); column height detection in metallic nanoparticles using a CNN; and, electric vehicle battery state-of-charge estimation using different ML methods. 
Electrical and Computer Engineering researchers recently started running mathematical models of ML algorithms and training language models using long-short term memory (LSTMs). 
In 2020, we began implementing a GPUoE (GPU over Ethernet) prototype. Using a GPU expansion chassis connected to compute nodes over Ethernet, we were able to compose remote GPUs into our existing composable infrastructure. Our APIs developed for composable infrastructure have been extended to support these remote GPUs. 
In 2021, we enhanced COMPaaS with a public-facing JupyterHub server providing secure web-based access to experiments, a modern PCIe fabric and a supporting expansion chassis, thereby making composable co-location available to our users. As familiarity with composable infrastructure grows, the idea of using composable co-location instead of traditional servers co-located in racks is possible. By composable co-location, we mean the integration of emerging hardware needs without changing the existing core infrastructure. Researchers can now co-locate their own accelerators or compute nodes and leverage existing COMPaaS resources natively.

COMPaaS DLV User Community

In today&rsquo;s Big Data era, data scientists require modern workflows to quickly analyze large-scale datasets using complex codes to maintain the rate of scientific progress. These scientists often rely on available campus resources or off-the-shelf computational systems for their applications. Unified infrastructure or over-provisioned servers can quickly become
bottlenecks for specific tasks, wasting time and resources. Composable infrastructure helps solve these problems by providing users with new ways to increase resource utilization. Composable infrastructure disaggregates a computer&rsquo;s components – CPU, GPU (accelerators), storage and networking - into fluid pools of resources, but typically relies upon infrastructure engineers to
architect individual machines. Infrastructure is either managed with specialized command-line utilities, user interfaces or specification files. These management models are cumbersome and difficult to incorporate into data-science workflows. We
developed a high-level software API, Composastructure, which, when integrated into modern workflows, can be used by infrastructure engineers as well as data scientists to reorganize composable resources on demand. Composastructure enables infrastructures to be programmable, secure, persistent and reproducible. Our API composes machines, frees resources, supports multi-rack operations, and includes a Python module for Jupyter Notebooks. 
Keywords - distributed systems, testbed implementation and deployment, composable infrastructure, deep learning, visualization, infrastructure as code

Composastructure System Architecture

Moving from Composable to Programmable

Composable infrastructure holds the promise of accelerating the pace of academic research and discovery by enabling researchers to tailor the resources of a machine (e.g., GPUs, storage, NICs), on-demand, to address application needs. We were first introduced to composable infrastructure in 2018, and at the same time, there was growing demand among our College of Engineering faculty for GPU systems for data science, artificial intelligence / machine learning / deep learning, and visualization. Many purchased their own individual desktop or deskside systems, a few pursued more costly cloud and HPC solutions, and others looked to the College or campus computer center for GPU resources which, at the time, were scarce. After surveying the diverse needs of our faculty and studying product offerings by a few nascent startups in the composable infrastructure sector, we applied for and received a grant from the National Science Foundation in November 2019 to purchase a mid-scale system, configured to our specifications, for use by faculty and students for research and research training. 
This paper describes our composable infrastructure solution and implementation for our academic community. Given how modern workflows are progressively moving to containers and cloud frameworks (using Kubernetes) and to programming notebooks (primarily Jupyter), both for ease of use and for ensuring reproducible experiments, we initially adapted these tools for our system. We have since made it simpler to use our system, and now provide our users with a public facing JupyterHub server. We also added an expansion chassis to our system to enable composable co-location, which is a shared central architecture in which our researchers can insert and integrate specialized resources (GPUs, accelerators, networking cards, etc.) needed for their research. 
In February 2020, installation of our system was finalized and made operational and we began providing access to faculty in the College of Engineering. Now, two years later, it is used by over 40 faculty and students plus some external collaborators for research and research training. Their use cases and experiences are briefly described in this paper. Composable infrastructure has proven to be a useful computational system for workload variability, uneven applications, and modern workflows in academic environments. 
Keywords - composable infrastructure, deep learning, visualization, resource management, workload management, user workflow, composable co-location, infrastructure as code

COMPaaS Hardware Racks

Composable Infrastructures for an Academic Research Environment: Lessons Learned

EVL is a technology Innovator, building industry and cross-institutional connections and partnerships. In 2022, EVL deployed two GPU FIONAs in partnership with the Pacific Research Platform (PRP) and the University of California San Diego. The PRP is a partnership of more than 50 institutions that establishes a science-driven high-capacity data-centric &lsquo;freeway system&rsquo;. PRP has support from the National Science Foundation, and includes the US Department of Energy&rsquo;s Energy Science Network (ESnet), as well as scores of research universities in the nation and around the world. PRP leverages Nautilus, a HyperCluster for running containerized Big Data Applications using Kubernetes, and FIONA, flash input/output network applications. A GPU FIONA is a rack-mounted PCs designed by the PRP team that is optimized for 100Gbps large data storage with Mellanox ConnectX-6 SmartNICs, a 24 core Epic Rome CPU, 256GB of memory and 8 NVIDIA RTX-3090 GPUs. Nautilus provides over 7500 CPU cores, over 500 GPUs and 4 PBs of storage. The addition of two Nautilus GPU FIONAs at EVL builds upon our 2018 DTN (Data Transfer Node) that has provided over 300GB of Ceph Storage to Nautilus and our CAVE2 GPU Nodes that are contributing 5 NVIDIA GPUS. EVL provides Nautilus with 300GB of Ceph Storage, 128 CPU cores, and 21 NVIDIA GPUs. The platform is utilized for a wide variety of applications, including applying advanced astronomy AI to microscopy workflow, simulations for natural disasters, climate modeling, virtual reality and even accelerating NSF&rsquo;s IceCube Photon Propagation. UIC is enabled to participate in these cross-institutional connections using UIC&rsquo;s HPRNet, a network environment tailored to the needs of high-performance science applications, built on a Science DMZ architecture. 
<a href="https://pacificresearchplatform.org/media/">Read more about PRP.</a> 
<a href="https://pacificresearchplatform.org/media/sc21-larry-smarr-on-the-rise-of-supernetwork-data-intensive-computing/">See media coverage from SC21 on PRP and Nautilus.</a>

FIONA

Pacific Research Platform (PRP) - Nautilus

In today&rsquo;s Big Data era, data scientists require modern workflows to quickly analyze large-scale datasets using complex codes to maintain the rate of scientific progress. These scientists often rely on available campus resources or off-the-shelf
computational systems for their applications. Unified infrastructure or over-provisioned servers can quickly become bottlenecks for specific tasks, wasting time and resources. Composable infrastructure helps solve these problems by providing users with new ways to increase resource utilization. Composable infrastructure disaggregates a computer&rsquo;s components - CPU, GPU (accelerators), storage and networking - into fluid pools of resources, but typically relies upon infrastructure engineers to architect individual machines. Infrastructure is either managed with specialized command-line utilities, user interfaces or specification files. These management models are cumbersome and difficult to incorporate into data-science workflows. We developed a high-level software API, Composastructure, which, when integrated into modern workflows, can be used by infrastructure engineers as well as data scientists to reorganize composable resources on demand. Composastructure enables infrastructures to be programmable, secure, persistent and reproducible. Our API composes machines, frees resources, supports multi-rack operations, and includes a Python module for Jupyter Notebooks. 
Keywords: distributed systems, testbed implementation and deployment, composable infrastructure, deep learning, visualization, infrastructure as code

Composable infrastructure holds the promise of accelerating the pace of academic research and discovery by enabling researchers to tailor the resources of a machine (e.g., GPUs, storage, NICs), on-demand, to address application needs. We were first introduced to composable infrastructure in 2018, and at the same time, there was growing demand among our College of Engineering faculty for GPU systems for data science, artificial intelligence / machine learning / deep learning, and visualization. Many purchased their own individual desktop or deskside systems, a few pursued more costly cloud and HPC solutions, and others looked to the College or campus computer center for GPU resources which, at the time, were scarce. After surveying the diverse needs of our faculty and studying product offerings by a few nascent startups in the composable infrastructure sector, we applied for and received a grant from the National Science Foundation in November 2019 to purchase a mid-scale system, configured to our specifications, for use by faculty and students for research and research training. 
This paper describes our composable infrastructure solution and implementation for our academic community. Given how modern workflows are progressively moving to containers and cloud frameworks (using Kubernetes) and to programming notebooks (primarily Jupyter), both for ease of use and for ensuring reproducible experiments, we initially adapted these tools for our system. We have since made it simpler to use our system, and now provide our users with a public facing JupyterHub server. We also added an expansion chassis to our system to enable composable co-location, which is a shared central architecture in which our researchers can insert and integrate specialized resources (GPUs, accelerators, networking cards, etc.) needed for their research. 
In February 2020, installation of our system was finalized and made operational and we began providing access to faculty in the College of Engineering. Now, two years later, it is used by over 40 faculty and students plus some external collaborators for research and research training. Their use cases and experiences are briefly described in this paper. Composable infrastructure has proven to be a useful computational system for workload variability, uneven applications, and modern workflows in academic environments. 
Keywords: composable infrastructure, deep learning, visualization, resource management, workload management, user workflow, composable co-location, infrastructure as code

This academic year 2023 EVL alum Mike Papka has joined UIC&rsquo;s Department of Computer Science and EVL faculty as a professor, bringing years of expertise in high performance computing to campus. For nearly three decades, Papka has held the position of Research Scientist at Argonne National Laboratory contributing to research in high performance computing, scientific visualization of large and complex datasets, information visualization related to high-performance computing, virtual reality for science and the integration of edge services. 
Since 2006, he’s served as Deputy Associate Laboratory Director, Computing, Environment and Life Sciences (CELS) Directorate, providing strategic planning and management of the Laboratory Directorate comprising Biosciences, Computational Science, Data Science and Learning, Environmental Sciences, Mathematics and Computer Science, a joint institute with Northwestern University, and the Argonne Leadership Computing Facility. In his role, Papka aids in executing a research program that aims to couple computing-related activities with several science domains whose future is tied to progress in computing.
 For the last 12 years, Papka has served and continues to serve as Division Director at Argonne Leadership Computing Facility (ALCF). He is responsible for planning and executing facility upgrades and integrated data strategies, risk assessment and management, operations reporting and reviews, staff promotion and retention, and safety. The ALCF is home to some of the world&rsquo;s fastest supercomputers: Theta; a Cray XC40 supercomputer based on Intel’s Xeon Phi KNL processor and NVIDIA DGX-3, Polaris; a HPE Apollo 6500 Gen10+ based on AMD Milan processor and NVIDIA A100s, and Aurora; a Intel CPU/GPU system currently being installed that support scientific and engineering open science research. 
We welcome Mike&rsquo;s return to UIC and EVL! He undoubtedly will be a valued asset to our research and educational enterprise.

Alum Michael E. Papka Joins EVL Faculty

Contrails are condensation trails generated from emitted particles by aircraft engines, which perturb Earth&rsquo;s radiation budget. Simulation modeling is used to interpret the formation and development of contrails. These simulations are computationally intensive and rely on high-performance computing solutions, and the contrail structures are not well defined. We propose a visual computing system to assist in defining contrails and their characteristics, as well as in the analysis of parameters for computer-generated aircraft engine simulations. The back-end of our system leverages a contrail-formation criterion and clustering methods to detect contrails&rsquo; shape and evolution and identify similar simulation runs. The front-end system helps analyze contrails and their parameters across multiple simulation runs. The evaluation with domain experts shows this approach successfully aids in contrail data investigation. 
Index Terms: Scalar Field Data, Physical & Environmental Sciences, Mathematics, Feature Detection, Tracking & Transformation

Run analysis.

Visual Analysis and Detection of Contrails in Aircraft Engine Simulations

The Electronic Visualization Laboratory at the University of Illinois Chicago acquired a 24 compute node, 64 GPU composable infrastructure compute cluster, COMPaaS DLV - Composable Platform as a Service: Instrument for Deep Learning & Visualization in 2019.  Since then 40+ users have designed and run computations on COMPaaS.  To track cluster utilization, evaluate hardware
performance, and simply analyze application resource utilization of this experimental composable infrastructure system we set out to
collect metrics on the hardware and software infrastructure.  Users deploy their applications on the cluster using Kubernetes, a container orchestration platform that enables users to deploy applications on the cluster without the need to have complete knowledge of the underlying hardware and with minimal assistance from system administrators.  Following this, we utilized tooling already adapted for Kubernetes-based clusters to collect hardware and software metrics.  The metrics provided by this tooling is useful for system administrators in evaluating cluster performance and utilization, as well as useful for users in determining their application&rsquo;s performance and discovering bottlenecks.

High Performance Computing

Bibliography

pubs (10)

Related Topics

events (3)

research (2)

Contact