Interactive, real-time exploration and correlation of multi-terabyte and petabyte datasets from multiple sources are critical to advancing scientific discovery in many disciplines, including climate modeling and prediction, biomedical imaging, geosciences, high-energy physics and homeland security. These data-intensive applications are now being enabled by the OptIPuter, a new paradigm that relies on multi-gigabit photonic networks to interconnect distributed storage, computing and visualization resources, thereby creating a planetary-scale supercomputer. Critical performance bottlenecks have been the access latencies associated with reading and/or writing data from/to storage systems and clusters, whether connected via local or wide-area networks. In the case of climate modeling and analysis, the NASA climate Modeling, Analysis, and Prediction (MAP) project noted that these latencies result in “…the supercomputers remaining idle for 25-50% of the execution time during the analysis phase.” Reducing I/O latency would allow researchers to run more complex models during the same timeframe, resulting in faster and more accurate weather prediction.

LambdaRAM is a high-performance, multi-dimensional, distributed cache that harnesses the memory of cluster nodes in one or more clusters that are interconnected by ultra-high-speed networks, providing data-intensive applications with rapid access to both local and remote data. LambdaRAM’s memory and data management enables applications to rapidly stride over multi-dimensional multi-terabyte scientific datasets.

It employs novel proactive latency mitigation heuristics, including presending and prefetching, based on the access patterns of an application, and data transfer protocols designed for high-bandwidth networks to mitigate data access latencies. Using LambdaRAM, climate analysis applications to compute wind shear, surface temperature and ozone thickness are able to rapidly stride through multi-terabyte datasets over both local and wide-area networks and achieve up to 20-fold improvement in performance.

PDF

Vishwanath, V., LambdaRAM: A High-Performance, Multi-Dimensional, Distributed Cache Over Ultra-High Speed Networks, Submitted as partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate College, University of Illinois at Chicago, December 15th, 2008.