Optical Network is Key to Next-Generation Research Cyberinfrastructure
Participants: Jason Leigh, Maxine Brown, Thomas A. DeFanti, Luc Renambot
Institutions: California Institute for Telecommunications and Information Technology (Calit2), San Diego Supercomputer Center
Las Vegas, NV
At TeraGrid ’08 Conference, UC San Diego’s Smarr Urges University Campuses to Remove Network Bottlenecks to Supercomputer Users
San Deigo, CA and Las Vegas, NV, June 11, 2008 - In a keynote talk at the TeraGrid ’08 Conference, the director of the California Institute for Telecommunications and Information Technology (Calit2) said today that all the pieces are in place for a revolution in the usability of remote high-performance computers to advance science across many disciplines. Toward that end, he urged scientists to create end-to-end dedicated lightpaths connecting remote supercomputers to their labs, greatly enhancing their local capability to visually analyze massive data generated by TeraGrid’s terascale to petascale computers.
“The last ten years have established the state, regional, national, and global optical networks needed for this revolution, but the bottleneck is on the user’s campus,” said Larry Smarr, director of Calit2, a partnership of UC San Diego and UC Irvine. However, Smarr said, there now is a clear path forward to removing this last bottleneck so that end users of the National Science Foundation-funded TeraGrid can begin to adopt these advanced technologies.
“The NSF-funded OptIPuter project [www.optiputer.net] has been exploring for six years how user-controlled, wide-area, 1- or 10-Gbps Internet protocol (IP) lightpaths - termed lambdas - on fiber optics can provide direct uncongested access to global data repositories, scientific instruments and high-performance computational resources from the researchers’ Linux clusters in their campus laboratories,” said Smarr. “This research is now being rapidly adopted because U.S. universities are beginning to acquire lambda access through state or regional optical networks interconnected with the National LambdaRail, the Internet2 Dynamic Circuit Network, and the Global Lambda Integrated Facility.”
Led by Calit2, the San Diego Supercomputer Center (SDSC), and the University of Illinois at Chicago’s Electronic Visualization Laboratory (EVL), the OptIPuter ties together the efforts of researchers from over a dozen campuses. “The OptIPuter essentially completes the Grid program,” said Smarr. “In addition to allowing the end user to discover, reserve, and integrate remote computers, storage, and instruments, the OptIPuter enables the user to do the same for dedicated lambdas, creating a high-performance LambdaGrid.”
A key OptIPuter technology, the OptIPortal, was prototyped by EVL and developed by Calit2 under the NSF-funded OptIPuter partnership. The OptIPortal is a networked and scalable, high-resolution LCD tiled display system, driven by a PC cluster. Designed for the user’s laboratory, each OptIPortal can be constructed with commodity commercial displays and processors. While most of the PC clusters run Linux, some run on Mac (Calit2@UCI and UCSD’s SIO) or on Windows (UCSD’s NCMIR) clusters.
“OptIPortals are the appropriate termination device for 10Gbps lightpaths, allowing the end user to choose the right amount of local storage, compute, and graphics capacity needed for their application,” said Smarr. “In addition, the tiled walls provide the scalable pixel real estate necessary to analyze visually the complexity of supercomputing runs.”
The OptIPuter project prefers OptIPortal clusters to run on SDSC’s Rocks, an open-source Linux cluster distribution that enables end users to easily build computational clusters, grid endpoints and visualization tiled-display walls. Rocks is developed under an NSF-funded SDCI project led by SDSC’s Philip Papadopoulos, also a co-principal investigator on the OptIPuter project.
OptIPortals range in size from four to 60 tiles, and provide screen resolutions ranging from 8 million pixels to the over ¼-billion-pixel HIPerSpace wall - the highest-resolution display system in the world, located in the Calit2 building on the UCSD campus.
To handle multi-gigabit data streams, OptIPuter researchers at EVL developed the Scalable Adaptive Graphics Environment (SAGE). Separately, Professor Falko Kuester’s UCSD Calit2 lab developed Cluster GLX, which allows OpenGL applications to be displayed on a visualization cluster like a tiled display. OptIPortal systems are now operating in over two dozen institutions in the United States and internationally.
To make effective use of the 10Gbps lightpaths from the TeraGrid resources to the campus gateways, Smarr said “the user’s campus must invest in the critical ‘missing link’ blocking widespread adoption of the OptIPuter / OptIPortal metacomputer - the equivalent of city ‘data freeway’ systems of switched optical fibers connecting the campus gateway to specific buildings and inside the buildings to the user’s lab.”
For example, UC San Diego has over 45 installed 10Gbps channels crisscrossing the campus - going up to 60 channels by the end of 2008. Geographically, these are located in seven different buildings, including 17 laboratories within these buildings. This vision was made possible with funds provided by the campus and an NSF-funded Major Research Instrumentation (MRI) grant Quartzite, which has SDSC’s Papadopoulos as PI and Calit2’s Smarr as one of the co-PIs.
“Quartzite provides the ‘golden spike’ which allows completion of end-to-end 10Gbps lightpaths running from TeraGrid sites to the remote user’s lab. When we were conceptualizing the OptIPuter seven years ago, I always thought that remote supercomputer users would provide the killer applications,” said Smarr, the founding director in 1985 of the National Center for Supercomputing Applications (NCSA).
“TeraGrid users are located in research campuses across the nation, but they all share the characteristic that they need to carry out interactive visual analysis of massive datasets generated by a remote supercomputer,” he said. “I believe that we will see early adopters step forward in the next year to set up prototypes of this cyberarchitecture.”
Smarr described the work of one such early adopter, Michael Norman, UCSD Professor of Physics, recently named SDSC’s Chief Scientific Officer. Norman is designing an OptIPortal in the new SDSC building, to be dedicated in October 2008. It will be connected over the UCSD optical complex described above to the TeraGrid 10Gbps backbone and National LambdaRail and used to visualize results from his cosmology simulations on the NSF’s Petascale Track II machines at the TACC and at the NICS.
Smarr also described how with the integration of high definition and digital cinema video streams, which easily fit inside of the 10Gbps lightpath, the OptIPuter architecture is rapidly creating an OptIPlanet Collaboratory in which multiple scientists can analyze a complex dataset while seeing and talking to each other as if they were physically in the same room. The most recent addition to the growing network - dubbed the “OzIPortal” - was deployed at the University of Melbourne earlier this year.
“After a decade of research carried out at dozens of institutions, we are seeing the OptIPuter take off on a global basis,” he said. “I look forward to working with many of the TeraGrid ’08 participants as they become early adopters of this innovative, high performance cyberinfrastructure-rebalancing the local analysis and network connectivity with the awesome growth NSF has made possible in the emerging petascale computers.”
In addition to Smarr and Papadopoulos, co-principal investigators on the OptIPuter initiative include Calit2’s Thomas DeFanti; Jason Leigh, from the University of Illinois at Chicago; and Mark Ellisman, from UC San Diego. The project manager is Maxine Brown, from the University of Illinois at Chicago. Andrew Chien, now Vice President of Research at Intel, served as the system software architect while he was at UCSD.
The OptIPuter project including 250 reports and publications.
California Institute for Telecommunications and Information Technology (Calit2)
San Diego Supercomputer Center (SDSC)
SDSC Rocks Project
Internet2 Dynamic Circuit Network
National Science Foundation
Doug Ramsey, Calit2 Communications
Laura Wolf, EVL / UIC
Warren R. Froelich, SDSC Communications
Jan Zverina, SDSC Communications
Date: June 11, 2008