Enhancements to SciStream, a platform to enable memory-to-memory data streaming between federated instruments

January 8th, 2024 - May 15th, 2025

Categories: Networking, Software, Supercomputing, Data Science, High Performance Computing


EVL graduate research assistants will work with Argonne National Laboratory scientists to develop enhancements to SciStream, a platform that enables memory-to-memory data streaming between federated scientific instruments residing in different security domains with no direct external network connectivity. We will explore utilizing advanced packet processing technologies such as DPDK, programmable processors such as DPU and programmable networking to improve the performance of SciStream data plane. We will also benchmark SciStream performance for different deployment scenarios including deploying SciStream alongside Globus Connect Servers on existing DTNs and deploying SciStream on dedicated streaming DTNs. We will also benchmark the performance of SciStream between Argonne’s Advanced Photon Source and the Argonne Leadership Computing Facility.

Research activities include:
1) Research and integrate advanced packet processing and programmable data processing in SciStream and characterize performance in testbed environments.
2) Evaluate the performance of SciStream in testbed and production environments for various deployment scenarios.
3) Create reports and contribute to the development of scholarly manuscripts.