I/O is one of the most challenging issues for many current petascale high performance computing systems. The issue is expected to become dire in future systems. In situ analyses has been proposed as a promising solution to glean faster insights and reduce the amount of data to storage. A critical challenge here is the sparse data pattern produced to be written out. We evaluate the performance of current I/O mechanisms on two diverse systems, namely the Mira IBM BG/Q system and the Hopper Cray Xe6 system, for sparse data patterns, and propose an I/O mechanism leveraging topology- aware data aggregation together with necessary abstraction to hide the underlying filesystem complexity to yield multi-fold performance up to 128K cores.

PDF

Bui, H., Vishwanath, V., Leigh, J., Papka M., Improving I/O Performance for Sparse Data Movement in Leadership Systems, The Proceedings of the 8th Parallel Data Storage Workshop, Denver, CO, (poster), ACM, IEEE, November 18th, 2013.