This half-day tutorial provides a comprehensive introduction to building blocks of complex scientific workflows, including I/O, remote data access, and visualization. The tutorial features live demonstrations involving a realistic simulation, data analysis and visualization, which illustrate how to use these tools in a plug-and-play manner. Attendees will see a workflow composed of simulation, data analysis and data visualization and run in various configurations of file-based and in situ setups.

Abstract: The tutorial will demonstrate how the presented tools can be used transparently for streaming workflows vs. file-based workflows. Furthermore, these tools can be used over the WAN to allow a client running at one location to access remote data for analyzing and visualizing the data locally. The tutorial will introduce the participants to four main US Department of Energy Exascale Computing Project tools and technologies including:

• ADIOS which provides a publish/subscribe I/O abstraction unifying storage and staging I/O.
• MGARD which is an error-controlled compression and refactor software for floating-point scientific data
• Fides a visualization data model and schema which allows ADIOS data to be easily ingested in a variety of visualization tools, and
• ParaView a scientific visualization framework for both file-based post-processing as well as in situ visualization.

These tools are integrated together to allow for both file-based and in-memory streaming analysis. Furthermore, the tutorial demonstrates a remote data access service, WANDS, that uses ADIOS to transfer UK Fusion experimental data to compute clusters at University of Cambridge.

Tutorial slides from a similar presentation from SC23 can be found here.

Related paper:
Streaming Data in HPC Workflows Using ADIOS
Greg Eisenhauer, Norbert Podhorszki, Ana Gainaru, Scott Klasky, et al,
arXiv preprint arXiv:2410.00178 2025