The Open Data Schema Workshop held in conjunction with the Data-driven Plasma Science and Engineering Workshop in Oxford, UK takes a first step at establishing consensus dor data formats across studies, experiments and simulations. The workshop gathers domain scientists and computer scientists from Europe and the US and provides an understanding of the I/O requirements of advanced HPC simulations and of the challenges unique to specific configurations(e.g. disruption avoidance and mitigation in tokamaks).

My talk focuses on surveying common data formats and unified solutions for code coupling between HPC and AI and the schema requirements to such workflows. Schemas in addition to self described data give the developer the ability to confine the domain of the file contents and protects data integrity by providing checks for writing data outside the input domain. The talk goes over challenges in designing general ML schemas, in connecting them to ontologies for attaching meaning to datasets and in handling HPC specific needs like parallel I/O, partial compression and real time visualization.

Link to the event: https://www.physics.ox.ac.uk/events/data-driven-plasma-science-and-engineering-workshop