Large Job Management in CDMS

The oceanographic community continues to need a generalized data collocation capability for satellite and/or in situ observations that is publicly accessible and provides flexibility and reproducibility for use cases ranging from open science to satellite retrieval calibration and validation. The Cloud-based Data Match-Up Service (CDMS) is a collaborative effort between NASA JPL, COAPS, NCAR, and Saildrone to address this need. With an exponential increase in the volume of satellite data, the CDMS architecture is designed to be scalable and to leverage the elasticity of the cloud. The differences between remote sensing data at various data processing levels and the heterogeneous nature of in situ data makes developing a generic system challenging, but CDMS is designed with the consideration of making it possible to efficiently onboard new datasets. Asynchronous processing serves as a critical component of the CDMS architecture to allow for efficient handling of large data requests. In addition to an overview of CDMS, the authors will demonstrate the asynchronous processing to address and analyze wind to SST relationships during an ocean heat wave.