Big Data Partners Data Broker Update

2019-08-29T11:49:49Z (GMT) by Jonathan Brannock Otis Brown
NOAA’s Big Data Project (2015 - 2019) was an experiment between the US government and five Cooperative Research and Development Agreement partners in utilization of Cloud resources for public data sets. NC State University's Cooperative Unit for Climate and Satellites - NC (CICS-NC) was the BDP Data Broker between the USG and Amazon Web Services, Google Cloud Services, IBM, Microsoft Azure and the Open Commons Consortium. In this role CICS-NC facilitated data transfers from NOAA to the Cloud partners, resolved operational problems, ensured data integrity, certified Cloud partner holdings, added new datasets, informed Cloud partners of source data irregularities including missing datasets, and, helped mesh contrasting cultures. In this presentation we review these efforts, including several different data transfer modalities utilized and present statistics on usage of Cloud partner BDP holdings. This presentation was given in July 2019 at the Earth Science Information Partners (ESIP) Summer Meeting held in Tacoma, Washington.