ESIP WM21 Hybrid Cloud Data Storage.pdf (917.27 kB)

Hybrid Cloud Data Storage

Download (917.27 kB)
posted on 21.01.2021, 16:53 by Ken Tanaka, David Neufeld
A hybrid approach to adding cloud storage to an existing on-premises data processing workflow with less effort will be described. We used a Python program to efficiently transfer data files from local directories to an AWS S3 bucket. Concepts of managing a manifest of files already processed to prevent sending duplicates as well as a description of filtering files will be covered. The end result is a collection of Crowdsourced Bathymetry data accessible for public use. This poster was presented at the 2021 Earth Science Information Partners (ESIP) Winter Meeting held virtually in January 2021.