ESIP WM21 Hybrid Cloud Data Storage.pdf (917.27 kB)

Hybrid Cloud Data Storage

Download (917.27 kB)
poster
posted on 21.01.2021, 16:53 by Ken Tanaka, David Neufeld
A hybrid approach to adding cloud storage to an existing on-premises data processing workflow with less effort will be described. We used a Python program to efficiently transfer data files from local directories to an AWS S3 bucket. Concepts of managing a manifest of files already processed to prevent sending duplicates as well as a description of filtering files will be covered. The end result is a collection of Crowdsourced Bathymetry data accessible for public use. This poster was presented at the 2021 Earth Science Information Partners (ESIP) Winter Meeting held virtually in January 2021.

History

Licence

Exports