You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a AWSDownloader that can fetch the data from AWS S3 storage. It should support an authentication token, ideally with the option to read it from an environment variable. See instructions for adding such a downloader in #382 (comment).
Original issue 👇🏾
Description of the desired feature:
Data can be stored in cloud hosted buckets, s3, google storage, Azure, ...
These can provide either urls (I believe per-signining is possible) or some bucket location + authentification for example see the boto3 s3 python SDK
Minio can also be used docker image to run s3 locally for testing if better
Are you willing to help implement and maintain this feature?
Not sure I know enough about pooch (first time contribution and usage) to be able to do anything of use but I could possibly help out with guidance / provide further info
The text was updated successfully, but these errors were encountered:
Based off a similar need, I created a custom GSDownloader that downloads files from Google Cloud Storage. It's focused on files that require authentication. It uses the google-cloud-storage API for the download. Not sure if this request was for a more generalizable BucketDownloader, or something specific for AWS, like S3Downloader, but I wanted to link it here given the high overlap.
Add a
AWSDownloader
that can fetch the data from AWS S3 storage. It should support an authentication token, ideally with the option to read it from an environment variable. See instructions for adding such a downloader in #382 (comment).Description of the desired feature:
Data can be stored in cloud hosted buckets, s3, google storage, Azure, ...
These can provide either urls (I believe per-signining is possible) or some bucket location + authentification for example see the boto3 s3 python SDK
I am not sure on the data size but here is an example of downloading public data from s3: https://github.com/planet-os/notebooks/blob/master/aws/era5-s3-via-boto.ipynb
Minio can also be used docker image to run s3 locally for testing if better
Are you willing to help implement and maintain this feature?
Not sure I know enough about
pooch
(first time contribution and usage) to be able to do anything of use but I could possibly help out with guidance / provide further infoThe text was updated successfully, but these errors were encountered: