Re: [thredds] Catalog example for AWS S3 resource?

• To: "H. Joe Lee" <hyoklee@xxxxxxxxxxxx>
• Subject: Re: [thredds] Catalog example for AWS S3 resource?
• From: Ethan Davis <edavis@xxxxxxxx>
• Date: Wed, 4 Mar 2020 09:52:45 -0700

Hi Joe,

[Sorry for the delayed response.]

The S3 work moved to the Unidata/netCDF-java repo in PR #173
<https://github.com/Unidata/netcdf-java/pull/173> ("S3 Support"). This PR
got merged into master a week or so ago and is available in the netCDF-Java
5.3.0-SNAPSHOT release (and will be in the upcoming 5.3.0 release). The
latest TDS code built with netCDF-Java 5.3.0-SNAPSHOT can be configured to
serve an individual netCDF file stored as an S3 object using a datasetRoot
configuration, e.g.

<?xml version="1.0" encoding="UTF-8"?>

<catalog name="Test TDS S3"

  xmlns="https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";

  xmlns:xlink="https://www.w3.org/1999/xlink";

  xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance";

  xsi:schemaLocation="
https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0

    https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.6.xsd";>

  <datasetRoot path="s3-test" location="s3://noaa-goes16" />

  <dataset name="Test GOES-16 S3" ID="testS3Grid"

     
urlPath="s3-test/ABI-L1b-RadC/2019/363/21/OR_ABI-L1b-RadC-M6C16_G16_s20193632101189_e20193632103574_c20193632104070.nc"

           dataType="Grid"/>

</catalog>

In this case, the datasetRoot location is the bucket name, and the urlPath
is the datasetRoot path combined with the key. We rely on the AWS Java SDK
(v2) to handle credentials, setting of region, etc. For now, you can set
the region by creating a credentials file ~/.aws/credentials that looked
like:

[default]

region=us-east-1

Which is how netCDF-java knows which region to use for bucket access. We
may look at other mechanisms to make that a bit more integrated into TDS
configuration but for now that should work.

Once the netCDF 5.3.0 release comes out, TDS snapshot builds will be built
with this capability. For now, you would need to build the TDS and
explicitly tell it to build with netCDF-Java 5.3.0-SNAPSHOT.

Cheers,

Ethan

On Tue, Feb 4, 2020 at 2:30 PM H. Joe Lee <hyoklee@xxxxxxxxxxxx> wrote:

> Hi,
>
>   Is it possible to serve netCDF data on AWS S3 using THREDDS?
>   I think it seems possible based on the S3 feature branch [1].
>
>   If so, can someone share an example THREDDS catalog configuration?
>
>   Regards,
>
> [1] https://github.com/Unidata/thredds/tree/feature/s3+hdfs
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
>

• Follow-Ups:
  • Re: [thredds] Catalog example for AWS S3 resource?
    • From: H. Joe Lee
  • Re: [thredds] Catalog example for AWS S3 resource?
    • From: David Neufeld

• References:
  • [thredds] Catalog example for AWS S3 resource?
    • From: H. Joe Lee