Re: [thredds] netcdf files (.nc) with data in them compressed with bzip2

• To: Artak Zakaryan <artak.zakaryan@xxxxxxxxxxxxxxx>
• Subject: Re: [thredds] netcdf files (.nc) with data in them compressed with bzip2
• From: "Antonio S. Cofino" <cofinoa@xxxxxxxxx>
• Date: Wed, 3 Sep 2025 23:21:09 +0200

Hi Artak,

By default, TDS/netCDF-Java only supports internal deflate (zlib)
compression. To use other HDF5 filters such as bzip2 you need to configure
TDS to run with the netCDF-C JNI backend. That requires installing netCDF-C
and HDF5 on the server, making sure the same plugin mechanism used when the
files were created is available, and setting the environment variable
HDF5_PLUGIN_PATH so that the external filters can be found at runtime. If
TDS is configured to use the same netCDF-C library and plugin environment
that was used to generate the files, it should be able to access them
transparently.

https://docs.unidata.ucar.edu/netcdf-c/current/nc_filters_quickstart.html

If you can share one of your NetCDF-4 files that uses bzip2 internally, I
can try to see if there is a viable solution with the current setup.

Best,
Antonio

On Wed, 3 Sept 2025, 09:13 Artak Zakaryan, <artak.zakaryan@xxxxxxxxxxxxxxx>
wrote:

> Hi Antonio,
>
>
>
> Thank you for your reply.
>
>
>
> My scenario is internal compression with bzip2. It seems THREDDS does not
> support it out of box, at least on our version of THREDDS.
>
> We are using THREDDS 5.5 with netcdf 5.4.1 and java 11.
>
> I wonder if there are prerequisites or specific settings/configurations to
> support bzip2 internal compression. Or is there documentation or
> information on this? I came across some documentation about compressed
> files but could not find information regarding internal compression.
>
>
>
> Regards,
>
>
>
> Artak
>
>
>
> *From:* Antonio S. Cofino <cofinoa@xxxxxxxxx>
> *Sent:* Tuesday, 2 September 2025 8:06 AM
> *To:* Artak Zakaryan <artak.zakaryan@xxxxxxxxxxxxxxx>
> *Cc:* THREDDS community <thredds@xxxxxxxxxxxxxxxx>
> *Subject:* Re: [thredds] netcdf files (.nc) with data in them compressed
> with bzip2
>
>
>
> Hi Artak,
>
>
>
> If you access a .nc.bz2 file through the HTTP File Service, TDS will just
> serve the compressed file as is.
>
>
>
> If you use data access services like OPeNDAP or the NetCDF Subset Service,
> netCDF Java will first decompress the .bz2 on the server and then serve the
> uncompressed dataset.
>
>
>
> Note that this is different from the internal compression available in
> NetCDF-4/HDF5 files (for example deflate/zlib), which is handled
> transparently and does not require any server side decompression step.
>
>
>
> Regards,
>
>
>
> Antonio
>
>
>
> On Mon, 1 Sept 2025, 09:02 Artak Zakaryan, <artak.zakaryan@xxxxxxxxxxxxxxx>
> wrote:
>
> Hello THREDDS Team,
>
>
>
> Can someone please advise if THREDDS server supports netcdf files (.nc)
> with data in them compressed with bzip2?
>
> The files in question are .nc files but the data in the files is
> compressed with bzip2. I have done some research, and it looks like THREDDS
> supports .bzip2 files but could not find any information about .nc files
> with data in them compressed with bzip2.
>
>
>
> Thank you.
>
>
>
> Artak
>
>
>
>
>
> ____________________________________________________________________
> NOTE: All exchanges posted to NSF Unidata maintained email lists are made
> publicly available online. Users who post to any of the lists we maintain
> are reminded to remove any personal information that they do not want to be
> made public.
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information, to unsubscribe, or change your membership options,
> visit: https://mailman.unidata.ucar.edu//listinfo/thredds/
>
>

• References:
  • [thredds] netcdf files (.nc) with data in them compressed with bzip2
    • From: Artak Zakaryan
  • Re: [thredds] netcdf files (.nc) with data in them compressed with bzip2
    • From: Antonio S. Cofino