NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi, Ward! I'm trying to convert a pure Zarr to GeoTIFF via GDAL and HDF5 via NCO using the new netCDF library. However, I have some trouble in following your NUG: "NCZarr versus Pure Zarr. The NCZARR format extends the pure Zarr format by adding extra objects such as _.nczarr_ and _.ncvar_. It is possible to suppress the use of these extensions so that the netcdf library can read and write a pure zarr formatted file. This is controlled by using mode=nczarr,zarr combination. The primary effects of using pure zarr are described in the Translation Section." I tried a few different modes but none worked for reading Zarr file: ~/src/netcdf-c-4.8.0/ncdump$ ./ncdump -h "file:///efs/power_901_constants.zarr/#mode=zarr,file" /home/ubuntu/src/netcdf-c-4.8.0/ncdump/.libs/ncdump: file:///efs/power_901_constants.zarr/#mode\ =zarr,file: NetCDF: internal library error; Please contact Unidata support ~/src/netcdf-c-4.8.0/ncdump$ ./ncdump -h "file:///efs/power_901_constants.zarr/#mode=nczarr,file" ncdump: zclose.c:223: zclose_type: Assertion `type && type->format_type_info != NULL' failed. Aborted (core dumped) Am I correct in using the "mode=zarr,file" for pure Zarr? By the way, ncdump works fine on sample test nczarr files. Regards, - Joe --- Deploy XaaS quickly using HDF in Distributed Cloud. On Tue, Mar 30, 2021 at 5:39 PM Ward Fisher <wfisher@xxxxxxxx> wrote: > Hello all, > > The Unidata netCDF team is happy to announce the availability of netCDF > 4.8.0, now with Zarr support (nczarr)! > > This release extends the netcdf-c library to provide access to cloud > storage (e.g. Amazon S3 by providing a mapping from a subset of the full > netCDF Enhanced (aka netCDF–4) data model to a variant of the Zarr data > model. The NetCDF variant is called NCZarr. It is possible for other > implementations, such as Python Zarr, to read this variant. This > implementation can optionally read and write using the pure version 2 Zarr > storage format. It also can optionally support the XArray _ARRAY_DIMENSIONS > extension to Zarr. > > In addition to this new functionality, the release notes appended to the > end of this announcement detail some of the other changes, improvements and > bug fixes included in this release. > > Please see the following links for more information: > > - Github release page: > https://github.com/Unidata/netcdf-c/releases/tag/v4.8.0 > - v4.8.0 Documentation: > https://www.unidata.ucar.edu/software/netcdf/docs > - NCZarr Documentation: > > https://www.unidata.ucar.edu/software/netcdf/documentation/NUG/nczarr_head.html > > Kudos to Dennis Heimbigner for the technical implementation of nczarr, and > our thanks to the members of the Zarr working group and our community who > have collaborated with us so patiently on this! > > Our next priority is to get caught up on the backlog of PRs and issues > blocking the next releases of netCDF-Fortran and netCDF-CXX4. > > Thank you all very much, have a great day! > > -Ward > > Ward Fisher > wfisher@xxxxxxxx > UCAR/Unidata - Software Engineer > NetCDF Team Lead > Full Release Notes4.8.0 - March 30, 2021 > > - [Enhancement] Bump the NC_DISPATCH_VERSION from 2 to 3, and as a > side effect, unify the definition of NC_DISPATCH_VERSION so it only needs > to be defined in CMakeLists.txt and configure.ac. See Github #1945 > <https://github.com/Unidata/netcdf-c/pull/1945> for more information. > - [Enhancement] Provide better cross platform path name management. > This converts paths for various platforms (e.g. Windows, MSYS, etc.) so > that they are in the proper format for the executing platform. See Github > #1958 <https://github.com/Unidata/netcdf-c/pull/1958> for more > information. > - [Bug Fixes] The nccopy program was treating -d0 as turning deflation > on rather than interpreting it as “turn off deflation”. See Github > #1944 <https://github.com/Unidata/netcdf-c/pull/1944> for more > information. > - [Enhancement] Add support for storing NCZarr data in zip files. See > Github > #1942 <https://github.com/Unidata/netcdf-c/pull/1942> for more > information. > - [Bug Fixes] Make fillmismatch the default for DAP2 and DAP4; too > many servers ignore this requirement. > - [Bug Fixes] Fix some memory leaks in NCZarr, fix a bug with long > strides in NCZarr. See Github #1913 > <https://github.com/Unidata/netcdf-c/pull/1913> for more information. > - [Enhancement] Add some optimizations to NCZarr, dosome cleanup of > code cruft, add some NCZarr test cases, add a performance test to NCZarr. > See Github #1908 <https://github.com/Unidata/netcdf-c/pull/1908> for > more information. > - [Bug Fix] Implement a better chunk cache system for NCZarr. The > cache now uses extendible hashing plus a linked list for provide a > combination of expandibility, fast access, and LRU behavior. See Github > #1887 <https://github.com/Unidata/netcdf-c/pull/1887> for more > information. > - [Enhancement] Provide .rc fields for S3 authentication: > HTTP.S3.ACCESSID and HTTP.S3.SECRETKEY. > - [Enhancement] Give the client control over what parts of a DAP2 URL > are URL encoded (i.e. %xx). This is to support the different decoding rules > that servers apply to incoming URLS. See Github #1884 > <https://github.com/Unidata/netcdf-c/pull/1844> for more information. > - [Bug Fix] Fix incorrect time offsets from ncdump -t, in some cases > when the time units attribute contains both a *non-zero* time-of-day, > and a time zone suffix containing the letter “T”, such as “UTC”. See Github > #1866 <https://github.com/Unidata/netcdf-c/pull/1866> for more > information. > - [Bug Fix] Cleanup the NCZarr S3 build options. See Github #1869 > <https://github.com/Unidata/netcdf-c/pull/1869> for more information. > - [Bug Fix] Support aligned access for selected ARM processors. See Github > #1871 <https://github.com/Unidata/netcdf-c/pull/1871> for more > information. > - [Documentation] Migrated the documents in the NUG/ directory to the > dedicated NUG repository found at https://github.com/Unidata/netcdf > - [Bug Fix] Revert the internal filter code to simplify it. From the > user’s point of view, the only visible change should be that (1) the > functions that convert text to filter specs have had their signature > reverted and renamed and have been moved to netcdf_aux.h, and (2) Some > filter API functions now return NC_ENOFILTER when inquiry is made about > some filter. Internally, the dispatch table has been modified to get rid of > the complex structures. > - [Bug Fix] If the HDF5 byte-range Virtual File Driver is available > )HDf5 1.10.6 or later) then use it because it has better performance than > the one currently built into the netcdf library. > - [Bug Fix] Fixed byte-range support with cURL > 7.69. See [ > https://github.com/Unidata/netcdf-c/pull/1798]. > - [Enhancement] Added new test for using compression with parallel > I/O: nc_test4/tst_h_par_compress.c. See [ > https://github.com/Unidata/netcdf-c/pull/1784]. > - [Bug Fix] Don’t return error for extra calls to nc_redef() for > netCDF/HDF5 files, unless classic model is in use. See [ > https://github.com/Unidata/netcdf-c/issues/1779]. > - [Enhancement] Added new parallel I/O benchmark program to mimic NOAA > UFS data writes, built when –enable-benchmarks is in configure. See [ > https://github.com/Unidata/netcdf-c/pull/1777]. > - [Bug Fix] Now allow szip to be used on variables with unlimited > dimension [https://github.com/Unidata/netcdf-c/issues/1774]. > - [Enhancement] Add support for cloud storage using a variant of the > Zarr storage format. Warning: this feature is highly experimental and is > subject to rapid evolution [ > > https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in > ]. > - [Bug Fix] Fix nccopy to properly set default chunking parameters > when not otherwise specified. This can significantly improve performance in > selected cases. Note that if seeing slow performance with nccopy, then, as > a work-around, specifically set the chunking parameters. [ > https://github.com/Unidata/netcdf-c/issues/1763]. > - [Bug Fix] Fix some protocol bugs/differences between the netcdf-c > library and the OPeNDAP Hyrax server. Also cleanup checksum handling [ > https://github.com/Unidata/netcdf-c/issues/1712].* [Bug Fix] > IMPORTANT: Ncgen was not properly handling large > data sections. The problem manifests as incorrect ordering of > data in the created file. Aside from examining the file with > ncdump, the error can be detected by running ncgen with the -lc > flag (to produce a C file). Examine the file to see if any > variable is written in pieces as opposed to a single call to > nc_put_vara. If multiple calls to nc_put_vara are used to write > a variable, then it is probable that the data order is > incorrect. Such multiple writes can occur for large variables > and especially when one of the dimensions is unlimited. > - [Bug Fix] Add necessary _ > *declspec declarations to allow compilation of netcdf library without > causing errors or (*declspec related) > warnings [https://github.com/Unidata/netcdf-c/issues/1725]. > - [Enhancement] When a filter is applied twice with different > parameters, then the second set is used for writing the dataset > [https://github.com/Unidata/netcdf-c/issues/1713]. > - [Bug Fix] Now larger cache settings are used for sequential HDF5 > file creates/opens on parallel I/O capable builds; see Github #1716 > <https://github.com/Unidata/netcdf-c/issues/1716> for more information . > - [Bug Fix] Add functions to libdispatch/dnotnc4.c to support > dispatch table operations that should work for any dispatch > table, even if they do not do anything; functions such as > nc_inq_var_filter [https://github.com/Unidata/netcdf-c/issues/1693]. > - [Bug Fix] Fixed a scalar annotation error when scalar == 0; see Github > #1707 <https://github.com/Unidata/netcdf-c/issues/1707> for more > information. > - [Bug Fix] Use proper CURLOPT values for VERIFYHOST and VERIFYPEER; > the semantics for VERIFYHOST in particular changed. Documented in > NUG/DAP2.md. See [https://github.com/Unidata/netcdf-c/issues/1684]. > - [Bug Fix][cmake] Correct an issue with parallel filter test logic in > CMake-based builds. > - [Bug Fix] Now allow nc_inq_var_deflate()/nc_inq_var_szip() to be > called for all formats, not just HDF5. Non-HDF5 files return NC_NOERR and > report no compression in use. This reverts behavior that was changed in the > 4.7.4 release. See [https://github.com/Unidata/netcdf-c/issues/1691]. > - [Bug Fix] Compiling on a big-endian machine exposes some missing > forward delcarations in dfilter.c. > > _______________________________________________ > NOTE: All exchanges posted to Unidata maintained email lists are > recorded in the Unidata inquiry tracking system and made publicly > available through the web. Users who post to any of the lists we > maintain are reminded to remove any personal information that they > do not want to be made public. > > > netcdfgroup mailing list > netcdfgroup@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > https://www.unidata.ucar.edu/mailing_lists/ >
netcdfgroup archives: