NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
<stephen.pascoe@xxxxxxxxxx> writes: > I'd like to voice a word of caution on the inclusion of optional szip > compression in netCDF4.1. Although I'm sure szip has technically merit > it further fragments the "netcdf format". The NetCDF FAQ lists 4 netCDF > format variants and szip will make that 6: > > 1. netCDF3 format > 2. netCDF3 64-bit offset format > 3. netCDF4 format (zlib or no compression) > 4. netCDF4 classic model (zlib or no compression) > 5. netCDF4 format (szip) > 6. netCDF4 classic model (szip) > > The reason I lump zlib with no compression is that when building NetCDF > libraries you need to decide whether to include szip in the build, with > the inherent license limitations it imposes. Therefore users will have > netCDF4 tools with or without szip support. > > This is bound to discourage the adoption of netCDF4 in general and could > cause many data archival problems down the line. For instance, from my > perspective I am now asking myself what the CMIP5 archive's policy > should be on netCDF4 szip. > > I'm not sure how to resolve this problem here but it's worth exposing. Howdy Stephen! The multiplicity of binary formats is certainly a concern, but the HDF5 format (which is the base for all the non-classic formats) is a well-known and stable software product, trusted in its own right by such organizations as NASA for long-term archive of scientific data. Restricting CMIP contributors from using szip makes perfect sense. (But everyone who asked me for it was from the climate community!) Yet the szlib library is freely available, and license restrictions on commercial writers of data will not apply to anyone in the CMIP5 community. The netCDF APIs (C, Fortran, C++, and Java) can all now read files that no longer fall into the strict format categories. This blurs the lines considerably. It is going to be up to managers of archives like CMIP to decide what formats and conventions are acceptable for each archive. For example, a HDF4 SD file can now be read exactly as if it were a classic format netCDF file. Assuming the proper CF conventions were used, this would then work with all software written for CMIP5 data. But I assume that you don't want people to send you HDF4 files, even though they are now, arguably, netCDF files too. I understand your point that this is the first case in which a user can produce a file with netCDF that cannot be read by other current netcdf libraries unless they are built with szlib. But why can't they be rebuilt, if necessary? This seems similar to the issue of netCDF-4/HDF5 files and pre-existing netCDF installations, which must upgrade their software to read the files. It is all freely available software. I have also heard that the szlib patent has expired, and that an unrestricted version may be available - I will try to run down that rumor... Thanks, Ed -- Ed Hartnett -- ed@xxxxxxxxxxxxxxxx
netcdfgroup archives: