[netcdfgroup] variable size limitation in the 64-bit Offset Format

• To: "netcdfgroup@xxxxxxxxxxxxxxxx List" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] variable size limitation in the 64-bit Offset Format
• From: Wei-keng Liao <wkliao@xxxxxxxxxxxxxxxxxxxxx>
• Date: Sun, 11 Jun 2017 13:49:31 -0500

Dear NetCDF community,

A question about the variable size limitation in NetCDF 64-bit Offset Format.

According to its format specification document (URL below), the last section
"6.3 The 64-bit Offset Format Variant" includes the following statement
"Each fixed-size variable and the data for one record's worth of each record
variable are still limited in size to a little less that 4 GiB. The rationale
for this limitation is to permit aggregate access to all the data in a netCDF
variable (or a record's worth of data) on 32-bit platforms."
https://cdn.earthdata.nasa.gov/conduit/upload/496/ESDS-RFC-011v2.00.pdf

When examining closely the Format Specification presented in Section 6.2,
I found that the 64-bit offset format can actually store more than one
large (> 4GiB) variable, as long as all the variable's dimensions are
smaller than 2^32-4. In fact, PnetCDF 1.5.0 ignores this limitation and
one can use it to create files storing more than one large variable. The
files can be read by the NetCDF utility programs, ncdump, and paraview
correctly. See discussion threads from a ticket reported in
http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2017-March/001911.html

It turns out NetCDF implementation of nc_open does not check this limitation
until a recent pull request https://github.com/Unidata/netcdf-c/pull/393

My question is the limitation rationale mentioned in the document.
I believe one can still access all the data of a variable or a record
on 32-bit machines when the limitation is lifted. Is this not the case?
Otherwise, I am wondering whether this is a limitation netCDF applications
prefer to have, for any practical reason that I am happy to learn.

regards,

Wei-keng Liao