NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Simon,
The underlying format of netcdf-4 is supported by the HDF5 library and
other support libraries. It is quite possible that the read timing
differences are coming from changes in the HDF5 layer, not the netcdf-C
library code itself. In your binary search, please keep track of which
HDF5 library versions are in use.
You might also pay attention to the libz version. I will be surprised if
it turns out you found a problem in libz.
--Dave
On Tue, Dec 13, 2016 at 5:55 PM, Simon (Vsevolod) Ilyushchenko <
simonf@xxxxxxxxxx> wrote:
> Sure. The rest is the same. I'll do binary search next.
>
> First:
>
> netcdf a {
> dimensions:
> depth = 2 ;
> lat = 2001 ;
> lon = 4500 ;
> time = UNLIMITED ; // (1 currently)
> variables:
>
<snip>
short water_u(time, depth, lat, lon) ;
> water_u:long_name = "Eastward Water Velocity" ;
> water_u:standard_name = "eastward_sea_water_velocity" ;
> water_u:units = "m/s" ;
> water_u:_FillValue = -30000s ;
> water_u:missing_value = -30000s ;
> water_u:scale_factor = 0.001f ;
> water_u:add_offset = 0.f ;
> water_u:NAVO_code = 17 ;
> water_u:_Storage = "chunked" ;
> water_u:_ChunkSizes = 1, 1, 976, 2196 ;
> water_u:_DeflateLevel = 9 ;
> water_u:_Endianness = "little" ;
>
<snip>
:history = "Fri Nov 20 10:01:58 2015: ncks -4 --deflate 9
> -v water_u,water_v -d depth,1,2,1 hycom_glb_911_2015110200_t000_uv3z.nc
> hycom_glb_911_2015110200_t000_uv3z-test.nc\narchv2ncdf3z" ;
> :NCO = "4.4.2" ;
> :_Format = "netCDF-4" ;
>
<snip>
On Tue, Dec 13, 2016 at 10:04 AM, Ed Hartnett <edwardjameshartnett@xxxxxxxxx
> > wrote:
>
>> Can you do an ncdump with the -s option, so we can see the chunksizes?
>>
>> On Tue, Dec 13, 2016 at 10:54 AM, Chris Barker <chris.barker@xxxxxxxx>
>> wrote:
>>
>>> if I understand the question, this is read times -- is this in fact with
>>> exactly the same files? In which case the chunking is already set.
>>>
>>> less performant decompression??
>>>
>>> sounds like the binary search is needed.
>>>
>>> Is EVERYTHING else the same? disk system, OS, etc?
>>>
>>> -CHB
>>>
>>>
>>> On Tue, Dec 13, 2016 at 9:21 AM, Charlie Zender <zender@xxxxxxx> wrote:
>>>
>>>> Hello Simon,
>>>>
>>>> Since both files are netCDF4 compressed that
>>>> means they use chunking. My wild guess is that
>>>> different chunking defaults cause the observed
>>>> change in dumping time. You can see the
>>>> chunk sizes employed with ncdump -s or ncks --hdn,
>>>> and you can play with the chunk sizes/policy
>>>> with either.
>>>>
>>>> Charlie
>>>>
>>>
netcdfgroup archives: