Re: [netcdfgroup] Unexpectedly large netCDF4 files from python

• To: "dmh@xxxxxxxx" <dmh@xxxxxxxx>
• Subject: Re: [netcdfgroup] Unexpectedly large netCDF4 files from python
• From: Val Schmidt <vschmidt@xxxxxxxxxxxx>
• Date: Tue, 5 Apr 2016 15:48:46 -0400

To answer your first two questions:
Yes, netcdf4
No, fill value is not set.

And this question:
Simply because I’m not storing strings in it. 

I’ve done a few more tests and if I specify the variable sizes explicitly the 
resulting file size is small, as I’d expect. So the overhead is coming from not 
specifying a fixed variable size. 

-Val


> On Apr 5, 2016, at 3:31 PM, dmh@xxxxxxxx wrote:
> 
> BTW and out of curiosity, why are you using vlen rather than string?
> =Dennis
> 
> 
> On 4/5/2016 12:44 PM, Val Schmidt wrote:
>> Hello netcdf folks,
>> 
>> I’m testing some python code for writing sets of timestamps and variable 
>> length binary blobs to a netcdf file and the resulting file size is 
>> perplexing to me.
>> 
>> The following segment of python code creates a file with just two variables, 
>> “timestamp” and “data”, populates the first entry of the timestamp variable 
>> with a float and the corresponding first entry of the data variable with an 
>> array of 100 unsigned 8-bit integers. The total amount of data is 108 bytes.
>> 
>> But the resulting file is over 73 MB in size. Does anyone know why this 
>> might be so large and what I might be doing to cause it?
>> 
>> Thanks,
>> 
>> Val
>> 
>> 
>> from netCDF4 import Dataset
>> import numpy
>> 
>> f = Dataset('scratch/text3.nc','w')
>> 
>> dim = f.createDimension('timestamp_dim',None)
>> data_dim = f.createDimension('data_dim',None)
>> 
>> data_t = f.createVLType('u1','variable_data_t’)
>> 
>> timestamp = f.createVariable('timestamp','d','timestamp_dim')
>> data = f.createVariable('data',data_t,'data_dim’)
>> 
>> timestamp[0] = time.time()
>> data[0] = uint8( numpy.ones(1,100))
>> 
>> f.close()
>> 
>> ------------------------------------------------------
>> Val Schmidt
>> CCOM/JHC
>> University of New Hampshire
>> Chase Ocean Engineering Lab
>> 24 Colovos Road
>> Durham, NH 03824
>> e: vschmidt [AT] ccom.unh.edu <http://ccom.unh.edu/> <http://ccom.unh.edu 
>> <http://ccom.unh.edu/>>
>> m: 614.286.3726
>> 
>> 
>> 
>> 
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
>> For list information or to unsubscribe,  visit: 
>> http://www.unidata.ucar.edu/mailing_lists/ 
>> <http://www.unidata.ucar.edu/mailing_lists/>
------------------------------------------------------
Val Schmidt
CCOM/JHC
University of New Hampshire
Chase Ocean Engineering Lab
24 Colovos Road
Durham, NH 03824
e: vschmidt [AT] ccom.unh.edu
m: 614.286.3726

• References:
  • [netcdfgroup] Unexpectedly large netCDF4 files from python
    • From: Val Schmidt
  • Re: [netcdfgroup] Unexpectedly large netCDF4 files from python
    • From: dmh@xxxxxxxx