Re: [netcdfgroup] reducing read time for large data

• To: Cathy Smith <cathy.smith@xxxxxxxx>
• Subject: Re: [netcdfgroup] reducing read time for large data
• From: Ed Hartnett <edwardjameshartnett@xxxxxxxxx>
• Date: Tue, 21 Jan 2020 13:52:43 -0700

The type matters because the size of the type matters.

For example, if you could store the data as NC_BYTE, then to read 1000
values means to read 1000 bytes. But if you store the same data as NC_INT,
then it's 4000 bytes for 1000 values, and it would take about 4 times as
long to read from the disk.

So you should store your data in the smallest type in which it will
comfortably fit. ;-)

Keep on netCDFing!

Ed Hartnett

On Tue, Jan 21, 2020 at 1:07 PM Cathy Smith via netcdfgroup <
netcdfgroup@xxxxxxxxxxxxxxxx> wrote:

> All
>
> The question came up if you have data that is in a smaller file but the
> same amount of data as a larger file (say both dimensioned
> 3600,1800,24levels,12months,40years), would it be quicker to read? I am
> not talking about netCDF4 compression at all but instead (I assume)
> variable type. So, if I wanted to reduce read time, should I store the
> data as byte or integer if that is possible, keeping the precision of
> the data? Or does the variable type that is used to store the data not
> matter in netCDF as far as read speed? I am using linux if that matters.
>
> Thanks for any advice.
>
> Cathy Smith
>
> --
> ----------------------------------------------
> NOAA/ESRL PSD and CU CIRES
> 303-497-6263
> https://www.esrl.noaa.gov/psd/people/cathy.smith/
>
> Emails about data/webpages may get quicker responses from emailing
> esrl.psd.data@xxxxxxxx
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
>

• References:
  • [netcdfgroup] reducing read time for large data
    • From: Cathy Smith