Re: using netCDF for storing data from realtime source

• To: "Thaden, John J" <ThadenJohnJ@xxxxxxxx>
• Subject: Re: using netCDF for storing data from realtime source
• From: Ed Hartnett <ed@xxxxxxxxxxxxxxxx>
• Date: Mon, 29 Jan 2007 10:51:52 -0700

"Thaden, John J" <ThadenJohnJ@xxxxxxxx> writes:

> I must say this topic has caught me by surprise.  I'd gotten the impression
>>From reading that the netCDF binary file type was created specifically to
> gracefully handle real time acquisition.  Those variables having as one of
> their dimensions the unlimited dimension are, to my understanding, located
> physically at the bottom of the file so that new data can be appended to them
> without rewriting existing portions of the file, that, if one such variable
> is to receive new data that will cause it to exceed the current actual length
> of the unlimited dimension, then all variables sharing the unlimited
> dimension will be lengthened, the one with an actual datum, others with a
> value representing "missing data", to be replaced in later if real data
> became available for those variables as well.  The processing overhead of
> writing missing values would seem to be a small price to pay for having a
> file that nicely organizes a collection of variables, and array-type
> variables, such that all can be addressably accessed.  Can this be done with
> text files?    
>
> John Thaden
> Research Assistant Professor of Geriatrics
> College of Medicine
> University of Arkansas for Medical Sciences
> Little Rock
>

I believe the best answer would be to simply try it and see.

As Russ points out in another message, the difference between writing
a netCDF and a binary file (other than the initial set-up of metadata
in the file) is that the netcdf file will write fill-values and will
keep a record count up to date.

If I were really concerned about performance I would turn off the use
of fill values.

I would suspect that any ASCII output scheme is going to be slower
than netCDF.

But mostly I would just get good enough hardware to handle the data
flow.

Thanks,

Ed
-- 
Ed Hartnett  -- ed@xxxxxxxxxxxxxxxx

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================

• References:
  • using netCDF for storing data from realtime source
    • From: Thaden, John J