Re: [netcdfgroup] Structure at the file level?

• To: Rick Hedin <rickh@xxxxxxxxxx>
• Subject: Re: [netcdfgroup] Structure at the file level?
• From: robl@xxxxxxxxxxx (Robert Latham)
• Date: Fri, 14 Dec 2007 16:30:56 -0600

On Thu, Dec 13, 2007 at 10:14:04AM -0600, Rick Hedin wrote:
> Greetings! 
> 
> I am making an information collector that will collect information from 
> a bunch of processes on a bunch of machines.  Initially, each process 
> will write to it's own file on it's machine.  Eventually, each process 
> will contribute to a collector process on one machine using UDP. 

Have you considered using MPI for the data exchange?  Might be easier
and more portable than UDP.

> I'd like to use netCDF because of its self-describing nature, and 
> because the information is uniform across machine architectures.  But I 
> wonder if netCDF can interpret a record in isolation, or if it depends 
> on an overall file structure.  I will end up with one big file 
> containing records from a bunch of different sources. 
>
> If a netCDF file is just a few records of common header information at 
> the top, followed by a bunch of records, each of which is in an allowed 
> format, but which are not in any particular order, I don't anticipate 
> any problems. 
> 
> Has anybody collected records from a bunch of sources and put them all 
> together in one file? 

This is the usual mode of operation for a parallel (MPI using)
application.  I think it's much less common if you have serial netcdf:
usually in serial netcdf a master process coordinates and controlls
all access to the netcdf dataset.

If your application makes sense for MPI (and it might not..), you
might want to use parallel-netcdf (www.mcs.anl.gov/parallel-netcdf) to
create the dataset.  you'll still be able to use the whole ecosystem
of netcdf viewers. 

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B

• References:
  • [netcdfgroup] Structure at the file level?
    • From: Rick Hedin