Re: HDF vs. netCDF (long)

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: HDF vs. netCDF (long)
• From: kac@xxxxxxxxxxx (Kevin Campbell)
• Date: Sat, 25 Sep 93 23:00:00 PDT

Hi,
  I received many responses to my post about the differences between
HDF and netCDF.  Thaks to everyone who responded, and sorry it has
taken me so long to get back to all of you.  School has gotten pretty
hectic lately.  This will be a summary of the responses I received
along with new questions from me :)  Hopefully, this will also serve as
a starting point for new threads of discussion, as I feel that this is
an important topic.

Here we go:

chouck@xxxxxxxxxxxxx (Chris Houck) writes:
>The HDF / netCDF project (aka HDF 3.3) was designed to be completely
>transparent to the programmer.  HDF 3.3 supports a new "multi-file"
>SDS interface and the complete netCDF interface as defined by
>Unidata netCDF Release 2.3.2.

Is this anything like the "VSETS" mentioned later on?

more from Chris Houck
>Using either interface, you are able to read XDR-based netCDF files,
>HDF-based netCDF files and pre-HDF3.3 HDF files.  The library 
>figures out what type of file is being accessed and handles it 
>appropriately.  Any of the above types of files may be modified, 
>however, the library will only create new files based on HDF 
>(you can't create new XDR-based netCDF files).

Hmm, it seems to me that writing the files in XDR format is a very important
feature.  Especially with the architectural diversity here at LBL.  How have
other users of HDF dealt with this problem?  That brings up another point
that was made:  are HDF and netCDF really distinct, or can should they be
used together?  An exerpt follows.

lysakowski@xxxxxxxxxxxxxxxxxxx (Rich Lysakowski) writes:
>What is your point of positioning netCDF versus HDF?  There is much
>information available on the netCDF/HDF interface.  I discourage anyone
>from describing them in your newsletter as "competing products", but rather
>discuss them as to how they complement each other.  I encourage you to take
>the time to understand each of them, because there is a LOT of rich
>functionality in each one.  They each have their place too.  They are
>complementary toolkits - witness the inclusion of the "standard", simple
>API of netCDF into HDF. 

I apologize for implying that netCDF and HDF should be used exclusive of
each other.  I am very new to both of these products, and barely know what
they can do, much less the subtleties of their differences.  This is 
something I am very interested in learning more about.  How would HDF and
netCDF complement each other in regards to the network transparency problem?
I suppose a more general question and less easily answered question would
be "what features in each of the formats are duplicated, and which are
unique?"

I will be working with one user soon, but I haven't spoken with him yet
about his specific problem.  What I would like to do, if it's ok with
everyone, is to post his problem and get feedback from the group on possible
solutions to it, as well as discuss how it relates to the above paragraph.

hankin@xxxxxxxxxx (Steve Hankin) writes:
>what is the future direction of this union [between HDF and netCDF]?  Will 
>the HDF version remain compatible with changes made in netCDF?

This is a good question.  I seem to remember reading in the "History"
section of the netCDF manual that HDF and netCDF are now independent
projects, but if it would be useful (and possible) to really integrate the 
two, perhaps that could change.

More from (Rich Lysakowski):
>There is still much more work that needs to be done to take the best of
>these two approaches and merge them, but each one has its strengths for
>different data types and for different reasons.  We are very supportive of 
>the netCDF and HDF approaches.  Personally I'd like to see the inclusion 
>of more of IBM's Explorer concepts, and U.Ill. Urbana's "Envision" concepts 
>in both public-domain toolkits (netCDF and HDF). 

Could you give a couple examples on when each of the two toolkits would be
appropriate?  Also, could you elaborate more on which Explorer and Envision
features you would like to see?

That's it for the summary.  By the way, the deadline for my article is
pretty far off, probably 3-4 weeks at least, so I'll have time to put a
good amount of thought into it.  It's not an extensive article, it's really
just designed to inform the user community at LBL of the existence of new
software and give a brief outline of what it is useful for.  I would not
be averse to going into more depth later on, to write a "how-to" to help
them understand the difference between the two packages and how they can
be used together.  I will put anything I come up with on the mailing list.
Input is welcome!

One last thing, regarding the other point in my original message.  A few
people gave me the names of the NCSA support crew (thanks!), so I'll be
getting in touch with them and I'll let you all in on what I learn.

Thanks,
Kevin