NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
http://cf-trac.llnl.gov/trac/ticket/145The point is made there, that there are already plenty of "datasets" that are in practice stored in multiple files, such as individual timesteps, or different data fields that are stored separately. In those cases, the files are valid files in and of themselves, but the model run, or whatever it is, is contained in multiple files.
These kinds of proposals go further by storing dimension variables or other information related to the grid in different files. Now if the separately-stored information is missing or gets moved, interpreting the file contents becomes impossible.
-Ansley On 7/13/2016 2:40 PM, Roy Mendelssohn - NOAA Federal wrote:
that if one of those underlying files is missing, then I assume the whole dataset is unreadable.I have been wondering that myself. Is there anyone from HDF that can answer that? -RoyOn Jul 13, 2016, at 2:33 PM, dmh@xxxxxxxx wrote: If I understand it correctly, you are correct: from the API point of view, it is invisible. But that is not the issue for me. The issue is that now one needs multiple files in order to acccess data rather than having everything kept in a single file. That means, for example, that if one of those underlying files is missing, then I assume the whole dataset is unreadable. =Dennis Heimbigner Unidata On 7/13/2016 3:25 PM, Roy Mendelssohn - NOAA Federal wrote:Hi Dennis: I asked about this also awhile back. The key question to my mind is how the virtual dataset works in HDF5. Perhaps the interface is such is that it is transparent to higher level calls, much like when you create virtual datasets in TDS. In that case, the calls would be to as if there was one file. -RoyOn Jul 13, 2016, at 2:09 PM, dmh@xxxxxxxx wrote: To this point in time, it has been an implicit assumption that a netcdf dataset encompasses a single file. This functionality effectively challenges that assumption. So, I would solicit community input on this. What problems are created if a netcdf dataset now requires multiple files? =Dennis Heimmbigner Unidata On 7/13/2016 5:09 AM, Eugen Betke wrote:Dear NetCDF-Group, we have been working on NetCDF external link functionality. This allows NetCDF applications to create dimension variables which values are stored in an external file. Therefore, it uses the HDF5 virtual dataset (VDS) functionality. This is useful for, e.g., climate applications that rely on a variable per file and timestep configuration. The idea is to store the grid in a separate file and link our data to this grid. We already have our first working version. You find the patch and the examples on our page: http://wr.informatik.uni-hamburg.de/research/projects/bullio/netcdf_external_links/start Under the hood it uses HDF5 virtual datasets. VDS has the advantage of being compatible to the functions that are supported by oridinary datasets. Therefore, files containing VDS should be supported by the most software. There is a minor issue related to HDF5, the call H5F_try_close function fails, when ncdump trys to read data from an external dimension. So far we found a workaround, but we will fix this issue. It would be great if external link functionality could be supported by netCDF at some timepoint. We would like to improve our patch and for that reason we need your feedback. If you have some idea to the issue above, we would be grateful for each hint. Regards, Eugen _______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/_______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center ***Note new address and phone*** 110 Shaffer Road Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center ***Note new address and phone*** 110 Shaffer Road Santa Cruz, CA 95060 Phone: (831)-420-3666 Fax: (831) 420-3980 e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr. _______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
netcdfgroup archives: