NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Andrew: Its unlikely that having that many groups is a good design, although there might be exceptions. What are you using them for? John On Mon, Nov 17, 2014 at 5:18 PM, Pedro Vicente < pedro.vicente@xxxxxxxxxxxxxxxxxx> wrote: > Andrew > > If you are writing a new format, my guess is that you are writing an API > for it, and since I have been writing these High-Level APIs for years > (starting with the very own HDF5 High-Level API), and I am writing one now, > here's my advice for it :-) > > 1) Do not use groups as a means to access your data (more on this later). > 2) Encapsulate all HDF5 IDs. > > > For my case, I am using a model where datasets are accessed for read/write > using their full path name, e.g "/path/to/dataset". > > So, "/path/to/" is the group name and "dataset" is the dataset relative > name. > > API usage for read/write is > > write( "/path/to/dataset", void * data buffer) > > so, the model to read/write is just the full dataset name as a string and > a buffer with data. > > The group part is implicit in the string, as you can see no HDF5 group IDs > or group paths to read/write (that is what I mean with > "not use groups as a means to access your data ") > > > How can you accomplish this? > > Here's an example of functions part of a C++ class , regarding this part > > void create_group(const std::string& group_name); > > void > create_dataset(const std::string& group_path, const std::string & > dataset_name, other parameters here like dataset size or chunk size > > void > write(const std::string& path, const void* buf); > > Let's start with the create_group function > > > Here just create the group using the HDF5 C API (I use the C API even in a > C++ program, the HDF5 C++ API adds nothing of use here) > > > gid = H5Gcreate2(this->m_fid, group_name.c_str(), other parameters > H5Gclose(gid) > As can be seen the group is created and closed immediately. > > This means, you will have no groups open at all during your read/write > calls (typically the core part of a program execution). > > Since groups are closed immediately, you can even exceed the number of > maximum open groups Russ mentioned. > > For the dataset creation > > > create_dataset( > const std::string& group_path, const std::string& dataset_name, > concatenate the group path with the dataset name and store it somehow in > your C++ class > > > > std::string absolute_dataset_name = > group_path + "/" + dataset_name; > In my case , I am using a map with dataset ID/ full path of dataset, > because I want to keep the datasets open (in memory), but any other way to > store the path will do > > > //map with dataset name (full path) / HDF5 dataset ID > > std:: > map<std::string, hid_t> m_map_datasets; > > > here is the HDF5 create call > > > did = H5Dcreate2( > this->m_fid, absolute_dataset_name.c_str() > > The trick here is to use the file ID (that must be stored in the class) > and the full path. This takes group IDs out of the equation. > > For the write/read call use simply > > > write( > const std::string& path, const void* buf) > > > > get the dataset ID from full path > > //get dataset ID from map <path/ID> > > hid_t did = this->m_map_datasets[path]; > > use it in the write call > > H5Dwrite(did, other parameters > > > > Hope this helps and ask any further questions if needed. > > > > btw, I downloaded your file with only groups, it has a size of 37MB, lots > of group metadata here. > > -Pedro > > ---------------------- > Pedro Vicente > pedro.vicente@xxxxxxxxxxxxxxxxxx > http://www.space-research.org/ > > ----- Original Message ----- > *From:* Russ Rew <russ@xxxxxxxxxxxxxxxx> > *To:* Andrew Dowsey <andrew.dowsey@xxxxxxxxxxxxxxxx> > *Cc:* netcdfgroup@xxxxxxxxxxxxxxxx > *Sent:* Monday, November 17, 2014 2:11 PM > *Subject:* Re: [netcdfgroup] Problem with nc_inq_grps and ncdump reading > HDF5file > > Hi Andrew, > > You've run across a limitation in the number of simultaneously open groups > permitted in netCDF-4. The groups documentation > <http://www.unidata.ucar.edu/netcdf/docs/group__groups.html> says > > ... Encoding both the open file id and group id in a single integer > currently limits the number of groups per netCDF-4 file to no more than > 32767. Similarly, the number of simultaneously open netCDF-4 files in one > program context is limited to 32767. > > > I think both those limits should actually be 65535 (== 2**16 - 1), but in > any case, your HDF5 file has 119254 groups, which is too many for netCDF-4 > to handle. > > The only workaround I can think of would be to close some groups if you > don't need to have them all open simultaneously. > > --Russ > > > On Mon, Nov 17, 2014 at 8:14 AM, Andrew Dowsey < > andrew.dowsey@xxxxxxxxxxxxxxxx> wrote: > >> Hi, >> >> I’m trying to create HDF5 files that can be read by NetCDF4 and I’ve run >> into a problem in that nc_inq_grps seems to report some bad ids. ncdump >> bails with this error too. h5dump works fine. The problem is deterministic >> but I haven’t been able to figure out what causes it because slightly >> different HDF5 files work fine. I have created a test file that has this >> problem, which contains nothing but groups. It can be downloaded from >> http://personalpages.manchester.ac.uk/staff/andrew.dowsey/test.h5 >> >> I am creating a new format for a type of instrument data we use, and for >> flexibility I would like it to be writeable/readable both by HDF5 and >> netCDF4 libraries. >> >> Any insight would be greatly appreciated! >> >> Kind regards, >> Andy >> >> >> *Andrew Dowsey PhD* >> *Lecturer and CADET Bioinformatics Research Lead* >> Institute of Human Development, The University of Manchester >> >> t: +44 161 701 0244 >> f: +44 161 701 0242 <%2B44%20161%20701%200242> >> http://www.manchester.ac.uk/research/andrew.dowsey >> >> Centre for Advanced Discovery and Experimental Therapeutics (CADET) >> Central Manchester University Hospitals NHS Foundation Trust >> Oxford Road >> Manchester M13 9WL >> UK >> >> >> _______________________________________________ >> netcdfgroup mailing list >> netcdfgroup@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ >> > > ------------------------------ > > _______________________________________________ > netcdfgroup mailing list > netcdfgroup@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > > > _______________________________________________ > netcdfgroup mailing list > netcdfgroup@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ >
netcdfgroup archives: