NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Pedro, Is there a way to represent compound HDF5 objects in your scheme? I see STAR_INT8, STAR_FLOAT, etc., but no STAR_COMPOUND. In any event, FYI, I have implemented a similar thing but in a narrower domain. I needed to map VOTable [1] (an xml format with a defined data model for astronomy) to HDF5. Along the way, I also mapped it (with varying levels of fidelity) to JSON, CSV, HTML, FITS, and plain old ascii. This makes my work very table focused, with a particular eye towards what relational databases like to output. The code is on github [2]. It is mainly meant to be used as a library, but the code also builds a simple program that converts files between formats. In case you are interested, I am attaching the same file in plain text, JSON, and HDF5. Cheers, Walter Landry [1] http://www.ivoa.net/documents/VOTable/ [2] https://github.com/Caltech-IPAC/tablator Pedro Vicente <pedro.vicente@xxxxxxxxxxxxxxxxxx> wrote: > Hi Charlie ! > > > So, I am doing that exact same thing. > > I wrote > > 1) The specification to convert netCDF/HDF5 to "a" JSON format (note > the "a" here) > > 2) I wrote the parsing of the JSON format using an open source C JSON > library (janson). > > http://www.digip.org/jansson/ > > 3) left to do: the actual C code of reading/writing netCDF/HDF5 to > JSON and vice-versa (the straightforward part). > > The "a" above means that JSON is not really a format in the sense of > netCDF but really a format that allows to define formats , for a lack > of a better > explanation. > > This means that anyone that writes this tool has to write code that > write in a particular JSON representation , only valid for that tool. > > Like you , I searched and there was not a good one, so I wrote one. > > The first criteria was that it had to be obvious for anyone looking at > the JSON text file, that that was indeed a netCDF/HDF5 file: hierarchy > clearly show, metadata and data > clearly shown > > My first look was HDF5-JSON > > http://hdf5-json.readthedocs.io/en/latest/ > > but the format seemed like a mess to look at > > example > > http://hdf5-json.readthedocs.io/en/latest/examples/nullspace_dset.html > > > and the reader is written in Python > > @John Readey > > (why Phyton? HDF5 developer tools should be all about writing in > C/C++) > > > > The specification is here > > http://www.space-research.org/ > > Click on menu > "Code blog", > then > "netCDF/HDF5 to JSON and vice-versa" > > > In the process I learned all about JSON and it is a neat format to > represent data . > > In particular, it allows nested structures and arrays, which suits > perfectly for netCDF > > here are two nested groups > > { > "group_name1": > { > "group_name2": "group" > } > } > > a dataset > > { > "dset1" : ["dataset", "STAR_INT32", 2, [3, 4], [1, 2, 3, 4, 5, 6, 7, > 8, 9, 10, 11, 12]] > } > > > > This is still under development, > > I would like to make this some kind of "official" netCDF/HDF5 JSON > format for the community, so I encourage anyone to read the > specification > > direct link > > http://www.space-research.org/blog/star_json.html > > If you see any flaw in the design or antything in the design that you > would like to have change please let me know now > > At the moment it only (intentionally) uses common generic features of > both netCDF and HDF5, which are the numeric atomic types and strings. > > Enjoy > > > ---------------------- > Pedro Vicente > pedro.vicente@xxxxxxxxxxxxxxxxxx > http://www.space-research.org/ > > > > > ----- Original Message ----- > From: "Charlie Zender" <zender@xxxxxxx> > To: "netCDF Mail List" <netcdfgroup@xxxxxxxxxxxxxxxx> > Sent: Thursday, October 13, 2016 11:10 PM > Subject: [netcdfgroup] How to dump netCDF to JSON? > > >> Hello netCDFers, >> >> A project I am working on wants to convert netCDF files to JSON. >> The requirements are to dump an arbitrary netCDF-extended file >> (with groups but without funky vlen/compound types) to JSON. >> The first few solutions that we googled (ncdump-json, netcdf2json.py) >> do not satisfy these requirements. What is the most robust and easy >> command-line tool (not web-service) that dumps netCDF to JSON? >> Ideally it would be somewhat configurable like ncdump -h/-x or >> ncks --cdl/--xml. >> >> Charlie >> -- >> Charlie Zender, Earth System Sci. & Computer Sci. >> University of California, Irvine 949-891-2429 )'( >> >> _______________________________________________ >> NOTE: All exchanges posted to Unidata maintained email lists are >> recorded in the Unidata inquiry tracking system and made publicly >> available through the web. Users who post to any of the lists we >> maintain are reminded to remove any personal information that they >> do not want to be made public. >> >> >> netcdfgroup mailing list >> netcdfgroup@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum@xxxxxxxxxxxxxxxxxx > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5
| object | ra | dec | htm20
| htm7 | htm3 | shtm20 | shtm7 | shtm3 |
flags | SSO |
| char | double | real | ulong
| uint | ushort | long | int | short |
byte | bool |
118289arstratraetratratsrastratsrastrats 359.88703 50.832570
16446744073709551616 3294967296 12000 8223372036854775808 1147483648
12000 122 0
113368 344.41273 -29.622250
8446744073709551616 294967296 43002 -7223372036854775808 -2047483648
13002 0xf2 true
113368 344.41273 -29.622250
8446744073709551616 294967296 43002 -7223372036854775808 -2047483648
-23002 211 False
113368 344.41273 -29.622250
8446744073709551616 294967296 43002 -7223372036854775808 -2047483648
-31002 211 1
{
"VOTABLE":
{
"<xmlattr>":
{
"version": "1.3",
"xmlns:xsi": "http:\/\/www.w3.org\/2001\/XMLSchema-instance",
"xmlns": "http:\/\/www.ivoa.net\/xml\/VOTable\/v1.3",
"xmlns:stc": "http:\/\/www.ivoa.net\/xml\/STC\/v1.30"
},
"RESOURCE":
{
"TABLE":
{
"FIELD":
{
"<xmlattr>":
{
"name": "object",
"datatype": "char",
"arraysize": "*"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "ra",
"datatype": "double"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "dec",
"datatype": "double"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm20",
"datatype": "ulong"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm7",
"datatype": "uint"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm3",
"datatype": "ushort"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm20",
"datatype": "long"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm7",
"datatype": "int"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm3",
"datatype": "short"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "flags",
"datatype": "unsignedByte"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "SSO",
"datatype": "boolean"
}
},
"DATA":
{
"TABLEDATA":
[
[
"118289arstratraetratratsrastratsrastrats",
"359.88702999999998",
"50.832569999999997",
"16446744073709551616",
"3294967296",
"12000",
"8223372036854775808",
"1147483648",
"12000",
"0x7a",
"0"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"13002",
"0xf2",
"1"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"-23002",
"0xd3",
"0"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"-31002",
"0xd3",
"1"
]
]
}
}
}
}
}
�HDF