NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
I knew that something was left unanswered @Walter
Is there a way to represent compound HDF5 objects in your scheme? I see STAR_INT8, STAR_FLOAT, etc., but no STAR_COMPOUND.
no, only the atomic types. see spec http://www.space-research.org/blog/star_json.html
In any event, FYI, I have implemented a similar thing but in a narrower domain. I needed to map VOTable [1] (an xml format with a defined data model for astronomy) to HDF5. The code is on github [2]. It is mainly meant to be used as a library, but the code also builds a simple program that converts files between formats.
ok, thanks, I did a clone of https://github.com/Caltech-IPAC/tablator but it seems there is not a build system? by the way HDF5 has its own table API, it seems you did not use it https://support.hdfgroup.org/HDF5/doc/HL/RM_H5TB.html -Pedro----- Original Message ----- From: "Walter Landry" <wlandry@xxxxxxxxxxx>
To: <hdf-forum@xxxxxxxxxxxxxxxxxx>; <pedro.vicente@xxxxxxxxxxxxxxxxxx> Cc: <netcdfgroup@xxxxxxxxxxxxxxxx>; <zender@xxxxxxx> Sent: Friday, October 14, 2016 2:10 AM Subject: Re: [Hdf-forum] [netcdfgroup] How to dump netCDF to JSON?
Hi Pedro, Is there a way to represent compound HDF5 objects in your scheme? I see STAR_INT8, STAR_FLOAT, etc., but no STAR_COMPOUND. In any event, FYI, I have implemented a similar thing but in a narrower domain. I needed to map VOTable [1] (an xml format with a defined data model for astronomy) to HDF5. Along the way, I also mapped it (with varying levels of fidelity) to JSON, CSV, HTML, FITS, and plain old ascii. This makes my work very table focused, with a particular eye towards what relational databases like to output. The code is on github [2]. It is mainly meant to be used as a library, but the code also builds a simple program that converts files between formats. In case you are interested, I am attaching the same file in plain text, JSON, and HDF5. Cheers, Walter Landry [1] http://www.ivoa.net/documents/VOTable/ [2] https://github.com/Caltech-IPAC/tablator Pedro Vicente <pedro.vicente@xxxxxxxxxxxxxxxxxx> wrote:Hi Charlie ! So, I am doing that exact same thing. I wrote 1) The specification to convert netCDF/HDF5 to "a" JSON format (note the "a" here) 2) I wrote the parsing of the JSON format using an open source C JSON library (janson). http://www.digip.org/jansson/ 3) left to do: the actual C code of reading/writing netCDF/HDF5 to JSON and vice-versa (the straightforward part). The "a" above means that JSON is not really a format in the sense of netCDF but really a format that allows to define formats , for a lack of a better explanation. This means that anyone that writes this tool has to write code that write in a particular JSON representation , only valid for that tool. Like you , I searched and there was not a good one, so I wrote one. The first criteria was that it had to be obvious for anyone looking at the JSON text file, that that was indeed a netCDF/HDF5 file: hierarchy clearly show, metadata and data clearly shown My first look was HDF5-JSON http://hdf5-json.readthedocs.io/en/latest/ but the format seemed like a mess to look at example http://hdf5-json.readthedocs.io/en/latest/examples/nullspace_dset.html and the reader is written in Python @John Readey (why Phyton? HDF5 developer tools should be all about writing in C/C++) The specification is here http://www.space-research.org/ Click on menu "Code blog", then "netCDF/HDF5 to JSON and vice-versa" In the process I learned all about JSON and it is a neat format to represent data . In particular, it allows nested structures and arrays, which suits perfectly for netCDF here are two nested groups { "group_name1": { "group_name2": "group" } } a dataset { "dset1" : ["dataset", "STAR_INT32", 2, [3, 4], [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]] } This is still under development, I would like to make this some kind of "official" netCDF/HDF5 JSON format for the community, so I encourage anyone to read the specification direct link http://www.space-research.org/blog/star_json.html If you see any flaw in the design or antything in the design that you would like to have change please let me know now At the moment it only (intentionally) uses common generic features of both netCDF and HDF5, which are the numeric atomic types and strings. Enjoy ---------------------- Pedro Vicente pedro.vicente@xxxxxxxxxxxxxxxxxx http://www.space-research.org/----- Original Message ----- From: "Charlie Zender" <zender@xxxxxxx>To: "netCDF Mail List" <netcdfgroup@xxxxxxxxxxxxxxxx> Sent: Thursday, October 13, 2016 11:10 PM Subject: [netcdfgroup] How to dump netCDF to JSON?Hello netCDFers, A project I am working on wants to convert netCDF files to JSON. The requirements are to dump an arbitrary netCDF-extended file (with groups but without funky vlen/compound types) to JSON. The first few solutions that we googled (ncdump-json, netcdf2json.py) do not satisfy these requirements. What is the most robust and easy command-line tool (not web-service) that dumps netCDF to JSON? Ideally it would be somewhat configurable like ncdump -h/-x or ncks --cdl/--xml. Charlie -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'( _______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@xxxxxxxxxxxxxxxxxx http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
--------------------------------------------------------------------------------
| object | ra | dec | htm20 | htm7 | htm3 | shtm20 | shtm7 | shtm3 | flags | SSO | | char | double | real | ulong | uint | ushort | long | int | short | byte | bool | 118289arstratraetratratsrastratsrastrats 359.88703 50.832570 16446744073709551616 3294967296 12000 8223372036854775808 1147483648 12000 122 0 113368 344.41273 -29.622250 8446744073709551616 294967296 43002 -7223372036854775808 -2047483648 13002 0xf2 true 113368 344.41273 -29.622250 8446744073709551616 294967296 43002 -7223372036854775808 -2047483648 -23002 211 False 113368 344.41273 -29.622250 8446744073709551616 294967296 43002 -7223372036854775808 -2047483648 -31002 211 1
--------------------------------------------------------------------------------
{
"VOTABLE":
{
"<xmlattr>":
{
"version": "1.3",
"xmlns:xsi": "http:\/\/www.w3.org\/2001\/XMLSchema-instance",
"xmlns": "http:\/\/www.ivoa.net\/xml\/VOTable\/v1.3",
"xmlns:stc": "http:\/\/www.ivoa.net\/xml\/STC\/v1.30"
},
"RESOURCE":
{
"TABLE":
{
"FIELD":
{
"<xmlattr>":
{
"name": "object",
"datatype": "char",
"arraysize": "*"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "ra",
"datatype": "double"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "dec",
"datatype": "double"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm20",
"datatype": "ulong"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm7",
"datatype": "uint"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm3",
"datatype": "ushort"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm20",
"datatype": "long"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm7",
"datatype": "int"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm3",
"datatype": "short"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "flags",
"datatype": "unsignedByte"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "SSO",
"datatype": "boolean"
}
},
"DATA":
{
"TABLEDATA":
[
[
"118289arstratraetratratsrastratsrastrats",
"359.88702999999998",
"50.832569999999997",
"16446744073709551616",
"3294967296",
"12000",
"8223372036854775808",
"1147483648",
"12000",
"0x7a",
"0"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"13002",
"0xf2",
"1"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"-23002",
"0xd3",
"0"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"-31002",
"0xd3",
"1"
]
]
}
}
}
}
}
--------------------------------------------------------------------------------
?HDF ÿÿÿÿÿÿÿÿ?0ÓGâAOHDR TrXTrXTrXTrXxÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿtableÃD"%=?OHDR!TrXTrXTrXTrX 6 Xnull_bitfield_flags:object(ra* ?@4 4ÿdec2 ?@4 4ÿhtm20:@htm7B htm3Fshtm20H@shtm7P shtm3TflagsVSSOW 0`ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ¼ -METADATA960namevalueattributes 96 namevalue4) ±118289arstratraetratratsrastratsrastrats"?^F1~v@»Ð\§'jI@8±~'>ä6eÄà.oXLI r6eDà.z113368@ÁÅSs?u@zï§ÆKY=ÀvûÜ8uØ"ú§ÈNgmÁ>áõ.Ê2ò113368@ÁÅSs?u@zï§ÆKY=ÀvûÜ8uØ"ú§ÈNgmÁ>áõ.&¦Ó113368@ÁÅSs?u@zï§ÆKY=ÀvûÜ8uØ"ú§ÈNgmÁ>áõ.æ?Ó
netcdfgroup archives: