NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
One problem that you are up against is that there is no defined conversion between opendap(dap2) and netcdf-4. There is a defined conversion between dap2 and netcdf-3 (classic). So when you make a opendap(dap2) url request, the resulting data is downloaded and is accessible (read-only) thru the netcdf-3 (classic) API an appears largely the same as any other netcdf-3 file. If you need netcdf-4, then you will need to use DAP4. =Dennis Heimbigner Unidata On 5/27/2016 9:38 AM, Nathan Potter wrote:
Ashwin, I believe that’s correct. To be honest I am not well versed on what can/cannot be done with the Java and C implementations of the NetCDF library once the data have been subset and retrieved. But given that nccopy/ncdump/etc. are “client” programs that are built on the NetCDF lib to achieve their functionality I imagine that source code for these programs will provide examples of how to save the response as a NetCDF file on your local disk. And I suspect that the Java NetCDF library must have similar “client” tools that could be reviewed to find the analogous examples in Java. Antonio Cofino also makes the salient point (in an email that is now orphaned from this response) that you should be thoughtful and subset/retrieve the coordinate/map/shared_dimension variables associated with the core data so that all of the relevant information is captured in the resulting NetCDF result. Without this the downstream software may face additional challenges. You might write to the operators of www.esrl.noaa.gov (esrl.psd.data@xxxxxxxx) and ask them if they have enabled or disabled the TDS feature that will allow you to make the subset request in such a way as to prompt the TDS to return a NetCDF file rather than a DAP2 or ASCII encoded response. Sincerely, NathanOn May 27, 2016, at 7:30 AM, ashwinD12 . <winash12@xxxxxxxxx> wrote: Nathan, That is a very important point you bring up. If I want to write my own Java client for instance I have to convert the OpenDAP 2 response into a netcdf 4 file on my disk. It is very likely that the THREDDS server does not serve netcdf files but I can get the response which is OpenDAP and convert it into a netcdf 4 file. Correct ? Regards, Ashwin. On Fri, May 27, 2016 at 7:53 PM, Nathan Potter <ndp@xxxxxxxxxxx <mailto:ndp@xxxxxxxxxxx>> wrote: Ashwin, I’m pleased! I want to clarify what’s happen though - The data is held in a NetCDF file on the server. However, what is transported is NOT a NetCDF file, but rather the response is encoded as a DAP2 metadata/data response which is in turn loaded into the (client side) NetCDF library and then written to local disk as a NetCDF file. DAP2 is the transport protocol/encoding. Some servers (Hyrax being my favorite example ;) are able to encode the response as a NetCDF file at the server end. This behavior can be triggered (in Hyrax) by replacing the original “.ascii” component of the path URL with “.nc”. For example: These ASCII request URLs: http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.ascii <http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.ascii> http://test.opendap.org:8080/opendap/data/hdf5/grid_1_2d.h5.ascii <http://test.opendap.org:8080/opendap/data/hdf5/grid_1_2d.h5.ascii> Become NetCDF request URLs: http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.nc <http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.nc> http://test.opendap.org:8080/opendap/data/hdf5/grid_1_2d.h5.nc <http://test.opendap.org:8080/opendap/data/hdf5/grid_1_2d.h5.nc> I believe that the TDS (which is the server software in play in your example) also supports a similar feature, albeit through a somewhat different URL scheme. And I think that earlier in the thread someone mentioned that this feature is disabled on the server in question. Cheers, NathanOn May 27, 2016, at 7:05 AM, ashwinD12 . <winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>> wrote: Nathan, Beautiful ! That is exactly what I am looking for. I can come up with my own version of nccopy in Java or Python if I so wish. So that means DAP2 can download netCDF files using the PSD THREDDS server and I can pass OpeNDAP constraint expressions. Regards, Ashwin. On Fri, May 27, 2016 at 7:17 PM, Nathan Potter <ndp@xxxxxxxxxxx <mailto:ndp@xxxxxxxxxxx>> wrote: Ashwin, The NetCDF library may be useful for your desired outcome. Because the NetCDF library supports access via the DAP2 protocol it is a natural for your probelm. I was able to use the NetCDF command line tools to accomplish your task. Note that be dropping the “.ascii” suffix from the URL path you return to the core “DAP2 URL” which is an important step in the following example. Retrieving the data subset and saving it as a NetCDF-4 file: ./nccopy -4 "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40] <http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40]>” foo.nc <http://foo.nc/> Which was easily read by ncdump: ./ncdump -h foo.nc <http://foo.nc/> netcdf foo { dimensions: time = UNLIMITED ; // (124 currently) lat = 15 ; level = 1 ; lon = 15 ; variables: short air(time, level, lat, lon) ; air:long_name = "6-hourly Air Temperature on Pressure Levels" ; air:valid_range = -32765s, -10260s ; air:unpacked_valid_range = 137.5f, 362.5f ; air:actual_range = 180.6f, 325.8f ; air:units = "degK" ; air:add_offset = 465.15f ; air:scale_factor = 0.01f ; air:missing_value = 32766s ; air:_FillValue = -32767s ; air:precision = 2s ; air:least_significant_digit = 1s ; air:GRIB_id = 11s ; air:GRIB_name = "TMP" ; air:var_desc = "Air temperature" ; air:dataset = "NCEP/DOE AMIP-II Reanalysis (Reanalysis-2)" ; air:level_desc = "Pressure Levels" ; air:statistic = "Individual Obs" ; air:parent_stat = "Other" ; air:standard_name = "air_temperature" ; // global attributes: :Conventions = "CF-1.0" ; :title = "4x Daily NCEP/DOE Reanalysis 2" ; :history = "created 2015/02 by NOAA/ESRL/PSD" ; :comments = "Data is from \nNCEP/DOE AMIP-II Reanalysis (Reanalysis-2)\n(4x/day). It consists of most variables interpolated to\npressure surfaces from model (sigma) surfaces." ; :platform = "Model" ; :source = "NCEP/DOE AMIP-II Reanalysis (Reanalysis-2) Model" ; :institution = "National Centers for Environmental Prediction" ; :References = "http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis2.html <http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis2.html>" ; :dataset_title = "NCEP-DOE AMIP-II Reanalysis" ; :DODS_EXTRA.Unlimited_Dimension = "time" ; } I am sure that there are several other ways to achieve a similar result, but this one does seem simple and amenable to programatic (well, at least script level) use. I believe that both the NetCDF C and Java APIs will also happily detect that a supplied file name is a URL and then dereference and retrieve data from it. Does this in any way address your problem? Regards, NathanOn May 27, 2016, at 2:13 AM, ashwinD12 . <winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>> wrote: Regarding the specifics of what you have written in relation to the programming language of choice 1) If I use java netcdf(the latest version) I can use the OPeNDAP URL that James explained to me and send it to the URL - http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40] <http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]> (as an example) the data returned will be ascii but the java netcdf library will allow me to generate a netcdf file containing that data precisely that can be written to disk. Am I correct in this ? 2) The other option is to use the URL you gave me and generate the URL with the required parameters and then use Python and pycurl and write out the netcdf file to disk. Your input is appreciated, Regards, Ashwin.= = = Nathan Potter ndp at opendap.org <http://opendap.org/> OPeNDAP, Inc. +1.541.231.3317= = = Nathan Potter ndp at opendap.org <http://opendap.org/> OPeNDAP, Inc. +1.541.231.3317= = = Nathan Potter ndp at opendap.org OPeNDAP, Inc. +1.541.231.3317 _______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. thredds mailing list thredds@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
thredds archives: