NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Thanks Sean! Stuck in meetings all Friday so I didn't get to reply sooner, but really this is a big help and the linked gist got me through the steps. Thanks again for all your hard work! -Tom ________________________________ From: Sean Arms <sarms@xxxxxxxx> Sent: Thursday, January 5, 2017 4:55 PM To: Hagman,Tom Cc: netcdf-java@xxxxxxxxxxxxxxxx Subject: Re: [netcdf-java] AggregateExisting with multiple Time dimensions present CAUTION - EXTERNAL EMAIL Greetings Tom! Ah GRIB...good times, or something :-) Here is a mailing list thread discussing some of the issues you outlined about the name of the time coordinate: http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2015/msg00133.html TL;DR; The GRIB files you've downloaded are essentially a collection of GRIB records concatenated into a single file, and the order of those GRIB records can change from file to file. The naming of the dimensions netcdf-java creates is based on when they are first encountered in a file, and thus can change from file to file. Unfortunately, NcML aggregation with GRIB files won't really work anymore (as of netcdf-java 4.3, I do believe). However, there is a way to aggregate GRIB files using a GribFeatureCollection. The code is a bit lower level, but it is sort of what the THREDDS Data Server uses when serving out collections of GRIB files. Here is a code example: https://gist.github.com/lesserwhirls/5eea2ffa0f1e66b50fb87b971fa111b5 The code example in the gist is using non-public parts of the netCDF-Java API, but should work. It is my hope to extend NcML to be able to do GRIB aggregations in the next version of netCDF-Java (v5), but this is what we have currently. Finally, while we just released the 4.6.7 version of netCDF-Java, I forgot one small but critical step in linking to the NCEP GRIB2 tables after doing a table update, and as a result this code won't work smoothly with 4.6.7 (some variable names will look like VAR0-7-192_FROM_7-0--1_surface...yuck...). We are in the process of cutting a new release that has the problem fixed. In the meantime, you may use the 4.6.6 version of the netCDF-All.jar file from here: https://github.com/Unidata/thredds/releases/tag/v4.6.6 Cheers, Sean On Thu, Jan 5, 2017 at 1:29 PM, Hagman,Tom <Tom.Hagman@xxxxxxxxxxxxxx> wrote: > Hi all, > > > Firstly, thank you all so much for your hard work on this library. > > > For my brain teaser, I've searched through the docs and mailing list > archives awhile and am having a hard time putting together the steps I need > to handle this aggregation. > > CFSR 1 hour data files data from here : > http://rda.ucar.edu/datasets/ds094.0/ > > cdas_20161215_0000_f00000_G4.grib2 > cdas_20161215_0000_f00100_G4 > cdas_20161215_0000_f00200_G4 > cdas_20161215_0000_f00300_G4 > etc... > The hourly files declare 2 time dimensions, one with bounds set and another > without. > > cdas_20161215_0000_f00300_G4.grib2 > double time(time=1); > :units = "Hour since 2016-12-15T00:00:00Z"; > :standard_name = "time"; > :long_name = "GRIB forecast or observation time"; > :calendar = "proleptic_gregorian"; > :bounds = "time_bounds"; > double time_bounds(time=1, 2); > :units = "Hour since 2016-12-15T00:00:00Z"; > :long_name = "bounds for time"; > double time1(time1=1); > :units = "Hour since 2016-12-15T00:00:00Z"; > :standard_name = "time"; > :long_name = "GRIB forecast or observation time"; > :calendar = "proleptic_gregorian"; > > The problem is that when I step through each dataset creation, different > hourly files will swap names for the 2 time dimension names. So then > AggregationExisting is unable to find the dimension name 'time' for certain > files, e.g. on the u-component_of_wind_isobaric variable in the 0300 file > because it was swapped to time1 instead. > > > Time dimension name issue illustrated in 2 files: > > //cdas_20161215_0000_f00300_G4.grib2 > > float u-component_of_wind_isobaric(time1=1, isobaric3=37, lat=361, lon=720); > :long_name = "u-component of wind @ Isobaric surface"; > :units = "m/s"; > :abbreviation = "UGRD"; > :missing_value = NaNf; // float > :grid_mapping = "LatLon_Projection"; > :coordinates = "reftime time1 isobaric3 lat lon "; > :Grib_Variable_Id = "VAR_0-2-2_L100"; > :Grib2_Parameter = 0, 2, 2; // int > :Grib2_Parameter_Discipline = "Meteorological products"; > :Grib2_Parameter_Category = "Momentum"; > :Grib2_Parameter_Name = "u-component of wind"; > :Grib2_Level_Type = "Isobaric surface"; > :Grib2_Generating_Process_Type = "Forecast"; > > > //cdas_20161215_0000_f00200_G4.grib2 > > float u-component_of_wind_isobaric(time=1, isobaric3=37, lat=361, lon=720); > :long_name = "u-component of wind @ Isobaric surface"; > :units = "m/s"; > :abbreviation = "UGRD"; > :missing_value = NaNf; // float > :grid_mapping = "LatLon_Projection"; > :coordinates = "reftime time isobaric3 lat lon "; > :Grib_Variable_Id = "VAR_0-2-2_L100"; > :Grib2_Parameter = 0, 2, 2; // int > :Grib2_Parameter_Discipline = "Meteorological products"; > :Grib2_Parameter_Category = "Momentum"; > :Grib2_Parameter_Name = "u-component of wind"; > :Grib2_Level_Type = "Isobaric surface"; > :Grib2_Generating_Process_Type = "Forecast"; > > Code I'm calling: > > List<String> variableNames = > Arrays.asList("u-component_of_wind_isobaric","u-component_of_wind_height_above_ground","v-component_of_wind_isobaric","v-component_of_wind_height_above_ground","Pressure_reduced_to_MSL_msl","Geopotential_height_isobaric"); > NetcdfDataset netcdfDataset = new NetcdfDataset(); > //here i'm trying to aggregate on a dimension called 'time' > AggregationExisting aggregationExisting = new > AggregationExisting(netcdfDataset, "time", null); > aggregationExisting.addDatasetScan(null, > "/cfsr-gribs/201612/", > "G4.grib2", > null, > null, > NetcdfDataset.getDefaultEnhanceMode(), > "false", > null); > aggregationExisting.persistWrite(); > aggregationExisting.finish(new CancelTaskImpl()); > GridDataset gridDataset = new GridDataset(netcdfDataset); > //....spline data in merged dataset in memory > //writer is the NetcdfFileWriter with the merged file path > writer.setRedefineMode(true); > CFGridWriter2.writeFile(gridDataset, variableNames, > gridDataset.getBoundingBox(), null, 1, null, null, 1, true, writer); > > This is my first NetCDF library use so I'm shopping for some preprocessing > tools to get these datasets merged that have this quirk. Could I move all > the variables into the same time dimension and rename it, for instance? Even > a link to an example I missed would be helpful. Otherwise I'm guessing I > will look into manually declaring dimensions and using readDataSlice() to > manually copy in the data into a new merged file. > > > > Thanks! > -Tom Hagman > > > ________________________________ > > CONFIDENTIALITY NOTICE: This e-mail message is for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. Any unauthorized review, use, disclosure or distribution of any > kind is strictly prohibited. If you are not the intended recipient, please > contact the sender via reply e-mail and destroy all copies of the original > message. Thank you. > > _______________________________________________ > NOTE: All exchanges posted to Unidata maintained email lists are > recorded in the Unidata inquiry tracking system and made publicly > available through the web. Users who post to any of the lists we > maintain are reminded to remove any personal information that they > do not want to be made public. > > > netcdf-java mailing list > netcdf-java@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/
netcdf-java archives: