NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Greetings Spicer, I dug into this more over the weekend, and it turns out two files are missing the project variable: FRF_20161116_1128_FRF_NAVD88_LARC_GPS_UTC_v20170320_grid_latlon.nc FRF_20200110_1180_FRF_NAVD88_LARC_GPS_UTC_v20200113_grid_latlon.nc If you add a project variable to those files, the aggregation works (tested locally with your files, ncml, and original python code). One thing I noticed - there are several files with the same time value, so in the aggregation you end up with duplicate time values without a way for users to distinguish where they came from (i.e. which version). A list of those files are at the end of this message. Cheers! Sean Files with the same time: FRF_19950420_0743_FRF_NAVD88_CRAB_Geodimeter_UTC_v20151115_grid_latlon.nc, FRF_19950420_0743_FRF_NAVD88_CRAB_Geodimeter_UTC_v20190326_grid_latlon.nc FRF_20150429_1100_FRF_NAVD88_LARC_GPS_UTC_v20160323_grid_latlon.nc, FRF_20150429_1100_FRF_NAVD88_LARC_GPS_UTC_v20190326_grid_latlon.nc FRF_20150618_1102_FRF_NAVD88_LARC_GPS_UTC_v20170328_grid_latlon.nc, FRF_20150618_1102_FRF_NAVD88_LARC_GPS_UTC_v20190326_grid_latlon.nc FRF_20151014_1108_FRF_NAVD88_LARC_GPS_UTC_v20170328_grid_latlon.nc, FRF_20151014_1108_FRF_NAVD88_LARC_GPS_UTC_v20190330_grid_latlon.nc FRF_20151221_1115_FRF_NAVD88_LARC_GPS_UTC_v20170320_grid_latlon.nc, FRF_20151221_1115_FRF_NAVD88_LARC_GPS_UTC_v20190326_grid_latlon.nc FRF_20160817_1122_FRF_NAVD88_LARC_GPS_UTC_v20170320_grid_latlon.nc, FRF_20160817_1122_FRF_NAVD88_LARC_GPS_UTC_v20190326_grid_latlon.nc FRF_20160926_1124_FRF_NAVD88_LARC_GPS_UTC_v20170320_grid_latlon.nc, FRF_20160926_1124_FRF_NAVD88_LARC_GPS_UTC_v20190330_grid_latlon.nc FRF_20161003_1125_FRF_NAVD88_LARC_GPS_UTC_v20170320_grid_latlon.nc, FRF_20161003_1125_FRF_NAVD88_LARC_GPS_UTC_v20190330_grid_latlon.nc FRF_20161020_1126_FRF_NAVD88_LARC_GPS_UTC_v20170320_grid_latlon.nc, FRF_20161020_1126_FRF_NAVD88_LARC_GPS_UTC_v20190330_grid_latlon.nc FRF_20161116_1128_FRF_NAVD88_LARC_GPS_UTC_v20170320_grid_latlon.nc (also missing project variable), FRF_20161116_1128_FRF_NAVD88_LARC_GPS_UTC_v20190330_grid_latlon.nc FRF_20170105_1129_FRF_NAVD88_LARC_GPS_UTC_v20170320_grid_latlon.nc, FRF_20170105_1129_FRF_NAVD88_LARC_GPS_UTC_v20190326_grid_latlon.nc FRF_20171011_1142_FRF_NAVD88_LARC_GPS_UTC_v20171012_grid_latlon.nc, FRF_20171011_1143_FRF_NAVD88_LARC_GPS_UTC_v20171221_grid_latlon.nc FRF_20171121_1143_FRF_NAVD88_LARC_GPS_UTC_v20171129_grid_latlon.nc, FRF_20171121_1144_FRF_NAVD88_LARC_GPS_UTC_v20171221_grid_latlon.nc, FRF_20171121_1144_FRF_NAVD88_LARC_GPS_UTC_v20180130_grid_latlon.nc FRF_20180418_1149_FRF_NAVD88_LARC_GPS_UTC_v20180427_grid_latlon.nc, FRF_20180418_1149_FRF_NAVD88_LARC_GPS_UTC_v20190326_grid_latlon.nc FRF_20190917_1170_FRF_NAVD88_CRAB_GPS_UTC_v20190919_grid_latlon.nc, FRF_20190917_1170_FRF_NAVD88_CRAB_GPS_UTC_v20191029_grid_latlon.nc On Fri, Jan 31, 2020 at 3:42 PM Sean Arms <sarms@xxxxxxxx> wrote: > Greetings Spicer, > > I think there is an issue with your new project variable. In previous > files, it's a float, > > > https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/FRF_20191206_1179_FRF_NAVD88_LARC_GPS_UTC_v20191209_grid_latlon.nc.ascii?project%5B0:1:0%5D > > Dataset { > Float64 project[time = 1]; > } > frf/geomorphology/DEMs/surveyDEM/FRF_20191206_1179_FRF_NAVD88_LARC_GPS_UTC_v20191209_grid_latlon.nc; > --------------------------------------------- > project[1] > -999.0 > > but in the new latest file, it's a string: > > > https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/FRF_20200110_1180_FRF_NAVD88_LARC_GPS_UTC_v20200113_grid_latlon.nc.ascii?project > > Dataset { > String project; > } > frf/geomorphology/DEMs/surveyDEM/FRF_20200110_1180_FRF_NAVD88_LARC_GPS_UTC_v20200113_grid_latlon.nc; > --------------------------------------------- > project, "F" > > That might cause new kinds of issues for a full variable read. > > Cheers! > > Sean > > > On Fri, Jan 31, 2020 at 3:33 PM Spicer Bak <spicer.bak.frf@xxxxxxxxx> > wrote: > >> Hey Sean, >> Glad we were able to help find that bug, but I don't think the "project" >> variable (or lack of) is the root of our problem as i chose your option 3 >> (my mistake, this was supposed to be the same after the last one) and i >> have similar response. Good news, when i add the #noprefetch option, it >> seems to fix it. hopefully this helps provide answers. Demonstrated by >> below code. >> >> # Failure with python (matlab as well) >> import netCDF4 as nc >> for url in urls: >> print(nc.Dataset(url)['time']) >> variables= nc.Dataset(url).variables.keys() >> for var in variables: >> try: >> nc.Dataset(url)[var][0] >> print('Success! {} from {}'.format(var, url)) >> except IndexError: >> print("won't load variable {} from {}".format(var, url)) >> url += "#noprefetch" >> nc.Dataset(url)[var][0] >> print('Success! {} from {}'.format(var, url)) >> except IndexError as e: >> print("FAIL: load variable {} from {}".format(var, url)) >> >> print(' {}'.format(e)) >> >> >> >> On Fri, Jan 31, 2020 at 4:25 PM Sean Arms <sarms@xxxxxxxx> wrote: >> >>> I thought it was working for me, but for the wrong reasons. Sorry about >>> that. But, now I have it. >>> >>> The error message from the server is...well...garbage. That's something >>> we need to look into. The underlying problem here is that the new data file >>> does not include the variable project. Because variable "project" exists in >>> the other files (or more specifically, the first file of the aggregation), >>> and specifically because it has time as it's outer dimension, currently it >>> must exist in all files of the aggregation. So, we can make an ascii >>> request for that variable specifically, and request everything up to the >>> last value, and it works [0:1:424]: >>> >>> >>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml.ascii?project%5B0:1:424%5D >>> >>> >>> However, once we ask for the last value, it bombs [0:1:424] : >>> >>> >>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml.ascii?project%5B0:1:425%5D >>> >>> It's only like this for the full read code path, though. If I slice it >>> from [1:1:424] (skip the first value), it works >>> >>> >>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml.ascii?project%5B1:1:425%5D >>> >>> ...and the missing value gets filled with a zero, because why not (well, >>> ok, I can think of a few reasons). So, from the TDS side, and more >>> specifically netCDF-Java, this is a bug (well, bugs, because returning zero >>> isn't quite the thing to do here either). >>> >>> From what I can understand, netCDF-C tries to preload some data from >>> remote servers (if the variable is considered "small enough"), and because >>> the variable "project" is "small enough", the library tries to grab it all >>> and the request bombs out. You can tell the C library to not do that by >>> adding "#noprefetch" at the end of the url when opening the dataset, but >>> that just delays things (unless you or the underlying library you are using >>> never tries to fully read the variable "project"). So, in your sample >>> python code, pass Netcdf4Python's Dataset the following URL and give it a >>> spin: >>> >>> >>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml#noprefetch >>> >>> Cool...so what to do until I can get this fixed. Three things off the >>> top of my head: >>> >>> 1. Add "#noprefetch" to your URL (and anyone else using the dataset). >>> Barf. >>> 2. Use NcML in the aggregation to remove the variable "project" all >>> together, as it seems to be -999.0 when it does exist anyways. >>> 3. Add the variable "project" to the latest file (NcML or by rewriting >>> it). >>> >>> Sorry this took so long to debug. There will be a fix, it will just take >>> some time (maybe early next week since I believe I know exactly what needs >>> done and where). >>> >>> Cheers, >>> >>> Sean >>> >>> >>> On Fri, Jan 31, 2020 at 12:27 PM Spicer Bak <spicer.bak.frf@xxxxxxxxx> >>> wrote: >>> >>>> Hey Sean, >>>> Thanks for getting back to me. I was still getting the same symptoms >>>> earlier today. Did it work on second inquiry for you? >>>> >>>> I did make a change to that last file so the dim's match up with >>>> previous. After I did that, checked again to make sure that wasn't causing >>>> the problem. Seems i still am getting the same IndexError i was getting, >>>> so this (as you mentioned) is not the root of the problem. >>>> >>>> On Fri, Jan 31, 2020 at 11:03 AM Sean Arms <sarms@xxxxxxxx> wrote: >>>> >>>>> Greetings Spicer, >>>>> >>>>> It looks like you have solved the issue? I was having problems the >>>>> other day as well. I downloaded the files locally to see if I could >>>>> reproduce the issue, and I am unable to do so now. However, I do see >>>>> something that might cause an issue down the road. The dimension order on >>>>> some of the variables in the latest file does not match what is in other >>>>> files. For example, if we compare the latest file: >>>>> >>>>> >>>>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/FRF_20200110_1180_FRF_NAVD88_LARC_GPS_UTC_v20200113_grid_latlon.nc.dds >>>>> >>>>> with next latest file: >>>>> >>>>> >>>>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/FRF_20191206_1179_FRF_NAVD88_LARC_GPS_UTC_v20191209_grid_latlon.nc.dds >>>>> >>>>> The dimensionality of latitude, longitude, northing, and easting >>>>> switch from (xFRF, yFRF) (20191206) to (yFRF, xFRF) (20200113).Might not >>>>> be >>>>> a big issue overall since the use of an NcML file on the server (as >>>>> opposed >>>>> to NcML directly in the catalog) uses the first file of the aggregation as >>>>> template. >>>>> >>>>> Cheers, >>>>> >>>>> Sean >>>>> >>>>> >>>>> On Wed, Jan 29, 2020 at 3:03 PM Spicer Bak <spicer.bak.frf@xxxxxxxxx> >>>>> wrote: >>>>> >>>>>> hello TDS community, >>>>>> I have a problem I'm quite pickled on. We had a dataset that was >>>>>> working fine until (i think) we pushed the latest file. Our server is >>>>>> continually updated with new files and all of the other datasets seem to >>>>>> work fine. >>>>>> >>>>>> I'm able to get the data from the individual file URLS, but not the >>>>>> time concatenated ncml file. I can display the file or variables, but >>>>>> not >>>>>> obtain any of the data through OPeNDAP, but i'm able to see the data >>>>>> returned via the "get ASCII" button on the OPeNDAP page. The below >>>>>> python >>>>>> script demonstrates the problem: >>>>>> >>>>>> # success with ncml: >>>>>> >>>>>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml.ascii?latitude%5B0:1:0%5D%5B0:1:0%5D,time%5B0:1:425%5D,elevation%5B0:1:0%5D%5B0:1:0%5D%5B0:1:0%5D >>>>>> >>>>>> # Failure with python (matlab as well) >>>>>> import netCDF4 as nc >>>>>> for url in urls: >>>>>> print(nc.Dataset(url)['time']) >>>>>> variables= nc.Dataset(url).variables.keys() >>>>>> for var in variables: >>>>>> try: >>>>>> nc.Dataset(url)[var][0] >>>>>> print('Success! {} from {}'.format(var, url)) >>>>>> except IndexError as e: >>>>>> print("won't load variable {} from {}".format(var, url)) >>>>>> >>>>>> print(' {}'.format(e)) >>>>>> >>>>>> Any help would be much appreciated! >>>>>> >>>>>> -- >>>>>> +++++++++++++++++++++++++++ >>>>>> Spicer Bak, PhD >>>>>> USACE CHL Field Research Facility >>>>>> 252-305-9975 >>>>>> _______________________________________________ >>>>>> NOTE: All exchanges posted to Unidata maintained email lists are >>>>>> recorded in the Unidata inquiry tracking system and made publicly >>>>>> available through the web. Users who post to any of the lists we >>>>>> maintain are reminded to remove any personal information that they >>>>>> do not want to be made public. >>>>>> >>>>>> >>>>>> thredds mailing list >>>>>> thredds@xxxxxxxxxxxxxxxx >>>>>> For list information or to unsubscribe, visit: >>>>>> https://www.unidata.ucar.edu/mailing_lists/ >>>>>> >>>>> >>>> >>>> -- >>>> +++++++++++++++++++++++++++ >>>> Spicer Bak, PhD >>>> USACE CHL Field Research Facility >>>> 252-305-9975 >>>> >>> >> >> -- >> +++++++++++++++++++++++++++ >> Spicer Bak, PhD >> USACE CHL Field Research Facility >> 252-305-9975 >> >
thredds archives: