NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [thredds] NetCDF Access failure with matlab and python

  • To: Antonio S. Cofiño <cofinoa@xxxxxxxxx>
  • Subject: Re: [thredds] NetCDF Access failure with matlab and python
  • From: Thomas Cook <tmcook@xxxxxxxx>
  • Date: Fri, 18 Jan 2019 11:08:22 -0800
This is great to know. Thanks!

On Fri, Jan 18, 2019 at 11:05 AM Antonio S. Cofiño <cofinoa@xxxxxxxxx> wrote:
>
> Tom,
>
> To show the logs from client side, try the following trick for the
> netcdf-c library (i.e matlab and python packages)
>
> Add a .dodsrc to your working directory with a HTTP.VERBOSE=1 line. For
> example:
>
> $ echo "HTTP.VERBOSE=1" > .dodsrc
>
> and from the same working directory execute your matlab/python code and
> the log from the requests and responses are shown in the standard output:
>
>  >>
> ncinfo('http://hfrnet-tds.ucsd.edu/thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd')
> * About to connect() to hfrnet-tds.ucsd.edu port 80 (#0)
> *   Trying 132.239.92.188... * connected
> * Connected to hfrnet-tds.ucsd.edu (132.239.92.188) port 80 (#0)
>  > GET
> /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dds
> HTTP/1.1
> User-Agent: libcurl-agent/1.0
> Host: hfrnet-tds.ucsd.edu
> Accept: */*
>
> < HTTP/1.1 200
> < Server: nginx/1.11.12
> < Date: Fri, 18 Jan 2019 18:55:49 GMT
> < Content-Type: text/plain
> < Transfer-Encoding: chunked
> < Connection: keep-alive
> < XDODS-Server: opendap/3.7
> < Content-Description: dods-dds
> < Access-Control-Allow-Origin: *
> < Access-Control-Allow-Methods: GET, POST, OPTIONS
> < Access-Control-Allow-Headers:
> DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type
> <
> * Connection #0 to host hfrnet-tds.ucsd.edu left intact
> * Re-using existing connection! (#0) with host hfrnet-tds.ucsd.edu
> * Connected to hfrnet-tds.ucsd.edu (132.239.92.188) port 80 (#0)
>  > GET
> /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.das
> HTTP/1.1
> User-Agent: libcurl-agent/1.0
> Host: hfrnet-tds.ucsd.edu
> Accept: */*
>
> < HTTP/1.1 200
> < Server: nginx/1.11.12
> < Date: Fri, 18 Jan 2019 18:55:49 GMT
> < Content-Type: text/plain
> < Transfer-Encoding: chunked
> < Connection: keep-alive
> < XDODS-Server: opendap/3.7
> < Content-Description: dods-das
> < Access-Control-Allow-Origin: *
> < Access-Control-Allow-Methods: GET, POST, OPTIONS
> < Access-Control-Allow-Headers:
> DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type
> <
> * Connection #0 to host hfrnet-tds.ucsd.edu left intact
> * Re-using existing connection! (#0) with host hfrnet-tds.ucsd.edu
> * Connected to hfrnet-tds.ucsd.edu (132.239.92.188) port 80 (#0)
>  > GET
> /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,procParams
> HTTP/1.1
> User-Agent: libcurl-agent/1.0
> Host: hfrnet-tds.ucsd.edu
> Accept: */*
>
> < HTTP/1.1 200
> < Server: nginx/1.11.12
> < Date: Fri, 18 Jan 2019 18:55:49 GMT
> < Content-Type: application/octet-stream
> < Transfer-Encoding: chunked
> < Connection: keep-alive
> < XDODS-Server: opendap/3.7
> < Content-Description: dods-data
> < Access-Control-Allow-Origin: *
> < Access-Control-Allow-Methods: GET, POST, OPTIONS
> < Access-Control-Allow-Headers:
> DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type
> <
> * Connection #0 to host hfrnet-tds.ucsd.edu left intact
> * Closing connection #0
>
> ans =
>
>        Filename:
> 'http://hfrnet-tds.ucsd.edu/thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd'
>            Name: '/'
>      Dimensions: [1x6 struct]
>       Variables: [1x14 struct]
>      Attributes: [1x24 struct]
>          Groups: []
>          Format: '64bit'
>
>  >>
>
> --
> Antonio S. Cofiño
> Dep. de Matemática Aplicada y
>          Ciencias de la Computación
> Universidad de Cantabria
> http://www.meteo.unican.es
>
> On 17/1/19 5:40, Thomas Cook wrote:
> > I do not know how to debug the matlab or python clients, but this is
> > what shows up in the threddsServlet.log
> >
> > Request: "GET 
> > /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,site%5fcode,site%5fnetCode,procParams
> > HTTP/1.0"
> >
> > Thanks
> > Tom
> >
> > On Wed, Jan 16, 2019 at 7:47 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:
> >> Is there anyway you can see the exact URL being sent to the server?
> >>
> >> On 1/16/2019 8:24 PM, Thomas Cook wrote:
> >>> Thanks Sean.
> >>> The server is http://hfrnet-tds.ucsd.edu/thredds (although I just
> >>> downgraded back to 4.6.12-SNAPSHOT). I will see about making my test
> >>> server accessible to you, but that will take a little time to
> >>> configure on my end.
> >>> Let me know if you think I should open an issue on the github page and
> >>> move the discussion there.
> >>> Tom
> >>>
> >>> On Wed, Jan 16, 2019 at 6:32 PM Sean Arms <sarms@xxxxxxxx> wrote:
> >>>> Hi Tom,
> >>>>
> >>>> So you are hitting up an opendap endpoint using all three libraries / 
> >>>> tools (python, matlab, and panoply)? Is this particular server publicly 
> >>>> accessible?
> >>>>
> >>>> I have a bad feeling that this is due to URL encoding (or lack thereof) 
> >>>> on the client side and changes made in tomcat 8.5, which it looks like 
> >>>> the thredds-docker repository bumped up to with the 4.6.12 release.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Sean
> >>>>
> >>>>
> >>>> On Wed, Jan 16, 2019 at 7:06 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> >>>>> HI I just wanted to add some details.
> >>>>>
> >>>>> The functionality works when I downgraded to the docker container
> >>>>> 4.6.12-SNAPSHOT
> >>>>>
> >>>>> My data is part of a FMRC collection.
> >>>>>
> >>>>> Thanks,
> >>>>> Tom
> >>>>>
> >>>>> On Wed, Jan 16, 2019 at 4:51 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> >>>>>> HI
> >>>>>> I just yesterday updated my thredds-docker to 4.6.12 from
> >>>>>> 4.6.12-snapshot. It seems to be running fine, except I'm getting
> >>>>>> errors using matlab ncread and python netcdf4.dataset. When I try to
> >>>>>> access any of the multi-dimensional (time, lat, lon) variables I get
> >>>>>> the following errors
> >>>>>> with matlab:
> >>>>>> Error using netcdflib
> >>>>>> The NetCDF library encountered an error during execution of
> >>>>>> 'getVarsShort' function - 'Access failure (-77)'.
> >>>>>>
> >>>>>> Error in netcdf.getVar (line 136)
> >>>>>>       data = netcdflib(funcstr,ncid,varid,varargin{:});
> >>>>>>
> >>>>>> Error in internal.matlab.imagesci.nc/read (line 635)
> >>>>>>                   data  = netcdf.getVar(gid, varid, ...
> >>>>>>
> >>>>>> Error in ncread (line 58)
> >>>>>> vardata = ncObj.read(varName, varargin{:});
> >>>>>>
> >>>>>> Error in readUVfromTDS (line 56)
> >>>>>> u=ncread(url, 'u_mean', [1 1 1], [2 2 2]);
> >>>>>>
> >>>>>>
> >>>>>> and with python:
> >>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()
> >>>>>>
> >>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()
> >>>>>>
> >>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
> >>>>>>
> >>>>>> RuntimeError: NetCDF: Access failure
> >>>>>>
> >>>>>> I know that the latest docker contains netcdf 4.6, where as the
> >>>>>> snapshot had 4.4. Could it be a version problem with the netCD4 python
> >>>>>> and matlab libraries? I should be running the latest netcdf python
> >>>>>> version and not too far behind with matlab versions.
> >>>>>>
> >>>>>> The weird things are that the index variables are accessible (time,
> >>>>>> lat, lon etc),  and the multiple-dimension data is accessible just
> >>>>>> fine using the OpenDap web interface and Panoply. So I'm a little
> >>>>>> confused. I'm addressing this list since I think there might be some
> >>>>>> matlab and python expertise here and I think there were some messages
> >>>>>> about the Access failure error in the past.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Tom
> >>>>> _______________________________________________
> >>>>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>>>> recorded in the Unidata inquiry tracking system and made publicly
> >>>>> available through the web.  Users who post to any of the lists we
> >>>>> maintain are reminded to remove any personal information that they
> >>>>> do not want to be made public.
> >>>>>
> >>>>>
> >>>>> thredds mailing list
> >>>>> thredds@xxxxxxxxxxxxxxxx
> >>>>> For list information or to unsubscribe,  visit: 
> >>>>> http://www.unidata.ucar.edu/mailing_lists/
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made publicly
> >>> available through the web.  Users who post to any of the lists we
> >>> maintain are reminded to remove any personal information that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> thredds mailing list
> >>> thredds@xxxxxxxxxxxxxxxx
> >>> For list information or to unsubscribe,  visit: 
> >>> http://www.unidata.ucar.edu/mailing_lists/
> >> _______________________________________________
> >> NOTE: All exchanges posted to Unidata maintained email lists are
> >> recorded in the Unidata inquiry tracking system and made publicly
> >> available through the web.  Users who post to any of the lists we
> >> maintain are reminded to remove any personal information that they
> >> do not want to be made public.
> >>
> >>
> >> thredds mailing list
> >> thredds@xxxxxxxxxxxxxxxx
> >> For list information or to unsubscribe,  visit: 
> >> http://www.unidata.ucar.edu/mailing_lists/
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web.  Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe,  visit: 
> > http://www.unidata.ucar.edu/mailing_lists/
  • 2019 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: