NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Greetings Nikhil, In your example output, we see the following message from the server: HTTP Status 400 – Bad Request: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986 This is generated by Tomcat, and most likely because the request contains unencoded square brackets. Using curl can simplify the troubleshooting process a bit. For example: curl -i http://138.194.55.191/thredds/dodsC/worldclim/historical/2.5m/prec/wc2.1_2.5m_prec.ncnetcdf.ascii?time will likely work just fine. Now, if we make a request that slices the time variable, like so: curl -i http://138.194.55.191/thredds/dodsC/worldclim/historical/2.5m/prec/wc2.1_2.5m_prec.ncnetcdf.ascii?time[0] this will fail. The issue here is that starting with Tomcat v9.0.8 (and also v8.5.31, v8.0.52 and v7.0.87), the presence of certain characters in the query portion of a URI (things after the question mark) will cause Tomcat to reject a request without question, and the underlying application will never see the request. Those characters are: < > [ \ ] ^ ` { | } In this case, it's the square brackets, and any time a client makes a subset request to OPeNDAP, square brackets are involved at some level. It turns out that square brackets were never considered "legal" for use in the query part of a URI, but Tomcat (and many others) have historically allowed them to pass through to the underlying application (in this case, the TDS), but that's no longer the case. If you want to make a request with those characters, you have to encode them first. When encoded, [ becomes %5B, and ] becomes %5D. If you replace the brackets with their encoded values and run the curl command again, I'm guessing things will work as expected: curl -i http://138.194.55.191/thredds/dodsC/worldclim/historical/2.5m/prec/wc2.1_2.5m_prec.ncnetcdf.ascii?time%5B0%5D Given all of that, there are basically two paths forward to get you up and running. 1. Allow Tomcat to let plain square brackets through. This is done on the server side, and will tell tomcat that square brackets are ok. At that point, the TDS will actually see the request and have a chance to return data. HOWEVER, there are reasons Tomcat chose to start blocking square brackets by default. Check out relaxedQueryChars at https://tomcat.apache.org/tomcat-9.0-doc/config/http.html. Some people view this as a security issue, and while I don't fully understand, I come down on the side of caution and would not recommend it in the general case. If you decide to go this route, research it well and know the risks - it's all going to be highly dependent on the environment in which you are running Tomcat (internal trusted clients only, which sounds like the case here, vs a server exposed to the world, for example). 2. Percent encode the query parts of a URL before making a request. It looks like NCO (or a library it uses) is making unencoded requests. I'm guessing NCO is relying on the netCDF-C library to make OPeNDAP requests. I believe a fix was introduced in netCDF-C last summer (https://github.com/Unidata/netcdf-c/pull/1439), so this may be something that can be fixed by rebuilding NCO with a more recent version of netCDF-C. Perhaps this version dependency is why NCO on one machine can make successful requests, while another cannot? Cheers, Sean Sean Arms, PhD Software Engineer UCAR/UCP/Unidata https://staff.ucar.edu/users/sarms On Mon, May 4, 2020 at 5:51 PM Nikhil Garg <nikhilgarg.gju@xxxxxxxxx> wrote: > > Hi, > > I am setting up a thredds data server on a local machine to serve some data > for private use within our research group. I have used the docker image > unidata/thredds-docker:latest to install and setup the server. I have also > managed to create and add a catalog. After setting up the server, I have > noticed a peculiar behaviour. If I use tools like nco etc. on the machine > where I have installed a thredds server, I am able to access the data. > However, when I try to access the data from a different machine, I get a > syntax error. > > I have tested to see if the file is accessible from the thredds server. > Below, you can see the output from ncdump. > > ncdump -t -v time > http://138.194.55.191/thredds/dodsC/worldclim/historical/2.5m/prec/wc2.1_2.5m_prec.ncnetcdf > wc2.1_2.5m_prec { > dimensions: > time = UNLIMITED ; // (12 currently) > lat = 4320 ; > lon = 8640 ; > variables: > double time(time) ; > time:standard_name = "time" ; > time:long_name = "time" ; > time:units = "months since 1985-01-01" ; > time:calendar = "standard" ; > time:axis = "T" ; > time:_ChunkSizes = 512 ; // "2027-09-01" > double lon(lon) ; > lon:standard_name = "longitude" ; > lon:long_name = "longitude" ; > lon:units = "degrees_east" ; > lon:axis = "X" ; > double lat(lat) ; > lat:standard_name = "latitude" ; > lat:long_name = "latitude" ; > lat:units = "degrees_north" ; > lat:axis = "Y" ; > float prec(time, lat, lon) ; > prec:units = "mm" ; > prec:_FillValue = -32768.f ; > prec:missing_value = -32768.f ; > prec:long_name = "precipitation" ; > prec:_ChunkSizes = 1, 7, 8640 ; > > // global attributes: > :CDI = "Climate Data Interface version 1.9.3 (http://mpimet.mpg.de/cdi)" ; > :history = "Thu Apr 30 17:43:57 2020: GDAL CreateCopy( > ./wc2.1_2.5m_prec_01.nc, ... )" ; > :GDAL_AREA_OR_POINT = "Area" ; > :GDAL = "GDAL 2.1.3, released 2017/20/01" ; > :CDO = "Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/cdo)" ; > :Conventions = "CF-1.5" ; > :DODS_EXTRA.Unlimited_Dimension = "time" ; > data: > > time = "1985-01-01", "1985-02-01", "1985-03-01", "1985-04-01", "1985-05-01", > "1985-06-01", "1985-07-01", "1985-08-01", "1985-09-01", "1985-10-01", > "1985-11-01", "1985-12-01" ; > } > > When I try to access the file using ncks e.g., > > ncks -d time,0,1,1 -v prec > http://138.194.55.191/thredds/dodsC/worldclim/historical/2.5m/prec/wc2.1_2.5m_prec.nc > temp.nc > > I get the following error > > syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or > SCAN_ERROR > context: <!doctype^ html><html lang="en"><head><title>HTTP Status 400 – Bad > Request</title><style type="text/css">body > {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b > {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 > {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} > .line > {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP > Status 400 – Bad Request</h1><hr class="line" /><p><b>Type</b> Exception > Report</p><p><b>Message</b> Invalid character found in the request target. > The valid characters are defined in RFC 7230 and RFC > 3986</p><p><b>Description</b> The server cannot or will not process the > request due to something that is perceived to be a client error (e.g., > malformed request syntax, invalid request message framing, or deceptive > request > routing).</p><p><b>Exception</b></p><pre>java.lang.IllegalArgumentException: > Invalid character found in the request target. The valid characters are > defined in RFC 7230 and RFC 3986 > org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:502) > org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:502) > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) > > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) > > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623) > > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > java.lang.Thread.run(Thread.java:748)</pre><p><b>Note</b> The full stack > trace of the root cause is available in the server logs.</p><hr class="line" > /><h3>Apache Tomcat</h3></body></html> > > I am not sure what I am missing here. I would appreciate some help in getting > this sorted. > > -- > Regards > > Nikhil > _______________________________________________ > NOTE: All exchanges posted to Unidata maintained email lists are > recorded in the Unidata inquiry tracking system and made publicly > available through the web. Users who post to any of the lists we > maintain are reminded to remove any personal information that they > do not want to be made public. > > > thredds mailing list > thredds@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > https://www.unidata.ucar.edu/mailing_lists/
thredds archives: