NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Greetings John!
On Wed, Apr 29, 2020 at 3:42 PM John Maurer <jmaurer@xxxxxxxxxx> wrote:
> Is there an easy way to determine the binLimit setting for a remote TDS?
>
>
Currently there isn't a way to get those limits a priori. We could probably
add something that exposes those limits without the need to make a request,
but even then, interpreting those limits can get tricky when compression is
involved (see below).
> Also, when binLimit is exceed in OPeNDAP, does it return an error message?
> I know an ASCII request spits out a 403 message like the following:
>
>> Error {
>> code = 403;
>> message = "Request too big=1275173.181384 Mbytes, max=1000.0";
>> };
>>
>> I guess how that message gets conveyed depends on the client, too (e.g.,
> Python, Matlab, etc.)?
>
>
A binary request over the max limit will contain a similar message body.
It's up to the client to check the message and expose it to the user. There
was a recent github issue over on the netCDF-C repository which I believe
led to having the C library at least return the message body, so now that
kind of message should be visible:
https://github.com/Unidata/netcdf-c/issues/1667
That issue also outlines the trickiness of compression I mention above.
What would be even better is to fix the underlying issue that causes the
need for having a limit on the server side. The performance section of the
TDS docs mentions
"The OPeNDAP-Java layer of the server currently has to read the entire data
request into memory before sending it to the client (we hope to get a
streaming I/O solution working eventually). Generally clients only request
subsets of large files, but if you need to support large data requests,
make sure that the -Xmx parameter above is set accordingly." (
https://www.unidata.ucar.edu/software/tds/current/reference/Performance.html
)
It's on my long list of fires, but I need to get TDS 5 out the door before
trying to address that one. On a related note, Unidata is hiring if you
know of anyone looking!
https://ucar.wd5.myworkdayjobs.com/en-US/UCAR_Careers/job/Foothills-Lab-4/THREDDS-Developer---Software-Engineer-II_REQ-2020-107-1
Certainly a candidate with the right skillset could dive into
this...although we might have to reopen the position shortly after (the
code gets pretty complicated).
On a related note, we're grappling with a TDS that returns all 0's when
> binLimit is exceeded, rather than returning an error status. Anybody seen
> this? I've seen mention online of a netcdf-c timeout issue?
>
I'd be interested in knowing more about this. Exceeding the binLimit should
always result in a 403.
I hope all i s well in Hawaii!
Sean
> Thanks for any insights!,
> John Maurer
> Data System Engineer
> Pacific Islands Ocean Observing System (PacIOOS)
> University of Hawaii at Manoa
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
thredds archives: