Re: [thredds] TDS NCSS: issue with netCDF4 files as input and netCDF4 requested as output

• To: Ryan May <rmay@xxxxxxxx>
• Subject: Re: [thredds] TDS NCSS: issue with netCDF4 files as input and netCDF4 requested as output
• From: Jolibois Tony <tjolibois@xxxxxx>
• Date: Mon, 14 Dec 2015 10:25:36 +0000

Dear Ryan,
Ok, new issue. Do you have a planning for the correction ? More generally, do 
you have a roadmap for Thredds Data Server developments ? The next versions, 
what will be corrected etc ?
Thanks.
Best regards,
Tony

De : Ryan May [mailto:rmay@xxxxxxxx]
Envoyé : samedi 12 décembre 2015 00:30
À : Jolibois Tony
Cc : THREDDS community; joan.sala@xxxxxxxxxxxxxxxxxxxxxxxx
Objet : Re: [thredds] TDS NCSS: issue with netCDF4 files as input and netCDF4 
requested as output

Thanks for the files, I'm able to reproduce the problem locally. Unfortunately, 
it doesn't appear to be related to the issue #301 as I thought.

I've opened a new issue: https://github.com/Unidata/thredds/issues/330

You can follow any progress there as I continue to dig.

Thanks,

Ryan

On Fri, Dec 11, 2015 at 1:45 AM, <tjolibois@xxxxxx<mailto:tjolibois@xxxxxx>> 
wrote:
Hi Christian,
Please find some FTP link in the end of this email, to download some netCDF4 
files we've used here for the tests. I've also added part of catalina.out, the 
java_opts used, and the Thredds configuration (simple joinExisting aggregation).
Note that it's only when you request netCDF4 as output that we have an error.
Best regards,
Tony



Hi Tony,

Can you provide an example of the NetCDF-4 input file so that we can recreate 
the problem locally?

Thanks,
Christian
On Thu, Dec 10, 2015 at 8:29 AM, Jolibois Tony wrote:
Hi Thredds team,

We’re currently testing TDS netCDF Subset Service, its performance, 
robustness.. These tests include different netCDF format, to study the impact 
on performance and robustness with different compression levels.
To study this impact, we run 4 scenarios for input/output data files for TDS 
NCSS.
Input netCDF3 => output netCDF3
Input netCDF4 => output netCDF3
Input netCDF3 => output netCDF4
Input netCDF4 => output netCDF4
We’re facing an error only with the 4th scenario, netCDF4 => netCDF4 (NCSS REST 
parameter &accept=netcdf4).

java.io.IOException: -101: NetCDF: HDF error
at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2821) 
~[netcdf4-4.6.3.jar:4.6.3]
at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2729) 
~[netcdf4-4.6.3.jar:4.6.3]
at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:953) 
~[cdm-4.6.3.jar:4.6.3]
at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:937) 
~[cdm-4.6.3.jar:4.6.3]
at ucar.nc2.FileWriter2.copyAll(FileWriter2.java:427) ~[cdm-4.6.3.jar:4.6.3]
at ucar.nc2.FileWriter2.copyVarData(FileWriter2.java:380) ~[cdm-4.6.3.jar:4.6.3]
at ucar.nc2.dt.grid.CFGridWriter2.writeOrTestSize(CFGridWriter2.java:244) 
~[cdm-4.6.3.jar:4.6.3]
at ucar.nc2.dt.grid.CFGridWriter2.writeFile(CFGridWriter2.java:112) 
~[cdm-4.6.3.jar:4.6.3]
at 
thredds.server.ncss.controller.GridResponder.writeGridFile(GridResponder.java:250)
 ~[classes/:4.6.3]
at 
thredds.server.ncss.controller.GridResponder.writeLatLonSubset(GridResponder.java:166)
 ~[classes/:4.6.3]
at 
thredds.server.ncss.controller.GridResponder.getResponseFile(GridResponder.java:97)
 ~[classes/:4.6.3]
at 
thredds.server.ncss.controller.NcssController.handleRequestGrid(NcssController.java:175)
 ~[classes/:4.6.3]
at 
thredds.server.ncss.controller.NcssController.handleRequest(NcssController.java:140)
 ~[classes/:4.6.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_65]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_65]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]

The error comes from netcdf4 java library used inside TDS version 4.6.3 (latest 
stable version).
The error appears only if we have a geographical subset (works with no subset 
or temporal aggregation).
Different NetCDF-4 input files have been tested:
• original Netcdf3 files converted with the command nccopy –k4 –d3 
input.nc<http://input.nc> output.nc4
• original Netcdf4 generated with python library
For information on software versions:
misgw-qt-new:/home/misgw# ls -ltr
total 246300
-rw-r--r-- 1 root root 9130958 oct 7 21:09 apache-tomcat-8.0.28.tar.gz
-rw-rw-r-- 1 misgw misgw 181260798 nov 9 10:48 jdk-8u65-linux-x64.gz
-rw-r--r-- 1 root root 61808104 nov 9 11:03 thredds_4.6.3.war

Thanks for the support.
Tony Jolibois

-
Tony Jolibois
Product Dissemination Services
Space Oceanography
+33 5 61 39 37 97<tel:%2B33%205%2061%2039%2037%2097>
Files attached to this message
Filename

Size

netCDF4.zip

13 MB

catalina.out

6 KB

java_opts.sh

1012.0 B

m_LR_OBS_nc4.xml

2 KB

m_Surcouf_C4.xml

2 KB


Please click on the following link to download the attachments: 
http://ftp-mail.cls.fr/message/XY07Uo3bGqdx38DKHGNgyD

The attachments are available until: Friday, 18 December.

Message ID: XY07Uo3
CLS — Secure Filetransfer Appliance<http://www.allardsoft.com/filetransfer>: 
http://ftp-mail.cls.fr/

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx<mailto:thredds@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/



--
Ryan May
Software Engineer
UCAR/Unidata
Boulder, CO


Cliquez 
ici<https://www.mailcontrol.com/sr/EHc7LJiOdsvGX2PQPOmvUtiZD!0EqjaRqtHcKzZIQrbuR32cwTUJp3AJPte9EuoTxiDOCmOH7hfLq66BGWNXYg==>
 si ce message est indésirable (pourriel).

• References:
  • Re: [thredds] TDS NCSS: issue with netCDF4 files as input and netCDF4 requested as output
    • From: tjolibois
  • Re: [thredds] TDS NCSS: issue with netCDF4 files as input and netCDF4 requested as output
    • From: Ryan May