Re: [thredds] TDS NCSS: issue with netCDF4 files as input and netCDF4 requested as output

• To: Jolibois Tony <tjolibois@xxxxxx>
• Subject: Re: [thredds] TDS NCSS: issue with netCDF4 files as input and netCDF4 requested as output
• From: Ryan May <rmay@xxxxxxxx>
• Date: Fri, 11 Dec 2015 16:29:41 -0700

Thanks for the files, I'm able to reproduce the problem locally.
Unfortunately, it doesn't appear to be related to the issue #301 as I
thought.

I've opened a new issue: https://github.com/Unidata/thredds/issues/330

You can follow any progress there as I continue to dig.

Thanks,

Ryan

On Fri, Dec 11, 2015 at 1:45 AM, <tjolibois@xxxxxx> wrote:

> Hi Christian,
> Please find some FTP link in the end of this email, to download some
> netCDF4 files we've used here for the tests. I've also added part of
> catalina.out, the java_opts used, and the Thredds configuration (simple
> joinExisting aggregation).
> Note that it's only when you request netCDF4 as output that we have an
> error.
> Best regards,
> Tony
>
>
>
> Hi Tony,
>
> Can you provide an example of the NetCDF-4 input file so that we can
> recreate the problem locally?
>
> Thanks,
> Christian
>
> On Thu, Dec 10, 2015 at 8:29 AM, Jolibois Tony wrote:
> Hi Thredds team,
>
> We’re currently testing TDS netCDF Subset Service, its performance,
> robustness.. These tests include different netCDF format, to study the
> impact on performance and robustness with different compression levels.
> To study this impact, we run 4 scenarios for input/output data files for
> TDS NCSS.
> Input netCDF3 => output netCDF3
> Input netCDF4 => output netCDF3
> Input netCDF3 => output netCDF4
> Input netCDF4 => output netCDF4
> We’re facing an error only with the 4th scenario, netCDF4 => netCDF4 (NCSS
> REST parameter &accept=netcdf4).
>
> java.io.IOException: -101: NetCDF: HDF error
> at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2821)
> ~[netcdf4-4.6.3.jar:4.6.3]
> at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2729)
> ~[netcdf4-4.6.3.jar:4.6.3]
> at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:953)
> ~[cdm-4.6.3.jar:4.6.3]
> at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:937)
> ~[cdm-4.6.3.jar:4.6.3]
> at ucar.nc2.FileWriter2.copyAll(FileWriter2.java:427)
> ~[cdm-4.6.3.jar:4.6.3]
> at ucar.nc2.FileWriter2.copyVarData(FileWriter2.java:380)
> ~[cdm-4.6.3.jar:4.6.3]
> at ucar.nc2.dt.grid.CFGridWriter2.writeOrTestSize(CFGridWriter2.java:244)
> ~[cdm-4.6.3.jar:4.6.3]
> at ucar.nc2.dt.grid.CFGridWriter2.writeFile(CFGridWriter2.java:112)
> ~[cdm-4.6.3.jar:4.6.3]
> at
> thredds.server.ncss.controller.GridResponder.writeGridFile(GridResponder.java:250)
> ~[classes/:4.6.3]
> at
> thredds.server.ncss.controller.GridResponder.writeLatLonSubset(GridResponder.java:166)
> ~[classes/:4.6.3]
> at
> thredds.server.ncss.controller.GridResponder.getResponseFile(GridResponder.java:97)
> ~[classes/:4.6.3]
> at
> thredds.server.ncss.controller.NcssController.handleRequestGrid(NcssController.java:175)
> ~[classes/:4.6.3]
> at
> thredds.server.ncss.controller.NcssController.handleRequest(NcssController.java:140)
> ~[classes/:4.6.3]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_65]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_65]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_65]
> at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]
>
> The error comes from netcdf4 java library used inside TDS version 4.6.3
> (latest stable version).
> The error appears only if we have a geographical subset (works with no
> subset or temporal aggregation).
> Different NetCDF-4 input files have been tested:
> • original Netcdf3 files converted with the command nccopy –k4 –d3
> input.nc output.nc4
> • original Netcdf4 generated with python library
> For information on software versions:
> misgw-qt-new:/home/misgw# ls -ltr
> total 246300
> -rw-r--r-- 1 root root 9130958 oct 7 21:09 apache-tomcat-8.0.28.tar.gz
> -rw-rw-r-- 1 misgw misgw 181260798 nov 9 10:48 jdk-8u65-linux-x64.gz
> -rw-r--r-- 1 root root 61808104 nov 9 11:03 thredds_4.6.3.war
>
> Thanks for the support.
> Tony Jolibois
>
> -
> Tony Jolibois
> Product Dissemination Services
> Space Oceanography
> +33 5 61 39 37 97
> Files attached to this message Filename Size netCDF4.zip 13 MB
> catalina.out 6 KB java_opts.sh 1012.0 B m_LR_OBS_nc4.xml 2 KB
> m_Surcouf_C4.xml 2 KB
>
> Please click on the following link to download the attachments:
> http://ftp-mail.cls.fr/message/XY07Uo3bGqdx38DKHGNgyD
>
> The attachments are available until: *Friday, 18 December.*
>
> Message ID: XY07Uo3
> CLS — Secure Filetransfer Appliance
> <http://www.allardsoft.com/filetransfer>: http://ftp-mail.cls.fr/
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
>



-- 
Ryan May
Software Engineer
UCAR/Unidata
Boulder, CO

• Follow-Ups:
  • Re: [thredds] TDS NCSS: issue with netCDF4 files as input and netCDF4 requested as output
    • From: Jolibois Tony

• References:
  • Re: [thredds] TDS NCSS: issue with netCDF4 files as input and netCDF4 requested as output
    • From: tjolibois