Re: [netcdf-java] Erroneous data from linked HDF files

• To: Christopher Mueller <cmueller@xxxxxxxxxxxxxx>
• Subject: Re: [netcdf-java] Erroneous data from linked HDF files
• From: Christian Ward-Garrison <cwardgar@xxxxxxxx>
• Date: Fri, 1 Aug 2014 17:17:28 -0600

Hi Chris,

First off, let me just say that this is an absolutely fantastic bug report.
I wish I had better news for you, but the simple answer is that NetCDF-Java
doesn't support linked HDF files. Frankly, you're the first use that's even
mentioned them to us. Is there a particular reason that you prefer linked
files to normal files?

Regards,
Christian


On Tue, Jul 15, 2014 at 1:29 PM, Christopher Mueller <
cmueller@xxxxxxxxxxxxxx> wrote:

>   *tl;dr* There appears to be a bug in NetCDF Java with respect to
> reading linked HDF4 files which results in data that is read from the
> linked file(s) to be erroneous.
>  Resources
>
>    - ToolsUI
>    - HDFView
>    - The files mentioned below can be retrieved directly from OceanColor
>    
> <http://oceancolor.gsfc.nasa.gov/cgi/l3/A20021822013212.L3b_MC_RRS.main.bz2?sub=bin>
>  (one
>    at a time), or (for convenience) as one tar.gz file fromhere
>    
> <https://drive.google.com/uc?id=0B6UT7Mn4GZQhMjdLNDBBMFE0TTA&export=download>
>
>  Details
>
> I'm reading data from the Aqua MODIS L3 Binned products available from the 
> NASA
> OceanColor <http://oceancolor.gsfc.nasa.gov/> website. It should be noted
> that these files are HDF4 (4.2.9 according to NetCDF Java - ncdump). Many
> of the products, such as chlorophyll, Particulate Inorganic Carbon, and Sea
> Surface Temperature, come as a single file. The NetCDF library reads these
> files without any difficulty.
>
> However, one of the datasets of interest is the Remote Sensing Reflectance
> data, which is NOT provided as a single file, but as a "main" file and a
> set of subordinate files which are read via the "main" file as needed (see
> here for more information
> <http://oceancolor.gsfc.nasa.gov/PRODUCTS/modis_binned.html>):
>
>    - A20021822013212.L3b_MC_RRS.main
>    - A20021822013212.L3b_MC_RRS.x00
>    - A20021822013212.L3b_MC_RRS.x01
>    - A20021822013212.L3b_MC_RRS.x02
>    - A20021822013212.L3b_MC_RRS.x03
>    - A20021822013212.L3b_MC_RRS.x04
>    - A20021822013212.L3b_MC_RRS.x05
>    - A20021822013212.L3b_MC_RRS.x06
>    - A20021822013212.L3b_MC_RRS.x07
>    - A20021822013212.L3b_MC_RRS.x08
>    - A20021822013212.L3b_MC_RRS.x09
>    - A20021822013212.L3b_MC_RRS.x10
>    - A20021822013212.L3b_MC_RRS.x11
>
>  NetCDF Java (via ToolsUI) loads the .main file without issue, and permits
> reading of data variables (i.e. Rrs_412) without raising any errors.
> However, the data returned is not accurate. Below is a comparison of the
> data returned by ToolsUI and the same data returned by HDFView (which uses
> the HDF-java JNI <http://www.hdfgroup.org/products/java/JNI/> library):
>
> Retrieving the first 10 values for variable "Rrs_412"
>  HDFView
>
> Screen Capture <http://cl.ly/WZnD>
>
> Opening the .main file in HDFView and looking at the Rrs_412 dataset gives
> a very different set of data:
>
> 0.0055423053, 0.0106070135, 0.006894292, -0.0040368317, -0.0020879991, 
> -0.0020279996, 0.009794002, 0.011879213, 0.010874448, 0.012330733
>
>  ToolsUI
>
> Screen Capture <http://cl.ly/WZMW>
>
> Opening the .main file and performing an *Ncdump Data* of variable:
> "Level-3_Binned_Data/Rrs_412(0:10:1).Rrs_412_sum"
>
> Returns:
>
> float Rrs_412_sum;
>
>  data:
>
>   {1.86057E-40, 9.403955E-38, 6.4099753E-10, 2.6076459E-9, 1.0297978E21, 
> 5.6431478E-11, 0.0, -2.9699963E36, 4.59183E-40, 3.67343E-40, 2.60329423E11}
>
>  Also, in ToolsUI, *all of the other data variables* (i.e. angstrom,
> aot_869 & Rrs_*) all display very very similar (most are identical) values
> as the Rrs_412. This is not the case for HDFView.
>
> Incidentally, reading the data via OceanColor's SeaDas
> <http://seadas.gsfc.nasa.gov/> application (which uses NetCDF Java under
> the hood) results in the same data as ToolsUI.
>  Wrap-up
>
> The evidence above appears to indicate that there is a bug in NetCDF Java
> related to linked HDF files which results in incorrect data reads from
> linked files.
>
> Does anyone have any idea:
>
> *a)* what could be causing the issue?
> *b)* how could it be addressed?
>
>
>
>  Thanks in advance,
> Chris
>
> _______________________________________________
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>

• Follow-Ups:
  • Re: [netcdf-java] Erroneous data from linked HDF files
    • From: Christopher Mueller

• References:
  • [netcdf-java] Erroneous data from linked HDF files
    • From: Christopher Mueller