NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Ben, Thanks very much indeed for this. When I foretold the overflow in my comments I never seriously expected it to happen! Since then I've come across a few really big datasets - out of interest, what are your datasets? High-res remote sensing? I'll incorporate your code into ncWMS, and it will find its way into THREDDS at the next sync point (not sure when this will be - up to Unidata largely). Many thanks also for providing a test case! Cheers, Jon -----Original Message----- From: Ben Caradoc-Davies [mailto:Ben.Caradoc-Davies@xxxxxxxx] Sent: 05 September 2011 10:13 To: Jon Blower Cc: thredds mailing list Subject: Thredds WMS support for large source grids Jon, I tested WMS in thredds 4.2.6 with large NetCDF source grids and encountered an integer overflow in ncwms PixelMap. (You foretold this in the comments!) The attached patch fixes this defect at the cost of a small increase in memory use. You might remember writing (in PixelMap): // Calculate a single integer representing this grid point in the source grid // TODO: watch out for overflows (would only happen with a very large grid!) int sourceGridIndex = j * this.sourceGridISize + i; The integer overflow appears when the source grid has more than 2**31-1 points. For example, this limit is exceeded with a 26 GB NetCDF file with a single ubyte variable on a 92255x301081 grid. The attached patch includes Xiangtan Lin's CdmUtils fix to force DataReadingStrategy.SCANLINE for HDF5: http://mailman.unidata.ucar.edu/mailing_lists/archives/thredds/2011/msg00312.html The PixelMap change replaces the single integer array representing source and target grid offsets integers packed into a single long with two long arrays, one for source and one for target. This costs extra memory but may, in addition to supporting large grids, improve performance by avoiding packing an unpacking. It also includes: - a minor CdmUtils static initialiser change to appease ecj (the Eclipse compiler) - access changes in HorizontalCoordSys to support unit testing - a fix for axis sizes needed when LatLonCoordSys is explicitly instantiated in the unit test (otherwise they can never be set) - a unit test in which only the small() test method passes before the patch is applied (to ensure existing behaviour is preserved for small grids); all test methods ensure the expected source grid offset monotonicity The patch is against the ncwms-src.jar distributed with thredds 4.2.6 (I'm guessing the ncwms tds4.2-20101102 branch). With this patch applied and the replacement ncwms.jar installed in WEB-INF/lib, thredds 4.2.6 can serve a test 647 GB NetCDF4/HDF5 file via WMS: http://siss2.anu.edu.au/thredds/godiva2/godiva2.html?server=http://siss2.anu.edu.au/thredds/wms/ga/test/PRISM_UTM55_wgs84.nc The test file has a single ubyte variable on a 461276x1505407 grid. Performance is better than I expected; the aligned source and target grids plus the nearest-point mapping from target to source seem to do the trick. Kind regards, -- Ben Caradoc-Davies <Ben.Caradoc-Davies@xxxxxxxx> Software Engineering Team Leader CSIRO Earth Science and Resource Engineering Australian Resources Research Centre
thredds archives: