Re: Problems reading NetCDF data - get NaNs

• To: Jon Blower <jdb@xxxxxxxxxxxxxxxxxxxx>
• Subject: Re: Problems reading NetCDF data - get NaNs
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Thu, 19 Oct 2006 17:34:41 -0600

Jon Blower wrote:

Hi John,

we no longer allow uploading (*&!%# security). Can you put it on an HTTP or FTP server on your end?

Unfortunately I'm working at home so my only link to my FTP server is
through a slow (encrypted) VPN connection - it's an 80MB file and
would take about an hour to upload.  I'll start uploading it and email
you when it's there (might be a little while, I'll wait till I knock
off for the day before starting the upload or it kills my network
connection).

We are working on new docs for our upcoming workshop. what are the gist of your questions?

OK, here goes - As you might know, a Web Map Service is a service that
allows clients to request maps, which might be in a number of
coordinate reference systems (CRSs), bounding boxes and sizes.  I want
to write a WMS (which I'm calling ncWMS) that produces map images from
CF-compliant NetCDF files.  These files will themselves have data in a
number of CRSs (lat-lon, rotated-pole, equal-area etc).

I know that nj22 has support for various CRSs and a pluggable
architecture for others.  The issue with ncWMS is that the images must
be produced quickly, in order to support interactive websites like the
one I sent you the link to before.  The problems with many existing
WMS implementations is either poor NetCDF support, or they are very
slow at producing images.

I'm trying to think of a way to produce images efficiently for general
CRSs (it's easy for some simple cases).  The problem seems to hinge on
being able to extract from disk:
1) as little data as possible (data extraction is often the slowest
step).  I.e. I don't want to extract a 1000x1000 area of data for a
100x100 picture.
2) contiguous chunks of data where possible

A key part of the software is being able to map between points in the
CRS of the *request* to equivalent (nearest) points in the CRS of the
*data object*.  To do this I translate all the points in the request
CRS into lat-lon coordinates (which is sometimes trivial), then find
the corresponding points in the data, i.e. there's an intermediate
step.

That's all background.  I have some specific questions:

1)  Can I translate directly from one CRS to another without going
through an intermediate lat-lon point?  I guess I'm asking if
Coordinate Transforms can be combined (matrix multiplication?)

No, these projections are highly nonlinear, and so dont compose.

2) If I have a dataset in a "funny" CRS such as rotated-pole and I
want to find the data point at a given lat-lon point, how do I do it?
I'm aware of the method GridCoordSys.findXYCoordElement() but this
takes coordinate points as its input, not lat-lon.  I guess I need
another method before this to translate from lat-lon to coordinate
space?

yes, you can use projection.latLonToProj(LatLonPoint latlon), then use 
GridCoordSys.findXYCoordElement()
but that will be very slow if you do it at every point!

the problem is you want to reproject from the CRS of the data to some other 
CRS. I dont know how to do that most efficiently. Dont they also give you the 
resolution and the bounds of the image that they want? If so, I might treat it 
as an image drawing problem. I used to have some code that created gridded 
images for any data in an arbitrary CRS. It was good enough to compute the 4 
corners of each grid cell in the new CRS, and drawing them to a BufferedImage. 
Better would be to resample/interpolate each pixel. there may be some way to 
pregenerate some mapping polynomials for all the possible transformations that 
you need.

sorry i dont know, you could try asking some graphics groups, im sure there are 
some known solutions out there.

3) I notice some changes between 2.2.16 and 2.2.17.  Would you
recommend that I use 2.2.17 now?

yes, its almost ready to be stable version 2.2.18, and has lots of bug fixes.

4) Just a point rather than a question: the
CoordinateAxis1D.findCoordElement() method is inefficient - in the
case of regular axes the method could be very quick and even for
irregular axes it could be much faster with a binary search.  I can
implement this and submit the changes to you if you would like - I
will have to make the changes anyway because at the moment the
findCoordElement() method is a rate-limiting step in my code! ;-)

sure, that'd be great.

have you looked at findCoordElement(double pos, int lastIndex) ?

That's probably it for now (I probably have more)!  I could implement
these things in an ad hoc way, but I'd much rather build on the nj22
libs as much as possible - that way my app can stay simple.  Thanks
very much for all your work on nj22, I think it's excellent.

thanks

• References:
  • Problems reading NetCDF data - get NaNs
    • From: Jon Blower