NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi John, > It is possible that for various reasons, Java will > be "several times slower" than C code, so you'll have to decide if the > increase in productivity is worth it. In what circumstances would Java-NetCDF be "several times slower" than C? I've always found nj to be very close to our old C++ code in terms of speed, and nj4 appears to be faster than nj2. (In a server environment with a persistent, modern JVM - as you say, running a one-off Java program has a big overhead in bootstrapping the JVM and loading classes.) I'd just like to know if there are any particular things I should look out for that might badly affect performance. Cheers, Jon On Wed, May 6, 2009 at 11:21 PM, John Caron <caron@xxxxxxxxxxxxxxxx> wrote: > Hi Bill: > > I made a few mods to your program (attached) > > 1) removed the print statements, which are notoriously slow. > 2) did the whole open/read/close loop 100 times > 3) added timing, and got: > > that took 1248.659775 millisecs > > which is about 13 msecs per call. When I get a chance I will try to compare > to the C code. > > None of this is all that definitive, its very hard to get accurate timings > on small programs. For one thing, the java compiler happens at runtime, and > its somewhat indeterministic. so running a program once will very likely > look very bad. If you are doing a CGI type server, where the java > application starts up for each request, that will be very slow. > > I can pretty much promise you that java performance is within a factor of 2 > of C code, and more likely within 20% of C code, in a long-running server > environment. There are certain things it can do faster, like memory > allocation and multithreading. > > Anyway, I could look at your actual production code to see if there are some > ways to help speed it up. It is possible that for various reasons, Java will > be "several times slower" than C code, so you'll have to decide if the > increase in productivity is worth it. > > Bill Moninger wrote: >> >> Hi John, >> >> sorry for the delay in getting back to you. >> >> I've attached two programs that read a netcdf file of RUC output in a >> hybrid coordinate system, on a 40km grid. One's in C, and one's in java. >> >> A typical netcdf file that they read may be found at >> http://ruc.noaa.gov/ruc_native_40.nc (53 M in size). >> >> Generally, I read 6 variables from this file to generate soundings (SkewT >> plots), and my C program reads the file and generates the sounding in far >> less than a second. The java version takes several times longer, and since >> we have many hits on our web page that generates on-the-fly soundings, it >> would be a big increase in load on our server to switch to java-netCDF. >> >> I've stripped both the C and java programs down to the minimum. Each reads >> one vertical column of data from the 'vpt' variable and prints it out for >> each level. (The production program reads 6 variables from one vertical >> column and generates a soundings for that column.) >> >> On my web server, the java program takes 0.58 seconds, and the C program >> takes 0.01 seconds. >> >> It may be for my simple processing netcdf-java is just overkill. But if it >> can be sped up to approach the time of C, I'd like to use it because I can >> use the same code (with different config files) to read netCDF, grib, and >> grib2, and things would be much easier to manage. >> >> Any thoughts you have will be gratefully received. >> >> -Bill >> >> On 4/4/2009 12:43 PM, John Caron wrote: >>> >>> >>> Bill Moninger wrote: >>>> >>>> Hi Robb, >>>> >>>> thanks for the information. I'll take a look at regenerating the gbx >>>> files. >>>> >>>> For what its worth--the *biggest* percentage slowdown is not with grib >>>> or grib2 files, but with netCDF files, surprisingly enough. My c routine >>>> (using an earlier version of netCDF) reads the files almost instantly--the >>>> java-netcdf4 arrangement reads the file much more slowly. >>> >>> Thats interesting. When you say "read" do you mean read all the data, or >>> just opening the file? I assume netcdf 3 formatted files? >>> >>> Can you send a sample program that has this slowdown? Are you comparing >>> against a C program or earlier versions of java-netcdf? >>> >>> BTW, java 1.6 should be 20-30% faster than java 1.5, particularly if you >>> use the -server option. >>> >>>> >>>> -Bill >>>> >>>> On 3/31/2009 12:56 PM, Robb Kambic wrote: >>>>> >>>>> On Fri, 27 Mar 2009, Bill Moninger wrote: >>>>> >>>>>> Hello netcdf-java folks, >>>>>> >>>>>> Thanks to good help from the netcdf-java staff, I'm now able to read >>>>>> and generate soundings from RUC files in netCDF, grib, and grib2 format. >>>>>> Its >>>>>> really nice to be able to use the same code for all three formats. >>>>>> >>>>>> Unfortunately, I find that, at least as I've implemented it, >>>>>> netcdf-java is 20% to 50% slower than my previous methods (using C). >>>>>> >>>>> >>>>> Bill, >>>>> >>>>> If you use the the grib index file, those are the files with the gbx >>>>> suffix that are usually in the same dir as the grib file. You should >>>>> delete >>>>> them all and then regenerate them. The new index file read in much >>>>> quicker. >>>>> Currently, i working on grib performance issues >>>>> >>>>> Robb... >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Moreover, it appears that java 1.6 is slower than 1.5 (though I >>>>>> haven't recompiled the underlying UCAR code in 1.6--only my code). >>>>>> >>>>>> If folks have any thoughts about how to speed things up, I will be >>>>>> much obliged to hear them. >>>>>> >>>>>> -Bill >>>>>> -- >>>>>> William R. Moninger http://www-frd.fsl.noaa.gov/~moninger/ >>>>>> NOAA / Earth Systems Research Laboratory / Global Systems Division >>>>>> 325 Broadway, R/GSD1 voice: 303-497-6435 >>>>>> Boulder, CO 80305 fax: 303-497-3329 >>>>>> >>>>>> _______________________________________________ >>>>>> netcdf-java mailing list >>>>>> netcdf-java@xxxxxxxxxxxxxxxx >>>>>> For list information or to unsubscribe, visit: >>>>>> http://www.unidata.ucar.edu/mailing_lists/ >>>>> >>>>> >>>>> =============================================================================== >>>>> Robb Kambic Unidata Program Center >>>>> Software Engineer III Univ. Corp for Atmospheric Research >>>>> rkambic@xxxxxxxxxxxxxxxx WWW: http://www.unidata.ucar.edu/ >>>>> >>>>> =============================================================================== >>>> >>> >>> _______________________________________________ >>> netcdf-java mailing list >>> netcdf-java@xxxxxxxxxxxxxxxx >>> For list information or to unsubscribe, visit: >>> http://www.unidata.ucar.edu/mailing_lists/ >> > > > import ucar.nc2.iosp.grib.*; > import ucar.nc2.iosp.grib.GribServiceProvider.*; > > import java.*; > import java.io.*; > import java.util.*; > > import ucar.ma2.*; > import ucar.nc2.*; > import ucar.nc2.util.*; > import ucar.nc2.units.DateFormatter; > > public class Tester { > private static final String VERSION = "0.01"; > > double[] Tvar; > > public static void main(String[] args) { > Tester gs = new Tester(args); > System.exit(0); > } > > public Tester(String[] args) { > String filename = "D:\\work\\moninger\\ruc_native_40.nc"; > > long start = System.nanoTime(); > int tuv_levels = 0; > > for (int count = 0; count < 100; count++) { > NetcdfFile ncfile = null; > try { > > ncfile = NetcdfFile.open(filename); > //System.out.println("ncfile is "+ncfile); > Array data4D; > Variable v = null; > Attribute a = null; > > // get grid parameters for most variables > Dimension d = ncfile.findDimension("z"); > if (d == null) { > System.out.println("Bad dimension for z"); > System.exit(1); > } > tuv_levels = d.getLength(); > > // get variables > int[] origin = new int[]{0, 0, 40, 50}; > int[] tuv_size = new int[]{1, tuv_levels, 1, 1}; > v = ncfile.findVariable("vpt"); > data4D = v.read(origin, tuv_size); > Tvar = (double[]) > data4D.reduce().get1DJavaArray(double.class); > //System.out.println("successfully read " + filename); > > } catch (Exception e) { > System.out.println("Exception: " + filename + " " + e); > e.printStackTrace(); > System.exit(1); > > } finally { > if (null != ncfile) try { > ncfile.close(); > //System.out.println("closed file"); > } catch (IOException ioe) { > System.out.println("trying to close " + filename + " " + > ioe); > } > } > > /* for (int i = 0; i < tuv_levels; i++) { > System.out.println("i: " + i + " t " + Tvar[i]); > } */ > > > } > > long stop = System.nanoTime(); > System.out.printf("that took %f millisecs %n", (stop - start) / > 1000.0 / 1000.0); > > > } > } > > > > _______________________________________________ > netcdf-java mailing list > netcdf-java@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > -- Dr Jon Blower Technical Director, Reading e-Science Centre Environmental Systems Science Centre University of Reading Harry Pitt Building, 3 Earley Gate Reading RG6 6AL. UK Tel: +44 (0)118 378 5213 Fax: +44 (0)118 378 6413 j.d.blower@xxxxxxxxxxxxx http://www.nerc-essc.ac.uk/People/Staff/Blower_J.htm
netcdf-java archives: