NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Christian, I'm still trying to sort out the issue of parsing .grib2 files. I need to be able to open .grib2 files without using the netcdfAll bundle. I'm taking the approach of gathering all the CDM Dependancies <http://www.unidata.ucar.edu/software/thredds/v4.5/netcdf-java/reference/JarDependencies.html#logging> from Maven Central, however, I'm still getting errors. Have you successfully opened a .grib2 files by providing all necessary .jar files at your shell command? If so, could you pass along an example? Many thanks, Annie On Wed, Jul 30, 2014 at 10:00 AM, <netcdf-java-request@xxxxxxxxxxxxxxxx> wrote: > Send netcdf-java mailing list submissions to > netcdf-java@xxxxxxxxxxxxxxxx > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.unidata.ucar.edu/mailman/listinfo/netcdf-java > or, via email, send a message with subject or body 'help' to > netcdf-java-request@xxxxxxxxxxxxxxxx > > You can reach the person managing the list at > netcdf-java-owner@xxxxxxxxxxxxxxxx > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of netcdf-java digest..." > > > Today's Topics: > > 1. Re: grib2 (Dennis Heimbigner) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 29 Jul 2014 15:04:15 -0600 > From: Dennis Heimbigner <dmh@xxxxxxxx> > To: Christian Ward-Garrison <cwardgar@xxxxxxxx> > Cc: Annie Bryant <anniebryant.burgess@xxxxxxxxx>, > netcdf-java@xxxxxxxxxxxxxxxx > Subject: Re: [netcdf-java] grib2 > Message-ID: <53D80C4F.3010007@xxxxxxxx> > Content-Type: text/plain; charset=UTF-8; format=flowed > > There may be a way to fix this. > You might 'unjar' tika-app-1.6-SNAPSHOT.jar > and replace the old netcdf in that jar with the > newer version, then rejar tika-app-1.6-SNAPSHOT.jar. > Assuming no interface problems, this has a fair chance > to work. > =Dennis Heimbigner > Unidata > > Christian Ward-Garrison wrote: > > Hi Annie, > > > > It turns out that this is an issue with Tika. tika-app-1.6-SNAPSHOT.jar > > actually includes netcdf-4.2.20, so you have 2 different versions of the > > same library on the classpath, which is always bad. And unfortunately, > > using Tika's bundled netcdf-4.2.20 alone won't work, because support for > > reading GRIB files was only added to NetCDF-Java in 4.3+. > > > > This may be a problem that only the Tika developers know how to fix. I > > suggest opening a ticket with them. > > > > Good luck! > > Christian Ward-Garrison > > > > > > On Mon, Jul 28, 2014 at 3:19 PM, Annie Burgess <anniebryant@xxxxxxxxx> > > wrote: > > > >> Hi Christian, > >> > >> Your code words great - no problems when I compile and run. I've > modified > >> my code to use NetcdfDataset rather than NetcdfFile to open the .grib2 > >> file. However, I'm still getting an error in my code: > >> > >> % javac -classpath > >> > ../../../../tika-core/target/tika-core-1.6-SNAPSHOT.jar:../../../../netcdfAll-4.3.jar > >> org/apache/tika/parser/grib/GribParser.java > >> > >> % java -classpath > >> > tika-app/target/tika-app-1.6-SNAPSHOT.jar:annie-parsers.jar:netcdfAll-4.3.jar > >> org.apache.tika.cli.TikaCLI --text gdas1.forecmwf.2014062612.grib2 > >> > >> > >> -----------------------GripParser.java------------------------ > >> import java.io.ByteArrayOutputStream; > >> import java.io.IOException; > >> import java.io.InputStream; > >> import java.util.Collections; > >> import java.util.Set; > >> import java.io.File; > >> > >> import org.apache.tika.exception.TikaException; > >> > >> import org.apache.tika.mime.MediaType; > >> import org.apache.tika.parser.AbstractParser; > >> import org.apache.tika.parser.ParseContext; > >> import org.apache.tika.parser.Parser; > >> import org.apache.tika.sax.XHTMLContentHandler; > >> import org.xml.sax.ContentHandler; > >> import org.xml.sax.SAXException; > >> > >> import ucar.nc2.NetcdfFile; > >> import ucar.nc2.dataset.NetcdfDataset;/** > >> > >> > >> public class GribParser extends AbstractParser { > >> > >> private final Set<MediaType> SUPPORTED_TYPES = > >> Collections.singleton(MediaType.application("x-grib2")); > >> > >> public Set<MediaType> getSupportedTypes(ParseContext context) { > >> return SUPPORTED_TYPES; > >> } > >> > >> public void parse(InputStream stream, ContentHandler handler, > >> Metadata metadata, ParseContext context) throws IOException, > >> SAXException, TikaException { > >> > >> System.err.println(" Check 1 "); > >> > >> File gribFile = new File("gdas1.forecmwf.2014062612.grib2"); > >> > >> NetcdfFile ncFile = > NetcdfDataset.openFile(gribFile.getAbsolutePath(), > >> null); > >> > >> System.err.println(" Check 2 "); > >> > >> } > >> } > >> > >> ---------------------OUTPUT--------------------- > >> > >> Check 1 > >> Exception in thread "main" org.apache.tika.exception.TikaException: > >> TIKA-198: Illegal IOException from > >> org.apache.tika.parser.grib.GribParser@483ad415 > >> at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:249) > >> > >> at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243) > >> at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:121) > >> at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:141) > >> at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:420) > >> at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:111) > >> Caused by: java.io.IOException: Cant read > >> /Users/IGSWAHWSWBURGESS/Development/tikadev/tika/gdas1.forecmwf. > 2014062612.grib2: > >> not a valid CDM file. > >> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:734) > >> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:384) > >> at > ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:687) > >> at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:564) > >> at org.apache.tika.parser.grib.GribParser.parse(GribParser.java:82) > >> > >> at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243) > >> ... 5 more > >> > >> --------------------------------------------- > >> > >> netcdfALL and .grib2 files are in the same director where I am running > >> GribParser.java. This could totally be a classpath issue... I'm just > >> stumped. > >> > >> Any other ideas? > >> > >> Thanks! > >> Annie > >> > >> > >> On Mon, Jul 28, 2014 at 12:40 PM, Christian Ward-Garrison < > >> cwardgar@xxxxxxxx> wrote: > >> > >>> Hi Annie, > >>> > >>> I see you're using netcdfAll-4.3.jar. That actually already contains > the > >>> grib module, so it should be all you need. I created a minimal example: > >>> > >>> ------------------------------------- Foo.java > >>> ------------------------------------- > >>> > >>> import java.io.IOException; > >>> import java.io.File; > >>> import ucar.nc2.NetcdfFile; > >>> import ucar.nc2.dataset.NetcdfDataset; > >>> > >>> public class Foo { > >>> public static void main(String[] args) throws IOException { > >>> File gribFile = new File("foo.grib2"); > >>> NetcdfFile ncFile = > >>> NetcdfDataset.openFile(gribFile.getAbsolutePath(), null); > >>> try { > >>> System.out.println(ncFile.toString()); > >>> } finally { > >>> ncFile.close(); > >>> } > >>> } > >>> } > >>> > >>> ------------------------------------- Shell commands > >>> ------------------------------------- > >>> > >>> javac -cp netcdfAll-4.3.jar Foo.java > >>> > >>> java -cp .;netcdfAll-4.3.jar Foo > >>> > >>> > >>> That should work as long as netcdfAll-4.3.jar and a file named > >>> "foo.grib2" are in the same directory as Foo.java. If you move things, > >>> you'll obviously need to modify the commands. Does this example work > for > >>> you? > >>> > >>> Cheers, > >>> Christian > >>> > >>> > >>> > >>> On Mon, Jul 28, 2014 at 1:52 PM, Annie Burgess <anniebryant@xxxxxxxxx> > >>> wrote: > >>> > >>>> Hi Christian, > >>>> > >>>> Thanks for your response. I've cut down the code (pasted below) to a > >>>> sort of bare-bones version that is ONLY trying to open the .grib2 > file as > >>>> if it were a .nc file. > >>>> > >>>> I build apache tika from: > >>>> > >>>> svn co http://svn.apache.org/repos/asf/tika/trunk tika > >>>> mvn install > >>>> > >>>> I pulled netcdfAll and toolsUI .jar files from: > >>>> http://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp > >>>> > >>>> I pulled the grib .jar from: > >>>> http://mvnrepository.com/artifact/edu.ucar/grib/8.0.29 > >>>> > >>>> I compile the code as: > >>>> [asc-227-196:src/main/java] AB% javac -classpath > >>>> > ../../../../tika-core/target/tika-core-1.6-SNAPSHOT.jar:../../../../toolsUI-4.3.jar:../../../../netcdfAll-4.3.jar:../../../../grib-8.0.29.jar > >>>> org/apache/tika/parser/grib/GribParser.java > >>>> > >>>> I run the code as: > >>>> [asc-227-196:~/Development/tikadev/tika] AB% java -classpath > >>>> > tika-app/target/tika-app-1.6-SNAPSHOT.jar:annie-parsers.jar:netcdfAll-4.3.jar:grib-8.0.29.jar:toolsUI-4.3.jar > >>>> org.apache.tika.cli.TikaCLI --text gdas1.forecmwf.2014062612.grib2 > >>>> > >>>> CODE: > >>>> > >>>> -------------------------------------------------- > >>>> package org.apache.tika.parser.grib; > >>>> > >>>> import java.io.ByteArrayOutputStream; > >>>> import java.io.IOException; > >>>> import java.io.InputStream; > >>>> import java.util.Collections; > >>>> import java.util.Set; > >>>> import java.util.List; > >>>> import java.util.Iterator; > >>>> > >>>> //JDK imports > >>>> import org.apache.tika.exception.TikaException; > >>>> import org.apache.tika.io.IOUtils; > >>>> import org.apache.tika.metadata.Metadata; > >>>> import org.apache.tika.metadata.Property; > >>>> import org.apache.tika.metadata.TikaCoreProperties; > >>>> import org.apache.tika.mime.MediaType; > >>>> import org.apache.tika.parser.AbstractParser; > >>>> import org.apache.tika.parser.ParseContext; > >>>> import org.apache.tika.parser.Parser; > >>>> import org.apache.tika.sax.XHTMLContentHandler; > >>>> import org.xml.sax.ContentHandler; > >>>> import org.xml.sax.SAXException; > >>>> > >>>> import ucar.grib.grib2.*; > >>>> import ucar.nc2.*; > >>>> > >>>> /** > >>>> * A {@link Parser} for <a > >>>> * href="http://www.unidata.ucar.edu/software/netcdf/index.html > >>>> ">NetCDF</a> > >>>> * files using the UCAR, MIT-licensed <a > >>>> * href="http://www.unidata.ucar.edu/software/netcdf-java/";>NetCDF > for > >>>> Java</a> > >>>> * API. > >>>> */ > >>>> public class GribParser extends AbstractParser { > >>>> > >>>> private final Set<MediaType> SUPPORTED_TYPES = > >>>> Collections.singleton(MediaType.application("x-grib2")); > >>>> /* > >>>> * (non-Javadoc) > >>>> * > >>>> * @see > >>>> * > >>>> org.apache.tika.parser.Parser#getSupportedTypes(org.apache.tika.parser > >>>> * .ParseContext) > >>>> */ > >>>> public Set<MediaType> getSupportedTypes(ParseContext context) { > >>>> return SUPPORTED_TYPES; > >>>> } > >>>> /* > >>>> * (non-Javadoc) > >>>> * > >>>> * @see org.apache.tika.parser.Parser#parse(java.io.InputStream, > >>>> * org.xml.sax.ContentHandler, org.apache.tika.metadata.Metadata, > >>>> * org.apache.tika.parser.ParseContext) > >>>> */ > >>>> public void parse(InputStream stream, ContentHandler handler, > >>>> Metadata metadata, ParseContext context) throws > IOException, > >>>> SAXException, TikaException { > >>>> > >>>> System.err.println(" Check 1 "); > >>>> String name = "/Users/IGSWAHWSWBURGESS/POLARCYBER/gdas1.forecmwf. > >>>> 2014062612.grib2"; > >>>> > >>>> if (name == null) { > >>>> name = ""; > >>>> } > >>>> > >>>> NetcdfFile ncFile = NetcdfFile.open(name, null); > >>>> System.err.println(" Check 2 "); > >>>> } > >>>> } > >>>> > >>>> > >>>> OUTPUT: > >>>> > >>>> Check 1 > >>>> > >>>> Exception in thread "main" org.apache.tika.exception.TikaException: > >>>> Unexpected RuntimeException from > >>>> org.apache.tika.parser.grib.GribParser@261a53b9 > >>>> at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:245) > >>>> at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243) > >>>> at > >>>> > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:121) > >>>> at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:141) > >>>> at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:420) > >>>> at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:111) > >>>> Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: > >>>> > ucar.grib.grib2.Grib2WriteIndex.writeGribIndex(Ljava/io/File;Ljava/lang/String;Lucar/unidata/io/RandomAccessFile;Z)Lucar/grid/GridIndex; > >>>> at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1326) > >>>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:744) > >>>> at ucar.nc2.NetcdfFile.openInMemory(NetcdfFile.java:670) > >>>> at org.apache.tika.parser.grib.GribParser.parse(GribParser.java:93) > >>>> at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243) > >>>> ... 5 more > >>>> Caused by: java.lang.NoSuchMethodError: > >>>> > ucar.grib.grib2.Grib2WriteIndex.writeGribIndex(Ljava/io/File;Ljava/lang/String;Lucar/unidata/io/RandomAccessFile;Z)Lucar/grid/GridIndex; > >>>> at > >>>> > ucar.nc2.iosp.grib.GribGridServiceProvider.writeIndex(GribGridServiceProvider.java:348) > >>>> at > >>>> > ucar.nc2.iosp.grib.GribGridServiceProvider.getIndex(GribGridServiceProvider.java:292) > >>>> at > >>>> > ucar.nc2.iosp.grib.GribGridServiceProvider.open(GribGridServiceProvider.java:118) > >>>> at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1308) > >>>> ... 9 more > >>>> > >>>> > >>>> > >>>> Note, if I use a .nc file the code runs successfully. > >>>> > >>>> OUTPUT: > >>>> > >>>> Check 1 > >>>> Check 2 > >>>> > >>>> > >>>> I am sort of a java newbie, so please let me know if I've left out any > >>>> critical information! > >>>> > >>>> Thank you for any help/insight you can give. > >>>> > >>>> Annie > >>>> > >>>> > >>>> On Sun, Jul 27, 2014 at 10:44 PM, Christian Ward-Garrison < > >>>> cwardgar@xxxxxxxx> wrote: > >>>> > >>>>> Hi Annie, > >>>>> > >>>>> This is the result of the GRIB module not being on the classpath when > >>>>> you execute your Java program. Can you give me more details about > your > >>>>> setup? Can you provide you build file (Maven, Ant, Gradle, etc)? > >>>>> > >>>>> Cheers, > >>>>> Christian > >>>>> > >>>>> > >>>>> On Wed, Jul 23, 2014 at 5:06 PM, Annie Burgess < > anniebryant@xxxxxxxxx> > >>>>> wrote: > >>>>> > >>>>>> Greetings all, > >>>>>> > >>>>>> I am trying to create a script that will mimic the output of NCDump. > >>>>>> I have successfully done this for NetCDF files, and now I am > trying to > >>>>>> apply it to grib2 files. I am using the NetCDF-java library in > conjunction > >>>>>> with Apache Tika to do this. Other posts have indicated I should > be able > >>>>>> to open my grib2 files, just as if they were .nc files. However, I > >>>>>> continue to get the following error: > >>>>>> > >>>>>> "Caused by: java.io.IOException: Cant read gdas1.forecmwf. > 2014062612.grib2: > >>>>>> not a valid CDM file." > >>>>>> > >>>>>> To open the .nc files, this is the bit of code I use (with the > >>>>>> exception of changing the .nc file to a .grib2 file): > >>>>>> > >>>>>> String name = "gdas1.forecmwf.2014062612.grib2"; > >>>>>> > >>>>>> if (name == null) { > >>>>>> name = ""; > >>>>>> } > >>>>>> > >>>>>> try NetcdfFile ncFile = NetcdfFile.openInMemory(name, > >>>>>> os.toByteArray()); > >>>>>> // first parse out the set of global attributes > >>>>>> for (Attribute attr : ncFile.getGlobalAttributes()) { > >>>>>> Property property = > resolveMetadataKey(attr.getName()); > >>>>>> if (attr.getDataType().isString()) { > >>>>>> metadata.add(property, attr.getStringValue()); > >>>>>> } else if (attr.getDataType().isNumeric()) { > >>>>>> int value = attr.getNumericValue().intValue(); > >>>>>> metadata.add(property, String.valueOf(value)); > >>>>>> } > >>>>>> } > >>>>>> > >>>>>> Also, I am using the netcdfAll-4.3.jar at the command line. Does > >>>>>> anyone have any insight as to *why *I'd be getting the 'not a valid > >>>>>> CDM' error. I have checked the file using the NetCDF (4.3) GUI and > the > >>>>>> file looks good. > >>>>>> > >>>>>> Thank you for any insight you can give. > >>>>>> > >>>>>> Annie > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> netcdf-java mailing list > >>>>>> netcdf-java@xxxxxxxxxxxxxxxx > >>>>>> For list information or to unsubscribe, visit: > >>>>>> http://www.unidata.ucar.edu/mailing_lists/ > >>>>>> > >>>>> > >> > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > netcdf-java mailing list > > netcdf-java@xxxxxxxxxxxxxxxx > > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > > > > ------------------------------ > > _______________________________________________ > netcdf-java mailing list > netcdf-java@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > > End of netcdf-java Digest, Vol 66, Issue 12 > ******************************************* > -- ------------------------------------------------------------------------------------------ Ann Bryant Burgess, PhD Postdoctoral Fellow Computer Science Department University of Southern California Viterbi School of Engineering Los Angeles, CA Alaska Science Center/USGS Anchorage, AK Cell: (585) 738-7549 Office: (907) 786-7059 Fax: (907) 786-7150 E-mail: anniebryant.burgess@xxxxxxxxx Office Address: 4210 University Dr., Anchorage, AK 99508-4626 -------------------------------------------------------------------------------------------
netcdf-java archives: