Re: [netcdf-java] Reading sequences from BUFR with NetcdfDataset

• To: netty.jawn@xxxxxx
• Subject: Re: [netcdf-java] Reading sequences from BUFR with NetcdfDataset
• From: John Caron <jcaron1129@xxxxxxxxx>
• Date: Mon, 2 Sep 2019 19:55:09 -0600

Hi netty:

Sorry for the crappy documents and confusion. Sequences are strange, they
dont behave like normal Arrays. As sort-of described here:

https://www.unidata.ucar.edu/software/netcdf-java/current/reference/Cookbook.html#readSequences

you cant call read(), you can only call getStructureIterator() and iterate,
because sequences dont have a fixed size. You have nested sequences: a BUFR
file consists of a variable number of BUFR messages (top level sequence),
and each BUFR message has a variable number of
Long_time_period_or_displacement fields (nested sequence). Heres some code
that can get you started:

public void testReadSequenceEnhanced() throws IOException {
  try (NetcdfDataset ncd =  NetcdfDataset.openDataset("yourFilename")) {
    SequenceDS record = (SequenceDS) ncd.findVariable("obs");

    try (StructureDataIterator iter = record.getStructureIterator()) {
      int recordCount = 0;
      while (iter.hasNext()) {
        StructureData sdata = iter.next();
        Assert.assertNotNull(sdata);
        recordCount++;

        ArraySequence nestedSequence =
sdata.getArraySequence("Long_time_period_or_displacement");
        try (StructureDataIterator nestedIter =
nestedSequence.getStructureDataIterator()) {
          int nestedCount = 0;
          while (nestedIter.hasNext()) {
            StructureData nestedData = nestedIter.next();
            Assert.assertNotNull(nestedData);
            NCdumpW.printStructureData(new PrintWriter(System.out), nestedData);
            nestedCount++;
          }
          Assert.assertEquals(1685, nestedCount);
        }
      }
      Assert.assertEquals(1, recordCount);
    }
  }
}


Theres also this doc that may be useful:

https://www.unidata.ucar.edu/software/netcdf-java/current/reference/StructureData.html

Good luck!


On Mon, Sep 2, 2019 at 8:18 AM <netty.jawn@xxxxxx> wrote:

> Hi,
>
> I have to read the attached BUFR-File with netcdf-java and it works a
> little bit.
>
> I opened my BUFR-File with ToolsUI to examine the content. Here was the
> first surprise: The icon
> https://docs.unidata.ucar.edu/netcdf-java/5.0/userguide/images/netcdf-java/tutorial/cdmdatasets/enhanceButton.jpg
> shows "add Coordinates..." as tooltip. But it is actually the
> enable/disable of the enhance-mode, as stated here
> https://docs.unidata.ucar.edu/netcdf-java/5.0/userguide/netcdf_dataset.html
> The tooltip should be changed.
>
> Reading variables was easy thanks to the enhance-mode:
> try (NetcdfDataset ncFile = NetcdfDataset.openDataset(filename);) {
>   VariableDS variable = (VariableDS)
> ncFile.findVariable("obs.Software_identification_and_version_number");
>   System.out.println(variable.readScalarString());
>   }
> Float and all other types are correctly enhanced.
>
> Naively I thought reading sequences is easy too. I have to read the
> sequence "obs.Long_time_period_or_displacement" (as an example). Aunt
> Google showed me the function "showNestedSequence" from
> https://www.unidata.ucar.edu/software/netcdf-java/current/reference/Cookbook.html
> (can't find it in the current documentation). This function shows me all
> values, but not enhanced. Furthermore 2 nested loops were used. I would
> prefer ONE loop (if possible).
>
> After (I think) thousands of failed attempts, I came to this
> try (NetcdfDataset ncFile = NetcdfDataset.openDataset(filename);) {
>   VariableDS variable = (VariableDS)
> ncFile.findVariable("obs.Long_time_period_or_displacement");
>   System.out.println("Type: " + variable.getDataType());
>   Array value = variable.read();
>   System.out.println("THIS LINE IS NOT PRINTED");
>   }
> The following exception is thrown:
> java.lang.ClassCastException: class ucar.ma2.ArrayObject$D1 cannot be cast
> to class ucar.ma2.ArrayStructure (ucar.ma2.ArrayObject$D1 and
> ucar.ma2.ArrayStructure are in unnamed module of loader
> org.apache.catalina.loader.ParallelWebappClassLoader @ebbfb89)
>         at ucar.nc2.dataset.StructureDS.convert(StructureDS.java:268)
>         at ucar.nc2.dataset.SequenceDS.read(SequenceDS.java:84)
>         at de.sample.BUFRTest.showNestedSequenceNew(BUFRTest.java:88)
> Line 88 is the "variable.read();".
> I called your netcdf-lib from a servlet, therefore the
> "org.apache.catalina", but this should not be the problem.
>
> So it seems this sentence is not correct:
> https://docs.unidata.ucar.edu/netcdf-java/5.0/userguide/netcdf_dataset.html
> "Note that NetcdfDataset is a subclass of NetcdfFile, and so can be used
> wherever a NetcdfFile is used."
>
> Someone suggested to have a look at NCdumpW, which I also did, but this
> program does not enhance any values.
>
> Since I don't want to open my file as NetcdfDataset and again as
> NetcdfFile, I would try to open it as NetcdfFile and manually enhance the
> values. But no idea how to do this. Any good solution for my problem is
> also appreciated.
>
> Thanks
>
> Kind regards
>
> Netty_______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>

• Follow-Ups:
  • Re: [netcdf-java] Reading sequences from BUFR with NetcdfDataset
    • From: netty . jawn

• References:
  • [netcdf-java] Reading sequences from BUFR with NetcdfDataset
    • From: netty . jawn