NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTE: The decoders
mailing list is no longer active. The list archives are made available for historical reasons.
============================================================================== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research rkambic@xxxxxxxxxxxxxxxx WWW: http://www.unidata.ucar.edu/ ============================================================================== ---------- Forwarded message ---------- Date: Fri, 26 Sep 2003 21:25:00 -0600 (MDT) From: ncdigest <owner-ncdigest@xxxxxxxxxxxxxxxx> To: ncdigest@xxxxxxxxxxxxxxxx Subject: ncdigest V1 #724 ncdigest Friday, September 26 2003 Volume 01 : Number 724 Today's Topics: Re: perl interface problem, plus ncdump oddity ---------------------------------------------------------------------- Date: Fri, 26 Sep 2003 15:54:55 -0400 From: Jason Thaxter <thaxter@xxxxxxxxxx> Subject: Re: perl interface problem, plus ncdump oddity - --SLDf9lqlvOQaIe6s Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Sep 16, 2003 at 11:20:13AM -0400, Jason Thaxter wrote:
I've found two odd problems with NetCDF tools recently. I suspect they are related to compiler and perl versions, but I don't really know. 1) ---------------------------------------------------- The Perl interface issue involves values retrieved by NetCDF::recget. The variable is a scalar of type BYTE. Perl always thinks it has a zero (0), even
I figured out this particular problem; the perl-netcdf library does not appear to be misbehaving at all, though it does things that might trap an unwary - and at least in this case, fairly wary - perl programmer. Variables of the NetCDF::BYTE type cannot be immediately used as perl variables. They will end up looking like "^A" or "^@" or something else when you try to use them as strings; worse, if you look at them as numbers, they will always appear to be zero. Suppose that $value is an actual single value retrieved from attget/varget/recget. To use a NetCDF::BYTE properly, do this $value = unpack('C',$value); I put this trick into a wrapper module I've written around NetCDF, called GoMOOS::NetCDF. There are a number of other tricks there, too - initializing arrays before passing them to NetCDF functions, retrieving variables by names instead of index number, etc. I wouldn't say it's complete or flawless; it's a little inconsistent about return values, and it's probably more useful for taking a record-oriented view than a variable-oriented view, and it's totally useless for writing NetCDF files. Nevertheless, anyone using perl-netcdf may find it useful and so I've attached it to this message. If the attachment doesn't make it through majordomo onto the list, feel free to e-mail me for a copy. Thanks, Jason - -- - ---------------------------------------------- Jason Thaxter GoMOOS, P.O. Box 4919, Portland, ME 04112-4919 Office Location: 1 Canal Plaza, 7th Floor Office: 207.773.0423 Fax: 207.773.8672 Email: thaxter@xxxxxxxxxx - ------------www.gomoos.org-------------------- - --SLDf9lqlvOQaIe6s Content-Type: application/x-perl Content-Disposition: attachment; filename="NetCDF.pm" Content-Transfer-Encoding: quoted-printable package GoMOOS::NetCDF;=0A=0A=3Dhead1 NAME=0A=0AGoMOOS::NetCDF - a library of routines for handling NetCDF files=0A=0A=3Dhead1 SYNOPSIS=0A=0AWraps bas ic NetCDF functions for use raw by scripts, or by other classes=0A(e.g. GoM OOS::Buoys::NetCDF). Note that many of the functions in this library=0Amay have a tendency to die suddenly. We try to check them, and we mark them =0Aif possible.=0A=0ANote that this is primarily a record-oriented library. =0A=0A=3Dcut=0Amy $version =3D '$Revision: 1.29 $';=0A=0Ause strict;=0Ause Carp 'cluck';=0A=0Ause NetCDF;=0A=0Amy @types;=0A$types[NetCDF::FLOAT] =3D 'float';=0A$types[NetCDF::DOUBLE] =3D 'double';=0A$types[NetCDF::LONG] =3D 'long';=0A$types[NetCDF::SHORT] =3D 'short';=0A$types[NetCDF::BYTE] =3D 'by te';=0A$types[NetCDF::CHAR] =3D 'char';=0Amy $ncGLOBAL =3D NetCDF::GLOBAL; =0A=0Amy $_MODE_RO =3D NetCDF::NOWRITE;=0A=0Asub new {=0A my $proto =3D shi ft;=0A my $class =3D ref($proto) || $proto;=0A my $self =3D {};=0A=0A if ( my $file =3D shift){=0A $self->{_FILE} =3D $file;=0A }=0A=0A bless($self,$ class);=0A return $self;=0A}=0A=0Asub open {=0A my ($self,$file) =3D @_;=0A =0A # may pass argument to open=0A if ($file){ $self->{_FILE} =3D $file; } =0A=0A unless(-e $self->{_FILE}){=0A cluck "No such file: " . $self->{_FIL E};=0A return; # NetCDF::open will terminate!! if file can't be opened=0A }=0A=0A # TODO: check to see if it looks like a netcdf file...=0A # it shou ld be a binary file:=0A # file tmp/test.nc =3D tmp/test.nc: data=0A # and the first three chars are 'CDF'=0A=0A # this next line will crash on fa il, can't eval!!=0A my $ncid =3D NetCDF::open($self->{_FILE},$_MODE_RO); # read-only mode should be default=0A $self->{_NCID} =3D $ncid;=0A=0A # Store basic info=0A my ($ndims,$nvars,$ngatts,$recdim);=0A NetCDF::inquire($ncid ,$ndims,$nvars,$ngatts,$recdim);=0A #warn "ndims=3D$ndims,nvars=3D$nvars,na tt=3D$ngatts,recdim=3D$recdim\n";=0A $self->{_NDIMS} =3D $ndims; #=0A $se lf->{_NVARS} =3D $nvars;=0A $self->{_NGATTS} =3D $ngatts;=0A $self->{_RECD IM} =3D $recdim;=0A=0A my ($numrecs,$dim);=0A=0A # STORE OUR DIMENSION SIZ ES=0A for (my $i=3D0;$i<$ndims;$i++){=0A my ($dname,$dsize);=0A if (NetCD F::diminq($ncid,$i,\$dname,\$dsize) =3D=3D 0){=0A # STORE FOR REFERENCE =0A $self->{_DIM}{$dname} =3D $dsize;=0A # STORE OUR RECORD DIMENSION: TELLS HOW MANY RECORDS WE HAVE=0A if ($i =3D=3D $recdim){=0A $numrecs =3D $dsize;=0A $self->{_NUMRECS} =3D $numrecs;=0A }=0A }=0A else { =0A cluck "NetCDF::diminq returned non-zero";=0A }=0A }=0A # TODO: this could cause other problems... maybe we should yell=0A return $self->{_NCID} unless ($self->{_NUMRECS});=0A=0A # CREATE LIST OF ALL VARIABLES=0A my @al l_vars =3D ();=0A for my $i (0 ... ($self->{_NVARS} - 1)){=0A my ($var,$ty pe,$ndims,@dimids,$natts);=0A NetCDF::varinq($ncid,$i,\$var,\$type,\$ndims ,\@dimids,\$natts);=0A push @all_vars,$var;=0A # also remember the type: used by get_rec_val=0A $self->{_VAR_TYPES}{$var} =3D $type;=0A }=0A $self-
{_ALLVARS} =3D \@all_vars;=0A=0A # CREATE LIST OF RECORD VARIABLES=0A # cu
rrently only works if =0A if (NetCDF::diminq($ncid,$self->{_RECDIM},\$dim,\ $numrecs) =3D=3D 0){=0A # Get record zero and store var names=0A my @recv arids =3D ();=0A my @recsizes =3D ();=0A my $nrecvars;=0A NetCDF::recinq ($self->{_NCID}, $nrecvars, \@recvarids, \@recsizes) =3D=3D 0 ||=0A cluck "Couldn't inquire about record variables\n";=0A=0A my @varnames =3D (); =0A if ($numrecs > 0){=0A my @record =3D ();=0A NetCDF::recget($self-> {_NCID}, 0, \@record) =3D=3D 0 || cluck "Couldn't get record zero\n";=0A # So we can immediately call get_next_rec()=0A $self->{_CURRENT_RECORD_NU M} =3D -1;=0A for my $k (0 ... $#record){=0A my $varid =3D $recvarids[ $k];=0A my ($var,$type,$ndims,@dimids,$natts);=0A NetCDF::varinq($nci d,$varid,\$var,\$type,\$ndims,\@dimids,\$natts);=0A push @varnames, $var ;=0A }=0A }=0A $self->{_RECVARS} =3D \@varnames;=0A }=0A=0A=0A return $self->{_NCID}; # return 'true' (!=3D0) value if succeeded=0A}=0A=0Asub clo se {=0A my $self =3D shift;=0A=0A unless($self->{_NCID}){=0A cluck "don't know how we can close NetCDF with NCID=3D" . $self->{_NCID};=0A return;=0A }=0A=0A my $rs =3D NetCDF::close($self->{_NCID});=0A=0A return !$rs;=0A} =0A=0Asub numrecs { # return number of records=0A my $self =3D shift;=0A r eturn $self->{_NUMRECS};=0A}=0A=0A=3Dhead2 get_rec_vars=0A=0AGet the list o f record variables for this file.=0A=0A=3Dcut=0Asub get_rec_vars {=0A my $s elf =3D shift;=0A my @vars =3D @{$self->{_RECVARS}};=0A=0A return @vars;=0A }=0A=0A=3Dhead2 get_all_vars=0A=0AGet the list of all variables for this fi le.=0A=0A=3Dcut=0A=0Asub get_all_vars {=0A my $self =3D shift;=0A my @vars =3D @{$self->{_ALLVARS}};=0A=0A return @vars;=0A}=0A=0A=0A=3Dhead2 get_gatt =0A=0AGet the values of a global attribute and return as a list. This is u seful=0Awhen you just want to extract the value of a given attribute. NOTE : it returns=0Aa list, even if there is only one value.=0A=0A=3Dcut=0Asub g et_gatt {=0A my ($self,$attr) =3D @_;=0A my ($type,$len);=0A=0A # First get the type of the global attribute so we know how to handle the result=0A my $ret =3D NetCDF::attinq($self->{_NCID}, NetCDF::GLOBAL, $attr, $type, $len );=0A=0A my @values =3D (); # NetCDF::attget will core if this is an undef =0A NetCDF::attget($self->{_NCID},NetCDF::GLOBAL,$attr,\@values);=0A @value s =3D perlify_values($type,\@values);=0A=0A #print "gvalues(" . scalar @val ues . ")=3D(" . join(",",@values) . ")\n";=0A return @values;=0A}=0A=0A=3Dh ead2 get_all_gatts=0A=0AGet all global attributes and return as a hash.=0A =0ASample:=0A my %gatts =3D $nc->get_all_gatts();=0A=0A foreach my $gatt ( sort keys %gatts){=0A my @vals =3D @{$gatts{$gatt}};=0A print "global att $gatt =3D " . join(",",@vals) . "\n";=0A }=0A=0A=3Dcut=0Asub get_all_gatts {=0A my $self =3D shift;=0A=0A my %gatts =3D ();=0A=0A for my $i (0 ... ($ self->{_NGATTS} - 1)){=0A=0A my ($name,$value,$type,$len);=0A NetCDF::att name($self->{_NCID}, NetCDF::GLOBAL, $i, \$name);=0A=0A # First get the ty pe of the global attribute so we know how to present=0A # the result=0A m y $ret =3D NetCDF::attinq($self->{_NCID}, NetCDF::GLOBAL, $name, $type, $le n);=0A=0A my @values =3D (); # NetCDF::attget will core if this is an unde f=0A NetCDF::attget($self->{_NCID},NetCDF::GLOBAL,$name,\@values);=0A @va lues =3D perlify_values($type,\@values);=0A $gatts{$name} =3D \@values;=0A }=0A=0A return %gatts;=0A}=0A=0A=3Dhead2 get_var_att=0A=0APass a name of a variable and the name of its attribute, and get back the value=0Aof the at tribute. This also checks to see if there is such an attribute, since=0Ath e charming NetCDF library will DIE if you try to look at a non-existent=0Aa ttribute. NOTE: it returns a list, even if there is only one value.=0A=0A =3Dcut=0Asub get_var_att {=0A=0A my ($self,$var,$attr) =3D @_;=0A my @value s =3D (); # NetCDF::attget will core if this is an undef=0A=0A # GET VARIAB LE ID=0A # which will kill you dead unless you have a valid variable=0A if ( !( grep { $_ eq $var; } @{$self->{_ALLVARS}}) ){=0A return undef;=0A } =0A my $varid =3D NetCDF::varid($self->{_NCID}, $var);=0A=0A # Now get the number of attributes it has=0A my ($var,$vtype,$ndims,@dimids,$natts);=0A N etCDF::varinq($self->{_NCID},$varid,\$var,\$vtype,\$ndims,\@dimids,\$natts) ;=0A #warn "varinq =3D $var,$vtype,$ndims,(" . join(",",@dimids) . "),$natt s" if $GoMOOS::DEBUG;=0A=0A # Find the correct attribute id... you'll see w hy in a minute=0A my $attid =3D -1;=0A for my $j (0 ... ($natts - 1)){=0A my $name;=0A NetCDF::attname($self->{_NCID}, $varid, $j, \$name);=0A if ( $name eq $attr){=0A $attid =3D $j;=0A #warn "$j:name =3D $name\n";=0A last;=0A }=0A }=0A # Return an empty list if there is no such attribute. =0A if ($attid =3D=3D -1){ return (); }=0A #warn "attid =3D $attr\n" if $Go MOOS::DEBUG;=0A=0A # Now go after the attribute itself.=0A my ($type,$len); =0A # This one dies with an untrappable error if $attr is bogus... (eval ju st dies)=0A # That's what all the above work is for!=0A NetCDF::attinq($sel f->{_NCID}, $varid, $attr, $type, $len);=0A # Now get the actual values=0A NetCDF::attget($self->{_NCID},$varid,$attr,\@values);=0A @values =3D perlif y_values($type,\@values);=0A=0A # Return single value as scalar for conveni ence=0A if (@values =3D=3D 1){=0A my $retval =3D $values[0];=0A return $r etval;=0A }=0A=0A #warn "attr values(" . scalar @values . ")=3D" . join("," ,@values) if $GoMOOS::DEBUG;=0A return @values;=0A}=0A=0A=3Dhead2 get_rec =0A=0APassed a record number: 0 to numrecs - 1 get the record and store it in _CURRENT_RECORD_NUM.=0ASee get_rec_val().=0A=0A=3Dcut=0A=0A=0Asub get_re c {=0A my ($self,$recnum, %opts) =3D @_;=0A=0A if( $self->{_NUMRECS} =3D=3D 0){=0A cluck "NO RECORDS";=0A return 0;=0A }=0A=0A if ($recnum < 0 || $r ecnum >=3D $self->{_NUMRECS}){=0A if (defined($self->{_NCID})){=0A unles s( $opts{INTERNAL}){=0A cluck "Invalid record number: $recnum";=0A } =0A cluck "Invalid record number: $recnum\n";=0A $self->{_CURRENT_RECOR D_NUM} =3D -1;=0A }=0A else {=0A #warn "No NCID: file not open!! (recnu m =3D $recnum)\n";=0A cluck "No NCID: file not open!! (recnum =3D $recnum )\n";=0A $self->{_CURRENT_RECORD_NUM} =3D -1;=0A }=0A return 0;=0A }=0A =0A # GET RECORD=0A my @current_record =3D ();=0A my $status =3D NetCDF::re cget($self->{_NCID}, $recnum, \@current_record);=0A if($status){=0A cluck "Could not get record $recnum\n";=0A $self->{_CURRENT_RECORD_NUM} =3D -1; =0A return 0;=0A }=0A=0A $self->{_CURRENT_RECORD_NUM} =3D $recnum;=0A $sel f->{_CURRENT_RECORD} =3D \@current_record;=0A=0A return 1;=0A}=0A=0A=3Dhead 2 get_rec_val=0A=0AReturn the values from current record returned by the la st call to get_rec().=0APass a variable name and get a scalar if the variab le reference is a scalar or=0Aif it is an array with only one value, also r eturn a scalar. If it is a reference to an=0Aarray just return the array re ference.=0A Sample:=0A my $sclr =3D $nc->get_rec_val('temperature_qc');=0A # time is an array but with only one value.=0A my $sclr_time =3D $nc->get_r ec_val('time');=0A Note: @arry =3D $nc->get_rec_val('time') will also work, arry will have one value.=0A my @arry =3D @{$nc->get_rec_val('current_u')} ;=0A=0A=3Dcut=0A=0Asub get_rec_val {=0A my ($self,$var) =3D @_;=0A my @recv ars =3D @{$self->{_RECVARS}};=0A my @current_record =3D @{$self->{_CURRENT_ RECORD}};=0A my $type =3D $self->{_VAR_TYPES}{$var};=0A die "WHAT? NO RECOR D VARIABLES in $self->{_FILE}: (@recvars)" unless @recvars;=0A=0A my $retva l =3D undef;=0A for my $k (0 ... $#recvars){=0A if ($recvars[$k] eq $var){ =0A $retval =3D perlify_values($type,$current_record[$k]);=0A last;=0A }=0A }=0A #if (!defined($retval)){ cluck "did not find $var\n"; }=0A=0A re turn $retval;=0A}=0A=0Asub get_next_rec {=0A my $self =3D shift;=0A=0A if( $self->{_NUMRECS} =3D=3D 0){ return 0; }=0A=0A $self->{_CURRENT_RECORD_NUM} ++;=0A=0A if ($self->{_CURRENT_RECORD_NUM} >=3D $self->{_NUMRECS}){ return 0; }=0A=0A return $self->get_rec($self->{_CURRENT_RECORD_NUM}, INTERNAL=3D> 1);=0A}=0A=0Asub get_prev_rec { =0A my $self =3D shift;=0A=0A if( $self->{_ NUMRECS} =3D=3D 0){ return 0; }=0A=0A $self->{_CURRENT_RECORD_NUM}--;=0A=0A if ($self->{_CURRENT_RECORD_NUM} < 0){ return 0; }=0A=0A return $self->get _rec($self->{_CURRENT_RECORD_NUM}, INTERNAL=3D>1);=0A}=0A=0Asub get_rewind { # reset get next record pointer=0A my ($self,%opts) =3D @_;=0A=0A if (def ined($opts{REVERSE})){=0A $self->{_CURRENT_RECORD_NUM} =3D $self->{_NUMREC S};=0A }=0A else {=0A $self->{_CURRENT_RECORD_NUM} =3D 0;=0A }=0A=0A retur n;=0A=0A}=0A=0A=3Dhead2 get_var ($var,[START=3D>$start,END=3D>$end,COUNT=3D
])=0A=0AReturn value(s) corresponding to a NetCDF variable. This accesses
variables in=0Aan array format, as opposed to the record format.=0A=0AThe parameters are: START, which tells which value (record) to start at (0 is t he=0Afirst); TO, which gives the index of the last value to return; and COU NT, which=0Atells a number of records to return. If none are specified, th e first record is=0Areturned. If both COUNT and TO are specified, COUNT ta kes precedence. If TO=0Ais -1, then values up to the last one are returned .=0A=0AIf this is called in a scalar context, then the first value is retur ned.=0AOtherwise, an array is returned. Caller beware.=0A=0A$start and $co unt are naturally an offset and a number; they default to 0 and 1,=0Awhich gives the first value only, which is useful when there is only one, which =0Ais the only time you wouldn't want to speficy it anyway. If $count is - 1 then it=0Aretrieves the entire array.=0A=0A=3Dcut=0Asub get_var {=0A my ( $self,$var,%range) =3D @_;=0A my @values =3D (); # NetCDF::varget might cor e if this is an undef=0A=0A # careful... NetCDF::varget will CORE if start or count are undef...=0A my ($start,$count);=0A $start =3D 0 if not defined ($range{START});=0A if ($range{COUNT}){=0A $count =3D $range{COUNT};=0A } =0A elsif ($range{TO}){=0A $count =3D $range{TO} - $range{START};=0A }=0A $count =3D 1 if not $count;=0A=0A # GET VARIABLE ID=0A # which will kill yo u dead unless you have a valid variable, so we check that=0A # it's valid b efore we do anything with it=0A if ( !( grep { $_ eq $var; } @{$self->{_ALL VARS}}) ){=0A return undef;=0A }=0A my $varid =3D NetCDF::varid($self->{_N CID}, $var);=0A=0A # GET NUMBER OF RECORDS IF POSSIBLE AND ADJUST COUNT IF NECESSARY=0A if ($self->{_DIM}{$var}){=0A if ($self->{_DIM}{$var} < ($star t + $count) or=0A ($range{TO} and not $range{COUNT})){=0A $count =3D $ self->{_DIM}{$var} - $start;=0A }=0A }=0A=0A # NOW CHECK START, COUNT, ETC .=0A # NOW GET THE VALUES=0A NetCDF::varget($self->{_NCID},$varid,\$start,\ $count,\@values);=0A # And don't forget to perlify them=0A @values =3D perl ify_values($self->{_VAR_TYPES}{$var},\@values);=0A=0A # Return single value as scalar for convenience=0A if (not wantarray){=0A my $retval =3D $value s[0];=0A # see get_rec_val: here it could also be a string, so we vary the test...=0A if (not $retval =3D~ /\w/ and $retval =3D=3D 0){ return 0; } =0A return $retval;=0A }=0A=0A return @values;=0A}=0A=0A=3Dhead2 perlify_v alues=0A=0AThe purpose of this function is to prettify the values extracted from NetCDF=0Afiles into something more like what a Perl programmer expect s - strings or=0Anumbers, not arrays of numbers representing char values or bytes that don't look=0Alike numbers.=0A=0AThis expects two arguments: the value's NetCDF type, and a reference to the=0Avalues. The values can be e ither an array OR scalar, depending on whether we've=0Ajust gotten the cont ents of a NetCDF variable - which itself can be either - or=0Athe content o f a variable's attribute.=0A=0AThis function returns either an array or a r eference to an array, e.g.=0A=0A @get_array =3D perlify_values($type,\@valu es);=0A $get_ref =3D perlify_values($type,\@values);=0A @use_array =3D @$ get_ref;=0A=0A=3Dcut=0Asub perlify_values {=0A my ($type,$input) =3D @_;=0A my @vals;=0A my $result;=0A=0A # the values can be an array reference, a s calar reference, or a scalar.=0A # just don't pass arrays.=0A if (ref($i nput) eq 'ARRAY') { @vals =3D @$input }=0A elsif (ref($input) eq 'SCALAR' ){ @vals =3D ($$input) }=0A else { @vals =3D ($input) }=0A=0A if ($type =3D=3D NetCDF::CHAR){=0A # weed out nulls, which can sneak in because NetC DF stores attribute strings=0A # as arrays of chars, not as null-terminate d strings=0A @vals =3D grep { $_ !=3D 0 } @vals;=0A # we also split it ba ck into an array of lines, which is the perl-ish way to=0A # look at strin gs=0A $result =3D [split "\n",(pack( 'c' x @vals, @vals))];=0A }=0A # the numeric types are easy... the input is the same as the output=0A elsif ($t ype =3D=3D NetCDF::FLOAT){ $result =3D $input }=0A elsif ($type =3D=3D NetC DF::DOUBLE){ $result =3D $input }=0A elsif ($type =3D=3D NetCDF::SHORT){ $r esult =3D $input }=0A elsif ($type =3D=3D NetCDF::LONG){ $result =3D $inpu t }=0A # except this one doesn't do what you'd expect in perl=0A elsif ($ty pe =3D=3D NetCDF::BYTE){=0A $result =3D [map { unpack('C',$_) } @vals];=0A }=0A else {=0A die "UNKNOWN NetCDF type=3D'$type'";=0A }=0A=0A return (wa ntarray ? @$result : (@$result =3D=3D 1 ? $result->[0] : $result));=0A}=0A =0A=3Dhead1 VERSION=0A=0ARevsion: $Revision: 1.29 $=0A=0A=3Dcut=0A=0A1;=0A =0A - --SLDf9lqlvOQaIe6s-- ------------------------------ End of ncdigest V1 #724 ***********************
decoders
archives: