NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

[thredds] Metadata_Conventions VS Conventions attributes



Greetings,
Following a fairly long discussion about identifying metadata conventions (1st message here: http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2010/msg00008.html ) we arrived at a road block due to differences in the way three documents describe the mechanism through which one would identify the list of metadata conventions used in the data: 


NetCDF Conventions: http://www.unidata.ucar.edu/software/netcdf/conventions.html

CF-1.4: 
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.4/cf-conventions.html#identification-of-conventions

NetCDF Dataset Discovery: 
http://www.unidata.ucar.edu/software/netcdf-java/formats/DataDiscoveryAttConvention.html



The problem:
The NetCDF COnventions page identifies an attribute called Conventions to be used to provide a list (space or comma separated) of one or more metadata conventions used in the file. In the CF-1.4 page, section 2.6.1 Identification of Conventions provides a description of the use of the Conventions attribute that can easily be (mistakenly) interpreted to mean that the Conventions attribute may not contain a list. In the NetCDF Dataset Discovery page the Conventions attribute is replaced by a Metadata_Conventions attribute that appears to be synonymous with the previously defined Conventions attribute. 


A solution:
1) Clarify the CF-1.4 page so that it is explicitly clear that a space or comma separated list of conventions is allowed.
2) Amend the NetCDF Dataset Discovery draft so that it uses Conventions. Either by replacing Metadata_Conventions with Conventions, or by explicitly stating that Metadata_Conventions and Conventions are synonyms (semantically identical) 

Can you guys get together and make that happen? Am I asking a lot?


Nathan







On Jan 21, 2010, at 3:10 PM, Bob Simons wrote:


My comments inline below.

On 1/21/2010 1:41 PM, Nathan Potter wrote:



My comments inline below.




On Jan 21, 2010, at 1:26 PM, Bob Simons wrote:


On 1/21/2010 12:56 PM, Kenneth Casey wrote:

Bob, Nathan,
Actually, the "Conventions" attribute can handle multiple conventions in 
the netCDF specification. See:
http://www.unidata.ucar.edu/software/netcdf/conventions.html

It says:

I*t is possible for a netCDF file to adhere to more than one set of
conventions, even when there is no inheritance relationship among the conventions. In this case, the value of the `Conventions' attribute may 
be a single text string containing a list of the convention names
separated by blank space (recommended) or commas (if a convention name 
contains blanks), for example *

* :Conventions = "XXX YYY" ;*


Yes. I had seen that. But it seemed like a tiny (but significant)
difference between the CF and the netCDF conventions documents.

But I hadn't noticed until now that according to this document the
"Conventions" value (e.g., "CF-1.4") is no longer used as a directory name (which had led me to think that a list of conventions would cause 
trouble).
While the CF conventions do say to use "CF-1.4" the text of the URL you sent doesn't say you CAN'T include other conventions. I think it would 
not be in the spirit of CF to tell someone not to include other
conventions if they are appropriate.


It had seemed to me that if the CF standard allowed a list of
Conventions and since CF tries to extend the COARDS conventions, the
CF standard would say the attribute should be
Conventions: COARDS, CF-1.4
not (as it does)
Conventions: CF-1.4
But maybe I'm reading too much between the lines.

In the end, I don't care if we (at ERD) use something like
Conventions: CF-1.4
Metadata_Conventions: COARDS, CF-1.4, Unidata Dataset Discovery v1.0
or just something like
Conventions: COARDS, CF-1.4, Unidata Dataset Discovery v1.0
We'll try to follow whatever the specs say.

It would be great if the CF standard could be modified a little, to
expressly forbid the use of a list of conventions, or expressly allow 
a list and specify the separator character.
So is the CF-1.4 implementing the NetCDF Conventions convention? If so is it really reasonable for it to redefine the meaning of the attribute 
oin such a way that locks out legitimate uses as defined in NetCDF
Conventions?


Sorry. I'm going to side-step the question.
I don't fully understand the relationship of the CF standard (which looks like a complete set of instructions) and the NetCDF Conventions (which seems to be just a list of different conventions from which one can choose). 
So which one takes precedence? I don't know.
I will note that for "Conventions", I never see "NetCDF", but I do see "CF-1.4". 
Is there even a disagreement between the standards? I don't know.
I don't really know the intent of the standards authors.

But this is all speculation.
In the end, the authors of the different standards need to agree on a consistent solution and modify their standards documents to clarify this situation. 






And if the CF standard allows a list, then it would be great if the
Attribute Conventions for Dataset Discovery's "Metadata_Conventions"
attribute were renamed to "Conventions".


Which I think brings us back to my original question...


Comments?


My original comment advocating the use of
 Conventions: CF-1.4
 Metadata_Conventions: [a comma-separated list]
was based on my interpretation of the standards documents.
Now I see it isn't necessarily so.
But it is not for me to decide.
Again, the authors of the different standards need to agree on a consistent solution and modify their standards documents to clarify this situation. Any solution that is consistent with all of the standards documents is fine with me. I'll work to modify ERD's datasets to comply with the standards. 




Nathan







Ken


On Jan 21, 2010, at 3:14 PM, Nathan Potter wrote:




Bob,

Thanks for the clarification.


Nathan



On Jan 21, 2010, at 12:05 PM, Bob Simons wrote:


On 1/21/2010 11:17 AM, Nathan Potter wrote:


Bob,
Roy indicated (via an offline conversation) that the "Conventions" metadata attribute is baked into the existing netCDF files at the ERD 
site, and that it is unlikely that a reprocessing of the data
would be
undertaken to correct the existing files. If that's the case then 
wouldn't this mean that it would be necessary to support both
interpretations of the "Conventions" attribute? Or was I
misunderstanding the extent of the issue w.r.t. changing the existing 
ERD holdings?


It is true that "Conventions" (and not "Metadata_Conventions") is
used in a large number of files at ERD and that it is unlikely (but 
possible) that those existing files will be changed. But we are
working to change our program that generates new .nc files so that 
new files will have both
Conventions: CF-1.4
Metadata_Conventions: COARDS, CF-1.4, Unidata Dataset Discovery v1.0
Since we have configured THREDDS to get metadata from the penultimate 
file, the new metadata will appear in almost all of our THREDDS
datasets sooner (many of our datasets are near-real-time) or later. 
We may have to change a few datasets by hand. But the plan is to
convert to using both "Conventions" and "Metadata_Conventions" for 
the reasons I stated earlier this morning.





Nathan




On Jan 21, 2010, at 10:57 AM, Bob Simons wrote:


Please leave "Metadata_Conventions" as is.
The problem is that the widely used CF convention specifies that the value of "Conventions" should be just "CF-1.4" and that the value is sometimes interpreted as a directory name. So it can't be a list of 
conventions. See
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.4/cf-conventions.html#identification-of-conventions
So it is useful to have a separate "Metadata_Conventions" attribute 
which can be a comma-separated list of convention names, e.g.,
"COARDS, CF-1.4, Unidata Dataset Discovery v1.0". See
http://www.unidata.ucar.edu/software/netcdf-java/formats/DataDiscoveryAttConvention.html

.
The misuse of "Conventions" at the ERD site was my mistake. I am 
working to change it to (for most datasets)
Conventions : CF-1.4
Metadata_Conventions: COARDS, CF-1.4, Unidata Dataset Discovery v1.0 

That's my $0.02.

On 1/21/2010 10:36 AM, Nathan Potter wrote:



Greetings,
Any idea how many people are using the UNIDATA's NetCDF Attribute 
Convention for Dataset Discovery??
If Roy's site is the primary (only?) example of it in actual use, 
and we
accept Benno's point that the name attribute
*Metadata_Conventions* is
redundant, should we not consider amending the draft to change the 
attribute name from *Metadata_Conventions* to *Conventions*?
(Assuming that it hasn't been push passed the draft spec: Dataset 
Discovery NetCDF Attribute Convention
<http://www.unidata.ucar.edu/software/netcdf-java/formats/DataDiscoveryAttConvention.html >) 




If we don't change it might we amend the draft to explicitly
make the
attributes *Metadata_Conventions* and *Conventions* synonymous? 


Nathan






On Jan 14, 2010, at 9:44 AM, Roy Mendelssohn wrote:


There are metadata in our files that follows all of those
conventions.
In trying to relay that act, we felt this was the best way to
relay
that fact. So you will find the metadata required for CF, the
Coastwatch conventions, and the Unidata Data Discovery
Conventions.

-Roy


On Jan 14, 2010, at 9:40 AM, Benno Blumenthal wrote:
OK, again trying desperately to understand your cryptic e- mails, 

http://thredds1.pfeg.noaa.gov/thredds/dodsC/satellite/SW/chla/1day.das


has many of those attributes. You did, however, say
String Conventions "COARDS, CF-1.0, Unidata Dataset Discovery 
v1.0,
CWHDF";

whereas the document suggested looking for
String Metadata_Conventions = "Unidata Dataset Discovery v1.0"; 


What you are doing, of course, makes sense. Did the original
document
get corrected over time?

Benno


On Thu, Jan 14, 2010 at 12:31 PM, Roy Mendelssohn
<Roy.Mendelssohn@xxxxxxxx <mailto:Roy.Mendelssohn@xxxxxxxx>
<mailto:Roy.Mendelssohn@xxxxxxxx>> wrote:
No we use it extensively. Look at any of he satellite data at 
our
TDS (http://oceanwatch.pfeg.noaa.gov/thredds/catalog.html).
It is
just since this is not automatically produced by the TDS that we 
have a program of our own that generates our catalogs.

-Roy

On Jan 14, 2010, at 9:18 AM, Benno Blumenthal wrote:
So, to be really concrete, despite this convention of using 
netcdf
attributes to insert metadata which you just pointed us to, you 
don't use it at all, you just scan your netcdf files
yourself and
generate the TDS XML directly.

Benno

On Thu, Jan 14, 2010 at 12:14 PM, Roy Mendelssohn
<Roy.Mendelssohn@xxxxxxxx <mailto:Roy.Mendelssohn@xxxxxxxx> 
<mailto:Roy.Mendelssohn@xxxxxxxx>> wrote:
We did this "by hand" (well actually we have a program that 
does
it) in generating our catalogs not certain ow others do it. 

-Roy

On Jan 14, 2010, at 8:56 AM, Nathan Potter wrote:




Roy,
Briefly scanning that page it looks like this gist of it is 
that
one could use NcML to add/complete a "Unidata Dataset
Discovery
v1.0" set of metadata to a data set. THis could then be
harvested
by the TDS, or by other software that understood the "Unidata 
Dataset Discovery v1.0" convention.

Is that the general idea?

N



On Jan 14, 2010, at 8:41 AM, Roy Mendelssohn wrote:


Hi Nathan:

Look at:

http://www.unidata.ucar.edu/software/netcdf-java/formats/DataDiscoveryAttConvention.html



-Roy

On Jan 14, 2010, at 8:33 AM, Nathan Potter wrote:





Greetings,

I have some questions about the origin of the
thredds:geospaptialCoverage element that is commonly
found in
THREDDS catalogs. It is my understanding that:

- The TDS can generate this metadata element by
interrogating
certain types of source data files.
- People that write their own THREDDS catalog files to be 
served
by the TDS can add the thredds:geospaptialCoverage metadata 
directly to their catalogs (using whatever process they
wish).

Is the thredds:geospaptialCoverage also something that
can be
added through normal channels in NcML augmented data sets? 



= = =
Nathan Potter ndp at opendap.org <http://opendap.org>
OPeNDAP, Inc. +1.541.231.3317




_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx > 
<mailto:thredds@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/


**********************
"The contents of this message do not reflect any position 
of the
U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx
<mailto:Roy.Mendelssohn@xxxxxxxx>
<mailto:Roy.Mendelssohn@xxxxxxxx> (Note new e-mail address) 
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 



= = =
Nathan Potter ndp at opendap.org <http://opendap.org>
OPeNDAP, Inc. +1.541.231.3317






**********************
"The contents of this message do not reflect any position of 
the
U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx
<mailto:Roy.Mendelssohn@xxxxxxxx>
<mailto:Roy.Mendelssohn@xxxxxxxx>
(Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 




--
Dr. M. Benno Blumenthal benno@xxxxxxxxxxxxxxxx
<mailto:benno@xxxxxxxxxxxxxxxx>
<mailto:benno@xxxxxxxxxxxxxxxx>
International Research Institute for climate and society
The Earth Institute at Columbia University
Lamont Campus, Palisades NY 10964-8000 (845) 680-4450


**********************
"The contents of this message do not reflect any position of the 
U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx
<mailto:Roy.Mendelssohn@xxxxxxxx>
<mailto:Roy.Mendelssohn@xxxxxxxx>
(Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 





--
Dr. M. Benno Blumenthal benno@xxxxxxxxxxxxxxxx
<mailto:benno@xxxxxxxxxxxxxxxx>
<mailto:benno@xxxxxxxxxxxxxxxx>
International Research Institute for climate and society
The Earth Institute at Columbia University
Lamont Campus, Palisades NY 10964-8000 (845) 680-4450


**********************
"The contents of this message do not reflect any position of
the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx
<mailto:Roy.Mendelssohn@xxxxxxxx>
<mailto:Roy.Mendelssohn@xxxxxxxx>
(Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"



= = =
Nathan Potter ndp at opendap.org <http://opendap.org>
OPeNDAP, Inc. +1.541.231.3317






_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/


--
Sincerely,

Bob Simons
IT Specialist
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
Phone: (831)658-3205
Fax: (831)648-8440
Email: bob.simons@xxxxxxxx <mailto:bob.simons@xxxxxxxx>

The contents of this message are mine personally and
do not necessarily reflect any position of the
Government or the National Oceanic and Atmospheric
Administration.
<>< <>< <>< <>< <>< <>< <>< <>< <><

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/


= = =
Nathan Potter ndp at opendap.org <http://opendap.org>
OPeNDAP, Inc. +1.541.231.3317






--
Sincerely,

Bob Simons
IT Specialist
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
Phone: (831)658-3205
Fax: (831)648-8440
Email: bob.simons@xxxxxxxx <mailto:bob.simons@xxxxxxxx>

The contents of this message are mine personally and
do not necessarily reflect any position of the
Government or the National Oceanic and Atmospheric
Administration.
<>< <>< <>< <>< <>< <>< <>< <>< <><

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/


= = =
Nathan Potter ndp at opendap.org <http://opendap.org>
OPeNDAP, Inc. +1.541.231.3317




_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/



[NOTE: The opinions expressed in this email are those of the author
alone and do not necessarily reflect official NOAA, Department of
Commerce, or US government policy.]

Kenneth S. Casey, Ph.D.
Technical Director
NOAA National Oceanographic Data Center
1315 East-West Highway
Silver Spring MD 20910
301-713-3272 ext 133
http://www.nodc.noaa.gov/ <http://www.nodc.noaa.gov/sog>





--
Sincerely,

Bob Simons
IT Specialist
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
Phone: (831)658-3205
Fax: (831)648-8440
Email: bob.simons@xxxxxxxx

The contents of this message are mine personally and
do not necessarily reflect any position of the
Government or the National Oceanic and Atmospheric
Administration.
<>< <>< <>< <>< <>< <>< <>< <>< <><


= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317






--
Sincerely,

Bob Simons
IT Specialist
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
Phone: (831)658-3205
Fax:   (831)648-8440
Email: bob.simons@xxxxxxxx

The contents of this message are mine personally and
do not necessarily reflect any position of the
Government or the National Oceanic and Atmospheric
Administration.
<>< <>< <>< <>< <>< <>< <>< <>< <><


= = =
Nathan Potter                        ndp at opendap.org
OPeNDAP, Inc.                        +1.541.231.3317
  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: