Hi All,
This is a really excellent discussion. At the end of it we'd make real
progress towards a shared sense of priorities if we could agree upon a
summary of it in bullet form.
At the heart of the discussion is the blunt question that Ben raised:
"abandon the CF-netCDF encoding spec. and ONLY be proposing
CF-OPeNDAP"? I'm not sure how to answer that directly. But if we
alter the question to be, "*_Should we consider the fusion of OPeNDAP
and WCS as a fundamental objective of GALEON?_*", I would say
emphatically YES!
Differences of opinion are so often due to different interpretations of
terminology and fundamental assumptions that have not been exposed.
Below I will try to spell out my own assumptions (presumably fairly
similar to Jon Blower's and Roy Mendelssohn's) in hopes of allowing
others to pin point where we may be starting from terminology confusions
or differing assumptions.
Terminology:
1. "netCDF-CF": we often refer to netCDF-CF in a manner so loose
that it becomes a source of confusion. NetCDF-CF is a standard
for data on complex grids -- curvilinear in XY; sigma and
density-related in Z; climatological and artificial calendars in
T; and heading towards "tile mosaics" and 5D forecast ensembles in
the near future. (The current ArcGIS release, as I understand it,
does not yet even fully support the vastly simpler COARDS
specification. There is a chasm separating ArcGIS from netCDF-CF
datasets.)
2. "OPeNDAP": for purposes of this discussion I would argue that
OPeNDAP is simply an approach that allows applications to access
remote, time-aggregated collections of netCDF-CF files (virtual
datasets -- often terabyte sized) through the unaltered netCDF API
-- as if they were local netCDF files. (There are opportunities
for endless interesting technical discussions about protocols,
mime-types, streaming, etc., but I think they miss the simpler
essence needed for this discussion.)
Assumptions:
1. The fundamental goal of GALEON is to promote interoperability.
There are two complementary forms of interoperability that must be
considered:
1. "inter-community" (between the fluid earth science community
and others) -- characterized by making complex fluid earth
science data accessible to users with a very broad range of
outlooks and levels of (mathematical) sophistication
2. "intra-community" (within the fluid earth science community)
-- characterized by exchanges between users of high
mathematical sophistication; data on complex grids (see
"netCDF-CF" definition, above); the need for high numerical
fidelity; and the need to "slice" the data arbitrarily along
all principal planes and axes in 4D space -- e.g. to extract
a depth-time slice from a 4D dataset
2. The current intention of the GALEON project is to incorporate the
full semantic richness of netCDF-CF into the WCS framework. (If
not, then what level of CF semantics is targeted by GALEON? and
in order to serve what classes of users?)
3. In order to succeed at achieving interoperability the applications
that users rely upon (in their distinct user communities) must be
able to request and read data and information in the proposed
standards and protocols
From T1 I conclude that many (most) OGC-friendly applications in use
outside of fluid earth science will never support the complexity of
netCDF-CF datasets -- even applications that might benefit from
occasional access to native fluid earth science data. Working with full
blown netCDF-CF datasets is not easy. To succeed requires the
involvement of programmers with mathematical skills similar to what we
find in the fluid earth science community. (Imagine an application used
by farmers planning crop rotations of their fields -- unlikely to be
able to handle curvilinear model output climate forecast data in
netCDF-CF, though there would be potential uses for this data. Or an
application for yachtsman to perform on-board navigation -- same deal.)
The importance of getting the WCS-OPeNDAP fusion into our conversation
is that we need to unify our community in order to promote
interoperability. We need to break down the barriers that currently
separate "camps": WCS versus OPeNDAP.
- Steve
=====================================================
stefano nativi wrote:
Dear Jon,
Thank you for the valuable and stimulating comments.
I am in serious agreement with several of the points you touched. I
particularly agree that "more clarity is needed about the use cases
that WCS is thought to satisfy". As matter of fact, Ben and I are
working on a position paper with the temporary title: "Which coverage
access services do we really need ?".
For me, the Web Coverage Service(s) is/are important for enabling
cross-disciplinary interoperability.
IMHO, this extension profile draft contributes to clarify this
question. In fact, mapping CF-NetCDF to WCS data model should provide
a useful indication of the complexity and rewards that we are going
to face. In fact, this exercise was done to support interoperability
and cross-disciplinary research/applications. This is important to
enable the CF-netCDF use in multi-disciplinary contexts.
From my perspective, WCS 1.1.x is a rather complex specification; it
will be replaced by WCS 1.2 which is trying to solve the complexity
problems. Thus, this extension specification must be considered as
propedeutic for the WCS 1.2 CF-netCDF extension.
I will try to provide more discussion points inline.
I have always regarded the OGC standards as a way in which the
met-ocean community can communicate its data to other communities.
Personally I do not look to OGC standards for providing methods for
exchanging data within our community. We already have
well-established tools, standards, protocols, data models and software
for doing this. When communicating outside ones own community, one
often has to accept that a certain amount of information will be lost.
I wonder if it is realistic to expect that we might use WCS in future
to communicate our data in all its subtlety?
Due to multiple Community specifications and conventions, I think
that a certain amount of information is often lost even when
communicating inside ones own community. In my opinion, the real
question is about the interoperability level required by applications to run.
For example, GEOSS is working on cross-disciplinary applications for
serving many SBAs (Societal Benefit Areas); in this case, what is the
exact meaning of "communicating outside ones Community"?
My other concern is that the WCS world is changing extremely rapidly,
with at least four versions of the specification in existence (1.0,
1.1, 1.2 and "plus"). This contrasts with the relatively small
numbers of real WCS servers "out there" in the wild (GALEON has done a
great job in encouraging people to stand up real systems but in
general, uptake of WCS is rather low). The ISO19123 Coverage model is
also likely to be revised as it is broken in places. Can we keep up
with this evolution?
I completely share your concerns. I made similar comments to the WCS RWG.
In my opinion, ISO 19123 Coverage model is best instrument we have
now to confront the different "disciplinary coverage models" and
design interoperability specifications for coverages.
Thirdly, the WCS1.2 "core plus extensions" model worries me a bit. I
understand that the "core" is small, implying that the different
community "extensions" will have little interoperability with each
other. Effectively, we'll end up with a lot of mutually-incompatible
versions of WCS that share some extremely limited features (perhaps
only their terminology). The "core plus extensions" model implies
that WCS has a desire to encompass all data, a scope that is arguably
too wide to be useful or realistic.
You raised an important issue that must be carefully addressed by the
SWG for WCS 1.2.
Actually, the WCS 1.2 "core" specification use case is very close to
the one you describe in your following comment (i.e. a WCS
specification considerably simplified, with a defined list of things
that it's good for and things that it isn't good for).
The question Ben and I are posing in our position paper, is about how
to decide the "list of things that it's good for". In fact, that
means to decide how "general purpose" the service will be to the
detriment of its effectiveness (i.e. semantic meaningfulness). The
latter was one of the main drawback outlined for the WCS 1.0.
Personally I would like to see the WCS specification considerably
simplified, with a defined list of things that it's good for and
things that it isn't good for. For the sake of argument, let's
imagine a WCS that only serves data as GeoTIFFs in a lat-lon
coordinate reference system (or lat-lon-elevation-time in 4D). Sure,
a lot of information would be lost across this interface (e.g. the
original data's grid) and certain specialist use cases could not be
satisfied (e.g. calculating heat transports). However, it would be
much simpler to implement (client and server) and would still satisfy
a large number of use cases in which met-ocean data is needed outside
our community.
This is very close to an enhanced WMS; in my opinion you presented a
valuable use case. However, the coverage concept and its types are
quite different from the map concept and its types. Thus, for the
sake of cross-disciplinary interoperability we must still wonder
whether this is the only service we need, or if we need "proper"
coverage access service(s), too.
I guess I'm saying that I don't think it's realistic or desirable to
develop a WCS specification to serve all kinds of geographic data,
which is the way things seem to be headed in the OGC world. The
effort required to support a small number of specialist use cases (in
the specification, in servers and in clients) is fearsome. I think
WCS should aim to complement, not replace, technologies such as
OPeNDAP by satisfying a different class of user. I think more clarity
is needed about the use cases that WCS is thought to satisfy.
In my opinion, to provide access to CF-netCDF data model and encoding
formats through "non-traditional" protocols should be a valuable
asset for the FES community.
Best regards,
Stefano
