NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><tt>Michael,</tt></p>
<p><tt>I'm commenting inline. </tt><tt><br>
</tt></p>
<div class="moz-cite-prefix"><tt>On 12/5/20 0:14, Michael McDonald
wrote:</tt><tt><br>
</tt></div>
<blockquote type="cite"
cite="mid:CAGZ4aURRNQGaEFqB_jEfxs71LT2=g6DaRtXgxQVkLNBLQma9uw@xxxxxxxxxxxxxx">
<pre class="moz-quote-pre" wrap="">Antonio,
While the "along_track" is the UNLIMITED dimension in all of these
swath files. Our attempts to just use this as the dimension for the
joinExisting aggregation does not throw any issues wrt THREDDS at
catalog init time, but when you actually try to load/use this dataset
it does not work and fails with the message in Panoply,</pre>
</blockquote>
<p><tt>Yes, that is normal when the error it's not in the
aggregation itself but is in the data aggregated. Usually
aggregation are not evaluated until their aggregated data are no
accessed. </tt><tt><br>
</tt></p>
<p><tt>Instead of using Panoply to access the aggregation using TDS
in between, test the aggregation itself using a ncml file, and
open it from Panoply. </tt><tt><br>
</tt></p>
<p><tt>This will allow you to check if the aggregation it's working.
The NCML tab on the ToolsUI it's very useful. </tt><tt><br>
</tt></p>
<blockquote type="cite"
cite="mid:CAGZ4aURRNQGaEFqB_jEfxs71LT2=g6DaRtXgxQVkLNBLQma9uw@xxxxxxxxxxxxxx">
<pre class="moz-quote-pre" wrap="">
"There was an error preparing the data: Failed creating the data
handler: Could not read lon axis array."
</pre>
</blockquote>
<p><tt>I don't from which part of Panoply it's this error coming
from. Usually the exception stacktrace it's providing more
useful info. </tt><tt><br>
</tt></p>
<p><tt><br>
</tt></p>
<blockquote type="cite"
cite="mid:CAGZ4aURRNQGaEFqB_jEfxs71LT2=g6DaRtXgxQVkLNBLQma9uw@xxxxxxxxxxxxxx">
<pre class="moz-quote-pre" wrap="">
e.g., joinExisting aggregation on along_track (fails with: Could not
read lon axis array.)
<a class="moz-txt-link-freetext"
href="http://tds.coaps.fsu.edu/thredds/dodsC/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily_TESTjoinExisting-along_track.html";>http://tds.coaps.fsu.edu/thredds/dodsC/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily_TESTjoinExisting-along_track.html</a>
<netcdf xmlns=<a class="moz-txt-link-rfc2396E"
href="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>"http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";</a>>
<aggregation type="joinExisting" dimName="along_track">
<scan
location="/Net/data/qscat/data/nc_L2B_v4p0_12/nc_L2B_12_daily/2009"
suffix=".nc" subdirs="false" />
</aggregation>
</netcdf>
</pre>
</blockquote>
<p><tt>Try to use "static" aggregation instead of dynamic (i.e.
scan) aggregation:</tt><tt><br>
</tt></p>
<p><tt><netcdf
xmlns=<a class="moz-txt-link-rfc2396E"
href="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>"http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";</a>><br>
<aggregation type="joinExisting" dimName="along_track"><br>
<netcdf
location="/Net/data/qscat/data/nc_L2B_v4p0_12/nc_L2B_12_daily/2009/QSCAT_12_2009324.nc"/><br>
<netcdf
location="/Net/data/qscat/data/nc_L2B_v4p0_12/nc_L2B_12_daily/2009/QSCAT_12_2009325.nc"/><br>
</aggregation><br>
</netcdf><br>
</tt></p>
<p>the reason, it's because the scan element doesn't guaranteed the
file order: </p>
<p><a
href="https://docs.oracle.com/javase/7/docs/api/java/io/File.html#listFiles()">https://docs.oracle.com/javase/7/docs/api/java/io/File.html#listFiles()</a></p>
<p>The ToolsUI NCML aggregation panel tab can show tha files
agregated and their order. <br>
</p>
<p>You <img src="cid:part2.48D04669.A43A1047@gmail.com" alt=""></p>
<p>The netcdf-java/panoply it's classifying the dataset as
curviliner, because the time coordinates has not been explicitly
declared on variables. <br>
</p>
<p>Yo can fix that adding the time to the coordinates attribute (I
have removed also the actual_range attribute for the time
variable):</p>
<p><tt><netcdf
xmlns=<a class="moz-txt-link-rfc2396E"
href="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>"http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";</a>><br>
<aggregation type="joinExisting" dimName="along_track"><br>
<netcdf
location="/Net/data/qscat/data/nc_L2B_v4p0_12/nc_L2B_12_daily/2009/QSCAT_12_2009324.nc"/><br>
<netcdf
location="/Net/data/qscat/data/nc_L2B_v4p0_12/nc_L2B_12_daily/2009/QSCAT_12_2009325.nc"/><br>
<variable name="time"><br>
<remove name="actual_range" type="attribute" /><br>
</variable><br>
<variable name="eastward_wind"><br>
<attribute name="coordinates" value="time lon lat" /><br>
</variable><br>
<variable name="northward_wind"><br>
<attribute name="coordinates" value="time lon lat" /><br>
</variable><br>
<variable name="wind_speed"><br>
<attribute name="coordinates" value="time lon lat" /><br>
</variable><br>
<variable name="wind_to_direction"><br>
<attribute name="coordinates" value="time lon lat" /><br>
</variable><br>
<variable name="wind_to_direction"><br>
<attribute name="coordinates" value="time lon lat" /><br>
</variable><br>
<variable name="simplified_wvc_quality_flag"><br>
<attribute name="coordinates" value="time lon lat" /><br>
</variable><br>
<variable name="rain_impact"><br>
<attribute name="coordinates" value="time lon lat" /><br>
</variable><br>
<variable name="distance_from_coast"><br>
<attribute name="coordinates" value="time lon lat" /><br>
</variable><br>
</aggregation><br>
</netcdf><br>
</tt></p>
<p>Then the dataset it's been classified as SWATH: <br>
</p>
<p><img src="cid:part3.B8AE5CF5.437BD1A7@gmail.com" alt=""></p>
<p>If the time it's *not* added as coordinate, then the netcd-java
is detecting as *just* curvilinear horizontal projection. <br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAGZ4aURRNQGaEFqB_jEfxs71LT2=g6DaRtXgxQVkLNBLQma9uw@xxxxxxxxxxxxxx">
<pre class="moz-quote-pre" wrap="">
This *simplified* product is a collection of multiple quikscat swath
passes in a 24 hour period to assemble a full coverage of the globe.
orbit_file_source:
<a class="moz-txt-link-freetext"
href="https://podaac-opendap.jpl.nasa.gov/opendap/allData/quikscat/L2B12/v4.0/";>https://podaac-opendap.jpl.nasa.gov/opendap/allData/quikscat/L2B12/v4.0/</a>
Which looks like this for a single day.
<a class="moz-txt-link-freetext"
href="http://tds.coaps.fsu.edu/thredds/wms/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily/2009/QSCAT_12_2009325.nc?STYLES=&SERVICE=WMS&FORMAT=image%2Fpng&REQUEST=GetMap&HEIGHT=512&WIDTH=1024&VERSION=1.1.1&BBOX=-180.0%2C-80.0%2C180.0%2C90.0&LAYERS=wind&SRS=EPSG%3A4326&TRANSPARENT=TRUE";>http://tds.coaps.fsu.edu/thredds/wms/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily/2009/QSCAT_12_2009325.nc?STYLES=&SERVICE=WMS&FORMAT=image%2Fpng&REQUEST=GetMap&HEIGHT=512&WIDTH=1024&VERSION=1.1.1&BBOX=-180.0%2C-80.0%2C180.0%2C90.0&LAYERS=wind&SRS=EPSG%3A4326&TRANSPARENT=TRUE</a>
Our *end goal* is to have a virtually aggregated collection of all
this simplified daily swath data from 1999 to 2009 for ingestion as an
OPeNDAP/WMS dataset into a web-based data visualizer like OWGIS
(<a class="moz-txt-link-freetext"
href="https://owgis.org/";>https://owgis.org/</a>), so that users can pick the
date(s) they want,
and view this in a web GUI.
</pre>
</blockquote>
<p>The above agregation it's the same kind of dataset, but with all
time steps aggregated. It will be an actual bit swath.<br>
</p>
<p>This a second order problem, but you will find some performance
problems due to the aggregation needs to read each file on the
aggregation, to find the size and the corresponding coordinates
values. If you use the static approach you can add both on the
ncml (or in third party file). <br>
</p>
<p><br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAGZ4aURRNQGaEFqB_jEfxs71LT2=g6DaRtXgxQVkLNBLQma9uw@xxxxxxxxxxxxxx">
<pre class="moz-quote-pre" wrap="">
We tried to use joinNew to aggregate all these like files on a new
dimension "T" and populate the time info using the filename
(dateFormatMark="QSCAT_#HH_yyyyDDD") and get the same error (fails
with: Could not read lon axis array.)
e.g., joinNew aggregation on new dimension T
<a class="moz-txt-link-freetext"
href="http://tds.coaps.fsu.edu/thredds/dodsC/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily_TESTjoinNew-time-lon-lat.html";>http://tds.coaps.fsu.edu/thredds/dodsC/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily_TESTjoinNew-time-lon-lat.html</a>
<netcdf xmlns=<a class="moz-txt-link-rfc2396E"
href="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>"http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";</a>>
<aggregation dimName="T" type="joinNew">
<variableAgg name="time"/>
<variableAgg name="lon"/>
<variableAgg name="lat"/>
<variableAgg name="eastward_wind"/>
<variableAgg name="northward_wind"/>
<variableAgg name="wind_speed"/>
<variableAgg name="wind_to_direction"/>
<variableAgg name="simplified_wvc_quality_flag"/>
<variableAgg name="rain_impact"/>
<variableAgg name="distance_from_coast"/>
<scan dateFormatMark="QSCAT_#HH_yyyyDDD"
location="/Net/data/qscat/data/nc_L2B_v4p0_12/nc_L2B_12_daily/2009"
suffix=".nc" subdirs="false" />
</aggregation>
</netcdf>
Any ideas what we are doing wrong in THREDDS, or is the issue in the
data files we simplified?
</pre>
</blockquote>
<p>It's not at the THREDDS itself, it's a problem on the dataset
attributes and aggregations. Another problem will be if this big
SWATH it's been accesible and usable from the Panoply tool or the
WMS service. <br>
</p>
<p>Hope this helps. <br>
</p>
<p>Antonio</p>
<pre class="moz-signature" cols="72">--
Antonio S. Cofiño
Dep. de Matemática Aplicada y
Ciencias de la Computación
Universidad de Cantabria
<a class="moz-txt-link-freetext"
href="http://www.meteo.unican.es";>http://www.meteo.unican.es</a></pre>
<p><br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAGZ4aURRNQGaEFqB_jEfxs71LT2=g6DaRtXgxQVkLNBLQma9uw@xxxxxxxxxxxxxx">
<pre class="moz-quote-pre" wrap="">
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">
Ian,,
What aggregation are you expecting? just on the 'along_track' dimension?
Antonio
On 5/5/20 19:46, Ian Dimitri wrote:
Hello All,
Adding onto Michael's existing thread, I am working closely with him attempting
to figure out the issue we are having with the aggregation of swath data that
has a two-dimensional grid with track and cross-track coordinates. We have
tried using FMRC, joinExisting, and joinNew file aggregation methods within
THREDDS, all with no success.
We tried using many different aggregations methods
that exist within the ncWMS2 interface. We have tried doing the simple wildcard
aggregation attempt by using the wildcard character (e.g.
/ASCAT/nc_L2B_25_daily/2020/*.nc).
We have also tried to use the "Dynamic Services" that is also available through
ncWMS2, but that also did not seem to work with the multi-dimensional swath
data that we are
trying to aggregate.
Finally another method we tried was through geoserver, where we attempted to
use the image mosaic option that it has but that option also proved to be a
dead end.
None of the methods mentioned above worked properly, and we are wondering if
anyone has had any success aggregating swath data with the methods, and would
like to share any
special steps that had to be taken to make said method work. Are there any
other methods I did not mention above that could possibly be able to aggregate
the multi-demensional swath data?
Any insight or help would be appreciated
unaggregated data
<a class="moz-txt-link-freetext"
href="http://tds.coaps.fsu.edu/thredds/catalog/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily/catalog.html";>http://tds.coaps.fsu.edu/thredds/catalog/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily/catalog.html</a>
attempted aggregation with joinNew
<a class="moz-txt-link-freetext"
href="http://tds.coaps.fsu.edu/thredds/satellite_testing.html?dataset=joinnew-TEST-coaps-qscat-nc_L2B_v4p0_12-nc_L2B_12_daily";>http://tds.coaps.fsu.edu/thredds/satellite_testing.html?dataset=joinnew-TEST-coaps-qscat-nc_L2B_v4p0_12-nc_L2B_12_daily</a>
</pre>
</blockquote>
</blockquote>
</body>
</html>
thredds archives: