NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Roy Mendelssohn wrote:
Hi John: Some questions on THREDDS aggregation. 1. Is there a limit to the number of files that can be aggregated over?
nope
2. Can aggregation occur over sub-directories of a directory structure?
using "scan" i assume?supposedly you can have multiple scan directives within the aggregation, but its not well tested. But I would try this if you need this feature.
the scan directive is still pretty primitive, we will continue to improve it, adding a "recurse" tag might be one way. remember that the aggregated files need to be pretty much homogenous.
3. For a lot of time periods, when aggregating fields over time. do you have any feel for the trade-off in speed of aggregation for the size of the netcdf file versus the number of files aggregated over (i.e. if we have 6-hourly data, should we produce 6-hourly files, daily files, weekly files, monthly files - and what would be the likely speed tradeoff if we want to extract a time series of a relatively small region?).
My intuition is that you want to create fewer large files, not lots of little files. It costs the same to open a big or a little file. My current rule of thumb is try to write files that are 50 - 200 Mbytes. In the future, we may add a feature that will try to open all the needed files in different threads. So that may argue for smaller file sizes, but its theoretical at this point.
TIA, -Roy
thredds archives: