NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Somewhat related to this thread: when is the practice of *not* using the -close option when using FILE in pqact recommended? Invariably I find that omitting its use leads to pqact not being able to keep up with the incoming stream. This is seen especially well when we are writing the GEFS GRIB2 files to disk. _____________________________________________ Kevin Tyle, Systems Administrator Dept. of Atmospheric & Environmental Sciences University at Albany Earth Science 235, 1400 Washington Avenue Albany, NY 12222 Email: ktyle@xxxxxxxxxx Phone: 518-442-4578 _____________________________________________ ________________________________________ From: ldm-users-bounces@xxxxxxxxxxxxxxxx <ldm-users-bounces@xxxxxxxxxxxxxxxx> on behalf of Michael Dross <mdross@xxxxxxxxxxxxxxxxxx> Sent: Wednesday, October 28, 2015 12:02 PM To: Tom Yoksas; ldm-users@xxxxxxxxxxxxxxxx Subject: Re: [ldm-users] 20151028: Pqact FILE writes with custom removal? Is there one type of filesystem that is more efficient at handling these types file large volumes of finds/creations/deletions? ie. EXT4, zfs. - Mike > On Oct 28, 2015, at 11:56 AM, Tom Yoksas <yoksas@xxxxxxxx> wrote: > > All, > > re: how to efficiently scour LDM-created data products from disk > > If one organizes one's data files into daily directories, the need to > 'find' each file that should be scoured can be pared down to finding > which directories+contents need to be scoured. This does not, however, > get around the fact that deleting large numbers of files > (e.g., millions for NEXRAD Level III files kept on disk for long > periods of time) is a _very_ costly endeavor. > > Cheers, > > Tom > >> On 10/28/2015 09:47 AM, Gerry Creager - NOAA Affiliate wrote: >> How often to you run your scour command? We've exercised scour pretty >> hard and were able to watch it keep up, if it were run frequently. >> There's a one-time penalty, if you've been running it, say, daily or >> less frequently, but if you run it hourly, it should keep up. >> >> gerry >> >> On Wed, Oct 28, 2015 at 10:39 AM, Steve Emmerson <emmerson@xxxxxxxx >> <mailto:emmerson@xxxxxxxx>> wrote: >> >> LDMers, >> >> On Wed, Oct 28, 2015 at 9:28 AM, Donna Cote <d-cote@xxxxxxxx >> <mailto:d-cote@xxxxxxxx>> wrote: >> >> I would like to ask for a more io-friendly scour utility. >> >> >> It's on my list (see <https://github.com/Unidata/LDM/issues/27>. >> Can't say when I'll get around to it, though. >> >> Regards, >> Steve Emmerson >> >> _______________________________________________ >> ldm-users mailing list >> ldm-users@xxxxxxxxxxxxxxxx <mailto:ldm-users@xxxxxxxxxxxxxxxx> >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ >> -- >> Gerry Creager >> NSSL/CIMMS >> 405.325.6371 >> ++++++++++++++++++++++ >> “Big whorls have little whorls, >> That feed on their velocity; >> And little whorls have lesser whorls, >> And so on to viscosity.” >> Lewis Fry Richardson (1881-1953) >> >> _______________________________________________ >> ldm-users mailing list >> ldm-users@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ > > -- > +----------------------------------------------------------------------+ > * Tom Yoksas UCAR Unidata Program * > * (303) 497-8642 (last resort) P.O. Box 3000 * > * yoksas@xxxxxxxx Boulder, CO 80307 * > * Unidata WWW Service http://www.unidata.ucar.edu/ * > +----------------------------------------------------------------------+ > > _______________________________________________ > ldm-users mailing list > ldm-users@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ _______________________________________________ ldm-users mailing list ldm-users@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
ldm-users archives: