NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
> Hi Murray, > > I agree with you 100% that creating TGZ files on the fly is the best way > to go and will bring it to the NODC leadership's attention. > > Charles > On 03/18/2012 10:43 AM, Brown, Murray wrote: > > Gang (sounds right), > > > > Charles' other product line is covered by a different exercise at > > http://marinedataliteracy.org/odv/gtspp_bestnc.htm. These are TGZ > > compressed nc profiles all loaded at once into the display program > > OceanDataView. The TGZ files are available per-ocean basin, and > > per-month. > > > > But with that approach and (if we had it) with multi-profile NC files, > > the more important question is how do we reduce the file load to the > > spatial and temporal envelope of interest? I know beans about the > > technology, but it seems like creating TGZ compilations or NC > > compilations on the fly would be a good bet. I say this mainly > > because I'm thinking of our IOC students in Pago Pago who have just > > slightly better technology than those old phones we used to stick into > > rubber cradles (yeah, I know you remember Omnet). The global > > operational game isn't won until we can offer everybody a reasonable > > shot at any kind of data. I'm very very grateful about just this past > > week's revelations. > > > > Murray > > > > ----- Original Message ----- From: "Rich Signell" <address@hidden> > > To: "Charles Sun" <address@hidden> > > Cc: "Brown, Murray" <address@hidden>; > > <address@hidden>; <address@hidden>; "Reed, Greg" > > <address@hidden> > > Sent: Sunday, March 18, 2012 10:02 AM > > Subject: Re: PC issues > > > > > > Gang, > > > > With the new CF Discrete Sampling Geometry featureTypes implemented in > > NetCDF-Java Common Data Model, you can save many profiles in a single > > file, rather than separate files. For example: > > > > http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8372832 > > > > > > But I don't know if IDV can read these files and do anything useful > > with them yet. > > > > -Rich This is the best solution for this kind of dataset, and the IDV can certainly handle it better in this approach. Yuan > > > > > > On Sun, Mar 18, 2012 at 9:29 AM, Charles Sun <address@hidden> > > wrote: > >> Hi Murray, > >> > >> It is a typical issue for using IDV or IDV-like applications for reading > >> large number of small size files, either from local hard drive or over > >> Internet. > >> > >> One other maybe unrelated issue that I would like to bring to your > >> attention > >> is a jar file, called gtspp.jar (attached) suggested by Yuen for > >> using IDV > >> with the GTSPP data, has two files, datasource.xml and gstpp.ncml. I > >> don't > >> know too much about IDV. Should the filename, gstpp.ncml, be > >> gtspp.ncml or > >> it does not matter? > >> > >> Regards, > >> > >> Charles > >> > >> On 03/18/2012 07:38 AM, Brown, Murray wrote: > >>> > >>> Dear Folks, > >>> > >>> Saturday morning I began to experience some very strange PC behavior > >>> (no, > >>> I didn't suddenly become a Republican). My laptop, admittedly an > >>> ancient > >>> beast (5 yrs, a Compaq) started running extremely slowly and just > >>> died on > >>> big programs. With no online shopping, wild browsing or other usual > >>> suspects for infection. Checking my memory usage I found that one > >>> instance > >>> of the notorious svchost.exe was steadily growing by the minute, and > >>> had > >>> reached 1.2 Gb. Yep. Rebooting just started the growth cycle again, > >>> and I > >>> could see the file grow about 1 Mb per second with nothing running. > >>> If I > >>> stop that particular svchost.exe (there are usually several) then > >>> some other > >>> stuff can run, but not IDV. It seems, and the Microsoft technical > >>> confirmed, that the runaway instance of svchost.exe is the one that > >>> usually > >>> gets associated with IDV (a little Google searching turned up that > >>> svchost.exe is the controller for dll's called by programs, among other > >>> functions). [Other instances of svchost.exe remain unaffected and > >>> seemingly > >>> benign.] But it starts the weird behavior always, even when IDV isn't > >>> started at all. I'm nuts, my PC is toast, or something weird was > >>> alive and > >>> well in the resources. > >>> > >>> The MS technician said they could clean svchost.exe, but they didn't > >>> know > >>> what was wrong with it and no virus checker (they used 3) could > >>> identify > >>> what it was. "It can be your old PC, or it can be the files you've > >>> worked > >>> with, but we just don't know." I've been working only with GTSPP > >>> files and > >>> IDV for the past 4 days or so (including the new jar), as you've all > >>> seen. > >>> But I'm up and running now, and the main svchost.exe is remaining > >>> stable at > >>> 28Kb. Bullet dodged. > >>> > >>> CHARLES: Sorry I sent that note about your servers, because now I > >>> realize > >>> it could have been the very beginning of my troubles. IDV loads > >>> http://data.nodc.noaa.gov/thredds/catalog/gtspp/realtime/2012/catalog.xml > >>> in > >>> about 2 mins. now and all seems fine. > >>> > >>> Murray > >>> > >>> > >>> ----- Original Message ----- From: "Unidata IDV Support" > >>> <address@hidden> > >>> To: <address@hidden> > >>> Cc: <address@hidden>; <address@hidden>; > >>> <address@hidden>; <address@hidden>; > >>> <address@hidden> > >>> Sent: Friday, March 16, 2012 4:12 PM > >>> Subject: [IDV #ILT-566886]: vertical profiles with depth, not altitude > >>> > >>> > >>>>> Wow. I am so grateful. But could you change the code/name so it is > >>>>> GTSPP? > >>>>> > >>>> > >>>> See the attached. > >>>> > >>>> Y > >>>>> > >>>>> ----- Original Message ----- > >>>>> From: "Unidata IDV Support" <address@hidden> > >>>>> To: <address@hidden> > >>>>> Cc: <address@hidden>; <address@hidden>; > >>>>> <address@hidden>; > >>>>> <address@hidden>; <address@hidden>; > >>>>> <address@hidden> > >>>>> Sent: Friday, March 16, 2012 1:11 PM > >>>>> Subject: [IDV #ILT-566886]: vertical profiles with depth, not > >>>>> altitude > >>>>> > >>>>> > >>>>> >> Yuan, > >>>>> >> > >>>>> >> I am aware of it. The typo was in my format conversion program > >>>>> to >> > >>>>> >> create > >>>>> >> the physical files. > >>>>> >> > >>>>> >> By the way, all files in the > >>>>> >> http://data.nodc.noaa.gov/gtspp/realtime/2012/03 are the > >>>>> corrected >> >> > >>>>> >> ones. > >>>>> >> > >>>>> >> Thanks. > >>>>> >> > >>>>> >> Charles > >>>>> >> On 3/16/2012 12:50 PM, Unidata IDV Support wrote: > >>>>> >> >> Yuan, > >>>>> >> >> > >>>>> >> >> This is Charles Sun from US National Oceanographic Data Center. > >>>>> >> >> > >>>>> >> >> We are replacing the old GTSPP netcdf conventions, currently > >>>>> >> >> >> >> > >>>>> >> >> located > >>>>> >> >> in > >>>>> >> >> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/, by moving all > >>>>> >> >> files > >>>>> >> >> below the folder, netcdf4.0, one level up. This will not > >>>>> happen >> >> >> > >>>>> >> >> >> till > >>>>> >> >> late next week. > >>>>> >> >> > >>>>> >> >> I will suggest use > >>>>> >> >> > >>>>> >> >> > >>>>> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/netcdf4.0/indian/2012/01/gtspp_13237135_te_111.nc > >>>>> >> >> for testing purpose. In addition, please be advised that a > >>>>> typo >> >> in > >>>>> >> >> >> >> the > >>>>> >> >> z-variable attribute was found by Rich, which causes the > >>>>> vertical > >>>>> >> >> profile displayed up-side down. > >>>>> >> > Charles, > >>>>> >> > I checked the physical file, and the typo is in the file. Not > >>>>> >> > sure how you plan to fix it. > >>>>> >> > > >>>>> >> > > >>>>> >> > Yuan > >>>>> >> > > >>>>> >> > > >>>>> >> > Murray, > >>>>> >> > See the attached jar file, this is a plugin created for this > >>>>> >> > kind of typo files located locally or remotely. Please > >>>>> install it > >>>>> >> > >> > and > >>>>> >> > restart the IDV, when you load the dataset, you need to > >>>>> select the > >>>>> >> > >> > data > >>>>> >> > source type as "GSTPP data file". You should get what you > >>>>> wanted. > >>>>> >> > > >>>>> > > >>>>> > Sorry I forgot to add the attached file in my last email. > >>>>> > > >>>>> > > >>>>> > Yuan > >>>>> >> > > >>>>> >> > > >>>>> >> > Yuan > >>>>> >> >> However, the typo has been corrected, again all files will > >>>>> be >> > >>>>> >> >> >> updated > >>>>> >> >> by > >>>>> >> >> late next week. > >>>>> >> >> > >>>>> >> >> Sorry for any inconvenience. > >>>>> >> >> > >>>>> >> >> Regards, > >>>>> >> >> > >>>>> >> >> Charles > >>>>> >> >> On 3/16/2012 12:17 PM, Unidata IDV Support wrote: > >>>>> >> >>>> Yuan, > >>>>> >> >>>> > >>>>> >> >>>> Please take a look at the attached figure (#1), which > >>>>> doesn't >> >>>> >> > >>>>> >> >>>> >>>> seem > >>>>> >> >>>> like > >>>>> >> >>>> much, but I'm so proud of it and so thankful to you and > >>>>> Rich >> > >>>>> >> >>>> >>>> Signell > >>>>> >> >>>> for > >>>>> >> >>>> helping me with all the tricks involved. It is based on the > >>>>> >> >>>> "corrected" > >>>>> >> >>>> NetCDF profile that Charles Sun provided (consisting of > >>>>> only a > >>>>> >> >>>> >> >>>> few > >>>>> >> >>>> points). > >>>>> >> >>>> But eventually, we could make these color-coded profiles > >>>>> for >> > >>>>> >> >>>> >>>> groups > >>>>> >> >>>> of GTSPP > >>>>> >> >>>> stations, grabbed online through THREDDS catalogs. > >>>>> >> >>> Murray, > >>>>> >> >>> The physical file (13237135.nc) is corrected from the > >>>>> >> >>> beginning. The problem is the THREDDS server need to be > >>>>> updated > >>>>> >> >>> (I > >>>>> >> >>> will check with TDS developer to make sure). There are two > >>>>> URLs > >>>>> >> >>> >> >>> here > >>>>> >> >>> and confused me: > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> > >>>>> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/indian/2012/01/13237135.nc > >>>>> > >>>>> >> >>> > >>>>> >> >>> and > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> > >>>>> http://data.nodc.noaa.gov/thredds/dodsC/gtspp/netcdf4.0/indian/2012/01/gtspp_13237135_te_111.nc > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> Which one are you planning to use? > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> Yuan > >>>>> >> >>>> This will be, for us in the water world, a huge step, in > >>>>> terms > >>>>> >> >>>> of > >>>>> >> >>>> efficiency, potential for synthesis, and compatibility > >>>>> with >> >>>> some > >>>>> >> >>>> training > >>>>> >> >>>> we're already doing. The hope (watch out Charles, here > >>>>> comes my > >>>>> >> >>>> soap box) > >>>>> >> >>>> is that we'll have user-defined delivery of spatially and >> > >>>>> >> >>>> >>>> temporally > >>>>> >> >>>> selected individual NetCDF profiles. When the files have the > >>>>> >> >>>> "positive" > >>>>> >> >>>> syntax correction, then the use of 0 to -160 depths gives > >>>>> >> >>>> really > >>>>> >> >>>> FINE > >>>>> >> >>>> figures. [I'm using Ocean Data View for mega-collections > >>>>> based > >>>>> >> >>>> >> >>>> on > >>>>> >> >>>> Charles' > >>>>> >> >>>> TGZ compressions...that's a different matter.] > >>>>> >> >>>> > >>>>> >> >>>> The second figure is one of the existing NC profiles on the > >>>>> >> >>>> GTSPP > >>>>> >> >>>> server, to > >>>>> >> >>>> show you how much better they begin to look with more data >> > >>>>> >> >>>> >>>> points. > >>>>> >> >>>> That > >>>>> >> >>>> figure comes directly from IDV/THREDDS, one click, without > >>>>> any > >>>>> >> >>>> >> >>>> ncml > >>>>> >> >>>> fix. So > >>>>> >> >>>> it's still upside down. I'll need some IDV help on applying a > >>>>> >> >>>> layout that > >>>>> >> >>>> makes the dots larger (perhaps scaled to temp); they ARE > >>>>> >> >> >>>> >>>> > >>>>> >> >>>> currently > >>>>> >> >>>> colored > >>>>> >> >>>> by temp, but it's not easy to see on these small figs. > >>>>> >> >>>> > >>>>> >> >>>> I really hope this dialogue, made necessary by my > >>>>> blundering >> > >>>>> >> >>>> >>>> around, > >>>>> >> >>>> might > >>>>> >> >>>> continue and lead to good IDV/UNIDATA-GTSPP/NODC discussions. > >>>>> >> >>>> It > >>>>> >> >>>> seems to > >>>>> >> >>>> me that a lot could be accomplished. > >>>>> >> >>>> > >>>>> >> >>>> Murray > >>>>> >> >>>> > >>>>> >> >>>> > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> Ticket Details > >>>>> >> >>> =================== > >>>>> >> >>> Ticket ID: ILT-566886 > >>>>> >> >>> Department: Support IDV > >>>>> >> >>> Priority: Normal > >>>>> >> >>> Status: Open > >>>>> >> >>> > >>>>> >> >> > >>>>> >> > > >>>>> >> > Ticket Details > >>>>> >> > =================== > >>>>> >> > Ticket ID: ILT-566886 > >>>>> >> > Department: Support IDV > >>>>> >> > Priority: Normal > >>>>> >> > Status: Closed > >>>>> >> > > >>>>> >> > >>>>> >> > >>>>> > > >>>>> > > >>>>> > Ticket Details > >>>>> > =================== > >>>>> > Ticket ID: ILT-566886 > >>>>> > Department: Support IDV > >>>>> > Priority: Normal > >>>>> > Status: Closed > >>>>> > >>>>> > >>>> > >>>> > >>>> Ticket Details > >>>> =================== > >>>> Ticket ID: ILT-566886 > >>>> Department: Support IDV > >>>> Priority: Normal > >>>> Status: Closed > >>> > >>> > >> > > > > > > > > Ticket Details =================== Ticket ID: ILT-566886 Department: Support IDV Priority: Normal Status: Closed