NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Alexa, Thanks for asking. Actually, I would be most interested in seeing the final NCML where you had to tweak the date/time stuff. Thanks again. -Julien > Are you interested in the rest of the HDF files (18 in total I think), and/or > the lightning data (as a csv)? -Alexa > > > > On Apr 5, 2016, at 8:16 AM, Unidata IDV Support <address@hidden> wrote: > > > > Excellent. Thank you. > > > >> Feel free to use this example. > >> > >> Best Regards, > >> Mike > >> > >> > >> Michael J. Pavolonis, PhD > >> Physical Scientist (NOAA/NESDIS) > >> Phone: +1.608.263.9597 > >> Fax: +1.608.262.5974 > >> address@hidden > >> > >> Duty Station and Mailing Address: > >> NOAA Cooperative Institute for Meteorological Satellite Studies > >> University of Wisconsin - Madison > >> 1225 W. Dayton St., Room 207 > >> Madison, WI 53706 USA > >> > >> > >> > >>> On Apr 4, 2016, at 1:57 PM, Alexa Van Eaton <address@hidden> wrote: > >>> > >>> Hi there, > >>> > >>> The person to ask about this would be Mike Pavolonis (Ccd), who created > >>> these brightness temperature files. Mike, could you read the message > >>> below and provide your thoughts? I’m attaching a screencast of what I’ve > >>> done with these files so far. The crosshairs are volcanic lighting > >>> detected by WWLLN, brought in through a point text data file. > >>> > >>> All the best > >>> > >>> Alexa > >>> > >>> <Kelut_MTSAT_Lightning.mov> > >>> > >>> > >>>> On Apr 4, 2016, at 11:41 AM, Unidata IDV Support <address@hidden> wrote: > >>>> > >>>> Alexa, > >>>> > >>>> Great! I am glad it worked. > >>>> > >>>> I was contemplating writing a blog entry for our Unidata Developer Blog > >>>> on this > >>>> experience as it makes a good case study. Would it be OK if I used your > >>>> data/ncml for this purpose? It is no problem if you would prefer not to. > >>>> > >>>> Best, > >>>> > >>>> Unidata IDV Support > >>>> > >>>>> Dear IDV Support (and Sean), > >>>>> > >>>>> This works perfectly. I realized the file naming scheme was actually > >>>>> actually giving year + day of the year, so I adjusted the > >>>>> dateFormatMark using the syntax in this example > >>>>> http://docs.opendap.org/index.php/Dynamic_Aggregation_Tutorial and now > >>>>> the dates/times are being read accurately. > >>>>> > >>>>> As a side note, I noticed that Projections—>Enable Adaptive Resolution > >>>>> is on by default in the IDV, which automatically subsamples the data. > >>>>> But once that function was unchecked and the data reloaded, everything > >>>>> looks good. Big thanks to the both of you. > >>>>> > >>>>> All the best > >>>>> > >>>>> Alexa > >>>>> > >>>>> > >>>>>> On Apr 1, 2016, at 12:06 PM, Unidata IDV Support <address@hidden> > >>>>>> wrote: > >>>>>> > >>>>>> Hi Alexa, > >>>>>> > >>>>>> What I was describing below had nothing to do with the problem. Please > >>>>>> see the > >>>>>> attached NCML. This file should correctly aggregate the variables over > >>>>>> the time dimension, and also it will extract the times from the file > >>>>>> name, so > >>>>>> you don't have to manually describe them. > >>>>>> > >>>>>> Let us know how it goes. > >>>>>> > >>>>>> Best, > >>>>>> > >>>>>> Unidata IDV Support > >>>>>> > >>>>>> PS Thanks to my colleague Sean Arms for the NCML lessons :-) > >>>>>> > >>>>>>> Sure thing, I’ll have a play and keep you posted. All the best, Alexa > >>>>>>> > >>>>>>> > >>>>>>>> On Mar 31, 2016, at 4:21 PM, Unidata IDV Support <address@hidden> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>> Alexa, > >>>>>>>> > >>>>>>>> I think I know what the problem is, but don't have time to try > >>>>>>>> before I go home. > >>>>>>>> > >>>>>>>> I think the problem is that we have only changed one variable when > >>>>>>>> we should > >>>>>>>> have changed them all, at least all those that are listed under the > >>>>>>>> Field > >>>>>>>> Selector. The IDV may be getting confused on this. > >>>>>>>> > >>>>>>>> Do you want to give that a shot and let me know what happens? > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> > >>>>>>>> Unidata IDV Support > >>>>>>>> > >>>>>>>>> Alexa, > >>>>>>>>> > >>>>>>>>> Yes, thanks for the clarification. (I think the ordering of our > >>>>>>>>> messages got > >>>>>>>>> jumbled there.) > >>>>>>>>> > >>>>>>>>> I see what you are saying, and when I load each individual file > >>>>>>>>> separately I can > >>>>>>>>> see the data are definitely not the same so something is amiss. > >>>>>>>>> We'll resume > >>>>>>>>> tomorrow. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> > >>>>>>>>> Unidata IDV Support > >>>>>>>>> > >>>>>>>>>> Hi there — thank you for for your reply. Using your syntax below, > >>>>>>>>>> I tried aggregating three files (edited script attached) — > >>>>>>>>>> different files this time. Again, the files are read into IDV with > >>>>>>>>>> the correct time stamps, and I can toggle between the times, but > >>>>>>>>>> the strange thing is the plot does not change with time. It’s like > >>>>>>>>>> it’s only plotting one time. -Alexa > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On Mar 31, 2016, at 3:46 PM, Unidata IDV Support <address@hidden> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Alexa, > >>>>>>>>>>> > >>>>>>>>>>> Great! Glad we are making progress and also that you are hanging > >>>>>>>>>>> in there. This > >>>>>>>>>>> stuff is not so easy. > >>>>>>>>>>> > >>>>>>>>>>> Replace the current values element with something like this: > >>>>>>>>>>> > >>>>>>>>>>> <values>0 3 11</values> > >>>>>>>>>>> > >>>>>>>>>>> But if that is too onerous because you have many file, we will > >>>>>>>>>>> have to go via > >>>>>>>>>>> the time extraction route. Let me know... > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> > >>>>>>>>>>> Unidata IDV Support > >>>>>>>>>>> > >>>>>>>>>>>> Hello — this looks great, and I appreciate your help with this. > >>>>>>>>>>>> Your ncml script worked for the two test files and the times > >>>>>>>>>>>> look right. However, since we have some missing satellite > >>>>>>>>>>>> images, not all the files are spaced exactly 10 mins apart. Is > >>>>>>>>>>>> there a way to specify time gaps between individual files begin > >>>>>>>>>>>> aggregated? Otherwise, perhaps extracting the time from the file > >>>>>>>>>>>> name would be good route. -Alexa > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> On Mar 31, 2016, at 3:29 PM, Unidata IDV Support > >>>>>>>>>>>>> <address@hidden> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi again, > >>>>>>>>>>>>> > >>>>>>>>>>>>> The embedded NcML is a bit hard to read so attached, you will > >>>>>>>>>>>>> find > >>>>>>>>>>>>> that same NcML file. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Unidata IDV Support > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Alexa, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> It is a little tricky but you can fix these problems with a > >>>>>>>>>>>>>> bit of > >>>>>>>>>>>>>> NcML. To get us going, let's look at the following NcML which > >>>>>>>>>>>>>> can be > >>>>>>>>>>>>>> read into the IDV and will allow you to view the > >>>>>>>>>>>>>> `channel_14_brightness_temperature' over time. This file > >>>>>>>>>>>>>> should be > >>>>>>>>>>>>>> located next to your `hdf' files. Please read the comments in > >>>>>>>>>>>>>> the NcML > >>>>>>>>>>>>>> for more information about what is going on here. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> ,---- > >>>>>>>>>>>>>> | <?xml version="1.0" encoding="UTF-8"?> > >>>>>>>>>>>>>> | <netcdf > >>>>>>>>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> > >>>>>>>>>>>>>> | <!-- Create a time coordinate variable that will be used > >>>>>>>>>>>>>> for the aggregation. --> > >>>>>>>>>>>>>> | <variable name="time" type="int" shape="time" > > >>>>>>>>>>>>>> | <!-- Let's add some units. Alexa: please make sure these > >>>>>>>>>>>>>> units are correct. --> > >>>>>>>>>>>>>> | <attribute name="units" value="minutes since 2014-04-04 > >>>>>>>>>>>>>> 19:09"/> > >>>>>>>>>>>>>> | <!-- Specify this variable as a time coordinate --> > >>>>>>>>>>>>>> | <attribute name="_CoordinateAxisType" value="Time" /> > >>>>>>>>>>>>>> | <!-- Increment each file listed below by 10 minutes. > >>>>>>>>>>>>>> Alexa, please make sure this --> > >>>>>>>>>>>>>> | <!-- assumption about the times is correct. --> > >>>>>>>>>>>>>> | <values start="0" increment="10" /> > >>>>>>>>>>>>>> | </variable> > >>>>>>>>>>>>>> | <!-- Specify the coordinates of time lat lon for this > >>>>>>>>>>>>>> one variable. Please use --> > >>>>>>>>>>>>>> | <!-- this as an example for other variables you are > >>>>>>>>>>>>>> interersted in. In other --> > >>>>>>>>>>>>>> | <!-- words, you can copy and paste this variable NcML > >>>>>>>>>>>>>> element for your other --> > >>>>>>>>>>>>>> | <!-- variables. --> > >>>>>>>>>>>>>> | <variable name="channel_14_brightness_temperature" > >>>>>>>>>>>>>> shape="lines elements" type="short"> > >>>>>>>>>>>>>> | <attribute name="coordinates" value="time pixel_latitude > >>>>>>>>>>>>>> pixel_longitude" /> > >>>>>>>>>>>>>> | </variable> > >>>>>>>>>>>>>> | <!-- Now you are ready to do your aggregation, which is > >>>>>>>>>>>>>> the easy part at this point --> > >>>>>>>>>>>>>> | <aggregation dimName="time" type="joinNew"> > >>>>>>>>>>>>>> | <variableAgg name="T"/> > >>>>>>>>>>>>>> | <netcdf location="geocatL1.MTSAT-1R.2014044.190900.hdf"/> > >>>>>>>>>>>>>> | <netcdf location="geocatL1.MTSAT-1R.2014044.191900.hdf"/> > >>>>>>>>>>>>>> | </aggregation> > >>>>>>>>>>>>>> | </netcdf> > >>>>>>>>>>>>>> `---- > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In summary, this NcML should aggregate over time the two of > >>>>>>>>>>>>>> HDF files > >>>>>>>>>>>>>> you provided. You will see only one variable that is > >>>>>>>>>>>>>> aggregated over > >>>>>>>>>>>>>> time; `channel_14_brightness_temperature'. I'll leave it up > >>>>>>>>>>>>>> to you to > >>>>>>>>>>>>>> do the others though do not hesitate to ask for help. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Also, if interested, there are also fancier tricks we can do > >>>>>>>>>>>>>> here such > >>>>>>>>>>>>>> as NcML can be smart enough to extract the times from the file > >>>>>>>>>>>>>> names. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Unidata IDV Support > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Ok sounds promising. Here is the link to another file, which > >>>>>>>>>>>>>>> should differ only in time — > >>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRDVyczU3N2NMbFU/view?usp=sharing > >>>>>>>>>>>>>>> -Alexa > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Mar 30, 2016, at 4:49 PM, Unidata IDV Support > >>>>>>>>>>>>>>>> <address@hidden> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Alexa, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Indeed we may be able to patch things up with a bit of ncml. > >>>>>>>>>>>>>>>> I only found one > >>>>>>>>>>>>>>>> file at the drive.google location. Would it be possible to > >>>>>>>>>>>>>>>> see more so that we > >>>>>>>>>>>>>>>> have something to aggregate? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Unidata IDV Support > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hello IDV Support, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I am working on reading in a number of HDF files using > >>>>>>>>>>>>>>>>> ‘aggregate grids by time’ or similar. I am able to read the > >>>>>>>>>>>>>>>>> files in individually without a problem, but it looks like > >>>>>>>>>>>>>>>>> the time variable is not recognized. The error message is > >>>>>>>>>>>>>>>>> "Grid data source failed making data set….Dimension time > >>>>>>>>>>>>>>>>> does not exist”. Any ideas for how to specify the time > >>>>>>>>>>>>>>>>> variable? Could this be achieved with an .ncml script? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Example file is located here: > >>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1jVlP9ZfAZCRFhkRVM2aVVRVGc/view?usp=sharing > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> All the best > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Alexa > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Ticket Details > >>>>>>>>>>>>>>>> =================== > >>>>>>>>>>>>>>>> Ticket ID: LPW-471935 > >>>>>>>>>>>>>>>> Department: Support IDV > >>>>>>>>>>>>>>>> Priority: Normal > >>>>>>>>>>>>>>>> Status: Closed > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Ticket Details > >>>>>>>>>>>>> =================== > >>>>>>>>>>>>> Ticket ID: LPW-471935 > >>>>>>>>>>>>> Department: Support IDV > >>>>>>>>>>>>> Priority: Normal > >>>>>>>>>>>>> Status: Closed<alexa.ncml> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Ticket Details > >>>>>>>>>>> =================== > >>>>>>>>>>> Ticket ID: LPW-471935 > >>>>>>>>>>> Department: Support IDV > >>>>>>>>>>> Priority: Normal > >>>>>>>>>>> Status: Closed > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Ticket Details > >>>>>>>> =================== > >>>>>>>> Ticket ID: LPW-471935 > >>>>>>>> Department: Support IDV > >>>>>>>> Priority: Normal > >>>>>>>> Status: Open > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> Ticket Details > >>>>>> =================== > >>>>>> Ticket ID: LPW-471935 > >>>>>> Department: Support IDV > >>>>>> Priority: Normal > >>>>>> Status: Closed<alexa.ncml> > >>>>> > >>>>> > >>>> > >>>> > >>>> Ticket Details > >>>> =================== > >>>> Ticket ID: LPW-471935 > >>>> Department: Support IDV > >>>> Priority: Normal > >>>> Status: Closed > >>>> > >>> > >> > >> > > > > > > Ticket Details > > =================== > > Ticket ID: LPW-471935 > > Department: Support IDV > > Priority: Normal > > Status: Closed > > > > Ticket Details =================== Ticket ID: LPW-471935 Department: Support IDV Priority: Normal Status: Closed