Re: [thredds] Mining Thredds Logs To Characterize Data Usage

• To: Jim Fluke <james.fluke@xxxxxxxxxxxxx>
• Subject: Re: [thredds] Mining Thredds Logs To Characterize Data Usage
• From: John Caron <jcaron1129@xxxxxxxxx>
• Date: Fri, 21 Jun 2024 13:17:21 -0600

Have a look at this as a start for what you might want to do:

https://docs.unidata.ucar.edu/tds/5.4/userguide/using_the_tdsmonitor_tool.html

Regards, John

On Thu, Jun 20, 2024 at 3:28 PM Jim Fluke <james.fluke@xxxxxxxxxxxxx> wrote:

> Jeremy,
>
> We here at the CloudSat DPC are very interested in doing the same thing.
> But I see that you have not received any replies with information on how to
> do it. Have you found anything useful outside of this list?
>
> And we would like to include associating users with their data requests.
> Hopefully that will show up in the threddsServlet logs when I enable
> authentication, and I'll be trying that soon. Have you tried that yet? I'll
> let you know how it works for us.
>
> Thanks,
> Jim
>
> On 5/6/24 13:51, Braun, Jeremy E ERDC-RDE-CHL-MS CIV via thredds wrote:
>
> ** Caution: EXTERNAL Sender **
>
> Hello,
>
> My colleague and I work at a Field Research Facility in Duck, NC and collect 
> a variety of Real-Time Oceanographic data that are publicly served via a 
> Thredds server. We have been exploring the possibility of quantifying our 
> data usage by characterizing things like how many data requests we get, which 
> data records are accessed most, etc. We've started exploring the logs on our 
> Thredds server and found where these requests are logged in the 
> threddsServlet logs along with the time, remote host IP, and a process ID.
>
> For example:
>         2024-03-19T00:12:19.445 -0500 [  35301761][    5849] INFO  - 
> threddsServlet - Remote host: 127.0.0.1 - Request: "GET 
> /thredds/dodsC/frf/oceanography/waves/waverider-17m/waverider-17m.ncml.dds 
> HTTP/1.0"
>         2024-03-19T00:12:19.447 -0500 [  35301763][    5849] INFO  - 
> threddsServlet - Request Completed - 200 - -1 - 2
>
> We are posting here to see if anyone has experience mining info in the logs 
> to characterize data usage and if we are on the right track looking in the 
> threddsServlet logs. This seems like something that has probably been done 
> before so we wanted to reach out to the community to see if anyone has 
> developed tools, or knows of a good way, to query the threddsServlet files or 
> any other files that might include the type of data we are interested in.
>
> Thanks in advance for the help.
>
> Jeremy Braun
> ------------------
> Jeremy E. Braun | Data Scientist | USACE Engineer Research and Development 
> Center
> Coastal and Hydraulics Laboratory Field Research Facility | 1261 Duck Rd, 
> Duck, NC 27949
> E: jeremy.e.braun@xxxxxxxxxxxxx  or jeremy.e.braun@xxxxxxxxxxxxxx | P: (203) 
> 675-5930
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing listthredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&data=05%7C02%7CJames.Fluke%40colostate.edu%7Cf772747d37534bf05d1d08dc6e05f490%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C638506219068777410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=kATWD%2BERu2dU8QE30op1kbYzcFhW%2BWnyzNIeIpUSDLs%3D&reserved=0
>  <https://www.unidata.ucar.edu/mailing_lists/>
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
>

• References:
  • [thredds] Mining Thredds Logs To Characterize Data Usage
    • From: Braun, Jeremy E ERDC-RDE-CHL-MS CIV
  • Re: [thredds] Mining Thredds Logs To Characterize Data Usage
    • From: Jim Fluke