[thredds] Mining Thredds Logs To Characterize Data Usage

• To: "thredds@xxxxxxxxxxxxxxxx" <thredds@xxxxxxxxxxxxxxxx>
• Subject: [thredds] Mining Thredds Logs To Characterize Data Usage
• From: "Braun, Jeremy E ERDC-RDE-CHL-MS CIV" <Jeremy.E.Braun@xxxxxxxxxxxxx>
• Date: Mon, 6 May 2024 19:51:34 +0000

Hello,

My colleague and I work at a Field Research Facility in Duck, NC and collect a 
variety of Real-Time Oceanographic data that are publicly served via a Thredds 
server. We have been exploring the possibility of quantifying our data usage by 
characterizing things like how many data requests we get, which data records 
are accessed most, etc. We've started exploring the logs on our Thredds server 
and found where these requests are logged in the threddsServlet logs along with 
the time, remote host IP, and a process ID. 

For example:
        2024-03-19T00:12:19.445 -0500 [  35301761][    5849] INFO  - 
threddsServlet - Remote host: 127.0.0.1 - Request: "GET 
/thredds/dodsC/frf/oceanography/waves/waverider-17m/waverider-17m.ncml.dds 
HTTP/1.0"
        2024-03-19T00:12:19.447 -0500 [  35301763][    5849] INFO  - 
threddsServlet - Request Completed - 200 - -1 - 2
        
We are posting here to see if anyone has experience mining info in the logs to 
characterize data usage and if we are on the right track looking in the 
threddsServlet logs. This seems like something that has probably been done 
before so we wanted to reach out to the community to see if anyone has 
developed tools, or knows of a good way, to query the threddsServlet files or 
any other files that might include the type of data we are interested in.

Thanks in advance for the help.

Jeremy Braun
------------------
Jeremy E. Braun | Data Scientist | USACE Engineer Research and Development 
Center 
Coastal and Hydraulics Laboratory Field Research Facility | 1261 Duck Rd, Duck, 
NC 27949 
E: jeremy.e.braun@xxxxxxxxxxxxx  or jeremy.e.braun@xxxxxxxxxxxxxx | P: (203) 
675-5930 

• Follow-Ups:
  • Re: [thredds] Mining Thredds Logs To Characterize Data Usage
    • From: Jim Fluke