Re: [thredds] Mining Thredds Logs To Characterize Data Usage

• To: thredds@xxxxxxxxxxxxxxxx
• Subject: Re: [thredds] Mining Thredds Logs To Characterize Data Usage
• From: Jim Fluke <james.fluke@xxxxxxxxxxxxx>
• Date: Thu, 20 Jun 2024 15:28:07 -0600
• Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=colostate.edu; dmarc=pass action=none header.from=colostate.edu; dkim=pass header.d=colostate.edu; arc=none
• Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TiKDm0yThEcOi9lXF9SjTaK8Zwjgz+rfHlNCSUXayog=; b=ApaBAd2wDPwvjUHUKMc/0vjstUIqU1NxpcZOHnIMli3qNfbEffGh6kff2RVJJh5QtcUxQd+uQ0+gHytORMutU1UbHeYhH6DSWnAT6OjPX/wj0Kynu7TbJeAwOAOWsySC4J2qX4qlEeVxeZrfQ1PNNvm+a35xWE40nSUsPU+erN47GXbsi/aOpjbL9aD0hvPFHT8xzkUgIcGbYPbM1Uk+uscFVo4qROQ+x5crMQK8dqjqcQL6nL7Xlw42oyyRBCFhmPKUkbFkUH887ESazVCwy3dsBbf2fL6WY5TJe/tqvR7AyAKtv5M3DLurg38BvxXcdl/foisRFqsbOIZ+vMwhOg==
• Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lz8ljWkAdrIFPt4INEPQReVzEIgf6u8Y7RhlGTfV3YQTXr8mLAgIpzlI4FNsAsfC0/C2CWXU1C3fqjMP0xXa/5/unyjOdfo5KnhhJWA0HPnUJLKvPlfC6pHO+fRJNxEs5TnyMuGFg7pa0mGeekFP4RVMu0b2h1jfCyqOCWqpMIdYCyzAaEeOTnfQmCOfCBHf5Ov45xILWxCNUjuTkzYycAtNV5WrVLajzrdlnZ2eYoYpxwxqGA670D4jsSX0Q5DcJfvDbDU1LBrWoPdTaZGLNRgS6iGVJ8bXWMaIwoJcxL9+SAbeF6QbTw4i0G2w2MifWzWbUD6SXN514I5A2YH3yA==
• Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=colostate.edu;

Jeremy,

Thanks,
Jim

On 5/6/24 13:51, Braun, Jeremy E ERDC-RDE-CHL-MS CIV via thredds wrote:

** Caution: EXTERNAL Sender **

Hello,

My colleague and I work at a Field Research Facility in Duck, NC and collect a 
variety of Real-Time Oceanographic data that are publicly served via a Thredds 
server. We have been exploring the possibility of quantifying our data usage by 
characterizing things like how many data requests we get, which data records 
are accessed most, etc. We've started exploring the logs on our Thredds server 
and found where these requests are logged in the threddsServlet logs along with 
the time, remote host IP, and a process ID.

For example:
         2024-03-19T00:12:19.445 -0500 [  35301761][    5849] INFO  - threddsServlet - 
Remote host: 127.0.0.1 - Request: "GET 
/thredds/dodsC/frf/oceanography/waves/waverider-17m/waverider-17m.ncml.dds HTTP/1.0"
         2024-03-19T00:12:19.447 -0500 [  35301763][    5849] INFO  - 
threddsServlet - Request Completed - 200 - -1 - 2

We are posting here to see if anyone has experience mining info in the logs to 
characterize data usage and if we are on the right track looking in the 
threddsServlet logs. This seems like something that has probably been done 
before so we wanted to reach out to the community to see if anyone has 
developed tools, or knows of a good way, to query the threddsServlet files or 
any other files that might include the type of data we are interested in.

Thanks in advance for the help.

Jeremy Braun
------------------
Jeremy E. Braun | Data Scientist | USACE Engineer Research and Development 
Center
Coastal and Hydraulics Laboratory Field Research Facility | 1261 Duck Rd, Duck, 
NC 27949
E:jeremy.e.braun@xxxxxxxxxxxxx   orjeremy.e.braun@xxxxxxxxxxxxxx  | P: (203) 
675-5930


_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  
visit:https://www.unidata.ucar.edu/mailing_lists/

• Follow-Ups:
  • Re: [thredds] Mining Thredds Logs To Characterize Data Usage
    • From: John Caron

• References:
  • [thredds] Mining Thredds Logs To Characterize Data Usage
    • From: Braun, Jeremy E ERDC-RDE-CHL-MS CIV