[python-users] Questions about THREDDS Data Server

• To: "python-users@xxxxxxxxxxxxxxxx" <python-users@xxxxxxxxxxxxxxxx>
• Subject: [python-users] Questions about THREDDS Data Server
• From: "Keem, Munsung" <munsung-keem@xxxxxxxxx>
• Date: Thu, 11 Feb 2016 22:50:56 +0000
• Authentication-results: unidata.ucar.edu; dkim=none (message not signed) header.d=none; unidata.ucar.edu; dmarc=none action=none header.from=uiowa.edu;
• Spamdiagnosticmetadata: NSPM
• Spamdiagnosticoutput: 1:23

Dear Ryan May

Good afternoon.

Thank you very much for your help, Ryan.

Thanks to your help, we can test several things about the availability of 
THREDDS Data Server and AWS to our rainfall processing system.

However, while testing them, other questions arose to me.

Now, our rainfall processing algorithms which are developed  fully with c 
language have used the LEVEL II data obtained from NCDC, and

read and process them based on NASA's rsl c library. Due to this, LEVEL II data 
structure and conventions in our programs are the same as those defined in rsl 
library.

Since I realized the advantages of TDS, I have tried to find out a way to 
access LEVEL II data through TDS and process them with our algorithms written 
in C without downloads of the data.

However, the data structures offered by TDS (OpenDAP and CdmRemote) look 
different from LEVEL II data from NCDC; I mean it is not readable using rsl 
library.

I think the data offered by HTTPSever have the same structure as those from 
NCDC, but it can only be downloaded, not accessed without downloads. Am I right?

So, my question is,

1.       Is there any way to read and process LEVEL II data using rls c library 
in c program through TDS without downloads of the data?
As I know, if we use OpenDAP c API, it is possible to access LEVEL II data on 
OpendDAP server through c program.
However, in this case, as I said, data structure looks different from raw LEVEL 
II data, so we need to modify our algorithms a lot since our algorithms are 
developed based on rsl c library.
The most ideal case for us is that we access raw LEVEL II data through TDS, 
read the data using rsl c library, process them using our c programs, and 
download the final result file.
I am wondering whether or not this method is possible. Could you share your 
expertise about this problem?


2.       If this way is impossible, another possible way may be to use python 
to read LEVEL II data through TDS, and connect them to c programs.

This way is similar to the way to use OpenDAP c API, but in this case, all 
necessary programs are already developed.
Also, as I know, there is a way to deliver python objects to c program. Am I 
right?
However, I question the execution speed of this method.
I think simply downloading through TDS and processing LEVEL II data with our c 
programs could be faster, but I cannot assure it. Could you tell me how you 
think of this way?
(For this issue, I tested the reading speed using pyart packages. When I access 
LEVEL II data through TDS and read them through pyart.io.read_nexrad_cdm,
It takes more time than simply downloading)


3.       Finally, I would like to know when I download LEVEL II data through 
TDS into AWS EC2 instance, is there any cost for downloading data regardless of 
Availability Zone and Region?
I wonder the difference between using TDS and AWS S3 for downloads of the data. 
In the case AWS S3, if my EC2 instance is located in different Zone or Region, 
some cost is charged.
However, I am not sure about TDS.


I am sorry to list too many things, but if you give me your advice about my 
questions, it should be really helpful for me.

Thank you for reading this email and for considering the contents.



Thank you.


Best regards,


Munsung Keem

IIHR-Hydroscience & Engineering
Department of Civil & Environmental Engineering
The University of Iowa

• Follow-Ups:
  • Re: [python-users] Questions about THREDDS Data Server
    • From: Ryan May