[netcdfgroup] NetCDF NC_CHAR file double the size of ASCII file

• To: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] NetCDF NC_CHAR file double the size of ASCII file
• From: Timothy Stitt <Timothy.Stitt.9@xxxxxx>
• Date: Mon, 19 May 2014 10:58:49 -0400

Hi all,

I’ve been trying to convert a large (40GB) ASCII text file (composed of 
multiple records of 4 line ASCII strings about 90 characters long) into NetCDF 
format. My plan was to rewrite the original serial code to use parallel NetCDF 
to have many MPI processes concurrently read records and process them in 
parallel.

I was able to write some code to convert the ASCII records into 
[unlimited][4][90] NetCDF NC_CHAR arrays, which I was able to read concurrently 
via parallel NetCDF routines. My question is related to the size of the 
converted NetCDF file.

I notice that the converted NetCDF file is always double the size of the ASCII 
file whereas I was hoping for it be to much reduced. I was therefore wondering 
if this is expected or is more due to my bad representation in NetCDF of the 
ASCII records? I am using nc_put_vara_text() to write my records. Maybe I need 
to introduce compression that I’m not doing already?

Thanks in advance for any advice you can provide.

Regards,

Tim.

• References:
  • [netcdfgroup] NetCDF NC_CHAR file double the size of ASCII file
    • From: Timothy Stitt