NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [netcdfgroup] How convert csv files to a netcdf file

  • To: Roy Mendelssohn - NOAA Federal <roy.mendelssohn@xxxxxxxx>
  • Subject: Re: [netcdfgroup] How convert csv files to a netcdf file
  • From: Bob Simons - NOAA Federal <bob.simons@xxxxxxxx>
  • Date: Tue, 15 Dec 2020 14:14:08 -0800
It is correct that NCCSV came from Steve Hankin's request for a way to
convert CSV files to .nc files. But NCCSV is not the solution to the
problem below because it is CSV-oriented: it is a format that supports
metadata + a CSV file in a way that survives editing in Excel (without
being changed) and can be converted (currently by ERDDAP only) into a .nc
file. It is not intended to be a 2-way system that would allow any .nc file
to be represented in a CSV file and vice versa. (That is nonsensical
because of the inherent nature of a standard CSV file. As Chris says, CSV
files are 1D.)

Roy and I have seen 2D CSV files like the one described below. We both
wrote little programs which read the file and made a .nc file from them. In
both cases, we added additional information based on our knowledge of what
was in the file, so they aren't general purpose tools.

I agree with Chris: this task is probably best done by writing a little
program in the language of your choice. I'm not aware of any tool to handle
it (although one might exist).

Best wishes.




On Tue, Dec 15, 2020 at 12:41 PM Roy Mendelssohn - NOAA Federal <
roy.mendelssohn@xxxxxxxx> wrote:

> A question for Bob.  He is copied on this.
>
> -Roy
>
> > On Dec 15, 2020, at 11:34 AM, Chris Barker <chris.barker@xxxxxxxx>
> wrote:
> >
> > Thanks Roy! I'm glad that got finished.
> >
> > I had a student back in the day get started on a Python library. that
> supported it -- but never finished it. Anyone know of software (other than
> ERDAP)
> > that supports NCCSV?
> >
> > As for NCO: I can't find any reference to "CSV" on the docs :-(
> >
> > One question: it seems to support only 1-D arrays -- is that the case?
> If so, then the use case that started this thread isn't supported.
> >
> > -CHB
> >
> >
> >
> >
> > On Tue, Dec 15, 2020 at 11:10 AM Roy Mendelssohn - NOAA Federal <
> roy.mendelssohn@xxxxxxxx> wrote:
> > Look at any recent ERDDAP server and ask for the .nccsv format such as
> in:
> >
> >
> https://coastwatch.pfeg.noaa.gov/erddap/griddap/erdMWchla1day.nccsv?chlorophyll[(2020-12-14T12:00:00Z):1:(2020-12-14T12:00:00Z)][(0.0):1:(0.0)%5D%5B(22.0):1:(51.0)][(205.0):1:(255.0)]
> >
> > I believe but don't want to be quoted on it that NCO has this also.
> More info on it can be found at:
> >
> > https://coastwatch.pfeg.noaa.gov/erddap/download/NCCSV.html
> >
> > -Roy
> >
> > > On Dec 15, 2020, at 10:54 AM, Chris Barker via netcdfgroup <
> netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
> > >
> > > Steve Hankin was looking at specifying a CSV format that would map to
> netCDF.
> > >
> > > I'm not sure what came of that.
> > >
> > > In the meantime, how to do this now depends on what tools you are
> familiar with:
> > >
> > > I would use Python (and probably xarray) to write a translator -- it
> would be quite straightforward if you are familiar with Python.
> > >
> > > Other options are:
> > > - Any number of other programming languages
> > > - some way to translate to CDL -- given that the meta data is similar
> for all this data, that could be not too hard.
> > >
> > > HTH,
> > > -CHB
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Dec 15, 2020 at 9:59 AM nazaniti6 <nazitkk1375@xxxxxxxxx>
> wrote:
> > > Hello,
> > > I have six CSV files which I have sent in this link
> > > (
> https://drive.google.com/drive/folders/1GQtyY1mI1YrK8GFP9SEo7dZfFHexKro3?usp=sharing
> ).
> > > If you open them you will understand that each of these files has 720
> rows
> > > and 360 columns which indicates longitude and latitude respectively. I
> > > should tell you that the first point of latitude is -89.75, the first
> point
> > > of longitude is -179.75, and the resolution of these data is 0.5
> degrees. I
> > > want to convert them to a NetCDF file which has the following
> properties:
> > > Dimensions:
> > > longitude = 720
> > > latitude = 360
> > > time = 6
> > > Variables:
> > > longitude
> > > Size: 720x1
> > > Dimensions: longitude
> > > Datatype: single
> > > Attributes:
> > > long_name = 'longitude'
> > > units = 'degrees_east'
> > > latitude
> > > Size: 360x1
> > > Dimensions: latitude
> > > Datatype: single
> > > Attributes:
> > > long_name = 'latitude'
> > > units = 'degrees_north'
> > > time
> > > Size: 6x1
> > > Dimensions: time
> > > Datatype: single
> > > Attributes:
> > > long_name = 'time'
> > > units = 'days since 1900-1-1'
> > > calendar = 'gregorian'
> > > tmp
> > > Size: 720x360x6
> > > Dimensions: longitude, latitude,time
> > > Datatype: single
> > > Attributes:
> > > long_name = 'monthly mean temperature'
> > > units = 'degrees Celsius'
> > > correlation_decay_distance = 1200
> > > _FillValue = 9.969209968386869e+36
> > > missing_value = 9.969209968386869e+36
> > > Could you please help me?
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Sent from: http://netcdf-group.1586084.n2.nabble.com/
> > >
> > > _______________________________________________
> > > NOTE: All exchanges posted to Unidata maintained email lists are
> > > recorded in the Unidata inquiry tracking system and made publicly
> > > available through the web.  Users who post to any of the lists we
> > > maintain are reminded to remove any personal information that they
> > > do not want to be made public.
> > >
> > >
> > > netcdfgroup mailing list
> > > netcdfgroup@xxxxxxxxxxxxxxxx
> > > For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
> > >
> > >
> > > --
> > >
> > > Christopher Barker, Ph.D.
> > > Oceanographer
> > >
> > > Emergency Response Division
> > > NOAA/NOS/OR&R            (206) 526-6959   voice
> > > 7600 Sand Point Way NE   (206) 526-6329   fax
> > > Seattle, WA  98115       (206) 526-6317   main reception
> > >
> > > Chris.Barker@xxxxxxxx
> > > _______________________________________________
> > > NOTE: All exchanges posted to Unidata maintained email lists are
> > > recorded in the Unidata inquiry tracking system and made publicly
> > > available through the web.  Users who post to any of the lists we
> > > maintain are reminded to remove any personal information that they
> > > do not want to be made public.
> > >
> > >
> > > netcdfgroup mailing list
> > > netcdfgroup@xxxxxxxxxxxxxxxx
> > > For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
> >
> > **********************
> > "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> > **********************
> > Roy Mendelssohn
> > Supervisory Operations Research Analyst
> > NOAA/NMFS
> > Environmental Research Division
> > Southwest Fisheries Science Center
> > ***Note new street address***
> > 110 McAllister Way
> > Santa Cruz, CA 95060
> > Phone: (831)-420-3666
> > Fax: (831) 420-3980
> > e-mail: Roy.Mendelssohn@xxxxxxxx www: https://www.pfeg.noaa.gov/
> >
> > "Old age and treachery will overcome youth and skill."
> > "From those who have been given much, much will be expected"
> > "the arc of the moral universe is long, but it bends toward justice"
> -MLK Jr.
> >
> >
> >
> > --
> >
> > Christopher Barker, Ph.D.
> > Oceanographer
> >
> > Emergency Response Division
> > NOAA/NOS/OR&R            (206) 526-6959   voice
> > 7600 Sand Point Way NE   (206) 526-6329   fax
> > Seattle, WA  98115       (206) 526-6317   main reception
> >
> > Chris.Barker@xxxxxxxx
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: https://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK
> Jr.
>
>
  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: