Re: [python-users] Trying to save case data for replay or later visualization

• To: Bryan Guarente <guarente@xxxxxxxx>
• Subject: Re: [python-users] Trying to save case data for replay or later visualization
• From: Ryan May <rmay@xxxxxxxx>
• Date: Wed, 9 Oct 2019 21:29:53 -0600

Bryan,

So this *should* be as easy as:

from siphon.catalog import TDSCatalog
cat = TDSCatalog('
http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_onedeg/catalog.xml
')
cat.latest.download()

However, because I was foolish, if you try to do this with a big file (i.e.
more than your memory), you will run out of memory because it tries to
download all the bytes first and then writes them to the file. As a
workaround, you can use this function (from here:
https://stackoverflow.com/questions/16694907/download-large-file-in-python-with-requests
):

import requests
import shutil

def download_file(url):
    local_filename = url.split('/')[-1]
    with requests.get(url, stream=True) as r:
        with open(local_filename, 'wb') as f:
            shutil.copyfileobj(r.raw, f)

    return local_filename

and do:

download_file(cat.latest.access_urls['HTTPServer'])

This should be fixed in the upcoming Siphon 0.9 release.

Hope this helps,

Ryan

On Wed, Oct 9, 2019 at 3:02 PM Bryan Guarente <guarente@xxxxxxxx> wrote:

> Ryan,
> I am looking for a way to grab all of today's model data as grb files.  We
> had talked about grabbing the "latest" catalog and checking it every time a
> new model updates to download more data.  I thought siphon or even trying
> python-awips would work to get some of this done.  Pointing to an EDEX
> server seemed smart to me, but I am sure you have different ideas.
>
> Thanks.
> Bryan
> --
> Bryan Guarente
> Meteorologist/Instructional Designer
> The COMET Program
> Boulder, CO
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> python-users mailing list
> python-users@xxxxxxxxxxxxxxxx
> For list information, to unsubscribe, or change your membership options,
> visit: https://www.unidata.ucar.edu/mailing_lists/
>


-- 
Ryan May, Ph.D.
Software Engineer
UCAR/Unidata
Boulder, CO

• References:
  • [python-users] Trying to save case data for replay or later visualization
    • From: Bryan Guarente