NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
> > > This is not a problem with pqact(1) *per se*, but with inefficient > decoders (pure Python, for example, is about three orders of magnitude > slower than C). > While Python is certainly slower than C (though I strongly question 1000x slower), I don't want anyone taking away from this that Python is a poor choice for an LDM decoder. The entire infrastructure ingesting *all* of the NEXRAD Level 2 feed into the Amazon S3 archive (and the realtime chunks bucket) is powered by a single pure Python process that takes 30% of a single CPU. We're also using Python to stitch together in realtime the GOES-16/17 imagery tiles being sent over NOAAPORT. These are not small data feeds. I'll note these applications are using long-running, persistent processes that have data continually piped in. Where Python becomes a problem is if you are starting up lots of individual processes, say one per product when you have 10s of products per second. In that case the startup/shutdown time of the Python interpreter will kill you. I tried this the other day with the NEXRAD3 feed and managed to get a machine up to a load average of 185. :) Ryan -- Ryan May, Ph.D. Software Engineer UCAR/Unidata Boulder, CO
ldm-users archives: