Calling and executable with arguements and getting its STDOUT more efficiently in a Python Script

Question

I'm writing a script that takes longitude and latitude and runs them through an executable called gdallocationinfo. The executable takes the latitude and longitude as its arguments and returns its STDOUT as the value for that coordinate. I've been reading about sub-processes and I was wondering if this is the most efficient way to implement this if I want to run alot of points. It seems to be taking a really long time.

def gdalgetpointdata(longitude,latitude):
    proc = subprocess.Popen(["gdallocationinfo","C:\Users\data\pop","-wgs84","-valonly","{0}".format(longitude),"{0}".format(latitude)], stdout=subprocess.PIPE, shell=True)
    (out, err) = proc.communicate()
    return int(out)

Is there a better way to call this executable without having to make a new sub-process every time I run my function? Is there something I could do to speed it up?

As a side note I know that if you run the executable from the command line it will keep accepting STDIN and giving ouputs in STDOUT until you tell it to quit()

Answer 1

The utility can do multiple coordinates with one call. For example, prepare a simple coords.txt text file with coordinate pairs:

1.0 2.0
3.0 4.0
5.0 6.0

Then pipe it in and out of gdallocationinfo from an OSGeo4W shell:

gdallocationinfo -wgs84 -valonly raster.tif < coords.txt > values.txt

which will make a values.txt file with the value for each coordinate. You can do the same with Popen with PIPE stdin and stdout arguments.

coords = [(1.0, 2.0), (3.0, 4.0), (5.0, 6.0)]
coord_txt = ''.join(['{0} {1}\n'.format(*c) for c in coords])
p = subprocess.Popen(
    ['gdallocationinfo', r'C:\path\to\raster.tif', '-wgs84', '-valonly'],
    stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True, shell=True)
values_txt, err = p.communicate(coord_txt)
values = values_txt.splitlines()

values will be a list of values, the same length as coords.

Answer 2

Here's a version of @Mike T's answer that passes input, reads output dynamically (not tested):

#!/usr/bin/env python
from __future__ import print_function
from subprocess import Popen, PIPE
from threading import Thread

def pump_input(pipe, coords):
    with pipe:
        for longitude, latitude in coords:
            print(longitude, latitude, file=pipe)

p = Popen(['gdallocationinfo', r'C:\path\to\raster.tif', '-wgs84', '-valonly'],
          shell=True, #NOTE: don't use a list argument with shell=True on POSIX
          stdin=PIPE, stdout=PIPE, bufsize=-1,
          universal_newlines=True)
Thread(target=pump_input, args=[p.stdin, coords]).start()
with p.stdout:
    for line in p.stdout:
        print(int(line))
if p.wait() != 0:
   raise Error

It might be more efficient to use GDAL's Python bindings instead.