Concurrent.futures - Returns error that imported module is not defined

1.1k views Asked by At

I am using concurrent.futures to multithread an app I am writing.

I start the app by importing IPAddress from netaddr:

from netaddr import IPNetwork, IPAddress

Next, I take some input files and I pass them all into my function being multithreaded:

with open(options.filename) as f:
    contents = f.readlines()
    executor = concurrent.futures.ProcessPoolExecutor(threads)
    futures = [executor.submit(ip_compare, ip, scope_list) for ip in contents]

Then I wait get the results as completed and append them to an output var:

for future in concurrent.futures.as_completed(futures):
    output.append(future.results()

The issue I have is that I keep getting the excetion from future:

global name 'IPAddress' is not defined

Here is the ip_compare function:

def ip_compare(ip_addr, scope_list):
    ip_addr = ip_addr.rstrip()
    if not is_ipv4(ip_addr):
        try:
            ip = socket.gethostbyname(ip_addr)
        except:
            return "error," + ip_addr + ",,," + str(sys.exc_info()[0]).replace(',',';') + "\r\n"
    else:
        ip = ip_addr
    for scope in scope_list:
        if IPAddress(ip) in IPNetwork(scope):
            return "in," + ip_addr + "," + ip + "," + scope + ",\r\n"
    return "out," + ip_addr + "," + ip + "," + ",,\r\n"

Any idea why futures is not recognizing the loaded module?

When my IDE stops the execution of the script because of the error, I can clearly see that IPAddress is defined in the memory:

IPAddress = {type} <class 'netaddr.ip.IPAddress'>
1

There are 1 answers

1
Sugitime On

Ok so the issue was that I was importing netaddr from within main:

if __name__=="__main__":
try:
    from netaddr import IPNetwork, IPAddress
except ImportError as error:
    print "Please install netaddr.\r\npip install netaddr\r\n\r\nIf pip is not installed, install pip\r\nhttps://pip.pypa.io/en/latest/installing.html"

I moved this to the top of the script and everything worked fine. I am curious as to why this worked though, if anyone can answer that.