parameter error in python script & TOR proxy server

133 views Asked by At

I am Noob in Python. My Boss instructed me to run this python script with TOR proxy server running. He told me to pass this parameter this way: python DownloadYP.py /Users/myfolder/ japan http://www.jpyellow.com/company 1 222299

He configured it on MAC. I am using windows. So my parameter is this way: python DownloadYP.py C:\rrrb japan http://www.jpyellow.com/company 1 222299

But I am getting error:

> Traceback (most recent call last):
File "C:\Users\USER\yp1\code\DownloadYP.py", line 92, in <module>
WebPage(path, country, url, lowerlimit,upperlimit)
File "C:\Users\USER\yp1\code\DownloadYP.py", line 23, in __init__
fout = open(self.dir+"/limit.txt",'wb')
IOError: [Errno 2] No such file or directory: 'C:\\rrr/japan/limit.txt'

My Code is:

  • DownloadYP.py

    def __init__(self, path, country, url, lower=0,upper=9999):
        self.dir = str(path)+"/"+ str(country)
        self.url = url
        try:
          fin = open(self.dir+"/limit.txt",'r')
            limit = fin.readline()
            limits = str(limit).split(",")
            lower = int(limits[0])
            upper = int(limits[1])
            fin.close()
        except:
    

    this is line 23

              **fout = open(self.dir+"/limit.txt",'wb')**
            limits = str(lower)+","+str(upper)
            fout.write(limits)
            fout.close()  
        self.process_instances(lower,upper)
    
    
        def process_instances(self,lower,upper):
                try:
                    os.stat(self.dir)
                except:
                    os.mkdir(self.dir)
                for count in range(lower,upper+1):
                    if count == upper:
                        print "all downloaded, quitting the app!!"
                        break
                    targetURL = self.url+"/"+str(count)
                    print "Downloading :" + targetURL
                    req = urllib2.Request(targetURL)
                    try:
                        response = urllib2.urlopen(req)
                        the_page = response.read()  
                        if the_page.find("Your IP suspended")>=0:
                            print "The IP is suspended"
                            fout = open(self.dir+"/limit.txt",'wb')
                            limits = str(count)+","+str(upper)
                            fout.write(limits)
                            fout.close()  
                            break
                        if the_page.find("Too many requests")>=0:
                            print "Too many requests"
                            print "Renew IP...."
                            fout = open(self.dir+"/limit.txt",'wb')
                            limits = str(count)+","+str(upper)
                            fout.write(limits)
                            fout.close()
                            break
    
    
    #subprocess.Popen("sudo  /Users/myfolder/Documents/workspace/DataMining/ip_renew.py", shell=True)
                        if the_page.find("404 error")>=0:
                            print "the page not exist"
                            continue
                        self.saveHTML(count, the_page)
                    except:
                        print "The URL cannot be fetched"
                        pass
                        #raise
    
        def saveHTML(self,count, content):
            fout = open(self.dir+"/"+str(count)+".html",'wb')
            fout.write(content)
            fout.close()
    
        if __name__ == '__main__':
    
        if len(sys.argv) !=6:
            print "cannot process!!! Five Parameters are required to run the          process."
            print "Parameter 1 should be the path where to save the data, eg, /Users/myfolder/data/"
            print "Parameter 2 should be the name of the country for which data is collected, eg, japan"
            print "Parameter 3 should be the URL from which the data to collect, eg, http://www.jpyellow.com/company/"
            print "Parameter 4 should be the lower limit of the company id, eg, 11 "
            print "Parameter 5 should be the upper limit of the company id, eg, 1000 "
            print "The output will be saved as the HTML file for each company in the target folder's country"
            exit()
        else:
            path = str(sys.argv[1])
            country = str(sys.argv[2])
            url = str(sys.argv[3])
            lowerlimit = int(sys.argv[4])
            upperlimit = int(sys.argv[5])
    

    This is LINE 92

            **WebPage(path, country, url, lowerlimit,upperlimit)**
    

I have downloaded TorVPN to run a Proxy server.. and running this script. So why the error is occurring? This is the script which can download a website.

1

There are 1 answers

0
Anand S Kumar On BEST ANSWER

The issue is in DownloadYP.py -

You do not have the files - C:\\rrr\japan\limit.txt

I would suggest creating a dummy file in the above directory with that name, and try running the script again.

Also, on a side note - You are mixing the os path separaters from unix and using it in windows , instead you need to use os.path.join() function so that python is able to handle the os path separaters accross platforms. The code would be like -

import os
self.dir = os.path.join(str(path),str(country))

Also, when openning the file, you need to use os.path.join , instead of directly specifying the path separators -

fin = open(os.path.join(self.dir,"limit.txt"),'r')