I am Noob in Python. My Boss instructed me to run this python script with TOR proxy server running. He told me to pass this parameter this way: python DownloadYP.py /Users/myfolder/ japan http://www.jpyellow.com/company 1 222299
He configured it on MAC. I am using windows. So my parameter is this way: python DownloadYP.py C:\rrrb japan http://www.jpyellow.com/company 1 222299
But I am getting error:
> Traceback (most recent call last):
File "C:\Users\USER\yp1\code\DownloadYP.py", line 92, in <module>
WebPage(path, country, url, lowerlimit,upperlimit)
File "C:\Users\USER\yp1\code\DownloadYP.py", line 23, in __init__
fout = open(self.dir+"/limit.txt",'wb')
IOError: [Errno 2] No such file or directory: 'C:\\rrr/japan/limit.txt'
My Code is:
DownloadYP.py
def __init__(self, path, country, url, lower=0,upper=9999): self.dir = str(path)+"/"+ str(country) self.url = url try: fin = open(self.dir+"/limit.txt",'r') limit = fin.readline() limits = str(limit).split(",") lower = int(limits[0]) upper = int(limits[1]) fin.close() except:
this is line 23
**fout = open(self.dir+"/limit.txt",'wb')** limits = str(lower)+","+str(upper) fout.write(limits) fout.close() self.process_instances(lower,upper) def process_instances(self,lower,upper): try: os.stat(self.dir) except: os.mkdir(self.dir) for count in range(lower,upper+1): if count == upper: print "all downloaded, quitting the app!!" break targetURL = self.url+"/"+str(count) print "Downloading :" + targetURL req = urllib2.Request(targetURL) try: response = urllib2.urlopen(req) the_page = response.read() if the_page.find("Your IP suspended")>=0: print "The IP is suspended" fout = open(self.dir+"/limit.txt",'wb') limits = str(count)+","+str(upper) fout.write(limits) fout.close() break if the_page.find("Too many requests")>=0: print "Too many requests" print "Renew IP...." fout = open(self.dir+"/limit.txt",'wb') limits = str(count)+","+str(upper) fout.write(limits) fout.close() break #subprocess.Popen("sudo /Users/myfolder/Documents/workspace/DataMining/ip_renew.py", shell=True) if the_page.find("404 error")>=0: print "the page not exist" continue self.saveHTML(count, the_page) except: print "The URL cannot be fetched" pass #raise def saveHTML(self,count, content): fout = open(self.dir+"/"+str(count)+".html",'wb') fout.write(content) fout.close() if __name__ == '__main__': if len(sys.argv) !=6: print "cannot process!!! Five Parameters are required to run the process." print "Parameter 1 should be the path where to save the data, eg, /Users/myfolder/data/" print "Parameter 2 should be the name of the country for which data is collected, eg, japan" print "Parameter 3 should be the URL from which the data to collect, eg, http://www.jpyellow.com/company/" print "Parameter 4 should be the lower limit of the company id, eg, 11 " print "Parameter 5 should be the upper limit of the company id, eg, 1000 " print "The output will be saved as the HTML file for each company in the target folder's country" exit() else: path = str(sys.argv[1]) country = str(sys.argv[2]) url = str(sys.argv[3]) lowerlimit = int(sys.argv[4]) upperlimit = int(sys.argv[5])
This is LINE 92
**WebPage(path, country, url, lowerlimit,upperlimit)**
I have downloaded TorVPN to run a Proxy server.. and running this script. So why the error is occurring? This is the script which can download a website.
The issue is in
DownloadYP.py
-You do not have the files -
C:\\rrr\japan\limit.txt
I would suggest creating a dummy file in the above directory with that name, and try running the script again.
Also, on a side note - You are mixing the os path separaters from unix and using it in windows , instead you need to use
os.path.join()
function so that python is able to handle the os path separaters accross platforms. The code would be like -Also, when openning the file, you need to use
os.path.join
, instead of directly specifying the path separators -