python code not seeing a class variable iniialized in the __init__() function

Question

I am running into a weird error with spynner, though the question is a generic one. Spynner is the stateful web-browser module for python. It works fine when it works but I almost with every run I get a failure saying this --

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/spynner-2.16.dev0-py2.7.egg/spynner/browser.py", line 1651, in createRequest
    self.cookies,
AttributeError: 'Browser' object has no attribute 'cookies'
Segmentation fault (core dumped)

The problem here is its segfaulting and not letting me continue.

Looking at the code for spynner I see that the cookies variable is in fact initialized in the __init__() function for the Browser class like this:

self.cookies = []

Now on failure its really saying that the __init__() is not run since its not seeing the cookies variable. I do not understand how that can be possible. Without restricting to the spynner module can someone venture a guess as to how a python object could fail with an error like this?

EDIT: I definitely would have pasted my code here except its not all in one place for me to compactly show it. I should have done it earlier but here is the overall structure and how I instantiate and use spynner.

# helper class to get url data
class C:
   def __init__(self):
       self.browser = spynner.Browser()

   def get_data(self, url):
       try:
           self.browser.load(url)
           return self.browser.html
       except:
           raise

# class that does other stuff among saving url data to disk
class B:
    def save_url_to_disk(self, url):
        urlObj = C()
        html = urlObj.get_data(url)
        # do stuff with html


# class that drives everything
class A:
    def do_stuff_and_save_url_data(self, url):
       fileObj = B()
       fileObj.save_url_to_disk(url)

driver = A()
# call this function for multiple URLs.
driver.do_stuff_and_save_url_data(url)

The way I run it is ---

# xvfb-run python myfile.py

The segfault is probably something else I am doing. May be its because of the xvfb I am using and not handling properly? I don't know yet. I need to mention that I am relatively new to python.

I noticed that when I run the code above with say 'http://www.google.com' I get the segfault every other time.

Answer 1

The code block of do_stuff_and_save_url_data() doesn't use the reference self:
then the execution of this function doesn't depend on driver.

The code block of save_url_to_disk() also doesn't use the reference self:
then the execution of this second function doesn't depend on the object fileObj.

Only the code block of get_data() uses the reference self, and more precisely the reference self.browser:
so its execution and result depends on the attribute browser of the instance urlObj from class C. This attribute is in fact a browser instance named browser of the spynner.Browser class.

In the end, you "do stuff with html" with just the data outputed by spynner.Browser().html. And creation of driver and fileObj aren't mandatory in any way.

.

Another point is that
when the instruction driver.do_stuff_and_save_url_data(url) is executed,
the method driver.do_stuff_and_save_url_data(url) is first created, then executed, and finally "destroyed" (or more precisely forgot somewhere in the RAM) because it hasn't been assigned to any identifier.

Then the identifier fileObj, which is an identifier belonging to the local namespace of the function driver.do_stuff_and_save_url_data() , is lost too, which means the instance fileObj of class B is also lost for ulterior use since it has no more assigned identifier alive.

It's the same for save_url_to_disk():
after the creation and execution of the method fileObj.save_url_to_disk(url), the object urlObj of class C , which contains an instance of browser ( an object created by spynner.Browser() ), is lost: the created browser and all its data is lost.

I wonder if this isn't because of this destruction of the browser instance after each execution of do_stuff_and_save_url_data() and save_url_to_disk() that the cookies information wouldn't be destroyed before an ulterior call.

.

So, in my opinion, your code only embeds two functions in two definitions of classes A and B and they are used as being considered functions , not as methods.

1/ I don't think it is a good coding pattern. When one wants only plain functions, they must be written outside of any class.

2/ The problem is that if operations are triggered by functions, a new browser is created each time these functions are activated , even if they have the mantle of methods.

You will say me that you want these functions to act with data provided by the browser defined by spynny.Browser().
That's why I think that they must not be functions embeded in classes as now, but real methods attached to a stable instance of a browser. That's the aim of object to keep in the same namespace the data and the tools to treat the data.

-

.

All that said, I would personnally write:

class C(spynner.Browser):
   def __init__(self):
       spynner.Browser.__init__(self)

   def get_data(self, url):
       try:
           self.html = self.load(url).html
       except:
           raise

    # method that does other stuff among saving url data to disk
    def save_url_to_disk(self, url):
        get_data(url)
        # do stuff with self.html

    # method that drives everything
    def do_stuff_and_save_url_data(self, url):
        self.save_url_to_disk(url)


driver = C()
driver.do_stuff_and_save_url_data(url)

But I'm not sure to have well undesrtood all your considerations, and I warn that I didn't know spynner before reading your post. All that I've written could be stupid relatively to your real problem. Keep a critic eye on my post, please.