PyInstance_NewRaw() with old and new style classes

Question

Recently I faced a problem in a C-based python extension while trying to instantiate objects without calling its constructor -- which is a requirement of the extension.

The class to be used to create instances is obtained dynamically: at some point, I have an instance x whose class I wish to use to create other instances, so I store x.__class__ for later use -- let this value be klass.

At a later point, I invoke PyInstance_NewRaw(klass, PyDict_New()) and then, the problem arises. It seems that if klass is an old-style class, the result of that call is the desired new instance. However, if it is a new-style class, the result is NULL and the exception raised is:

SystemError: ../Objects/classobject.c:521: bad argument to internal function

For the record, I'm using Python version 2.7.5. Googling around, I observed no more than one other person looking for a solution (and it seemed to me he was doing a workaround, but didn't detailed it).

For the record #2: the instances the extension is creating are proxies for these same x instances -- the x.__class__ and x.__dict__'s are known, so the extension is spawning new instances based on __class__ (using the aforementioned C function) and setting the respective __dict__ to the new instance (those __dict__'s have inter-process shared-memory data). Not only is conceptually problematic to call an instance's __init__ a second time (first: it's state is already know, second: the expected behavior for ctors is that they should be called exactly once for each instance), it is also impractical, since the extension cannot figure out the arguments and their order to call the __init__() for each instance in the system. Also, changing the __init__ of each class in the system whose instances may be proxies and making them aware there is a proxy mechanism they will be subjected to is conceptually problematic (they shouldn't know about it) and impractical.

So, my question is: how to perform the same behavior of PyInstance_NewRaw regardless of the instance's class style?

Answer 1

The type of new-style classes isn't instance, it's the class itself. So, the PyInstance_* methods aren't even meaningful for new-style classes.

In fact, the documentation explicitly explains this:

Note that the class objects described here represent old-style classes, which will go away in Python 3. When creating new types for extension modules, you will want to work with type objects (section Type Objects).

So, you will have to write code that checks whether klass is an old-style or new-style class and does the appropriate thing for each case. An old-style class's type is PyClass_Type, while a new-style class's type is either PyType_Type, or a custom metaclass.

Meanwhile, there is no direct equivalent of PyInstance_NewRaw for new-style classes. Or, rather, the direct equivalent—calling its tp_alloc slot and then adding a dict—will give you a non-functional class. You could try to duplicate all the other appropriate work, but that's going to be tricky. Alternatively, you could use tp_new, but that will do the wrong thing if there's a custom __new__ function in the class (or any of its bases). See the rejected patches from #5180 for some ideas.

But really, what you're trying to do is probably not a good idea in the first place. Maybe if you explained why this is a requirement, and what you're trying to do, there would be a better way to do it.

---

If the goal is to build objects by creating a new uninitialized instance of the class, then copying over its _dict__ from an initialized prototype, there's a much easier solution that I think will work for you:

__class__ is a writeable attribute. So (showing it in Python; the C API is basically the same, just a lot more verbose, and I'd probably screw up the refcounting somewhere):

class NewStyleDummy(object):
    pass
def make_instance(cls, instance_dict):
    if isinstance(cls, types.ClassType):
        obj = do_old_style_thing(cls)
    else:
        obj = NewStyleDummy()
        obj.__class__ = cls
    obj.__dict__ = instance_dict
    return obj

The new object will be an instance of cls—in particular, it will have the same class dictionary, including the MRO, metaclass, etc.

This won't work if cls has a metaclass that's required for its construction, or a custom __new__ method, or __slots__… but then your design of copying over the __dict__ doesn't make any sense in those cases anyway. I believe that in any case where anything could possibly work, this simple solution will work.

---

Calling cls.__new__ seems like a good solution at first, but it actually isn't. Let me explain the background.

When you do this:

foo = Foo(1, 2)

(where Foo is a new-style class), it gets converted into something like this pseudocode:

foo = Foo.__new__(1, 2)
if isinstance(foo, Foo):
    foo.__init__(1, 2)

The problem is that, if Foo or one of its bases has defined a __new__ method, it will expect to get the arguments from the constructor call, just like an __init__ method will.

As you explained in your question, you don't know the constructor call arguments—in fact, that's the main reason you can't call the normal __init__ method in the first place. So, you can't call __new__ either.

The base implementation of __new__ accepts and ignores any arguments it's given. So, if none of your classes has a __new__ override or a __metaclass__, you will happen to get away with this, because of a quirk in object.__new__ (a quirk which works differently in Python 3.x, by the way). But those are the exact same cases the previous solution can handle, and that solution works for much more obvious reason.

Put another way: The previous solution depends on nobody defining __new__ because it never calls __new__. This solution depends on nobody defining __new__ because it calls __new__ with the wrong arguments.