Can python abstract base classes inherit from C extensions?

Question

It seems as if that when I have an abstract base class that inherits from gevent.Greenlet (which inherits from the C extension module greenlet: https://github.com/python-greenlet/greenlet) then classes that implement it do not raise any of the abc errors about unimplemented methods.

class ActorBase(gevent.Greenlet):
    __metaclass__ = abc.ABCMeta

    @abc.abstractmethod
    def foo(self):
        print "foo"

class ActorBaseTest(ActorBase):
    def bar(self):
        print "bar"

abt = ActorBaseTest()  # no errors!

If I inherit from object it fails as expected:

class ActorBase(object):
    __metaclass__ = abc.ABCMeta

    @abc.abstractmethod
    def foo(self):
        print "foo"

class ActorBaseTest(ActorBase):
    def bar(self):
        print "bar"

>>> abt = ActorBaseTest()
Traceback (most recent call last):
  File "/home/dw/.virtualenvs/prj/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2827, in run_code
exec code_obj in self.user_global_ns, self.user_ns
  File "<ipython-input-6-d67a142e7297>", line 1, in <module>
    abt = ActorBaseTest()
TypeError: Can't instantiate abstract class ActorBaseTest with abstract methods foo

What is the right way to implement this functionality?

Answer 1

The cause of your problem is that it's the object.__new__ method that does the check for instantiation of an abstract class, and in this case object.__new__ isn't being invoked: gevent.Greenlet inherits from greenlet.greenlet, and greenlet.greenlet is a C extension type whose __new__ implementation doesn't call object.__new__ at any point (see the green_new function in the greenlet C source).

You can see the same effect by subclassing some of the other builtin types that implement their own __new__ method and don't refer back to object.__new__ (the float type, for example). The issue is not particular to C extension types, though: you can also replicate it with pure Python types. Consider the code below:

import abc

class A(object):
    def __new__(cls):
        # self = object.__new__(cls)
        return 42

class B(A):
    __metaclass__ = abc.ABCMeta

    @abc.abstractmethod
    def foo(self):
        pass

b = B()  # No exception.

The class B is correctly registered as an abstract class (internally, its Py_TPFLAGS_IS_ABSTRACT bit is set in the tp_flags field), but object.__new__ is never called, so there's no error when B is instantiated. However, if you uncomment the self = object.__new__(cls) method call in A, you'll see the expected error on instantiation.

As far as the 'right way' to implement this, unfortunately I think the right way is to fix the greenlet type so that its __new__ method calls object.__new__. I guess you could add a __new__ method to ActorBase that explicitly calls both the base class __new__ and object.__new__ (and throws away the result of the latter), but I'd consider that an ugly workaround rather than the 'right way'. (EDIT: And on top of that, it doesn't work. I get TypeError: object.__new__(ActorBase) is not safe, use greenlet.greenlet.__new__() from the object.__new__ call.) I've opened an issue on the greenlet tracker.

---

EDIT: This problem seemed somewhat familiar, and I just did some digging into the Enthought Traits source, which defines a CHasTraits class implemented in C which does play nicely with ABCs. And its __new__ method starts like this (comments are from the original source, not mine):

PyObject *
has_traits_new ( PyTypeObject * type, PyObject * args, PyObject * kwds ) {

    // Call PyBaseObject_Type.tp_new to do the actual construction.
    // This allows things like ABCMeta machinery to work correctly
    // which is implemented at the C level.
    has_traits_object * obj = (has_traits_object *) PyBaseObject_Type.tp_new(type, empty_tuple, empty_dict);

So perhaps the long-term solution is to persuade the greenlet folks to do something similar.