Python's __radd__ doesn't work for C-defined types

Question

When creating a python (2.7.5) extension that defines a noddy.Noddy type with __radd__ method, it gets a different behavior from a (otherwise equivalent) python defined-class object with a custom __radd__ (the former does not work, while the latter works). Example:

class PythonClass():
    def __radd__(self, other):
        return 'indeed!'

w = PythonClass()
d = noddy.Noddy()

print(w.__radd__)
print(d.__radd__)

print('the following works:')
print([1] + w)
print('the following does not work:')
print([1] + d)

And the corresponding output:

<bound method PythonClass.__radd__ of <__main__.PythonClass instance at 0xf6e9792c>>
<built-in method __radd__ of noddy.Noddy object at 0xf749d4b8>
the following works:
indeed!
the following does not work:
Traceback (most recent call last):
  File "examples/2.py", line 44, in <module>
    print([1] + d)
TypeError: can only concatenate list (not "noddy.Noddy") to list

The method d.__radd__ is not called, but w.__radd__ is. Any ideas as to why this is so? The behavior of [1] + x where x is a PythonClass instance seems to be in accord with the documentaton, and I would expect noddy.Noddy to work as well. Also, both are types unrelated to list.

Workarounds are welcome. I've already tried patching list.__radd__ with forbiddenfruit, with no success, though I have brought this issue to the author's attention, who happens to be a close friend of mine.

EDIT

...and here is a picture of the C land:

typedef struct {
    PyObject_HEAD
} Noddy;


static PyObject*
Noddy_radd(PyObject* _self, PyObject* args) {
    printf("Noddy_radd!\n");
    return NULL;
}

static PyObject*
Noddy_add(PyObject* _self, PyObject* args) {
  printf("Noddy_add\n");
  return NULL;
}

PyNumberMethods noddy_nums = {
  Noddy_add,         /* binaryfunc nb_add;         /* __add__ */
    0,               /* binaryfunc nb_subtract;    /* __sub__ */
    0,               /* binaryfunc nb_multiply;    /* __mul__ */
    0,               /* binaryfunc nb_divide;      /* __div__ */
    0,               /* binaryfunc nb_remainder;   /* __mod__ */
    0,               /* binaryfunc nb_divmod;      /* __divmod__ */
    0,               /* ternaryfunc nb_power;      /* __pow__ */
    0,               /* unaryfunc nb_negative;     /* __neg__ */
    0,               /* unaryfunc nb_positive;     /* __pos__ */
    0,               /* unaryfunc nb_absolute;     /* __abs__ */
    0,               /* inquiry nb_nonzero;        /* __nonzero__ */
    0,               /* unaryfunc nb_invert;       /* __invert__ */
    0,               /* binaryfunc nb_lshift;      /* __lshift__ */
    0,               /* binaryfunc nb_rshift;      /* __rshift__ */
    0,               /* binaryfunc nb_and;         /* __and__ */
    0,               /* binaryfunc nb_xor;         /* __xor__ */
    0,               /* binaryfunc nb_or;          /* __or__ */
    0,               /* coercion nb_coerce;        /* __coerce__ */
    0,               /* unaryfunc nb_int;          /* __int__ */
    0,               /* unaryfunc nb_long;         /* __long__ */
    0,               /* unaryfunc nb_float;        /* __float__ */
    0,               /* unaryfunc nb_oct;          /* __oct__ */
    0,               /* unaryfunc nb_hex;          /* __hex__ */
};

static PyMethodDef Noddy_methods[] = {
    {"__radd__", (PyCFunction)Noddy_radd, METH_VARARGS,
     "__radd__ function"},
    {NULL}  /* Sentinel */
};


static PyTypeObject NoddyType = {
    PyObject_HEAD_INIT(NULL)
    0,                         /*ob_size*/
    "noddy.Noddy",             /*tp_name*/
    sizeof(Noddy),             /*tp_basicsize*/
    0,                         /*tp_itemsize*/
    0,                         /*tp_dealloc*/
    0,                         /*tp_print*/
    0,                         /*tp_getattr*/
    0,                         /*tp_setattr*/
    0,                         /*tp_compare*/
    0,                         /*tp_repr*/
    &noddy_nums,               /*tp_as_number*/
    0,                         /*tp_as_sequence*/
    0,                         /*tp_as_mapping*/
    0,                         /*tp_hash */
    0,                         /*tp_call*/
    0,                         /*tp_str*/
    0,                         /*tp_getattro*/
    0,                         /*tp_setattro*/
    0,                         /*tp_as_buffer*/
    Py_TPFLAGS_DEFAULT |
      Py_TPFLAGS_HAVE_SEQUENCE_IN | /* tp_flags */
      Py_TPFLAGS_HAVE_ITER,
    "Noddy objects",           /* tp_doc */
    0,                     /* tp_traverse */
    0,                     /* tp_clear */
    0,                     /* tp_richcompare */
    0,                     /* tp_weaklistoffset */
    0,                     /* tp_iter */
    0,                     /* tp_iternext */
    Noddy_methods,             /* tp_methods */
    0,                         /* tp_members */
    0,                         /* tp_getset */
    0,                         /* tp_base */
    0,                         /* tp_dict */
    0,                         /* tp_descr_get */
    0,                         /* tp_descr_set */
    0,                         /* tp_dictoffset */
    0,                         /* tp_init */
    0,                         /* tp_alloc */
    PyType_GenericNew,         /* tp_new */
};

Answer 1

Python's getattr is a tricky guy. The __radd__ method is part of the [in]famous magic methods. They're not stored in the same array as the regular methods (ob_type->tp_methods), it is part of the ob_type->tp_as_number, managed separately by the Number Protocol.

Forbidden fruit has an issue requesting the ability to monkey patch those methods. This effort is documented here

Answer 2

How did you implement __radd__?

For C extensions, __radd__ isn't explicitly defined. There is a single slot called nb_add that supports a pointer to function that accepts two arguments. In a Python class method, the first argument is always the instance (i.e. self). So both normal and reflected methods are required. This is not true for C extensions. nb_add can be called with the instance as either argument.

EDIT

It will probably be easier to understand if you rewrite the signature of Noddy_add as Noddy_add(PyObject* a, PyObject* b). Let d be an instance of a custom C type. Then [1] + d is processed as follows (ignore the abuse of syntax and some special cases):

PyNumber_Add([1], d) is called. It first tries ListType.nb_add([1], d) which fails because the ListType doesn't implement nb_add. Then it tries NoddyType.nb_add([1], d) and this is the call you want to handle. If this call fails, then ListType.sq_concat([1], d) is called.

When you evaluate d + [1], the same sequence ends with NoddyType.nb_add(d, [1]). ListType.sq_concat is not called unless you implement the sequence methods for NoddyType.

You need to modify Noddy_add so it can be called with a reference to a list as the first parameter and a reference to a NoddyType as the second parameter. Calling nb_add with the arguments reversed is equivalent to __radd__ in Python code.

For details, see PyNumber_Add in Objects/abstract.c

Answer 3

From the Python Docs, look at the footer.

For operands of the same type, it is assumed that if the non-reflected method (such as add()) fails the operation is not supported, which is why the reflected method is not called.