swig: How to make a QList<T> iterable, like std::vector

Question

I'm using SWIG to generate Python Bindings for my qt app. I have several places where I use QLists and I would like to integrate those QLists like std::vector from the SWIG Library (see http://www.swig.org/Doc1.3/Library.html#Library_nn15).
This means:

• The QList objects should be iterable from python (= they must be an iterable python object)
• It should be possible to pass a python list to a function which takes a qlist
• ... and all the other features listed in the SWIG Library for std::vector

To achieve that I use the following Code: https://github.com/osmandapp/OsmAnd-core/blob/master/swig/java/QList.i
Later in my classes using QLists, I add code like:

%import "qlist.i"
%template(listfilter) QList<Interface_Filter*>;

class A {
    public:
    //.....
    QList<Interface_Filter*> get_filters();
};

This works so far, but it doesn't give me the kind of integration I get with std::vector. I'm having trouble finding out which parts of std_vector.i, std_container.i,... make an object iterable.
How do I need to extend the QList interface file to make my QList's iterable?

Answer 1

What you are asking for -- a qlist.i swig file that achieves the same level of integration for QList in python as std_vector.i does for std::vector -- is a non-trivial task.

I provide a very basic extended qlist.i file (and qlisttest.i to show you how to use it) and will try to explain what steps are required.

qlist.i:

%{
#include <QList>
%}

%pythoncode %{
class QListIterator:
    def __init__(self, qlist):
        self.index = 0
        self.qlist = qlist
    def __iter__(self):
        return self
    def next(self):
        if self.index >= self.qlist.size():
            raise StopIteration;
        ret = self.qlist.get(self.index)
        self.index += 1
        return ret
    __next__ = next
%}

template<class T> class QList {
public:
class iterator;
typedef size_t size_type;
typedef T value_type;
typedef const value_type& const_reference;
QList();
size_type size() const;
void reserve(size_type n);
%rename(isEmpty) empty;
bool empty() const;
void clear();
%rename(add) push_back;
void push_back(const value_type& x);
%extend {
    const_reference get(int i) throw (std::out_of_range) {
    int size = int(self->size());
    if (i>=0 && i<size)
        return (*self)[i];
    else
        throw std::out_of_range("QList index out of range");
    }
    void set(int i, const value_type& val) throw (std::out_of_range) {
    int size = int(self->size());
    if (i>=0 && i<size)
        (*self)[i] = val;
    else
        throw std::out_of_range("QList index out of range");
    }
    int __len__() {
    return self->size();
    }
    const_reference __getitem__(int i) throw (std::out_of_range) {
    int size = int(self->size());
    if (i>=0 && i<size)
        return (*self)[i];
    else
        throw std::out_of_range("QList index out of range");
    }
    %pythoncode %{
    def __iter__(self):
        return QListIterator(self)
    %}
}
};

%define %qlist_conversions(Type...)
%typemap(in) const QList< Type > & (bool free_qlist)
{
free_qlist = false;
if ((SWIG_ConvertPtr($input, (void **) &$1, $1_descriptor, 0)) == -1) {
    if (!PyList_Check($input)) {
    PyErr_Format(PyExc_TypeError, "QList or python list required.");
    SWIG_fail;
    }
    Py_ssize_t len = PyList_Size($input);
    QList< Type > * qlist = new QList< Type >();
    free_qlist = true;
    qlist->reserve(len);
    for (Py_ssize_t index = 0; index < len; ++index) {
    PyObject *item = PyList_GetItem($input,index);
    Type* c_item;
    if ((SWIG_ConvertPtr(item, (void **) &c_item, $descriptor(Type *),0)) == -1) {
        delete qlist;
        free_qlist = false;
        PyErr_Format(PyExc_TypeError, "List element of wrong type encountered.");
        SWIG_fail;
    }
    qlist->append(*c_item);
    }
    $1 = qlist;
}
}
%typemap(freearg) const QList< Type > &
{ if (free_qlist$argnum and $1) delete $1; }    
%enddef

qlisttest.i:

%module qlist;
%include "qlist.i"

%inline %{
class Foo {
public:
    int foo;
};
%}

%template(QList_Foo) QList<Foo>;
%qlist_conversions(Foo);

%inline %{  
int sumList(const QList<Foo> & list) {
    int sum = 0;
    for (int i = 0; i < list.size(); ++i) {
        sum += list[i].foo;
    }
    return sum;
}
%}

1. Wrapping of QList to make it and its methods accessible from python
   This is achieved by making the (partial) class definition available to swig. That is what your current qlist.i does.
   Note: You might need to add a "template specialization" for the case QList<T*> that typedefs const_reference as const T* since you are using a QList of pointers. Otherwise, QList<T*>::const_reference will be const T*&, which apparently might confuse swig. (see swig/Lib/std/std_vector.i)

2. Automatic conversion between python list and QList
   This is generally achieved by using swig typemaps. For instance, if you want a function f(const QList<int>& list) to be able to accept a python list, you need to specify an input typemap that performs the conversion from a python list to a QList<int>:

   %typemap(in) const QList<int> &
   {
       PyObject * py_list = $input;
       [check if py_list is really a python list of integers]
       QList<int>* qlist = new QList<int>();
       [copy the data from the py_list to the qlist]
       $1 = qlist;
   }
   %typemap(freearg) const QList<int> &
   { if ($1) delete $1; }
   

   Here, the situation is more difficult in several ways:

   • You want to be able to pass a python lists or a wrapped QList: For this to work, you need to handle both cases in the typemap.
   • You want to convert a python list of wrapped type T to a QList<T>:
     This also involves a conversion for every element of the list from the wrapped type T to the plain T. This is achieved by the swig function SWIG_ConvertPtr.
   • I am not sure if you can specify typemaps with template arguments. Therefore, I wrote a swig macro %qlist_conversions(Type) that you can use to attach the typemap to the QList<Type> for a specific Type.

   For the other conversion direction (QList -> python list) you should first consider what you want. Consider a C++ function that returns a QList<int>. Calling this from python, should this return a wrapped QList object, or should it automatically convert the QList to a python list?

3. Accessing the wrapped QList as a python sequence, i.e., make len and [] work from python
   For this, you need to extend the QList class in the qlist.i file using %extend { ... } and implement __len__ and __getitem__ methods.

   If slicing should also work, you need to provide a __getitem__(PySliceObject *slice)__ member method and input and "typecheck" typemaps for PySliceObjects.

   If you want to be able to modify values in the wrapped QList using [] from python, you need to implement __setitem__.

   For a list of all the useful methods you can implement to achieve better integration, see the python documentation on "builtin types" and "abstract base classes for containers".

   Note: If you use the swig -builtin feature, then you need to additionally register the above functions to the appropriate "slots" using e.g.

   %feature("python:slot", "sq_length", functype="lenfunc") __len__;
   

4. Making the wrapped QList iterable from python
   For this you need to extend the QList class and implement an __iter__() method that returns a python iterator object.

   A python iterator object is an object that provides the methods __iter__() and __next__() (next() for older python), where __next__() returns the next value and raises the python exception StopIteration to signal the end.

   As mentioned before, you can implement the iterator object in python or C++. I show an example of doing this in python.

I hope this helps as a basis for you to tweak the functionality that you require.

Answer 2

As described in the docs, you need the following:

• QList should have a method in python __iter__() that returns an iterator object (tp_iter if you implement it in C).
• The iterator object should implement __iter__() and return itself
• The iterator object should implement next() that returns the next item or raises StopIteration when it's done.

It's probably easiest to do in python, but you can implement it in C as well.

Another option is to use python generators to avoid implementing an iterator type. To do this you QList needs to implement __iter__() but instead of returning an iterator you simply yield the values.

The methods mentioned only need to be visible to python. You don't have to make them available in C/Java.

See also SWIG interfacing C library to Python (Creating 'iterable' Python data type from C 'sequence' struct)

Answer 3

You provided an answer to the question "How to make a python Object iterable", but I asked for "How do I need to extend the QList interface file to make my QList's iterable?" which is more a SWIG, than a python related question.

I tested the example from http://www.swig.org/Doc1.3/Library.html#Library_nn15 with Java, C# and Python. Only Python and C# provide iterators. The generated interface of Java doesn't implement Iterable or something like that. As far as I can see your question is related to the target language.

Maybe extending MutableSequence is an option for you. The only methods you have to implement are __getitem__, __setitem__, __delitem__, __len__ and insert by delegating them to the corresponding methods of QList. Afterwards your generated class is iterable.