Error trying to get uids of emails in large folders using IMAP and Twisted

Question

I am writing an IMAP client using the last version of Twisted.

I am having trouble using two different ways for getting email UIDs.

First i have tried to use the search method this way :

@inlineCallbacks
def getEmailList(self):
    for f in folder_list:
        rep = yield self.examine(f)
        uids_list = yield self.search(imap4.Query(all=True), uid=True)

This works but when I try to use it on large folders (containing more than 10.000 messages) the command fails.

I got an error like :

Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/twisted/protocols/policies.py", line 120, in dataReceived
    self.wrappedProtocol.dataReceived(data)
  File "/usr/local/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 571, in dataReceived
    why = self.lineReceived(line)
  File "/usr/local/lib/python2.7/dist-packages/twisted/mail/imap4.py", line 2360, in lineReceived
    self._regularDispatch(line)
  File "/usr/local/lib/python2.7/dist-packages/twisted/mail/imap4.py", line 2388, in _regularDispatch
    self.dispatchCommand(tag, rest)
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/dist-packages/twisted/mail/imap4.py", line 2417, in dispatchCommand
    f(tag, rest)
  File "/usr/local/lib/python2.7/dist-packages/twisted/mail/imap4.py", line 2446, in response_UNAUTH
    self._defaultHandler(tag, rest)
  File "/usr/local/lib/python2.7/dist-packages/twisted/mail/imap4.py", line 2467, in _defaultHandler
    raise IllegalServerResponse(tag + ' ' + rest)
twisted.mail.imap4.IllegalServerResponse: 1428 111429 111430 111431 111432

So am I doing wrong ? Or can I handle the server answer in a better way ?

I tried something else for passing through this problem. Instead of using search method I tried to replace :

uids_list = yield self.search(imap4.Query(all=True), uid=True)

by

rep = yield self.fetchUID('1:*')
uids_list = set([x['UID'] for x in rep.values()])

This works too (slower than previous method sometimes) but when I try to use it on imap.mail.yahoo.com it fails with this error :

[Failure instance: Traceback (failure with no frames): <class'twisted.mail.imap4.IMAP4Exception'>: [CLIENTBUG] FETCH Bad sequence in the command]

This is strange because when I run the same command using imaplib I don't get any error so did I miss something ?

EDIT : I resolve this problem. The IMAP4 server of yahoo seems to have a pretty strange implementation. In fact, if we ask for a sequence that does not exists in the folder. For example (was my case) if the folder is empty and we send:

FETCH 1:* (UID)

The server fails with :

BAD [CLIENTBUG] FETCH Bad sequence in the command

So for bypassing this error I just check the answer of examine like that :

rep = yield self.examine(f)
if rep['EXISTS'] != 0:
    rep = yield self.fetchUID('1:*')
    uids_list = set([x['UID'] for x in rep.values()])

Thanks by advance for any hint or answer,

Answer 1

That is a client bug. If you refer to message number 1 and the server has told you that there are 0 messages, the bug is plainly yours. The same applies if the server has told you that there are 1000 messages and you refer to number 1001.

Listen to the EXISTS responses. Or switch to using UID (and listen to the UIDNEXT response codes).