PermanentTaskFailure: __new__() takes exactly 4 arguments (due to DateTimePropety?)

Question

Following the "Background work with the deferred library" article, I created a set of scheduled tasks to help maintain our app. They each flush old entities of a specific type.

In one case, instead of using the standard ndb model key property (for tracking which records have been processed), we use a non-key property of type DateTimeProperty. When we do this, and we hit the DeadlineExceededError, the deferred instance of the method dies during the unpickling of the task arguments made in the deferred.run method.

Relevant code in our base class:

def _continue(self, start_key, batch_size):
    # ...
    except DeadlineExceededError:
        if self._numBatches > 0:
            self.Log("Deferring to a new instance of the process.")
            deferred.defer(self._continue, start_key, batchSize)
        else:
            self.LogWarning("No batches were completely processed. This process will terminate to prevent continuous operation.")
            raise deferred.PermanentTaskFailure("DeadlineExceededError occurred before any batches could be completely processed.")
    self.Finish()

When the deferred copy of the method kicks in, it produces this error:

Traceback (most recent call last):
    File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 310, in post
        self.run_from_request()
    File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 305, in run_from_request
        run(self.request.body)
    File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 145, in run
        raise PermanentTaskFailure(e)
    PermanentTaskFailure: __new__() takes exactly 4 arguments (1 given)

More digging (pdb through the deferred and pickle library) unveils a bit more details:

1080        def load_newobj(self):
1081            args = self.stack.pop()
1082            cls = self.stack[-1]
1083 ->         obj = cls.__new__(cls, *args)
1084            self.stack[-1] = obj
1085        dispatch[NEWOBJ] = load_newobj

and

(Pdb) pp args
      ()
(Pdb) pp cls
      <class 'google.appengine.ext.ndb.query.FilterNode'>
(Pdb) n
      TypeError: '__new__() takes exactly 4 arguments (1 given)'
      > /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py(1083)load_newobj()

I'll be digging more, but the basic questions now seem to be:

1. Aside from pdb, how could one gain more info on a failure happening during this stage (ie: deferred.run and unpickling) ?
2. I found some hints online about the ability to override deferred.TaskHandler (introduced in v.1.6.3 - Feb 28, 2012), which may help with the first question, but could not find any example on how to do this.
3. Seems like ndb FilterNodes with a DateTimeProperty may not be picklable by default (or properly handled). Is that so?

Answer 1

As can be seen from the source of deferred.py, you've analyzed quite well together with @rdodev to realize that the problem is when attempting to un-pickle the data argument to run(). That in turn was being called from run_from_request(), so the thing that's un-pickled here is the request payload itself, which is created when you call deferred.defer().

It's possible that for the DateTimeProperty to be un-pickled, the pickle module needs to call the constructor for it (__new__), but this isn't available or presents a different method signature at that time than when it was pickled, resulting in the exception you see, which bubbles up to be caught and presents to you as PermanentTaskFailure.

Overall, this might be worth posting as a defect report to the public issue tracker if you can produce an example app which reliably reproduces the behaviour.

Other than that, as a work-around for now, I'd use something which doesn't cause the error, such as the string representation of the DateTimeProperty, or a regular datetime.datetime object, or even another ndb.Property class which doesn't cause the issue. You could also try to use the taskqueue library itself, which is quite easy to use and shouldn't represent a significant departure from your current use-case.