How to log APScheduler warnings with more information?

Question

I'm using the Python APScheduler to run a bunch of recurring tasks, but for a couple tasks I now get errors such as:

WARNING:apscheduler.scheduler:Execution of job "getAndStoreExchangeRate (trigger: cron[minute='*'], next run at: 2014-11-06 18:48:00 CET)" skipped: maximum number of running instances reached (1)

In addition to that, the amount of used memory of this APscheduler instance rises suddenly after about ten days of running, from around 47MiB to 1200MiB, after which all sorts of processes stop because the machine is out of memory.

So to get to the root of the problem, I need to understand exactly which calls cause these warnings. As far as I understand, this error occurs because the previous call (a minute before this one) didn't end yet. The information in this error is rather vague though. the name of the function is given (getAndStoreExchangeRate), but I've got multiple files with a function named like this. So what I would like to know is:

• in which file on which line is this function for which the warning occurs?
• which arguments were given to this function?

Does anybody know how I can log this information somewhere? All tips are welcome!

[EDIT]

So I subclassed the BackgroundScheduler and overwrote the _process_jobs() method in which I changed the self._logger.warning in this snippet:

try:
    executor.submit_job(job, run_times)
except MaxInstancesReachedError:
    self._logger.warning(
        'Execution of job "%s" skipped: maximum number of running instances reached (%d)',
        job, job.max_instances
    )
except:
    self._logger.exception('Error submitting job "%s" to executor "%s"', job, job.executor)

to this:

'Execution of job "%s" skipped: maximum number of running instances reached (%d) =-= ARGS: ' + str(job.args) + ' - KWARGS: ' + str(job.kwargs),

This works in that it shows me the arguments given to the function, but I still don't know the most important thing: in which file and on which line this function for which the warning occurs is defined. Does anybody know how I could show that?

Answer 1

The easiest way would be to add an extra logging statement in the "except MaxInstancesReachedError:" block on apscheduler.schedulers.base module in the _process_jobs() method. There you have access to job.func which is the target function of the job.

I suggest you subclass your scheduler of choice and copy over the _process_jobs method code. Then modify the except MaxInstancesReachedError block:

try:
    executor.submit_job(job, run_times)
except MaxInstancesReachedError:
    self._logger.warning('Execution of job "%s" skipped: maximum number of running instances reached (%d)', job.func, job.max_instances)

EDIT:

The function's file name: job.func.__code__.co_filename

Line number: job.func.__code__.co_firstlineno