Python Gearman get data from background task

Question

I'm try use gearman with backgroud tasks and get data progress from worker. In documentation I'm see methods: send_job_data and send_job_status, but with background first method not work (I'm not see data in job.data_updates), but status changes in job.status.

I'm use this code for test workers: from gearman import GearmanWorker import time

worker = GearmanWorker(['192.168.1.79:4730'])

def long_task(work, job):
  work.send_job_data(job, 'long task')
  work.send_job_status(job, 0, 3)
  time.sleep(60)
  work.send_job_data(job, 'long task2')
  work.send_job_status(job, 1,3)
  time.sleep(120)
  work.send_job_status(job,3,3)
  return "COMPLETE ALL"

worker.register_task('pool', long_task)
worker.work()

And this code from client: from gearman import GearmanClient client = GearmanClient(['192.168.1.79:4730'])

This code (blocking) work normal:

In [6]: pool = client.submit_job('pool', '')
In [7]: pool.result
Out[7]: 'COMPLETE ALL'
In [8]: pool.data_updates
Out[8]: deque(['long task', 'long task2'])
In [9]: pool.status
Out[9]: 
{'denominator': 3,
 'handle': 'H:dhcp94:22',
 'known': True,
 'numerator': 3,
 'running': True,
 'time_received': 1322755490.691739}

And this client not work normal (not update status for task and not get data/result) :(

In [10]: pool = client.submit_job('pool', '', background=True)
In [11]: pool = client.get_job_status(pool)
In [12]: pool.status
Out[12]: 
{'denominator': 3,
 'handle': 'H:dhcp94:23',
 'known': True,
 'numerator': 0,
 'running': True,
 'time_received': 1322755604.695123}
In [13]: pool.data_updates
Out[13]: deque([])

In [14]: pool = client.get_job_status(pool)
In [15]: pool.data_updates
Out[15]: deque([])
In [16]: pool.status
Out[16]: 
{'denominator': 0,
 'handle': 'H:dhcp94:23',
 'known': False,
 'numerator': 0,
 'running': False,
 'time_received': 1322755863.306605}

How I'm can normal get this data? Because my background task will work few hours and send information about our status in messages.

Answer 1

background tasks are called such because they allow the client that submitted them to un-block and work disconnected. They do not keep a channel of communication open to the client, so you won't get any of those status updates. They essentially go into the bit-bucket. If you want a background task to communicate its progress, you need to have some other channel for it to communicate with interested programs.

If you want the client to keep running and get updates, but not block on them, you can use the "task" method where you add a bunch of tasks, and then wait for any of them to provide status or be completed. I'm not sure if the pure python gearman interface has this, but the libgearman interface does. Its available in source form here https://launchpad.net/gearman-interface or in some versions of Ubuntu/Debian as python-gearman.libgearman.