Django Celery throws error after changing databases

Question

I recently was forced to switch databases to mysql from sqlite3 due to high concurrent write problems when running my celery tasks (these worked fine with sqlite3). Now that I have the mysql database configured, It throws a

C:\Sites\deltafy-massive-feed-backend>celery -A backend worker -l info

-------------- celery@Robert-PC v3.1.11 (Cipater)
---- **** -----
--- * ***  * -- Windows-7-6.1.7601-SP1
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         backend:0x2d69b10
- ** ---------- .> transport:   django://localhost//
- ** ---------- .> results:     djcelery.backends.cache:CacheBackend
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
--------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . backend.celery.debug_task
  . private_api.tasks.gilt_script
  . private_api.tasks.nordstrom_script
  . private_api.tasks.zappos_script

[2014-06-14 01:37:13,625: ERROR/MainProcess] Unrecoverable error: WindowsError(123,             'The filename, directory name, or volume label syntax is incorrect')
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\celery\worker\__init__.py", line 206, in start
    self.blueprint.start(self)
  File "C:\Python27\lib\site-packages\celery\bootsteps.py", line 123, in start
    step.start(parent)
  File "C:\Python27\lib\site-packages\celery\bootsteps.py", line 373, in start
    return self.obj.start()
  File "C:\Python27\lib\site-packages\celery\concurrency\base.py", line 131, in start
    self.on_start()
  File "C:\Python27\lib\site-packages\celery\concurrency\prefork.py", line 117, in on_start
    **self.options)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 925, in __init__
    self._setup_queues()
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 1267, in _setup_queues
    self._inqueue = SimpleQueue()
  File "C:\Python27\lib\site-packages\billiard\queues.py", line 369, in __init__
    self._reader, self._writer = Pipe(duplex=False)
  File "C:\Python27\lib\site-packages\billiard\__init__.py", line 97, in Pipe
    return Pipe(duplex, rnonblock, wnonblock)
  File "C:\Python27\lib\site-packages\billiard\py2\connection.py", line 233, in Pipe
    1, obsize, ibsize, win32.NMPWAIT_WAIT_FOREVER, win32.NULL
WindowsError: [Error 123] The filename, directory name, or volume label syntax is     incorrect`

Also, I have a hunch that the address is the problem and contains invalid characters, so I went into connection.py and printed it. The address is:

c:\\.\pipe\pyc-5532-0-7ix5od

Any ideas as to what this is doing, or any solution to the problem is much appreciated! I haven't a clue even from my hours of searching.

Answer 1

This issue is a bug in billiard, the multiprocessing module fork used by Celery. Here is a link to the issue.

You may confirm this by running python -c "from billiard.connection import Pipe; p = Pipe(duplex=False)" which should produce the same error.

Cogniva has just pushed a fixed of this to their fork of billiard. At the time of writing this post this fix has been submitted though not merged into celery/billiard. This issue will be resolved in version 3.4.0.0 of billiard.

To install the fix :

$ pip uninstall billiard
$ git clone [email protected]:cogniva/billiard.git
$ cd billiard 
$ git checkout bb6169ff1bdd8bc7c5520f7c47085063bc51d4c7
$ python setup.py install

To confirm this fix works run python -c "from billiard.connection import Pipe; p = Pipe(duplex=False)". This time you should not see an error message.