Crash in billiard after celery upgrade to 3

1k views Asked by At

I recently did a django-celery upgrade using on a ubuntu 10.4 server. It upgraded celery to 3.0.1 and then also installed billiards. Output as follows.

sudo easy_install --upgrade django-celery
Searching for django-celery
Reading http://pypi.python.org/simple/django-celery/
Reading http://celeryproject.org
Best match: django-celery 3.0.1
Downloading http://pypi.python.org/packages/source/d/django-celery/django-celery-3.0.1.tar.gz#md5=16c59b6a39676ca3fed6676e16e80bb3
Processing django-celery-3.0.1.tar.gz
Running django-celery-3.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-8H1yhX/django-celery-3.0.1/egg-dist-tmp-t1kBSJ
warning: no files found matching 'FAQ'
warning: no files found matching 'README.rst'
no previously-included directories found matching 'bin/*.pyc'
no previously-included directories found matching 'tests/*.pyc'
no previously-included directories found matching 'docs/*.pyc'
no previously-included directories found matching 'extra/*.pyc'
no previously-included directories found matching 'djcelery/*.pyc'
no previously-included directories found matching 'docs/.build'
no previously-included directories found matching 'examples/*.pyc'
Removing django-celery 2.3.3 from easy-install.pth file
Adding django-celery 3.0.1 to easy-install.pth file
Installing djcelerymon script to /usr/local/bin
Installing djcelerymon script to /usr/local/bin

Installed /usr/local/lib/python2.6/dist-packages/django_celery-3.0.1-py2.6.egg
Processing dependencies for django-celery
Searching for celery>=3.0.1
Reading http://pypi.python.org/simple/celery/
Reading http://github.com/ask/celery/
Best match: celery 3.0.1
Downloading http://pypi.python.org/packages/source/c/celery/celery-3.0.1.tar.gz#md5=64cd59537ddda0b32ef3e54935650771
Processing celery-3.0.1.tar.gz
Running celery-3.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-uL6a4S/celery-3.0.1/egg-dist-tmp-5hrIRJ
- force upgrading previous installation
  - removing '/usr/local/lib/python2.6/dist-packages/celery-2.3.1-py2.6.egg/celery/app/task' package...
no previously-included directories found matching '*.pyc'
Removing celery 2.3.1 from easy-install.pth file
Adding celery 3.0.1 to easy-install.pth file
Installing celeryctl script to /usr/local/bin
Installing celeryd script to /usr/local/bin
Installing camqadm script to /usr/local/bin
Installing celeryev script to /usr/local/bin
Installing celery script to /usr/local/bin
Installing celeryd-multi script to /usr/local/bin
Installing celerybeat script to /usr/local/bin

Installed /usr/local/lib/python2.6/dist-packages/celery-3.0.1-py2.6.egg
Searching for kombu>=2.2.5,<3.0
Reading http://pypi.python.org/simple/kombu/
Reading http://github.com/ask/kombu/
Reading http://kombu.readthedocs.org
Best match: kombu 2.2.6
Downloading http://pypi.python.org/packages/source/k/kombu/kombu-2.2.6.tar.gz#md5=3eac58bf4b54f28abb2be5e30404b46a
Processing kombu-2.2.6.tar.gz
Running kombu-2.2.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-iu3AZk/kombu-2.2.6/egg-dist-tmp-mD0pZA
warning: no files found matching 'FAQ'
warning: no files found matching 'README.rst'
warning: no files found matching 'README'
Removing kombu 1.2.1 from easy-install.pth file
Adding kombu 2.2.6 to easy-install.pth file

Installed /usr/local/lib/python2.6/dist-packages/kombu-2.2.6-py2.6.egg
Searching for billiard>=2.7.3.10
Reading http://pypi.python.org/simple/billiard/
Reading http://github.com/ask/billiard
Reading http://github.com/celery/billiard
Best match: billiard 2.7.3.10
Downloading http://pypi.python.org/packages/source/b/billiard/billiard-2.7.3.10.tar.gz#md5=c45ef2790bdaad4183e903d516c996f4
Processing billiard-2.7.3.10.tar.gz
Running billiard-2.7.3.10/setup.py -q bdist_egg --dist-dir /tmp/easy_install-I7GOq6/billiard-2.7.3.10/egg-dist-tmp-ktFy4f
warning: no files found matching '*.py' under directory 'Lib'
Adding billiard 2.7.3.10 to easy-install.pth file

Installed /usr/local/lib/python2.6/dist-packages/billiard-2.7.3.10-py2.6-linux-i686.egg
Searching for amqplib>=1.0.2
Reading http://pypi.python.org/simple/amqplib/
Reading http://barryp.org/software/py-amqplib/
Reading http://code.google.com/p/py-amqplib/
Best match: amqplib 1.0.2
Downloading http://py-amqplib.googlecode.com/files/amqplib-1.0.2.tgz
Processing amqplib-1.0.2.tgz
Running amqplib-1.0.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-64sBF8/amqplib-1.0.2/egg-dist-tmp-qSYI3b
zip_safe flag not set; analyzing archive contents...
Removing amqplib 1.0.0 from easy-install.pth file
Adding amqplib 1.0.2 to easy-install.pth file

Installed /usr/local/lib/python2.6/dist-packages/amqplib-1.0.2-py2.6.egg
Searching for anyjson>=0.3.3
Reading http://pypi.python.org/simple/anyjson/
Reading http://bitbucket.org/runeh/anyjson
Reading http://bitbucket.org/runeh/anyjson/
Best match: anyjson 0.3.3
Downloading http://pypi.python.org/packages/source/a/anyjson/anyjson-0.3.3.tar.gz#md5=2ea28d6ec311aeeebaf993cb3008b27c
Processing anyjson-0.3.3.tar.gz
Running anyjson-0.3.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ZziFNI/anyjson-0.3.3/egg-dist-tmp-iwK4uD
Removing anyjson 0.3.1 from easy-install.pth file
Adding anyjson 0.3.3 to easy-install.pth file

Installed /usr/local/lib/python2.6/dist-packages/anyjson-0.3.3-py2.6.egg
Finished processing dependencies for django-celery

Now when i try to run celery, it crashes with some error in billiard saying invalid pointer issue. What i am missing?

*** glibc detected *** /usr/bin/python: free(): invalid pointer: 0x089b4724 ***
======= Backtrace: =========
/lib/tls/i686/nosegneg/libc.so.6(+0x6c321)[0xb74e0321]
/lib/tls/i686/nosegneg/libc.so.6(+0x6db78)[0xb74e1b78]
/lib/tls/i686/nosegneg/libc.so.6(cfree+0x6d)[0xb74e4c9d]
/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.10-py2.6-linux-i686.egg/_billiard.so(+0x3824)[0xb7369824]
/usr/bin/python[0x8090304]
/usr/bin/python[0x80ab611]
/usr/bin/python[0x8090304]
/usr/bin/python[0x80ab611]
/usr/bin/python[0x808308e]
/usr/bin/python[0x816c36c]
/usr/bin/python[0x8090304]
/usr/bin/python[0x806c88b]
/usr/bin/python[0x8169a4a]
/usr/bin/python[0x81077d4]
/usr/bin/python[0x81077e4]
/usr/bin/python[0x81077e4]
/usr/bin/python[0x81077e4]
/usr/bin/python[0x808ecd9]
/usr/bin/python(PyDict_SetItem+0x87)[0x8091037]
/usr/bin/python(PyDict_SetItemString+0x43)[0x80911f3]
/usr/bin/python(PyImport_Cleanup+0x101)[0x80f2be1]
/usr/bin/python(Py_Finalize+0x10c)[0x80ffdec]
/usr/bin/python(Py_Main+0x4a6)[0x805d826]
/usr/bin/python(main+0x1b)[0x805d03b]
/lib/tls/i686/nosegneg/libc.so.6(__libc_start_main+0xe6)[0xb748abd6]
/usr/bin/python[0x805cf81]
======= Memory map: ========
08048000-08228000 r-xp 00000000 08:01 475601     /usr/bin/python2.6
08228000-08229000 r--p 001df000 08:01 475601     /usr/bin/python2.6
08229000-08278000 rw-p 001e0000 08:01 475601     /usr/bin/python2.6
08278000-08281000 rw-p 00000000 00:00 0 
08712000-08c04000 rw-p 00000000 00:00 0          [heap]
b7100000-b7121000 rw-p 00000000 00:00 0 
b7121000-b7200000 ---p 00000000 00:00 0 
b7209000-b7226000 r-xp 00000000 08:01 73921      /lib/libgcc_s.so.1
b7226000-b7227000 r--p 0001c000 08:01 73921      /lib/libgcc_s.so.1
b7227000-b7228000 rw-p 0001d000 08:01 73921      /lib/libgcc_s.so.1
b7230000-b7237000 r-xp 00000000 08:01 41665      /usr/local/lib/python2.6/dist-packages/simplejson-2.0.9-py2.6-linux-i686.egg/simplejson/_speedups.so
b7237000-b7238000 r--p 00006000 08:01 41665      /usr/local/lib/python2.6/dist-packages/simplejson-2.0.9-py2.6-linux-i686.egg/simplejson/_speedups.so
b7238000-b7239000 rw-p 00007000 08:01 41665      /usr/local/lib/python2.6/dist-packages/simplejson-2.0.9-py2.6-linux-i686.egg/simplejson/_speedups.so
b7239000-b723b000 r-xp 00000000 08:01 486931     /usr/lib/python2.6/lib-dynload/resource.so
b723b000-b723c000 r--p 00001000 08:01 486931     /usr/lib/python2.6/lib-dynload/resource.so
b723c000-b723d000 rw-p 00002000 08:01 486931     /usr/lib/python2.6/lib-dynload/resource.so
b723e000-b723f000 rw-s 00000000 00:0f 11605      /dev/shm/sem.mp-025f3c38-1141-2
b723f000-b7242000 r-xp 00000000 08:01 73973      /lib/libuuid.so.1.3.0
b7242000-b7243000 r--p 00002000 08:01 73973      /lib/libuuid.so.1.3.0
b7243000-b7244000 rw-p 00003000 08:01 73973      /lib/libuuid.so.1.3.0
b7244000-b725b000 r-xp 00000000 08:01 486899     /usr/lib/python2.6/lib-dynload/_ctypes.so
b725b000-b725c000 r--p 00017000 08:01 486899     /usr/lib/python2.6/lib-dynload/_ctypes.so
b725c000-b725f000 rw-p 00018000 08:01 486899     /usr/lib/python2.6/lib-dynload/_ctypes.so
b725f000-b726e000 r-xp 00000000 08:01 486917     /usr/lib/python2.6/lib-dynload/datetime.so
b726e000-b726f000 r--p 0000e000 08:01 486917     /usr/lib/python2.6/lib-dynload/datetime.so
b726f000-b7272000 rw-p 0000f000 08:01 486917     /usr/lib/python2.6/lib-dynload/datetime.so
b7272000-b7355000 rw-p 00000000 00:00 0 
b7355000-b735c000 r-xp 00000000 08:01 83780      /lib/tls/i686/nosegneg/librt-2.11.1.so
b735c000-b735d000 r--p 00006000 08:01 83780      /lib/tls/i686/nosegneg/librt-2.11.1.so
b735d000-b735e000 rw-p 00007000 08:01 83780      /lib/tls/i686/nosegneg/librt-2.11.1.so
b735e000-b735f000 rw-s 00000000 00:0f 11603      /dev/shm/sem.mp-025f3c38-1141-1
b735f000-b7360000 rw-s 00000000 00:0f 11601      /dev/shm/sem.mp-025f3c38-1141-0
b7360000-b7361000 rw-p 00000000 00:00 0 
b7361000-b7363000 r-xp 00000000 08:01 486905     /usr/lib/python2.6/lib-dynload/_heapq.so
b7363000-b7364000 r--p 00001000 08:01 486905     /usr/lib/python2.6/lib-dynload/_heapq.so
b7364000-b7366000 rw-p 00002000 08:01 486905     /usr/lib/python2.6/lib-dynload/_heapq.so
b7366000-b736b000 r-xp 00000000 08:01 50186      /usr/local/lib/python2.6/dist-packages/billiard-2.7.3.10-py2.6-linux-i686.egg/_billiard.so
b736b000-b736c000 r--p 00004000 08:01 50186      /usr/local/lib/python2.6/dist-packages/billiard-2.7.3.10-py2.6-linux-i686.egg/_billiard.so
b736c000-b736d000 rw-p 00005000 08:01 50186      /usr/local/lib/python2.6/dist-packages/billiard-2.7.3.10-py2.6-linux-i686.egg/_billiard.so
b736d000-b73ef000 rw-p 00000000 00:00 0 
b73ef000-b73f0000 rwxp 00000000 00:00 0 
b73f0000-b7474000 rw-p 00000000 00:00 0 
b7474000-b75cb000 r-xp 00000000 08:01 83764      /lib/tls/i686/nosegneg/libc-2.11.1.so
b75cb000-b75cc000 ---p 00157000 08:01 83764      /lib/tls/i686/nosegneg/libc-2.11.1.so
b75cc000-b75ce000 r--p 00157000 08:01 83764      /lib/tls/i686/nosegneg/libc-2.11.1.so
b75ce000-b75cf000 rw-p 00159000 08:01 83764      /lib/tls/i686/nosegneg/libc-2.11.1.so
b75cf000-b75d2000 rw-p 00000000 00:00 0 
b75d2000-b75f6000 r-xp 00000000 08:01 83768      /lib/tls/i686/nosegneg/libm-2.11.1.so
b75f6000-b75f7000 r--p 00023000 08:01 83768      /lib/tls/i686/nosegneg/libm-2.11.1.so
b75f7000-b75f8000 rw-p 00024000 08:01 83768      /lib/tls/i686/nosegneg/libm-2.11.1.so
b75f8000-b760b000 r-xp 00000000 08:01 73977      /lib/libz.so.1.2.3.3
b760b000-b760c000 r--p 00012000 08:01 73977      /lib/libz.so.1.2.3.3
b760c000-b760d000 rw-p 00013000 08:01 73977      /lib/libz.so.1.2.3.3
b760d000-b7745000 r-xp 00000000 08:01 74561      /lib/i686/cmov/libcrypto.so.0.9.8
b7745000-b774d000 r--p 00137000 08:01 74561      /lib/i686/cmov/libcrypto.so.0.9.8
b774d000-b775b000 rw-p 0013f000 08:01 74561      /lib/i686/cmov/libcrypto.so.0.9.8
b775b000-b775f000 rw-p 00000000 00:00 0 
b775f000-b77a3000 r-xp 00000000 08:01 74562      /lib/i686/cmov/libssl.so.0.9.8
b77a3000-b77a4000 r--p 00044000 08:01 74562      /lib/i686/cmov/libssl.so.0.9.8
b77a4000-b77a7000 rw-p 00045000 08:01 74562      /lib/i686/cmov/libssl.so.0.9.8
b77a7000-b77a8000 rw-p 00000000 00:00 0 
b77a8000-b77aa000 r-xp 00000000 08:01 83782      /lib/tls/i686/nosegneg/libutil-2.11.1.so
b77aa000-b77ab000 r--p 00001000 08:01 83782      /lib/tls/i686/nosegneg/libutil-2.11.1.so
b77ab000-b77ac000 rw-p 00002000 08:01 83782      /lib/tls/i686/nosegneg/libutil-2.11.1.so
b77ac000-b77ae000 r-xp 00000000 08:01 83767      /lib/tls/i686/nosegneg/libdl-2.11.1.so
b77ae000-b77af000 r--p 00001000 08:01 83767      /lib/tls/i686/nosegneg/libdl-2.11.1.so
b77af000-b77b0000 rw-p 00002000 08:01 83767      /lib/tls/i686/nosegneg/libdl-2.11.1.so
b77b0000-b77c5000 r-xp 00000000 08:01 83778      /lib/tls/i686/nosegneg/libpthread-2.11.1.so
b77c5000-b77c6000 r--p 00014000 08:01 83778      /lib/tls/i686/nosegneg/libpthread-2.11.1.so

Do i need to upgrade libc? How can i upgrade it.

1

There are 1 answers

0
JiminyCricket On

I was having a similar issue. Installing the python-devel package fixed it though.

yum install python-devel