Django - possible leak of file descriptors?

460 views Asked by At

I am developing application in Django where I want to use multiprocessing in order to perform effective computation but somehow python/Django keeps open file descriptors to local database what makes errors after while.

In file models.py I've got class:

class VM(models.Model):
    name = models.CharField(max_length=30, primary_key=True)
    status = models.CharField(max_length=10)

In second file I've got:

from multiprocessing.dummy import Pool as ThreadPool
from myapp.models import VM

def multi(vm): 
    summary = vm.summary
    state = summary.runtime.powerState
    vm_name = summary.config.name
    p=VM(name=vm_name, status = state)
    p.save()

Method save() is built-in Django.models method for inserting rows in db and I think that it opens file descriptors to db.

def my_function():
    x=0
    while(x<100):
        pool = ThreadPool(3)
        pool.map(multi, vms)
        pool.close()
        pool.join()
        x=x+1
        sleep(1)`

Basically my_function refresh status of Vms and update it in database. My problem is that even after closing pool there are open file descriptors to db.sqlite3 and after 20 minutes I've got I/O exception because there are more than 1000 open descriptors to one file. I don't know why Django or python doesn't close it. This is what lsof command returns:

[root@PC ~]# lsof | grep db.sqlite3

python 6924 root 6u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 7u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 8u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 9u REG 253,0 2576 1581352 /opt/projects/mail/mvp/db.sqlite3-journal

python 6924 root 10u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 11u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 12r REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 14u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 16u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 17u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 18u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 19u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3

python 6924 root 20u REG 253,0 1143808 1581361 /opt/projects/mail/mvp/db.sqlite3`

And longer the app works , more descriptors are opened. Is there any solution for that? Is there any way to close this descriptors and clean after pool workers?

0

There are 0 answers