Unusual error in Django tests wrongly thinking instance doesn't exist

Question

I have very customised Django app that checking to see if a user can change a ForeignKey to certain values.

In this instance, a User belongs to a Workgroup and an Item can also belong to a Workgroup, and consequently, when a User makes an Item, they can only put it in Workgroups they belong to. Complicating matters Item is a parent class, so there are lots of types of "Item".

At present I have a custom admin form setup to check this:

class AdminConceptForm(autocomplete_light.ModelForm):
    def __init__(self, *args, **kwargs):
        #... other code
        self.fields['workgroup'].queryset = self.request.user.profile.editable_workgroups.all()

The important bits of this test are:

def setUp(self):
    from django.test import Client

    self.client = Client()
    self.wg1 = models.Workgroup.objects.create(name="Test WG 1") # Editor is member

def test_editor_change_item(self):
    self.login_editor()
    response = self.client.get(reverse("admin:%s_%s_change"%(self.itemType._meta.app_label,self.itemType._meta.model_name),args=[self.item1.pk]))
    self.assertResponseStatusCodeEqual(response,200)

    updated_item = dict((k,v) for (k,v) in model_to_dict(self.item1).items() if v is not None)
    updated_name = updated_item['name'] + " updated!"
    updated_item['name'] = updated_name

    updated_item.update({
        'statuses-TOTAL_FORMS': 0, 'statuses-INITIAL_FORMS': 0 #no statuses
    })
    updated_item.update(self.form_defaults)
    self.assertTrue(self.wg1 in self.editor.profile.myWorkgroups)

    self.assertEqual([self.wg1],list(response.context['adminform'].form.fields['workgroup'].queryset))

    self.assertTrue(perms.user_can_edit(self.editor,self.item1))
    self.assertTrue(self.item1.workgroup in self.editor.profile.editable_workgroups.all())

    response = self.client.post(
            reverse("admin:%s_%s_change"%(self.itemType._meta.app_label,self.itemType._meta.model_name),args=[self.item1.pk]),
            updated_item
            )

# HERE IS WHERE THE FAILURE IS!!!
    self.assertResponseStatusCodeEqual(response,302)

    self.item1 = self.itemType.objects.get(pk=self.item1.pk)
    self.assertEqual(self.item1.name,updated_name)

But sometimes (and intermittently), when I run the test suite, I post to this form to test saving content, and I get this error:

======================================================================
FAIL: test_editor_change_item (aristotle_mdr.tests.test_extension_api.QuestionAdmin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/aristotle-mdr/aristotle-metadata-registry/aristotle_mdr/tests/test_admin_pages.py", line 285, in test_editor_change_item
    self.assertResponseStatusCodeEqual(response,302)
  File "/home/travis/build/aristotle-mdr/aristotle-metadata-registry/aristotle_mdr/tests/utils.py", line 501, in assertResponseStatusCodeEqual
    self.assertEqual(response.status_code, code)
AssertionError: 200 != 302

Because of the nature of this, if posted successfully, the page should redirect, and I have some code that just spits of the response HTML if this isn't the case, and in those cases I get this:

<label class="required" for="id_workgroup">Workgroup</label>
<select id="id_workgroup" name="workgroup">
    <option value="">---------</option>
    <option value="17" selected="selected">Test WG 1</option>
</select>
<ul class="errorlist">
    <li>workgroup instance with pk 17 does not exist.</li>
</ul>

However, when this error fires, not every item type will throw the error, only one or two. But if you look at the select field, the workgroup with id (or pk) 17 is there! Plus, when I rerun the test suite it will be fine (sometimes afer a few "warm-up" goes). I've also never encountered this in a non-test site.

I think this might be due to the way Django tests are kept in transactions? I'm starting to get annoyed with this, as it used to be quite intermittent but now its getting more frequent - but still random.

---

So this is still failing and I can say what doesn't fix it:

• Using a file-based SQLite instance instead of in-memory
• Using PostgreSQL for tests
• Switching from TestCase to TransactionTestCase

What I know:

• Tests run fine on the development serve, but the same tests fail on Travis-CI
• Its not just the calls to the test web Client, but also some other querysets work
• It might be Transaction based, but I'm not sure.

And for the ultra curious here is the issue I'm trying to quash but can't.

---

Edit: 2015-06-11

I have built a failing, self contained example!! SQLite consistently works, Postgres consistently fails.

It seems that for some reason this code is consistently bad:

    def test_bar(self):
        # This test will always work
        print("Do Bar")
        self.do_foo()
        print("Bar done")
    def test_foo(self):
        # This test will always work
        print("Do Foo")
        self.login_editor()
        response = self.client.get(reverse("admin:%s_%s_changelist"%(self.itemType._meta.app_label,self.itemType._meta.model_name)))
        self.assertResponseStatusCodeEqual(response,200)
        self.do_foo()
        print("Foo done")
    def test_zip(self):
        # This test will always FAIL
        print("Do Zip")
        self.do_foo()
        print("Zip done")

In fact, calling an admin changelist view will always cause any subsequent admin pages to fail on Postgres when trying to save as the Workgroup no longer appears in querysets. Now, why is that?

The full code:

class MinimalExample(TestCase):
    itemType=models.ObjectClass
    form_defaults = {}
    create_defaults = {}
    def setUp(self):
        self.wg1 = models.Workgroup.objects.create(name="Test WG")

        self.editor = User.objects.create_user('eddie','','editor')
        self.editor.is_staff=True
        self.editor.save()

        self.wg1.submitters.add(self.editor)

        self.assertEqual(self.editor.profile.editable_workgroups.count(),1)
        self.item1 = self.itemType.objects.create(name="admin_page_test_oc",description=" ",workgroup=self.wg1,**self.create_defaults)
    def logout(self):
        self.client.post(reverse('django.contrib.auth.views.logout'), {})

    def login_editor(self):
        self.logout()
        response = self.client.post(reverse('friendly_login'), {'username': 'eddie', 'password': 'editor'})
        self.assertEqual(response.status_code,302)
        return response
    def assertResponseStatusCodeEqual(self,response,code):
        self.assertEqual(response.status_code, code)

    def test_bar(self):
        print("Do Bar")
        self.do_foo()
        print("Bar done")
    def test_foo(self):
        print("Do Foo")
        self.login_editor()
        response = self.client.get(reverse("admin:%s_%s_changelist"%(self.itemType._meta.app_label,self.itemType._meta.model_name)))
        self.assertResponseStatusCodeEqual(response,200)
        self.do_foo()
        print("Foo done")
    def test_zip(self):
        print("Do Zip")
        self.do_foo()
        print("Zip done")
    def do_foo(self):
        url_bits = (self.itemType._meta.app_label,self.itemType._meta.model_name)
        response = self.client.post(reverse('friendly_login'), {'username': 'eddie', 'password': 'editor'})

        response = self.client.get(reverse("admin:%s_%s_add"%url_bits))

        data = {'name':"admin_page_test_oc",'description':"test","workgroup":self.wg1.id,
                    'statuses-TOTAL_FORMS': 0, 'statuses-INITIAL_FORMS': 0 #no substatuses
                }
        response = self.client.post(reverse("admin:%s_%s_add"%url_bits),data)
        self.item1 = self.itemType.objects.first()
        response = self.client.get(reverse("admin:%s_%s_change"%url_bits,args=[self.item1.id]))

        data['name'] = "updated"
        # Re post the same data
        response = self.client.post(
                reverse("admin:%s_%s_change"%url_bits,args=[self.item1.id]),
                data
                )
        print response
        self.item1 = self.itemType.objects.first() # decache
        self.assertTrue(self.item1.name == "updated")

Answer 1

Wow, what a wild and crazy ride!!

It turns out it was an issue in the RelatedListFilter with a custom queryset that was causing problems. The comments on this answer point out that:

The filter lookups are getting cached in some way

and

field and field.rel objects will persist between requests

Even across transaction rollbacks in some cases it seems!

Moral of the story, be careful when using ListFilters in the admin!

Answer 2

I'd start by moving around where you load that workgroup. Try moving the workgroup creation out of setUp and into setUpClass. This has the effect of leaving the workgroup in place for all tests in the TestCase, which is probably what you want.

def setUp(self):
    from django.test import Client
    self.client = Client()

@classmethod
def setUpClass(cls):
    cls.wg1 = models.Workgroup.objects.create(name="Test WG 1") # Editor is member
    super().setUpClass() # Python 3 version

Fixtures are generally a nightmare, but if this doesn't help I'd be interested to see if moving that workgroup to a fixture and seeing if the problem goes away.