Why some testcases fail in Pytest even after their dependencies have run successfully before them?

413 views Asked by At

I have been trying to run some testcases which are dependent on other testcases. So, I made sure to use the pytest.mark.dependency marker and also made sure that the dependencies (the ones upon which the cases are dependent) execute before the dependents. Below is a snippet of my code:

class TestB(TestCase):
   
    @pytest.mark.dependency(depends=["test_third"])
    def test_fourth(self):
        assert True

    @pytest.mark.dependency()
    def test_third(self):
        assert True

    def test_fifth(self):
        assert True

In the above code, test_fourth is dependent upon test_third, so I found a code snippet that would help in reordering the cases on the basis of their dependencies. Below is the snippet:

conftest.py/

def pytest_collection_modifyitems(config, items):
    """
    After collection, this will reorder the function test ordering by dependency.
    Here's what we're gonna do:
        bucket the functions by their parents
        For each parent bucket, iterate over all functions, keeping track of dependency tree.
        Start doing BFS down the dependency tree.
    TODO: check for circular dependency
    TODO: make the files test in the same order
    """
    # creating the parent buckets
    function_buckets = {}
    for item in items:
        print(item.parent.name)
        if item.parent.name not in function_buckets:
            function_buckets[item.parent.name] = []
        function_buckets[item.parent.name].append(item)

    master_sorted_list = []

    # sorting the functions within each file
    # f_items = file items: test functions within this test_file
    for parent, f_items in function_buckets.items():
        item_dict = {item.name: item for item in f_items}  # item.name -> item
        # removing duplicate function names. Python >3.7 preserves ordering for
        #   dict insertion
        func_names = list(dict.fromkeys([item.name for item in f_items]))

        # building the dependency tree
        dependency_tree = {}  # item.name --> [names]
        for item in f_items:
            dependencies = [
                f.kwargs["depends"]
                for f in item.iter_markers(name="dependency")
                if "depends" in f.kwargs
            ]
            dependency_tree[item.name] = dependencies
            
        # BFS through the list
        # we're gonna use the items (funcs within this file)
        #   as the set of functions we iterate through
        #       we don't end until this list is empty
        # for each function: it either depends on a function in our remaining set, or it doesn't
        #   if it doesn't: put it in the list:
        #       none of the functions this function depends on will run after this.
        #   if it does: put at the end of the remaining functions (items)
        #       this way, we can fit the dependent functions first
        sorted_list = []
        while func_names:
            func_name = func_names[0]
            dependent_functions = dependency_tree[func_name]

            if len(dependent_functions) == 1:
                dependent_functions = dependent_functions[0]

            if all([f not in func_names for f in dependent_functions]):
                sorted_list.append(item_dict[func_names[0]])
                func_names = func_names[1:]  # purge it cause we're done with it
            else:
                # shuffle this entry to the back
                func_names = func_names[1:] + [func_names[0]]

        master_sorted_list.extend(sorted_list)

    assert len(master_sorted_list) == len(items)
    items[:] = master_sorted_list

Though it does reorder the cases and test_third runs before test_fourth, test_fourth is skipped even though test_third passed SUCCESSFULLY! I have attached the results below:

>> pytest -s -v 
================================= test session starts ===========================================
platform linux -- Python 3.9.7, pytest-7.0.1, pluggy-1.0.0 -- /usr/local/python/python-
collected 3 items                                                                                                                
...

tests/test_practise.py::TestB::test_fifth PASSED
tests/test_practise.py::TestB::test_third PASSED
tests/test_practise.py::TestB::test_fourth SKIPPED (test_fourth depends on test_third)

Note: My testcases are in a class and are not standalone cases, since keeping the cases in the classes is my use case. For standalone cases this code seems to work fine.

Can anyone understand the problem they can see here, or is there any mistake in my conftest.py code snippet because of which it's not able to cater to my usecase?

0

There are 0 answers