The order of test is not maintained if test case uses parameterise and pytest_generate_tests.

[PShiw]

While running test which user "pytest_generate_tests" from conftest.py and local parameter, the order of test cases are not appropriate.

Refer below example.

conftest.py

import pytest


def pytest_addoption(parser):
    parser.addoption("--param1", action="append", default=[])


def pytest_generate_tests(metafunc):
    if 'param1' in metafunc.fixturenames:
        param = metafunc.config.getoption('param1')
        param = list(set(param))
        metafunc.parametrize('param1', param, scope='session')


@pytest.fixture
def do_with_param(param1):
    print "running test with param1"

parameters.py

sub_param1 = {
    "test1": "Do test1",
    "test2": "Do test2",
    "test3": "Do teat3"}

test_param1.py

import pytest
from parameters import sub_param1

@pytest.mark.parametrize('sub_param', sub_param1.keys())
def test_csae1(do_with_param, sub_param):
    print "running sub_param = %s " % sub_param1[sub_param]
    assert True

test_param2.py

import pytest


def test_csae2(do_with_param):
    print "running with no sub_param"
    assert True

Refer below o/p after execution of test cases.

$ pytest -v --param1 "t1" --param1 "t2"
=== test session starts ===
platform darwin -- Python 2.7.10, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python
cachedir: .cache
sensitiveurl: .*
metadata: {'Python': '2.7.10', 'Driver': None, 'Capabilities': {}, 'Base URL': '', 'Platform': 'Darwin-17.7.0-x86_64-i386-64bit', 'Plugins': {'cov': '2.5.1', 'variables': '1.7.0', 'selenium': '1.11.1', 'html': '1.16.0', 'base-url': '1.4.1', 'metadata': '1.5.0'}, 'Packages': {'py': '1.4.34', 'pytest': '3.2.3', 'pluggy': '0.4.0'}}
rootdir: /Users/pankaj.shiwotia/eclipse-workspace/Test_Codes, inifile:
plugins: variables-1.7.0, selenium-1.11.1, metadata-1.5.0, html-1.16.0, cov-2.5.1, base-url-1.4.1
collected 8 items                                                                                                                                                                                                                       

test_param1.py::test_csae1[t2-test1] PASSED
test_param2.py::test_csae2[t2] PASSED
test_param1.py::test_csae1[t2-test3] PASSED
test_param2.py::test_csae2[t1] PASSED
test_param1.py::test_csae1[t2-test2] PASSED
test_param1.py::test_csae1[t1-test1] PASSED
test_param1.py::test_csae1[t1-test3] PASSED
test_param1.py::test_csae1[t1-test2] PASSED

==== 8 passed in 0.06 seconds ===

Here all test cases should have been executed for "t1" then for "t2" or vice-versa

Also, all test under test_param1.py should have been executed first and then should execute for test_param2.py. The behavior is observed even if the test was run using a single command line parameter. For details refer below o/p.

$ pytest -v --param1 "t1" 
=== test session starts ===
platform darwin -- Python 2.7.10, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python
cachedir: .cache
sensitiveurl: .*
metadata: {'Python': '2.7.10', 'Driver': None, 'Capabilities': {}, 'Base URL': '', 'Platform': 'Darwin-17.7.0-x86_64-i386-64bit', 'Plugins': {'cov': '2.5.1', 'variables': '1.7.0', 'selenium': '1.11.1', 'html': '1.16.0', 'base-url': '1.4.1', 'metadata': '1.5.0'}, 'Packages': {'py': '1.4.34', 'pytest': '3.2.3', 'pluggy': '0.4.0'}}
rootdir: /Users/pankaj.shiwotia/eclipse-workspace/Test_Codes, inifile:
plugins: variables-1.7.0, selenium-1.11.1, metadata-1.5.0, html-1.16.0, cov-2.5.1, base-url-1.4.1
collected 4 items                                                                                                                                                                                                                       

test_param1.py::test_csae1[t1-test1] PASSED
test_param2.py::test_csae2[t1] PASSED
test_param1.py::test_csae1[t1-test3] PASSED
test_param1.py::test_csae1[t1-test2] PASSED

==== 4 passed in 0.03 seconds ===

Here expected order is

test_param1.py::test_csae1[t1-test1] PASSED
test_param1.py::test_csae1[t1-test3] PASSED
test_param1.py::test_csae1[t1-test2] PASSED
test_param2.py::test_csae2[t1] PASSED

[asottile]

I'm having a really difficult time following the example but two things stick out to me:

@pytest.mark.parametrize('sub_param', sub_param1.keys())

param = list(set(param))

Both of these are unordered in python2.7, I suspect that is contributing to the confusion?

[PShiw]

@asottile Even if the @pytest.mark.parametrize('sub_param', sub_param1.keys()) is unordered, it should complete it first instead of going to another test.

[kam1sh]

I have the same issue in python 3.6 and pytest 3.8.1 =/
Right now I have short code example:

import pytest
def pytest_generate_tests(metafunc):
    metafunc.parametrize("x", [1, 2], scope='session')
@pytest.mark.parametrize("y", [1, 2])
def test_1(x, y):
    pass
def test_2(x):
    pass

Result:

test_1[1-1] PASSED
test_2[1] PASSED  
test_1[1-2] PASSED
test_2[2] PASSED  
test_1[2-1] PASSED
test_1[2-2] PASSED

Expected:

test_1[1-1] PASSED
test_1[1-2] PASSED
test_2[1] PASSED  
test_1[2-1] PASSED
test_1[2-2] PASSED
test_2[2] PASSED  

I'll try tomorrow write more than just test case.

[RonnyPfannschmidt]

fixture usage with scopes reorders tests to minimize setup/teardown based on fixutres and scopes

[kam1sh]

fixture usage with scopes reorders tests to minimize setup/teardown based on fixutres and scopes

Yes, but I still don't understand, what's here being optimized. If we look at ordering by usage of fixture x -- nothing changed, because of its session scope). By usage y param tests being reordered, yes, but there's nothing wrong with that, because that's just the parameter. Or a fixture with function scope, idk.

Can you explain?

[kam1sh]

I looked at source code, and that's interesting. Some intermediate order items_by_argkey could lead to the final order, and it looks like the root of the cause.

pytest/src/_pytest/fixtures.py

Lines 213 to 226 in 4474bee

	def reorder_items(items):
	argkeys_cache = {}
	items_by_argkey = {}
	for scopenum in range(0, scopenum_function):
	argkeys_cache[scopenum] = d = {}
	items_by_argkey[scopenum] = item_d = defaultdict(deque)
	for item in items:
	keys = OrderedDict.fromkeys(get_parametrized_fixture_keys(item, scopenum))
	if keys:
	d[item] = keys
	for key in keys:
	item_d[key].append(item)
	items = OrderedDict.fromkeys(items)
	return list(reorder_items_atscope(items, argkeys_cache, items_by_argkey, 0))

I'm not sure though. Fixing code is harder than writing tests =)
I'll try my best to explore more later.

[RonnyPfannschmidt]

the topoligical ordering pytest tries is tricky - i wish we had something more comprehensible instead

[blueyed]

@kam1sh
Just out of curiosity: what tool are you using in the screenshot?

[kam1sh]

@blueyed
It's birdseye, cool and useful thing.
https://github.com/alexmojaki/birdseye