Flutter/Dart slow unit tests: each file taking >4s to start running

Question

My one test file with 7 empty tests is taking 5+ seconds to start running. I've used both flutter test and the Dart test runner in VS Code, and they both take about the same amount of time.

My whole test suite of ~150 tests takes over 30 seconds to complete, because each test file is taking 3-7 seconds just to start running.

launch.json (for Dart test runner)

    {
      "name": "Dart: Run all tests",
      "type": "dart",
      "request": "launch",
      "program": "./test/"
    },

Timed test run:

00:05 +1 -7: Some tests failed.
real    0m9.532s
user    0m0.075s
sys     0m0.138s

I am using nested group blocks, but I don't believe this should cause such an enormous delay that makes TDD extremely slow

Example section of code

void main() {
  group('AuthCubit:', () {

    late AuthCubit          authCubit;
    late MockLogIn          logIn;
    late MockLogOut         logOut;
    late MockRestoreSession restoreSession;

    setUp(() {
      logIn         = MockLogIn();
      logOut        = MockLogOut();
      restoreSession= MockRestoreSession();
      authCubit     = AuthCubit(
        logIn         : logIn,
        logOut        : logOut,
        restoreSession: restoreSession,
      );
    });

    tearDown(() {
      authCubit.close();
    });

    test('initial state is unauthenticated', () async {
      expect(authCubit.state, equals(const AuthStateUnauthenticated()));
    });
    group('logIn():', () {
      //? Create late variables that all stages use, if needed
      setUp(() {});
      //? Create subgroups for stages this function will step through
      group('On failure:', () {
        setUp(() {});
        test('emits unauthenticated state', () async { throw UnimplementedError(); });
      });
      group('On success:', () {
        setUp(() {});
        test('emits authenticated state', () async { throw UnimplementedError(); });
      });
    });
  });
  // ... + similar empty tests for other functions ...
}

---

I've tried even testing two empty example test files both separately and combined (see below code) both take 4+ seconds to run individually. If I combine both test files into one file, the execution time is the almost the same as just one test.

The problem seems to be around launching each test file, not the tests themselves.

Tests

test_test_1.dart

import 'package:flutter_test/flutter_test.dart';

//? Create Mocks, Fakes, etc, if needed

void main() {
  group('Test File 1:', () {
    //? Create late variables that all functions use, if needed
    test('empty test 1', () async { throw UnimplementedError(); });
    test('empty test 2', () async { throw UnimplementedError(); });
    test('empty test 3', () async { throw UnimplementedError(); });
    test('empty test 4', () async { throw UnimplementedError(); });
    test('empty test 5', () async { throw UnimplementedError(); });
  });
}

test_test_2.dart

import 'package:flutter_test/flutter_test.dart';

//? Create Mocks, Fakes, etc, if needed

void main() {
  group('Test File 2:', () {
    //? Create late variables that all functions use, if needed
    test('empty test 1', () async { throw UnimplementedError(); });
    test('empty test 2', () async { throw UnimplementedError(); });
    test('empty test 3', () async { throw UnimplementedError(); });
    test('empty test 4', () async { throw UnimplementedError(); });
    test('empty test 5', () async { throw UnimplementedError(); });
  });
}

test_test_1_2.dart

import 'package:flutter_test/flutter_test.dart';

//? Create Mocks, Fakes, etc, if needed

void main() {
  group('Test File 1:', () {
    //? Create late variables that all functions use, if needed
    test('empty test 1', () async { throw UnimplementedError(); });
    test('empty test 2', () async { throw UnimplementedError(); });
    test('empty test 3', () async { throw UnimplementedError(); });
    test('empty test 4', () async { throw UnimplementedError(); });
    test('empty test 5', () async { throw UnimplementedError(); });
  });
  group('Test File 2:', () {
    //? Create late variables that all functions use, if needed
    test('empty test 1', () async { throw UnimplementedError(); });
    test('empty test 2', () async { throw UnimplementedError(); });
    test('empty test 3', () async { throw UnimplementedError(); });
    test('empty test 4', () async { throw UnimplementedError(); });
    test('empty test 5', () async { throw UnimplementedError(); });
  });
}

Results

test_test_1.dart individual result

00:04 +0 -5: Some tests failed.

real    0m8.743s
user    0m0.060s
sys     0m0.167s

`test_test_2.dart` individual result

00:05 +0 -5: Some tests failed.

real    0m8.982s
user    0m0.137s
sys     0m0.106s

`test_test_1_2.dart` individual result

00:04 +0 -10: Some tests failed.

real    0m8.602s << Note this is actually FASTER than the smaller test files
user    0m0.045s << same ^
sys     0m0.200s

All 3 test file results (in one run)

00:08 +0 -20: Some tests failed.

real    0m12.696s
user    0m0.015s << Weirdly, this is the smallest value out of all test cases
sys     0m0.152s

Questions

What could be the issue with my code, setup (software or hardware), with Dart/Flutter, or anything else?

Pretend I know nothing about Flutter, Dart, or VS Code; what questions would help find the potential causes and solutions for this?

Answer 1

jensjoha on GitHub helped me find a workaround to this by creating "nested aggregate test files"

On a higher level note I should probably add that for each file you have tests in, running tests are going to be slower because it has to do a compile, launch a new process etc. Similarly - as we saw - getting to the point where it actually starts doing something with the tests takes quite a while, so running flutter test specificfile.dart (at least for many specific files) isn't great. As somewhat of a workaround - when you want to run all tests - you could try to create one test file like done in #86722 (comment) and only run that.

Example

// test_group_1_test.dart

import test_file_1_test.dart as testFile1;
import test_file_2_test.dart as testFile2;
// ...

void main() {
  group("test file 1", testFile1.main);
  group("test file 2", testFile2.main);
  // ...
}

// everything_test.dart

import test_group_1_test.dart as testGroup1
import test_group_2_test.dart as testGroup2
// ...

void main() {
  group("test group 1", testGroup1.main);
  group("test group 2", testGroup2.main);
  // ...
}

With this structure, I can run all (or any group of) test files within the same time it takes to run a single test file, which for my case is ~75% faster.