What is the purpose of setup, teardown and cycles in Benchmark.js?

Question

I know the formal descriptions:

Setup: create the expected state for the test.
Teardown: Do necessary clean-up operations.

However, why is this necessary, especially in Benchmark.js ? Why the need for different test cycles (as defined in Benchmark.js in this article) ? I've observed that in all cases I can think of (and I think in all other cases as well), you can just move the set-up code to the preparation code (code outside of the benchmark/test), and perhaps the teardown code to the end of your code, and the functionality is essentially the same (I've looked at a few jsperf.com tests as well, and as far as I know, this is true of them as well).

For example, here is a benchmark I created, this version uses Setups and Teardowns:

const bench = new Benchmark(
  'TicTacToeBenchmark',
  // The function to test
  () => {
    ticTacToe.addEvent(
      'turn',
      player => {
        turnText.innerHTML =
          'It\'s ' + (player['id'] === 1 ? 'X' : 'O') + '\'s turn.';
      }
    );
  },
  {
    'setup': () => {
      const players = [
        {
          char: '✕',
          className: 'playerX',
          id: 1,
        },
        {
          char: '◯',
          className: 'playerY',
          id: 2,
        },
      ];
      const ticTacToe = new TicTacToe(3, players);
    }
  }
);

bench.run();

console.log(bench); // 'mean' is 5e-7 seconds

Same example, except, everything necessary for the test is declared with the rest of the page:

const players = [
  {
    char: '✕',
    className: 'playerX',
    id: 1,
  },
  {
    char: '◯',
    className: 'playerY',
    id: 2,
  },
];
const ticTacToe = new TicTacToe(3, players);

const bench = new Benchmark(
  'TicTacToeBenchmark',
  // The function to test
  () => {
    ticTacToe.addEvent(
      'turn',
      player => {
        turnText.innerHTML =
          'It\'s ' + (player['id'] === 1 ? 'X' : 'O') + '\'s turn.';
      }
    );
  }
);

bench.run();

console.log(bench); // 'mean' is 7e-7 seconds

Mayhaps the difference is more discernible in Unit Testing ? I don't know. Could you please provide a few cases where this would be different ? Or explain why the tests have to be run within iterations which are run within cycles (essentially, like 2 loops, cycles being the outer one) ?

All content that I can find online on this topic is basically a re-utterance of the definitions of Setup and Teardown with different wording, and unfortunately, there is no Wikipedia entry for this topic.

Answer 1

Setup and teardown is where you put code that (1) needs to run before or after the function you want to benchmark but (2) don't want to include in the benchmark's measurement.

For example, let's say you have a text search library. Usage is as follows:

1. searchEngine = new New SearchEngine(pathToLargeCorpusOfText)
2. searchEngine.search(queryString)
3. searchEngine.close() - deallocates memory

If you want to benchmark the search() method in isolation, i.e. without (1) and (3) affecting the results, you put them in setup and teardown, respectively.

To accurately benchmark a function, it must be run many times. Benchmark.js runs many iterations (one call of search()) each cycle, and runs many cycles for each benchmark. I can't explain the reasons any better than the authors of Benchmark.js: Bulletproof JavaScript benchmarks by Mathias Bynens and John-David Dalton.