When I run ember test and visit /tests the results are inconsistent, how can I troubleshoot why these are different?

Question

I have been working with ember for a little over a month now and I have yet to find a solution to some testing inconsistencies I have been experiencing.

The problem is that when I run ember test from the command line and visit /tests in the browser sometimes I see a different total number of tests. It seems like ember test with phantomjs as the test runner is skipping some tests. On top of that the results seem to be inconsistent as well.

For instance, I have a simple acceptance test:

import Ember from 'ember';
import startApp from '../helpers/start-app';

var App;

module('Acceptance: Login', {
  setup: function() {
    App = startApp();
  },
  teardown: function() {
    Ember.run(App, 'destroy');
  }
});

test('Page contents', function() {
  visit('/login');

  andThen(function() {
    equal(find('form.login').length, 1);
  });
});

When I visit /tests, all of my tests pass, however when I run Ember test I get one failure:

not ok 1 PhantomJS 1.9 - Acceptance: Login: Page contents
---
    actual: >
        0
    expected: >
        1
    Log: >
...

Thanks in advance for any help.

Answer 1

I had the same frustration as you until I looked a bit closer at what was being counted.

When I run my tests in a browser, it shows how many assertions are being run. When I run phantomjs (via 'ember test' command line), the log only reports how many tests are run. There can be many assertions in a test.

If I scroll to the very bottom of the page after a test run is complete in a browser, I see that the number next to the final test matches the total number of tests run in phantomjs.

As for why your test is breaking in phantomjs, it could be due to a number of things. Without seeing your handlebars and implementation it can be hard to tell, but I've seen problems with timing and also jquery binding issues that fail only in a headless browser (aka phantomjs).

If you post some more specifics, I may be able to help.