Why are my tests failing when run on the build server?

Question

My team is building out a regression suite of SpecFlow tests for one of our websites and have the tests working locally. We've been trying to get the build server (TFS 2013 build server) to be able to run these tests when we check in code as well. We haven't been able to get this to work.

Locally we have our tests running on the NUnit framework, which we thought would work the same on the build server. It seems the build server only wants to run the tests on VS Test Runner which is failing to run our SpecFlow tests every time.

If I log into the build server remotely and run the SpecFlow tests in VS against SpecRun the tests work and pass. We've been playing around with the build definition as well, but haven't had any luck. What needs to be set up to get this to run when code is checked in?

Here is our pattern matching for our SpecFlow test project:

And here is an image of our build output as VS Test Runner is trying to execute one of our SpecFlow tests:

UPDATE: We've got our tests to run on the build server. However, our tests that are actually trying to hit our web sites keep generating the following error:

Test method AccessTheNDCHomePage threw exception: OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:7055/hub/session/ae10bda6-c46c-4c35-bd9f-f2f9400767ef/url timed out after 60 seconds. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)

After looking around online it seems like this could potentially be because of Firefox update to version 38 and/or a recent Selenium/WebDriver update. Does anyone have any information or tips on fixing this issue?

Answer 1

If you are already able, with the same solution, to build and run other nUnit tests with the build then you likley only have a pattern match issue.

The build specified a pattern of assembly names to go find test in. If you change It to included "spec" then your tests will run.

In XAML build It is on the process tab of the build definition editor. In the new build system you select the "VS Test" task and edit It there.

Answer 2

Firstly you need to understand what SpecFlow is. It is basically a unit test generation tool. It generates the unit tests in the language defined in the config.

if you wrote plain NUnit tests in your solution what would you need to do to get them to run on TFS? you would have to have a NUnit runner on the the build server and have TFS configured to use this. It's no different for SpecFlow tests.

So the solution? Either generate your tests in MSTest format locally, then when you check these tests in TFS will run them. If you don't want to do that, then set TFS up to be able to run NUnit flavour tests.

EDIT

It sounds as though when you say 'fail to run' the specflow tests, these are not failing, but not being found. Probably you need to adjust either the name of the assemblies that contain your specflow tests so that the current patterns in tfs find the test assemblies, or add filters to allow TFS to find your new assemblies.

Answer 3

Here were some of the issues and solutions we found to set up SpecFlow on our build server:

• The build server wouldn't always recognize our SpecFlow test project - Setting the Platform target of the SpecFlow test to x86 seemed to fix this issue

• WebDriver time out after 60 seconds error - I am still getting this locally and it still happens when we try to run our tests via the browser on our build server. We were able to run our tests against our Selenium Grid as well as against SauceLabs via the build server. If I find a solution to this I will update my answer.

Answer 4

I have had the same issue in the past, the answer is to make sure that your build agent is configured to run interactive.

The build agent must be configured to run interactive in order to run the selenium/coded UI tests against a browser.

https://msdn.microsoft.com/en-us/library/ms181712.aspx#interactive