How does testing a connection to a third-party API fit into continuous integration?

Question

I wrote a test some time ago that tests an integration I wrote between my code and a third-party API. The test makes sure that the integration works properly and that we get back the expected results.

The formal build failed today because the test received a 500 error while trying to connect to the third-party API.

Does it make sense to test a situation like this?

Answer 1

In my opinion integration tests are okay to fail if the 3rd party (db, webservice etc.) isn't available. If you don't want to test the integration itself and just the plain functionality you could mock the result of your API and test against them. In that scenario your test are not depending on the 3rd party availability anymore.

I generally mark unit tests depending on a 3rd party availability with a group attribute like "Integration" and exclude them from the continous integration process. Instead I let them run in a nightly build. Integration tests are generally expensive in time and the whole point of continuous integration is to provide rapid feedback.

Answer 2

No, it isn't helpful for your test suite to fail when a third-party service is down. But you do want to fully test your integration with the service. The following strategy has worked well for me:

• Isolate the third-party service in a module (let's say a class, but however your language modularizes is fine) which does as little as possible besides encapsulating the connection to the service. Write methods on the class so that it's possible to tell the difference between errors which are the service's fault and errors which are your fault. For example, give service-down exceptions a common superclass.

• Write unit tests of the service class so that

  • if a service-down error occurs, tests pass but log the error.

  • if any other error occurs, fail as usual.

  Write these tests so that they run against the live service. There should be a small number of these tests, so they should not create an issue for test suite runtime.

• Stub or mock the service class out of all other tests, including integration tests. (By "integration tests" here I'm talking about tests that test all layers of your own code, not whether they interact with the third-party service.)

  When stubbing or mocking the service class out of integration tests, don't stub or mock the public API of the service class, but rather stub or mock some lower level, perhaps an internal method of the service class or the library that you use to connect to the service. This prevents integration bugs when calling the service class. In unit tests, stub or mock the service class's public API as you would any class.

As Martin Buberl suggests, it might be helpful to have a separate test run of integration tests where the service class is not stubbed or mocked out. To be honest, this is something that has been discussed in multiple projects I've been involved in but I don't think it's ever been done. When a third-party service fails, we always find out about it from production errors (reported by monitoring or customer support) before we find out about it from tests.