Mocking/Testing HTTP Get Request

Question

I'm trying to write unit tests for my program and use mock data. I'm a little confused on how to intercept an HTTP Get request to a URL.

My program calls a URL to our API and it is returned a simple XML file. I would like the test to instead of getting the XML file from the API online to receive a predetermined XML file from me so that I can compare the output to the expected output and determine if everything is working correctly.

I was pointed to Mockito and have been seeing many different examples such as this SO post, How to use mockito for testing a REST service? but it's not becoming clear to me how to set it all up and how to mock the data (i.e., return my own xml file whenever the call to the URL is made).

The only thing I can think of is having another program made that's running locally on Tomcat and in my test pass a special URL that calls the locally running program on Tomcat and then return the xml file that I want to test with. But that just seems like overkill and I don't think that would be acceptable. Could someone please point me in the right direction.

private static InputStream getContent(String uri) {

    HttpURLConnection connection = null;

    try {
        URL url = new URL(uri);
        connection = (HttpURLConnection) url.openConnection();
        connection.setRequestMethod("GET");
        connection.setRequestProperty("Accept", "application/xml");

        return connection.getInputStream();
    } catch (MalformedURLException e) {
        LOGGER.error("internal error", e);
    } catch (IOException e) {
        LOGGER.error("internal error", e);
    } finally {
        if (connection != null) {
            connection.disconnect();
        }
    }

    return null;
}

I am using Spring Boot and other parts of the Spring Framework if that helps.

Answer 1

Part of the problem is that you're not breaking things down into interfaces. You need to wrap getContent into an interface and provide a concrete class implementing the interface. This concrete class will then need to be passed into any class that uses the original getContent. (This is essentially dependency inversion.) Your code will end up looking something like this.

public interface IUrlStreamSource {
    InputStream getContent(String uri)
}

public class SimpleUrlStreamSource implements IUrlStreamSource {
    protected final Logger LOGGER;

    public SimpleUrlStreamSource(Logger LOGGER) {
      this.LOGGER = LOGGER;
    }

    // pulled out to allow test classes to provide
    // a version that returns mock objects
    protected URL stringToUrl(String uri) throws MalformedURLException {
        return new URL(uri);
    }

    public InputStream getContent(String uri) {

        HttpURLConnection connection = null;

        try {
            Url url = stringToUrl(uri);
            connection = (HttpURLConnection) url.openConnection();
            connection.setRequestMethod("GET");
            connection.setRequestProperty("Accept", "application/xml");

            return connection.getInputStream();
        } catch (MalformedURLException e) {
            LOGGER.error("internal error", e);
        } catch (IOException e) {
            LOGGER.error("internal error", e);
        } finally {
            if (connection != null) {
                connection.disconnect();
            }
        }

        return null;
    }
}

Now code that was using the static getContent should go through a IUrlStreamSource instances getContent(). You then provide to the object that you want to test a mocked IUrlStreamSource rather than a SimpleUrlStreamSource.

If you want to test SimpleUrlStreamSource (but there's not much to test), then you can create a derived class that provides an implementation of stringToUrl that returns a mock (or throws an exception).

Answer 2

The answer depends on what you are testing.

If you need to test the processing of the InputStream

If getContent() is called by some code that processes the data returned by the InputStream, and you want to test how the processing code handles specific sets of input, then you need to create a seam to enable testing. I would simply move getContent() into a new class, and inject that class into the class that does the processing:

public interface ContentSource {

  InputStream getContent(String uri);
}

You could create a HttpContentSource that uses URL.openConnection() (or, better yet, the Apache HttpClientcode).

Then you would inject the ContentSource into the processor:

public class Processor {
  private final ContentSource contentSource;

  @Inject
  public Processor(ContentSource contentSource) {
    this.contentSource = contentSource;
  }

  ...
}

The code in Processor could be tested with a mock ContentSource.

If you need to test the fetching of the content

If you want to make sure that getContent() works, you could create a test that starts a lightweight in-memory HTTP server that serves the expected content, and have getContent() talk to that server. That does seem overkill.

If you need to test a large subset of the system with fake data

If you want to make sure things work end to end, write an end to end system test. Since you indicated you use Spring, you can use Spring to wire together parts of the system (or to wire the entire system, but with different properties). You have two choices

1. Have the system test start a local HTTP server, and when you have your test create your system, configure it to talk to that server. See the answers to this question for ways to start the HTTP server.

2. Configure spring to use a fake implementation of ContentSource. This gets you slightly less confidence that everything works end-to-end, but it will be faster and less flaky.

Answer 3

The other answers in here advise you to refactor your code to using a sort of provider which you can replace during your tests - which is the better approach.

If that isn't a possibility for whatever reason you can install a custom URLStreamHandlerFactory that intercepts the URLs you want to "mock" and falls back to the standard implementation for URLs that shouldn't be intercepted.

Note that this is irreversible, so you can't remove the InterceptingUrlStreamHandlerFactory once it's installed - the only way to get rid of it is to restart the JVM. You could implement a flag in it to disable it and return null for all lookups - which would produce the same results.

URLInterceptionDemo.java:

public class URLInterceptionDemo {

    private static final String INTERCEPT_HOST = "dummy-host.com";

    public static void main(String[] args) throws IOException {
        // Install our own stream handler factory
        URL.setURLStreamHandlerFactory(new InterceptingUrlStreamHandlerFactory());

        // Fetch an intercepted URL
        printUrlContents(new URL("http://dummy-host.com/message.txt"));
        // Fetch another URL that shouldn't be intercepted
        printUrlContents(new URL("http://httpbin.org/user-agent"));
    }

    private static void printUrlContents(URL url) throws IOException {
        try(InputStream stream = url.openStream();
            BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
            String line;
            while((line = reader.readLine()) != null) {
                System.out.println(line);
            }
        }
    }

    private static class InterceptingUrlStreamHandlerFactory implements URLStreamHandlerFactory {

        @Override
        public URLStreamHandler createURLStreamHandler(final String protocol) {
            if("http".equalsIgnoreCase(protocol)) {
                // Intercept HTTP requests
                return new InterceptingHttpUrlStreamHandler();
            }
            return null;
        }
    }

    private static class InterceptingHttpUrlStreamHandler extends URLStreamHandler {

        @Override
        protected URLConnection openConnection(final URL u) throws IOException {
            if(INTERCEPT_HOST.equals(u.getHost())) {
                // This URL should be intercepted, return the file from the classpath
                return URLInterceptionDemo.class.getResource(u.getHost() + "/" + u.getPath()).openConnection();
            }
            // Fall back to the default handler, by passing the default handler here we won't end up
            // in the factory again - which would trigger infinite recursion
            return new URL(null, u.toString(), new sun.net.www.protocol.http.Handler()).openConnection();
        }
    }
}

dummy-host.com/message.txt:

Hello World!

When run, this app will output:

Hello World!
{
  "user-agent": "Java/1.8.0_45"
}

It's pretty easy to change the criteria of how you decide which URLs to intercept and what you return instead.