WebClient apears to add headers to request on its own. Where does it happen?

Question

Here's a failing test:

// these are pretty basic dependencies so I won't include imports or pom
    @Test
    void testIfNoHeadersAreAddedToRequestImplicitly() {
        Mono<Map<String, Object>> responseMono = WebClient.builder()
                .baseUrl("https://httpbin.org")
                .build()
                .get()
                .uri("/headers")
                .retrieve()
                .bodyToMono(new ParameterizedTypeReference<>() {});
        StepVerifier.create(responseMono.map(m -> m.get("headers")).cast(Map.class))
                .expectNextMatches(Map::isEmpty)
                .verifyComplete();
    }

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.1.5</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
<!-- ... -->
    <properties>
        <java.version>17</java.version>
        <spring-cloud.version>2022.0.4</spring-cloud.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter</artifactId>
        </dependency>
        <!-- ↓ includes WebFlux starter -->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-gateway</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.wiremock</groupId>
            <artifactId>wiremock-standalone</artifactId>
            <version>3.5.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>io.projectreactor</groupId>
            <artifactId>reactor-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

I'm pretty sure WebClient adds some default request headers, such as HOST, and I want to see where it happens. It so happens I need to disable this feature, but I'm just as eager to see the cause

So far, my debugging of this issue was unsuccessful – including debugging of DefaultWebClient.exchange() which seems to work as expected

However, I discovered that a wrong request is passed to this callback

// org.springframework.http.client.reactive.ReactorClientHttpConnector.connect(..)

        return requestSender
                // put a breakpoint on the lambda
                .send((request, outbound) -> requestCallback.apply(adaptRequest(method, uri, request, outbound)))

When I dug deeper, Netty just pretended those headers were all along (once any request appears, it already has those default headers)

The issue is reproduced with WireMock too so it's unlikely to have anything to do with httpbin (btw, in case you're unfamiliar with it, you may visit the API's page)

Answer 1

The default headers are explicitly added by Netty when a request is prepared for sending. The main point of interest is here:

// io.netty.handler.codec.http.HttpHeaders
public interface HttpHeaders extends Iterable<Map.Entry<String, String>> {
    HttpHeaders add(String name, Object value);
    // ...
}

As you might know, Spring's WebClient uses Project Reactor and Netty under the hood for making web requests. When you create a WebClient instance and build a request with it, the headers do not appear immediately. However, once the request is about to be sent out (when Netty's HttpClientOperations comes into play), Netty adds these default headers (like HOST, Accept-Encoding, etc) even if none were explicitly set by you.

Here's a look into part of HttpClientOperations from Netty on GitHub:

// io.netty.handler.codec.http.HttpClientOperations
@Override
protected void onOutboundSubscribe(Subscriber<? super ByteBuf> s) {
    if (!HttpRequestEncoder.encoderHeader(headers(), method(), uri())) {
        if (log.isDebugEnabled()) {
            log.debug(format(ctx.channel(), "Dropped {}"), this);
        }
        return;
    }
    //...
}

The HttpRequestEncoder.encoderHeader(...) part is where the headers are prepared and added if they don't exist. You might take a look at the encodeHeaders method too for deeper insights.

Here is the point-in-time in HttpClientOperations where headers are added:

// io.netty.handler.codec.http.HttpHeaders
public interface HttpHeaders extends Iterable<Map.Entry<String, String>> {
    HttpHeaders add(String name, Object value);
    // ...
}

Unfortunately, WebClient does not provide an out-of-the-box solution to prevent Netty from adding these headers. However, you can create a custom filter and add it to your WebClient to intentionally remove these headers each time a request is built:

WebClient.builder()
    .baseUrl("https://httpbin.org")
    .filter((clientRequest, next) -> {
        ClientRequest mutatedRequest = ClientRequest.from(clientRequest)
            .headers(httpHeaders -> {
                httpHeaders.remove("Host");
                // remove other headers if needed
                })
            .build();
        return next.exchange(mutatedRequest);
    })
    .build()
    .get()
    // continue with your code for .uri, etc.

This way, each time you build a request, the filter will remove the Host header (and others if you wish) before sending it off. Please note that this might not be the best practice as some servers might require the Host header or others for processing the request.

Answer 2

Using Malvin Lok's answer as a lead, I located io.netty.handler.codec.http.HttpHeaders and put a breakpoint for each set*(..) and add*(..) method. Here's what I found

Default headers are added one by one in different places of Netty code. It's hard-coded, there are no injected strategies or configs. There's apparently nothing you can do about it

1. Accept-Encoding

// reactor.netty.http.client.HttpClient

    public final HttpClient compress(boolean compressionEnabled) {
        if (compressionEnabled) {
            if (!configuration().acceptGzip) {
                HttpClient dup = duplicate();
                HttpHeaders headers = configuration().headers.copy();
                headers.add(HttpHeaderNames.ACCEPT_ENCODING, HttpHeaderValues.GZIP);
                dup.configuration().headers = headers;
                dup.configuration().acceptGzip = true;
                return dup;
            }
        }

2, 3, 4. User-Agent, Host, Accept

// reactor.netty.http.client.HttpClientConnect.HttpClientHandler

        Publisher<Void> requestWithBody(HttpClientOperations ch) {
            try {
                ch.resourceUrl = this.resourceUrl;
                ch.responseTimeout = responseTimeout;

                UriEndpoint uri = toURI;
                HttpHeaders headers = ch.getNettyRequest()
                                        .setUri(uri.getPathAndQuery())
                                        .setMethod(method)
                                        .setProtocolVersion(HttpVersion.HTTP_1_1)
                                        .headers();

                ch.path = HttpOperations.resolvePath(ch.uri());

                if (!defaultHeaders.isEmpty()) {
                    headers.set(defaultHeaders);
                }

                if (!headers.contains(HttpHeaderNames.USER_AGENT)) {
                    headers.set(HttpHeaderNames.USER_AGENT, USER_AGENT);
                }

                SocketAddress remoteAddress = uri.getRemoteAddress();
                if (!headers.contains(HttpHeaderNames.HOST)) {
                    headers.set(HttpHeaderNames.HOST, resolveHostHeaderValue(remoteAddress));
                }

                if (!headers.contains(HttpHeaderNames.ACCEPT)) {
                    headers.set(HttpHeaderNames.ACCEPT, ALL);
                }

If you specify your own Host header

// like so

        Mono<Map<String, Object>> responseMono = WebClient.builder()
                .baseUrl("https://httpbin.org")
                .build()
                .get()
                .uri("/headers")
                .header(HttpHeaders.HOST, "fake-host")
                .retrieve()
                .bodyToMono(new ParameterizedTypeReference<>() {});

Netty will be unaware of it (that if will still evaluate to true), but the server will still get your "fake" host because at some point, your custom headers and Netty's default headers are merged by Spring. Custom headers overwrite those of Netty's

In the code below, Spring creates a Flux of "commit actions" which Netty will subscribe to (indirectly, through multiple wrapper layers). By doing so, it will trigger execution of all commit action Runnables, including the one that merges headers (see applyHeaders())

// org.springframework.http.client.reactive.AbstractClientHttpRequest

    /**
     * Apply {@link #beforeCommit(Supplier) beforeCommit} actions, apply the
     * request headers/cookies, and write the request body.
     * @param writeAction the action to write the request body (may be {@code null})
     * @return a completion publisher
     */
    protected Mono<Void> doCommit(@Nullable Supplier<? extends Publisher<Void>> writeAction) {
        if (!this.state.compareAndSet(State.NEW, State.COMMITTING)) {
            return Mono.empty();
        }

        this.commitActions.add(() ->
                Mono.fromRunnable(() -> {
                    applyHeaders();
                    applyCookies();
                    this.state.set(State.COMMITTED);
                }));

        if (writeAction != null) {
            this.commitActions.add(writeAction);
        }

        List<Publisher<Void>> actions = new ArrayList<>(this.commitActions.size());
        for (Supplier<? extends Publisher<Void>> commitAction : this.commitActions) {
            actions.add(commitAction.get());
        }

        return Flux.concat(actions).then();
    }

// org.springframework.http.client.reactive.ReactorClientHttpRequest

    @Override
    protected void applyHeaders() {
        getHeaders().forEach((key, value) -> this.request.requestHeaders().set(key, value));
    }