Unexpected Content-Type Alterations -- consumes / produces Errors in Web Service Controller - Spring Boot 3.2.1

Question

I am encountering two problems with Content-Type handling on a Spring Boot web service migrated from Spring Boot 2.4.5 to 3.2.1. The service exposes typical create / retrieve / update / delete methods. The POST and PUT accept a JSON object and POST, GET and PUT all return a JSON object reflecting a "VoltageReading" object.

{"reading_id":379,
"espidentifier":"mdh",
"readingmv":3312,
"readingdatetime":"2024-01-02 23:24:00",
"cleardatetime":"2024-01-17 02:17:59"
}

All four methods functioned as expected under Spring Boot 2.4.5 but fail under Spring Boot 3.2.1 due to errors related to consuming incoming JSON or producing outgoing JSON responses. For incoming POST and PUT actions, the root issue appears to be that for incoming JSON, the servlet layer is ALTERING the incoming "Content-Type: application/json" to "Content-Type: application/json;charset=UTF-8" then my controller method doesn't think it can handle that, even when explicitly altered to specify consumes= "application/json;charset=UTF-8". For the GET and DELETE, the root issue appears to be that something in servlet processing is incorrectly setting Content-Type to null, which causes the Jackson library doing the serializing to not find any suitable mapper (JSON or XML).

QUESTIONS:

1. do the new Spring Boot 3.2.1 and Spring Core 6.1.13 libraries require additional annotations to produce prior default JSON processing?
2. is there a way to STOP Spring Boot from appending charset=UTF-8 to incoming "Content-Type" headers?
3. why is the mapping for incoming payloads not matching even when "application/json;charset=UTF-8" is explicitly configured?
4. what could be causing Content-Type to be passed as null when serializing responses? My underlying DAO class IS returning an object to my controller method which can serialize it in a log statement.

DETAILS:

The controller method signatures specify consumes / produces values reflecting application/json:

    @RequestMapping(value = "/voltagereading/",
                    method = RequestMethod.POST,
                    consumes = MediaType.APPLICATION_JSON,
                    produces = MediaType.APPLICATION_JSON)
    public @ResponseBody VoltageReading createVoltageReading(@RequestBody final VoltageReading newReading)  throws SQLException, Exception {

    @RequestMapping(value = "/voltagereading/{queryid}",
                    method = RequestMethod.GET,
                    produces = MediaType.APPLICATION_JSON)
    public @ResponseBody VoltageReading retrieveVoltageReading(
           @PathVariable("queryid") final int queryid,
           final HttpServletRequest theQuery,
           final HttpServletResponse theReply)  throws SQLException, Exception {

    @RequestMapping(value = "/voltagereading/{queryid}",
                      method = RequestMethod.PUT,
                      produces = MediaType.APPLICATION_JSON)
    public @ResponseBody VoltageReading updateVoltageReading(
             @PathVariable("queryid") final int queryid,
             @RequestBody final VoltageReading newReading)  throws SQLException, Exception {

    @RequestMapping(value = "/voltagereading/{queryid}",
                     method = RequestMethod.DELETE,
                     produces = MediaType.APPLICATION_JSON)
    public int deleteVoltageReading(
              @PathVariable("queryid") final int queryid)  throws SQLException, Exception {

These curl commands returned expected results under Spring Boot 2.4.5. For example, here is the GET of an individual VoltageReading:

mdh@fedora1:~/gitwork/esp32service_sb2basic $ **curl -v -X GET -u esp32controller:weakpassword http://192.168.99.10:7780/esp32service/api/voltagereading/4**
Note: Unnecessary use of -X or --request, GET is already inferred.
* processing: http://192.168.99.10:7780/esp32service/api/voltagereading/4
*   Trying 192.168.99.10:7780...
* Connected to 192.168.99.10 (192.168.99.10) port 7780
* Server auth using Basic with user 'esp32controller'
> GET /esp32service/api/voltagereading/4 HTTP/1.1
> Host: 192.168.99.10:7780
> Authorization: Basic ZXNwMzJjb250cm9sbGVyOndlYWtwYXNzd29yZA==
> User-Agent: curl/8.2.1
> Accept: */*
>
< HTTP/1.1 200
< Set-Cookie: JSESSIONID=F26931BAC03C2ABF1CB61A46E30AE386; Path=/esp32service/api; HttpOnly
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Wed, 17 Jan 2024 16:02:49 GMT
<
* Connection #0 to host 192.168.99.10 left intact
{"reading_id":4,"espidentifier":"queryme","readingmv":2112,"readingdatetime":"2024-01-11 16:29:59","cleardatetime":"2024-01-11 16:30:00"}
mdh@fedora1:~/gitwork/esp32service_sb2basic $
mdh@fedora1:~/gitwork/esp32service_sb2basic $

Here is a curl test of a POST creating a new VoltageReading:

mdh@fedora1:~/gitwork/esp32service_sb2basic $ **curl -v -X POST -d @"./test/test.reading.new.json" -u esp32controller:weakpassword --header "Content-type:application/json" http://192.168.99.10:7780/esp32service/api/voltagereading/**
Note: Unnecessary use of -X or --request, POST is already inferred.
* processing: http://192.168.99.10:7780/esp32service/api/voltagereading/
*   Trying 192.168.99.10:7780...
* Connected to 192.168.99.10 (192.168.99.10) port 7780
* Server auth using Basic with user 'esp32controller'
> POST /esp32service/api/voltagereading/ HTTP/1.1
> Host: 192.168.99.10:7780
> Authorization: Basic ZXNwMzJjb250cm9sbGVyOndlYWtwYXNzd29yZA==
> User-Agent: curl/8.2.1
> Accept: */*
> Content-type:application/json
> Content-Length: 121
>
< HTTP/1.1 200
< Set-Cookie: JSESSIONID=77281A463EDD54C89BAAFAF46DC2111D; Path=/esp32service/api; HttpOnly
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Wed, 17 Jan 2024 16:04:46 GMT
<
* Connection #0 to host 192.168.99.10 left intact
{"reading_id":381,"espidentifier":"testesp","readingmv":3312,"readingdatetime":"2024-01-02 23:24:00","cleardatetime":""}
mdh@fedora1:~/gitwork/esp32service_sb2basic $

After migrating the service to Spring Boot 3.2.1 (latest as of 1/17/2024), those same test requests encounter two different errors but both are related to interpreting Content-Type. For a GET which simply has to produce output as application/json, my underlying DAO does execute and return a VoltageReading to my controller class. The controller class then returns the object through the servlet which fails to serialize it into JSON because it cannot find a converter for a null Content-Type. Here is the exact set of logs:

2024-01-17 10:09:33 INFO com.mdhlabs.esp32service.services.Esp32ServiceController QUERY action=retrieveVoltageReading method=get reading_id=4
 2024-01-17 10:09:33 INFO com.mdhlabs.esp32service.dao.VoltageReadingsDAO retrieveVoltageReading() reading_id=4
 2024-01-17 10:09:33 INFO com.mdhlabs.esp32service.dao.VoltageReadingsDAO retrieveVoltageReading() reading_id=4 status=FOUND
 2024-01-17 10:09:33 INFO com.mdhlabs.esp32service.dao.VoltageReadingsDAO retrieveVoltageReading() ---> VoltageReading [reading_id=4 espidentifier=queryme readingmv=2112 readingdatetime=2024-01-11 16:29:59 cleardatetime=2024-01-11 16:30:00]
 2024-01-17 10:09:33 INFO com.mdhlabs.esp32service.services.Esp32ServiceController REPLY action=retrieveVoltageReading method=get reading_id=4 status=FOUND
 2024-01-17 10:09:33 ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/esp32service/api].[dispatcherServlet] Servlet.service() for servlet [dispatcherServlet] in context with path [/esp32service/api] threw exception [Request processing failed: org.springframework.http.converter.HttpMessageNotWritableException: No converter for [class com.mdhlabs.esp32service.models.VoltageReading] with preset Content-Type 'null'] with root cause
 org.springframework.http.converter.HttpMessageNotWritableException: No converter for [class com.mdhlabs.esp32service.models.VoltageReading] with preset Content-Type 'null'

Here is the curl command to confirm it originated the exact same query as before with appropriate Content-Type and Accepts values:

mdh@fedora1:~/gitwork/esp32service_sb3basic $ **curl -v -X GET -u esp32controller:weakpassword http://192.168.99.10:7780/esp32service/api/voltagereading/4**
Note: Unnecessary use of -X or --request, GET is already inferred.
* processing: http://192.168.99.10:7780/esp32service/api/voltagereading/4
*   Trying 192.168.99.10:7780...
* Connected to 192.168.99.10 (192.168.99.10) port 7780
* Server auth using Basic with user 'esp32controller'
> GET /esp32service/api/voltagereading/4 HTTP/1.1
> Host: 192.168.99.10:7780
> Authorization: Basic ZXNwMzJjb250cm9sbGVyOndlYWtwYXNzd29yZA==
> User-Agent: curl/8.2.1
> Accept: */*
>
< HTTP/1.1 500
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 0
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 455
< Date: Wed, 17 Jan 2024 16:31:59 GMT
< Connection: close
<
* Closing connection
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1></body></html>
mdh@fedora1:~/gitwork/esp32service_sb3basic $

For the POST (create) and PUT (update) methods, the tests fail because the servlet is altering the Content-Type from application/json sent by curl to application/json;charset=UTF-8. I tried explicitly altering the consumes attribute of the methods like this

    @RequestMapping(value = "/voltagereading/",
                    method = RequestMethod.POST,
                    consumes = "application/json;charset=UTF-8",
                    produces = MediaType.APPLICATION_JSON)
    public @ResponseBody VoltageReading createVoltageReading(@RequestBody final VoltageReading newReading)  throws SQLException, Exception {

but that does not fix the problem. I'll omit the curl output for brevity but the request sent by curl was identical to the request that worked with the service running under Spring Boot 2.4.5.

Searching for fixes for this problem mention several corner cases with faulty construction of the object class being serialized that can cause this problem:

1. Lack of a null constructor method for the object class. My VoltageReading.java class DOES have a null constructor for use by Jackson and its mappers.

2. Accidental mapping of two different fields in the object to the same external ID. My VoltageReading.java class does not have any JSON annotations and none were apparently needed under Spring Boot 2.4.5.

I have also seen solutions where an alternate character set begins appearing rather than UTF-8 which can be fixed by adding lines like this to ./src/main/resources/application.properties

server.servlet.encoding.charset=UTF-8
server.servlet.encoding.force-response=true

In this case, the problem isn't the wrong charset, it's a problem with the correct character set being rejected when explicitly appearing in Content-Type. I even tried setting the force-response to false with no impact on the problem.

Here is a summary of all of the versions of libraries and tools involved:

mdh@fedora1:~/gitwork/esp32service_sb3basic $ java --version
openjdk 20 2023-03-21
OpenJDK Runtime Environment (build 20+36-2344)
OpenJDK 64-Bit Server VM (build 20+36-2344, mixed mode, sharing)
mdh@fedora1:~/gitwork/esp32service_sb3basic $


mdh@fedora1:~/gitwork/esp32service_sb3basic $ cat pom.xml | grep artifactId | grep spring
   <artifactId>spring-boot-starter-parent</artifactId> <version>3.2.1</version>
       <artifactId>spring-boot-properties-migrator</artifactId>
       <artifactId>spring-expression</artifactId> <version>6.1.3</version>
       <artifactId>spring-context</artifactId> <version>6.1.3</version>
       <artifactId>spring-aop</artifactId> <version>6.1.3</version>
       <artifactId>spring-core</artifactId> <version>6.1.3</version>
       <artifactId>spring-beans</artifactId> <version>6.1.3</version>
     <artifactId>spring-web</artifactId> <version>6.1.3</version>
     <artifactId>spring-boot-starter-web</artifactId>    <version>3.2.1</version>
           <artifactId>spring-boot-starter-logging</artifactId>
      <artifactId>spring-boot-starter-log4j2</artifactId> <version>3.2.1</version>
       <artifactId>spring-boot-starter</artifactId>   <version>3.2.1</version>
             <artifactId>spring-boot-starter-logging</artifactId>
       <artifactId>spring-boot-starter-tomcat</artifactId> <version>3.2.1</version>
       <artifactId>spring-boot-starter-jdbc</artifactId> <version>3.2.1</version>
       <artifactId>spring-boot-starter-data-jpa</artifactId> <version>3.2.1</version>
       <artifactId>spring-boot-starter-security</artifactId> <version>3.2.1</version>
       <artifactId>spring-boot</artifactId>
       <artifactId>spring-boot-autoconfigure</artifactId>
               <artifactId>spring-boot-maven-plugin</artifactId>
                      <artifactId>springloaded</artifactId>
mdh@fedora1:~/gitwork/esp32service_sb3basic $

Here are all of the Jackson libraries used in the Spring Boot 3.2.1 based build:

mdh@fedora1:~/gitwork/esp32service_sb3basic $ cat pom.xml | grep artifactId | grep jackson
       <artifactId>jackson-core</artifactId> <version>2.16.1</version>
      <artifactId>jackson-databind</artifactId> <version>2.16.1</version>
      <artifactId>jackson-annotations</artifactId> <version>2.16.1</version>
mdh@fedora1:~/gitwork/esp32service_sb3basic $

One difference I noticed is that the old build used this Jackson module

   <dependency>
      <groupId>org.codehaus.jackson</groupId>
      <artifactId>jackson-mapper-asl</artifactId> <version>1.9.13</version>
   </dependency>

But that module appears to have been folded into

   <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId> <version>2.16.1</version>
   </dependency>

And that is obviously related to data binding functionality.

I'm sure this is something obvious. I may even find the problem twenty minutes after posting this question (in which case, I'll post a follow-up). Until then, I'm stumped.

Answer 1

Found the problem. My application's configuration class included these annotations.

@SpringBootApplication
@ComponentScan(basePackages="com.mdhlabs.esp32service")
@EnableAutoConfiguration(exclude = {WebMvcAutoConfiguration.class})
@EnableWebSecurity

I had previously disabled WebMvcAutoConfiguration in Spring Boot 2.x for some reason that escapes me now. However, eliminating that exclusion solved the problem.

@SpringBootApplication
@ComponentScan(basePackages="com.mdhlabs.esp32service")
@EnableAutoConfiguration
@EnableWebSecurity

Posting a new object to the service now results in success in the logs and a full JSON reply sent back to the client.

2024-01-17 16:19:11 INFO com.mdhlabs.esp32service.services.Esp32ServiceController QUERY action=createVoltageReading method=post reading_id=NEWREADING
 2024-01-17 16:19:11 INFO com.mdhlabs.esp32service.dao.VoltageReadingsDAO createReading() inputReading = VoltageReading [reading_id=0 espidentifier=testesp readingmv=3312 readingdatetime=2024-01-02 23:24:00 cleardatetime=]
 2024-01-17 16:19:11 INFO com.mdhlabs.esp32service.services.Esp32ServiceController REPLY action=readingCreate method=post reading_id=382

Here is the response to the curl client.

mdh@fedora1:~/gitwork/esp32service_sb3basic $ curl -v -X POST -d @"./test/test.reading.new.json" -u esp32controller:weakpassword --header "Content-type:application/json" http://192.168.99.10:7780/esp32service/api/voltagereading/
Note: Unnecessary use of -X or --request, POST is already inferred.
* processing: http://192.168.99.10:7780/esp32service/api/voltagereading/
*   Trying 192.168.99.10:7780...
* Connected to 192.168.99.10 (192.168.99.10) port 7780
* Server auth using Basic with user 'esp32controller'
> POST /esp32service/api/voltagereading/ HTTP/1.1
> Host: 192.168.99.10:7780
> Authorization: Basic ZXNwMzJjb250cm9sbGVyOndlYWtwYXNzd29yZA==
> User-Agent: curl/8.2.1
> Accept: */*
> Content-type:application/json
> Content-Length: 121
>
< HTTP/1.1 200
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 0
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Wed, 17 Jan 2024 22:19:11 GMT
<
* Connection #0 to host 192.168.99.10 left intact
{"reading_id":382,"espidentifier":"testesp","readingmv":3312,"readingdatetime":"2024-01-02 23:24:00","cleardatetime":""}
mdh@fedora1:~/gitwork/esp32service_sb3basic $

Sorry for the internet noise. Hopefully this link of unmapped Content-type exceptions and missing WebMvc autoconfiguration will save someone a few hours.