Mysterious Http 408 errors in AWS elasticbeanstalk-access_log

Question

The elasticbeanstalk-access_log log-file in our AWS EBS instances are full of 408 errors, like these:

172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:59 +0000] "-" 408 - "-" "-"

They appear randomly, sometimes there are a few minutes between them, sometimes there are 4-6 errors within a few seconds. These errors also happen on our non-public staging environment when there is no any real traffic on the server, so the source of these requests is probably one of AWS's own service.

Answer 1

I played with the settings for hours on different environments, and here is the solution:

When I turned off the 'Connection Draining' under Configuration -> Load Balancing, the errors are vanished from the logs.

And here comes the best part: When I turned on the Connection Draining again, the errors didn't come back!

So it seems that turning it off and on works on AWS Load-Balancer too (not only on Windows...)

Answer 2

We faced the same issue and the suggestion made at the bottom of this AWS forum thread resolved it. In short, you should make sure that the idle time configured on the Elastic Loadbalancer is slightly lower than the idle timeout configured for the Apache httpd running on each of the instances.

Answer 3

EDIT: Do you have a classic load balancer? Change to the application load balancer by creating a new environment with the Elastic Beanstalk cli and select application load balancer. This will solve this issue.

---

ELB has a mechanism called pre-open connections. The ELB does this so that your requests can be served faster, i.e. your new requests will not have to wait at the ELB for the extra time it will take to open up a fresh connection to the backend before the requests get sent to the backends. If you have a lower keep-alive timeout, causing pre-opened connections to be closed quicker which will make your backend generate a 408 error response to indicate that the connection was closed because the client (ELB) timeout expired without the ELB sending any request on that particular connection.

If you have modified the ELB idle connection timeout, then you need to make sure your http keep-alive timeout value is greater than the ELB idle connection timeout value. If this is not the case enable keep alive timeouts and make sure the value is greater than the ELB idle connection timeout.

You can change the keepalive timeout in apache by adding a .config file in the ebextensions folder with the following code:

files:
  "/etc/httpd/conf.d/keepalive.conf" :
  mode: "000644"
  owner: root
  group: root
  content: |
    # Enable TCP keepalive
    Timeout 300
    KeepAlive On
    MaxKeepAliveRequests 100
    KeepAliveTimeout 300
    <IfModule mod_reqtimeout.c>
     RequestReadTimeout header=300 body=300
    </IfModule>

Answer 4

I had the similar issue: HTTP error code 408 on AWS Elastic Beanstalk. The solution I had to implement in order to fix this was changing the Instance port and protocol on the Classic LB itself to 80 and HTTP.

Initially both ports and protocols were set to 443 and HTTPS. So make sure you have the instance port and protocol set to 80 even though you have the LB port and protocol set to 443 and HTTPS.

Answer 5

It is probably because instance keepalaive timeout is shorter than load balancer. It has to be few seconds longer than load balancer's idle timeout, to do that create config into your .ebextensions folder with this content replacing "YOUR-TIME" with time longer than your instance load balancer's timeout:

files:
  "/etc/httpd/conf.d/mod_reqtimeout.conf" :
    mode: "000644"
    owner: root
    group: root
    content: |
      <IfModule reqtimeout_module>
        RequestReadTimeout header=YOUR-TIME,MinRate=500 body=YOUR-TIME,MinRate=500
      </IfModule>