Strange nginx behavior specific for bingbot requests

Question

we have found something strange in nginx logs. Some requests only from bingbot goes directly to backend, but if we fire this requests from curl, they processed as usually. Please take a look of configuration.

I've got two servers, one (server1) if frontend and it's proxying some requests to server2 and some requests process by himself.

Please take a look Server 1 config:

server {   
  listen  80;# default_server;

  proxy_set_header        X-Real-IP       $remote_addr; 
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;   
  proxy_set_header        Host            $host;

  server_name myservername.com;

     location / {
         index  index.html;
         root /var/www/apps/delta/shared/public;

         if (-f $request_filename/index.html) {
           rewrite (.*) $1/index.html break;
         }

         if (-f $request_filename.html) {
           rewrite (.*) $1.html break;
         }

         if (!-f $request_filename) {
           proxy_pass http://cloud;
           break;
         }   
    }

    location /store/index/ {
      index  index.html; 
      root /var/www/apps/delta/shared/public;

      if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
      }

      if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
      }

      if (!-f $request_filename) {
        proxy_pass http://delta;
        break;
      }
   }

}

Config file for server2 is simple, process requests at upstream and nothing interesting.

As it must be with this configuration: nginx need to process requests started from '/store/index' like http://myservername.com/store/index/category/898/author/989 at upstream "delta", and he do it. BUT: for bing requests he ignores location /store/index and proxy'ing request to upstream cloud (server2)

Take a look: Server1 log:

157.55.39.83 - - [04/May/2016:15:33:45 +0300] "GET /store/index/category/67/author/6 HTTP/1.1" 200 12150 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Server2 log, little bit custom log, x.x.x.x - my frontend ip:

x.x.x.x - - [04/May/2016:15:33:45 +0300]  "GET /store/index/category/67/author/6 HTTP/1.0" 200 12137 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "myservername.com" 5.045

What we see:

1. nginx proxy'ing request to server2
2. bingbot request http 1.0 version
3. this is one proxed request, and not two from bingbot to server1, and same time to server2.
4. after I make the same request from another server: curl -0 -v http://myservername.com/store/index/category/67/author/6 all is ok, only server1 process this request and nginx not proxy'ing this to server2.

I'm totally confused. Location is working like I need for usually user requests. And by unknown reason and only for bingbot requests it's proxy'ing them to server2. Please tell me maybe bingbot send some headers or what wrong in config? Thank's a lot!

Answer 1

It seems that bing request via. through https version - I edit my https config and everything is well now. Thanks