Why does Googlebot crawl for /mobile/* and /m/* pages that are not referenced anywhere?

Question

Since the end of may, I have a lot of new 404 errors in the Smartphone Crawl Errors page in Webmaster Tools / Google search console. All of them starts with /m/ or /mobile/, none of which are existing nor linked to anywhere on the site.

For example, I have a 404 error for http://www.example.com/mobile/foo-bar/ and http://www.example.com/m/foo-bar pages. According to the Search Console, those page are linked in the existing page http://www.example.com/foo-bar/, but they are not.

Is Googlebot deciding on its own to look for a mobile version of every page ? Can I disable this behavior ? Is this because my site is not mobile-friendly yet (a problem for which I received another warning message from Google).

Answer 1

You could block those directories in your robots.txt, but that's a bit of a workaround. The better option would be to figure out where exactly Googlebot is picking up those URLs from.

If you shared an example page URL where Google says you have links to the /mobile pages, I could look at it and figure out where that's being picked up.

And no, Google doesn't just invent directories to crawl on the off-chance that you might have snuck in a mobile page randomly :)

Answer 2

I am experiencing the same issue since December 2016. Googlebot is constantly trying to crawl my website pages with the /m/ and /mobile/ prefixes.

All those urls cause the 404 errors and get listed in Google Webmaster Tools as errors.

The automatic email was received from GWT on January 2nd, 2017 stating

Googlebot for smartphones identified a significant increase in the number of URLs on http://example.com that return a 404 (not found) error. If these pages exist on your desktop site, showing an error for mobile users can be a bad user experience. This misconfiguration can also prevent Google from showing the correct page in mobile search results. If these URLs don't exist, no action is necessary.

This is done by a mobile crawler:

*Ip: 66.249.65.124

Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1)

Browser: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1)*

You are not alone, therefore. Take it easy. It's a Google bug :)

Answer 3

As for redirecting /m and /mobile pages back to the originals, here's a snippet for nginx:

location /m/ {
     rewrite ^/[^/]+(/.*)$ $1 permanent;
}

location /mobile/ {
    rewrite ^/[^/]+(/.*)$ $1 permanent;
}

One can also redirect everything to the root:

location /m/ {
    return 301 $scheme://$host/;
}

location /mobile/ {
    return 301 $scheme://$host/;
}

Answer 4

As @Jonny 5 mentioned in a comment, this seems to be happening as a result of Google guessing that you may have a mobile version of your site in the /m and/or /mobile directories. From what I have read, they will only try those directories if they decided that the pages they initially indexed were not mobile-friendly/responsive. More info on this behavior can be found in these Google Product Forum threads:

• https://productforums.google.com/forum/#!topic/webmasters/k3TFeCkFE0Q
• https://productforums.google.com/forum/#!topic/webmasters/56CNFxZBFwE

Another helpful comment came from @user29671, who pointed out that your website does in fact have some URLs with /m and /mobile indexed. I found that the same was true for my website, so this behavior may also be limited to sites that Google has (for whatever reason) indexed a /m and/or /mobile URL for. To test if this is true for your site, go to the following URLs and replace example.com with your website's domain:

• https://www.google.com/search?q=site:example.com/m/&filter=0
• https://www.google.com/search?q=site:example.com/mobile/&filter=0

As far as preventing this goes, your best bet is either creating a mobile-friendly version of your site or redirecting /m and /mobile pages back to the originals.