Why can't Googlebot crawl this robots.txt?

380 views Asked by At

I don't understand why Googlebot can't crawl and index a WordPress website I created. This is what I have for my robots.txt:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-content/
Disallow: /content/
Allow: /content/uploads/
2

There are 2 answers

1
Luis Cardenas On

sorry i dont have permission to comment , so maybe sounds silly but you can check in admin panel

go to settings --> reading --> and Search Engine Visibility got to be unchecked

also in your frontend press ctrl+u and look for a meta

     <meta name="robots" content="noindex">

maybe a plugin is setting the meta robots,

0
Jaakko Karhu On

Go to Google's webmaster tools and add your site there. On left, go to tab 'Crawl' -> Fetch as Google. Now you should see the error, or then button 'Submit to index'. Submit your page and it should be indexed in couple of days.