I am facing an issue with Bingbot not crawling my website effectively. Despite taking several measures, including disabling the iThemes and Wordfence plugins, turning off Cloudflare, and resubmitting the sitemap to Bing, I am still encountering crawl errors when inspecting my main page URL. Strangely, the live URL does not show any critical errors. I also reached out to Bing support for assistance, but unfortunately, I have not received any responses thus far. I have installed all of Bing's plugins and configured them to submit my main page URL daily.
this is a error in bing crawl i see
Additional information:
- My website is built on WordPress.
- I have checked the error logs, but no relevant errors or warnings were found.
- Other search engines, such as Google, are crawling and indexing my website without any issues.
- I have thoroughly searched online forums and documentation but have not found a solution to this specific problem.
I kindly ask for any insights, suggestions, or potential solutions to help resolve the issue with Bingbot's failure to crawl my website despite disabling plugins, resubmitting the sitemap, and using Bing plugins for submission. Any assistance in resolving this matter would be greatly appreciated.
Steps taken
- Disabled iThemes and Wordfence plugins on my website.
- Temporarily disabled Cloudflare to eliminate any potential conflicts.
- Resubmitted the sitemap to Bing for better indexing.
- Installed and configured all of Bing's plugins, requesting daily submission of my main page URL.
- Added a code in the .htaccess file to allow Bingbot crawling.the code is "Allow from env=BingRobot "
- Verified that my robots.txt file is allowing Bingbot crawling. here is a image of my robots.txt
- Noted that all of my sitemaps have been crawled and indexed successfully. here is a image that shows my site maps status in bing
- Observed that my website's images show up in the image section of Bing's search results when searching by site name. However, my website does not appear in the general sites list, even when using a site: search operator.
Expected outcome:
After disabling the plugins, resubmitting the sitemap, and utilizing Bing plugins for submission, I expected Bingbot to successfully crawl my website without any errors.
I kindly request your assistance in resolving the issue with Bingbot's crawl errors on my website. Despite disabling plugins, resubmitting the sitemap, using Bing's plugins, adding appropriate code in the .htaccess file, and ensuring proper configuration in the robots.txt file, Bingbot is still encountering crawl errors on my main page URL. I would appreciate any insights, suggestions, or potential solutions to help resolve this matter. Thank you in advance for your assistance.