topic Re: Is Hostopia blocking bingbot web crawler intentionally? in Everything else

Is Hostopia blocking bingbot web crawler intentionally?

jtonline — Fri, 28 Oct 2022 02:15:17 GMT

I can't get my little Plusnet website crawled and therefore indexed by the Bing search engine.

Bing Webmaster Tools says it can't find my robots.txt file, and yet I can browse to it and Googlebot can see it.

Submitted sitemaps via Webmaster Tools are coming back with a 403 error and yet Googlebot can see them and validate them OK.

Getting Webmaster Tools to Inspect a URL comes back with "Discovered but not crawled".

My thoughts are that Hostopia is blocking the bingbot web crawler robot at server level, or they're only allowing bingbot from a set range of IP addresses that are out of date.

Can anyone confirm if they're experiencing the same issue?

@bobpullen are you able to ask your contacts at Hostopia on my behalf?

My website is http://www.jtonline.info. I'm going to start adding some new content & Bing search results would be nice.

Re: Is Hostopia blocking bingbot web crawler intentionally?

bobpullen — Fri, 28 Oct 2022 10:47:32 GMT

@jtonline wrote:
My thoughts are that Hostopia is blocking the bingbot web crawler robot at server level, or they're only allowing bingbot from a set range of IP addresses that are out of date.

Well, if it's the latter, I imagine you're out of luck given they don't seem to publish what these IP's are. From here: -

"You can identify Bing crawlers with the user agent string. But user agent strings are easy to spoof, so not every request with these user agent strings may be coming from a real Bing crawler. As a rule, Bing does not share the IP addresses from which we crawl the web, but you can always use the Verify Bingbot tool to check if a crawler belongs to Bing."

I can try asking the question of our webhost partner. Assume it's the main 'Bingbot' crawler that you're concerned with?

My suspicion is that the crawling is being blocked by a Web Application Firewall but that's pure conjecture at this point.

Re: Is Hostopia blocking bingbot web crawler intentionally?

jtonline — Fri, 28 Oct 2022 13:27:44 GMT

@bobpullen wrote:

I can try asking the question of our webhost partner. Assume it's the main 'Bingbot' crawler that you're concerned with?

My suspicion is that the crawling is being blocked by a Web Application Firewall but that's pure conjecture at this point.

Thanks for your reply, Bob.
If you could ask the question that would be great. Yes, Bingbot seems to be the main one.
I've reached out to Bing Webmaster Tools support team too via their contact form, and await a reply, but I suspect they're going to say 'contact the webhost'.