AI crawler list · Microsoft · AI search index

Bingbot

Microsoft's search crawler. Bing says a robots.txt rule that blocks Bingbot can keep you from being seen in its index, and Bing's AI answers (launched as Bing Chat, now Copilot) draw on that content unless you use Bing's NOCACHE or NOARCHIVE controls.

Facts checked against Microsoft docs on Oct 18, 2026.

Operator

Microsoft

Used for

Bing search and Copilot

Type

AI search index

Obeys robots.txt

Yes

Is Bingbot allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Always allow it.

If you block it: Bing can't crawl your site, which can keep your pages out of Bing's index.

Bing Webmaster Tools has a Crawl Control setting to make Bingbot crawl faster or slower for each hour of the day.

robots.txt rules

Allow Bingbot

User-agent: Bingbot
Allow: /

Block Bingbot

User-agent: Bingbot
Disallow: /

Disallow Bingbot from one folder only

User-agent: Bingbot
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

How to verify it's really Bingbot

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "Bingbot" access.log

Microsoft's documentation does not publish the full user agent string for Bingbot, so treat a token match as a starting point and confirm the IP.

Reverse DNS check (Microsoft documents this): look up the IP from your log, confirm the hostname ends in search.msn.com, then look up that hostname and confirm it returns the same IP.

host 203.0.113.7          # replace with the IP from your log
host <hostname-from-above>  # must return the same IP

Published IP list: www.bing.com/toolbox/bingbot.json

Check one IP against the list (prints True or False):

curl -s https://www.bing.com/toolbox/bingbot.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7

Or paste the IP into Bing's Verify Bingbot tool.

Common mistakes

Blocking Bingbot to keep content out of Copilot answers. Bing's documented controls for its AI answers are the NOCACHE and NOARCHIVE robots meta tags, and pages with either tag stay in Bing search results.
Writing a User-agent: Bingbot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the Bingbot group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "Bingbot" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is Bingbot?

Does Bingbot respect robots.txt?

Yes, according to Microsoft.

What happens if I block Bingbot?

Bing can't crawl your site, which can keep your pages out of Bing's index.

Should I block Bingbot?

Always allow it.

How do I keep my content out of Copilot answers but stay in Bing search?

Bing announced two robots meta tags for its AI answers (then called Bing Chat). With NOCACHE, answers may use only the URL, title and snippet. With NOARCHIVE, the page is not used or linked in the AI answers and not used for training. Pages with either tag still appear in Bing search results. Blocking Bingbot removes you from both.

How do I verify Bingbot?

Do a reverse DNS lookup on the IP: the hostname should end in search.msn.com. Then do a forward lookup on that hostname and confirm it returns the same IP. Bing also offers a Verify Bingbot tool at bing.com/toolbox/verify-bingbot.

Can I slow Bingbot down?

Yes. Bing Webmaster Tools has a Crawl Control feature that lets you make Bingbot crawl faster or slower for each of the 24 hours in a day.

Sources

Related AI crawlers

OAI-SearchBot · Claude-SearchBot · PerplexityBot · Googlebot · Applebot · meta-webindexer · Amzn-SearchBot · DuckAssistBot

Facts checked against Microsoft docs on Oct 18, 2026. Spot an error? rankroot.support@proton.me