ChatGPT's web search function uses Microsoft Bing search technology and its crawler is called OAI-SearchBot
Brook.X
OpenAI has recently unveiled the ChatGPT Search web search feature, marking its direct entry into the competitive landscape dominated by Google Search. This addition essentially categorizes it as a search engine.
In terms of data collection, OpenAI employs a dual approach. The search technology is powered by Microsoft Bing, and simultaneously, OpenAI has initiated its own content crawling efforts. All collected data is sorted using a specific algorithm to facilitate the return of search results within ChatGPT.
Although not explicitly disclosed in OpenAI's blog posts, the use of Microsoft Bing for search technology was confirmed by its engineers in a Reddit forum discussion. For webmasters aiming to optimize for search traffic via ChatGPT, tailoring SEO strategies for Bing becomes imperative.
OpenAI operates three distinct crawlers:
- GPTBot: This bot crawls the internet to gather data for training OpenAI's AI models. It is designed not to affect website search traffic negatively.
- ChatGPT-User: This bot retrieves data from the web to provide source links in response to user queries, without collecting web page data.
- OAI-SearchBot: Specifically designed for the ChatGPT Search functionality, this crawler gathers web data without using it for AI model training.
Websites wishing to prevent their content from being used for AI training but still want to benefit from ChatGPT Search traffic (albeit currently minimal) can choose to block GPTBot but allow OAI-SearchBot.
More about OAI-SearchBot:
Complete User-Agent (UA) String: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
IP Addresses: 20.42.10.176/28, 172.203.190.128/28, 51.8.102.0/24
To guard against malicious bots impersonating OAI-SearchBot, webmasters are advised to verify the bot's IP addresses. Any crawler not operating within the specified IP ranges should be considered fake and blocked accordingly.
