Reddit blocks all search engines except Google because they don't want to pay

Previously, BlueDotNet mentioned that Zhihu, a Chinese knowledge Q&A community, blocked all search engine crawlers except for Baidu and Sogou, preventing these crawlers from fetching and indexing Zhihu's content. At the same time, Zhihu also used garbled characters to intentionally disrupt these search engines to prevent its content from being captured for training artificial intelligence models.

Now, the globally renowned Reddit forum has taken similar measures. Recently, Reddit has blocked all search engine crawlers except for Google, including Microsoft Bing and Yandex, preventing these search engines from fetching content.

The reason is undoubtedly to avoid forum posts and comments being captured for training AI models. Allowing Google to continue crawling is because Google had previously reached an agreement with Reddit, paying $60 million annually to obtain all posts and comments on Reddit in real time for training artificial intelligence.

A Reddit official spokesperson claimed that the blocking of other search engines is unrelated to the agreement reached with Google. The blocking is because these search engines cannot or are unwilling to commit that their fetched content will not be used for AI training, which is the fundamental reason for the blockage.

That is, if a search engine can promise to fetch Reddit content and compile it into an index, but not use its data and content for training artificial intelligence models, then it is still possible to negotiate with Reddit to continue fetching content.

However, this situation currently has a huge negative impact on the entire internet. Whether it's Reddit, Zhihu, or other content websites, the measures they take to avoid data being fetched for AI training are to block, which leads to a reduction in content available through search engines. In fact, this is not conducive to the development of the internet.

On the other hand, the internet is seeing an increasing amount of junk content generated by artificial intelligence. For example, many websites use AI to generate nonsensical content in bulk and attract search engine crawls through SEO techniques. However, these pieces of junk content do not provide any assistance to users and waste their time with every click.

In the long run, the internet will no longer become more open; on the contrary, it will become more closed. People might gradually give up using search engines to find content and instead turn to various artificial intelligence chatbots for answers, which also have the potential to provide incorrect information, leaving users without a way to verify it.