Bing blocked by Reddit, breaking search results for everyone but Google
By Sean Endicott,
2024-07-25
What you need to know
Reddit has blocked Bing from crawling the social media site.
The change took place on July 1, 2024, after which Microsoft's AI tools had to stop crawling Reddit to index content.
It appears that all search engines apart from Google are now unable to scrape Reddit for data, which means those search engines cannot show results from Reddit's website.
Bing will no longer show search results from Reddit. If you navigate to Microsoft's search engine and try to filter results to be from Reddit or simply include Reddit in a search query, you will not be presented by any results from Reddit's website from the past week. The same thing happens if you try to get Reddit results from DuckDuckGo or other non-Google search engines.
Reddit now includes a robots.txt file that says sites are not allowed to scrape data from the website. You can check out the specific page that mentions scraping to see the code in question.
"We respect the robots.txt standard. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.”
Despite the change file, Google is able to display search results from Reddit. A Reddit spokesperson told The Verge that Bing being blocked is not related to Reddit's partnership with Google:
"This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI."
Note that the spokesperson did not deny the existence of a partnership between Google and Reddit, just if the partnership is the reason for Bing being blocked. Google's deal with Reddit was reportedly worth $60 million and provides Google with real-time access to content from Reddit.
A blessing in disguise?
Bing, and other search engines, have always been handy for searching topics on Reddit. In the past, I've found searching for content and adding "Reddit" to the search or specifically filtering results to be from Reddit has been effective. Search engines are often better at finding content than Reddit's built-in search functionality.
While crawling Reddit may be useful for finding specific Reddit threads or content, Bing not having access to Reddit may be a blessing in disguise when it comes to generating responses with AI. When Google gained the rights to use Reddit content for its search engine, Google recommended people eat glue to keep cheese on a pizza. Reports that Google recommended a depressed person jump off a bridge were worrying. Google also suggested people consume rocks daily, which is not sound health advice.
Thank you to Liam ( @djsonicwarrior on X ) for the tip about Bing being blocked by Reddit.
Get updates delivered to you daily. Free and customizable.
It’s essential to note our commitment to transparency:
Our Terms of Use acknowledge that our services may not always be error-free, and our Community Standards emphasize our discretion in enforcing policies. As a platform hosting over 100,000 pieces of content published daily, we cannot pre-vet content, but we strive to foster a dynamic environment for free expression and robust discourse through safety guardrails of human and AI moderation.
Comments / 0