Reddit is taking a stand against AI corporations — or not less than asking them to pay up.
Earlier this week, Reddit announced that it’s changing its Robots Exclusion Protocol, also referred to as its robots.txt file. This dry-sounding edit is an element of a bigger negotiation/battle between the AI corporations which are hungry for content they’ll use to coach their language models, and the businesses that truly own the content.
“Robots.txt” is how web sites communicate to 3rd parties how an internet site could be crawled — the classic example being web sites that allow Google to crawl them in order that they could be included in search results.
Within the case of AI, the worth exchange is rather a lot less obvious. While you run an internet site whose business model involves attracting clicks and eyeballs, there’s not much appeal in letting AI corporations hoover up your content after which they don’t send you any traffic — and in some cases, they outright plagiarize your work.
So by changing its robots.txt file, and likewise by continuing to rate limit and block unknown bots and crawlers, Reddit appears to be working to stop the practices that corporations like Perplexity AI have been criticized for.
Hit play to learn more, then tell us what you’re thinking that within the comments!
