Cloudflare, Inc. (NYSE: NET), the leading connectivity cloud company, announced a major update to protect original online content. The company revealed it is now the first Internet infrastructure provider to block AI crawlers by default.

This move aims to stop AI companies from accessing web content without permission or compensation. Starting immediately, website owners can choose whether AI crawlers can access their content and how it can be used.

AI companies must now clearly state if their crawlers are used for training, inference, or search. This transparency helps content creators make informed decisions.

Cloudflare reported that this new setting marks a step toward a more sustainable future for both creators and AI innovators. For years, search engines directed users to original websites, generating traffic and ad revenue. That model is now disrupted.

AI crawlers collect content like text and images but often provide no referral traffic. This change, Cloudflare warned, risks removing incentives to produce high-quality content.

Matthew Prince, co-founder and CEO of Cloudflare, stated, “If the Internet is going to survive the age of AI, we need to give publishers control and build a new economic model.” He emphasized that original content powers the Internet and must be protected.

Prince added, “Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”

Several publishing and tech leaders voiced strong support for Cloudflare’s move:

• Roger Lynch, CEO of Condé Nast, called it a “game-changer” for publishers.
• Neil Vogel, CEO of Dotdash Meredith, said AI platforms must fairly compensate creators.
• Renn Turiano of Gannett Media praised the technology for combating unauthorized scraping.

Pinterest CEO Bill Ready said the update helps ensure content is used as intended. Reddit’s CEO Steve Huffman agreed, stating that crawling should be more transparent and controlled.

Vivek Shah, CEO of Ziff Davis, applauded the step as a boost for a sustainable digital ecosystem.

Cloudflare revealed it manages 20% of global web traffic and handles trillions of requests daily. In September 2024, the company introduced an option to block AI crawlers with one click. Over one million customers enabled this option.

Now, Cloudflare is enforcing a permission-based model. New domains must choose whether to allow AI crawlers during setup. This removes the need to manually opt out later.

Cloudflare also announced new protocols to help AI bots authenticate themselves. These protocols will make it easier for website owners to identify AI crawlers and control access.

The company reported support from global publishers and media organizations including TIME, Condé Nast, Gannett Media, Reddit, and Pinterest.

Cloudflare stated this is about protecting intellectual property while still enabling innovation. The update supports a fairer Internet where content creators retain control and receive proper recognition.

Related post

View all