0:02let's get started welcome everyone thank you to coming to my talk today anyone here was patching PHP applications at 6 a .m. this morning just the two there you go a few more so I'm a little tired but we'll get through it it's not going to take the full 45 minutes topic for today it's a big one if you were at the keynote yesterday there's actually a bit of pre This is a bit of a longer form version of that and actually some tactical examples of what we're doing to build, of course, an open agentic internet that helps everyone, content producers, website owners, as well as the frontier model providers, which helped me a lot this morning because Claude did patch a bunch of my servers.

0:49Right, so the premise to the whole situation. You can actually check this up on radar.cloud4.com. If you haven't seen the website, I highly advise to take a look. There's a whole bunch of really interesting filters that you can filter by region, activity, bot type, purpose of crawling. The headline, of course, is the majority of the traffic we're seeing across the Cloud for Proxy, give or take, we're talking about tens of millions of websites at this point, is mostly non-human or better, bot and automated. And we also know that the increase over the past you know 12 to 18 months has mostly been driven by all the new use cases for crawling training being the prime examples everyone is building a bot everyone is fetching content via their agents and this is the output of that consequence was mentioned at the keynote the order magnitude difference over the next couple of years it's just going to grow so take an example of you running a small little website you may be receiving a couple thousand requests today, you can probably expect that over the next years, that number is going to go way higher, and it's all going to be automated, because everyone is hungry for data.

2:02We zoom in. Again, this is a screenshot from radar.com. And just as I said earlier, the number one use case. And we do this by actually identifying, where possible, what bot is accessing the content behind the proxy. You may have heard of WebBotOff. Another really cool thing on Radar you can go and see bot directory, and you have a list of hundreds of bots we recognize, and a bunch of them, especially the big provider ones, submit a cryptographically verified HTTP header so we can say, yes, this is OpenAI, this is Anthropic, this is Google, this is Muse from Facebook and similar. And based on the bot, most of the providers split their bots based on use case. We can tell that, you know, number one case is training or some sort of agentic behavior to power our conversation.

2:48with our new bot overlords. Search is still there, not negligible, about 13% of traffic. That is your traditional search that you might see in Google, Bing, Yandex, you name it. Pretty constant, but not definitely more than the agentic use cases. What does this mean? So we started a year and a half ago, we had Content Independence Day. when we started noticing the behavior changing on internet traffic and the canary of the whole situation of the business model is changing, of course, is websites that produce original novel content. Okay, I actually run, as a side, one of the little news agency for my own town, local town, my hometown in Italy.

3:35And I've actually seen this myself on the server looking at the logs. Human traffic is dropping a lot. considerably. And I run the website on behalf of a friend who's actually the owner of the website. And just like these big brands here on the slide, his revenue from ads is just going down. His revenue from subscriptions is going down. Why? Because today you open your OpenAI Chat GPT, what's the latest news? Even this morning I was looking for the latest, when was the patch out for PHP? I could have probably checked php.net. given it was the front page article on the site, but I actually asked Claude, and Claude fetched the page for me and gave me the information I needed.

4:20Did I, as a human, go to php.net? No, right? And you can see how this is changing behavior for everyone, and those who make money from human visits are suffering the most. Pretty impressed, digital trends. This is actually published by a third-party organization that looks up this data, so not even cloud for data. 97% less human traffic hitting the site. This is just complete disruption of their business model. Okay, so some would argue, you know, historically, you get a crawler accessing your site, getting your site indexed, and that's good. A bunch of us probably have an experience with search engine optimization. Is this bringing any value? We can actually see, looking at referrer headers, and again, this is public, go to radar.cloudflow.com, how much are these providers crawling?

5:08And how many humans are they sending back to their site? Sometimes you actually may see, you know, the chat agent is giving you a referral link and you may click on it and actually visit the site, consume the ads, buy a subscription or something. But it's pretty bleak, let's be honest. Look at perplexity. For every 3 ,400 visits their agent accesses any given site, they're sending one human visit back. And it doesn't get better throughout the board. And this means more resources, more load, more servers that you need to step up to keep your sites in mind. We track this over time. If you actually click on Radar, you can actually see the difference over the course of the past couple of months. And these, of course, are just the top ones that we picked, If you go to the bot directory, you can actually see on any given bot that you might be interested in many more providers than the ones listed on the slide.

6:01So the big question is, what do we do now? an internet community or people that benefit and enjoy exploring and using the internet as part of our daily lives. We need to essentially make sure that the business model stays up so that the internet doesn't devolve into something that's worse, let's say worse over time. As I said earlier, the Canaries here, of course the news publication websites, they're suffering the most, but you can see how this trickles through for example. companies that are publishing research, journals, news, weather companies. And eventually, it will probably hit everyone, including e-commerce installations, small blog providers. I have a personal blog that I keep updated, and I'm probably going to see the consequences of that.

6:49Not that I get many visitors in the first place, but it's only going to get worse. The other thing, some of you may not know this, some of the browsers are actually embedding or provide you, some of the Frontier Model providers, have their own AI-native browser. Someone who heard about Comet, for example, is one example. When you're accessing web pages using those browsers, sometimes they're actually doing crawls in the back end, because they come across as, of course, legitimate humans. And it's harder for bot detection providers, such as Cloudflare, to actually detect them and stop them, and as a consequence, actually making it worse for everyone. Good news. You know, new business models, this is just the way history evolves.

7:35Companies and industries have been disrupted before. So for us, it's just a matter of figuring out what is the next. A couple of, you know, very well-known examples here, the Napsters of the world. Anyone here used to use Napster? Yeah, they've disappeared, replaced by Spotify with a subscription model. We can debate forever if that's good for the artists or not. Some will say yes, some will say no. There's been some very interesting scams and similar going on some of the subscription-based platforms. I still remember as a kid going to Blogposter, the Italian version of it. Long gone, replaced by Netflix. And of course, I did see someone actually with a paper -based newspaper in the tube in London the other day. But fair to say, that's the minority, not the majority.

8:20Everyone's reading news online.

8:26The new business model, we're starting to see it take shape. Some of the large content producers, for example, have already been able to, and this actually started early in the year and into even the late end of last year, signing direct deals with the big frontier model providers such as OpenAI, Anthropic, and similar. The good news is they actually want to sign these deals with the content producers because there's a benefit for both parties, right? If I... sign a deal with a big news corp, and I'm a frontier model provider, I can probably optimize how I receive the data from them so that I have to crawl a thousand times or continuously refresh and fight against them to try and bypass their mitigations so that I can provide a better service to my end users. This is the start.

9:11I would say at this point, the majority of the large news publishers, even if they haven't maybe publicly declared or not, like People Inc, have actually now got in place deals with the Frontier model providers for them to access content, especially because freshness of content is extremely important, right? You wanna know what's going on in your local town, you ask the chat agent, it should be able to give you really up-to-date information. It's also important that they get up-to-date information from the agents because we as humans are starting to rely on them for our everyday tasks. And if the agent doesn't have fresh information. We've all heard of hallucinations, right? We may actually get misguided, misaligned, get the wrong content from the content producers.

9:59So actually, it's good for everyone if both parties work and find a good way of working together. This is just an example of, of course, Google AI overviews maybe putting people at risk on health advice, because sometimes they have just out of date or not access to the latest best content with the correct information.

10:21So, at Cloudflare, you've heard this at the keynote, we're trying our best to be the impartial party, figure out how can we help both content producers, website owners, or Frontier model Of course, at some point, we need to talk about money, so monetization is arguably one of the end goals, but we cannot just do that, right? We need to make it easy for both sides to achieve their end goal. So, these are a few . tactical examples of things we've been working on. Number one, making it easier to crawl. Now, arguably lots of people here do not want to be crawled, but fair to say some of us do want to be crawled. And if you have to crawl me, I want to make sure you're as efficient as possible whilst doing that, so you're not taking my website down or your crawling doesn't look like a DDoS attack against my infrastructure.

11:08I've actually had a couple of sites go down because I was getting hit quite heavily by... ink crawlers a while back. I had to block them even if I was trying to give them product catalogs and similar. If you go to the Cloudflare dashboard and you have a website on Cloudflare, who here has a website on Cloudflare? Okay, quite a few of you. If you want to give access, there's a little crawl option. The proxy will automatically issue a new endpoint, which will only be accessible, by the way, by a signed, verified bot. So we're trying to keep things. as clean as possible, and we will parse the content out of your site and actually provide a much easier to crawl markdown syntax to the bot so they can access the content over and over again if they so wish to do so without hitting your backend server or increasing load to your application.

11:58Okay, so this is one of the ways we're trying to make it better for both parties.

12:06Second one, make it easier to search. You may have noticed if you do a prompt on a Frontier model provider, on Cloud, Opus, or whatever, very often the first thing the agent will do is kick off a bunch of sub-agents that will actually go on and perform searches on the open internet. And this is one of the initiatives, but fair to say are still very much under development, with a plan of building an open index for all the sites sitting behind the Cloud4Proxy, give or take, we're talking about 20% of the internet, so that the Frontier model providers or whoever, if you want to build one tomorrow, again, it's about having diversity in the ecosystem, can actually refer to the index and actually get content without having to search for it, which is a lot more resource intensive.

12:52Even this, by the way, is not opt-in. By default, you can go to the dashboard, you can decide to turn it on and opt-in into the index or opt-out. It's your content, it's your application under your control. Plan is, make it easier. so you don't receive the load. And of course, it's a safe path to AI adoption over the long term.

13:13Last one and not least. Search is part of it, and the index definitely helps. Whilst we are building out this index, of course, the search engines and the crawlers have a discovery problem. I remember I used to work at a company called NetGraph before I joined Cloudflare. One of the things we were doing was an internet service survey. Discovering new domains and sites for the purpose of serving the internet was one of our biggest challenges. And we actually used to partner with the likes of Microsoft to get a list of domain names that were signed up or by registries. Similar sort of incentive, we actually did a research, and you can find it on our Cloudflare blog. We partner with OpenAI in this case to give them signals of new domains that are being added behind the Cloudflare proxy with the crawl endpoint turned on.

14:02into being discovered. We did on a very small percentage of traffic, so not the entire application set behind the Cloud4Network. And you may have heard at the keynote, this resulted in OpenAI, in this case, discovering several million new domains and applications without having to necessarily bombard all of us with their crawls. We think this is the win, especially for the smaller players on the internet. So let's say you're starting up a new site tomorrow. You will be no one on the landscape, but with something like this, the providers will be able to find you more easily, and you actually show up in their search results or within the agent, so you can get access to your end users.

14:44There's a lot more we're doing. These are, I think, three of the best concrete examples they're actively working on. So homework, go in the Cloud4Dashboard, find the options. You can make the decisions to access or not. thing about the slash crawl endpoint which actually powers some of the research we did with open AI there's actually a standard called content crawl control which we only let bots who are at least is the trust based system wanting to access your content for a given purpose to actually give them that content be a training inference or for example requiring referral back to your site or not and all of these options are for you to configure on the side we're actually pushing for standard to be done by the industry. There's still some work to be done there, but over time we're confident we'll make it happen.

15:34Okay, so frontier models, crawlers, building a new one yourselves, you're a content producer, trying to connect the dots for everyone. Now we can start thinking about monetization. This is the last two things of my presentation today. Number one, pay-for-use. There's a little video as a background.

15:55you as a content provider or content producer want to get you know some part of the pie when it comes to making money at the end of the day there's two big challenges here number one you could actually get paid every time a crawler tries to access your content and you may have heard of our monetization gateway which is my next slide everything is good and great you can set a price you can say you access this API endpoint I want to be paid ten cents and there's a way to make it happen. However, there's a bigger problem, which is what we call pay-per-use, which is still under development. We're figuring out how to make it scale up, which is a Frontier Model provider only needs to crawl you once, and they may reuse that content a hundred thousand times, right? So you need to think about how do we make sure that every time the content provider uses your content, you get a little portion of that value back in your site.

16:44Okay, so if you're wondering what's the difference between pay-per-use and monetization gateway, this is the cracks of it, is reuses out of scope versus accessing your content. Pay-per-use is based on the trust system for now, so it needs to be signed agents accessing your content. You set the price in the dashboard, OpenAI comes along, we take a look, we say yes, this is a price we're willing to pay, no we're not, and then they will add the content to their search engine, to their agent on front. share model providers, and in theory, every time they use the content, you'll receive a little value back. Everyone following so far? Second part is, it goes to the next slide.

17:33Seem to be stuck.

17:36Could you help me at the back, advancing the slide?

17:43Monetization gateway. And the video, there you go. Second part. monetization gateway. This is in private beta. If you go to the cloud for dashboard, there's a little on the left of the account level. There's monetization. Click on that. One click to request access to the private beta. This is essentially the technology built on top of the proxy so that you can then define at access time how much you want to be paid. Easier from a technological standpoint because we're not relying on the trust. system. You define the prices, a signed agent comes along, of course will receive a 402 payment required response from the server, and assuming it's a signed agent and they've implemented the integration with our protocol, they will submit the payment to the relevant payment provider and then re -access the content with proof that they have paid.

18:33This is the token. Proof happens, and then they access the content that you're providing. And in this case, it can be an API, right? So if you expect you've had a service, and you're not building necessarily content, but it's an API service, you may want to get paid one cent for every API call. Monetization gateway, which by the way is an open standard, is a tool for you to achieve that. So you can scale out payments to agents without having to worry about collecting money from each one. Probably very useful for the smaller players that cannot go off and have big signed agreements with the Frontier Model providers. I advise you take a look in the dashboard, so sign up for the private beta. You can actually set up rules on all sorts of filters you wish. You can decide, for example, to only make portions of your API requiring payment versus different filters per region.

19:25You control the rules. We implement them as part of the proxy. Under the hood, who is here instead of the X402 Foundation? A couple of you, not too many. with Coinbase and a couple of other technology companies to come up with a standard and the protocol that makes it happen transparently. I'm not going to go into the details of this right now. Again, we are one of the providers that implement the standard. There are others out there, of course, with the best ones that made it happen. Up to you, you choose. You can even implement it actually on your own servers without a proxy, if so, if you wish to do so. Everything is verified. The only challenge here for us has been over time is how do we scale up this concept of micropayments.

20:11You may have also heard that we issued and are building out a stable coin called NetDollar, which is part of one piece of the puzzle to hopefully make this happen as it scales up to everyone using the internet at large. And yes, and hopefully, maybe by the time we meet again next year at a similar conference, we'll start seeing the new business model of internet. and taking shape. With that, I promised it was gonna be sweet and short. So thank you for listening to me today. And yes, any questions from the group, I'm happy to answer them. I think we have a mic so everyone hears you.

20:55On the monetization side, are you guys thinking about bid asks? because I think most people have no idea how to price. Good question. So think about the two use cases again. Subsequent usage of content that was crawled versus payment at access time. When you go into the monetization gateway, which is, again, fully built, you just need to access the private beta. Once the team approves, you get access to the dashboard. You just define the price for the endpoints, and that's the price you set. You can figure out what you want. and the bot can decide if to pay or not. For the pay-per-use, which is something we're still having fully rolled out to everyone, which we're trialing with some of the big content providers, that is more of a bidding system, because it's a bit more complicated to give value to something that gets reused over time, especially if it's not a crawl time, but that does have a bidding system where the providers can say, I want to pay you X, and then you can decide the rules of, I accept, or this is the range I'm willing to go, and we try to opt a little bit like Google AdWords, you can imagine, right?

22:02To get you the best value out of your content usage. Having said that, this is well under development, so I'm sure there's gonna be lots of subtleties that need to get worked out still.

22:15Down there.

22:21So, since you're targeting content providers, are you looking at ways where some content provider, let's say, New York Times, right? I can allow OpenAI through crawl to get the headlines, but if the user then says, okay, now I want to read the full article, then I go in, click, and then I do a paper. That's a very good question. So you could easily build that yourself today by using the monetization gateway. Let's say you wanted to get paid even for the headline. Because sometimes, you know, lots of readers just read the headline and they're happy and they move on. I've actually done something similar on one of my sites. You essentially would have an API endpoint that responds to the signed agents with a summary of each articles.

23:09And then you can apply monetization gateway on top of that. And as part of that summary, you have the referral back to your content. And then if the user clicks more, then they would subscribe through your standard process. The challenge with that, by the way, becomes... just generic bot mitigation because what we're seeing okay we're doing all the right things and the big providers are actually willing and wanting to make this new business model work out there's a very long tail of other providers or potential crawlers that are trying to bypass all of these mechanisms to the point we've seen sophisticated agents even sign up to subscriptions crawl all of the content and then redistribute it right there's actually services out there legitimate businesses in many cases that are doing that on behalf of, you know, content providers. So, the complexity becomes in locking down your content in a way that you're happy it's not leaking, right?

23:58One of the side effects of this whole new business model has been the web archive, for example, right? Because they're essentially a very legitimate good reason to exist, but now they're getting abused because they're just echoing content that you may not want to get leaked, right? Any other questions on that?

24:22Hi, very interesting models and just curious how do you balance the monetization versus discoverability because isn't that your adding hurdle or gateways for free indexing content? So our current position is you as the website owner and content provider make the decision. Okay, it's near impossible. for us to make a judgment on your behalf. We do not want to make the judgment. We're providing, or we think we can help everyone solve their problem by providing the tools. And then it's gonna be in each individual or business that makes the decision. I want to block bots and agents versus I want to allow them for free. That's something that many companies and businesses want to do versus I want to monetize.

25:11It gets complicated if you start thinking about it because I agree with you, it's a hard problem. a smaller news agent may want to give access to the agents for free, because ultimately they're smaller and they're trying to grow. Maybe that's a good thing, whilst the big company that has big bills to pay wants to start monetizing from day one, and it creates very odd balances. But it's your decision at the end of the day, and we'll figure out how to help you the best.

25:41Hi, how are you? So do website... Publishers need to structure their website content or data in any special way, whether it be meta-tagging or markdowns or hierarchy to maximize their monetization opportunities? Yes, absolutely. This is actually what the slash crawl endpoint does. Of course, it works better for smaller sites. If you're talking about a very large publication like BBC, we're going to be building something custom. Markdown is the language that a lot of agents are accepting, lower weight. Having said that, modern agents are able to parse anything out. You may have seen our announcement for our new model, Cleric, that we just launched. The whole point of that model is to convert unstructured content into structured content, which is solving sort of that problem.

26:28Having said that, if you look at the big content producers, they all have alternatives to their standard human readable sites, which are optimized for agents. And this is you know, the new CEO of the agents, right? And agent optimization is all part of that, making it easily readable by robots. Question at the back?

26:56Hey, I just set up AI search on one of my sites while you were chatting, very cool. How do you think of the AI search functionality along with some of the other things, like the crawl. so the crawl feature is kind of creating the markdown file the AI search is a different feature I just created an MCP and put it in my cloud so I can interact with it how do you think about those that universe kind of working together and not creating different functions so we're you know we're trying different things at the same time arguably to your point slash crawl versus search are complementary or overlap in some ways but the slash crawl assumes that the bot or the frontier model or the crawler builder knows that you exist as a website.

27:44Okay, so if I know your domain, when I try to access it with my signed verified agent and because I'm signed and verified, the team has made sure that you're doing the right thing, so I follow content control directives, I can access your slash crawl, get the content, you never see the request in your server, so you're happy and we echo that. If I don't know it, about you though, in the first place, I don't know where to go. This is what the search does. So the whole point of opting into the AI search is that you'll be added to an index that the content crawlers can separately access to discover new sites on the web, okay? So that's sort of their entry point. They may get content directly from there, which is even more efficient, but once they know about you, then they'll be able to hit your slash crawl.

28:29Okay, so that's the difference from a technological perspective between it. great questions by the way anymore yeah I have one about so you mentioned like there's a surgeons of like these resellers who are like just purchasing a subscription and then reselling it yeah so how do you kind of avoid this race to the bottom for like all these content providers like if anything takes off basically right then there's an incentive for someone to get it and then redistribute it to agents via that. So is there like a... That's why this problem is hard and that's why we exist and we're trying to make it better. You've all maybe have heard of residential proxies.

29:14Yeah, so this is where this company is out there. You go there, you want to crawl something, you get blocked. You go to another company, they give you a bunch of residential IP proxies. That's a simple way we detect that, for example. But that's a simple way where people are trying to bypass protection. it is a race to the bottom if you're open and you're not don't have visibility of what's going on in your site that's why again we're for Cloudflare we have our bot detection product is trying to detect every occasion that someone is trying to bypass your controls to make sure you're creating scarcity and then you can have a business conversation but yes that's the that's the challenge we have because it's a race to the bottom otherwise and the whole business model of apart yeah and residential proxies by the way it's relatively easy now there's way more complicated case you know cases of people bypassing detections questions at the front yeah so I'm one of those people that we want to get crawled I'm with Maricopa County elections and we want to get the information out there right right because we want the right information out I'm curious as we're updating throughout the day if we do that is that a constant scrape and they update, or is that just, what's the frequency on?

30:28On the slash crawl, you mean? Yeah, on the crawler, is it kind of pushing out, oh, hey, something just changed, or is it very passive still? That is the idea of the, so some of this is still in development, we're still figuring out. The idea of the search index, ultimately, is also to be able to push to the content creator a notification of, this content has changed, and therefore we're optimizing. the whole system from the ground up. Right now, it's still not happening at large. So even if you enable the slash crawl, you're gonna get hit on the slash crawl. At least we're taking the load, not you as the content provider. You may have the challenge that you do want to get crawled and therefore you need a way to notify that your content has changed. That's something we're actively working on.

31:13We're trying to solve with the search index and also with an integration with the relevant providers to ping them back. Sort of subscribe to the content. this is where you get the fresh version. It's an ecosystem that we're building up.

31:27Any more questions? One more?

31:32So there's this view that a lot of the pre -2023 content was disproportionately more useful than content that's maybe being created now, in the sense that it's already in training corpuses and either through open deals or just through like... up scraping I'm curious how you think about the value for like information that is being generated created and put on the internet now versus what's previously existed and sort of the economic value of forward-facing information versus like what's already been scraped and crawled and it's a little bit so yeah I don't have the perfect answer to that question what we do know is the subsequent reuse is important to some monetization value there.

32:18That's why paper use is technically a different concept from monetization gateway. Think about Spotify, right? Every time you play a music, a song from an artist, they will get the relevant charge for the service, right? And the same applies to internet content, or we believe should apply. That's why paper use is the thing. Content definitely value over time goes down. This is why for paper use, the question Ben was asking at the start. There's more of a, we're building out a sort of bid slash response over time with the content value changing, and we're trying to build a system that optimizes that for the content creator. We're also in a stage where the internet is blowing up with content and replicas of each. I don't know how that's going to play out. As we build this out, hopefully we'll start fine-tuning what works best so we maximize and everyone's happy.

33:10It will take some time to get there. One more question. Anyone?

33:17Okay, so thank you for coming. This was a great set of questions. And homework, check out the features, especially monetization gateway in the dashboard, and enjoy the rest of Connect. Thank you.