0:02Okay, I see my timer has started going, so we're going to jump into this. As you may have just heard me said, I just finished my slides rewriting some of the tail end, so we're gonna go through this experience together for the first time, so hang on. So, quick poll, does anyone already know what Artifacts is or is using Artifacts? Oh, hell yeah. Okay, so we... All my users are here. This is... So today I want to talk about artifacts, obviously, and how we basically went from zero, no product, to being in closed beta in three weeks. And it was a trip. So to set the scene a little bit, before I was working on artifacts, I was on the dashboard agent team working on Agently.

0:50We were working on changing a bunch of architecture for that, rewriting it, getting ready for Agents Week. I'm sure many of you have seen us. around our agents week. It's kind of like very high pressure, high rush. Everyone ships everything all at once. So, like, my mind was very in the zone on, like, I need to really make Agent Lee a great experience. And then I was sitting on my patio on a Friday afternoon, and I get a phone call from the one and only Dane Connect. He says, how soon can you stop working on Agent Lee and hand it off to the rest of the team? And that was a alarming. thing to hear. He then told me that we had a large customer that had come to us with a need for basically Git and code storage built on the edge. So just a quick outline of what I want to try to get through here is why artifacts? Why is it on Git? And why now?

1:40Kind of a deep dive. I would love to go longer. We only have 18 minutes left at this point. Giddy, which is our internal engine that we have implemented Git in. We recently just completed a refactor from Zig and Wasm back to JS. A mistake I made early in the project, and just some takeaways and lessons from having built this whole thing out on Cloudflare and basically AI writing all of the code. So why artifacts, why Git, and why now? As I mentioned, a large multi-tenant customer came to us and had a strong need for . and storage on the edge. We had Cloudflora had talked about this over the years, but the time, the problem was never right.

2:25But given some of the struggles from other Git providers and the nature of needing small ephemeral, not ephemeral, but small repos that agents can operate over quickly, was a growing need, and it was the right time. We chose Git because Git is already in the training data. I feel like this is starting to be less of a problem as agents become more capable. but we started this basically when Opus 4.5 was the main model out there. It's ubiquitous. It works everywhere. Every developer already has Git clients on their computer, so you can interact with it there. You have JustBash, Git's in there, so it kind of just works anywhere you need it. And something kind of interesting over this is we found out that it's, like, better. It's good for things more than just code.

3:12It's good for just version storage in general. And yeah. Agents are writing more code than ever. So we had two deadlines. We had three weeks to get to Agents Week for closed beta, and then this customer was launching their own products, closed beta, in four weeks. And one of the constraints that this customer came to us with was that they needed to be able to create 1,000 repos per second. Yeah, for a single tenant in a single namespace. And I am happy to say we did meet that benchmark. You can blast away repos at a given name. space and hit 1,000 per second. So the original team was Matt Carey. He was really focused on the Git engine. He spent his winter break deep in AI psychosis, playing with Zig and Git, so we kind of had some work up front. That's how we landed on Zig and Wasm.

3:58Thomas Ancorn, he's on our observability team, and just all around Cloudflare Ball Knower. He helped get a lot of process and teams aligned on getting this out the door so quickly. And then I was working on the control plane, DX, getting the right architecture to scale the control plane, et cetera. And then a whole bunch more people have contributed over the months. So a quick architecture deep dive. And to start with some language here, this is what one of your remotes look like if you build an artifact repo. You have your account ID, the rest of the URL, slash git, slash namespace, repo name, and a namespace is pretty much like a GitHub org or user. So for anyone, I mean, a lot of you are already using artifacts, so I assume that you know how durable objects work. but for anyone, we're going to have a quick refresher. DurableObject is a tiny server with a SQLite database.

4:45It can handle about 1,000 requests per second per instance. It comes with 10 gigabytes of SQLite storage. It's single-threaded, and it has 128 megabytes of memory shared across the isolate. So how do we track keeping track of every repo? Knowing which repos belong to which customer? Access tokens? durable objects. The Git data itself, believe it or not, also durable objects. So we kind of had four design principles at the start of this, and they were basically it would be unacceptable to corrupt a user's Git data. We should never, ever be in a position where we can lose or corrupt a user's data. Stream everything, never buffer, especially because we are constrained by edge compute, 128 megabytes, 1,000 requests per second, single thread.

5:37do as little CPU work in a durable object as possible, and we can't put any one durable object in the hot path. So this is what the control plane architecture looks like. We have our core API worker. We have a kind of per user or per account, I should say, we have this thing called the account namespace index. This keeps track of basically all the namespaces that that account has, or that has the number of repos in that namespace and the jurisdiction. Then we have the registry shard. The registry shards basically keep track of all the repos in a given namespace. And this is sharded to help absorb that load of needing 1,000 creates per second.

6:20Because we can't put any one DO in the hot path, we can't write to the account namespace index in the create path. That would bottleneck everything. We would be constrained to much lower RPS. So every time one of the reg- grocery shards, gets something written to it, marks itself as dirty, alarms, batches new writes, writes that back to the account namespace index. And then at creation time, we also create a token manager. So every repo has a DO that has all the tokens that are minted, like the read or rewrite tokens. They live in a DO. And then the Git server itself is a durable object. And then we have KV just for some nice indexing for reverse indexing on the token ID. And currently, we're using R2 for forking, but that's soon changing.

7:06If you peep the worker desource code, you'll see that a clone API is coming, but nobody heard me say that. So what happens when you create a repo? We hash the repo name, and we basically generate a number 0 through 4. That's how it picks the shard, right? We write the repo row in the control plane. That gets marked as dirty. It spins up this to basically what I just described. I'm not going to waste time on the rest of this slide. So in any given namespace, this comes out to a theoretical throughput of 4,000 crates per second. We chose four just because it gave us wiggle room and buffer to hit that constraint of 1,000. So that's currently what we're sitting at today.

7:49And obviously, the cost of that sharding those, the registry shards, is eventual consistency. So you'll see if. If you create a repo and then try to query a list of all your repos, there's going to be a delay there. But that's just, you know, it's a trade-off we have to make. The data plane, because we, again, can't put any one durable object in a hot path, it never touches the control plane. So your Git client comes in, hits a worker, that talks to the token manager, validates that, make sure you have the correct access, make sure if, like, you're trying to do a push operation, you have a read-only token that it's going to fail, things of that nature. it goes off to the Git server and streams through. So what's inside a Git server? Let's see here. We're going to go through this.

8:36Essentially, the stream connects through the worker into the durable object, and we basically chunk it into two megabyte rows into the SQLite database. So inside that 10 gigabyte database in any given Git server, you'll find every file, every delta, every pack file compressed with ZSTD, all the things in there.

9:02So when you push, which streams through, we check each file, and the main compression algorithm that Git uses is ZSTD, and that compress in there. I should note that today, one of the limitations we have is that artifact repos are constrained to 10 gigs at max. We are going to. hopefully support Git LFS in the future. We have ideas for sharding the Git server itself or possibly using, like, lazy pointers or refs to R2 and putting, like, worker cache in front of that. Git has a nice property that almost all of the data is content addressable, so very useful for caching. So hopefully we'll be able to support larger repos here in the future. Again, clone basically the opposite, but going in the opposite direction.

9:49We actually have some clever optimization tricks about using deltas and chaining those together rather than storing all of the objects together and taking up more memory, which I would love to dive deeper We owe you a blog post on the deep dive of our Git engine, and the team hopes to open source it in the near future.

10:10So yeah, as I mentioned earlier, one of the core principles is don't lose customer data ever. Don't corrupt Git data. And the way we do this is we can that is as the data is streaming in, like you do a new commit and push, as that's streaming in and it's chunking into two megabytes into the rows, we write them to like a quarantine table. And once the full push is in and it lives in the quarantine tables, we basically use another atomic transaction to copy all that data from the quarantine into the main tables and clear it out. So that's how we guarantee that we're never going to lose your data on a push. If something fails, we clear the quarantine, and your prod data still lives in the main tables. So our Git engine, as I mentioned, Matt built it over the holidays during Opus 4.5 psychosis.

10:58It gave us a really nice head start on already having something that can compile a WASM and run in workers. However, as you'll come to see here, our manager, Steve, we rewrote an internal thing from Zig to JavaScript. and it got faster and more reliable, the obvious conclusion is that language doesn't matter. Rewriting does. You should be writing everything all the time. And I think maybe at the beginning of the year, I would have been skeptical at a take like that. But now I am fully convinced that that is true, especially if you're in an organization where you have a decent token budget. Throw inference at improving things and rewriting things. And yeah.

11:42The Git engine basically was 100 kilobytes of Wasm. So it was very small, helped with our cold start times. And it basically communicated to JavaScript through callbacks in Wasm.

11:56Basically, we had the main problems that we run into with Zig over time was that mem copy and data serialization across that boundary really killed us with CPU and memory utilization. So it worked for a couple months, and then as we scaled up and as we added more and more customers into the closed beta, went open beta, as more multi-tenant customers started onboarding their own customers to this, we were pushing, like, probably at the time getting close to 10 million repos. When we started this rewrite, I think we're probably close, if not past 20 million repos today. We were getting hit with. reliability issues with memory issues um so it was either like write it in rust and get it into the runtime of worker d or try to solve it with javascript and one of those things is much easier than the other uh so we did it we uh michael hart one another principal engineer at cloudflare basically worked used auto research used agents and got the full thing rewritten we basically tested it against our entire test suite, conformance test suites.

13:07Nice thing about Git is that there is a lot of tests out there for compliance or conformance. So that makes doing something like this much easier for correctness. You can kind of just let an agent churn until it's correct.

13:25So for a benchmark, WorkerD, it's kind of a maybe over 10-year-old repo now, quite large. For the benchmark, at peak memory, we dropped from... 359 megs down to 40. A push of an empty artifact repo went from 54 seconds to 27 seconds, and the clone time from 17 seconds to 8 seconds. And basically, the memory kind of stays flat as the repo grows over time. And this is basically like a function of

13:56building up in the WASM land from having to do the mem copies and doing the serialization back and forth. So some of you, if you follow me on Twitter, may have seen me tweet about a thorn in my side, a mistake I made. I considered two big technical mistakes I made in my career, one on artifacts, one when I was a very junior dev and I chose Mongo over NoSQL because I didn't understand the difference and lived with the consequence of that for a long time. So the mistake I made was that I forgot to lowercase the repo names when we created it, which led to a situation where you could have repoA and repoA, and there are different places. to different durable objects. And we never had the time to really go and fix this. And fortunately, our current customers had the right access patterns that they weren't causing this problem to exasperate.

14:44You can kind of see why this is hard, as I demonstrated here. That difference in case sensitivity goes to different places. I knew how I wanted to fix it. I knew the right way to do this. I knew the right architecture. I knew the right design. I knew how to migrate the database. I never had the time to do it by hand. We were working. We had a big push internally to get jurisdiction support rolled out. And we weren't having customers run into this problem at this point. I was checking this every day when I logged into work. And agents weren't able to do it. I probably once a week tried to throw an agent at solving this problem and gave it very descriptive ways of how I wanted it solved. And none of them could get it done along chunking up the PR. to make sure that we didn't have version skew. And then Fable 5.1 came out and basically one-shot it.

15:32And that's how the problem got solved. So basically, the fix was lower casing the name when you came in. We had to, on each of the registry shards, add a unique index, making sure that it doesn't clash. And the trade-off that sucks here is that in some of the read pass, we have to basically now fan out and or for repos created before a certain date, we check basically the spelling that comes in and the lowercase version of it. So that's kind of how we got around it. There's a lot more machinery there under the covers, which made this hard to do, but that's kind of where we landed. So takeaways and lessons. Small Agile teams are the future. That's bullshit.

16:18Small AI native teams are the future. Artifacts... built by a team of three in as many weeks. And to my knowledge, we were the first product at Cloudflare that was written entirely by agents. We have not contributed any code by hand.

16:35Wherever you can, cut process, friction, and slow feedback Take humans out of the loop wherever possible and build tooling and automation that lets you trust what they're doing, build trust in what they're doing, verify what they're doing, let them verify. their own work. And honestly, be more ambitious. The models are getting good enough. If you would have told me a year ago we started shipping a project like this, and then three, four months in, we decided to rewrite a Git engine from Zig to JavaScript, I would have told you you were crazy. But that is the world we live in now.

17:15Hot take, maybe not a hot take, but... Writing code by hand is dead, and we should all kind of adjust. And, you know, I've talked about this a lot on Twitter, on our podcast, The Next Token that Cloudflare sponsors. Like, it's a hard thing to grapple with and go through, and I think we actually don't do a very good job at talking about, like, the human experience of that, but the practicality is that this is the reality and the future we're going towards. And, yeah, I think we have to rethink things like code review. things like CICD. This is where currently kind of like where I think my team is feeling bottlenecks is like we're still very much reading every line of code that the agent produces. But especially with Opus 5 .5, we're starting to trust that code more and starting to have to knit it or correct it less and less.

18:04And now the bottleneck is very much becoming us reviewing our own code. And we get into a situation where it's like, oh, I'll have my agent review my co-workers PR and then it's just like agents going back and forth. CICD, like if agents are verifying their work locally, is it really worth spending all the compute to be rerunning CICD all over the place? I don't know. And also deployments. We're experimenting with building our own basically health mediated deployment rollout where it's orchestrated by an an agent, like a normal LM, and something like Clef, give it all our logs, observability, traces, and if it detects at any cohorts of, like, rollout from Free, Pago, Enterprise, you know, up the tiers, it has the autonomy to roll back.

19:00Yeah, and overall, I think Cloudflare is a great place to have, this was the first huge project I built on Cloudflare. I've been at the company about a year. And I've built a lot of stuff on durable objects over the year, but nothing at the scale of creating 1,000 Git rebos a second. And the primitives have gotten pretty good and are very good for agents, whether that's preview deployments, having queues, having durable workflows. Like, there's something there to solve most problems.

19:33And, yeah, Artifacts was built on the shoulders of giants. Like, there is nothing in Artifacts that is built in the . run time, like any one of you could throw your agent at it and replicate the exact design I had here. And yeah, be more ambitious. Thank you. I don't know if we have time for questions, but.

19:59Does this have an integration of their OS? I do not think Artifacts is being used inside. the core version of Cloudflare OS directly. Don't quote me on that, but I don't think so. Does it need a layer to make that integration happen, or is that just not the done yet?

20:21I don't, I actually don't know. It would not take a layer of integration. Like, you could just wire up a binding, and it would be right there accessible in the code the agent's writing, but I don't know if they're using artifacts in the core product.

20:36Can you talk more about your takeaways about how software is dead and what that does to the industry in general? That's a... How much time do we have, everybody? I talk about this a lot in our podcast called The Next Token. Sunil Pai, Reese Sullivan, and I. I think the overall takeaway is that there's going to be more software, and at least today, it still needs, you know... someone that has the experience of building software, having taken down prod, having made mistakes, having shipped bugs. Like, I have the existential dread of, like, how do we onboard new devs into this world? And I don't know what the answer there is.

21:21But I know to stay competitive and to increase the quality of our products, because we should, at the same time that we're using more inference, we should be using that inference to ensure higher bars of quality, like higher thresholds of monitoring and tracing and performance and content layout shift in UIs. But I wish I had the answer.