Cursor: Git at any scale

GitHub's system, Spokes, has been the industry reference since around 2013, and the article is fair to it: its three central choices were correct for the problem as it stood. Replicate at the packfile level rather than trying to distribute Git itself, so ordinary Git tooling keeps working. Keep each replica as a real Git repository on local NVMe, so reads have the access pattern Git expects. And hold the replicas strongly consistent with three-phase commit, so a push is either everywhere or nowhere.

Three-phase commit is worth understanding rather than treating as a label, because its cost is the whole reason for the article. A coordinator asks every replica whether it can accept the push, waits for all of them to say yes, tells them to prepare, waits again, then tells them to commit. Every phase waits for every participant, which means the latency of a push is set by the slowest replica in the set, not the median and not the fastest.

That gives the architecture a property nobody wants: adding replicas makes writes worse. Three is the standard set, and three is plenty for a repository whose readers are humans. It is not plenty for a large enterprise monorepo whose readers are continuous integration jobs, where the read load is enormous and would happily be spread over dozens of machines. But going from three replicas to twelve means every push waits on twelve, so the thing that would fix reads directly damages writes. Read capacity and write throughput are coupled by the consistency protocol, and the coupling has the wrong sign.

There is a second cost that is operational rather than architectural, and the article's phrase for it is the memorable one: the repositories are pets rather than cattle. Because a replica holds state that exists nowhere else in a directly usable form, the system has to know exactly which machines hold which repository, keep that mapping in an external database, and continuously check that each replica is healthy and in agreement. Every one of those is a component that can be wrong, and a stale mapping is a repository nobody can find.

None of this is a design error. It is what happens when a set of correct decisions meets a load nobody was designing for, which in this case is two loads at once: monorepos whose CI fan-out wants many more readers than three, and a newer shape entirely, millions of small repositories created by agents, where the per-repository overhead of tracking and health-checking a pet dominates everything else about them.

The replacement inverts what is authoritative. Instead of the repositories on disk being the truth and the object store being a backup, an append-only write-ahead log in S3-compatible object storage is the truth, and every repository on every disk is derived from it. Each entry in that log describes one accepted change, and the packfiles it refers to are stored as their own objects beside it, so the log stays small and the bulk data is separate.

That is a familiar move with an unfamiliar substrate. A database's write-ahead log records the intention before the change is applied, so recovery is a matter of replaying it; here the same idea is applied across machines rather than across a crash, so a machine that has fallen behind is not repaired, it simply replays. What makes it possible at all is that object storage acquired the one primitive the design needs.

That primitive is an atomic compare-and-swap.

S3 now supports conditional writes: put this object only if it does not already exist, or only if its current version matches the one I read. That turns appending to the log into a race that exactly one participant wins, which is the ordering the mutable half of Git needs. So the consensus that Spokes ran between its own replicas is delegated to the object store, which already provides it, rather than being implemented and operated.

The consistency claim that follows is unusually strong for something built this way, and the article states it plainly: no push is acknowledged until it has been fully persisted, all pushes are linearizable through those atomic operations, and every view of every repository is fully consistent. The order in that first clause is the important part. Persist, then acknowledge, which is the same discipline as a messaging server that stores a message before sending the tick, and for the same reason: the window between accepting and durably storing is the window in which a crash loses something the client believes is safe.

The price is a round trip to object storage in the write path, and it is worth being clear that this is a real cost rather than a free lunch. What it buys is that durability and ordering both come from a service whose availability and durability are somebody else's problem, at eleven nines, with no cluster of your own to keep in agreement. For a workload where pushes are frequent but not latency-critical in the way a database commit is, that is an excellent trade, and the throughput numbers later in the piece are what make the case.

Once the log is authoritative, the repositories on disk change status entirely. They are no longer state that must be protected, tracked and health-checked; they are a warm cache, materialised from the log when a node needs one and discardable at any moment. The article's word for what that buys is stateless: nothing on a node is irreplaceable, so a node is interchangeable with any other node.

That is the change that turns pets into cattle, and the consequences are mostly about what stops being necessary. No external database recording which machine holds which repository, because any machine can hold any repository. No continuous validation that replicas agree, because agreement is checked on every read against the log rather than maintained between peers. No careful repair of a divergent replica, because a divergent replica is thrown away and rebuilt from the log, which is strictly cheaper than reconciling it.

Which node materialises which repository is then decided by rendezvous hashing rather than by a lookup. Every node can compute, from the repository's name and the current node list, which nodes should hold it: hash the name together with each node's name, sort the scores, take the top few. Because it is a computation rather than a stored mapping, there is nothing to keep current and nothing to be stale, and when the node list changes only the repositories whose top scores moved need to relocate. It is consistent hashing's cousin, and it handles weighting more cleanly.

The replica count then becomes a per-repository dial rather than an architectural constant, which is the payoff for all of this. A large monorepo whose CI generates enormous read load can be materialised on hundreds of nodes, because replicas no longer participate in a consensus round and therefore no longer slow pushes down. At the other end, a tiny repository created by an agent can live on exactly one node, because the object store is the availability guarantee: if that node goes away, another materialises the repository from the log.

It is worth noticing what the trade actually is. A read served by a node that does not yet hold the repository must wait for it to be materialised, which is a cold start, and the cost of that is proportional to the repository's size. The design does not eliminate that cost, it relocates it: cold starts happen on the read path instead of divergence happening on the write path, and a cold start is at least a failure mode that resolves itself.

This is the part of the design that repays the most attention, because it is where the correctness actually comes from. When a replica is asked to serve a read, it does not assume that whatever it holds is up to date. It asks the object store, using a conditional GET with the ETag it already has for the log's head.

There are exactly two answers and both are cheap. A 304 Not Modified means nothing has changed since the version the replica holds, so it can serve immediately from its warm on-disk repository; the article reports that this metadata-only operation takes less than 10ms on average. A 200 means the log has advanced and comes with the latest index, so the replica catches up on the entries it is missing before answering, and is then current by construction.

So the guarantee is not that replication succeeded. It is that a stale replica cannot serve a stale answer, because it finds out that it is stale before it answers. That is a materially different property from the usual arrangement, where a replica serves whatever it has and the system's consistency is whatever the replication lag happens to be, and it is why the design can claim that every view of every repository is fully consistent while replicating optimistically.

The cost is one small round trip to the object store per read, and this is the trade to weigh rather than skim. Ten milliseconds is a great deal next to a local disk read and almost nothing next to cloning a repository, which is what makes it acceptable here: Git operations are large enough that a fixed 10ms verification disappears into them, while the same overhead on a per-key cache lookup would be absurd. The technique is not universal; it fits because of the size of the operations it protects.

It is also what makes the write path's promise mean something. Because pushes are linearizable through the log and every read verifies against the log, a client that has just been told its push succeeded will see that push from any replica it subsequently reads from, which is read-your-own-writes without pinning anybody to anything. The two halves of the design are one mechanism seen from two directions.

Given that every read verifies against the log, replication no longer has a correctness job. Its only job is to make the common case fast by getting updates to replicas before anybody asks for them, and something whose only job is to be an optimisation can be built very differently from something that has to be right.

So it is gossip over UDP. A node that accepts a push tells the others, best effort, in a datagram that may simply not arrive. There is no acknowledgement, no retransmission and no delivery guarantee, and the article is direct about why that is acceptable: a lost packet costs a replica one 200 instead of one 304 on its next read, which is to say it costs one catch-up that would have happened anyway. Nothing is incorrect, only slightly slower.

The sentence worth taking from this section is the design goal it states: always correct when degraded, and always fast when healthy. Those two properties usually pull against each other, and the reason they do not here is that they have been assigned to different mechanisms. Correctness lives entirely in the conditional read against the log. Speed lives entirely in the gossip. Neither mechanism is trying to provide both, so neither has to be compromised for the other.

That separation is the transferable idea, and it inverts the usual instinct. Most systems make replication reliable and then read from replicas hopefully; this one makes replication unreliable and reads carefully. The second arrangement is easier to build, because an unreliable broadcast has no failure modes worth the name, and it degrades better, because a replication outage produces slower reads rather than wrong ones. It is also what makes hundreds of replicas practical. Reliable delivery to hundreds of nodes is an increasingly expensive proposition, with acknowledgement tracking and retry state per peer, and it is exactly the cost that made adding replicas painful in the consensus design. Fire-and-forget datagrams cost the sender almost nothing per additional recipient, so the replica count stops being a factor in what replication costs.

The measurements are what turn the argument into evidence, and there are three worth holding. On S3 Standard the system sustains up to 120 pushes per second while simultaneously compacting and replicating the compacted data to every other node. On S3 Express One Zone, the lower-latency variant, it exceeds 300 pushes per second. And stress testing showed linear scaling for reads up to 100 replicas with no regression in push throughput at all.

That third number is the claim the whole architecture was built to make. In the consensus design, replicas and push throughput were coupled by the protocol, so reads could only be scaled by making writes worse. Here they are independent, and the measurement says so: a hundredfold increase in read capacity for no measurable write cost. Everything else in the design is machinery in service of that one property.

The second number is interesting for a different reason: what it says about where the limit now sits. Moving to faster storage roughly tripled push throughput and then stopped being the constraint, because the system became bottlenecked by the speed at which Git can compact the on-disk data. The bottleneck moved out of the storage layer and into Git itself, which is the honest sign that the storage problem has been solved as far as this design can solve it. Optimising the object store further would now buy nothing.

Compaction is also where the design gets one more benefit that is easy to skip past. Only the primary compacts, and the result is applied to both its on-disk repository and the log at once, so every other replica downloads the already-compacted packs from the object store rather than repacking the same data itself. The article's phrase for this is trading bandwidth for CPU, and it removes a genuine hazard from the previous generation, where repacking a large repository across several nodes was an availability risk on all of them at the same time.

One last property is worth naming because it comes free and is the sort of thing you only appreciate after an incident. Every push is in the log, permanently, which means the system has a complete and authoritative record of what happened without any external database keeping it. The article notes this as what lets a Git bug be analysed and remediated retrospectively, and that is a genuine operational difference: the question of what state a repository was in last Tuesday is a lookup rather than an investigation.