Event-driven architecture
Events against commands, the outbox, choreography and the debugging problem nobody mentions.
Events, commands and who decides
intermediateA command says do this and names a recipient. An event says this happened and names nobody. That grammatical difference is the whole architecture: a command couples the sender to the receiver and to the outcome, and an event couples the publisher to nothing at all, because it is a statement of fact that has already occurred.
The consequence is where the decision lives. With commands, the caller decides what should happen next, so adding a behaviour means changing the caller. With events, each consumer decides whether it cares, so adding a behaviour means adding a subscriber and changing nothing that already works. That is the property people are buying when they say decoupled, and it is real.
It is also why events are named in the past tense and describe facts rather than intentions. OrderPlaced is an event; SendConfirmationEmail is a command wearing an event's clothing, and publishing it means the publisher has decided what the consumer does, which is the coupling it was supposed to remove. If the publisher would be upset that nobody handled it, it was a command.
Event content is the next decision and it has two ends. A thin event carrying only an identifier forces every consumer to call back for the details, which is simple and puts read load on the publisher and reintroduces a runtime dependency. A fat event carrying the full state removes that call and means the payload is a copy that can be stale and must be versioned. The middle, an identifier plus the fields consumers actually need, is where most systems end up after trying both.
The honest summary is that events move the complexity rather than removing it. What was an explicit call becomes an implicit contract, and the system gains the ability to add consumers freely and loses the ability to read one file and know what happens next.
The choice is about who decides what happens next. Commands put that decision in the caller and keep it visible; events put it in each consumer and make it extensible. Both are legitimate, and the failure is publishing commands as events and expecting the decoupling anyway.
The naming test is the practical one: an event is a past-tense fact and a command is an imperative with a recipient. Anything called something like SendEmail published to a topic is a command, and the publisher will find out when nobody handles it.
Why are events named in the past tense?
The outbox, and why dual writes fail
advancedThe most common bug in an event-driven system is not in the broker. It is that a service updates its database and then publishes an event, and those are two commits with nothing tying them together. The process can die in between. The broker can be unreachable for the second. The result is a database that says the order is placed and a world that never heard about it, or the reverse.
Retrying does not fix it, because the failure is that there is no atomicity, not that the publish was unlucky. Publishing first and writing second has the same problem with the halves swapped. Wrapping both in a database transaction does nothing, since the broker is not a participant, and adding a distributed transaction across the two is the two-phase commit that everybody avoided for good reasons.
The outbox pattern removes the problem rather than mitigating it. The event is written into a table in the same transaction as the state change, so either both are committed or neither is. A separate process then reads that table and publishes, marking rows as sent. There is still exactly one commit that matters, and the publisher becomes a retryable background job rather than part of the request.
Delivery is then at least once by construction, because a crash after publishing and before marking the row means the event goes out twice. That is the correct trade: duplicates are survivable with an idempotent consumer, and a lost event is not. Anyone who wants exactly once is asking for something the network does not offer, and the honest version is at least once plus deduplication at the consumer.
Change data capture is the same idea with less code, reading the database's replication log and turning committed changes into events without an outbox table at all. It removes the polling process and adds a coupling to the physical schema, which is why teams that want a designed event contract keep the outbox and teams that want the plumbing to disappear reach for the log.
Two commits with no atomicity is the whole problem, and no retry policy repairs it. The outbox turns it into one commit plus a retryable delivery, which is the same move as every other durable-work design: write the intention transactionally, then perform it separately.
Debezium reading a replication log into Kafka is the common implementation of the log-based variant, and the outbox table is the version teams choose when they want the published event to be a designed contract rather than a mirror of their columns.
Why is outbox delivery at least once rather than exactly once?
Schemas, versioning and the contract nobody owns
advancedA synchronous API has an owner, a version and a small set of callers who can be found. An event topic has a publisher, an unknown set of consumers, and messages that may be replayed from months ago. That combination makes schema evolution harder than in a request-response system, and it is the part most teams discover late.
The rules that hold are the familiar compatibility rules applied strictly. Adding an optional field is safe. Removing a field or renaming one breaks any consumer reading it, and you cannot know who that is. Changing the meaning of a field while keeping its name is the change no schema check catches, and in an event stream it also silently rewrites the meaning of history.
A schema registry is the mechanism that turns this from a convention into a check. Producers register a schema and the registry refuses an incompatible change, so the break happens at deploy time rather than in a consumer nobody remembered. Formats with defined evolution rules, Avro and Protocol Buffers among them, exist largely because this problem is old and the rules are known.
Replay makes versioning permanent in a way request-response does not. If events are retained so a new consumer can rebuild its state from the beginning, then every schema version that was ever written is a version some consumer must still be able to read. Either the old messages are upgraded on read, or the consumers keep the code for every version, and choosing that deliberately is much cheaper than discovering it during a rebuild.
The organisational point sits underneath the technical one. An event is a public interface published by a team who often does not know who depends on it, so the discipline that makes it work is treating the schema as a product with an owner, a documented meaning per field, and a deprecation process, rather than as a serialisation detail of whatever the producing service happens to store.
An event schema is a public interface with anonymous consumers and a history that can be replayed, which makes it stricter than an API rather than looser. The registry is what converts the compatibility rules from good intentions into a deployment that fails.
Confluent's schema registry enforces compatibility on producers, refusing a change that would break existing consumers, and Avro and Protocol Buffers both define what counts as compatible. Those tools exist because the problem is old enough to have known answers.
Why is event schema evolution stricter than evolving an HTTP API?
Debugging a system with no call stack
advancedThe cost nobody quotes when adopting events is that the flow disappears. In a synchronous system a request has a call stack and a trace, and you can read the code to see what happens next. In an event-driven system what happens next is the set of everything subscribed, which lives in configuration across several repositories, and there is no single place that describes the sequence.
So tracing has to be deliberate rather than emergent. A correlation identifier created at the edge must be carried in every event and logged by every consumer, or a business flow spanning six services is six unrelated log streams. Modern tracing standards define how to propagate context through messaging as well as HTTP, and using them is what makes an event-driven system observable rather than merely instrumented.
The failure modes are different too, and they are quieter. A consumer that is down is not an error anybody sees, it is a lag number that has to be watched. A poison message stalls a partition, which appears as one entity's updates having stopped while everything else looks healthy. A consumer that processes events out of order writes a state nobody can explain from the events themselves. None of these produces a failed request, so none of them alerts unless somebody built the alert.
Consumer lag is therefore the central metric, and the useful version is the age of the oldest unprocessed event rather than the count. Depth is ambiguous, because ten thousand messages is fine at high throughput and an incident at low, while age answers the question a user would ask, which is how far behind reality this system currently is.
The last practice is to make replay a designed capability rather than an emergency measure. Being able to reprocess a topic from a position, into a fresh consumer, with idempotent handlers, is what turns a bug in a consumer into a re-run rather than a manual reconciliation. Systems that can do this recover from consumer bugs in an afternoon, and systems that cannot spend a week writing scripts to repair state by hand.
Removing the call stack removes the thing that made debugging tractable, and nothing replaces it by default. Correlation identifiers, lag measured as age, and a rehearsed replay path are what put back the ability to answer what happened to this order.
OpenTelemetry defines context propagation for messaging as well as for HTTP, which is what allows one trace to span a publish and its consumers. Without it, a flow across six services is six log streams with no shared key.
A consumer stops processing. Why might nothing alert?