Financial systems · Rust · Node.js · gRPC · MemoryDB · PostgreSQL
TL;DR: for a prediction market we built a trading engine in Rust whose state depends only on an ordered, durable log of commands. Orders are applied in memory by a single thread, so there are no locks and no garbage collector pauses at the end-of-event peak, and the order of the commands is set by a managed log on Amazon MemoryDB instead of a consensus protocol. In our benchmarks the core applies about 250,000 orders per second on a single thread. The result is a set of verifiable guarantees: every command is applied at most once, from the app to the engine; every replica reaches the same state; every money movement is a double-entry record, reconciled with the database every hour; new rules are switched on by a command in the log. The tests compare the system with deliberately simple references.
A prediction market is an exchange where forecasts are traded. For every event (“who wins the match?”) people buy and sell contracts on the possible outcomes: a YES contract pays 1 if the outcome happens, a NO contract pays 1 if it does not, and the YES price, between 0.01 and 0.99, is the probability the market assigns to that outcome. The platform never bets against its users: it matches those who believe in an outcome with those who believe the opposite, and earns a fee on every trade.
For this client we built the whole system: backend, apps for iOS, Android and the web, back office, infrastructure and trading engine. The engine is the component where a mistake costs the most. An order applied twice, a wrong rounding or two servers that disagree on the state are not cosmetic defects, but money created or destroyed, in a regulated product where users’ funds must be reconciled. And the load is not even: it concentrates in the last minutes of every event and on the same outcomes, exactly when users are most active and the platform earns the most.
The most common way to build a system like this is to handle the hard cases one by one: a lock here, an idempotency table there, a job that reconciles overnight. We took the opposite approach: an engine in Rust whose state depends only on an ordered, durable sequence of commands. If the same commands always produce the same state, replicas, restarts, retries and duplicate messages stop being cases to handle and become harmless by construction. And the same choices that make it correct also make it fast: in our benchmarks the core applies about 250,000 orders per second.
The bottleneck is a contended book, and adding servers does not fix it
Faced with a traffic peak, the natural reflex is to scale out. Here it does not work: the orders of a book must be applied one after the other, in an order everyone has to share, and a user’s balance is shared across every book they trade on. A relational database with a lock per order would turn the final burst into a queue of conflicting transactions, exactly when it matters most.
So we split the responsibilities. The Node.js backend handles everything that is “application”: authentication, catalogue, regulatory checks, notifications to the apps. The engine keeps books, balances, contracts, fees and spending limits in memory, and applies commands one at a time, on a single thread. It looks like a step backwards, but it is the choice of systems such as LMAX and TigerBeetle: with no locks and no coordination between threads, a complete order, from validation to the settlement of both sides, takes a few microseconds.
The language, too, came out of a measurement, not a preference. We compared the time needed to process a block of 1,000 orders with the same simplified engine and a large state:
| Language | Median | p99.9 | Worst case |
|---|---|---|---|
| Rust | 0.85 ms | 1.42 ms | 1.6 ms |
| Node.js | 4 ms | 21 ms | 658 ms |
The decisive figure is the last column. Node.js’s 658 ms are a garbage collector pause, and in an exchange a pause like that tends to land right on the peak, because that is where allocation is heaviest. In Rust, with no garbage collector, the worst case stays close to the typical one: the order placed in the last second is treated like the one placed at leisure.
One book per outcome: more liquidity for users, verifiable accounts for the platform
The most obvious model treats YES and NO as two separate securities, each with its own book. But then someone buying YES at 0.40 and someone buying NO at 0.60, who together form exactly a pair worth 1, would never meet: it would take a counterparty in the middle, which is the role the platform chose not to play, and the result would be thinner books and worse prices.
We gave every outcome a single book, in which every price is a YES price. Buying NO at 0.60 is equivalent to bidding 0.40 on the opposite side, and selling a YES you hold has the same effect on the book as buying a NO. All orders end up on just two sides, and every match creates a new pair of contracts, passes it from one user to another, or closes it and returns the money.
The benefit goes beyond liquidity. A simple invariant follows from this model: for every outcome, the YES contracts in circulation equal the NO contracts, and every pair is backed by one unit of deposited money. The money that came in minus the money that went out always equals the balances, plus the money locked by open orders, plus the fees, plus the value of the pairs in circulation. Accounting becomes a conservation law, and a conservation law can be checked. Fees are designed to respect it too: a configuration in which a match would pay out more than it collects is refused.
Determinism is a discipline, and we make it verifiable
A single thread guarantees an order, not the same result on different machines. Divergence between replicas rarely comes from a matching error: it comes from details that look harmless, such as the system time, the order in which a map is iterated or a rounding. That is why the core is a pure library, and each of its rules closes one of these paths.
Time arrives with the command, assigned by the log, never from the machine’s clock: two replicas cannot decide differently whether an order arrived before or after the close. Money is an integer throughout the system, and an invalid amount is refused, never rounded. Every operation checks everything before writing, so a refused command leaves the state exactly as it was. On top of these rules there is a safety net: if an internal check fails while the engine applies a command, the engine stops instead of carrying on, because a wrong balance would be replicated everywhere and presented as correct. The rules are designed so that this does not happen, and the tests verify it after every command.
Two tools turn the discipline into something that can be verified. An audit recomputes every invariant from scratch, from the conservation of money to the parity between YES and NO: the tests run it after every command, and at runtime every replica runs it on every snapshot it saves, so an anomaly surfaces within minutes, not at the end of the month. A fingerprint (digest) of the whole state lets us compare two replicas at the same log position with a single number, and a snapshot that yields a different fingerprint once reloaded is discarded. The fingerprint is built so that a field added to the state without being included does not compile: the check stays valid after the next change, whoever makes it.
A managed log instead of a consensus protocol
If the state is a function of the commands, it is enough to make the commands durable. What remains is deciding who sets their order. The textbook answer is a consensus protocol such as Raft, in which a leader elected by the nodes receives the commands and replicates them. We evaluated it in the form of Aeron Cluster: sub-millisecond commits, but an operational cost, between nodes with a stable identity and orchestrated releases, that the client would have paid forever.
We chose to let the log set the order instead of a leader. Every command is written to a Redis stream hosted on Amazon MemoryDB, which acknowledges a write only after recording it in its own transaction log, spread across multiple availability zones: a failover does not lose an acknowledged command. Every engine replica accepts commands, appends them to the log, reads the whole log and computes the whole state. There is no leader to elect, and during a release the old and the new version can run side by side, as long as they apply the same rules: we will see below how we guarantee that.
Removing the leader leaves three problems open, and each has a simple answer.
The first is throughput: writing commands to the log one at a time soon saturates the replica. Instead, every replica groups the commands that arrive while the previous write is in flight and writes them together, the same principle as the group commit of databases. At low traffic a command goes out immediately, under load the groups grow with the traffic, and in our benchmarks throughput nearly doubles. Replicas serve availability and reads, and the log sits behind an interface that lets us replace it without rewriting the core, on the day more capacity is needed.
The second is time: if the engine does not read the clock, something has to tell it that an event must be closed. When a deadline arrives, every replica appends a “tick” command: the first one applied runs the close, the others find nothing to do. An order arriving even one millisecond after the close finds the event already closed, so nobody can trade past the closing time, and there is no cron job in the backend that could get the time wrong or contradict the engine.
The third is reads. A replica can be a few milliseconds behind, and a user must not see the order they just placed disappear. Every write returns its position in the log, the app sends it back with the next read, and the replica waits, if needed, until it has applied it: on any replica, the user never sees a state older than one already seen.
Hundreds of thousands of orders per second: correctness does not cost speed
Guarantees this strong are usually paid for in performance. Here the opposite happens: with no locks, no garbage collector and no consensus protocol to go through, the sensitive work comes down to a few microseconds of in-memory computation. In our benchmarks, where every order pays for everything a real order pays for, from validation to the settlement of both sides:
| Scenario | Orders per second |
|---|---|
| All orders on the same outcome | about 250,000 |
| 1,000 open outcomes and one million users | over 150,000 |
| Complete engine, with every command durably written | about 80,000 |
The figure that matters most is the first: the scenario every exchange worries about, everyone on the same outcome in the last minutes, is the one in which the core is fastest. And to keep performance from degrading over time, memory and instructions per order are CI thresholds: a pull request that makes the engine noticeably slower does not pass.
Retries are always safe, from the app to the engine
A user who taps “buy” on a flaky network does not know whether the order went through. If they try again, they must get the order exactly once, and for a withdrawal a duplicate costs even more. The most common solution is an idempotency table in the backend, but here the effect happens in the engine, and an external table always leaves a window between the execution and the recording of the outcome. We moved the guarantee to where the effect happens: the command id is born from the user’s intent, and the engine does the deduplication.
The app generates a key for every intent and reuses it until the outcome is certain. The backend derives the command id from it, also bound to the user, so a key can never point to someone else’s order. The engine remembers every applied id for a period of time together with the reply it gave, refusals included, and answers a repeated command with the original reply and no effect. The backend can therefore retry with the same id, even on another replica, and if it runs out of time it answers honestly: “outcome unknown, try again”. Whether the write failed, the reply was lost or the replica went down right after the write, the command is applied at most once.
There is one constraint that usually nobody writes down: every automatic retry, from the app to the jobs that move money, must finish before the engine stops remembering that id. The backend checks this at startup and, if the configuration violates it, refuses to start. It is the kind of error that would otherwise be discovered in production, on the day someone lengthens a timeout.
Every money movement has an identity, and a duplicate costs nothing
A regulated platform does not just need correct balances: it needs to be able to show how they were reached. That is why every movement, of money or of contracts, leaves the engine as a transfer between two accounts, in double entry: the transfers alone are enough to rebuild every balance, and the tests verify it after every command.
Bringing these movements into the database is where a replicated system usually gets complicated: who exports, how duplicates are avoided, what happens if the exporter fails halfway. Determinism makes it possible to answer without coordination, because every event has a deterministic identity: its position in the log plus a sequence number, the same on every replica and on every replay. PostgreSQL uses it as a unique key and discards duplicates, so a standard SQS queue, which may deliver a message more than once, is enough. Only one replica at a time exports, but only for efficiency: if two replicas export together for a moment, the database just receives duplicates, and no coordination protocol is needed.
On these foundations reconciliation becomes routine: every hour a job compares the engine’s totals with the sum of the accounting records in PostgreSQL at the same log position, and any difference is flagged. Before the log is compacted it is archived, so every archived period can be replayed from scratch for a check.
Changing the rules of an exchange while it is open
A deterministic engine raises a problem that traditional systems do not have. In a rolling release the new replica and the old one run side by side on the same log: if the new one applied a different fee or a new check, the two would diverge. And at startup the new version replays commands that the old one had already applied and confirmed to users: recomputing them with new rules would mean rewriting the past.
We separated the release from the activation. The version of the rules is part of the state and changes only through a command in the log, sent by an operator from the back office: every replica switches rules at exactly that entry, and a replay applies every entry with the rules of its moment. Until the new version is active, the new binary computes and writes exactly what the previous one did, so the release changes nothing and can be rolled back. A replica that meets a version it does not know stops before applying it, instead of diverging, and restarts with the new binary. Every release includes a state and a stretch of log written by the previous binary, and on every pull request the CI verifies that the new one reads them back and writes them out identically: the ability to roll back is verified, not assumed.
This is how the engine grows with the product. The cost of each position including the fees paid, which changes what the engine computes for every user and what the app shows in the profile, was released this way: switched off, ready to be activated by a command in the log, without resetting the state or rewriting history.
Tests that compare the system with a simple reference
Testing a distributed system that moves money usually means writing many scenarios and hoping you thought of the right ones. Determinism allows something stronger: comparing the system with a reference simple enough to check by hand. Property-based tests generate random, reproducible sessions of hundreds of commands, and verify that the invariants hold after every command, that two engines with the same inputs agree on replies, events and fingerprint, and that matching gives the same results as a deliberately naive implementation.
The same principle applies to replicas. Clusters of 2 to 10 replicas, under random concurrent traffic with duplicate submissions, abrupt stops, releases and restarts, must produce exactly the same results as a single engine reading the log in order. In the failover tests of the log, the longest pause between two replies during a scheduled node switch was about 50 ms, and with the primary node suddenly frozen, writes resume about three seconds after the standby node is promoted, without losing any acknowledged command. In the backend, race conditions and replies lost after an executed order are reproduced deterministically against a real engine binary, so a concurrency defect fails the test on every run, not once in a hundred.
The value for the platform
For a prediction market the trading engine is the product: it decides who bought what, at what price and who gets paid at the end. What we built gives the client verifiable guarantees instead of promises. Money is conserved, and an audit checks it. Every operation is applied at most once, from the app to the log. Every replica reaches the same state, comparable with a single number. Every movement has an accounting record, and every hour the engine’s totals are reconciled with the database. And none of this costs speed: in benchmarks the core applies hundreds of thousands of orders per second, even with all the traffic on the same outcome. The same choices make the system simple to run and to evolve: the sensitive work lives in a single thread, durability is entrusted to a managed service, and new rules come into force without stopping the market.
The method applies beyond this project. Diverging replicas, lost replies, duplicate messages and half-finished restarts are not solved by handling them one by one, but by making the system deterministic and every one of its outputs identifiable. At that point repeating an operation is harmless, and correctness stops being a hope and becomes something the tests verify with every change.