{"canonical_url":"https://engineer.company/notes/optimise-before-you-shard/","content_markdown":"# Optimise the database before you shard it\n\n2026-10-14\n\nWhen a database slows down under load, the first proposal is usually more\nmachines. Sometimes that is right. More often the slow part is a handful of\nstatements nobody has read yet, and spreading the data across servers spreads\nthose statements over more hardware without making one of them cheaper. A\nproblem that looks unsolvable is often one query plan away from a neat answer,\nand horizontal scaling is sometimes chosen precisely so that nobody has to read\nthe database you already have *(read it before you buy more of it)*.\n\nThis is the order we work in before we split anything: look, tighten, take the\nreads off the primary, partition the tables around the hot spot, and shard only\nwhen one machine is full of good work. The flash sale, where every buyer writes\nto one row, is worked through in\n[Selling the last ticket exactly once](/notes/selling-limited-stock-without-overselling/).\nThis note is the method around it, and its last section takes on the opposite\nproposal: sell from every server, and apologise for the units sold twice.\n\n## Look first\n\nPostgreSQL already records most of what there is to know. Before changing\nanything, ask it four questions.\n\n**Which statements cost the most in total?** `pg_stat_statements`, an extension\nloaded through `shared_preload_libraries`, keeps every normalised statement with\nits call count and its time. Sort by total, not by mean: a 2 ms query run a\nmillion times an hour costs more than a two-second report run twice.\n\n```sql\nSELECT calls,\n       round(total_exec_time) AS total_ms,\n       round(mean_exec_time::numeric, 1)\n         AS mean_ms,\n       left(query, 60) AS query\n  FROM pg_stat_statements\n ORDER BY total_exec_time DESC\n LIMIT 10;\n```\n\n**What does the top one actually do?** `EXPLAIN (ANALYZE, BUFFERS)` runs the\nstatement and shows the plan it chose, the rows each step really returned\nagainst the rows it expected, and how many pages it read. A scan of a million\nrows to return ten, or an estimate off by a factor of a hundred, is usually the\nwhole story. `ANALYZE` executes what it explains, so wrap a write in `BEGIN` and\n`ROLLBACK`.\n\n**Who is waiting, and on whom?**\n\n```sql\nSELECT pid, wait_event_type, wait_event,\n       pg_blocking_pids(pid) AS blocked_by,\n       left(query, 60) AS query\n  FROM pg_stat_activity\n WHERE wait_event_type = 'Lock';\n```\n\nRows here in a busy minute mean transactions are queueing for each other, not\nfor hardware, and a bigger machine does not remove the serialization of updates\nto one shared row. More application servers can add writers to that queue.\n\n**Is the server working or juggling?** Count the active sessions in\n`pg_stat_activity` against the cores. When many times more are active than\nthere are cores, the server spends its time switching between them, and a\nconnection pool fixes that before any hardware does. Then read `n_dead_tup` in\n`pg_stat_user_tables`: a table with more dead rows than live ones is being\nscanned through its own garbage, which is autovacuum falling behind, not a\ndatabase that is too small.\n\nWrite down what you find. It is the baseline every later step is measured\nagainst, and the honest answer to \"do we need to shard?\" is often in it\nalready.\n\n## Tighten what is there\n\nRoughly in order of payoff for the effort.\n\n**Indexes that match the questions.** The plans above name the scans. An index\non the columns a hot query filters by turns a scan into a lookup, and a partial\nindex holds only the rows a query can want — units still for sale, say — so it\nshrinks as a sale goes on instead of growing:\n\n```sql\nCREATE INDEX units_unsold\n    ON units (item_id)\n WHERE NOT sold;\n```\n\nEvery index is also paid for on every write, so drop the ones whose `idx_scan`\nin `pg_stat_user_indexes` stays at zero.\n\n**Types that say what the value is.** Schemas written fast, by a person or a\nmodel, reach for `TEXT` where a `SMALLINT` or a `BOOLEAN` would do. A `BOOLEAN`\nis one byte, a `SMALLINT` two, and `'yes'` stored as text is four. PostgreSQL\nalso pads each fixed-width column to its own alignment, so a column order chosen\nby accident costs bytes too. One unit of stock, written fast:\n\n```sql\nCREATE TABLE units (\n  sold        text,  -- 'yes' or 'no'\n  id          bigint,\n  reserved    text,  -- 'yes' or 'no'\n  reserved_at timestamptz,\n  item_id     bigint,\n  order_id    bigint\n);\n```\n\nAnd the same unit with types that say what the values are, widest first:\n\n```sql\nCREATE TABLE units (\n  id          bigint PRIMARY KEY,\n  reserved_at timestamptz,\n  order_id    bigint,\n  item_id     integer NOT NULL,\n  sold        boolean NOT NULL DEFAULT false,\n  reserved    boolean NOT NULL DEFAULT false\n);\n```\n\nOn PostgreSQL 18 the first row measures 72 bytes and the second 54, a quarter\nless, so more of the table fits in memory. The types also refuse values such as\n`'maybe'` that the application would otherwise have to defend against, and a\n`CHECK` constraint does the same for ranges: `quantity smallint CHECK (quantity \u003e 0)`.\n\n**Fewer round trips, and the work where the data is.** Every statement the\napplication sends costs a round trip and a parse. A loop that reads a hundred\nrows and updates them one by one is a hundred and one round trips where one\nset-based statement would do, and a page that fetches every unit of an item to\ncount the free ones ships thousands of rows to compute one number. The rule we\nuse: what has to see the data goes to the database — filtering, counting, a\nwrite that depends on what is there — and business rules stay in the\napplication, because the database's CPU is the hardest one in the system to add\nmore of.\n\n**Updates that stay on their page.** A row that changes many times a minute\nleaves a new version of itself each time. When an update changes no indexed\ncolumn and the page has room, PostgreSQL keeps the new version on the same page\nand touches no index at all: a heap-only tuple, or HOT update. So keep the\ncolumns a frequent update changes out of the indexes, and leave room on the\npages:\n\n```sql\nALTER TABLE units SET (fillfactor = 90);\n```\n\nThe setting applies to pages written from then on, so set it before the rows\nare loaded. On the table above, 2,000 claims that set `reserved`, `reserved_at`\nand `order_id` were all HOT, and of 2,000 updates that set `sold`, which the\npartial index reads, none were HOT. Selling happens once per unit; claiming can\nhappen many times. `n_tup_hot_upd` against `n_tup_upd` in `pg_stat_user_tables`\nshows the ratio on a live system.\n\n## One row says who was first\n\nA unit goes to the first buyer whose statement reaches it, and the claim runs\nout on the database's clock. Two columns carry it: `reserved`, a boolean, and\n`reserved_at`, the moment it was set.\n\n```sql\nUPDATE units\n   SET reserved = true,\n       reserved_at = now(),\n       order_id = :order\n WHERE id = :unit\n   AND NOT sold\n   AND (NOT reserved\n        OR reserved_at\n           \u003c now() - interval '10 minutes')\nRETURNING id;\n```\n\nOne row back: this buyer has it. None: someone else was first, or it is sold.\nTwo claims on the same unit queue on its row lock, and on READ COMMITTED,\nPostgreSQL's default, the second checks the `WHERE` clause again against the\nrow the first one committed, so exactly one of them gets a row back. A claim older than ten minutes counts as free, so nothing\nhas to sweep it up for the answer to be right.\n\nThat makes the boolean alone an incomplete answer. `reserved` stays true after\na claim lapses, and only the two columns together say whether a unit is held,\nso every query that asks has to ask both. A periodic sweep that resets lapsed\nclaims keeps the table honest for anyone reading `reserved` on its own, but\ncorrectness never waits for it. A constraint keeps the pair consistent:\n\n```sql\nALTER TABLE units\n  ADD CONSTRAINT reserved_has_time\n  CHECK (NOT reserved\n         OR reserved_at IS NOT NULL);\n```\n\nThe ticket note stores the deadline, `held_until`, rather than the start; it is\nthe same design. When the buyer picks the unit — a seat, a numbered print — the\nstatement names it, as here. When any unit will do, that note's `SKIP LOCKED`\nform picks one nobody else is holding.\n\n## Reads off the primary\n\nMost of a shop's traffic is reading: the product page, the \"3 left\", the\nbasket, the order history. None of it needs the primary, the one machine that\nhas to stay free for writes. A streaming replica takes those reads, and adding\none is configuration where a shard is a project.\n\nTwo cautions come with it. A replica runs a little behind, so a buyer who has\njust claimed a unit and then reads their basket from it may not see the claim:\nread what a user has just written from the primary, or send that user to the\nprimary for a few seconds after each write. And a long report on a replica can\nbe cancelled when it holds up the changes the replica is replaying. The\nPostgreSQL documentation calls these query conflicts and lists the settings\nthat trade replica lag against them.\n\nThe cheapest read is one that never reaches a database. When every unit of an\nitem is claimed, nobody can buy it until a claim lapses or is released, and the\ndatabase can say exactly when that is:\n\n```sql\nSELECT count(*) FILTER (\n         WHERE NOT reserved\n            OR reserved_at\n               \u003c now() - interval '10 minutes')\n         AS free,\n       min(reserved_at) + interval '10 minutes'\n         AS until\n  FROM units\n WHERE item_id = :item\n   AND NOT sold;\n```\n\nWhen `free` is zero, `until` is the moment the earliest claim lapses: cache\n\"fully reserved\" with that as its expiry, never longer, and every visitor until\nthen is turned away by the cache rather than the database. When `free` is zero\nand `until` is empty, every unit is sold and the answer is final. A claim\nreleased early — a declined card, an abandoned basket — deletes the cache\nentry, so the unit is back on sale at once.\n\nThe cache only ever says no. A yes always comes from the conditional `UPDATE`\nabove, so a stale cache can cost a sale for a few seconds and can never sell a\nunit twice. It is the ticket note's cached \"sold out\", with a deadline.\n\n## Partition the tables around the hot row\n\nPartitioning splits one table into several by a key — a month, a tenant —\nbehind one name. It is worth being plain about what it does not do: it does not\nmake one contended row any faster. That row still lives in one partition, on\none machine, behind one lock.\n\nWhat it does is keep the tables around the hot spot small. Orders, claim\nhistory and events grow for ever, and most queries want the recent part:\n\n```sql\nCREATE TABLE orders (\n  id          bigint NOT NULL,\n  created_at  timestamptz NOT NULL\n              DEFAULT now(),\n  item_id     integer NOT NULL,\n  quantity    smallint NOT NULL\n              CHECK (quantity \u003e 0),\n  paid        boolean NOT NULL DEFAULT false,\n  PRIMARY KEY (id, created_at)\n) PARTITION BY RANGE (created_at);\n\nCREATE TABLE orders_2026_10\n  PARTITION OF orders\n  FOR VALUES FROM ('2026-10-01')\n              TO ('2026-11-01');\n```\n\nA query that names a date range reads only the partitions inside it, and this\nmonth's indexes stay small enough to live in memory. Retention stops being a\n`DELETE` of millions of rows, with the bloat and the vacuuming that follow, and\nbecomes one statement:\n\n```sql\nALTER TABLE orders\n  DETACH PARTITION orders_2025_10;\n```\n\nThe primary key has to include the partition key, which is why it is\n`(id, created_at)`. A query that does not name the key reads every partition,\nso partition by what the queries already filter on.\n\n## When sharding is reasonable\n\nThere is one limit sharding cannot move: every item has one owner. A unit\ncannot be claimed on two servers at once, so sharding moves whole items, never\nhalf of one. When a single item outgrows its machine, the ticket note's buckets\nsplit its stock into smaller pieces, and each piece again has exactly one\nowner.\n\nWithin that, we call sharding reasonable when one of these is true, and not\nbefore:\n\n- **One machine is full of good work.** The statements are already lean — one\n  conditional write each, nothing external inside a transaction, updates that\n  stay on their page — and the primary's CPU or write-ahead log is still\n  saturated, on the largest machine worth buying, across many items rather than\n  one. One hot item is a bucket problem, not a shard problem.\n- **The data has outgrown its maintenance.** A backup cannot be restored inside\n  the time promised to the business, an index cannot be built or a table\n  vacuumed in the quiet hours, or the working set no longer fits in memory.\n- **Isolation is the requirement.** A tenant whose contract or regulator says\n  their data lives apart, or whose sale must not share a disk with anyone\n  else's.\n\nThe cost lands on everything else. Migrations, backups, restores and reports\nhave to account for separate shards; a join across shards may move into\napplication code; and a query that does not know its shard key may have to ask\nall of them. That is the bill the sections above exist to avoid, and once it is\npaid it is hard to get back.\n\n## When an apology is cheaper than a lock\n\nThere is a serious argument for going the other way. Let every server sell from\nthe stock it can see, accept that now and then a unit goes to two buyers, and\nlet customer support refund one of them with an apology. Business logic becomes\neventually consistent, built for the 99.9% of requests that never collide, and\nan asynchronous loop finds the collisions afterwards and makes them good. In\nreturn the shop never has to say \"try again\". Helland and Campbell make the\nstrongest form of the case in _Building on Quicksand_: a system that\nacknowledges work before its backup has it makes every promise probabilistic,\nand businesses have always settled the promises they could not keep with\napologies.\n\nWe agree with most of that. We disagree about where it is spent.\n\n**Most of a shop should be eventually consistent.** The catalogue, the search\nindex, the \"3 left\" badge, the basket, the recommendations, the emails, the\nwarehouse export and the reports can all run seconds behind, and making them\nwait for the primary costs speed and buys nothing. Dynamo was built for exactly\nthis: Amazon's shopping cart never forgets or rejects an \"add to cart\", even\nwhile machines fail, and the price is that a merge can bring back an item the\nbuyer had removed. After the claim, the ticket note's checkout is already\nasynchronous: a saga of local steps, messages delivered at least once, and\nconsumers that are idempotent, keyed by the order. A reconciliation loop belongs\nin any design that talks to a payment provider, because webhooks get lost.\n\n**The claim is the one place it does not hold.** \"Rare\" assumes collisions are\nrare, and in a flash sale the collisions are the sale. A server that sells from\nits own view keeps selling until it hears what the others sold, so the oversell\nis roughly the attempt rate times the replication lag. In a simple simulation,\ntwo regions split 2,000 attempts a second, each sells from its own count and\nhears of the other's sales 200 milliseconds later. Across five runs of a\n10,000-unit sale, each run sold between 191 and 205 units that did not exist.\nThe arithmetic predicts 200: the attempt rate, times the lag, times the share of\nsales made in the other region. With one owner per item, the same model sold\nexactly the stock. That happens on every sell-out, whether anything failed or\nnot, and it lands at the climax, when the most buyers are watching.\n\n**The uptime is smaller than it sounds.** Gilbert and Lynch proved the trade\nthat eventual consistency makes: while the network is partitioned, a system can\nanswer or it can be consistent, not both. Abadi's PACELC adds the other half.\nWhen nothing is partitioned, the trade is latency against consistency, and one\nowner per item, with lean statements, is already fast. Neither design reaches\n100%: the payment provider, DNS and the networks between them have outages of\ntheir own. What the strict design gives up is narrow. When the owner of one\nitem's stock cannot be reached, that item stops selling for the seconds a\nfailover takes, and the ticket note's section on pausing the item scopes the\npause to the items on the failed machine. The rest of the shop keeps selling.\n\n**An apology is not a refund.** The refund is the cheap part. The buyer was told\nyes, and may already have booked the train, taken the day off and told their\nfriends. Hearing \"sold out\" a second too early is a disappointment. Hearing that\na confirmed order has been withdrawn is a broken promise. Support is staffed for\nan ordinary day, and oversells arrive together, in the busiest minute of the\nyear. Before planning to withdraw confirmed orders, ask a lawyer what a\nconfirmation commits the shop to.\n\n**Keep the asynchronous part, and move it before the yes.** What the proposal\ngets right is that the buyer does not have to wait on a lock. Accept every click\ninto a durable, replicated log, one partition per item, and tell the buyer the\nanswer is coming. Accepting a click then depends only on the log, and a log\nreplicated across three machines keeps taking writes when one of them fails. One\nowner per item reads the log in order and decides, at memory speed, and the\nanswer goes back as a message within seconds. When that owner fails over, the\nrequests wait in the log instead of being refused or sold twice. The yes is a\nhold, and payment follows it as in the ticket note's checkout, so nobody pays\nfor a unit they did not get. The reconciliation still happens, but before the\npromise instead of after it.\n\n**Share the stock, not the count.** When the business needs regions to sell\nwithout asking each other, give each region a quota of the stock and let it sell\nfrom its own quota without asking anyone. A region whose quota runs dry asks for\nmore, and near the end of the sale the quotas collapse back to one owner. Bailis\nand colleagues work out which rules servers can keep without talking to each\nother, and a stock that must not go below zero is not one of them. Quotas do not\nremove that conversation; they move it off the hot path, to the rare moment a\nquota is refilled. That is the eventually consistent design that keeps the\ninvariant. Its one cost is the tail, where a buyer is turned away in one region\nwhile another region still has stock.\n\n**When an oversell is the right policy.** Some goods can be made again: a print\non demand, a T-shirt the supplier still holds, a licence key. Selling past the\ncount is then a backorder with an honest date, and eventual consistency is the\nright design. Airlines overbook on purpose. They know their no-show rates, price\nthe risk and compensate by a rule written in advance. That is the test. An\noversell is acceptable when the business chose its size and its remedy before\nthe sale. It is a defect when it falls on whoever happened to click last.\n\n**If you choose apologies, build them like payments.** The loop the proposal\ndescribes is a refund process, and it needs what anything that moves money\nneeds:\n\n- **An id on every sale,** so a retry is the same sale, and an append-only\n  ledger of sales. The stock number is derived from the ledger, never edited.\n- **A rule for who keeps the unit** that you could explain to the buyer who\n  lost. Order by the log, not by the clocks of different servers, where a few\n  milliseconds of skew would decide who gets the refund.\n- **Compensations that are idempotent and logged:** cancel, void or refund, tell\n  the buyer. They are the saga's undo steps, run by a machine.\n- **A deadline.** The loop has to find every oversell before anything ships, and\n  before the buyer finds it.\n- **A budget, measured.** Oversells per sale, against the number the business\n  signed for, next to the invariant on the same dashboard, with an alert when\n  the budget is spent.\n\nOur answer is to be eventually consistent everywhere around the claim, strict at\nthe claim, and asynchronous in how the answer reaches the buyer. Design for the\ncommon path, but in a flash sale the collision is the common path. If the\nbusiness wants a few oversells in return for something, make it a number the\nbusiness chose, with a remedy written before the sale, and measure it.\n\n## What we would tell the next person\n\n- **Read before you buy.** `pg_stat_statements` and one `EXPLAIN` answer more\n  capacity questions than a new server does.\n- **Make the types say what the data is.** A boolean stored as a boolean is\n  smaller, and it refuses nonsense.\n- **Send the database what has to see the data, and nothing else.** Filtering\n  and counting belong there. Business rules do not.\n- **The cache says no; only the database says yes.** Then a stale cache costs\n  seconds, never a unit.\n- **Partitioning is housekeeping.** It keeps the tables around a hot row small\n  and does nothing for the row.\n- **Shard when one machine is full of good work,** not when it is full of\n  waiting.\n- **Reconcile before the yes, not after.** Everything around the claim can be\n  eventually consistent. The claim itself, made that way, oversells by the\n  attempt rate times the lag, on every sale.\n\n## Sources\n\n- [PostgreSQL: pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html)\n- [PostgreSQL: Using EXPLAIN](https://www.postgresql.org/docs/current/using-explain.html)\n- [PostgreSQL: the cumulative statistics system](https://www.postgresql.org/docs/current/monitoring-stats.html)\n- [PostgreSQL: numeric types](https://www.postgresql.org/docs/current/datatype-numeric.html)\n- [PostgreSQL: character types](https://www.postgresql.org/docs/current/datatype-character.html)\n- [PostgreSQL: Boolean type](https://www.postgresql.org/docs/current/datatype-boolean.html)\n- [PostgreSQL: partial indexes](https://www.postgresql.org/docs/current/indexes-partial.html)\n- [PostgreSQL: heap-only tuples (HOT)](https://www.postgresql.org/docs/current/storage-hot.html)\n- [PostgreSQL: table partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html)\n- [PostgreSQL: hot standby and query conflicts](https://www.postgresql.org/docs/current/hot-standby.html)\n- [Pat Helland and Dave Campbell, Building on Quicksand](https://arxiv.org/abs/0909.1788)\n- [Seth Gilbert and Nancy Lynch, Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services](https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf)\n- [Daniel Abadi, Consistency Tradeoffs in Modern Distributed Database System Design](https://www.cs.umd.edu/~abadi/papers/abadi-pacelc.pdf)\n- [DeCandia et al., Dynamo: Amazon's Highly Available Key-value Store](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)\n- [Hector Garcia-Molina and Kenneth Salem, Sagas](https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf)\n- [Bailis et al., Coordination Avoidance in Database Systems](https://www.vldb.org/pvldb/vol8/p185-bailis.pdf)\n\n\u003chttps://engineer.company/notes/optimise-before-you-shard/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"Most databases that seem to need sharding need reading first. The order we work in: measure, tighten, move reads off, partition, then shard.","formats":{"gemtext":"https://data.engineer.company/notes/optimise-before-you-shard/index.gmi","gopher":"https://data.engineer.company/notes/optimise-before-you-shard/gophermap","html":"https://data.engineer.company/notes/optimise-before-you-shard/","json":"https://data.engineer.company/notes/optimise-before-you-shard/index.json","markdown":"https://data.engineer.company/notes/optimise-before-you-shard/index.md","pdf":"https://data.engineer.company/notes/optimise-before-you-shard/index.pdf"},"kind":"page","language":"en","title":"Optimise the database before you shard it","url":"https://data.engineer.company/notes/optimise-before-you-shard/","version":1}