<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[How They Scaled]]></title><description><![CDATA[How They Scaled]]></description><link>https://howtheyscaled.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sun, 01 Nov 2026 20:43:52 GMT</lastBuildDate><atom:link href="https://howtheyscaled.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[How Figma, Notion and Slack handled sharding]]></title><description><![CDATA[Horizontal sharding is the move every team hopes to put off. You take one database, split its rows across many, and suddenly every query, migration and…]]></description><link>https://howtheyscaled.hashnode.dev/how-figma-notion-and-slack-handled-sharding</link><guid isPermaLink="true">https://howtheyscaled.hashnode.dev/how-figma-notion-and-slack-handled-sharding</guid><category><![CDATA[database]]></category><category><![CDATA[sharding]]></category><category><![CDATA[scalability]]></category><category><![CDATA[architecture]]></category><dc:creator><![CDATA[How They Scaled]]></dc:creator><pubDate>Mon, 19 Oct 2026 18:57:05 GMT</pubDate><content:encoded><![CDATA[<p>Horizontal sharding is the move every team hopes to put off. You take one database, split its rows across many, and suddenly every query, migration and on-call runbook has to know where the data lives.</p>
<p>I've been collecting first-hand scaling write-ups, and five of them are tagged horizontal sharding: Figma, Notion, Slack, Shopify and PlanetScale. Figma, Notion and Slack are the most useful to compare, because they took noticeably different routes. Shopify and PlanetScale fill in the rest of the picture.</p>
<h2>Figma: buy time first, then shard logically</h2>
<p>Figma's story is really two posts.</p>
<p>The first, <a href="https://www.figma.com/blog/how-figma-scaled-to-multiple-databases/">The growing pains of database architecture</a> (2023), is the "put it off" chapter. Figma moved from <strong>one Postgres instance to multiple databases by vertically partitioning table groups</strong>, using read replicas and connection pooling along the way. No sharding yet, just moving whole groups of tables to their own databases.</p>
<p>The second, <a href="https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/">How Figma's Databases Team Lived to Tell the Scale</a> (2024), is the real thing: a <strong>nine-month effort to horizontally shard</strong> its Postgres stack on AWS RDS. Two ideas stand out:</p>
<ul>
<li><strong>A custom query proxy.</strong> The application talks to a proxy, and the proxy works out which shard a query belongs to. That keeps routing logic out of every service.</li>
<li><strong>Logical sharding before physical splits.</strong> Figma first made the system <em>behave</em> as if it were sharded, then moved data onto separate physical databases afterwards.</li>
</ul>
<p>That second point is the one I'd copy. It splits a terrifying project into two smaller ones. You prove your routing and query patterns work while all the data still sits in one place, and only then do the risky data move.</p>
<h2>Notion: shard by workspace, then do it again</h2>
<p>Notion's <a href="https://www.notion.com/blog/sharding-postgres-at-notion">Herding elephants: Lessons learned from sharding Postgres at Notion</a> (2021) is a classic for a reason. Notion split its Postgres monolith into <strong>a fleet of logical shards keyed by workspace</strong>.</p>
<p>The workspace is a natural shard key for a product like Notion: most reads and writes stay inside one workspace, so most queries only ever touch one shard.</p>
<p>To move the data, Notion used <strong>double writes and backfills</strong>: write new data to both the old and new locations, copy the historical data over, then switch reads once the two match.</p>
<p>Then the sequel. In <a href="https://www.notion.com/blog/the-great-re-shard">The Great Re-shard: adding Postgres capacity (again) with zero downtime</a> (2023), Notion <strong>tripled its Postgres fleet</strong> by re-sharding with <strong>logical replication</strong> and careful <strong>PgBouncer cutovers</strong>, with <strong>no downtime</strong>.</p>
<p>The lesson here is that sharding isn't a one-time project. If you pick the right key and use logical shards, adding capacity later becomes "move shards to more machines" rather than "redesign the schema."</p>
<h2>Slack: shard by any key, not just workspace</h2>
<p>Slack's <a href="https://slack.engineering/scaling-datastores-at-slack-with-vitess/">Scaling Datastores at Slack with Vitess</a> (2020) is the interesting contrast. Slack ran MySQL as <strong>active-active pairs</strong> and spent <strong>several years</strong> migrating to Vitess, a sharding layer for MySQL. The post cites <strong>2.3 million QPS</strong>.</p>
<p>The key reason: Slack wanted to <strong>shard data by any key, not only by workspace</strong>.</p>
<p>That's the opposite lesson from Notion, and both are right for their products. Notion's data mostly lives inside a workspace. Slack has data that doesn't fit that model cleanly, so a workspace-only scheme became a limit. Vitess let Slack pick the shard key per table instead of forcing one for everything.</p>
<p>If you're choosing a shard key today, ask: what's the biggest thing in my product that <em>crosses</em> tenant boundaries? If the answer is "not much," shard by tenant. If it's "a lot," plan for more than one key.</p>
<h2>What Shopify adds: design the keyspace, then keep rebalancing</h2>
<p>Shopify has two sharding posts on MySQL.</p>
<ul>
<li><a href="https://shopify.engineering/horizontally-scaling-the-rails-backend-of-shop-app-with-vitess">Horizontally scaling the Rails backend of Shop app with Vitess</a> (2024) covers how the Shop app adopted Vitess to shard a Rails backend, including <strong>keyspace design and the migration plan</strong>.</li>
<li><a href="https://shopify.engineering/mysql-database-shard-balancing-terabyte-scale">Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale</a> (2021) covers the day-two problem: shards get uneven, so Shopify <strong>moves individual shops between shards online</strong> with its Ghostferry tool, at terabyte scale.</li>
</ul>
<p>Together they make the same point Notion's re-shard does. Getting sharded is the start. You also need a safe, boring way to move one tenant at a time.</p>
<h2>What PlanetScale adds: a ceiling check</h2>
<p><a href="https://planetscale.com/blog/one-million-queries-per-second-with-mysql">One million queries per second with MySQL</a> (2022) is a <strong>benchmark</strong>, not a production story. PlanetScale load-tested a horizontally sharded Vitess cluster up to <strong>one million QPS</strong>. It's useful as a sanity check on how far the Slack and Shopify approach can stretch, as long as you read it as a load test.</p>
<h2>Five patterns across all five stories</h2>
<ol>
<li><strong>Delay it honestly.</strong> Figma vertically partitioned first. Splitting table groups apart is often enough for a while.</li>
<li><strong>Put a routing layer in the middle.</strong> Figma built a query proxy. Slack, Shopify and PlanetScale use Vitess. Either way, the app shouldn't hard-code where data lives.</li>
<li><strong>Separate logical from physical.</strong> Figma sharded logically before splitting physically, and Notion's logical shards made its re-shard possible.</li>
<li><strong>Choose the key from your product, not the database.</strong> Workspace worked for Notion. Slack needed more flexibility.</li>
<li><strong>Budget for moving data forever.</strong> Double writes and backfills (Notion), logical replication (Notion again), Ghostferry (Shopify). Every team ended up needing a way to move data while the product stayed up.</li>
</ol>
<h2>The honest summary</h2>
<p>None of these teams sharded early, and none of them describe it as easy. Figma's took nine months. Slack's took years. What they share is that they made the data move a separate, testable step, and they planned for doing it again.</p>
<p>All five horizontal sharding stories, with stack and scale details, are on one page: <a href="https://howtheyscaled.com/technique/horizontal-sharding/">Horizontal sharding: real scaling case studies</a>. There's a broader <a href="https://howtheyscaled.com/technique/sharding/">sharding page</a> too, with Discord's guild-sharded Elasticsearch search, Meta's ZippyDB, Stripe's DocDB and Datadog's Husky.</p>
<p>Have you sharded by tenant and regretted it, or gone multi-key and wished you hadn't? I'd love to hear about it in the comments.</p>
]]></content:encoded></item><item><title><![CDATA[What 11 PostgreSQL scaling stories taught me]]></title><description><![CDATA[I've been collecting first-hand "how we scaled" posts from engineering blogs, and PostgreSQL shows up more than any other database: 11 stories from 7…]]></description><link>https://howtheyscaled.hashnode.dev/what-11-postgresql-scaling-stories-taught-me</link><guid isPermaLink="true">https://howtheyscaled.hashnode.dev/what-11-postgresql-scaling-stories-taught-me</guid><category><![CDATA[PostgreSQL]]></category><category><![CDATA[database]]></category><category><![CDATA[scalability]]></category><category><![CDATA[backend]]></category><dc:creator><![CDATA[How They Scaled]]></dc:creator><pubDate>Sun, 18 Oct 2026 03:42:33 GMT</pubDate><content:encoded><![CDATA[<p>I've been collecting first-hand "how we scaled" posts from engineering blogs, and PostgreSQL shows up more than any other database: 11 stories from 7 companies so far. Read them back to back and a pattern jumps out. Almost nobody starts by sharding. They squeeze a lot out of one database first, then split by domain, and only shard when they really have to.</p>
<p>Here's what each story adds, in roughly the order you'd hit these problems.</p>
<h2>1. One primary can go a very long way</h2>
<p>The most surprising entry is also the newest. In <a href="https://openai.com/index/scaling-postgresql/">Scaling PostgreSQL to power 800 million ChatGPT users</a> (January 2026), OpenAI describes running ChatGPT on a <strong>single-primary Postgres with nearly 50 global read replicas</strong>, serving millions of queries per second.</p>
<p>The trick isn't one clever feature. It's a stack of protections around that primary:</p>
<ul>
<li>reads are pushed to replicas,</li>
<li>write-heavy workloads that <em>can</em> be sharded move to Cosmos DB instead of landing on Postgres,</li>
<li>PgBouncer handles connection pooling,</li>
<li>cache leasing and multi-layer rate limits stop traffic spikes from reaching the database.</li>
</ul>
<p>My takeaway: before you design a sharding scheme, ask which workloads actually need to live on the primary. Moving the shardable, write-heavy ones somewhere else can buy you years.</p>
<h2>2. Connections are a resource, so pool them</h2>
<p>Connection count is a quiet limit that bites early. Two stories lean on PgBouncer (OpenAI above, and Notion's re-shard below), and Supabase went further and built its own pooler.</p>
<p>In <a href="https://supabase.com/blog/supavisor-1-million">Supavisor: Scaling Postgres to 1 Million Connections</a> (2023), Supabase load-tested Supavisor, a Postgres pooler written in Elixir, with <strong>one million client connections</strong> at 20,000 QPS. Keep in mind this is a load test, not a production incident report, but it shows how far you can push the "many clients, few real connections" model.</p>
<h2>3. Watch for small features with big side effects</h2>
<p><a href="https://about.gitlab.com/blog/2021/09/29/why-we-spent-the-last-month-eliminating-postgresql-subtransactions/">Why we spent the last month eliminating PostgreSQL subtransactions</a> is my favourite kind of post: a whole month spent on something that looks tiny. GitLab traced <strong>stalls on its read replicas</strong> to Postgres subtransaction overflow, and the fix was removing subtransactions from its code.</p>
<p>The lesson I take from it: if your replicas lag or stall under load and nothing obvious changed, look at what your ORM or app code is doing per transaction, not just at hardware.</p>
<h2>4. Split by domain before you shard</h2>
<p>Two teams solved "one database is too busy" without sharding at all:</p>
<ul>
<li><strong>Figma</strong>, in <a href="https://www.figma.com/blog/how-figma-scaled-to-multiple-databases/">The growing pains of database architecture</a> (2023), moved from one Postgres instance to several databases by <strong>vertically partitioning groups of tables</strong>, alongside read replicas and connection pooling.</li>
<li><strong>GitLab</strong>, in <a href="https://about.gitlab.com/blog/2022/06/02/splitting-database-into-main-and-ci/">We are splitting our database into Main and CI</a> (2022), moved its CI tables into <strong>their own Postgres database</strong> to take load off the main one.</li>
</ul>
<p>Vertical partitioning works when your tables naturally fall into groups that rarely join each other. It's usually less risky than sharding because each group still lives in one ordinary database.</p>
<h2>5. When you do shard, pick the key your product already has</h2>
<p>Eventually Figma and Notion both had to shard horizontally.</p>
<ul>
<li>In <a href="https://www.notion.com/blog/sharding-postgres-at-notion">Herding elephants</a> (2021), Notion split its Postgres monolith into a fleet of <strong>logical shards keyed by workspace</strong>, migrating with <strong>double writes and backfills</strong>.</li>
<li>In <a href="https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/">How Figma's Databases Team Lived to Tell the Scale</a> (2024), Figma describes a <strong>nine-month</strong> effort built on a custom query proxy, with <strong>logical sharding before physical splits</strong>.</li>
</ul>
<p>That "logical first" idea is worth stealing. If your app already routes queries by shard key while the data still sits in one place, the physical move becomes a separate, smaller project.</p>
<h2>6. Plan for sharding twice</h2>
<p>Sharding isn't one and done. In <a href="https://www.notion.com/blog/the-great-re-shard">The Great Re-shard</a> (2023), Notion <strong>tripled its Postgres fleet</strong> by re-sharding with <strong>logical replication</strong> and careful PgBouncer cutovers, <strong>with no downtime</strong>.</p>
<p>If you shard, design for moving data between shards from day one, because you'll need to.</p>
<h2>7. The replication log is an API</h2>
<p>Two stories treat Postgres replication as more than a backup:</p>
<ul>
<li>Figma's <a href="https://www.figma.com/blog/livegraph-real-time-data-fetching-at-figma/">LiveGraph</a> (2021) <strong>tails the Postgres replication log</strong> so the UI can subscribe to live query results.</li>
<li>Notion's re-shard (above) used <strong>logical replication</strong> to copy data into new shards.</li>
</ul>
<p>Once you see the log as a stream you can build on, real-time features and migrations get a lot easier.</p>
<h2>8. Shape the data for how it's queried</h2>
<p><a href="https://www.tigerdata.com/blog/how-we-scaled-postgresql-to-350-tb-with-10b-new-records-day">How We Scaled PostgreSQL to 350 TB+</a> (2023) is Timescale using its own Postgres extension for a database-observability product. The dataset is <strong>350 TB+ with 10 billion new records a day</strong>, kept queryable with hypertables (time-based partitioning), columnar compression and tiered storage.</p>
<p>For append-heavy time-series data, partitioning by time plus compressing and tiering older data is often a better move than adding more servers.</p>
<h2>9. Sometimes the scale is the number of databases</h2>
<p>The last story flips the usual problem. In <a href="https://www.atlassian.com/blog/atlassian-engineering/migrating-jira-database-platform-to-aws-aurora">Migrating the Jira Database Platform to AWS Aurora</a> (2025), Atlassian moved <strong>about four million Jira databases</strong> from RDS Postgres to Aurora, weighing reliability, downtime and cost along the way.</p>
<p>When you run a database per tenant, the hard part is fleet automation, not query tuning.</p>
<h2>The pattern, in one list</h2>
<p>If I had to turn these 11 stories into a checklist:</p>
<ol>
<li>Offload reads to replicas and move shardable write-heavy work off the primary.</li>
<li>Pool connections early.</li>
<li>Profile what your app does inside transactions.</li>
<li>Split table groups into separate databases before sharding.</li>
<li>Shard by the tenant key your product already has, logically first.</li>
<li>Build for re-sharding.</li>
<li>Use logical replication for migrations and real-time features.</li>
<li>Partition, compress and tier time-series data.</li>
</ol>
<p>None of this is new on its own. What's useful is seeing which steps real teams took, and in what order.</p>
<p>All 11 stories, with stack, techniques and the scale numbers each post states, are on one page: <a href="https://howtheyscaled.com/tech/postgresql/">How teams scaled PostgreSQL</a>. If you know a good first-hand Postgres scaling post I've missed, I'd love to hear about it in the comments.</p>
]]></content:encoded></item></channel></rss>