How Big Tech Generates Unique IDs at Scale (Twitter Snowflake, Instagram)

UUIDs solve the coordination problem — any node can generate a value with negligible collision risk, no central authority required. But at the scale of Twitter or Instagram, coordination-free generation isn't the only requirement: IDs also need to be small (8 bytes, not 16), time-sortable, and cheap to route to the correct database shard. Neither company adopted UUIDs for their primary content IDs. Both built custom 64-bit schemes instead. This post breaks down how each one works, why the specific bit allocations were chosen, and how the two approaches compare to UUIDs and to each other.

---

The Problem Neither Auto-Increment Nor UUIDs Solved

A single Postgres or MySQL AUTO_INCREMENT column works perfectly — until you need more than one write node. Once data is sharded across multiple database servers, each with its own auto-increment sequence, two different rows on two different shards will eventually generate the same ID. You can't merge or reference across shards without a collision.

UUIDv4 fixes the collision problem but introduces new ones at this scale:

• 16 bytes vs. 8 bytes — doubles index and foreign-key storage across billions of rows.
• No embedded time-ordering (for v4) — every insert lands at a random position in a clustered index, which is exactly the fragmentation problem covered in our benchmark post.
• No routing information — a UUID alone doesn't tell you which shard holds the row, so you still need a separate lookup service or a directory table for basic reads.

Twitter and Instagram independently arrived at the same underlying idea: a 64-bit integer, generated without central coordination, with a timestamp in the high bits for sortability and structured lower bits for uniqueness. They differ mainly in what goes in those lower bits.

---

Twitter Snowflake

Snowflake, introduced in 2010, generates a 64-bit signed integer per ID with four packed fields:

 0                   41                46      51                 63
+---+---------------------------------------+-------+-------+------------+
| 0 |         41-bit timestamp (ms)          | DC(5) | WK(5) | Seq (12)   |
+---+---------------------------------------+-------+-------+------------+

Why this specific split

• 41 bits of timestamp gives roughly 69 years of range from the chosen epoch — a custom epoch (not 1970) is used specifically so those 41 bits aren't wasted on decades before the system existed.
• 10 bits total for machine identity (5+5) supports up to 1,024 concurrent ID-generating workers, split into datacenter and worker components mainly for operational clarity — you can tell at a glance where an ID came from.
• 12-bit sequence allows 4,096 IDs per millisecond, per worker, before the generator has to stall and wait for the next millisecond tick. Across 1,024 workers, that's a theoretical ceiling in the billions of IDs per second.

Reference implementation

public class SnowflakeGenerator
{
    private const long Epoch = 1288834974657L; // Nov 4, 2010 01:42:54.657 UTC
    private const int WorkerIdBits = 5;
    private const int DatacenterIdBits = 5;
    private const int SequenceBits = 12;

    private readonly long _workerId;
    private readonly long _datacenterId;
    private long _sequence = 0L;
    private long _lastTimestamp = -1L;
    private readonly object _lock = new();

    public SnowflakeGenerator(long datacenterId, long workerId)
    {
        _datacenterId = datacenterId;
        _workerId = workerId;
    }

    public long NextId()
    {
        lock (_lock)
        {
            long timestamp = CurrentTimeMillis();

            if (timestamp < _lastTimestamp)
                throw new InvalidOperationException("Clock moved backwards — refusing to generate ID.");

            if (timestamp == _lastTimestamp)
            {
                _sequence = (_sequence + 1) & 0xFFF; // mask to 12 bits
                if (_sequence == 0)
                    timestamp = WaitNextMillis(_lastTimestamp); // sequence exhausted, spin to next ms
            }
            else
            {
                _sequence = 0L;
            }

            _lastTimestamp = timestamp;

            return ((timestamp - Epoch) << (WorkerIdBits + DatacenterIdBits + SequenceBits))
                 | (_datacenterId << (WorkerIdBits + SequenceBits))
                 | (_workerId << SequenceBits)
                 | _sequence;
        }
    }

    private static long CurrentTimeMillis() =>
        DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();

    private long WaitNextMillis(long lastTimestamp)
    {
        long ts;
        do { ts = CurrentTimeMillis(); } while (ts <= lastTimestamp);
        return ts;
    }
}

-- Storing Snowflake IDs is just a BIGINT — no special type needed
CREATE TABLE Tweets (
    Id BIGINT PRIMARY KEY,       -- generated application-side, not IDENTITY
    AuthorId BIGINT NOT NULL,
    Body NVARCHAR(280) NOT NULL,
    -- CreatedAt can be derived from Id, but an explicit column is often kept for query ergonomics
    CreatedAt DATETIME2 NOT NULL
);

Extracting the embedded timestamp is a pure bit-shift, with no table lookup required:

const EPOCH = 1288834974657n;

function extractTimestamp(snowflakeId) {
  const id = BigInt(snowflakeId);
  const timestampPart = id >> 22n; // shift past 5+5+12 = 22 bits
  return new Date(Number(timestampPart + EPOCH));
}

---

Instagram's Sharded ID Scheme

Instagram (2011) faced a related but distinct problem: they weren't just generating unique IDs, they were sharding Postgres and needed each ID to double as a routing key, so a given ID alone would tell you which physical shard held the row.

Rather than run a dedicated ID-generation service (Snowflake was originally a standalone Thrift service coordinated via ZooKeeper), Instagram generated IDs inside Postgres itself, using PL/pgSQL and each table's native auto-increment sequence, on a per-shard basis. Their 64-bit layout:

 0                                    41            54                 63
+---------------------------------------+-------------+------------------+
|         41-bit timestamp (ms)          | Shard (13)  |  Sequence (10)   |
+---------------------------------------+-------------+------------------+

The key architectural difference: logical vs. physical shards

Instagram deliberately created many more logical shards (thousands) than physical database servers. Each logical shard is a Postgres schema — a namespace within a database — and each sharded table (posts, likes, comments) exists once per schema. Because the mapping of logical shard → physical server lives in application config rather than being baked into the ID itself, they could move a batch of logical shards to a new physical server as they scaled, without ever having to regenerate or re-bucket existing IDs. The shard ID embedded in every row stays valid forever; only the routing table that says "logical shard 1341 lives on server 7" needs to change.

PL/pgSQL ID generator (Instagram's approach)

CREATE OR REPLACE FUNCTION insta5.next_id(OUT result BIGINT) AS $$
DECLARE
    our_epoch BIGINT := 1314220021721; -- custom epoch, ms since Unix epoch
    seq_id BIGINT;
    now_millis BIGINT;
    shard_id INT := 5; -- this schema/shard's fixed ID, 0-8191
BEGIN
    SELECT nextval('insta5.table_id_seq') % 1024 INTO seq_id;

    SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000) INTO now_millis;
    result := (now_millis - our_epoch) << 23;
    result := result | (shard_id << 10);
    result := result | (seq_id);
END;
$$ LANGUAGE PLPGSQL;

-- Each sharded table's default draws directly from the shard-local generator
CREATE TABLE insta5.likes (
    id BIGINT PRIMARY KEY DEFAULT insta5.next_id(),
    user_id BIGINT NOT NULL,
    photo_id BIGINT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Routing reads directly from the ID

Because the shard ID is embedded in bits 10–22, any service holding an ID can compute which shard to query without a directory lookup:

public static int ExtractShardId(long instagramId)
{
    return (int)((instagramId >> 10) & 0x1FFF); // mask to 13 bits
}

---

Snowflake vs. Instagram: Design Comparison

The core difference is what problem the identity bits solve. Snowflake's worker/datacenter bits exist purely to prevent collisions between concurrent generators — they say nothing about where the resulting data lives. Instagram's shard bits are load-bearing for routing: they're deliberately the same value used to pick which Postgres schema the row was inserted into, so the ID itself replaces what would otherwise be a separate lookup service.

---

Comparison Against UUIDs

The 64-bit schemes win decisively on size and routing capability, at the cost of needing synchronized clocks and pre-assigned identity ranges — operational overhead a pure UUID scheme doesn't have. That trade-off is exactly why these designs only tend to show up at companies operating enough infrastructure to manage worker/shard ID assignment reliably; for a smaller system, UUIDv7 gets you most of the sortability benefit with none of the coordination burden.

---

When to Reach for Each Approach

• UUIDv4 — no shared state between services, no need for time-ordering or routing info, simplest to reason about.
• UUIDv7 — same coordination-free generation, but you also want time-ordered, database-friendly clustering.
• Snowflake-style — you control a fleet of ID-generating nodes, need 8-byte (not 16-byte) keys, and don't need the ID to encode where the data physically lives.
• Instagram-style sharded IDs — you're already sharding your database and want the ID to double as a routing key, avoiding a separate shard-lookup service entirely.

Both Twitter's and Instagram's schemes are, at their core, the same insight applied to two different constraints: pack a timestamp into the high bits for free sortability, and use the remaining bits to solve whatever coordination problem your specific architecture actually has — worker identity for Snowflake, physical data location for Instagram.