Cloud Spanner splits, the automatic sharding mechanism, actually work by dynamically partitioning your data based on estimated row count and data distribution, not by explicit user-defined sharding keys.
Let’s watch Spanner rebalance a table. Imagine we have a Users table with UserId as the primary key, and we’ve inserted a million rows, all with UserIds starting from 1.
CREATE TABLE Users (
UserId INT64 NOT NULL,
Name STRING(100),
Email STRING(100)
) PRIMARY KEY (UserId);
Initially, Spanner might create a single split for this table.
Split 1: [ , )
Contains: Users table (approx. 1M rows)
As we insert more data, and especially if the distribution becomes skewed (e.g., all new users get sequential IDs), Spanner’s internal monitoring detects a growing split. It aims for splits to be roughly 2GB in size or contain around 10,000 rows, whichever comes first. When a split exceeds these thresholds, Spanner prepares to split it.
Here’s the magic: Spanner doesn’t stop writes. It performs a split online. It identifies a logical midpoint within the existing split. For our sequential UserId example, it might pick a UserId value like 500000.
Spanner then creates two new, empty splits and tells the orchestrator to start directing new writes for UserId > 500000 to the new split, while reads and writes for UserId <= 500000 continue to go to the original split.
Split 1: [ , 500000)
Contains: Users table (approx. 500k rows)
Split 2: [500000, )
Contains: Users table (approx. 500k rows)
Over time, as data continues to flow, the original split might become "stale" as all its data is logically copied to the new splits. Spanner then garbage collects the old split. This process happens continuously and transparently. The key point is that Spanner is constantly evaluating split sizes and rebalancing to maintain performance.
The actual data isn’t "copied" in a traditional sense during the split. Instead, Spanner’s distributed transaction manager (Paxos) coordinates the creation of new split boundaries. Existing data ranges are logically re-assigned to these new boundaries. Reads and writes are transparently routed to the correct split based on the current metadata.
The primary levers you control are your schema design and your primary key choice. A well-distributed primary key (like a UUID or a hash of sequential IDs) prevents hot splits. If you have a monotonically increasing primary key, Spanner will aggressively split it, but the writes will still be concentrated on the latest split, potentially leading to hot spots.
You can observe splits using gcloud spanner databases ddl describe <database-name> --instance=<instance-name>. This will show you the current split boundaries for your tables.
The underlying mechanism for split creation and management is a distributed consensus protocol, typically Paxos, ensuring consistency across replicas. When a split occurs, the metadata about the new split boundaries and which splits are responsible for which key ranges is updated atomically.
The surprising truth is that Spanner’s automatic sharding is not about predicting future data distribution but about reacting to observed data distribution and size, constantly optimizing for balance. It’s a reactive system, not a predictive one. This allows it to handle massive, unpredictable workloads without manual intervention.
The next concept you’ll run into is understanding how these splits impact latency, especially during rebalancing operations or when a single split becomes a hot spot.