Distributed Search Clusters: Sharding Strategies, Replication & Raft Consensus

Scaling search to billions of documents requires distributing inverted indexes across hundreds of cluster nodes while maintaining high availability and consensus.

1. Sharding & Document Partitioning

Large indexes are split into primary shards using consistent hashing. Routing keys allow co-locating multi-tenant customer data onto dedicated shards to eliminate cross-cluster network latency.

2. Scatter-Gather Query Execution

When a search query arrives at a coordinating node, it is broadcast to one replica of every primary shard in parallel. Each shard computes localized top-K scores; the coordinator collects and merges intermediate lists into the final global ranking.

3. Cluster Consensus via Raft

Cluster metadata (index mappings, shard allocation, routing tables) is managed via the Raft distributed consensus protocol, ensuring leader election and state machine replication survive unexpected node crashes without data loss.