Near-Duplicate Detection at Web Scale: SimHash, MinHash & Locality-Sensitive Hashing

Up to 30% of pages crawled on the public internet represent duplicate or near-duplicate copies of existing content. Filtering redundant pages preserves crawling bandwidth and prevents index bloat.

1. SimHash: 64-Bit Semantic Fingerprints

SimHash produces 64-bit integer fingerprints where similar documents have tiny Hamming distances. Documents with a Hamming distance ≤ 3 bits are categorized as near-duplicates, allowing rapid duplicate identification across billions of documents using table permutation lookup.

2. MinHash & Jaccard Set Similarity

MinHash estimates the Jaccard similarity between document n-gram token sets by calculating the probability that random hash functions map both documents to the same minimum value.

3. Locality-Sensitive Hashing (LSH)

LSH groups similar MinHash signatures into hash buckets, reducing near-duplicate candidate discovery from O(N^2) pairwise comparisons to O(N) sub-linear lookups.