Distributed Crawling, Inverted Indexing & BM25 Ranking Monograph
Technical deep-dives into high-throughput web crawling, bloom-filter URL deduplication, WAND top-k query evaluation, and zero-telemetry search.
Every query through NetSearch Proxy is executed server-side. We strip your IP address, browser User-Agent, cookies, and ad-tech tracking parameters (utm_*, fbclid, gclid) before fetching results.
Technical deep-dives into high-throughput web crawling, bloom-filter URL deduplication, WAND top-k query evaluation, and zero-telemetry search.
Explore the decentralized multi-vertical index powering NetSearch: zero-profile query routing, inverted index sharding, and crawl signal filtering.
How NetSearch Systems isolates client browsers from behavioral search telemetry through an ephemeral HTTP header scrubber, URL parameter sanitizer, and server-side Okapi BM25 federated retrieval relay.
Traditional search engines bind your search history to your IP address, browser fingerprint, and advertising ID. NetSearch breaks the surveillance chain in three architectural layers:
Before any query leaves our edge node, we strip X-Forwarded-For, User-Agent, Cookie, Accept-Language, and Client Hints (Sec-CH-UA). Upstream indexes only ever see our generic server identity.
Result URLs are parsed and sanitized in real time to strip 16+ surveillance parameters including utm_source, fbclid, gclid, msclkid, and _ga tokens before you click.
Queries are processed in volatile memory with Cache-Control: no-store and Referrer-Policy: no-referrer. We never write search terms or client IPs to disk.
| Privacy Capability | NetSearch Proxy (/anonymous-search) |
Big Tech Search Engines | Client-Side Browser Extensions |
|---|---|---|---|
| Hides Client IP from Search Index | ✅ Yes (Server-Side Relay) | ❌ No (Logged & Geo-Profiled) | ❌ No (Direct Client Connection) |
| Strips Browser Fingerprint & UA | ✅ Yes (Normalized Proxy UA) | ❌ No (Full Client Hints Captured) | ⚠️ Partial (Spoofed in JS) |
| Scrubs Outbound Ad Tracking Params | ✅ Yes (16+ UTM/Click IDs Removed) | ❌ No (Appends Redirect Trackers) | ⚠️ Varies by Rule List |
| Programmatic JSON API Available | ✅ GET /api/netsearch/search?q=... |
❌ Paid / Rate-Gated | ❌ Browser-Only |