Query Processing & Lexical Analysis: Tokenization, Stemming & Spell Correction

A search engine's query understanding pipeline bridges ambiguous human input with structured database retrieval algorithms.

1. Lexical Tokenization & Unicode Normalization

The query processor normalizes input strings using Unicode NFKC normalization, strips punctuation, and handles compound word splitting (e.g., German compounding or alphanumeric identifiers like iPhone15Pro).

2. Morphological Stemming vs Lemmatization

Stemming algorithms (such as the Porter and Snowball stemmers) apply heuristic suffix-stripping rules to reduce inflected words to their root base. In contrast, dictionary-based lemmatizers maintain morphological accuracy at slightly higher computational overhead.

3. High-Speed Fuzzy Spell Correction

Real-time spell suggestion evaluates candidate words within Levenshtein edit distance ≤ 2 using symmetric delete (SymSpell) hashing, generating accurate typographical corrections in under 5 microseconds per keystroke.