Search Engine Evaluation Metrics: NDCG@10, MRR, MAP, Precision & Recall

In search engine engineering, algorithmic modifications—such as tuning BM25 hyperparameters, integrating dense neural vector embeddings, or deploying machine-learned rankers (Learning to Rank)—cannot be evaluated based on intuition or anecdotal queries. Evaluating whether an algorithm elevates or degrades search quality requires rigorous, mathematically grounded information retrieval evaluation metrics. Search quality teams measure performance across both offline test collections (using Precision@K, Recall@K, MAP, MRR, and NDCG@K) and online real-time user engagement experiments (via Team Draft Interleaving and A/B testing).

1. Core Information Retrieval Evaluation Metrics Matrix

Evaluating the mathematical characteristics and ideal application contexts for the standard search evaluation toolkit:

2. Deconstructing NDCG@K: DCG, IDCG, and Graded Relevance

Normalized Discounted Cumulative Gain (NDCG) evaluates ranked results using graded relevance judgments (typically: $0 = ext{Irrelevant}$, $1 = ext{Marginal}$, $2 = ext{Relevant}$, $3 = ext{Highly Relevant}$, $4 = ext{Perfect Match}$):

1. Discounted Cumulative Gain (DCG@K): Sums the exponential gain of each document discounted by its position:

   ext{DCG}@K = sum_{i=1}^{K} rac{2^{rel_i} - 1}{log_2(i + 1)}

2. Ideal DCG (IDCG@K): Sorts the ground-truth relevance labels of all available documents in descending order and calculates the maximum possible DCG achievable by a hypothetical perfect search engine.
3. Normalization: $ ext{NDCG}@K = rac{ ext{DCG}@K}{ ext{IDCG}@K}$. This normalizes scores into a clean range between $0.0$ and $1.0$, allowing engineers to average performance across queries with differing numbers of relevant candidates.

3. Mean Reciprocal Rank (MRR) for Navigational Search

For navigational or single-answer search queries (e.g. searching "IRS tax bracket table 2026" or "GitHub settings"), the user seeks exactly one correct destination. Subsequent hits carry zero value:

• If the first relevant result appears at rank #1, Reciprocal Rank is $1/1 = 1.0$.
• If the first relevant result appears at rank #2, Reciprocal Rank drops to $1/2 = 0.5$.
• If pushed to rank #5, it falls to $1/5 = 0.2$. Averaging across hundreds of thousands of test queries yields the system's Mean Reciprocal Rank (MRR).

4. Online Evaluation: Team Draft Interleaving vs. A/B Testing

While offline metrics evaluate historical ground-truth datasets, online user preferences are validated via live production experiments:

• A/B Testing: 50% of users receive Model A; 50% receive Model B. Measuring click-through rates (CTR) and abandonment requires millions of queries to achieve statistical significance due to immense variance across query distributions.
• Team Draft Interleaving: Generates a single composite search result page by interleaving results from Algorithm A and Algorithm B like a sports team draft (tossing a coin at each slot to select between algorithms). Because every user evaluates both models simultaneously on the identical query, interleaving requires $100 imes$ fewer queries to detect subtle $1%$ relevance gains with high statistical power.

5. Production Implementation: NDCG@10 & MRR Calculation Engine

The following TypeScript blueprint demonstrates calculating NDCG@10 and Reciprocal Rank across candidate search rankings:

export class SearchMetricsEvaluator {
  // Compute Discounted Cumulative Gain up to rank K
  static computeDCG(relevanceScores: number[], k: number = 10): number {
    let dcg = 0;
    const limit = Math.min(relevanceScores.length, k);

    for (let i = 0; i < limit; i++) {
      const rel = relevanceScores[i];
      // Formula: (2^rel - 1) / log2(rank + 1)
      const gain = Math.pow(2, rel) - 1;
      const discount = Math.log2((i + 1) + 1); // rank starts at 1, so (i+1)+1
      dcg += gain / discount;
    }

    return dcg;
  }

  // Compute Normalized Discounted Cumulative Gain (NDCG@K)
  static computeNDCG(actualRelevance: number[], idealRelevance: number[], k: number = 10): number {
    const dcg = this.computeDCG(actualRelevance, k);
    // Sort ideal relevance descending to compute IDCG
    const sortedIdeal = [...idealRelevance].sort((a, b) => b - a);
    const idcg = this.computeDCG(sortedIdeal, k);

    if (idcg === 0) return 0;
    return dcg / idcg;
  }

  // Compute Reciprocal Rank for navigational queries (binary relevance)
  static computeReciprocalRank(binaryRelevance: boolean[]): number {
    for (let i = 0; i < binaryRelevance.length; i++) {
      if (binaryRelevance[i]) {
        return 1 / (i + 1); // 1-based rank position
      }
    }
    return 0; // No relevant document found
  }
}