Your eval's confidence interval assumes independent examples. Yours are clustered.

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren't.

I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong.

Why clustering shrinks your real sample size

Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn't, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half.

The survey-statistics name for this is the design effect (Kish, "Survey Sampling," 1965). For clustered data it is approximately:

Deff = 1 + (m̄ - 1) · ICC

Enter fullscreen mode Exit fullscreen mode

where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is:

n_eff = n / Deff

Enter fullscreen mode Exit fullscreen mode

That is the number of independent examples your clustered set is actually worth.

The number

Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for "questions from the same document" (I have measured higher).

Deff  = 1 + (5 - 1) · 0.3 = 2.2
n_eff = 200 / 2.2 ≈ 91

Enter fullscreen mode Exit fullscreen mode

Your 200-example eval is worth about 91 independent examples. The correct confidence interval is √2.2 ≈ 1.48 times wider than the naive one. So the interval you proudly reported as plus or minus 3.5 points is really plus or minus 5.2. The 4-point improvement you shipped last sprint, the one that "cleared the CI," may not clear the corrected interval at all.

And note the direction of the error. Clustering never makes your interval too wide. It always makes it too narrow. So the bias is always toward false confidence, toward shipping a change that didn't actually beat baseline.

The three clusterings I check for

1. Source clustering. Multiple examples generated from or grounded in the same document, table, or context. This is the big one for RAG evals, where synthetic questions are minted per-document. ICC here is often 0.2 to 0.4.

2. Conversation clustering. Multiple turns scored from the same multi-turn session. Turns within a session share the same user, same goal, same accumulated context, so they correlate hard. Scoring 10 turns from 20 conversations is not 200 independent points.

3. Template clustering. Paraphrases or perturbations of the same underlying case. If you augmented 50 seed cases into 150 by rewording, your n is closer to 50 for the purpose of the interval. The rewordings measure robustness to phrasing, not 150 independent capabilities.

The fix: resample clusters, not rows

The cleanest correction that does not require you to estimate ICC by hand is a cluster bootstrap. Instead of resampling individual examples (which assumes independence, reintroducing the exact bug), you resample whole clusters with replacement.

import numpy as np

def cluster_bootstrap_ci(scores, cluster_ids, B=5000, alpha=0.05, seed=0):
    rng = np.random.default_rng(seed)
    clusters = {}
    for s, c in zip(scores, cluster_ids):
        clusters.setdefault(c, []).append(s)
    keys = list(clusters)
    means = np.empty(B)
    for b in range(B):
        drawn = rng.choice(keys, size=len(keys), replace=True)   # resample CLUSTERS
        pooled = [s for k in drawn for s in clusters[k]]
        means[b] = np.mean(pooled)
    lo, hi = np.percentile(means, [100*alpha/2, 100*(1-alpha/2)])
    return float(lo), float(hi)

Enter fullscreen mode Exit fullscreen mode

Resampling at the cluster level automatically bakes in the correlation structure. You do not have to estimate the ICC. The interval it returns is the honest one, and it will be wider than the binomial interval your CI currently prints. The wider interval is not an artifact of the bootstrap. The correlation in your data was always there, and this is just the first interval that stops ignoring it.

FAQ

What if my examples really are independent? Then the cluster bootstrap and the binomial interval will roughly agree, and you have lost nothing by checking. The check costs a few seconds of compute. Shipping a change that never actually beat baseline costs a lot more than that.

Do I need to know the ICC? No. The cluster bootstrap sidesteps it. You only need the design-effect formula if you want a back-of-envelope sense of how bad the problem is before you code anything.

Does this change my point estimate? No. The mean pass rate is unchanged. Only the interval around it widens. This is purely about how much you should trust the number, not the number itself.

Is this the same as just using a bigger eval set? No, and this is the trap. Adding 50 more questions from the same 8 documents barely helps, because you are adding within-cluster examples that carry little new information. To tighten a clustered interval you need more clusters (more documents, more conversations), not more examples per cluster.

Open question

I do not have a good rule of thumb for the minimum number of clusters before a cluster bootstrap itself becomes unstable. With very few clusters (say under 15) the bootstrap distribution gets lumpy and the interval is itself uncertain. I have been using a soft floor of 20 clusters and falling back to reporting the design-effect-adjusted interval below that, but I have not seen a principled threshold for eval-sized data. If you know the literature here better than I do, point me at it.