Method · Statistics

Confidence interval: what it is, the formula, and how we use it to measure AI visibility

By Kristian Stig Henriksen · Dear Future

A confidence interval is the range within which an unknown number lies with a stated level of certainty, typically 95%, when you have only measured a sample. It does not give you one number; it tells you how precisely you know the number. When we measure how often AI assistants mention a brand, the interval is what separates a real change from random noise.

What is a confidence interval?

When you measure a sample you get an estimate: 12 of 100 answers mention the brand, so 12%. But a new sample of 100 answers would give a slightly different number. The confidence interval puts bounds on how far away the true number can reasonably be. It has two parts: an estimate and an uncertainty on either side of it. The more measurements, the narrower the interval.

The level, 95% for example, describes the method, not the single interval: if you repeated the measurement many times and computed an interval each time, about 95 of 100 intervals would contain the true number. In practice it reads like this: the number lies somewhere between 5.6% and 18.4%, and we would be surprised if it fell outside.

What does a 95% confidence interval mean?

95% is a convention, not a law of nature. A 90% interval is narrower and wrong more often; a 99% interval is wider and wrong less often. The width depends on three things: how sure you want to be, how much the data spread, and how many measurements you have. Only the last one is yours to change, and it costs time.

The formula for a confidence interval

For a proportion, meaning the share of answers that mention the brand, the classic formula is:

p̂ ± z · √( p̂ · (1 − p̂) / n )

where p̂ is the proportion in the sample, n is the number of measurements, and z is 1.96 at 95% (1.645 at 90% and 2.576 at 99%). Worked example: 12 mentions in 100 answers gives p̂ = 0.12. The square root of 0.12 · 0.88 / 100 is 0.0325. Multiplied by 1.96 that is 0.064, so plus or minus 6.4 percentage points. The interval is 5.6% to 18.4%.

For a mean, an average position for instance, the fraction under the square root is replaced by the sample's standard deviation divided by the square root of n. For a small sample, below about 30, the t-distribution is used instead of z.

Why we use the Wilson interval

The classic formula breaks when the proportion is small or the count is low, and that is exactly the situation in AI visibility, where a brand is often mentioned in under 5% of answers. Two examples from our own measurement of aivisibility.dk show the problem.

3 citations in 240 answers gives p̂ = 1.25%. The classic formula gives plus or minus 1.4 percentage points, an interval from minus 0.2% to 2.7%. A negative proportion does not exist. The Wilson interval, which adjusts the centre and the width for small samples, gives 0.4% to 3.6%. That is both possible and more honest.

0 citations in 240 answers gives, with the classic formula, the interval 0% to 0%: complete certainty that the brand is never mentioned. The Wilson interval gives 0% to 1.6%. Zero is not zero; zero is below about 1.6%. That is why the dashboard shows an interval even when the count is zero.

The Wilson interval was described in 1927 and is recommended in the statistical literature over the classic formula precisely for small proportions. The formula is a little longer, but it fits in one spreadsheet, and we write it out below.

How we use it when we measure AI visibility

We ask the same 120 questions of two assistants every day. That gives 240 answers a day, and every answer is one measurement: mentioned or not, cited or not. Share of voice is a proportion, and it gets a Wilson interval on every window.

A single answer carries almost no information. A study of 12,933 AI answers found a within-brand correlation of 0.0146, which in practice means the same question asked again gives an almost independent answer. So we never read one day; we read 7-day windows and compare them.

We only call a change real when three things hold: the intervals for before and after do not overlap, the direction is the same across the days in the window, and the questions we did not work on, the control group, have not moved the same way. Otherwise it is noise, however good the number looks.

That also sets a timeframe. For the interval around a proportion near 2% to be plus or minus 0.5 percentage points, the formula needs about 3,000 answers, so 12 to 13 days at 240 answers a day. That is why we read a change after two weeks and not after two days.

Statistical significance, p-value and significance level

Significance and confidence intervals are two sides of the same thing. A result is statistically significant at the 5% level when it is so unlikely under the assumption of no difference that it would occur in fewer than 5% of cases. The p-value is exactly that probability. And a 95% interval that does not contain the no-difference value corresponds to a p-value below 0.05.

Significant does not mean important. A change from 1.0% to 1.1% can be significant with enough data and still mean nothing for the business. Conversely, a large change can be real without being significant yet, because it has not been measured for long enough. The interval shows both at once: how big the change is, and how sure you can be.

Confidence intervals in Excel and Google Sheets

For a mean, the function CONFIDENCE.NORM(alpha, standard deviation, n) returns the half-width of the interval, where alpha is 0.05 at 95%. The interval is then the mean minus and plus that number. Google Sheets uses the same function name.

For a proportion there is no built-in Wilson function, but it can be written with ordinary cells. With the proportion in A1, the count in A2 and z = 1.96 in A3:

Centre: (A1 + A3^2/(2*A2)) / (1 + A3^2/A2)

Half-width: A3 / (1 + A3^2/A2) * SQRT(A1*(1-A1)/A2 + A3^2/(4*A2^2))

The interval is the centre minus and plus the half-width.

Read on: the method behind the number, share of voice in AI and how to measure AI visibility.

← Back to the front page