Funnel Chart Builder

A funnel chart answers one plain question: of the people who started, how many were still there at each later stage? Twelve thousand arrive, fewer look at something, fewer still begin a form, and a small remainder finish. The chart exists to show where the crowd thins.

That is a modest job, and it is easy to do badly in two ways at once. The picture can mislead, because the familiar tapering shape is usually drawn for looks and not from the numbers. And the rates printed beside it can mislead, because a single percentage hides how much or how little evidence stands behind it.

Both faults have the same cure. Decide what the drawing encodes, state it, and make the geometry obey. Then give every rate with its base and a range. The rest of this essay works through each half with one set of numbers.

The shape that gave the chart its name

Take five stages with made-up round counts: 12,000 arrived, 5,400 looked, 1,800 started, 900 finished and 360 returned. The usual tapered drawing gives each stage a band of the same height and makes the band as wide at its top as its count, narrowing to the width of the next stage. It looks like a funnel. It also quietly changes the data.

The trouble is that a reader sees a band, and a band has area. A trapezoid’s area is its height times the average of its two widths, so an equal-height band is sized by the average of its own count and the next one. Work that through for the five stages and the bands hold 59.4, 24.6, 9.2, 4.3 and 2.5 percent of the drawn area. The counts themselves are 58.7, 26.4, 8.8, 4.4 and 1.8 percent of their total. The second stage has been shrunk and the last one inflated by more than a third, and nothing on the chart says so.

Many funnel pictures are worse than that, because their shape is fixed in advance and the numbers are only printed on top. Such a picture is a list with decoration. If the labels were swapped, the drawing would not change.

A tapered funnel can be drawn truthfully. Keep each band’s top edge as wide as its count, and then choose each band’s height so that its area comes out proportional to the count as well. For the example the heights come to 1.38, 1.50, 1.33, 1.43 and 1.00 units. Width at every boundary is right and area is right. The price is that the bands are no longer equally tall, so height by itself means nothing, and the chart has to say that in words.

Even so, the plain alternative is better. William Cleveland and Robert McGill argued in 1984 that the judgments a reader makes from a graph can be ordered by how accurately people make them. They tested that ordering by experiment and concluded that graphs should use the judgments highest in the order. They proposed radical surgery on several popular chart forms. Bars that start from one shared edge ask for the simplest comparison there is. Their length is proportional to the count, and nothing else about them varies. That is why bars are the default here and the taper is offered beside them, not instead of them.

One more scale deserves a warning. When the first stage is a thousand times the last, the final bars shrink to slivers, and a logarithmic axis brings them back into view. On that axis position still tells the count but length no longer does. A solid bar would invite exactly the comparison the scale forbids, so a dot on a labelled axis is the honest mark, and a count of zero has no place on the axis at all.

A rate is a count over a count

Every percentage on a funnel is one count divided by another. The step rate divides a stage by the stage before; the overall rate divides it by the first. In the example, 5,400 of 12,000 is 45.0 percent and 360 of 12,000 is 3.0 percent. A percentage printed without its base cannot be checked and cannot be weighed, so the base belongs beside it every time.

The base matters because the same percentage can rest on very different evidence. Forty percent might be 400 of 1,000 or 4 of 10. The usual textbook range, the rate plus or minus a multiple of its estimated standard error, handles the small case poorly. The NIST handbook points out that it can return a lower limit below zero, an impossible value for a proportion. At the extremes it fails in the other direction: when none of twenty convert, it reports a range of no width at all, falsely implying certainty.

Edwin Wilson published a better interval in 1927. It comes from turning the standard test around: find every true rate that the observed count would not contradict. Its limits stay between 0 and 1, and it is lopsided near the ends, as it should be. Two later reviews are the reason it is so widely recommended now. Brown, Cai and DasGupta called the behavior of the textbook interval erratic and its usual safety rules misleading, and recommended Wilson’s interval for small samples. Agresti and Coull found its coverage close to the stated level even for very small samples.

Now the worked example can be read properly. For 5,400 of 12,000 the 95 percent Wilson interval runs from 44.1 to 45.9 percent: a tight range, because the base is large. For the last step, 360 of 900, it is 36.8 to 43.2. Shrink the same kind of step to 4 of 10 and the interval becomes 16.8 to 68.7, which says almost nothing beyond “somewhere in the middle”. For none of twenty it runs from 0 to 16.1 percent, not from 0 to 0. NIST’s own software reference gives 8 of 30 as 14.2 to 44.4 percent, and a correct implementation should reproduce that.

No such interval is exact. Counts come in whole numbers, so the true chance that a nominal 95 percent interval covers the real rate wobbles with the base and the rate. Summing the binomial distribution exactly over thirty-six cases, with bases from 10 to 1,000 and true rates from 1 to 50 percent, puts Wilson’s coverage between 90.4 and 98.4 percent, a little under 95 on average. That is a good record and not a guarantee. The interval also assumes each person is counted once and acts independently, which real traffic only approximates.

Two funnels and many chances to be fooled

Sooner or later two funnels are laid side by side: this month and last, one page design and another. The honest question is not which one won. It is how large the difference in each step rate could plausibly be, in either direction.

For that, the tool reports an interval for the difference. It uses the adjustment of Agresti and Caffo, which adds one success and one failure to each group before applying the familiar formula. Their paper reports that with ten observations per group the unadjusted interval fell below 93 percent coverage in most of the settings they tried, and the adjusted one almost never did. Newcombe reached a similar verdict about the simple methods. Alongside the interval sits the large-sample test from the NIST handbook, which pools the two groups to estimate a common rate; the handbook itself points to an exact test when samples are small.

In the example, the last step is 360 of 900 for one funnel and 330 of 960 for the other: 40.0 against 34.4 percent, a difference of 5.6 points with an interval from 1.2 to 10.0. That interval leaves out zero. It also leaves a wide choice of sizes, from barely anything to ten points, and that range is the finding.

A funnel with five stages produces four such comparisons, and each one is a fresh chance for noise to look like signal. Bonferroni’s inequality gives the ceiling: with four intervals at 95 percent, the chance that at least one misleads can be as high as four times 5, or 20 percent. Building each interval at 98.75 percent restores the overall level. Either way, one step that stands out among several deserves less trust than the same result would on its own.

What the shape cannot say

A funnel is a row of counts, and a row of counts leaves out time. It does not say how long anyone took, or whether those who left were about to return. Reporting systems that build funnels make the missing choices explicit. One widely used analytics reporting interface lets a funnel be open, so that people may enter at any step, or closed, so that they must enter at the first. It lets a step be required to follow the last one directly, or allow other events between. It also lets a step be limited to a set duration after the one before. Each choice changes the counts, and none of them shows in the finished picture.

It also assumes the stages are nested. If the first stage counts visits and the second counts people, or if a later stage can be reached by a side door, a stage can come out larger than the one before it. That is not a rate above 100 percent. It is a sign that the two numbers are not parts of one whole. The honest response is to say why, show no rate across the gap, and break the drawing there so it does not pretend to be a step.

Presentation has limits of its own. Shading stages in different colors is harmless only if nothing depends on it, since color must never be the only way a chart conveys information; labels on the marks and a real table underneath carry the same facts for everyone. When that table is exported as comma-separated text, a stage name that begins with an equals sign can be run as a formula by a spreadsheet, so it should be written in a way that keeps it as text.

What remains after all these cautions is still useful. A funnel drawn to scale shows where most people stop being counted, and a rate with its base and its interval shows how firmly that is known. It will not say why they stopped, and it predicts nothing. Those are questions for the people behind the counts.