Long-Tail Query Explorer

Ask anyone who runs a website which searches bring people in, and they will name a handful of obvious phrases. Then open the export of real queries and scroll. The handful is there at the top, but the list keeps going for hundreds or thousands of lines, most of them typed once or twice. That second part is the long tail, and it is usually where the surprises live.

The idea did not start with search. In statistics, the long tail of a distribution is the part with many occurrences far from the head, the region of frequent, popular values. Chris Anderson popularized the phrase in a magazine article in October 2004 and later in a book subtitled “Why the Future of Business Is Selling Less of More.” Search queries behave the same way, and for the same reason: there is no shelf limit on what people type.

Why the rare queries add up

A familiar shorthand for lopsided data is the 80-20 rule, which says that a fifth of the causes produce four fifths of the effect. It is not a law of nature. In the Pareto distribution it holds only for one particular shape value, log base 4 of 5, which is about 1.16. Change the shape and the split changes. Zipf’s law, which describes word frequencies in text, makes a related claim: the item of rank n is often roughly inversely proportional to n in frequency. Both describe a steep head followed by a slow, stubborn tail.

The arithmetic of a slow tail is the whole point. If the top query brings 940 clicks and the hundredth brings 3, the hundredth looks worthless. But there are many queries near the hundredth and very few near the first. Add up a few hundred small numbers and the total can rival the head. Researchers studying online retail made the same observation from the other side: Erik Brynjolfsson, Yu Hu and Duncan Simester, in a 2006 working paper titled “Goodbye Pareto Principle, Hello Long Tail,” found that lowering search costs could raise the collective share of hard-to-find products.

So the question is never whether a tail exists. The question is how heavy it is for a particular site and a particular topic, and that is an empirical question. A rule of thumb cannot answer it. Neither can a list of plausible phrases written by someone, or something, that has never seen the site’s data.

Reading your own data instead of guessing

A query export is the honest place to start. If your site tools let you download the searches that showed or clicked through to your pages, the file usually holds a query column and a few numbers, such as clicks, views and an average position. It is a record of what really happened. It cannot tell you about phrases nobody typed, but every line in it is a real search.

Reading that file correctly is less trivial than it sounds. The common format, described in RFC 4180, wraps a field in double quotes when it holds a comma, a quote or a line break, and writes a quote inside such a field as two quotes. A query like “kites, paper or plastic” is one field, not two. Some exports use tabs or semicolons instead of commas, a header row is optional, and numbers may carry grouping commas or a percent sign. A careful reader states how it handles each case and lists every row it refuses, rather than quietly dropping lines.

Then come the near misses of text itself. Unicode allows the same visible letter to be stored in more than one way, and Unicode equivalence defines normal forms that make such strings compare equal. Form KC goes further and folds compatibility characters, such as full-width letters, into their plain forms. After that, lower case, trimming stray punctuation and collapsing spaces make “Paper Kite For Kids” and “paper kite, for kids” the same query. Splitting the result into words is best left to the word segmenter built into browsers, which marks which pieces of a string are words and which are spaces or punctuation.

Patterns, families and the worked example

Once queries are clean, a seed such as “paper kite” picks out every query that extends it. Two readings are useful. In order, the seed’s words must appear side by side, as in “easy paper kite”. In any order, they only need to appear somewhere, which also catches “kite made of paper”. Neither reading is right in general; they answer different questions, and the choice should be visible.

The extra words around the seed fall into recognizable families. A question word at the start marks a how or what search. Words such as “vs” or “review” mark comparison. A preposition right after the seed, as in “paper kite for kids”, narrows the audience. Modifiers such as “easy” or “free” shape the request, intent words such as “buy” or “price” signal a purchase, and a four-digit year signals freshness. These are word patterns, close cousins of the n-grams of text analysis, which are simply runs of adjacent words.

Here is a worked example you can reproduce, because it is the made-up export the tool opens with. It has 141 readable rows; two more are rejected on purpose, one with an empty query and one whose click cell says “n/a”. Normalising merges two spellings into “paper kite for kids”, so 139 distinct queries remain, and 130 of them contain “paper kite” in order. They hold 3,557 clicks between them.

Rank those 130 by clicks and ask for the smallest head that holds half the total. It takes only three queries: “paper kite” with 940 clicks, “how to make a paper kite” with 610, and “paper kite for kids” with 361 once its three spellings are merged. That is 1,911 clicks, so the head is 3 of 130 queries. The other 127 queries hold 1,646 clicks, or 46.3% of the total. Switch the rule to call every query of four words or more the tail, and 112 queries hold 49.2%.

The families tell a parallel story. Fifteen of the matched queries open with a question word and bring 893 clicks. Thirty-eight have a preposition right after the seed. Sixty-four belong to no family at all, which is a useful warning: lists of patterns describe part of what people type, never all of it.

Near duplicates, and what an export cannot tell you

Long tails are full of variants. Some are the same words in another order, some differ by a typing slip, and some differ by a single extra word. Treating each as a separate opportunity inflates the count of ideas; merging everything that looks alike hides real differences. The only defensible approach is a rule you can read and change.

Three rules cover most cases. Two queries with exactly the same set of words are linked. Two queries whose word sets overlap heavily are linked, measured by the Jaccard index, the size of the intersection divided by the size of the union. And two queries that differ by a small number of character edits are linked. The Levenshtein distance counts insertions, deletions and substitutions; the Damerau-Levenshtein family adds the swap of two adjacent letters. The restricted form, called optimal string alignment, edits no substring more than once, so it is not a true metric.

Links then need to become groups. If A is linked to B and B to C, all three belong together, even when A and C look nothing alike. That is the definition of a connected component in a graph, and a disjoint-set forest finds components in near-constant time per operation. It also means the groups do not depend on the order of the rows, which a greedy method that compares each query only with the first member of an existing group cannot promise.

Thresholds matter more than any algorithm. In the worked example, the defaults of 1 edit and a 90% overlap join “paper kite for kid” and “for kids paper kite” to “paper kite for kids”, and fix two typing slips in “instructions” and “tutorial”. Allow two edits and “buy paper kite” joins “diy paper kite”, while “big” and “bird” kites merge in seven pairs. Lower the overlap to 85% and “how to make a bird paper kite” joins “how to make a paper kite”, because six of their seven distinct words match.

Notice what was not joined: “how to make paper kite” and “how to make a paper kite”. The missing “a” is a stop word, and the stop word entry is frank that no list of such words suits every purpose. One extra edit would join them; so would ignoring small words. Both choices have costs, and that is exactly why the rule should be stated rather than buried.

A query export also has blind spots worth saying aloud. It counts searches that reached you, so a phrase that is absent may still be searched by people who went elsewhere. Clicks and views belong to your pages and your position at the time, not to the phrase in general. And when you save results to a spreadsheet, any cell beginning with an equals or plus sign can be read as a formula, so careful exports put a harmless apostrophe in front.

None of this needs a guess. The tail of search is not a forecast to be generated but a record to be counted, one line at a time, with the rules on the table.