Co-occurrence Term Finder

Put a word in a sentence and it picks up company. “Chain” sits beside “gears” and “oil” in a bicycle manual and beside “store” and “brand” in a business report. Linguists have long treated that company as evidence of meaning: the distributional idea, popularised by J. R. Firth in the 1950s, holds that a word is characterised by the company it keeps, and that words used in similar contexts tend to mean similar things.

That idea is sound, and it can be measured. What it cannot do is hand anyone a secret list of words that search engines want to see. This page is about the difference between those two things.

Counting company honestly

The simplest measurement is a count. Choose a seed word, choose a window, and count how often each other word shares that window with it. The window matters more than people expect. “Same sentence” asks whether two words were part of one thought. “Same document” asks whether they belong to one topic. “Within five words” asks whether they sit in one phrase. The same pair can look close under one window and distant under another.

A raw count is easy to read but hard to compare, because common words win everything. A word that appears in every sentence will share a sentence with your seed simply by being everywhere. The usual correction is pointwise mutual information, or PMI. It compares how often two words actually occur together with how often they would if they were independent: the logarithm of p(x, y) divided by p(x) times p(y). In base 2 the answer is in bits. Zero means the pair turns up exactly as often as chance predicts; positive means more often; negative means less.

The Wikipedia article on PMI gives a small table that makes this concrete. With p(x, y) of 0.15, p(x) of 0.2 and p(y) of 0.25, the score is 1.584963 bits. It also lists word pairs counted in the first fifty million words of an English Wikipedia dump. “Puerto” and “rico” score above 10 on a natural logarithm. “Of” and “and” score below minus 3, because those two everywhere-words share a window less often than their sheer frequency predicts. The article cites a 1990 paper by Kenneth Church and Patrick Hanks on word association norms and mutual information, and traces the concept itself to Robert Fano in 1961.

Here is a worked example you can reproduce on this page, since it is the example that loads first. Thirteen short texts about baking, gardening and cycling hold 32 sentences. “Chain” appears in 5 of them, “gears” in 2, and the two share 2 sentences. PMI is log2(2 x 32 / (5 x 2)), which is log2(6.4), or 2.678 bits.

Switch the window to “within k words” with k at 5 and the event changes: every pair of word positions at most five apart becomes one event, 3,230 of them in all. The score becomes log2(2 x 3230 / (45 x 20)), or 2.844. The counts behind every score are shown beside it, so nobody has to trust a number they cannot rebuild.

One detail is easy to get wrong. The joint count and the single counts must come from the same set of events. If you count pairs within five words but count each word’s frequency by documents, the ratio stops being a probability ratio at all, and the scores drift for no reason a reader could see. This tool computes both from one event space, and its test suite includes a deliberately broken version that mixes the two, to show that the checks catch it.

The trouble with rare words

PMI has a famous weakness, and anyone using it should meet it early. It flatters rare pairs. Two words that each appear once, in the same sentence, get the highest score their counts allow, because nothing contradicts their perfect association. The PMI article cites published work noting this tendency to give higher scores to low-frequency events, and describes variants designed to correct it.

You can watch it happen. Set the “least times together” control on this page to 1 and the seed “chain” gains 22 partners instead of 4. Words like “adjust”, “cleanly” and “drop”, each seen with “chain” exactly once, now tie with “gears” at 2.678. Nothing about bicycles has changed. One sentence is simply not enough evidence to tell a real association from an accident of phrasing. A minimum count is a blunt but honest defence; it hides weak evidence instead of pretending it is strong.

Positive PMI, or PPMI, takes a related shortcut: it sets every negative score to zero. The reasoning quoted in the PMI article is that negative values tend to be unreliable unless the corpus is enormous. That is a useful reminder about scale in general. Statistics about language need a lot of language.

A second classic weight answers a different question. Instead of asking which words travel with the seed, tf-idf asks which words mark out one document from the rest. Term frequency counts how often a word appears in a document, as a share of its words; inverse document frequency, the idea Karen Sparck Jones published in 1972, is the logarithm of the number of documents divided by the number that contain the word. A word found everywhere gets an idf of zero. In the worked example of the tf-idf article, “example” appears 3 times in a 7-word document and in one of two documents, so its tf-idf is 3/7 times log10(2), about 0.129.

The 1990 idea: a smaller matrix

Counting pairs one at a time misses something. “Bike” and “bicycle” may never share a sentence, because writers rarely use both at once, yet they keep the same company: chains, tyres, brakes, rides. Latent semantic analysis was built to notice exactly that. The Wikipedia article says the technique was patented in 1988 and first applied to text at Bellcore in the late 1980s, and that it is called latent semantic indexing when used for retrieval. The paper that set it out, “Indexing by latent semantic analysis” by Scott Deerwester and colleagues, appeared in 1990.

The method starts with a term-document matrix: one row per term, one column per document, each cell a weighted count. It then factors that matrix with the singular value decomposition, writing it as U times a diagonal matrix of singular values times V transposed. Keeping only the largest k singular values gives the closest matrix of rank k, closest in the sense of the Frobenius norm. That is the Eckart-Young result.

The low-rank approximation article states its price exactly: the error is the square root of the sum of the squared singular values that were dropped. Terms are then compared as rows of U times the kept singular values, usually by cosine similarity, the cosine of the angle between two vectors.

On the example texts the matrix is 40 terms by 13 documents. Keeping 3 of its 13 singular values keeps 51.8% of the squared norm. “Bike” appears in two documents and “bicycle” in two others, never together, so their rows in the original matrix have a cosine of exactly 0. At rank 3 the cosine is 0.987. With all 13 dimensions kept, the matrix is rebuilt exactly and the cosine falls back to 0. The rank cut is what lets two words that never met be judged alike, and a different rank tells a different story.

The decomposition on this page is computed by a one-sided Jacobi method, which repeatedly rotates pairs of columns until all of them are at right angles. It is checked on hundreds of random matrices, including tall, wide and rank-deficient ones, and on the four by five example printed in the SVD article, whose singular values are 15, 6, 5 and 0.

A demonstration, not a finding

Here the honest part matters most. Thirteen short paragraphs are a demonstration. A matrix of 40 by 13 has too little data for its third dimension to mean anything beyond these particular sentences, and a near-perfect cosine between “chain” and “pump” says more about one paragraph than about bicycles. The same is true of most pasted corpora: a dozen product pages or a handful of articles. The method is the same one used on large collections; the evidence is not.

The LSA article lists other limits worth keeping in view. Each word is a single point, so a word with two meanings is averaged into one place; the article calls LSA only partly able to capture polysemy. A document is treated as a bag of words, so word order inside it is ignored entirely. And the article warns that the dimensions can be difficult to interpret. Nothing guarantees that dimension 2 means “kitchen versus shed”; it is a direction in which these documents happen to differ most after the first.

The tokens themselves carry choices too. Words are split by the browser’s own segmenter and folded with Unicode case folding, so “Chain” and “CHAIN” count as one. Stop words come from the 318-word English list of the open-source scikit-learn project, which includes a few content words such as “system” and “interest”. The Porter stemmer of 1980 merges “rides” with “ride”, and occasionally merges words that are only alike in spelling. Each choice can be switched off, because each one changes the numbers.

The long afterlife of a phrase

Somewhere along the way, “LSI keywords” became a phrase in marketing advice: lists of related words to sprinkle into a page so that it seems more relevant. The method itself offers no support for that. Latent semantic indexing works on a fixed collection and its own matrix. A writer cannot compute a search engine’s matrix, because the writer does not have the collection; and the method says nothing about rankings at all.

Google’s own SEO starter guide, read for this page on 21 October 2026, does not mention latent semantic indexing. It does address several topics it says it has seen circulating: it says Google Search does not use the keywords meta tag, that keyword stuffing is against its spam policies, and that word count has no magic target. Its advice on vocabulary is plainer. Think about the words different readers might use, and write naturally.

That advice and the real method point the same way. Co-occurrence statistics are a fine way to study your own texts: to see which terms your documents lean on, which pairs never appear together, and which words keep the same company without ever meeting. They are a poor way to chase a ranking, and this page makes no claim that any term it shows will change one. Every number here is a count, a score or a distance that can be checked by hand, which is the most a small corpus can honestly offer.