Zipf Karaoke

Words fade by how common they are; press play and each one decays at its own rate.
0 / 0words visible
100%meaning retained
allthreshold
1 token2345+pieces underlined in alternating colours
What is this?

Zipf's law says a word's frequency is roughly inversely proportional to its rank, so a handful of words ("the", "of", "and") do most of the work while the long tail of rare words carries most of the surprise. The slider fades words by their log rank in a corpus: push it right and only the rare, information-heavy words remain; push it left and only the grammatical skeleton of function words survives. Press play and the text is sung word by word, each word pitched by its rarity and lingering for a time set by its frequency, so a common word vanishes within a beat while a rare one holds for the rest of the song.

The Tokens lens shows the same text as a language model chops it: each word is coloured by how many BPE pieces it becomes. "Common in..." swaps the frequency table, so a word rare in general English can be common in this blog. Everything (text, lens, corpus, slider, tempo) lives in the URL, so the address bar is a link to what you see. Space plays and pauses; click a word to hear its note.

Frequencies: the top 50,000 words of Peter Norvig's count_1w.txt (Google Web Trillion Word Corpus, MIT licence). Tokenizer: gpt-tokenizer (cl100k_base) from jsDelivr, with a length-based estimate if the CDN is unreachable. Texts: public domain.

Text

Edit text

Lens

keep commonshow allkeep rare

Karaoke

150 wpm
Common words go low and vanish within a beat; rare words ring high and stay.