World Language Similarity · 133 official languages · 40 basic words

How Similar Are Languages?

How alike are the world's official languages if you only listen to their most basic words? Every pair is scored from 0% to 100% similar, using 40 everyday words written in one shared sound alphabet.

Interesting findings

    What the similarity score means

    How the score works

    For each pair of languages we line up the same 40 basic words (I, you, two, water, eye, star, name…) and ask how much of the sound is shared. The answer is a similarity score from 0% to 100%, averaged across the 40 words. 100% means the two word lists sound identical. Even completely unrelated languages share a few sounds by accident, so they rarely reach 0%: they average about 11%. Anything well above that points to shared ancestry or heavy borrowing.

    What it does and does not tell you

    A score is not a measure of whether speakers can understand each other. It tracks shared ancestry in everyday vocabulary, which is what linguists use to sort languages into families.

    • 75% and above: very close relatives, often called dialects of one language (Czech and Slovak).
    • Around 35 to 60%: the same branch of a family (Spanish and Portuguese, English and German).
    • Around 10 to 30%: distant relatives in the same family (Hungarian and Finnish).
    • Under about 10%: no detectable relationship (Japanese and Korean).

    Scores use PPW, the default method explained below. Grey rows are averages over all pairs in the world set.

    Four ways to compare two words

    All four methods line up two words sound by sound and ask how much of one you would have to change to get the other. They differ in what a change costs and in how they treat chance resemblance. Every score on this page uses PPW unless you pick another method on the Similarity Web or Similarity Matrix tabs.

    PPW · phonetic penalised weighted · default

    Count sound changes, then drop weak matches

    PPW starts from the phonetic weighted score of each word pair (next card). If a pair is at least 50% similar, it counts in full. If it is below 50%, the two words probably do not share an ancestor and only resemble each other by accident, so PPW scores it 0. The language score is the plain average over the 40 words, so it effectively measures how many basic words two languages share and how closely those shared words match.

    History

    PPW was designed for this project. It borrows an idea from automatic cognate detection, where a computer decides whether two words come from the same ancestor word: pairs below a similarity cut-off are treated as unrelated. PPW applies that cut-off directly, and it replaces a separate chance correction: accidental look-alikes almost never reach 50%, so they simply drop out.

    The maths

    sw = phonetic weighted similarity of word w
    f(s) = s        if s ≥ 0.5
    f(s) = 0        if s < 0.5
    PPW = (1/n) · Σw f(sw)  over the n shared concepts
    person: Czech osoba vs Slovak tʃlovek
    weighted 40%, below 50% → counts 0%
    bone: kost vs kosc 85% → counts 85%
    Read about cognates on Wikipedia →
    Phonetic weighted

    Edits that know which sounds are related

    Like Levenshtein, but each change has a price that depends on the sounds involved. Swapping a sound for a close cousin (p and b, t and d, s and š) costs a quarter or 60% of a full change, a vowel for another vowel costs 40%, and dropping a vowel or h is cheaper than dropping a consonant. Languages drift along exactly these paths, so related words stay close. Finally the score is corrected for chance: it is compared with how alike words of different meaning are in the same two languages, and only similarity above that level counts.

    History

    Edit distances with unequal costs were formalised by Robert Wagner and Michael Fischer in 1974, whose algorithm still computes them. Grouping consonants into classes that readily turn into one another goes back to Aharon Dolgopolsky's sound classes of 1964. The chance correction follows the ASJP team (Wichmann, Holman, Bakker and Brown, 2010).

    The maths

    D[i,j] = min( D[i−1,j] + del(ai),
             D[i,j−1] + del(bj),
             D[i−1,j−1] + sub(ai,bj) )
    sub: same 0 · same class 0.25 · vowel 0.4 · related class 0.6 · other 1
    d = D / max(|a|,|b|)
    S = 1 − mean(d same meaning) / mean(d different meaning)
    bone: kost vs kosc
    t → ts is a related sound: costs 0.6
    1 − 0.6 / 4 = 85%
    Read about edit distance on Wikipedia →
    Levenshtein

    Count the edits

    The number of single-sound changes (swap one sound, add one, remove one) needed to turn one word into the other, divided by the length of the longer word. Every change costs the same, and nothing is subtracted for chance, so even unrelated languages score 10% or more.

    History

    Soviet mathematician Vladimir Levenshtein defined the measure in 1965 while studying codes that correct transmission errors. Brett Kessler first used it to compare dialects in 1995, measuring Irish Gaelic, and the ASJP project has used it on the world's languages since about 2008.

    The maths

    D[i,j] = min( D[i−1,j] + 1,
             D[i,j−1] + 1,
             D[i−1,j−1] + [ai ≠ bj] )
    similarity = 1 − D / max(|a|,|b|)
    language score = mean over the 40 words
    bone: kost → kosc
    1 swap out of 4 sounds
    1 − 1/4 = 75%
    Read about Levenshtein distance on Wikipedia →
    Jaro-Winkler

    Shared letters, bonus for the start

    Counts the sounds two words share in roughly the same position, penalises shared sounds that appear in swapped order, and adds a bonus when the words begin the same way. It was built to catch typing errors in names, so it is generous with short, unrelated words.

    History

    Matthew Jaro of the US Census Bureau introduced the measure in 1989 for record linkage, matching names across records of the 1985 census of Tampa, Florida. William Winkler added the bonus for a shared beginning in 1990.

    The maths

    m = sounds matching within ⌊max(|a|,|b|)/2⌋ − 1 positions
    t = half the matched sounds out of order
    J = ⅓ · ( m/|a| + m/|b| + (m − t)/m )
    JW = J + ℓ · 0.1 · (1 − J),  ℓ = shared start, at most 4
    bone: kost vs kosc = 88%
    English vs Japanese overall: 33%,
    though the two are unrelated
    Read about Jaro-Winkler on Wikipedia →

    How the speaker numbers are defined

    Bubble sizes on the Similarity Web show how many people speak each language worldwide, counting first and second language speakers. Each language gets one number for the whole world; it is not split by country.

    The figures come from the Unicode Common Locale Data Repository (CLDR), release 48, the dataset phones and computers use for language settings. For every country, CLDR estimates the share of the population that uses each language. We multiply that share by the country's population and add up all countries.

    • Second-language speakers are included, so English (about 1.7 billion) and French come out far larger than native counts.
    • Shares are estimates of different ages and quality from country to country; treat them as rough sizes, not censuses.
    • CLDR has no separate figure for dialects, so a language such as Arabic is one bubble.

    Which languages are included

    • Every national official language listed by CLDR (official or official in practice) for every country, each placed on the continents of the countries where it is official.
    • Five state languages of India, added on request: Telugu, Kannada, Malayalam, Punjabi and Sanskrit. Punjabi also counts for Pakistan, where most of its speakers live. Sanskrit uses ASJP's classical word list.
    • South Africa's 11 official languages are included; CLDR marks them as regional by mistake.
    • Countries spanning two continents (Turkey, Russia, Kazakhstan, Azerbaijan, Georgia, Armenia, Cyprus) count in both Europe and Asia.
    • Six regional languages from the earlier European study are kept: Basque, Welsh, Breton, Scottish Gaelic, Galician and Northern Sami.
    • Missing: Manx and Fiji Hindi, which have no usable ASJP word list.
    • National varieties (English in India against English in the USA, say) cannot be compared: ASJP holds one list per language, usually recorded in its homeland.

    Where the words come from

    The word lists

    In 1952 the linguist Morris Swadesh proposed a list of basic meanings (pronouns, numbers, body parts, nature) that every language has words for and that are rarely borrowed. His final 100-word version appeared in 1971, and the Swadesh list became the standard tool for measuring how related languages are.

    The lists here come from the Automated Similarity Judgment Program (ASJP), founded around 2008 by Cecil Brown, Søren Wichmann, Eric Holman and colleagues, via its Lexibank release. ASJP records the 40 most stable Swadesh concepts in a simple 41-symbol sound alphabet, so spelling systems drop out.

    How the scores and charts are built

    Words flagged as borrowed are left out. When a language has several words for one concept, the closest pair is used.

    In the Similarity Web, clusters are found with Louvain community detection on each language's three most similar links. In the Similarity Matrix, rows are ordered by average-linkage clustering so that similar languages sit next to each other.

    About the author

    Michael Brabec

    A data analyst at heart with a keen interest in linguistics. This project started with a simple question, how close are the languages we speak, and grew into a map of 133 of them.

    Would you like to see a language added, spotted something that looks off, or fancy collaborating? I would love to hear from you.

    Connect on LinkedIn linkedin.com/in/michael-brabec-uk
    Continents
    Method
    Colour by

    Similarity Web

    vs
    –

    Word by word

    ConceptABSimilarity

    Missing a language?

    Tell me which one and I will do my best to add it. Fill in the two boxes and your email app opens with the message ready to send.

    Email my request or write to info@michaelbrabec.com