Wiktextract

This is a utility and Python package for extracting data from Wiktionary.

Please report issues on github and we'll try to address them reasonably soon.

Some extracted Wiktionary editions data are available for browsing and downloading at https://kaikki.org, the website will be updated every few days.

Note: extracting all data for all languages from the English Wiktionary may take from an hour to several days, depending on your computer. Expanding Lua modules is not cheap, but it enables superior extraction quality and maintainability! You may want to look at the data downloads instead of running it yourself.

Overview

This is a Python package and tool for extracting information from various Wiktionary data dumps, most notably and completely the English edition (enwiktionary). Note that an edition of Wiktionary contains extensive dictionaries and inflectional information for many languages, not just the language it has been written in.

One thing that distinguishes this tool from any system we're aware of is that this tool expands templates and Lua macros in Wiktionary. That enables much more accurate rendering and extraction of glosses, word senses, inflected forms, and pronunciations. It also makes the system much easier to maintain. All this results in much higher extraction quality and accuracy.

The English edition extraction 'module' extracts glosses, parts-of-speech, declension/conjugation information when available, translations for all languages when available, pronunciations (including audio file links), qualifiers including usage notes, word forms, links between words including hypernyms, hyponyms, holonyms, meronyms, related words, derived terms, compounds, alternative forms, etc. Links to Wikipedia pages, Wikidata identifiers, and other such data are also extracted when available. For many classes of words, a word sense is annotated with specific information such as what word it is a form of, what is the RGB value of the color it represents, what is the numeric value of a number, what SI unit it represents, etc.

Other editions are less complete (or the Wiktionary edition itself doesn't necessarily have the same width of data), but we try to cover the basics.

This tool extracts information for all languages that have data in the wiktionary edition. It also extracts translingual data and information about characters (anything that has an entry in Wiktionary).

This tool reads a <language-code>wiktionary-<date>-pages-articles.xml.bz2 dump file and outputs JSONL-format (json objects separated with newlines) dictionaries containing most of the information in Wiktionary. The dump files can be downloaded from https:// dumps.wikimedia.org.

This utility will be useful for many natural language processing, semantic parsing, machine translation, and language generation applications both in research and industry.

The tool can be used to extract machine translation dictionaries, language understanding dictionaries, semantically annotated dictionaries, and morphological dictionaries with declension/conjugation information (where this information is available for the target language). Dozens of languages have extensive vocabulary in enwiktionary, and several thousand languages have partial coverage.

The wiktwords script makes extracting the information for use by other tools trivial without writing a single line of code. It extracts the information specified by command options for languages specified on the command line, and writes the extracted data to a file or standard output in JSONL format (json objects separated with newlines) for processing by other tools.

As far as we know, this is the most comprehensive tool available for extracting information from Wiktionary as of December 2020.

If you find this tool and/or the pre-extracted data helpful, please give this a star on github!

Pre-extracted data

For most people, it may be easiest to just download pre-expanded data. Please see https://kaikki.org/dictionary/rawdata.html. The raw wiktextract data, extracted category tree, extracted templates and modules, as well as a bulk download of audio files for pronunciations in both .ogg and .mp3 formats are available.

There is a also download link at the bottom of every page and a button to view the JSON produced for each page. You can download all data, data for a specific language, data for just a single word, or data for a list of related words (e.g., a particular part-of-speech or words relating to a particular topic or having a particular inflectional form). All downloads are in JSON Lines format (each line is a separate JSON object). The bigger downloads are also available in compressed form.

Some people have asked for the full data as a single JSON object (instead of the current one JSON object per line format). I've decided to keep it as a JSON object per line, because loading all the data into Python requires about 120 GB of memory. It is much easier to process the data line-by-line, especially if you are only interested in a part of the information. You can easily read the files using the following code:

import json

with open("filename.json", encoding="utf-8") as f:
    for line in f:
        data = json.loads(line)
        ... # parse the data in this record

If you want to collect all the data into a list, you can read the file into a list with:

import json

lst = []
with open("filename.json", encoding="utf-8") as f:
    for line in f:
        data = json.loads(line)
        lst.append(data)

You can also easily pretty-print the data into a more human-readable form using:

print(json.dumps(data, indent=2, sort_keys=True, ensure_ascii=False))

Non-en editions have JSON schema files at https://tatuylonen.github.io/wiktextract/. The English edition doesn't have a JSON schema but the fields are listed at https://kaikki.org/dictionary/errors/mapping/index.html, and has TypedDict models in type_utils.py.

Here is a pretty-printed example of an extracted word entry for the word thrill as an English verb (only one part-of-speech is shown here):

{
  "categories": [
    "Emotions"
  ],
  "derived": [
    {
      "word": "enthrill"
    }
  ],
  "forms": [
    {
      "form": "thrills",
      "tags": [
        "present",
        "simple",
        "singular",
        "third-person"
      ]
    },
    {
      "form": "thrilling",
      "tags": [
        "present"
      ]
    },
    {
      "form": "thrilled",
      "tags": [
        "participle",
        "past",
        "simple"
      ]
    }
  ],
  "head_templates": [
    {
      "args": {},
      "expansion": "thrill (third-person singular simple present thrills, present participle thrilling, simple past and past participle thrilled)",
      "name": "en-verb"
    }
  ],
  "lang": "English",
  "lang_code": "en",
  "pos": "verb",
  "senses": [
    {
      "glosses": [
        "To suddenly excite someone, or to give someone great pleasure; to electrify; to experience such a sensation."
      ],
      "tags": [
        "ergative",
        "figuratively"
      ]
    },
    {
      "glosses": [
        "To (cause something to) tremble or quiver."
      ],
      "tags": [
        "ergative"
      ]
    },
    {
      "glosses": [
        "To perforate by a pointed instrument; to bore; to transfix; to drill."
      ],
      "tags": [
        "obsolete"
      ]
    },
    {
      "glosses": [
        "To hurl; to throw; to cast."
      ],
      "tags": [
        "obsolete"
      ]
    }
  ],
  "sounds": [
    {
      "ipa": "/\u03b8\u0279\u026al/"
    },
    {
      "ipa": "[\u03b8\u027e\u032a\u030a\u026a\u026b]",
      "tags": [
        "UK",
        "US"
      ]
    },
    {
      "ipa": "[\u03b8\u027e\u032a\u030a\u026al]",
      "tags": [
        "Ireland"
      ]
    },
    {
      "ipa": "[t\u032a\u027e\u032a\u030a\u026al]",
      "tags": [
        "Ireland"
      ]
    },
    {
      "rhymes": "-\u026al"
    },
    {
      "audio": "en-us-thrill.ogg",
      "mp3_url": "https://upload.wikimedia.org/wikipedia/commons/transcoded/d/db/En-us-thrill.ogg/En-us-thrill.ogg.mp3",
      "ogg_url": "https://upload.wikimedia.org/wikipedia/commons/d/db/En-us-thrill.ogg",
      "tags": [
        "US"
      ],
      "text": "Audio (US)"
    }
  ],
  "translations": [
    {
      "code": "nl",
      "lang": "Dutch",
      "sense": "suddenly excite someone, or to give someone great pleasure; to electrify",
      "word": "opwinden"
    },
    {
      "code": "fi",
      "lang": "Finnish",
      "sense": "suddenly excite someone, or to give someone great pleasure; to electrify",
      "word": "syk\u00e4hdytt\u00e4\u00e4"
    },
    {
      "code": "fi",
      "lang": "Finnish",
      "sense": "suddenly excite someone, or to give someone great pleasure; to electrify",
      "word": "riemastuttaa"
    },
...
    {
      "code": "tr",
      "lang": "Turkish",
      "sense": "slight quivering of the heart that accompanies a cardiac murmur",
      "word": "\u00e7arp\u0131nt\u0131"
    }
  ],
  "wikipedia": [
    "thrill"
  ],
  "word": "thrill"
}

Getting started

Installing

Use container:

$ podman run -v /data:/data -it --rm ghcr.io/tatuylonen/wiktextract --all --all-languages --out /data/fr-20250101.jsonl --edition fr /data/frwiktionary-20250101-pages-articles.xml.bz2

Install from source:

On Linux (example from Ubuntu 20.04), you may need to first install the build-essential and python3-dev packages with apt update && apt install build-essential python3-dev python3-pip lbzip2.

git clone https://github.com/tatuylonen/wiktextract.git
cd wiktextract
python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e .

Use pip install command's --force-reinstall and -e option to reinstall the wikitextprocessor package from source in editable mode if you want to update both packages' code with git pull.

Running tests

This package includes tests written using the unittest framework. The test dependencies can be installed with the command python -m pip install -e .[dev].

To run the tests, use the following command in the top-level directory:

make test

Expected performance

Extracting all data for all languages from English Wiktionary takes about 1.25 hours on a 128-core dual AMD EPYC 7702 system. The performance is expected to be approximately linear with the number of processor cores, provided you have enough memory (about 10GB/core or 5GB/hyperthread recommended).

As the extractor expands, these times will change.

You can control the number of parallel processes to use with the --num-processes option; the default is to use the number of available cores/hyperthreads.

You can download the full pre-extracted data from kaikki.org. The pre-extraction is updated regularly with the latest Wiktionary dump. Using the pre-extracted data may be the easiest option unless you have special needs or want to modify the code.

Using the command-line tool

The wiktwords script is the easiest way to extract data from Wiktionary. Just download the data dump file from dumps.wikimedia.org and run the script. The correct dump file the name enwiktionary-<date>-pages-articles.xml.bz2.

An example of a typical invocation for extracting all data would be:

wiktwords --all --all-languages --out data.json --edition en enwiktionary-20230801-pages-articles.xml.bz2

If you wish to modify the code or test processing individual pages, the following may also be useful:

1. Pass a path to save database file that you can use for quickly processing individual pages:

wiktwords --db-path en_20230801.db --edition en enwiktionary-20230801-pages-articles.xml.bz2

1. To process a single page and produce a human-readable output file for debugging:

wiktwords --db-path en_20230801.db --edition en --all --all-languages --out outfile --page page_title

The following command-line options can be used to control its operation:

• --out FILE: specifies the name of the file to write (specifying "-" as the file writes to stdout)
• --all-languages: extract words for all available languages
• --language-code LANGUAGE_CODE: extracts the given language (this option may be specified multiple times; defaults to dump file language code and mul(Translingual))
• --language-name LANGUAGE_NAME: Similar to --language-code except this option accepts language name
• --edition LANGUAGE_CODE: specifies the language code for the Wiktionary edition that the dump file is for (supported editions are listed in -h help descriptions)
• --skip-extraction: Used to create a database file from the dump file without waiting for the extraction process to complete.
• --all: causes all data to be captured for the selected languages
• --translations: causes translations to be captured
• --pronunciation: causes pronunciation information to be captured
• --linkages: causes linkages (synonyms etc.) to be captured
• --examples: causes usage examples to be captured
• --etymologies: causes etymology information to be captured
• --descendants: causes descendants information to be captured
• --inflections: causes inflection tables to be captured
• --redirects: causes redirects to be extracted
• --pages-dir DIR: save all wiktionary pages under this directory (mostly for debugging)
• --db-path PATH: save/use database from this path (for debugging)
• --page FILE or TITLE: read page from file or database, can be specified multiple times(first line must be "TITLE: pagetitle"; file should use UTF-8 encoding)
• --num-processes PROCESSES: use this many parallel processes (needs 4GB/process)
• --human-readable: print human-readable JSON with indentation (no longer machine-readable)
• --override PATH: override pages with files in this directory (first line of the file must be TITLE: pagetitle)
• --templates-file: extract Template namespace to this tar file
• --modules-file: extract Module namespace to this tar file
• --categories-file: extract Wiktionary category tree into this file as JSON (see description below)
• --inflection_tables_file: extract and expand tables into this file as wikitext; use this to create tests
• --help: displays help text (with some more options than listed here)

Calling the library

While this package has been mostly intended to be used using the wiktwords command, it is also possible to call this as a library. Underneath, this uses the wikitextprocessor module. For more usage examples please read the wiktwords.py and [wiktionary.