Text Preparation

Prepare text data with language detection, spell checking and text cleaning.

This capability is provided by the Text Preparation plugin, which you need to install. Please see Installing plugins.

It allows you to:

• Detect dominant languages among 114 supported languages. If you have multilingual data, this step is useful before applying language-specific processing.

• Identify and correct misspellings in 37 supported languages.

• Tokenize, filter, and lemmatize text data in 59 supported languages.

Note that languages are defined as per the ISO 639-1 standard with 2-letter codes.

How to use

Let’s assume that you have a Dataiku DSS project with a dataset containing raw text data in one or more languages. This text data must be stored in a dataset, inside a text column, with one row for each document.

Navigate to the Flow, click +RECIPE, and open the Natural Language Processing menu. If your dataset is selected, you can directly find the recipes in the right panel.

The recipes retain the input columns and prefix additional column names with the selected text column’s name. The output examples below assume that this column is named text. If an output column name already exists, a numeric suffix is added to avoid a conflict.

Language Detection recipe

Detect dominant languages among 114 languages.

Input

Text dataset: Dataset with a text column.

Settings

• Fill INPUT PARAMETERS

  • The Text column parameter lets you choose the column of your input dataset containing text data.

  • The Language scope parameter allows you to specify the set of languages in your specific dataset. You can leave it empty (default) to detect all 114 languages.

• (Optional) Review ADVANCED parameters.

You can activate Expert mode to access advanced parameters.

• The Minimum score parameter allows you to filter detected languages based on the confidence score in the prediction. You can leave it at 0 (default) to apply no filtering.

• The Fallback language parameter is for cases where the detected language is not in your language scope, or the confidence score is below your specified minimum.

  • You can leave it at None (default) to output an empty cell.

  • If the fallback is used, the confidence score will be an empty cell.

Output

Output dataset: Dataset enriched with three columns.

• text_language_code for the ISO 639-1 language code.

• text_language_name for the ISO 639-1 language name.

• text_language_score for the predicted probability from 0 to 1.

Spell Checking recipe

Identify and correct misspellings in 37 languages.

Input

• Text dataset: Dataset with a text column

• (Optional) Custom vocabulary dataset

  • This dataset must contain a single column of DSS type string for words that should not be corrected.

  • This input is case-sensitive, so “NY” and “Ny” are considered as different words.

  • If your data contains special terms or slang which are not in the bundled dictionaries, we recommend using this dataset.

• (Optional) Custom corrections dataset

  • This dataset must contain exactly two columns of DSS type string, the first one for words and the second one for your custom correction.

  • Words are also case-sensitive.

  • If the same word is present in both custom vocabulary and corrections, the custom correction will overrule the custom vocabulary.

  • Use this dataset if you wish to correct the special terms or slang contained in your data.

Settings

• Fill INPUT PARAMETERS

  • The Text column parameter lets you choose the column of your input dataset containing text data.

  • The Language parameter lets you choose among 37 supported languages if your text is monolingual. Otherwise, the Multilingual option lets you set a Language column. This Language column can use the language-code output of the Language Detection recipe (for example, text_language_code) or ISO 639-1 language codes computed by other means. Codes must belong to the 37 languages supported by Spell Checking. Filter out rows with unsupported codes or handle them separately before running the recipe; unsupported codes cause the recipe to fail.

• (Optional) Review ADVANCED parameters

You can activate the Expert mode to access advanced parameters.

• The Edit distance parameter allows you to tune the maximum edit distance between a word and its potential correction.

• The Ignore pattern parameter lets you define a regular expression matching words which should not be corrected. This is useful if you work with special-domain data with acronyms and codes.

Output

• Output dataset: Dataset with four additional columns

  • text_corrected for corrected text.

  • text_misspellings for misspelled text.

  • text_misspelling_list for the list of unique misspellings.

  • text_misspelling_count for the number of misspellings.

• (Optional) Diagnosis dataset with spell checking information on each word

If you connect this optional output, the recipe will also produce a diagnosis dataset with spell checking information on each word. Note that computing this diagnosis dataset increases the recipe runtime.

Text cleaning recipe

Tokenize, filter, and lemmatize text data in 59 languages.

Input

Text dataset: Dataset with a text column.

Settings

• Fill INPUT PARAMETERS

  • The Text column parameter lets you choose the column of your input dataset containing text data.

  • The Language parameter lets you choose among 59 supported languages if your text is monolingual. Otherwise, the Multilingual option lets you set a Language column. This Language column can use the language-code output of the Language Detection recipe (for example, text_language_code) or ISO 639-1 language codes computed by other means. Codes must belong to the 59 languages supported by Text Cleaning. Filter out rows with unsupported codes or handle them separately before running the recipe; unsupported codes cause the recipe to fail.

• Review CLEANING PARAMETERS

  • Select which types of token to filter in the Filter tokens parameter:

    • Punctuation: if all token characters are within the Unicode “P” class

    • Stopword: if the token matches our per-language stopword lists. These stopword lists are based on spaCy and NLTK, with an additional human review done by native speakers at Dataiku.

    • Number: if the token contains only digits (e.g. 11) or matches the written form of a number in the corresponding language (e.g. eleven).

    • Currency sign: if all token characters are within the Unicode “Sc” class

    • Datetime: if the token matches this regular expression for tokens like “10:04” or “2/10/2049”.

    • Measure: if the token matches this rule for tokens like “17th” or “10km”.

    • URL: if the token matches this rule for tokens starting with “http://”, “https://”, “www.” or matching a correct domain name.

    • Email: if the token matches this regular expression.

    • Username: if the token begins with “@”.

    • Hashtag: if the token begins with “#”.

    • Emoji: if one of the token characters is recognized as a Unicode emoji.

    • Symbol: if the token is not an emoji and if all token characters are within the Unicode “M”, “S” classes.

  • Activate Lemmatize to simplify words to their lemma form, sometimes called “dictionary” form.

  It uses spaCy per-language lookup data under the hood. - Activate Lowercase to convert all words to lowercase.

• (Optional) Review ADVANCED parameters

You can activate the Expert mode to access advanced parameters.

• The Unicode normalization parameter allows you to add a post-processing step to apply one of the unicodedata normalization methods.

• If you activate the Keep filtered tokens parameter, you will get an additional column for each selected Token filter. This is useful if you want to analyze the usage of tokens like emojis or hashtags in your text data.

Output

• Output dataset: Dataset with additional columns

  • text_cleaned for the cleaned text after tokenization, filtering, and lemmatization

  • If the Keep filtered tokens parameter is activated, one additional column for each selected filter, such as text_punctuations, text_stopwords, and text_numbers.

Supported languages

Language detection

Here are the 114 supported languages and their ISO 639-1 format:

• Afrikaans (af)

• Albanian (sq)

• Amharic (am)

• Arabic (ar)

• Aragonese (an)

• Armenian (hy)

• Assamese (as)

• Azerbaijani (az)

• Basque (eu)

• Belarusian (be)

• Bengali (bn)

• Bosnian (bs)

• Breton (br)

• Bulgarian (bg)

• Burmese (my)

• Catalan (ca)

• Central Khmer (km)

• Chinese (zh)

• Croatian (hr)

• Czech (cs)

• Danish (da)

• Dutch (nl)

• Dzongkha (dz)

• English (en)

• Esperanto (eo)

• Estonian (et)

• Faroese (fo)

• Finnish (fi)

• French (fr)

• Galician (gl)

• Georgian (ka)

• German (de)

• Greek (el)

• Gujarati (gu)

• Haitian (ht)

• Hausa (ha)

• Hebrew (he)

• Hindi (hi)

• Hungarian (hu)

• Icelandic (is)

• Igbo (ig)

• Indonesian (id)

• Irish (ga)

• Italian (it)

• Japanese (ja)

• Javanese (jv)

• Kannada (kn)

• Kazakh (kk)

• Kinyarwanda (rw)

• Kirghiz (ky)

• Korean (ko)

• Kurdish (ku)

• Lao (lo)

• Latin (la)

• Latvian (lv)

• Lithuanian (lt)

• Luxembourgish (lb)

• Macedonian (mk)

• Malagasy (mg)

• Malay (ms)

• Malayalam (ml)

• Maltese (mt)

• Maori (mi)

• Marathi (mr)

• Mongolian (mn)

• Nepali (ne)

• Northern Sami (se)

• Norwegian (no)

• Norwegian Bokmål (nb)

• Norwegian Nynorsk (nn)

• Nyanja (ny)

• Occitan (oc)

• Oriya (or)

• Panjabi (pa)

• Persian (fa)

• Polish (pl)

• Portuguese (pt)

• Pushto (ps)

• Quechua (qu)

• Romanian (ro)

• Russian (ru)

• Samoan (sm)

• Scottish Gaelic (gd)

• Serbian (sr)

• Shona (sn)

• Sindhi (sd)

• Sinhala (si)

• Slovak (sk)

• Slovenian (sl)

• Somali (so)

• Southern Sotho (st)

• Spanish (es)

• Sundanese (su)

• Swahili (sw)

• Swedish (sv)

• Tagalog (tl)

• Tajik (tg)

• Tamil (ta)

• Telugu (te)

• Thai (th)

• Turkish (tr)

• Uighur (ug)

• Ukrainian (uk)

• Urdu (ur)

• Uzbek (uz)

• Vietnamese (vi)

• Volapük (vo)

• Walloon (wa)

• Welsh (cy)

• Western Frisian (fy)

• Xhosa (xh)

• Yiddish (yi)

• Yoruba (yo)

• Zulu (zu)

Spell checking

Here are the 37 supported languages and their ISO 639-1 format:

• Albanian (sq)

• Arabic (ar)

• Bulgarian (bg)

• Catalan (ca)

• Chinese (simplified) (zh)

• Croatian (hr)

• Czech (cs)

• Danish (da)

• Dutch (nl)

• English (en)

• Estonian (et)

• Finnish (fi)

• French (fr)

• German (de)

• Greek (el)

• Hebrew (he)

• Hungarian (hu)

• Icelandic (is)

• Indonesian (id)

• Italian (it)

• Japanese (ja)

• Latvian (lv)

• Lithuanian (lt)

• Persian (fa)

• Polish (pl)

• Portuguese (pt)

• Romanian (ro)

• Russian (ru)

• Serbian (sr)

• Slovak (sk)

• Slovenian (sl)

• Spanish (es)

• Swedish (sv)

• Thai (th)

• Turkish (tr)

• Ukrainian (uk)

• Vietnamese (vi)

Text cleaning

Here are the 59 supported languages and their ISO 639-1 format:

An asterisk marks languages where the Lemmatize option is not available.

• Afrikaans (af) *

• Albanian (sq) *

• Arabic (ar) *

• Armenian (hy) *

• Basque (eu) *

• Bengali (bn)

• Bulgarian (bg) *

• Catalan (ca)

• Chinese (simplified) (zh) *

• Croatian (hr)

• Czech (cs)

• Danish (da)

• Dutch (nl)

• English (en)

• Estonian (et) *

• Finnish (fi) *

• French (fr)

• German (de)

• Greek (el)

• Gujarati (gu) *

• Hebrew (he) *

• Hindi (hi) *

• Hungarian (hu)

• Icelandic (is) *

• Indonesian (id)

• Irish (ga) *

• Italian (it)

• Japanese (ja) *

• Kannada (kn) *

• Latvian (lv) *

• Lithuanian (lt)

• Luxembourgish (lb)

• Macedonian (mk)

• Malayalam (ml) *

• Marathi (mr) *

• Nepali (ne) *

• Norwegian Bokmål (nb)

• Persian (fa)

• Polish (pl)

• Portuguese (pt)

• Romanian (ro)

• Russian (ru)

• Sanskrit (sa) *

• Serbian (sr)

• Sinhala (si) *

• Slovak (sk) *

• Slovenian (sl) *

• Spanish (es)

• Swedish (sv)

• Tagalog (tl)

• Tamil (ta) *

• Tatar (tt) *

• Telugu (te) *

• Thai (th) *

• Turkish (tr)

• Ukrainian (uk) *

• Urdu (ur)

• Vietnamese (vi) *

• Yoruba (yo) *