Azure Cognitive Services

Analyze text using the Azure Cognitive Services Text Analytics APIs.

This capability is provided by the Azure Cognitive Services - Text Analytics plugin, which you need to install. Please see Installing plugins.

You can:

  • Detect the dominant language of a text (recommended first step)

  • Analyze the sentiment of a text (positive, negative, neutral, or mixed)

  • Recognize “real-world objects” (people names, dates, places, etc.) in a text

  • Extract key phrases in a text to identify the main points

Azure API usage is subject to your resource’s pricing tier. You can consult the API pricing page to evaluate the future cost.

How to set up

If you are a Dataiku and Azure admin user, follow these configuration steps right after you install the plugin. If you are not an admin, you can forward this to your admin and scroll down to the How to use section.

Create a new Azure Cognitive Services resource – in Microsoft Azure

You need an Azure subscription to create the resource.

Create an Azure resource that provides Text Analytics, following the resource setup steps in this Azure quickstart.

On the Resource creation page, you will be able to specify several elements, including the Azure location and the pricing tier. You can note down the chosen location, as this information will be needed later.

After your resource is successfully deployed, retrieve an API key from its Keys and endpoint page in the Azure portal.

Create an API configuration preset – in Dataiku

In Dataiku, open the Azure Cognitive Services - Text Analytics plugin from Plugins, navigate to Settings > API configuration, and create a preset.

Configure the preset – in Dataiku

  • Fill the AUTHENTICATION settings

    • Copy your resource’s API key into Azure API Key and its region into Azure region.

    • The format for the Azure region parameter must be its short name. You can find the mapping between the full and short name of your Azure region by running this command.

    • Alternatively, you may leave the fields empty so that the credentials are ascertained from the server environment.

    • If you choose this option, set AZURE_TEXT_ANALYTICS_KEY and AZURE_TEXT_ANALYTICS_ENDPOINT in the recipe execution environment. The endpoint must be the resource’s base URL, without a Text Analytics API path. Clear the default Azure region value (eastus) to use the endpoint environment variable.

  • (Optional) Review the API QUOTA and PARALLELIZATION settings

    • Rate limit sets the maximum number of requests per Period for each recipe run. The defaults are 300 requests per 60 seconds. These settings limit requests; they do not guarantee throughput.

    • Batch size sets the number of documents sent per request, with a default of 5.

    • Adjust these settings to your resource’s quota and the Azure service limits. Limits differ by feature and pricing tier; the plugin defaults do not guarantee that you stay within your resource’s quota.

    • Decrease Rate limit when multiple recipes or other applications share the same Azure resource. The preset does not coordinate quotas across recipe runs.

    • Concurrency sets how many requests can run in parallel, with a default of 4, within the per-run quota settings above.

    • Keep Batch size at 5 or below when sharing a preset across all four recipes, to respect the most restrictive document-count limit. Individual documents must also fit within Azure’s size limits.

  • Set the Permissions of your preset

    • You can declare yourself as Owner of this preset and make it available to everybody, or to a specific group of users.

    • Any user belonging to one of these groups on your Dataiku instance will be able to see and use this preset.

Your preset is now ready to be used.

Later, you (or another Dataiku admin) will be able to add more presets. This can be useful to segment usage by user group. For instance, you can create a Default preset for everyone and a High performance one for your Marketing team. Separate presets using the same Azure resource do not provide separate billing.

How to use

Let’s assume that you have a Dataiku project with a dataset containing text data. This text data must be stored in a dataset, inside a text column, with one row for each document.

As an example, we will use the Microsoft Research WikiQA Corpus. You can follow the same steps with your own data.

To create your first recipe, navigate to the Flow, click on the + RECIPE button and access the Natural Language Processing menu. If your dataset is selected, you can directly find the plugin on the right panel.

Select one of the four recipes below, choose the input dataset, and create or select an output dataset. Configure its settings and run the recipe. Each recipe sends the selected text to Azure and retains the input columns in the output dataset.

The output descriptions below assume the default Log error handling. In Fail mode, the error-message and error-type columns are omitted, and API errors cause the recipe to fail.

Language Detection

Input

Dataset with a text column

Output

Dataset with 6 additional columns

  • Language name detected by the API

  • Language code from the API in ISO 639 format

  • Confidence score of the API from 0 to 1

  • Raw response for the document in JSON format

  • Error message from the API if any

  • Azure error code or Python exception type, if any

Settings

  • Fill INPUT PARAMETERS

    • Specify the Text column parameter for your column containing text data.

  • Review CONFIGURATION parameters

    • Select an API configuration preset made available to you by your Dataiku admin.

  • (Optional) Review ADVANCED parameters

    • You can activate the Expert mode to access advanced parameters.

    • The optional Country hint parameter allows you to specify the origin country of your dataset.

    • The Error handling parameter determines how the recipe will behave if the API returns an error.

      • In “Log” error handling, this error will be logged to the output but it will not cause the recipe to fail.

      • We do not recommend to change this parameter to Fail mode unless this is the desired behaviour.

Sentiment Analysis

Input

Dataset with a text column.

Output

Dataset with 7 additional columns.

  • Sentiment prediction from the API (neutral, negative, positive, or mixed)

  • Confidence score in the NEUTRAL prediction from 0 to 1

  • Confidence score in the NEGATIVE prediction from 0 to 1

  • Confidence score in the POSITIVE prediction from 0 to 1

  • Raw response for the document in JSON format

  • Error message from the API if any

  • Azure error code or Python exception type, if any

Settings

Configure Text column and API configuration preset as for Language Detection. Expert mode exposes Error handling; Country hint is not available for this recipe.

Set Language to one of the languages available in the recipe; the default is English. For multilingual data, choose Detected language column, then set Language column to the column containing each document’s language code, such as the one produced by Language Detection. The available languages differ between Sentiment Analysis, Named Entity Recognition, and Key Phrase Extraction.

Named Entity Recognition

Input

Dataset with a text column.

Output

Dataset with additional columns

  • One column for each selected entity type, with a list of entities.

  • Raw response for the document in JSON format.

  • Error message from the API if any.

  • Azure error code or Python exception type, if any.

Settings

The parameters under INPUT PARAMETERS and CONFIGURATION are almost the same as the Sentiment Analysis recipe (see above). The additional parameters specific to this recipe are:

  • Under CONFIGURATION. Entity types: select one or more entity types from the list available in the recipe.

  • Under ADVANCED with Expert mode activated. Minimum score: retain only entities whose confidence score meets this threshold in the selected entity-type columns. The value must be between 0 and 1; the default is 0. The raw response remains unfiltered.

Key Phrase Extraction

Input

Dataset with a text column.

Output

Dataset with additional columns.

  • One column for each requested key phrase, see Number of key phrases. The columns contain the first phrases returned by Azure, with empty values if fewer phrases are returned. The recipe does not impose a word-count limit on each phrase.

  • Raw response for the document in JSON format.

  • Error message from the API if any.

  • Azure error code or Python exception type, if any.

Settings

The parameters under INPUT PARAMETERS and CONFIGURATION are almost the same as the Sentiment Analysis recipe (see above). The additional parameter specific to this recipe is:

Under CONFIGURATION: Number of key phrases: choose how many phrase columns to include, from 1 to 100. The default is 3. This setting does not change the raw response.

Visualization

Use the output datasets to create charts to analyze results from the API. For instance, you can:

  • filter documents to focus on one language

  • analyze the distribution of sentiment scores

  • identify which entities are mentioned

  • understand what are the key phrases used by responders

After crafting these charts, you can share them with business users in a dashboard.