Amazon Comprehend Medical¶
DSS provides recipes to analyze medical text records using the Amazon Comprehend Medical APIs.
This capability is provided by the Amazon Comprehend Medical plugin, which you need to install.
You can:
Extract Protected Health Information (PHI) in a medical text record
Recognize Medical Entities (medical condition, treatment, etc.) in a medical text record
Note
The Amazon Comprehend Medical API is a paid service. You can consult the API pricing page to evaluate the future cost.
How to set up¶
If you are a Dataiku and AWS admin user, follow these configuration steps right after you install the plugin. If you are not an admin, you can forward this to your admin and scroll down to the How to use section.
Create an IAM user with the Amazon Comprehend Medical policy – in AWS¶
Let’s assume that your AWS account has already been created and that you have full admin access. If not, please follow the AWS setup guide.
Start by creating a dedicated IAM user to centralize access to the Comprehend API, or select an existing one. Next, you will need to attach a policy to this user following this documentation. We recommend using the “ComprehendMedicalFullAccess” managed policy.
Alternatively, you can create a custom IAM policy to allow “comprehendmedical:*” actions. After completing this step, you will be able to retrieve the user Access key ID and Secret access key.
Create an API configuration preset – in Dataiku¶
In Dataiku, open Plugins, select Amazon Comprehend Medical, and open Settings > API configuration to create your first preset.
Configure the preset – in Dataiku¶
Fill AUTHENTICATION settings.
Copy-paste your access key ID and secret access key into AWS access key ID and AWS secret access key.
Set AWS region to a region in this list. The default is
us-east-1.Alternatively, leave the credentials empty to use credentials from the server environment. Leaving AWS region empty also uses the region from the environment. Follow this documentation on the server hosting Dataiku.
(Optional) Review the API QUOTA settings.
Rate limit is the maximum number of requests per Period, in seconds, for one recipe run. The defaults are 5 requests per 1 second.
These defaults are plugin settings, separate from the AWS service quotas. Keep the combined usage of concurrent recipes within your AWS quotas.
You may need to decrease the Rate limit parameter if you envision that multiple recipes will run concurrently to call the API. For instance, if you want to allow 5 concurrent Dataiku activities, you can set this parameter at 5/5 = 1 request per second.
(Optional) Review the PARALLELIZATION settings.
Set the Permissions of your preset.
You can declare yourself as Owner of this preset and make it available to everybody, or to a specific group of users.
Any user belonging to one of these groups on your Dataiku instance will be able to see and use this preset.
Your preset is now ready to be used.
Later, you (or another Dataiku admin) will be able to add more presets. This can be useful to segment plugin usage by user group. For instance, you can create a “Default” preset for everyone and a “High performance” one for your Marketing team, with separate billing for each team.
How to use¶
Let’s assume that you have a Dataiku project with a dataset containing medical records. These records must be stored in a dataset, inside a text column, with one row for each record.
As an example, we will use a sample of the PubMed dataset. You can follow the same steps with your own data.
To create your first recipe, navigate to the Flow, click on the + RECIPE button and access the Natural Language Processing menu. Select one of the recipes below, select the input dataset, and create an output dataset. Configure the recipe settings, then run it.
Protected Health Information Extraction¶
Input¶
Dataset with a text column
Output¶
Dataset retaining the input columns, with 12 additional columns in the default Log mode (10 in Fail mode)
Nine columns containing lists of extracted text for address, age, date, name, phone or fax, email, ID, URL, and profession. See the AWS PHI documentation for entity descriptions. A column is empty when no matching entities meet Minimum score.
Raw response from the API in JSON format.
Error message from the API if any, in Log mode only.
Error type (module and class name) if any, in Log mode only.
Settings¶
Fill INPUT PARAMETERS
You can specify the Text column parameter for your column containing text data.
Language is set to English, the only available choice.
Fill CONFIGURATION parameters
Select an API configuration preset available to you.
(Optional) Review ADVANCED parameters
You can activate the Expert mode to access advanced parameters.
Minimum score filters the extracted text lists by confidence score, from 0 to 1. The default is 0, so no filtering is applied. The raw API response remains unfiltered.
The Error handling parameter determines how the recipe will behave if the API returns an error.
Log (the default) writes API errors to the output without failing the recipe.
Fail stops the recipe on an API error and omits the error columns.
Blank or missing text is not sent to the API. In Log mode, these rows have empty results and no API error. In Fail mode, they cause the recipe to fail when parsing the empty response.
Medical Entity Recognition¶
Input¶
Dataset with a text column
Output¶
Dataset retaining the input columns, with additional columns
One column for each selected medical entity category, containing a list of matched text. A column is empty when no matching entities meet Minimum score.
Raw response from the API in JSON format.
Error message from the API if any, in Log mode only.
Error type (module and class name) if any, in Log mode only.
Settings¶
The parameters are almost the same as the Protected Health Information Extraction recipe (see above). The only change is the addition of the Entity types parameter under CONFIGURATION. Select one or more of Anatomy, Medical condition, Medication, Protected health information, Test treatment procedure, and Time expression. All are selected by default except Protected health information.