A complete walk through Amazon Comprehend..ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Motivation:</h1> \n",
    "<p>This jupyter notebook is meant to serve as a complementary material for my tutorial on How to use Amazon Comprehend.</p>\n",
    "\n",
    "\n",
    "<ul>\n",
    "    <h3> This notebook is designed to showcase how to use AMAZON COMPREHEND for three tasks.</h3>\n",
    "    <li> Sentiment Analysis.</li>\n",
    "    <li> Key Phrase Extraction. </li>\n",
    "    <li> Named Entities Extraction. </li>\n",
    "    <li> Language Detection</li>\n",
    "</ul>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import Boto3 and connect its client to the comprehend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "comprehend = boto3.client(service_name='comprehend', region_name='us-east-1',aws_access_key_id = 'Your Access Key Id Goes Here', \n",
    "                          aws_secret_access_key = 'You Secret Key Goes Here')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are all set to use Comprehend! \n",
    "\n",
    "\n",
    "<h1> 1) Comprehend for Sentiment Analysis.</h1>\n",
    "<p> It is simple. Just use the __detect_sentiment__ method of your client.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Sentiment': 'POSITIVE',\n",
       " 'SentimentScore': {'Positive': 0.9986692667007446,\n",
       "  'Negative': 5.039777897763997e-05,\n",
       "  'Neutral': 0.00023834838066250086,\n",
       "  'Mixed': 0.0010420106118544936},\n",
       " 'ResponseMetadata': {'RequestId': '83970793-e89e-11e8-97c2-ddc68243f710',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'date': 'Thu, 15 Nov 2018 06:20:14 GMT',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '167',\n",
       "   'connection': 'keep-alive',\n",
       "   'x-amzn-requestid': '83970793-e89e-11e8-97c2-ddc68243f710'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent = comprehend.detect_sentiment(Text = 'Amazon is a great company. I love working there.', LanguageCode= 'en')\n",
    "sent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>The output is in java script object notation jason format. In python, this is equivallent to a nested dictionary. The output consists of three main parts namely:</p> \n",
    "<ul>\n",
    "    <li>__Sentiment__: This is a word that describes the sentiment expressed in the Text.</li>\n",
    "    <li>__SentimentScore__: This is a dictionary of sentiments wiith their probabilities of represented in the Text.</li>\n",
    "    <li>__ResponseMetadata__: This is the metadata about the call to the comphrehend service.</li>\n",
    "</ul>\n",
    "<p> They are four valid sentiments namely: </p>\n",
    "## POSITIVE | NEGATIVE | NEUTRAL | MIXED\n",
    "\n",
    "<br> You can only have one sentiment per Text</br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> 2) Comprehend for Key Phrase Extraction.</h1>\n",
    "<p> It is simple. Just use the __detect_key_phrases__ method of your client.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the text that you want to get key phrases from\n",
    "txt = \"On the 10th of November 2016, at 6:30 pm, the Police arrested John Doe, the director of Free Health at his home in Madison. For several months now, he has been under investigation. John was accused of diverting company funds into his personal account. He has requested to speak with his lawyer as soon as possible.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "keyPhrases = comprehend.detect_key_phrases(Text = txt, LanguageCode = 'en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'KeyPhrases': [{'Score': 0.9808025360107422,\n",
       "   'Text': 'the 10th',\n",
       "   'BeginOffset': 3,\n",
       "   'EndOffset': 11},\n",
       "  {'Score': 0.9979576468467712,\n",
       "   'Text': 'November 2016',\n",
       "   'BeginOffset': 15,\n",
       "   'EndOffset': 28},\n",
       "  {'Score': 0.9829525351524353,\n",
       "   'Text': '6:30 pm',\n",
       "   'BeginOffset': 33,\n",
       "   'EndOffset': 40},\n",
       "  {'Score': 0.9958406090736389,\n",
       "   'Text': 'the Police',\n",
       "   'BeginOffset': 42,\n",
       "   'EndOffset': 52},\n",
       "  {'Score': 0.9991291761398315,\n",
       "   'Text': 'John Doe',\n",
       "   'BeginOffset': 62,\n",
       "   'EndOffset': 70},\n",
       "  {'Score': 0.999496579170227,\n",
       "   'Text': 'the director',\n",
       "   'BeginOffset': 72,\n",
       "   'EndOffset': 84},\n",
       "  {'Score': 0.9977135062217712,\n",
       "   'Text': 'Free Health',\n",
       "   'BeginOffset': 88,\n",
       "   'EndOffset': 99},\n",
       "  {'Score': 0.9993090629577637,\n",
       "   'Text': 'his home',\n",
       "   'BeginOffset': 103,\n",
       "   'EndOffset': 111},\n",
       "  {'Score': 0.9967743754386902,\n",
       "   'Text': 'Madison',\n",
       "   'BeginOffset': 115,\n",
       "   'EndOffset': 122},\n",
       "  {'Score': 0.9980320930480957,\n",
       "   'Text': 'several months',\n",
       "   'BeginOffset': 128,\n",
       "   'EndOffset': 142},\n",
       "  {'Score': 0.9969323873519897,\n",
       "   'Text': 'investigation',\n",
       "   'BeginOffset': 166,\n",
       "   'EndOffset': 179},\n",
       "  {'Score': 0.9997968077659607,\n",
       "   'Text': 'John',\n",
       "   'BeginOffset': 181,\n",
       "   'EndOffset': 185},\n",
       "  {'Score': 0.9966952800750732,\n",
       "   'Text': 'company\\xa0funds',\n",
       "   'BeginOffset': 211,\n",
       "   'EndOffset': 224},\n",
       "  {'Score': 0.9996809959411621,\n",
       "   'Text': 'his personal account',\n",
       "   'BeginOffset': 230,\n",
       "   'EndOffset': 250},\n",
       "  {'Score': 0.9976174235343933,\n",
       "   'Text': 'his lawyer',\n",
       "   'BeginOffset': 283,\n",
       "   'EndOffset': 293}],\n",
       " 'ResponseMetadata': {'RequestId': 'a2876925-e89e-11e8-90dd-75ee00afd85e',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'date': 'Thu, 15 Nov 2018 06:21:06 GMT',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '1254',\n",
       "   'connection': 'keep-alive',\n",
       "   'x-amzn-requestid': 'a2876925-e89e-11e8-90dd-75ee00afd85e'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keyPhrases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>The key phrases are in jave script object notation jason which in python is the equivallent of nested dictionaries. The output consists of KeyPhrases, and ResponseMetadata. We are only interested in the KeyPhrases. Each KeyPhrase is dictionary of four objects namely:\n",
    "</p>\n",
    "<ul>\n",
    "    <li>__Score__: The Score is measure of the confidence the algorithm has that that key phrase makes sense.</li>\n",
    "    <li>__Text__: This is our subject of interest. This is the text that make up the key phrase that we want.</li>\n",
    "    <li>__BeginOffset__: This is the index of the first character of the key phrase in the input Text.</li>\n",
    "    <li>__EndOffset__: This is the index of the Last character of the key phrase in the input Text.</li>\n",
    "<ul>\n",
    "    \n",
    "    \n",
    "<br> Below, I print the extracted key phrases, one on a line.</br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the 10th\n",
      "November 2016\n",
      "6:30 pm\n",
      "the Police\n",
      "John Doe\n",
      "the director\n",
      "Free Health\n",
      "his home\n",
      "Madison\n",
      "several months\n",
      "investigation\n",
      "John\n",
      "company funds\n",
      "his personal account\n",
      "his lawyer\n"
     ]
    }
   ],
   "source": [
    "for result in keyPhrases['KeyPhrases']:\n",
    "    print(result['Text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> 2) Comprehend for Named Entity Extraction.</h1>\n",
    "<p>It is simple. Just use the detect_named_entities method of your client.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "namedEntities = comprehend.detect_entities(Text = txt, LanguageCode = 'en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Entities': [{'Score': 0.984207272529602,\n",
       "   'Type': 'DATE',\n",
       "   'Text': '10th of November 2016',\n",
       "   'BeginOffset': 7,\n",
       "   'EndOffset': 28},\n",
       "  {'Score': 0.9976122975349426,\n",
       "   'Type': 'DATE',\n",
       "   'Text': '6:30 pm',\n",
       "   'BeginOffset': 33,\n",
       "   'EndOffset': 40},\n",
       "  {'Score': 0.9987821578979492,\n",
       "   'Type': 'PERSON',\n",
       "   'Text': 'John Doe',\n",
       "   'BeginOffset': 62,\n",
       "   'EndOffset': 70},\n",
       "  {'Score': 0.7706741094589233,\n",
       "   'Type': 'ORGANIZATION',\n",
       "   'Text': 'Free Health',\n",
       "   'BeginOffset': 88,\n",
       "   'EndOffset': 99},\n",
       "  {'Score': 0.9977292418479919,\n",
       "   'Type': 'LOCATION',\n",
       "   'Text': 'Madison',\n",
       "   'BeginOffset': 115,\n",
       "   'EndOffset': 122},\n",
       "  {'Score': 0.7375115752220154,\n",
       "   'Type': 'QUANTITY',\n",
       "   'Text': 'several months',\n",
       "   'BeginOffset': 128,\n",
       "   'EndOffset': 142},\n",
       "  {'Score': 0.9996035695075989,\n",
       "   'Type': 'PERSON',\n",
       "   'Text': 'John',\n",
       "   'BeginOffset': 181,\n",
       "   'EndOffset': 185}],\n",
       " 'ResponseMetadata': {'RequestId': 'fcc4feb2-e8a0-11e8-b6e5-5dbbcf180c8c',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'date': 'Thu, 15 Nov 2018 06:37:56 GMT',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '705',\n",
       "   'connection': 'keep-alive',\n",
       "   'x-amzn-requestid': 'fcc4feb2-e8a0-11e8-b6e5-5dbbcf180c8c'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "namedEntities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Just like in Key phrase extraction, the output is a json object. Each named entity consists of 5 parts namely:</p>\n",
    "<ul>\n",
    "    <li>__Score__: The Score is measure of the confidence the algorithm has that that key phrase makes sense.</li>\n",
    "    <li>__Type__: This is the type of named entity that our named entity is. They are 9 valid types.</li>\n",
    "    <li>__Text__: This is our subject of interest. This is the text that make up the key phrase that we want.</li>\n",
    "    <li>__BeginOffset__: This is the index of the first character of the key phrase in the input Text.</li>\n",
    "    <li>__EndOffset__: This is the index of the Last character of the key phrase in the input Text.</li>\n",
    "</ul>\n",
    "    \n",
    "<p>The valid named entity types are:</p>\n",
    "###  PERSON | LOCATION | ORGANIZATION | COMMERCIAL_ITEM | EVENT | DATE | QUANTITY | TITLE | OTHER\n",
    "<br> Below, I print the extracted named entities, and their types one on a line.</br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entity: 10th of November 2016 \t EntityType: DATE\n",
      "Entity: 6:30 pm \t EntityType: DATE\n",
      "Entity: John Doe \t EntityType: PERSON\n",
      "Entity: Free Health \t EntityType: ORGANIZATION\n",
      "Entity: Madison \t EntityType: LOCATION\n",
      "Entity: several months \t EntityType: QUANTITY\n",
      "Entity: John \t EntityType: PERSON\n"
     ]
    }
   ],
   "source": [
    "for result in namedEntities['Entities']:\n",
    "    print('Entity: {} \\t EntityType: {}'.format(result['Text'], result['Type']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> 3) Comprehend for Language Detection.</h1>\n",
    "<p>It is simple. Just use the __detect_dominant_language__ method of your client.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang = comprehend.detect_dominant_language(Text = txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Languages': [{'LanguageCode': 'en', 'Score': 0.9971848130226135}],\n",
       " 'ResponseMetadata': {'RequestId': 'f7b33f6a-e8a4-11e8-90dd-75ee00afd85e',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'date': 'Thu, 15 Nov 2018 07:06:26 GMT',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '64',\n",
       "   'connection': 'keep-alive',\n",
       "   'x-amzn-requestid': 'f7b33f6a-e8a4-11e8-90dd-75ee00afd85e'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lang ### Can you interprete the output in the cell above? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only languages supported by comprehend at this time are\n",
    "'en'|'es'|'fr'|'de'|'it'|'pt' for 'English'|'Spanish'|'French'|'German'|'Italian'|'Portuguese' respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> 4) Comprehend for Medical Entity detection.</h1>\n",
    "<p>It is simple. Just use the __detect_named_entities__ method of your client. Keep in mind that the `service_name` for comprehend medical is __comprehendmedical__.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "comprehend_medical = boto3.client(service_name = 'comprehendmedical', region_name = 'us-east-1', \n",
    "                                  aws_access_key_id = 'YourAccessKeyGoesHere', aws_secret_access_key = 'YourSecretKeyGoesHere')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt = '''A 62yo male states onset of chest pain with dizziness approximately 15 minutes before calling 9-1-1. Patient states he was mowing the yard when he had the onset of chest pain. Patient indicates no pain on palpation during examination of sternal and chest areas. Patient states chest pain radiates to right arm and denies difficulty breathing at this time. Patient’s skin is cool, pale and moist to touch. Pulse is equal on both wrists at 85, and is irregular.'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Entities': [{'Id': 0,\n",
       "   'BeginOffset': 2,\n",
       "   'EndOffset': 6,\n",
       "   'Score': 0.9998487234115601,\n",
       "   'Text': '62yo',\n",
       "   'Category': 'PROTECTED_HEALTH_INFORMATION',\n",
       "   'Type': 'AGE',\n",
       "   'Traits': []},\n",
       "  {'Id': 11,\n",
       "   'BeginOffset': 28,\n",
       "   'EndOffset': 33,\n",
       "   'Score': 0.9947172999382019,\n",
       "   'Text': 'chest',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'SYSTEM_ORGAN_SITE',\n",
       "   'Traits': []},\n",
       "  {'Id': 1,\n",
       "   'BeginOffset': 28,\n",
       "   'EndOffset': 38,\n",
       "   'Score': 0.95955890417099,\n",
       "   'Text': 'chest pain',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': [{'Name': 'SYMPTOM', 'Score': 0.8515509366989136}]},\n",
       "  {'Id': 2,\n",
       "   'BeginOffset': 44,\n",
       "   'EndOffset': 53,\n",
       "   'Score': 0.9893583059310913,\n",
       "   'Text': 'dizziness',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': [{'Name': 'SYMPTOM', 'Score': 0.8791248798370361}]},\n",
       "  {'Id': 12,\n",
       "   'BeginOffset': 164,\n",
       "   'EndOffset': 169,\n",
       "   'Score': 0.9959815740585327,\n",
       "   'Text': 'chest',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'SYSTEM_ORGAN_SITE',\n",
       "   'Traits': []},\n",
       "  {'Id': 3,\n",
       "   'BeginOffset': 164,\n",
       "   'EndOffset': 174,\n",
       "   'Score': 0.9412338137626648,\n",
       "   'Text': 'chest pain',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': [{'Name': 'SYMPTOM', 'Score': 0.9135958552360535}]},\n",
       "  {'Id': 4,\n",
       "   'BeginOffset': 197,\n",
       "   'EndOffset': 201,\n",
       "   'Score': 0.8807910084724426,\n",
       "   'Text': 'pain',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': [{'Name': 'SYMPTOM', 'Score': 0.8446396589279175},\n",
       "    {'Name': 'NEGATION', 'Score': 0.786529004573822}]},\n",
       "  {'Id': 13,\n",
       "   'BeginOffset': 237,\n",
       "   'EndOffset': 244,\n",
       "   'Score': 0.2318798452615738,\n",
       "   'Text': 'sternal',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'SYSTEM_ORGAN_SITE',\n",
       "   'Traits': []},\n",
       "  {'Id': 14,\n",
       "   'BeginOffset': 249,\n",
       "   'EndOffset': 260,\n",
       "   'Score': 0.6070584654808044,\n",
       "   'Text': 'chest areas',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'SYSTEM_ORGAN_SITE',\n",
       "   'Traits': []},\n",
       "  {'Id': 15,\n",
       "   'BeginOffset': 277,\n",
       "   'EndOffset': 282,\n",
       "   'Score': 0.9893577694892883,\n",
       "   'Text': 'chest',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'SYSTEM_ORGAN_SITE',\n",
       "   'Traits': []},\n",
       "  {'Id': 5,\n",
       "   'BeginOffset': 277,\n",
       "   'EndOffset': 287,\n",
       "   'Score': 0.8794683814048767,\n",
       "   'Text': 'chest pain',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': [{'Name': 'SYMPTOM', 'Score': 0.8732218146324158}]},\n",
       "  {'Id': 16,\n",
       "   'BeginOffset': 300,\n",
       "   'EndOffset': 305,\n",
       "   'Score': 0.9969936609268188,\n",
       "   'Text': 'right',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'DIRECTION',\n",
       "   'Traits': []},\n",
       "  {'Id': 17,\n",
       "   'BeginOffset': 306,\n",
       "   'EndOffset': 309,\n",
       "   'Score': 0.9933364391326904,\n",
       "   'Text': 'arm',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'SYSTEM_ORGAN_SITE',\n",
       "   'Traits': []},\n",
       "  {'Id': 6,\n",
       "   'BeginOffset': 321,\n",
       "   'EndOffset': 341,\n",
       "   'Score': 0.973494827747345,\n",
       "   'Text': 'difficulty breathing',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': [{'Name': 'SYMPTOM', 'Score': 0.9269325733184814},\n",
       "    {'Name': 'NEGATION', 'Score': 0.8731715679168701}]},\n",
       "  {'Id': 18,\n",
       "   'BeginOffset': 366,\n",
       "   'EndOffset': 370,\n",
       "   'Score': 0.9815253019332886,\n",
       "   'Text': 'skin',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'SYSTEM_ORGAN_SITE',\n",
       "   'Traits': []},\n",
       "  {'Id': 7,\n",
       "   'BeginOffset': 374,\n",
       "   'EndOffset': 378,\n",
       "   'Score': 0.6224351525306702,\n",
       "   'Text': 'cool',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': []},\n",
       "  {'Id': 8,\n",
       "   'BeginOffset': 380,\n",
       "   'EndOffset': 384,\n",
       "   'Score': 0.7666590213775635,\n",
       "   'Text': 'pale',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': []},\n",
       "  {'Id': 9,\n",
       "   'BeginOffset': 389,\n",
       "   'EndOffset': 403,\n",
       "   'Score': 0.8116811513900757,\n",
       "   'Text': 'moist to touch',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': []},\n",
       "  {'Id': 10,\n",
       "   'BeginOffset': 405,\n",
       "   'EndOffset': 419,\n",
       "   'Score': 0.5948115587234497,\n",
       "   'Text': 'Pulse is equal',\n",
       "   'Category': 'MEDICAL_CONDITION',\n",
       "   'Type': 'DX_NAME',\n",
       "   'Traits': [{'Name': 'SIGN', 'Score': 0.724435031414032}]},\n",
       "  {'Id': 19,\n",
       "   'BeginOffset': 423,\n",
       "   'EndOffset': 427,\n",
       "   'Score': 0.8222383260726929,\n",
       "   'Text': 'both',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'DIRECTION',\n",
       "   'Traits': []},\n",
       "  {'Id': 20,\n",
       "   'BeginOffset': 428,\n",
       "   'EndOffset': 434,\n",
       "   'Score': 0.9955358505249023,\n",
       "   'Text': 'wrists',\n",
       "   'Category': 'ANATOMY',\n",
       "   'Type': 'SYSTEM_ORGAN_SITE',\n",
       "   'Traits': []}],\n",
       " 'UnmappedAttributes': [],\n",
       " 'ResponseMetadata': {'RequestId': 'ba71605d-fbfd-11e8-a32c-31a9e264d6da',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n",
       "   'date': 'Sun, 09 Dec 2018 21:59:41 GMT',\n",
       "   'x-amzn-requestid': 'ba71605d-fbfd-11e8-a32c-31a9e264d6da',\n",
       "   'content-length': '3529',\n",
       "   'connection': 'keep-alive'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "medical_entities = comprehend_medical.detect_entities(Text = txt)\n",
    "medical_entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>As you can see, the output is very similar to that frome Named entity extraction and each named entity such as:</p>\n",
    "<ul>\n",
    "<li>__Entity:__ A textual reference to the unique name of a real-world object such as people, treatments, medications, and medical conditions, and to precise references to measures, such as dates and dosage. For example, \"Ibuprofen.\"</li>\n",
    "\n",
    "<li>__Category:__ The generalized grouping to which an detected entity belongs, for ease of understanding. For example, \"Ibuprofen\" is part of the MEDICATION category.</li>\n",
    "\n",
    "<li>__Type:__ The type of entity detected, scoped to a category. For example, \"Ibuprofen\" is of the GENERIC_NAME type of entity.</li>\n",
    "\n",
    "<li>__Attribute:__ Relevant information related to an detected entity, as in dosage is an attribute of a medication. For example \"200mg\" is an attribute of the \"Ibuprofen\" entity.</li>\n",
    "\n",
    "<li>__Trait:__ Something we understand about an entity, based on context. For instance, a medication is negated (NEGATION trait) if a patient is not taking it.</li>\n",
    "</ul>\n",
    "    \n",
    "<p>Checkout the valid __Categories__ and __attributes__ on __[this page](https://docs.aws.amazon.com/comprehend/latest/dg/extracted-med-info.html).__:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would be nice to see the output in a `pandas dataframe.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedicalEntity</th>\n",
       "      <th>Catetory</th>\n",
       "      <th>Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>62yo</td>\n",
       "      <td>PROTECTED_HEALTH_INFORMATION</td>\n",
       "      <td>AGE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chest</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>SYSTEM_ORGAN_SITE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>chest pain</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dizziness</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>chest</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>SYSTEM_ORGAN_SITE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>chest pain</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>pain</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>sternal</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>SYSTEM_ORGAN_SITE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>chest areas</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>SYSTEM_ORGAN_SITE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>chest</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>SYSTEM_ORGAN_SITE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>chest pain</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>right</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>DIRECTION</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>arm</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>SYSTEM_ORGAN_SITE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>difficulty breathing</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>skin</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>SYSTEM_ORGAN_SITE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>cool</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>pale</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>moist to touch</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Pulse is equal</td>\n",
       "      <td>MEDICAL_CONDITION</td>\n",
       "      <td>DX_NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>both</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>DIRECTION</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>wrists</td>\n",
       "      <td>ANATOMY</td>\n",
       "      <td>SYSTEM_ORGAN_SITE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           MedicalEntity                      Catetory               Type\n",
       "0                   62yo  PROTECTED_HEALTH_INFORMATION                AGE\n",
       "1                  chest                       ANATOMY  SYSTEM_ORGAN_SITE\n",
       "2             chest pain             MEDICAL_CONDITION            DX_NAME\n",
       "3              dizziness             MEDICAL_CONDITION            DX_NAME\n",
       "4                  chest                       ANATOMY  SYSTEM_ORGAN_SITE\n",
       "5             chest pain             MEDICAL_CONDITION            DX_NAME\n",
       "6                   pain             MEDICAL_CONDITION            DX_NAME\n",
       "7                sternal                       ANATOMY  SYSTEM_ORGAN_SITE\n",
       "8            chest areas                       ANATOMY  SYSTEM_ORGAN_SITE\n",
       "9                  chest                       ANATOMY  SYSTEM_ORGAN_SITE\n",
       "10            chest pain             MEDICAL_CONDITION            DX_NAME\n",
       "11                 right                       ANATOMY          DIRECTION\n",
       "12                   arm                       ANATOMY  SYSTEM_ORGAN_SITE\n",
       "13  difficulty breathing             MEDICAL_CONDITION            DX_NAME\n",
       "14                  skin                       ANATOMY  SYSTEM_ORGAN_SITE\n",
       "15                  cool             MEDICAL_CONDITION            DX_NAME\n",
       "16                  pale             MEDICAL_CONDITION            DX_NAME\n",
       "17        moist to touch             MEDICAL_CONDITION            DX_NAME\n",
       "18        Pulse is equal             MEDICAL_CONDITION            DX_NAME\n",
       "19                  both                       ANATOMY          DIRECTION\n",
       "20                wrists                       ANATOMY  SYSTEM_ORGAN_SITE"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame([(ent['Text'], ent['Category'], ent['Type']) for ent in medical_entities['Entities']], columns = 'MedicalEntity Catetory Type'.split())\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get all the __medical_entities__ for for our text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pain | pale | difficulty breathing | right | 62yo | chest | wrists | arm | chest areas | skin | dizziness | both | cool | Pulse is equal | moist to touch | chest pain | sternal'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' | '.join(set([ent['Text'] for ent in medical_entities['Entities']]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> RESOURCES </h1>\n",
    "<ul>\n",
    "    <li>__Comprehed [Sentiment Analysis](https://docs.aws.amazon.com/comprehend/latest/dg/how-sentiment.html) How to Page.__</li>\n",
    "    <li>__Comprehed [Key Phrase Extraction](https://docs.aws.amazon.com/comprehend/latest/dg/get-started-api-key-phrases.html) How to Page.__</li>\n",
    "    <li>__Comprehed [Named Entities Extraction](https://docs.aws.amazon.com/comprehend/latest/dg/API_Entity.html) How to Page.__</li>\n",
    "    <li>__Comprehed [Dominant Language Detection](https://docs.aws.amazon.com/comprehend/latest/dg/how-languages.html) How to Page.__</li>\n",
    "    <li>__Comprehed [Custom Classification](https://docs.aws.amazon.com/comprehend/latest/dg/how-document-classification.html) How to Page.__</li>\n",
    "    <li>__Comprehed [Custom Named Entity Rrecognition](https://docs.aws.amazon.com/comprehend/latest/dg/custom-entity-recognition.html) How to Page.__</li>\n",
    "    <li>__Comprehed [Medical](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/comprehendmedical.html) How to Page.__</li>\n",
    "    <li>__My Tutorial - [The Power of Comprehend](https://www.sammywealth.com/the-power-of-comprehend)__.</li>\n",
    "    \n",
    "</ul>\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}