samples/extraction/text-based/document-intelligence-openai.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Extraction - Azure AI Document Intelligence + Azure OpenAI GPT-4o\n",
    "\n",
    "This sample demonstrates how to extract structured data from any document using Azure AI Document Intelligence and Azure OpenAI GPT models.\n",
    "\n",
    "![Data Extraction](../../../images/extraction-document-intelligence-openai.png)\n",
    "\n",
    "This is achieved by the following process:\n",
    "\n",
    "- Analyze a document using Azure AI Document Intelligence's `prebuilt-layout` model to extract the structure as Markdown.\n",
    "- Construct a system prompt that defines the instruction for extracting structured data from documents.\n",
    "- Construct a user prompt that includes specific extraction instruction for the type of document, and the Markdown content of the document.\n",
    "- Use the Azure OpenAI chat completions API with the GPT-4o model to generate a structured output from the content.\n",
    "\n",
    "## Objectives\n",
    "\n",
    "By the end of this sample, you will have learned how to:\n",
    "\n",
    "- Convert a document to Markdown format using Azure AI Document Intelligence.\n",
    "- Use prompt engineering techniques to instruct GPT-4o to extract structured data from a type of document.\n",
    "- Use the [Structured Outputs feature](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure) to extract structured data from a document using Azure OpenAI's GPT-4o model.\n",
    "- Use the analysis result from Azure AI Document Intelligence to determine the confidence of the extracted structured output.\n",
    "- Use the [logprobs](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#request-body:~:text=False-,logprobs,-integer) parameter in an OpenAI request to determine the confidence of the extracted structured output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import modules\n",
    "\n",
    "This sample takes advantage of the following Python dependencies:\n",
    "\n",
    "- **azure-ai-documentintelligence** to interface with the Azure AI Document Intelligence API for analyzing documents.\n",
    "- **openai** to interface with the Azure OpenAI chat completions API to generate structured extraction outputs using the GPT-4o model.\n",
    "- **azure-identity** to securely authenticate with deployed Azure Services using Microsoft Entra ID credentials.\n",
    "\n",
    "The following local modules are also used:\n",
    "\n",
    "- **modules.app_settings** to access environment variables from the `.env` file.\n",
    "- **modules.comparison** to compare the output of the extraction process with expected results.\n",
    "- **modules.document_intelligence_confidence** to evaluate the confidence of the extraction process based on the extracted structured output and the analysis result from Azure AI Document Intelligence.\n",
    "- **modules.document_processing_result** to store the results of the extraction process as a file.\n",
    "- **modules.openai_confidence** to calculate the confidence of the classification process based on the `logprobs` response from the API request.\n",
    "- **modules.invoice** to provide the expected structured output JSON schema for invoice documents.\n",
    "- **modules.utils** `Stopwatch` to measure the end-to-end execution time for the classification process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../../') # Import local modules\n",
    "\n",
    "from IPython.display import display, Markdown\n",
    "import os\n",
    "import pandas as pd\n",
    "import json\n",
    "from dotenv import dotenv_values\n",
    "from azure.ai.documentintelligence import DocumentIntelligenceClient\n",
    "from azure.ai.documentintelligence.models import AnalyzeResult, ContentFormat\n",
    "from openai import AzureOpenAI\n",
    "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
    "import json\n",
    "\n",
    "from modules.app_settings import AppSettings\n",
    "from modules.utils import Stopwatch\n",
    "from modules.accuracy_evaluator import AccuracyEvaluator\n",
    "from modules.comparison import get_extraction_comparison\n",
    "from modules.confidence import merge_confidence_values\n",
    "from modules.document_intelligence_confidence import evaluate_confidence as di_evaluate_confidence\n",
    "from modules.openai_confidence import evaluate_confidence as oai_evaluate_confidence\n",
    "from modules.invoice import Invoice\n",
    "from modules.document_processing_result import DataExtractionResult"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the Azure services\n",
    "\n",
    "To use Azure AI Document Intelligence and Azure OpenAI, their SDKs are used to create client instances using a deployed endpoint and authentication credentials.\n",
    "\n",
    "For this sample, the credentials of the Azure CLI are used to authenticate with the deployed services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the working directory to the root of the repo\n",
    "working_dir = os.path.abspath('../../../')\n",
    "settings = AppSettings(dotenv_values(f\"{working_dir}/.env\"))\n",
    "\n",
    "# Configure the default credential for accessing Azure services using Azure CLI credentials\n",
    "credential = DefaultAzureCredential(\n",
    "    exclude_workload_identity_credential=True,\n",
    "    exclude_developer_cli_credential=True,\n",
    "    exclude_environment_credential=True,\n",
    "    exclude_managed_identity_credential=True,\n",
    "    exclude_powershell_credential=True,\n",
    "    exclude_shared_token_cache_credential=True,\n",
    "    exclude_interactive_browser_credential=True\n",
    ")\n",
    "\n",
    "openai_token_provider = get_bearer_token_provider(credential, 'https://cognitiveservices.azure.com/.default')\n",
    "\n",
    "openai_client = AzureOpenAI(\n",
    "    azure_endpoint=settings.openai_endpoint,\n",
    "    azure_ad_token_provider=openai_token_provider,\n",
    "    api_version=\"2024-10-01-preview\" # Requires the latest API version for structured outputs.\n",
    ")\n",
    "\n",
    "document_intelligence_client = DocumentIntelligenceClient(\n",
    "    endpoint=settings.ai_services_endpoint,\n",
    "    credential=credential\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Establish the expected output\n",
    "\n",
    "To compare the accuracy of the extraction process, the expected output of the extraction process has been defined in the following code block based on the details of an [Invoice](../../assets/invoices/invoice_1.pdf).\n",
    "\n",
    "> **Note**: More invoice examples can be found in the [assets folder](../../assets/invoices). These examples include the PDF file and an associated JSON metadata file that provides the expected structured output. You can add your own scenarios by following the same structure.\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"fname\": \"<name of the invoice file>\",\n",
    "    \"expected\": {\n",
    "        \"customer_name\": \"\",\n",
    "        \"customer_address\": {\n",
    "            \"street\": \"\",\n",
    "            \"city\": \"\",\n",
    "            \"state\": \"\",\n",
    "            \"postal_code\": \"\",\n",
    "            \"country\": \"\"\n",
    "        },\n",
    "        \"customer_tax_id\": \"\",\n",
    "        \"shipping_address\": \"\",\n",
    "        \"purchase_order\": \"\",\n",
    "        \"invoice_id\": \"\",\n",
    "        \"invoice_date\": \"\",\n",
    "        \"payable_by\": \"\",\n",
    "        \"vendor_name\": \"\",\n",
    "        \"vendor_address\": \"\",\n",
    "        \"vendor_tax_id\": \"\",\n",
    "        \"remittance_address\": \"\",\n",
    "        \"subtotal\": 0,\n",
    "        \"total_discount\": 0,\n",
    "        \"total_tax\": 0,\n",
    "        \"invoice_total\": 0,\n",
    "        \"payment_terms\": \"\",\n",
    "        \"items\": [\n",
    "            {\n",
    "                \"product_code\": \"\",\n",
    "                \"description\": \"\",\n",
    "                \"quantity\": 0,\n",
    "                \"tax\": 0,\n",
    "                \"tax_rate\": \"\",\n",
    "                \"unit_price\": 0,\n",
    "                \"total\": 0,\n",
    "                \"reason\": null\n",
    "            }\n",
    "        ],\n",
    "        \"total_item_quantity\": 0,\n",
    "        \"items_customer_signature\": {\n",
    "            \"signatory\": \"\",\n",
    "            \"is_signed\": true\n",
    "        },\n",
    "        \"items_vendor_signature\": {\n",
    "            \"signatory\": \"\",\n",
    "            \"is_signed\": true\n",
    "        },\n",
    "        \"returns\": [\n",
    "            {\n",
    "                \"product_code\": \"\",\n",
    "                \"description\": \"\",\n",
    "                \"quantity\": 0,\n",
    "                \"tax\": null,\n",
    "                \"tax_rate\": null,\n",
    "                \"unit_price\": null,\n",
    "                \"total\": null,\n",
    "                \"reason\": \"\"\n",
    "            }\n",
    "        ],\n",
    "        \"total_return_quantity\": 0,\n",
    "        \"returns_customer_signature\": {\n",
    "            \"signatory\": \"\",\n",
    "            \"is_signed\": true\n",
    "        },\n",
    "        \"returns_vendor_signature\": {\n",
    "            \"signatory\": \"\",\n",
    "            \"is_signed\": true\n",
    "        }\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "The expected output has been defined by a human evaluating the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = f\"{working_dir}/samples/assets/invoices/\"\n",
    "metadata_fname = \"invoice_1.json\" # Change this to the file you want to evaluate\n",
    "metadata_fpath = f\"{path}{metadata_fname}\"\n",
    "\n",
    "with open(metadata_fpath, \"r\") as f:\n",
    "    data = json.load(f)\n",
    "    \n",
    "expected = Invoice(**data['expected'])\n",
    "pdf_fname = data['fname']\n",
    "pdf_fpath = f\"{path}{pdf_fname}\"\n",
    "\n",
    "invoice_evaluator = AccuracyEvaluator(match_keys=['product_code', 'description'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract data from the document\n",
    "\n",
    "The following code block executes the data extraction process using Azure AI Document Intelligence and Azure OpenAI's GPT-4o model.\n",
    "\n",
    "It performs the following steps:\n",
    "\n",
    "1. Get the document bytes from the provided file path. _Note: In this example, we are processing a local document, however, you can use any document storage location of your choice, such as Azure Blob Storage._\n",
    "2. Use Azure AI Document Intelligence to analyze the structure of the document and convert it to Markdown format using the pre-built layout model.\n",
    "3. Using Azure OpenAI's GPT-4o model and its [Structured Outputs feature](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure), extract a structured data transfer object (DTO) from the content of the Markdown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Stopwatch() as di_stopwatch:\n",
    "    with open(pdf_fpath, \"rb\") as f:\n",
    "        poller = document_intelligence_client.begin_analyze_document(\n",
    "            \"prebuilt-layout\",\n",
    "            analyze_request=f,\n",
    "            output_content_format=ContentFormat.MARKDOWN,\n",
    "            content_type=\"application/pdf\"\n",
    "        )\n",
    "    \n",
    "    result: AnalyzeResult = poller.result()\n",
    "\n",
    "markdown = result.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Stopwatch() as oai_stopwatch:\n",
    "    completion = openai_client.beta.chat.completions.parse(\n",
    "        model=settings.gpt4o_model_deployment_name,\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": \"You are an AI assistant that extracts data from documents.\",\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": f\"\"\"Extract the data from this invoice. \n",
    "                - If a value is not present, provide null.\n",
    "                - Dates should be in the format YYYY-MM-DD.\"\"\",\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": markdown,\n",
    "            }\n",
    "        ],\n",
    "        response_format=Invoice,\n",
    "        max_tokens=4096,\n",
    "        temperature=0.1,\n",
    "        top_p=0.1,\n",
    "        logprobs=True # Enabled to determine the confidence of the response.\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the Structured Outputs JSON schema\n",
    "\n",
    "Using [Pydantic's JSON schema feature](https://docs.pydantic.dev/latest/concepts/json_schema/), the [Invoice](../../modules/invoice.py) data model is automatically converted to a JSON schema when applied to the `response_format` parameter of the OpenAI chat completions request.\n",
    "\n",
    "The JSON schema is used to instruct the GPT-4o model to generate a strict output that adheres to the structure defined. The approach using Pydantic makes it easier for developers to manage the data structure in code, with helpful descriptions and examples that will be included in the final JSON schema.\n",
    "\n",
    "Demonstrated below, you can see how the Invoice data model is understood by the OpenAI request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"$defs\": {\n",
      "    \"InvoiceAddress\": {\n",
      "      \"description\": \"A class representing an address in an invoice.\\n\\nAttributes:\\n    street: Street address\\n    city: City, e.g. New York\\n    state: State, e.g. NY\\n    postal_code: Postal code, e.g. 10001\\n    country: Country, e.g. USA\",\n",
      "      \"properties\": {\n",
      "        \"street\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Street address, e.g. 123 Main St.\",\n",
      "          \"title\": \"Street\"\n",
      "        },\n",
      "        \"city\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"City, e.g. New York\",\n",
      "          \"title\": \"City\"\n",
      "        },\n",
      "        \"state\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"State, e.g. NY\",\n",
      "          \"title\": \"State\"\n",
      "        },\n",
      "        \"postal_code\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Postal code, e.g. 10001\",\n",
      "          \"title\": \"Postal Code\"\n",
      "        },\n",
      "        \"country\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Country, e.g. USA\",\n",
      "          \"title\": \"Country\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"street\",\n",
      "        \"city\",\n",
      "        \"state\",\n",
      "        \"postal_code\",\n",
      "        \"country\"\n",
      "      ],\n",
      "      \"title\": \"InvoiceAddress\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"InvoiceItem\": {\n",
      "      \"description\": \"A class representing a line item in an invoice.\\n\\nAttributes:\\n    product_code: Product code, product number, or SKU associated with the line item.\\n    description: Description of the line item.\\n    quantity: Quantity of the line item.\\n    tax: Tax amount applied to the line item.\\n    tax_rate: Tax rate applied to the line item.\\n    unit_price: Net or gross price of one unit of the line item.\\n    total: The total charges associated with the line item.\\n    reason: Reason for returning the line item.\",\n",
      "      \"properties\": {\n",
      "        \"product_code\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Product code, product number, or SKU associated with the line item, e.g. 12345\",\n",
      "          \"title\": \"Product Code\"\n",
      "        },\n",
      "        \"description\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Description of the line item, e.g. Product A\",\n",
      "          \"title\": \"Description\"\n",
      "        },\n",
      "        \"quantity\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Quantity of the line item\",\n",
      "          \"title\": \"Quantity\"\n",
      "        },\n",
      "        \"tax\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"number\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Tax amount applied to the line item, e.g. 6.00\",\n",
      "          \"title\": \"Tax\"\n",
      "        },\n",
      "        \"tax_rate\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Tax rate applied to the line item, e.g. 18%\",\n",
      "          \"title\": \"Tax Rate\"\n",
      "        },\n",
      "        \"unit_price\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"number\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Net or gross price of one unit of the line item, e.g. 10.00\",\n",
      "          \"title\": \"Unit Price\"\n",
      "        },\n",
      "        \"total\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"number\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The total charges associated with the line item, e.g. 100.00\",\n",
      "          \"title\": \"Total\"\n",
      "        },\n",
      "        \"reason\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Reason for returning the line item, e.g. Damaged\",\n",
      "          \"title\": \"Reason\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"product_code\",\n",
      "        \"description\",\n",
      "        \"quantity\",\n",
      "        \"tax\",\n",
      "        \"tax_rate\",\n",
      "        \"unit_price\",\n",
      "        \"total\",\n",
      "        \"reason\"\n",
      "      ],\n",
      "      \"title\": \"InvoiceItem\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"InvoiceSignature\": {\n",
      "      \"description\": \"A class representing a signature for an invoice.\\n\\nAttributes:\\n    signatory: Name of the person who signed the invoice.\\n    is_signed: Indicates if the invoice is signed.\",\n",
      "      \"properties\": {\n",
      "        \"signatory\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Name of the person who signed the invoice\",\n",
      "          \"title\": \"Signatory\"\n",
      "        },\n",
      "        \"is_signed\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"boolean\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"Indicates if the invoice is signed\",\n",
      "          \"title\": \"Is Signed\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"signatory\",\n",
      "        \"is_signed\"\n",
      "      ],\n",
      "      \"title\": \"InvoiceSignature\",\n",
      "      \"type\": \"object\"\n",
      "    }\n",
      "  },\n",
      "  \"description\": \"A class representing an invoice.\\n\\nAttributes:\\n    customer_name: Name of the customer being invoiced.\\n    customer_address: Full address of the customer.\\n    customer_tax_id: Government tax ID of the customer.\\n    shipping_address: Full address of the shipping location for the customer.\\n    purchase_order: Purchase order reference number.\\n    invoice_id: Reference ID for the invoice.\\n    invoice_date: Date the invoice was issued.\\n    payable_by: Date when the invoice should be paid.\\n    vendor_name: Name of the vendor who created the invoice.\\n    vendor_address: Full address of the vendor.\\n    vendor_tax_id: Government tax ID of the vendor.\\n    remittance_address: Full address where the payment should be sent.\\n    subtotal: Subtotal of the invoice.\\n    total_discount: Total discount applied to the invoice.\\n    total_tax: Total tax applied to the invoice.\\n    invoice_total: Total charges associated with the invoice.\\n    payment_terms: Payment terms for the invoice.\\n    items: List of line items in the invoice.\\n    total_item_quantity: Total quantity of items in the invoice.\\n    items_customer_signature: Signature of the customer for the items in the invoice.\\n    items_vendor_signature: Signature of the vendor for the items in the invoice.\\n    returns: List of line items returned in the invoice.\\n    total_return_quantity: Total quantity of items returned in the invoice.\\n    returns_customer_signature: Signature of the customer for the returned items in the invoice.\\n    returns_vendor_signature: Signature of the vendor for the returned items in the invoice.\",\n",
      "  \"properties\": {\n",
      "    \"customer_name\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Name of the customer being invoiced, e.g. Company A\",\n",
      "      \"title\": \"Customer Name\"\n",
      "    },\n",
      "    \"customer_address\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/InvoiceAddress\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Full address of the customer, e.g. 123 Main St., City, Country\"\n",
      "    },\n",
      "    \"customer_tax_id\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Government tax ID of the customer, e.g. 123456789\",\n",
      "      \"title\": \"Customer Tax Id\"\n",
      "    },\n",
      "    \"shipping_address\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/InvoiceAddress\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Full address of the shipping location for the customer (null if the same as customer address), e.g. 123 Main St., City, Country\"\n",
      "    },\n",
      "    \"purchase_order\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Purchase order reference number, e.g. PO-1234\",\n",
      "      \"title\": \"Purchase Order\"\n",
      "    },\n",
      "    \"invoice_id\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Reference ID for the invoice (often invoice number), e.g. INV-1234\",\n",
      "      \"title\": \"Invoice Id\"\n",
      "    },\n",
      "    \"invoice_date\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Date the invoice was issued or delivered, e.g., 2021-01-01\",\n",
      "      \"title\": \"Invoice Date\"\n",
      "    },\n",
      "    \"payable_by\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Date when the invoice should be paid, e.g., 2021-01-15\",\n",
      "      \"title\": \"Payable By\"\n",
      "    },\n",
      "    \"vendor_name\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Name of the vendor who created the invoice, e.g. Company B\",\n",
      "      \"title\": \"Vendor Name\"\n",
      "    },\n",
      "    \"vendor_address\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/InvoiceAddress\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Full address of the vendor, e.g. 321 Main St., City, Country\"\n",
      "    },\n",
      "    \"vendor_tax_id\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Government tax ID of the vendor, e.g. 123456789\",\n",
      "      \"title\": \"Vendor Tax Id\"\n",
      "    },\n",
      "    \"remittance_address\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/InvoiceAddress\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Full address where the payment should be sent (null if the same as vendor address), e.g. 321 Main St., City, Country\"\n",
      "    },\n",
      "    \"subtotal\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"number\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Subtotal of the invoice, e.g. 100.00\",\n",
      "      \"title\": \"Subtotal\"\n",
      "    },\n",
      "    \"total_discount\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"number\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Total discount applied to the invoice, e.g. 10.00\",\n",
      "      \"title\": \"Total Discount\"\n",
      "    },\n",
      "    \"total_tax\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"number\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Total tax applied to the invoice, e.g. 5.00\",\n",
      "      \"title\": \"Total Tax\"\n",
      "    },\n",
      "    \"invoice_total\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"number\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Total charges associated with the invoice, e.g. 95.00\",\n",
      "      \"title\": \"Invoice Total\"\n",
      "    },\n",
      "    \"payment_terms\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Payment terms for the invoice, e.g. Net 90\",\n",
      "      \"title\": \"Payment Terms\"\n",
      "    },\n",
      "    \"items\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"items\": {\n",
      "            \"$ref\": \"#/$defs/InvoiceItem\"\n",
      "          },\n",
      "          \"type\": \"array\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"List of line items in the invoice\",\n",
      "      \"title\": \"Items\"\n",
      "    },\n",
      "    \"total_item_quantity\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"number\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Total quantity of items in the invoice\",\n",
      "      \"title\": \"Total Item Quantity\"\n",
      "    },\n",
      "    \"items_customer_signature\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/InvoiceSignature\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Signature of the customer for the items in the invoice\"\n",
      "    },\n",
      "    \"items_vendor_signature\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/InvoiceSignature\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Signature of the vendor for the items in the invoice\"\n",
      "    },\n",
      "    \"returns\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"items\": {\n",
      "            \"$ref\": \"#/$defs/InvoiceItem\"\n",
      "          },\n",
      "          \"type\": \"array\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"List of line items returned in the invoice\",\n",
      "      \"title\": \"Returns\"\n",
      "    },\n",
      "    \"total_return_quantity\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"number\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Total quantity of items returned in the invoice\",\n",
      "      \"title\": \"Total Return Quantity\"\n",
      "    },\n",
      "    \"returns_customer_signature\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/InvoiceSignature\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Signature of the customer for the returned items in the invoice\"\n",
      "    },\n",
      "    \"returns_vendor_signature\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/InvoiceSignature\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"Signature of the vendor for the returned items in the invoice\"\n",
      "    }\n",
      "  },\n",
      "  \"required\": [\n",
      "    \"customer_name\",\n",
      "    \"customer_address\",\n",
      "    \"customer_tax_id\",\n",
      "    \"shipping_address\",\n",
      "    \"purchase_order\",\n",
      "    \"invoice_id\",\n",
      "    \"invoice_date\",\n",
      "    \"payable_by\",\n",
      "    \"vendor_name\",\n",
      "    \"vendor_address\",\n",
      "    \"vendor_tax_id\",\n",
      "    \"remittance_address\",\n",
      "    \"subtotal\",\n",
      "    \"total_discount\",\n",
      "    \"total_tax\",\n",
      "    \"invoice_total\",\n",
      "    \"payment_terms\",\n",
      "    \"items\",\n",
      "    \"total_item_quantity\",\n",
      "    \"items_customer_signature\",\n",
      "    \"items_vendor_signature\",\n",
      "    \"returns\",\n",
      "    \"total_return_quantity\",\n",
      "    \"returns_customer_signature\",\n",
      "    \"returns_vendor_signature\"\n",
      "  ],\n",
      "  \"title\": \"Invoice\",\n",
      "  \"type\": \"object\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Highlight the schema sent to the OpenAI model\n",
    "print(json.dumps(Invoice.model_json_schema(), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the outputs\n",
    "\n",
    "To provide context for the execution of the code, the following code blocks visualize the outputs of the data extraction process.\n",
    "\n",
    "This includes:\n",
    "\n",
    "- The Markdown representation of the document structure as determined by Azure AI Document Intelligence.\n",
    "- The accuracy of the structured data extraction comparing the expected output with the output generated by Azure OpenAI's GPT-4o model.\n",
    "- The confidence score of the structured data extraction by comparing against the Azure AI Document Intelligence analysis.\n",
    "- The execution time of the end-to-end process.\n",
    "- The total number of tokens consumed by the GPT-4o model.\n",
    "- The side-by-side comparison of the expected output and the output generated by Azure OpenAI's GPT-4o model.\n",
    "\n",
    "### Understanding Accuracy vs Confidence\n",
    "\n",
    "When using AI to extract structured data, both confidence and accuracy are essential for different but complementary reasons.\n",
    "\n",
    "- **Accuracy** measures how close the AI model's output is to a ground truth or expected output. It reflects how well the model's predictions align with reality.\n",
    "  - Accuracy ensures consistency in the extraction process, which is crucial for downstream tasks using the data.\n",
    "- **Confidence** represents the AI model's internal assessment of how certain it is about its predictions.\n",
    "  - Confidence indicates that the model is certain about its predictions, which can be a useful indicator for human reviewers to step in for manual verification.\n",
    "\n",
    "High accuracy and high confidence are ideal, but in practice, there is often a trade-off between the two. While accuracy cannot always be self-assessed, confidence scores can and should be used to prioritize manual verification of low-confidence predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "NEXGEN\n",
       "\n",
       "Innovation drives progress\n",
       "\n",
       "\n",
       "# INVOICE\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Customer:</th>\n",
       "<th colspan=\"2\">Sharp Consulting</th>\n",
       "<th colspan=\"2\">Address:</th>\n",
       "<th colspan=\"3\">73 Regal Way, Leeds, LS1 5AB, UK</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Delv. Date:</td>\n",
       "<td>5/16/2024</td>\n",
       "<td colspan=\"2\">Invoice Number:</td>\n",
       "<td colspan=\"2\">3847193</td>\n",
       "<td>Purchase Order:</td>\n",
       "<td>15931</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Item Code</th>\n",
       "<th>Item Desc.</th>\n",
       "<th>Unit Price</th>\n",
       "<th>Quantity</th>\n",
       "<th>Total</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>MA197</td>\n",
       "<td>STRETCHWRAP ROLL</td>\n",
       "<td>16.62</td>\n",
       "<td>5</td>\n",
       "<td>83.10</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>ST4086</td>\n",
       "<td>BALLPOINT PEN MED.</td>\n",
       "<td>2.49</td>\n",
       "<td>10</td>\n",
       "<td>24.90</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>JF9912413BF</td>\n",
       "<td>BUBBLE FILM ROLL CL.</td>\n",
       "<td>15.46</td>\n",
       "<td>12</td>\n",
       "<td>185.52</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "NOTES\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Total Pcs .:</th>\n",
       "<th>27</th>\n",
       "<th>Total Price:</th>\n",
       "<th>293.52</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td colspan=\"4\">Payable on or before 5/24/2024</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Cust. Sig.</td>\n",
       "<td>Sfr</td>\n",
       "<td>Drivr. Sig.</td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Cust. Name:</td>\n",
       "<td>Sarah H</td>\n",
       "<td>Drivr. Name:</td>\n",
       "<td>James T</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<!-- PageBreak -->\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<td colspan=\"6\">NEXGEN Innovation drives progress</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td colspan=\"5\">RETURNS</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Customer:</td>\n",
       "<td>Sharp Consulting</td>\n",
       "<td colspan=\"2\">Address:</td>\n",
       "<td colspan=\"2\">73 Regal Way, Leeds, LS1 5AB, UK</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Coll. Date:</td>\n",
       "<td>5/16/2024</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"2\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Item Code</td>\n",
       "<td>Item Desc.</td>\n",
       "<td>Pcs.</td>\n",
       "<td colspan=\"3\">Reason</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>MA145</td>\n",
       "<td>POSTAL TUBE BROWN</td>\n",
       "<td>1</td>\n",
       "<td colspan=\"3\">This item was provided in previous order</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>JF7902</td>\n",
       "<td>MAILBOX 25PK</td>\n",
       "<td>1 7</td>\n",
       "<td colspan=\"3\">as replacement</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"3\">Not required</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"3\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td colspan=\"6\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>NOTES</td>\n",
       "<td></td>\n",
       "<td colspan=\"4\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"4\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td colspan=\"5\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td colspan=\"5\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td colspan=\"2\"></td>\n",
       "<td></td>\n",
       "<td colspan=\"3\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"4\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Total Pcs .:</td>\n",
       "<td>2</td>\n",
       "<td></td>\n",
       "<td colspan=\"3\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td colspan=\"6\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Cust. Sig.</td>\n",
       "<td>8</td>\n",
       "<td></td>\n",
       "<td colspan=\"2\">Drivr. Sig.</td>\n",
       "<td>&amp;</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Cust. Name:</td>\n",
       "<td>Sarah H</td>\n",
       "<td></td>\n",
       "<td colspan=\"2\">Drivr. Name:</td>\n",
       "<td>James T</td>\n",
       "</tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Displays the output of the Azure AI Document Intelligence pre-built layout analysis in Markdown format.\n",
    "display(Markdown(markdown))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the parsed Invoice object from the completion response.\n",
    "invoice = completion.choices[0].message.parsed\n",
    "\n",
    "expected_dict = expected.to_dict()\n",
    "invoice_dict = invoice.to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determines the accuracy of the extracted data against the expected values.\n",
    "accuracy = invoice_evaluator.evaluate(expected=expected_dict, actual=invoice_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determines the confidence of the extracted data using both the OpenAI and Azure Document Intelligence responses.\n",
    "di_confidence = di_evaluate_confidence(invoice_dict, result)\n",
    "oai_confidence = oai_evaluate_confidence(invoice_dict, completion.choices[0])\n",
    "\n",
    "confidence = merge_confidence_values(di_confidence, oai_confidence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the total execution time of the data extraction process.\n",
    "total_elapsed = di_stopwatch.elapsed + oai_stopwatch.elapsed\n",
    "\n",
    "# Gets the prompt tokens and completion tokens from the completion response.\n",
    "prompt_tokens = completion.usage.prompt_tokens\n",
    "completion_tokens = completion.usage.completion_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the output of the data extraction result.\n",
    "extraction_result = DataExtractionResult(invoice_dict, confidence, accuracy, prompt_tokens, completion_tokens, total_elapsed)\n",
    "\n",
    "with open(f\"{working_dir}/samples/extraction/text-based/document-intelligence-openai.{pdf_fname}.json\", \"w\") as f:\n",
    "    f.write(extraction_result.to_json(indent=4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Confidence</th>\n",
       "      <th>Execution Time</th>\n",
       "      <th>Document Intelligence Execution Time</th>\n",
       "      <th>OpenAI Execution Time</th>\n",
       "      <th>Prompt Tokens</th>\n",
       "      <th>Completion Tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>93.69%</td>\n",
       "      <td>93.54%</td>\n",
       "      <td>10.44 seconds</td>\n",
       "      <td>4.38 seconds</td>\n",
       "      <td>6.07 seconds</td>\n",
       "      <td>3391</td>\n",
       "      <td>445</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Accuracy Confidence Execution Time Document Intelligence Execution Time  \\\n",
       "0   93.69%     93.54%  10.44 seconds                         4.38 seconds   \n",
       "\n",
       "  OpenAI Execution Time  Prompt Tokens  Completion Tokens  \n",
       "0          6.07 seconds           3391                445  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_933ae_row0_col0, #T_933ae_row0_col1, #T_933ae_row0_col2, #T_933ae_row0_col3, #T_933ae_row0_col4, #T_933ae_row1_col0, #T_933ae_row1_col1, #T_933ae_row1_col2, #T_933ae_row1_col3, #T_933ae_row1_col4, #T_933ae_row2_col0, #T_933ae_row2_col1, #T_933ae_row2_col2, #T_933ae_row2_col3, #T_933ae_row2_col4, #T_933ae_row3_col0, #T_933ae_row3_col1, #T_933ae_row3_col2, #T_933ae_row3_col3, #T_933ae_row3_col4, #T_933ae_row4_col0, #T_933ae_row4_col1, #T_933ae_row4_col2, #T_933ae_row4_col3, #T_933ae_row4_col4, #T_933ae_row5_col0, #T_933ae_row5_col1, #T_933ae_row5_col2, #T_933ae_row5_col3, #T_933ae_row5_col4, #T_933ae_row6_col0, #T_933ae_row6_col1, #T_933ae_row6_col2, #T_933ae_row6_col3, #T_933ae_row6_col4, #T_933ae_row7_col0, #T_933ae_row7_col1, #T_933ae_row7_col2, #T_933ae_row7_col3, #T_933ae_row7_col4, #T_933ae_row8_col0, #T_933ae_row8_col1, #T_933ae_row8_col2, #T_933ae_row8_col3, #T_933ae_row8_col4, #T_933ae_row9_col0, #T_933ae_row9_col1, #T_933ae_row9_col2, #T_933ae_row9_col3, #T_933ae_row9_col4, #T_933ae_row10_col0, #T_933ae_row10_col1, #T_933ae_row10_col2, #T_933ae_row10_col3, #T_933ae_row10_col4, #T_933ae_row11_col0, #T_933ae_row11_col1, #T_933ae_row11_col2, #T_933ae_row11_col3, #T_933ae_row11_col4, #T_933ae_row12_col0, #T_933ae_row12_col1, #T_933ae_row12_col2, #T_933ae_row12_col3, #T_933ae_row12_col4, #T_933ae_row13_col0, #T_933ae_row13_col1, #T_933ae_row13_col2, #T_933ae_row13_col3, #T_933ae_row13_col4, #T_933ae_row14_col0, #T_933ae_row14_col1, #T_933ae_row14_col2, #T_933ae_row14_col3, #T_933ae_row14_col4, #T_933ae_row15_col0, #T_933ae_row15_col1, #T_933ae_row15_col2, #T_933ae_row15_col3, #T_933ae_row15_col4, #T_933ae_row16_col0, #T_933ae_row16_col1, #T_933ae_row16_col2, #T_933ae_row16_col3, #T_933ae_row16_col4, #T_933ae_row17_col0, #T_933ae_row17_col1, #T_933ae_row17_col2, #T_933ae_row17_col3, #T_933ae_row17_col4, #T_933ae_row18_col0, #T_933ae_row18_col1, #T_933ae_row18_col2, #T_933ae_row18_col3, #T_933ae_row18_col4, #T_933ae_row19_col0, #T_933ae_row19_col1, #T_933ae_row19_col2, #T_933ae_row19_col3, #T_933ae_row19_col4, #T_933ae_row20_col0, #T_933ae_row20_col1, #T_933ae_row20_col2, #T_933ae_row20_col3, #T_933ae_row20_col4, #T_933ae_row21_col0, #T_933ae_row21_col1, #T_933ae_row21_col2, #T_933ae_row21_col3, #T_933ae_row21_col4, #T_933ae_row22_col0, #T_933ae_row22_col1, #T_933ae_row22_col2, #T_933ae_row22_col3, #T_933ae_row22_col4, #T_933ae_row23_col0, #T_933ae_row23_col1, #T_933ae_row23_col2, #T_933ae_row23_col3, #T_933ae_row23_col4, #T_933ae_row24_col0, #T_933ae_row24_col1, #T_933ae_row24_col2, #T_933ae_row24_col3, #T_933ae_row24_col4, #T_933ae_row25_col0, #T_933ae_row25_col1, #T_933ae_row25_col2, #T_933ae_row25_col3, #T_933ae_row25_col4, #T_933ae_row26_col0, #T_933ae_row26_col1, #T_933ae_row26_col2, #T_933ae_row26_col3, #T_933ae_row26_col4, #T_933ae_row27_col0, #T_933ae_row27_col1, #T_933ae_row27_col2, #T_933ae_row27_col3, #T_933ae_row27_col4, #T_933ae_row28_col0, #T_933ae_row28_col1, #T_933ae_row28_col2, #T_933ae_row28_col3, #T_933ae_row28_col4, #T_933ae_row29_col0, #T_933ae_row29_col1, #T_933ae_row29_col2, #T_933ae_row29_col3, #T_933ae_row29_col4, #T_933ae_row30_col0, #T_933ae_row30_col1, #T_933ae_row30_col2, #T_933ae_row30_col3, #T_933ae_row30_col4, #T_933ae_row31_col0, #T_933ae_row31_col1, #T_933ae_row31_col2, #T_933ae_row31_col3, #T_933ae_row31_col4, #T_933ae_row32_col0, #T_933ae_row32_col1, #T_933ae_row32_col2, #T_933ae_row32_col3, #T_933ae_row32_col4, #T_933ae_row33_col0, #T_933ae_row33_col1, #T_933ae_row33_col2, #T_933ae_row33_col3, #T_933ae_row33_col4, #T_933ae_row34_col0, #T_933ae_row34_col1, #T_933ae_row34_col2, #T_933ae_row34_col3, #T_933ae_row34_col4, #T_933ae_row35_col0, #T_933ae_row35_col1, #T_933ae_row35_col2, #T_933ae_row35_col3, #T_933ae_row35_col4, #T_933ae_row37_col0, #T_933ae_row37_col1, #T_933ae_row37_col2, #T_933ae_row37_col3, #T_933ae_row37_col4, #T_933ae_row38_col0, #T_933ae_row38_col1, #T_933ae_row38_col2, #T_933ae_row38_col3, #T_933ae_row38_col4, #T_933ae_row39_col0, #T_933ae_row39_col1, #T_933ae_row39_col2, #T_933ae_row39_col3, #T_933ae_row39_col4, #T_933ae_row40_col0, #T_933ae_row40_col1, #T_933ae_row40_col2, #T_933ae_row40_col3, #T_933ae_row40_col4, #T_933ae_row41_col0, #T_933ae_row41_col1, #T_933ae_row41_col2, #T_933ae_row41_col3, #T_933ae_row41_col4, #T_933ae_row42_col0, #T_933ae_row42_col1, #T_933ae_row42_col2, #T_933ae_row42_col3, #T_933ae_row42_col4, #T_933ae_row43_col0, #T_933ae_row43_col1, #T_933ae_row43_col2, #T_933ae_row43_col3, #T_933ae_row43_col4, #T_933ae_row44_col0, #T_933ae_row44_col1, #T_933ae_row44_col2, #T_933ae_row44_col3, #T_933ae_row44_col4, #T_933ae_row46_col0, #T_933ae_row46_col1, #T_933ae_row46_col2, #T_933ae_row46_col3, #T_933ae_row46_col4, #T_933ae_row47_col0, #T_933ae_row47_col1, #T_933ae_row47_col2, #T_933ae_row47_col3, #T_933ae_row47_col4, #T_933ae_row48_col0, #T_933ae_row48_col1, #T_933ae_row48_col2, #T_933ae_row48_col3, #T_933ae_row48_col4, #T_933ae_row49_col0, #T_933ae_row49_col1, #T_933ae_row49_col2, #T_933ae_row49_col3, #T_933ae_row49_col4, #T_933ae_row50_col0, #T_933ae_row50_col1, #T_933ae_row50_col2, #T_933ae_row50_col3, #T_933ae_row50_col4, #T_933ae_row51_col0, #T_933ae_row51_col1, #T_933ae_row51_col2, #T_933ae_row51_col3, #T_933ae_row51_col4, #T_933ae_row54_col0, #T_933ae_row54_col1, #T_933ae_row54_col2, #T_933ae_row54_col3, #T_933ae_row54_col4, #T_933ae_row55_col0, #T_933ae_row55_col1, #T_933ae_row55_col2, #T_933ae_row55_col3, #T_933ae_row55_col4, #T_933ae_row56_col0, #T_933ae_row56_col1, #T_933ae_row56_col2, #T_933ae_row56_col3, #T_933ae_row56_col4, #T_933ae_row57_col0, #T_933ae_row57_col1, #T_933ae_row57_col2, #T_933ae_row57_col3, #T_933ae_row57_col4, #T_933ae_row58_col0, #T_933ae_row58_col1, #T_933ae_row58_col2, #T_933ae_row58_col3, #T_933ae_row58_col4, #T_933ae_row59_col0, #T_933ae_row59_col1, #T_933ae_row59_col2, #T_933ae_row59_col3, #T_933ae_row59_col4, #T_933ae_row60_col0, #T_933ae_row60_col1, #T_933ae_row60_col2, #T_933ae_row60_col3, #T_933ae_row60_col4, #T_933ae_row61_col0, #T_933ae_row61_col1, #T_933ae_row61_col2, #T_933ae_row61_col3, #T_933ae_row61_col4, #T_933ae_row62_col0, #T_933ae_row62_col1, #T_933ae_row62_col2, #T_933ae_row62_col3, #T_933ae_row62_col4, #T_933ae_row63_col0, #T_933ae_row63_col1, #T_933ae_row63_col2, #T_933ae_row63_col3, #T_933ae_row63_col4, #T_933ae_row64_col0, #T_933ae_row64_col1, #T_933ae_row64_col2, #T_933ae_row64_col3, #T_933ae_row64_col4, #T_933ae_row65_col0, #T_933ae_row65_col1, #T_933ae_row65_col2, #T_933ae_row65_col3, #T_933ae_row65_col4, #T_933ae_row66_col0, #T_933ae_row66_col1, #T_933ae_row66_col2, #T_933ae_row66_col3, #T_933ae_row66_col4, #T_933ae_row67_col0, #T_933ae_row67_col1, #T_933ae_row67_col2, #T_933ae_row67_col3, #T_933ae_row67_col4, #T_933ae_row68_col0, #T_933ae_row68_col1, #T_933ae_row68_col2, #T_933ae_row68_col3, #T_933ae_row68_col4, #T_933ae_row69_col0, #T_933ae_row69_col1, #T_933ae_row69_col2, #T_933ae_row69_col3, #T_933ae_row69_col4, #T_933ae_row70_col0, #T_933ae_row70_col1, #T_933ae_row70_col2, #T_933ae_row70_col3, #T_933ae_row70_col4 {\n",
       "  background-color: #66ff33;\n",
       "}\n",
       "#T_933ae_row36_col0, #T_933ae_row36_col1, #T_933ae_row36_col2, #T_933ae_row36_col3, #T_933ae_row36_col4, #T_933ae_row45_col0, #T_933ae_row45_col1, #T_933ae_row45_col2, #T_933ae_row45_col3, #T_933ae_row45_col4, #T_933ae_row52_col0, #T_933ae_row52_col1, #T_933ae_row52_col2, #T_933ae_row52_col3, #T_933ae_row52_col4, #T_933ae_row53_col0, #T_933ae_row53_col1, #T_933ae_row53_col2, #T_933ae_row53_col3, #T_933ae_row53_col4 {\n",
       "  background-color: #ff9999;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_933ae\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_933ae_level0_col0\" class=\"col_heading level0 col0\" >Field</th>\n",
       "      <th id=\"T_933ae_level0_col1\" class=\"col_heading level0 col1\" >Expected</th>\n",
       "      <th id=\"T_933ae_level0_col2\" class=\"col_heading level0 col2\" >Extracted</th>\n",
       "      <th id=\"T_933ae_level0_col3\" class=\"col_heading level0 col3\" >Confidence</th>\n",
       "      <th id=\"T_933ae_level0_col4\" class=\"col_heading level0 col4\" >Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_933ae_row0_col0\" class=\"data row0 col0\" >customer_address_city</td>\n",
       "      <td id=\"T_933ae_row0_col1\" class=\"data row0 col1\" >Leeds</td>\n",
       "      <td id=\"T_933ae_row0_col2\" class=\"data row0 col2\" >Leeds</td>\n",
       "      <td id=\"T_933ae_row0_col3\" class=\"data row0 col3\" >98.70%</td>\n",
       "      <td id=\"T_933ae_row0_col4\" class=\"data row0 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_933ae_row1_col0\" class=\"data row1 col0\" >customer_address_country</td>\n",
       "      <td id=\"T_933ae_row1_col1\" class=\"data row1 col1\" >UK</td>\n",
       "      <td id=\"T_933ae_row1_col2\" class=\"data row1 col2\" >UK</td>\n",
       "      <td id=\"T_933ae_row1_col3\" class=\"data row1 col3\" >98.70%</td>\n",
       "      <td id=\"T_933ae_row1_col4\" class=\"data row1 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_933ae_row2_col0\" class=\"data row2 col0\" >customer_address_postal_code</td>\n",
       "      <td id=\"T_933ae_row2_col1\" class=\"data row2 col1\" >LS1 5AB</td>\n",
       "      <td id=\"T_933ae_row2_col2\" class=\"data row2 col2\" >LS1 5AB</td>\n",
       "      <td id=\"T_933ae_row2_col3\" class=\"data row2 col3\" >98.70%</td>\n",
       "      <td id=\"T_933ae_row2_col4\" class=\"data row2 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_933ae_row3_col0\" class=\"data row3 col0\" >customer_address_state</td>\n",
       "      <td id=\"T_933ae_row3_col1\" class=\"data row3 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row3_col2\" class=\"data row3 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row3_col3\" class=\"data row3 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row3_col4\" class=\"data row3 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_933ae_row4_col0\" class=\"data row4 col0\" >customer_address_street</td>\n",
       "      <td id=\"T_933ae_row4_col1\" class=\"data row4 col1\" >73 Regal Way</td>\n",
       "      <td id=\"T_933ae_row4_col2\" class=\"data row4 col2\" >73 Regal Way</td>\n",
       "      <td id=\"T_933ae_row4_col3\" class=\"data row4 col3\" >98.70%</td>\n",
       "      <td id=\"T_933ae_row4_col4\" class=\"data row4 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_933ae_row5_col0\" class=\"data row5 col0\" >customer_name</td>\n",
       "      <td id=\"T_933ae_row5_col1\" class=\"data row5 col1\" >Sharp Consulting</td>\n",
       "      <td id=\"T_933ae_row5_col2\" class=\"data row5 col2\" >Sharp Consulting</td>\n",
       "      <td id=\"T_933ae_row5_col3\" class=\"data row5 col3\" >99.40%</td>\n",
       "      <td id=\"T_933ae_row5_col4\" class=\"data row5 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_933ae_row6_col0\" class=\"data row6 col0\" >customer_tax_id</td>\n",
       "      <td id=\"T_933ae_row6_col1\" class=\"data row6 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row6_col2\" class=\"data row6 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row6_col3\" class=\"data row6 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row6_col4\" class=\"data row6 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_933ae_row7_col0\" class=\"data row7 col0\" >invoice_date</td>\n",
       "      <td id=\"T_933ae_row7_col1\" class=\"data row7 col1\" >2024-05-16</td>\n",
       "      <td id=\"T_933ae_row7_col2\" class=\"data row7 col2\" >2024-05-16</td>\n",
       "      <td id=\"T_933ae_row7_col3\" class=\"data row7 col3\" >99.99%</td>\n",
       "      <td id=\"T_933ae_row7_col4\" class=\"data row7 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_933ae_row8_col0\" class=\"data row8 col0\" >invoice_id</td>\n",
       "      <td id=\"T_933ae_row8_col1\" class=\"data row8 col1\" >3847193</td>\n",
       "      <td id=\"T_933ae_row8_col2\" class=\"data row8 col2\" >3847193</td>\n",
       "      <td id=\"T_933ae_row8_col3\" class=\"data row8 col3\" >99.80%</td>\n",
       "      <td id=\"T_933ae_row8_col4\" class=\"data row8 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_933ae_row9_col0\" class=\"data row9 col0\" >invoice_total</td>\n",
       "      <td id=\"T_933ae_row9_col1\" class=\"data row9 col1\" >293.520000</td>\n",
       "      <td id=\"T_933ae_row9_col2\" class=\"data row9 col2\" >293.520000</td>\n",
       "      <td id=\"T_933ae_row9_col3\" class=\"data row9 col3\" >99.40%</td>\n",
       "      <td id=\"T_933ae_row9_col4\" class=\"data row9 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_933ae_row10_col0\" class=\"data row10 col0\" >items_0_description</td>\n",
       "      <td id=\"T_933ae_row10_col1\" class=\"data row10 col1\" >STRETCHWRAP ROLL</td>\n",
       "      <td id=\"T_933ae_row10_col2\" class=\"data row10 col2\" >STRETCHWRAP ROLL</td>\n",
       "      <td id=\"T_933ae_row10_col3\" class=\"data row10 col3\" >98.90%</td>\n",
       "      <td id=\"T_933ae_row10_col4\" class=\"data row10 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_933ae_row11_col0\" class=\"data row11 col0\" >items_0_product_code</td>\n",
       "      <td id=\"T_933ae_row11_col1\" class=\"data row11 col1\" >MA197</td>\n",
       "      <td id=\"T_933ae_row11_col2\" class=\"data row11 col2\" >MA197</td>\n",
       "      <td id=\"T_933ae_row11_col3\" class=\"data row11 col3\" >99.70%</td>\n",
       "      <td id=\"T_933ae_row11_col4\" class=\"data row11 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_933ae_row12_col0\" class=\"data row12 col0\" >items_0_quantity</td>\n",
       "      <td id=\"T_933ae_row12_col1\" class=\"data row12 col1\" >5</td>\n",
       "      <td id=\"T_933ae_row12_col2\" class=\"data row12 col2\" >5</td>\n",
       "      <td id=\"T_933ae_row12_col3\" class=\"data row12 col3\" >99.60%</td>\n",
       "      <td id=\"T_933ae_row12_col4\" class=\"data row12 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
       "      <td id=\"T_933ae_row13_col0\" class=\"data row13 col0\" >items_0_reason</td>\n",
       "      <td id=\"T_933ae_row13_col1\" class=\"data row13 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row13_col2\" class=\"data row13 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row13_col3\" class=\"data row13 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row13_col4\" class=\"data row13 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
       "      <td id=\"T_933ae_row14_col0\" class=\"data row14 col0\" >items_0_tax</td>\n",
       "      <td id=\"T_933ae_row14_col1\" class=\"data row14 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row14_col2\" class=\"data row14 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row14_col3\" class=\"data row14 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row14_col4\" class=\"data row14 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
       "      <td id=\"T_933ae_row15_col0\" class=\"data row15 col0\" >items_0_tax_rate</td>\n",
       "      <td id=\"T_933ae_row15_col1\" class=\"data row15 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row15_col2\" class=\"data row15 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row15_col3\" class=\"data row15 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row15_col4\" class=\"data row15 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
       "      <td id=\"T_933ae_row16_col0\" class=\"data row16 col0\" >items_0_total</td>\n",
       "      <td id=\"T_933ae_row16_col1\" class=\"data row16 col1\" >83.100000</td>\n",
       "      <td id=\"T_933ae_row16_col2\" class=\"data row16 col2\" >83.100000</td>\n",
       "      <td id=\"T_933ae_row16_col3\" class=\"data row16 col3\" >90.08%</td>\n",
       "      <td id=\"T_933ae_row16_col4\" class=\"data row16 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row17\" class=\"row_heading level0 row17\" >17</th>\n",
       "      <td id=\"T_933ae_row17_col0\" class=\"data row17 col0\" >items_0_unit_price</td>\n",
       "      <td id=\"T_933ae_row17_col1\" class=\"data row17 col1\" >16.620000</td>\n",
       "      <td id=\"T_933ae_row17_col2\" class=\"data row17 col2\" >16.620000</td>\n",
       "      <td id=\"T_933ae_row17_col3\" class=\"data row17 col3\" >99.70%</td>\n",
       "      <td id=\"T_933ae_row17_col4\" class=\"data row17 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row18\" class=\"row_heading level0 row18\" >18</th>\n",
       "      <td id=\"T_933ae_row18_col0\" class=\"data row18 col0\" >items_1_description</td>\n",
       "      <td id=\"T_933ae_row18_col1\" class=\"data row18 col1\" >BALLPOINT PEN MED.</td>\n",
       "      <td id=\"T_933ae_row18_col2\" class=\"data row18 col2\" >BALLPOINT PEN MED.</td>\n",
       "      <td id=\"T_933ae_row18_col3\" class=\"data row18 col3\" >99.20%</td>\n",
       "      <td id=\"T_933ae_row18_col4\" class=\"data row18 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row19\" class=\"row_heading level0 row19\" >19</th>\n",
       "      <td id=\"T_933ae_row19_col0\" class=\"data row19 col0\" >items_1_product_code</td>\n",
       "      <td id=\"T_933ae_row19_col1\" class=\"data row19 col1\" >ST4086</td>\n",
       "      <td id=\"T_933ae_row19_col2\" class=\"data row19 col2\" >ST4086</td>\n",
       "      <td id=\"T_933ae_row19_col3\" class=\"data row19 col3\" >99.70%</td>\n",
       "      <td id=\"T_933ae_row19_col4\" class=\"data row19 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row20\" class=\"row_heading level0 row20\" >20</th>\n",
       "      <td id=\"T_933ae_row20_col0\" class=\"data row20 col0\" >items_1_quantity</td>\n",
       "      <td id=\"T_933ae_row20_col1\" class=\"data row20 col1\" >10</td>\n",
       "      <td id=\"T_933ae_row20_col2\" class=\"data row20 col2\" >10</td>\n",
       "      <td id=\"T_933ae_row20_col3\" class=\"data row20 col3\" >99.80%</td>\n",
       "      <td id=\"T_933ae_row20_col4\" class=\"data row20 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row21\" class=\"row_heading level0 row21\" >21</th>\n",
       "      <td id=\"T_933ae_row21_col0\" class=\"data row21 col0\" >items_1_reason</td>\n",
       "      <td id=\"T_933ae_row21_col1\" class=\"data row21 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row21_col2\" class=\"data row21 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row21_col3\" class=\"data row21 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row21_col4\" class=\"data row21 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row22\" class=\"row_heading level0 row22\" >22</th>\n",
       "      <td id=\"T_933ae_row22_col0\" class=\"data row22 col0\" >items_1_tax</td>\n",
       "      <td id=\"T_933ae_row22_col1\" class=\"data row22 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row22_col2\" class=\"data row22 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row22_col3\" class=\"data row22 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row22_col4\" class=\"data row22 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row23\" class=\"row_heading level0 row23\" >23</th>\n",
       "      <td id=\"T_933ae_row23_col0\" class=\"data row23 col0\" >items_1_tax_rate</td>\n",
       "      <td id=\"T_933ae_row23_col1\" class=\"data row23 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row23_col2\" class=\"data row23 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row23_col3\" class=\"data row23 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row23_col4\" class=\"data row23 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row24\" class=\"row_heading level0 row24\" >24</th>\n",
       "      <td id=\"T_933ae_row24_col0\" class=\"data row24 col0\" >items_1_total</td>\n",
       "      <td id=\"T_933ae_row24_col1\" class=\"data row24 col1\" >24.900000</td>\n",
       "      <td id=\"T_933ae_row24_col2\" class=\"data row24 col2\" >24.900000</td>\n",
       "      <td id=\"T_933ae_row24_col3\" class=\"data row24 col3\" >99.60%</td>\n",
       "      <td id=\"T_933ae_row24_col4\" class=\"data row24 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row25\" class=\"row_heading level0 row25\" >25</th>\n",
       "      <td id=\"T_933ae_row25_col0\" class=\"data row25 col0\" >items_1_unit_price</td>\n",
       "      <td id=\"T_933ae_row25_col1\" class=\"data row25 col1\" >2.490000</td>\n",
       "      <td id=\"T_933ae_row25_col2\" class=\"data row25 col2\" >2.490000</td>\n",
       "      <td id=\"T_933ae_row25_col3\" class=\"data row25 col3\" >99.20%</td>\n",
       "      <td id=\"T_933ae_row25_col4\" class=\"data row25 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row26\" class=\"row_heading level0 row26\" >26</th>\n",
       "      <td id=\"T_933ae_row26_col0\" class=\"data row26 col0\" >items_2_description</td>\n",
       "      <td id=\"T_933ae_row26_col1\" class=\"data row26 col1\" >BUBBLE FILM ROLL CL.</td>\n",
       "      <td id=\"T_933ae_row26_col2\" class=\"data row26 col2\" >BUBBLE FILM ROLL CL.</td>\n",
       "      <td id=\"T_933ae_row26_col3\" class=\"data row26 col3\" >98.60%</td>\n",
       "      <td id=\"T_933ae_row26_col4\" class=\"data row26 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row27\" class=\"row_heading level0 row27\" >27</th>\n",
       "      <td id=\"T_933ae_row27_col0\" class=\"data row27 col0\" >items_2_product_code</td>\n",
       "      <td id=\"T_933ae_row27_col1\" class=\"data row27 col1\" >JF9912413BF</td>\n",
       "      <td id=\"T_933ae_row27_col2\" class=\"data row27 col2\" >JF9912413BF</td>\n",
       "      <td id=\"T_933ae_row27_col3\" class=\"data row27 col3\" >99.50%</td>\n",
       "      <td id=\"T_933ae_row27_col4\" class=\"data row27 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row28\" class=\"row_heading level0 row28\" >28</th>\n",
       "      <td id=\"T_933ae_row28_col0\" class=\"data row28 col0\" >items_2_quantity</td>\n",
       "      <td id=\"T_933ae_row28_col1\" class=\"data row28 col1\" >12</td>\n",
       "      <td id=\"T_933ae_row28_col2\" class=\"data row28 col2\" >12</td>\n",
       "      <td id=\"T_933ae_row28_col3\" class=\"data row28 col3\" >99.80%</td>\n",
       "      <td id=\"T_933ae_row28_col4\" class=\"data row28 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row29\" class=\"row_heading level0 row29\" >29</th>\n",
       "      <td id=\"T_933ae_row29_col0\" class=\"data row29 col0\" >items_2_reason</td>\n",
       "      <td id=\"T_933ae_row29_col1\" class=\"data row29 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row29_col2\" class=\"data row29 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row29_col3\" class=\"data row29 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row29_col4\" class=\"data row29 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row30\" class=\"row_heading level0 row30\" >30</th>\n",
       "      <td id=\"T_933ae_row30_col0\" class=\"data row30 col0\" >items_2_tax</td>\n",
       "      <td id=\"T_933ae_row30_col1\" class=\"data row30 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row30_col2\" class=\"data row30 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row30_col3\" class=\"data row30 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row30_col4\" class=\"data row30 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row31\" class=\"row_heading level0 row31\" >31</th>\n",
       "      <td id=\"T_933ae_row31_col0\" class=\"data row31 col0\" >items_2_tax_rate</td>\n",
       "      <td id=\"T_933ae_row31_col1\" class=\"data row31 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row31_col2\" class=\"data row31 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row31_col3\" class=\"data row31 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row31_col4\" class=\"data row31 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row32\" class=\"row_heading level0 row32\" >32</th>\n",
       "      <td id=\"T_933ae_row32_col0\" class=\"data row32 col0\" >items_2_total</td>\n",
       "      <td id=\"T_933ae_row32_col1\" class=\"data row32 col1\" >185.520000</td>\n",
       "      <td id=\"T_933ae_row32_col2\" class=\"data row32 col2\" >185.520000</td>\n",
       "      <td id=\"T_933ae_row32_col3\" class=\"data row32 col3\" >99.60%</td>\n",
       "      <td id=\"T_933ae_row32_col4\" class=\"data row32 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row33\" class=\"row_heading level0 row33\" >33</th>\n",
       "      <td id=\"T_933ae_row33_col0\" class=\"data row33 col0\" >items_2_unit_price</td>\n",
       "      <td id=\"T_933ae_row33_col1\" class=\"data row33 col1\" >15.460000</td>\n",
       "      <td id=\"T_933ae_row33_col2\" class=\"data row33 col2\" >15.460000</td>\n",
       "      <td id=\"T_933ae_row33_col3\" class=\"data row33 col3\" >99.80%</td>\n",
       "      <td id=\"T_933ae_row33_col4\" class=\"data row33 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row34\" class=\"row_heading level0 row34\" >34</th>\n",
       "      <td id=\"T_933ae_row34_col0\" class=\"data row34 col0\" >items_customer_signature_is_signed</td>\n",
       "      <td id=\"T_933ae_row34_col1\" class=\"data row34 col1\" >True</td>\n",
       "      <td id=\"T_933ae_row34_col2\" class=\"data row34 col2\" >True</td>\n",
       "      <td id=\"T_933ae_row34_col3\" class=\"data row34 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row34_col4\" class=\"data row34 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row35\" class=\"row_heading level0 row35\" >35</th>\n",
       "      <td id=\"T_933ae_row35_col0\" class=\"data row35 col0\" >items_customer_signature_signatory</td>\n",
       "      <td id=\"T_933ae_row35_col1\" class=\"data row35 col1\" >Sarah H</td>\n",
       "      <td id=\"T_933ae_row35_col2\" class=\"data row35 col2\" >Sarah H</td>\n",
       "      <td id=\"T_933ae_row35_col3\" class=\"data row35 col3\" >54.80%</td>\n",
       "      <td id=\"T_933ae_row35_col4\" class=\"data row35 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row36\" class=\"row_heading level0 row36\" >36</th>\n",
       "      <td id=\"T_933ae_row36_col0\" class=\"data row36 col0\" >items_vendor_signature_is_signed</td>\n",
       "      <td id=\"T_933ae_row36_col1\" class=\"data row36 col1\" >True</td>\n",
       "      <td id=\"T_933ae_row36_col2\" class=\"data row36 col2\" >False</td>\n",
       "      <td id=\"T_933ae_row36_col3\" class=\"data row36 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row36_col4\" class=\"data row36 col4\" >Mismatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row37\" class=\"row_heading level0 row37\" >37</th>\n",
       "      <td id=\"T_933ae_row37_col0\" class=\"data row37 col0\" >items_vendor_signature_signatory</td>\n",
       "      <td id=\"T_933ae_row37_col1\" class=\"data row37 col1\" >James T</td>\n",
       "      <td id=\"T_933ae_row37_col2\" class=\"data row37 col2\" >James T</td>\n",
       "      <td id=\"T_933ae_row37_col3\" class=\"data row37 col3\" >86.50%</td>\n",
       "      <td id=\"T_933ae_row37_col4\" class=\"data row37 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row38\" class=\"row_heading level0 row38\" >38</th>\n",
       "      <td id=\"T_933ae_row38_col0\" class=\"data row38 col0\" >payable_by</td>\n",
       "      <td id=\"T_933ae_row38_col1\" class=\"data row38 col1\" >2024-05-24</td>\n",
       "      <td id=\"T_933ae_row38_col2\" class=\"data row38 col2\" >2024-05-24</td>\n",
       "      <td id=\"T_933ae_row38_col3\" class=\"data row38 col3\" >100.00%</td>\n",
       "      <td id=\"T_933ae_row38_col4\" class=\"data row38 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row39\" class=\"row_heading level0 row39\" >39</th>\n",
       "      <td id=\"T_933ae_row39_col0\" class=\"data row39 col0\" >payment_terms</td>\n",
       "      <td id=\"T_933ae_row39_col1\" class=\"data row39 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row39_col2\" class=\"data row39 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row39_col3\" class=\"data row39 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row39_col4\" class=\"data row39 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row40\" class=\"row_heading level0 row40\" >40</th>\n",
       "      <td id=\"T_933ae_row40_col0\" class=\"data row40 col0\" >purchase_order</td>\n",
       "      <td id=\"T_933ae_row40_col1\" class=\"data row40 col1\" >15931</td>\n",
       "      <td id=\"T_933ae_row40_col2\" class=\"data row40 col2\" >15931</td>\n",
       "      <td id=\"T_933ae_row40_col3\" class=\"data row40 col3\" >99.50%</td>\n",
       "      <td id=\"T_933ae_row40_col4\" class=\"data row40 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row41\" class=\"row_heading level0 row41\" >41</th>\n",
       "      <td id=\"T_933ae_row41_col0\" class=\"data row41 col0\" >remittance_address</td>\n",
       "      <td id=\"T_933ae_row41_col1\" class=\"data row41 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row41_col2\" class=\"data row41 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row41_col3\" class=\"data row41 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row41_col4\" class=\"data row41 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row42\" class=\"row_heading level0 row42\" >42</th>\n",
       "      <td id=\"T_933ae_row42_col0\" class=\"data row42 col0\" >returns_0_description</td>\n",
       "      <td id=\"T_933ae_row42_col1\" class=\"data row42 col1\" >POSTAL TUBE BROWN</td>\n",
       "      <td id=\"T_933ae_row42_col2\" class=\"data row42 col2\" >POSTAL TUBE BROWN</td>\n",
       "      <td id=\"T_933ae_row42_col3\" class=\"data row42 col3\" >99.20%</td>\n",
       "      <td id=\"T_933ae_row42_col4\" class=\"data row42 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row43\" class=\"row_heading level0 row43\" >43</th>\n",
       "      <td id=\"T_933ae_row43_col0\" class=\"data row43 col0\" >returns_0_product_code</td>\n",
       "      <td id=\"T_933ae_row43_col1\" class=\"data row43 col1\" >MA145</td>\n",
       "      <td id=\"T_933ae_row43_col2\" class=\"data row43 col2\" >MA145</td>\n",
       "      <td id=\"T_933ae_row43_col3\" class=\"data row43 col3\" >99.60%</td>\n",
       "      <td id=\"T_933ae_row43_col4\" class=\"data row43 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row44\" class=\"row_heading level0 row44\" >44</th>\n",
       "      <td id=\"T_933ae_row44_col0\" class=\"data row44 col0\" >returns_0_quantity</td>\n",
       "      <td id=\"T_933ae_row44_col1\" class=\"data row44 col1\" >1</td>\n",
       "      <td id=\"T_933ae_row44_col2\" class=\"data row44 col2\" >1</td>\n",
       "      <td id=\"T_933ae_row44_col3\" class=\"data row44 col3\" >99.40%</td>\n",
       "      <td id=\"T_933ae_row44_col4\" class=\"data row44 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row45\" class=\"row_heading level0 row45\" >45</th>\n",
       "      <td id=\"T_933ae_row45_col0\" class=\"data row45 col0\" >returns_0_reason</td>\n",
       "      <td id=\"T_933ae_row45_col1\" class=\"data row45 col1\" >This item was provided in previous order as a replacement</td>\n",
       "      <td id=\"T_933ae_row45_col2\" class=\"data row45 col2\" >This item was provided in previous order</td>\n",
       "      <td id=\"T_933ae_row45_col3\" class=\"data row45 col3\" >66.20%</td>\n",
       "      <td id=\"T_933ae_row45_col4\" class=\"data row45 col4\" >Mismatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row46\" class=\"row_heading level0 row46\" >46</th>\n",
       "      <td id=\"T_933ae_row46_col0\" class=\"data row46 col0\" >returns_0_tax</td>\n",
       "      <td id=\"T_933ae_row46_col1\" class=\"data row46 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row46_col2\" class=\"data row46 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row46_col3\" class=\"data row46 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row46_col4\" class=\"data row46 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row47\" class=\"row_heading level0 row47\" >47</th>\n",
       "      <td id=\"T_933ae_row47_col0\" class=\"data row47 col0\" >returns_0_tax_rate</td>\n",
       "      <td id=\"T_933ae_row47_col1\" class=\"data row47 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row47_col2\" class=\"data row47 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row47_col3\" class=\"data row47 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row47_col4\" class=\"data row47 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row48\" class=\"row_heading level0 row48\" >48</th>\n",
       "      <td id=\"T_933ae_row48_col0\" class=\"data row48 col0\" >returns_0_total</td>\n",
       "      <td id=\"T_933ae_row48_col1\" class=\"data row48 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row48_col2\" class=\"data row48 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row48_col3\" class=\"data row48 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row48_col4\" class=\"data row48 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row49\" class=\"row_heading level0 row49\" >49</th>\n",
       "      <td id=\"T_933ae_row49_col0\" class=\"data row49 col0\" >returns_0_unit_price</td>\n",
       "      <td id=\"T_933ae_row49_col1\" class=\"data row49 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row49_col2\" class=\"data row49 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row49_col3\" class=\"data row49 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row49_col4\" class=\"data row49 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row50\" class=\"row_heading level0 row50\" >50</th>\n",
       "      <td id=\"T_933ae_row50_col0\" class=\"data row50 col0\" >returns_1_description</td>\n",
       "      <td id=\"T_933ae_row50_col1\" class=\"data row50 col1\" >MAILBOX 25PK</td>\n",
       "      <td id=\"T_933ae_row50_col2\" class=\"data row50 col2\" >MAILBOX 25PK</td>\n",
       "      <td id=\"T_933ae_row50_col3\" class=\"data row50 col3\" >99.20%</td>\n",
       "      <td id=\"T_933ae_row50_col4\" class=\"data row50 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row51\" class=\"row_heading level0 row51\" >51</th>\n",
       "      <td id=\"T_933ae_row51_col0\" class=\"data row51 col0\" >returns_1_product_code</td>\n",
       "      <td id=\"T_933ae_row51_col1\" class=\"data row51 col1\" >JF7902</td>\n",
       "      <td id=\"T_933ae_row51_col2\" class=\"data row51 col2\" >JF7902</td>\n",
       "      <td id=\"T_933ae_row51_col3\" class=\"data row51 col3\" >99.50%</td>\n",
       "      <td id=\"T_933ae_row51_col4\" class=\"data row51 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row52\" class=\"row_heading level0 row52\" >52</th>\n",
       "      <td id=\"T_933ae_row52_col0\" class=\"data row52 col0\" >returns_1_quantity</td>\n",
       "      <td id=\"T_933ae_row52_col1\" class=\"data row52 col1\" >1</td>\n",
       "      <td id=\"T_933ae_row52_col2\" class=\"data row52 col2\" >17</td>\n",
       "      <td id=\"T_933ae_row52_col3\" class=\"data row52 col3\" >60.58%</td>\n",
       "      <td id=\"T_933ae_row52_col4\" class=\"data row52 col4\" >Mismatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row53\" class=\"row_heading level0 row53\" >53</th>\n",
       "      <td id=\"T_933ae_row53_col0\" class=\"data row53 col0\" >returns_1_reason</td>\n",
       "      <td id=\"T_933ae_row53_col1\" class=\"data row53 col1\" >Not required</td>\n",
       "      <td id=\"T_933ae_row53_col2\" class=\"data row53 col2\" >as replacement</td>\n",
       "      <td id=\"T_933ae_row53_col3\" class=\"data row53 col3\" >83.40%</td>\n",
       "      <td id=\"T_933ae_row53_col4\" class=\"data row53 col4\" >Mismatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row54\" class=\"row_heading level0 row54\" >54</th>\n",
       "      <td id=\"T_933ae_row54_col0\" class=\"data row54 col0\" >returns_1_tax</td>\n",
       "      <td id=\"T_933ae_row54_col1\" class=\"data row54 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row54_col2\" class=\"data row54 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row54_col3\" class=\"data row54 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row54_col4\" class=\"data row54 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row55\" class=\"row_heading level0 row55\" >55</th>\n",
       "      <td id=\"T_933ae_row55_col0\" class=\"data row55 col0\" >returns_1_tax_rate</td>\n",
       "      <td id=\"T_933ae_row55_col1\" class=\"data row55 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row55_col2\" class=\"data row55 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row55_col3\" class=\"data row55 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row55_col4\" class=\"data row55 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row56\" class=\"row_heading level0 row56\" >56</th>\n",
       "      <td id=\"T_933ae_row56_col0\" class=\"data row56 col0\" >returns_1_total</td>\n",
       "      <td id=\"T_933ae_row56_col1\" class=\"data row56 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row56_col2\" class=\"data row56 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row56_col3\" class=\"data row56 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row56_col4\" class=\"data row56 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row57\" class=\"row_heading level0 row57\" >57</th>\n",
       "      <td id=\"T_933ae_row57_col0\" class=\"data row57 col0\" >returns_1_unit_price</td>\n",
       "      <td id=\"T_933ae_row57_col1\" class=\"data row57 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row57_col2\" class=\"data row57 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row57_col3\" class=\"data row57 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row57_col4\" class=\"data row57 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row58\" class=\"row_heading level0 row58\" >58</th>\n",
       "      <td id=\"T_933ae_row58_col0\" class=\"data row58 col0\" >returns_customer_signature_is_signed</td>\n",
       "      <td id=\"T_933ae_row58_col1\" class=\"data row58 col1\" >True</td>\n",
       "      <td id=\"T_933ae_row58_col2\" class=\"data row58 col2\" >True</td>\n",
       "      <td id=\"T_933ae_row58_col3\" class=\"data row58 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row58_col4\" class=\"data row58 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row59\" class=\"row_heading level0 row59\" >59</th>\n",
       "      <td id=\"T_933ae_row59_col0\" class=\"data row59 col0\" >returns_customer_signature_signatory</td>\n",
       "      <td id=\"T_933ae_row59_col1\" class=\"data row59 col1\" >Sarah H</td>\n",
       "      <td id=\"T_933ae_row59_col2\" class=\"data row59 col2\" >Sarah H</td>\n",
       "      <td id=\"T_933ae_row59_col3\" class=\"data row59 col3\" >54.80%</td>\n",
       "      <td id=\"T_933ae_row59_col4\" class=\"data row59 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row60\" class=\"row_heading level0 row60\" >60</th>\n",
       "      <td id=\"T_933ae_row60_col0\" class=\"data row60 col0\" >returns_vendor_signature_is_signed</td>\n",
       "      <td id=\"T_933ae_row60_col1\" class=\"data row60 col1\" >True</td>\n",
       "      <td id=\"T_933ae_row60_col2\" class=\"data row60 col2\" >True</td>\n",
       "      <td id=\"T_933ae_row60_col3\" class=\"data row60 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row60_col4\" class=\"data row60 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row61\" class=\"row_heading level0 row61\" >61</th>\n",
       "      <td id=\"T_933ae_row61_col0\" class=\"data row61 col0\" >returns_vendor_signature_signatory</td>\n",
       "      <td id=\"T_933ae_row61_col1\" class=\"data row61 col1\" >James T</td>\n",
       "      <td id=\"T_933ae_row61_col2\" class=\"data row61 col2\" >James T</td>\n",
       "      <td id=\"T_933ae_row61_col3\" class=\"data row61 col3\" >86.50%</td>\n",
       "      <td id=\"T_933ae_row61_col4\" class=\"data row61 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row62\" class=\"row_heading level0 row62\" >62</th>\n",
       "      <td id=\"T_933ae_row62_col0\" class=\"data row62 col0\" >shipping_address</td>\n",
       "      <td id=\"T_933ae_row62_col1\" class=\"data row62 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row62_col2\" class=\"data row62 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row62_col3\" class=\"data row62 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row62_col4\" class=\"data row62 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row63\" class=\"row_heading level0 row63\" >63</th>\n",
       "      <td id=\"T_933ae_row63_col0\" class=\"data row63 col0\" >subtotal</td>\n",
       "      <td id=\"T_933ae_row63_col1\" class=\"data row63 col1\" >293.520000</td>\n",
       "      <td id=\"T_933ae_row63_col2\" class=\"data row63 col2\" >293.520000</td>\n",
       "      <td id=\"T_933ae_row63_col3\" class=\"data row63 col3\" >83.77%</td>\n",
       "      <td id=\"T_933ae_row63_col4\" class=\"data row63 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row64\" class=\"row_heading level0 row64\" >64</th>\n",
       "      <td id=\"T_933ae_row64_col0\" class=\"data row64 col0\" >total_discount</td>\n",
       "      <td id=\"T_933ae_row64_col1\" class=\"data row64 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row64_col2\" class=\"data row64 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row64_col3\" class=\"data row64 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row64_col4\" class=\"data row64 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row65\" class=\"row_heading level0 row65\" >65</th>\n",
       "      <td id=\"T_933ae_row65_col0\" class=\"data row65 col0\" >total_item_quantity</td>\n",
       "      <td id=\"T_933ae_row65_col1\" class=\"data row65 col1\" >27.000000</td>\n",
       "      <td id=\"T_933ae_row65_col2\" class=\"data row65 col2\" >27.000000</td>\n",
       "      <td id=\"T_933ae_row65_col3\" class=\"data row65 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row65_col4\" class=\"data row65 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row66\" class=\"row_heading level0 row66\" >66</th>\n",
       "      <td id=\"T_933ae_row66_col0\" class=\"data row66 col0\" >total_return_quantity</td>\n",
       "      <td id=\"T_933ae_row66_col1\" class=\"data row66 col1\" >2.000000</td>\n",
       "      <td id=\"T_933ae_row66_col2\" class=\"data row66 col2\" >2.000000</td>\n",
       "      <td id=\"T_933ae_row66_col3\" class=\"data row66 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row66_col4\" class=\"data row66 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row67\" class=\"row_heading level0 row67\" >67</th>\n",
       "      <td id=\"T_933ae_row67_col0\" class=\"data row67 col0\" >total_tax</td>\n",
       "      <td id=\"T_933ae_row67_col1\" class=\"data row67 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row67_col2\" class=\"data row67 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row67_col3\" class=\"data row67 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row67_col4\" class=\"data row67 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row68\" class=\"row_heading level0 row68\" >68</th>\n",
       "      <td id=\"T_933ae_row68_col0\" class=\"data row68 col0\" >vendor_address</td>\n",
       "      <td id=\"T_933ae_row68_col1\" class=\"data row68 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row68_col2\" class=\"data row68 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row68_col3\" class=\"data row68 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row68_col4\" class=\"data row68 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row69\" class=\"row_heading level0 row69\" >69</th>\n",
       "      <td id=\"T_933ae_row69_col0\" class=\"data row69 col0\" >vendor_name</td>\n",
       "      <td id=\"T_933ae_row69_col1\" class=\"data row69 col1\" >NEXGEN</td>\n",
       "      <td id=\"T_933ae_row69_col2\" class=\"data row69 col2\" >NEXGEN</td>\n",
       "      <td id=\"T_933ae_row69_col3\" class=\"data row69 col3\" >99.01%</td>\n",
       "      <td id=\"T_933ae_row69_col4\" class=\"data row69 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_933ae_level0_row70\" class=\"row_heading level0 row70\" >70</th>\n",
       "      <td id=\"T_933ae_row70_col0\" class=\"data row70 col0\" >vendor_tax_id</td>\n",
       "      <td id=\"T_933ae_row70_col1\" class=\"data row70 col1\" >None</td>\n",
       "      <td id=\"T_933ae_row70_col2\" class=\"data row70 col2\" >None</td>\n",
       "      <td id=\"T_933ae_row70_col3\" class=\"data row70 col3\" >0.00%</td>\n",
       "      <td id=\"T_933ae_row70_col4\" class=\"data row70 col4\" >Match</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f8de3fa8d70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the outputs of the data extraction process.\n",
    "df = pd.DataFrame([\n",
    "    {\n",
    "        \"Accuracy\": f\"{accuracy['overall'] * 100:.2f}%\",\n",
    "        \"Confidence\": f\"{confidence['_overall'] * 100:.2f}%\",\n",
    "        \"Execution Time\": f\"{total_elapsed:.2f} seconds\",\n",
    "        \"Document Intelligence Execution Time\": f\"{di_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"OpenAI Execution Time\": f\"{oai_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Prompt Tokens\": prompt_tokens,\n",
    "        \"Completion Tokens\": completion_tokens\n",
    "    }\n",
    "])\n",
    "\n",
    "display(df)\n",
    "display(get_extraction_comparison(expected_dict, invoice_dict, confidence, accuracy['accuracy']))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}