Hackathon - Team April 3, 2023

Explore

Hackathon - Team April 3, 2023

Hackathon - Team April 3, 2023

Use Case 4 - OAI + Templatization

Document Automation

Objective

The main objective of this use case is to combine two extraction methods (OAI and Templatization) to ensure the best extraction results for Mode Transport. The extraction process will be the following:

Utilize OpenAI for complex extractions.

Implement Templatization for structured data.

Key Note: For this project, we have utilized Claude AI instead of OAI. As such we will mention Claude AI throughout this documentation.

Workflow

Extraction States

The way that we want to approach this is by taking the extraction from both Model Extraction and the templates and map the best extractions to each inputs at the end in the Create Payload state. The figure below shows the flow of our workflow states and where the results of each extraction method is generated.

⁠

Figure: State Flowchart

Figure: State Flowchart

⁠

The detail explanation of each of the output states are explained below:

Model Extraction

Text Extraction: Uses the default Universal Parser to extract document text from the pdf files.

Model Extraction: Uses Claude AI to extract desirable output using a collection of prompts.

Model Data Transformation: Uses Transformation Layer to convert the model’s output into suitable form-table data JSON structure for CP and Mapping.

Template Extraction

Extract Data Template: Uses Templatization to extract data and shows it in JSON format.

Payload Creation

By thorough comparison of outputs generated from both Model Extraction and Templatization, we will be prioritizing which input will have which extraction. The following figure shows the ways how different inputs in the Create Payload state will be getting the results.

⁠

Figure: Extraction Types

Figure: Extraction Types

⁠

Model Extraction: The first input will be show the extraction generated through Claude AI.

Templatization: The second input will be the extraction generated through Templatization.

If-Else: The third input will be an If-Else case where we will be checking if the extraction from Templatization is empty or not. In case the extraction is empty, we can assume that the document does not have the value and we will not be using model extraction as there are high chances that the model will hallucinate.

Integration

BOT Email : april@preprod.rippeybot.com

System Interaction Overview

This integration aims to improve the accuracy and flexibility of data extraction from various document types used by Mode Transport. The components involved are:

UFL Dashboard: We have our existing dashboard which we have used as the extraction platform. We have used our product, Document Automation, in pre-prod environment.

Claude Model: We have used Claude AI model for extracting complex, unstructured data that may span across multiple pages.

Templatization: We have utilized templatization for structured data extraction based on predefined templates.

Workflow Integration

Document Ingestion:

Documents are ingested into the system either by sending to the bot email or uploading in the dashboard itself, where they are first processed by the existing document automation system to identify the type of document and relevant fields.

The document is then passed through both the Claude AI and templatization pipelines for extraction.

Claude AI Integration:

API/Model Invocation: We have directed the dashboard to use the QA environment which has the Claude AI model integrated. The task used is ‘shipping_order_prompt_division_task_v1’ which has prompts with Claude AI model.

Templatization Integration:

Template Matching: Each template will have a unique identification which will match the document sent with that template. After the identification is true, then the rest of the fields are matched based on how the template has been created.

Data Extraction and Validation: We have used Regex for each field to extract data accurately for each field.

Decision-Making Process

Extraction Prioritization:

After thoroughly evaluating the extraction results for Claude AI and calculating the accuracy rates, we prioritize which field will be using which extraction method (Claude AI or Template) and map them accordingly in Create Payload state.

Fallback Mechanism: In case the system fails to extract through all methods, then the user will have to manually enter the data for mandatory fields.

Error Handling: We will be having different mechanisms to handle the errors that we face while payload creation. They are explained below:

Empty Value: In case there is an empty data for a mandatory field, we will have to manually enter data for critical fields that must not be left empty.

Detailed Logs and Alerts: In case a user tries to Execute the transaction with an empty mandatory field, proper alert will be shown indicating which field was problematic and why.

Testing

Before Integration

Our team has conducted the following methods to conduct testing to validate accuracy:

We prepared ground truth for 20+ documents of Mode.

Then we analyzed the accuracy for each fields using Claude AI.

Based on the accuracy rate, we made the following decisions:

High Accuracy: Use the extraction from Claude AI.

Low Accuracy: Use the Claude AI if it extracts a value, else use Templatization.

Zero Accuracy: Use the extraction from Templatization.

After Integration

After the setup of the whole dashboard, we conducted testing by sending documents in our dashboard and refining the results.

Results

We have successfully mapped the fields directly in the Create Payload state based on the prioritization and accuracy rates. Some fields are generated from model extraction whereas some fields are from templatization. We have mapped the following fields to have template extraction:

marks_and_numbers

weight_uom

customer_po_number

ncm_number

container_details

Challenges

During Integration

We encountered a significant challenge while working on this use case due to a Claude AI outage. Initially, we relied on Claude AI for model extraction, but when the daily limit was exceeded, we could not perform any tests for the whole day and had to wait for the next. This was an unexpected case as well. Consequently, when we think about the long run, this daily limit would definitely harm us on the product level.

Outside Integration

Using both the model and templatization for a single document certainly means increased cost. The question to ask here is: Is the increased accuracy rate worth the increased cost. This alone could enable our feature to be used in only a few niche cases.

Suggestion

Further suggestion that we want to give to improve are as follows:

The Claude AI “Outage” that we faced while working on this use case is not scalable for future use. Hence, we suggest on extending the daily limit if possible, or at least a failback solution where it would switch to OAI must it encounter the same issue.

Another suggestion is to have a visual indication of which extraction method has been used in the Create Payload state for each field. We can have a badge beside each input label as shown in the screenshot below:

Indication: ML for Model Extraction and TL for Templatization

For Form Data, we can have the badge inside each input box.

Figure: Form Data

Figure: Form Data

⁠

For Table Data, we can have the badge to indicate beside the table header.

Figure: Table Data

Figure: Table Data

⁠

Utilizing both O(Claude)AI and Templatization for a single document can significantly heighten our costs. Therefore, a suggestion could be to reserve the use of both methods for specific cases and document formats where it's truly necessary. Exceptions should be carefully defined, and Templatization should only be triggered when the model fails in those designated scenarios. This approach will optimize efficiency while managing costs effectively.