Why Production AI Workflows Need More Than an LLM Call

A demo that reads invoices can extract the right total and still leave a business with duplicate records or a backlog of corrections. Before relying on it, ask: who catches mistakes, and what happens next?

Answering those questions early helps control scope and cost. A practical first release can handle fewer document types while retaining the checks that make it dependable.

Consider an accounts payable workflow where invoices arrive by email. A large language model (LLM) extracts the vendor, total, purchase order number, and line items. Approved results enter an accounting system. Here is what that process needs around the model call.

Give the model a defined job

In this example, software receives the email, checks for duplicates, stores the attachment, and prepares the document. The model extracts fields. The workflow then validates them, requests review when needed, and writes approved records to the accounting system.

This follows the distinction in Anthropic's guidance on workflows and agents: workflows follow predefined paths, while agents can decide how to proceed and which tools to use.

An invoice should never be able to tell the system to bypass approval. Let the model read the document, while application code controls which actions are allowed.

Compare models using invoices like those the business receives. Measure how accurately they read the fields, what processing costs, and how much correction each result needs. A cheaper model call is not necessarily a cheaper process if it creates more manual corrections.

Validate the result before acting on it

Before the workflow can use the model's answer, it needs the information in a consistent format—for example, an invoice total returned as a number and a vendor name returned as text. Features called structured outputs help enforce that format, as described in Anthropic's documentation. But a correctly formatted answer can still contain the wrong amount or vendor, so the workflow needs to check the values too.

Check the result in three ways:

• Format: Is all required information present and in the expected format? An incomplete answer should not proceed to the accounting system.
• Business rules: Do the line items, taxes, discounts, and other charges add up to the total? Is the vendor recognized? Does the purchase order belong to that vendor?
• Source support: Can each extracted value be traced to the relevant part of the document?

For PDFs with selectable text, compare extracted values with the source, accounting for differences in spacing and number formatting. Text recognition can misread scans and photos, so a missing match should prompt review rather than automatically disqualify the result.

Even an exact match needs context: a document may contain several dates and totals. Check that the value belongs to the right field.

Define what happens when a check fails. One additional attempt, with the specific error explained, may help correct an extraction mistake. An unknown vendor may require a business decision instead. Check revised values against the source again; passing the business rule alone is not enough.

Make failures recoverable

The model service may respond too slowly, limit the number of requests it accepts, or become unavailable. Store incoming documents reliably, limit retries, and show the team which items still need attention.

Saving an invoice needs particular care. Suppose the accounting system creates the record, but the workflow never receives confirmation. Trying again without checking the outcome could create a duplicate.

Where the accounting system supports it, give each request to create a record a unique identifier and reuse it when retrying. The system can then recognize the repeated request and avoid creating another record. This is called idempotency, a pattern explained in AWS's guidance on safe retries. Otherwise, check whether the record was created before trying again; send uncertain cases for investigation. Keeping extraction and saving separate helps with recovery, but does not itself prevent duplicates.

If you switch to another model when the first is unavailable, check its results too. During an outage, holding invoices for later processing or human review may be appropriate.

Make human review useful and measurable

In healthcare, a similar workflow might extract information from a patient intake form for staff review. If a medication name is unclear or an allergy field is blank, flag it for confirmation. A blank field should remain "unknown," rather than become "no known allergies."

Give reviewers the source document beside the extracted fields, with a clear explanation of what needs checking. Let them correct the result without reconstructing the entire workflow, and record those corrections.

Monitor how long invoices wait, how long review takes, and how often corrections are needed. These measurements help determine whether the automation is reducing work or shifting it into a backlog.

Few corrections do not necessarily mean review is unnecessary. The model may be accurate, the invoices may be easy to read, or reviewers may be missing mistakes. Arrange a separate check of some approved records before expanding automatic processing, and continue spot checks afterward.

Record enough to investigate and improve

A workflow can finish successfully while producing an incorrect result. Link each invoice to its processing history: the prompt and model used, validation results, retries, accounting-system updates, processing time, and cost. Record whether it was approved, corrected, or rejected, along with who reviewed it.

Full prompts and responses can contain sensitive business data. Save only what you need to investigate problems, restrict who can see it, and set a deletion schedule. Routine logs should not copy unnecessary invoice details.

Use verified corrections to build a test set covering the documents you expect to receive: clean PDFs, scans, credit notes, unfamiliar layouts, and previous failures. Compare the model's answers with the correct values. Test the surrounding software too: calculations, permissions, retries, and where each invoice goes next.

Run those checks before releasing prompt or model changes. Include invoices you did not use to improve the prompt, and check results by document type and important fields such as the total. An overall accuracy score can hide a problem concentrated in one vendor's invoices.

Scope the first release around a complete process

A first release does not need to handle every vendor's invoices or choose its own next steps. It needs a clear plan for mistakes and interruptions in the work it does support.

Start with a few document types, clear checks, human approval, and safeguards against duplicate records. Keep enough history to investigate errors. Automate more as the results justify it.

For a project proposal, ask four questions:

• What happens when the model is wrong?
• How will someone discover the problem?
• Who owns correction and recovery?
• How will the team verify that a change improves the process?

Budget for reviewing invoices and resolving problems alongside connecting systems and paying for model usage. If the first release is too expensive, reduce the range of documents or actions it supports while keeping those controls intact. This is the same scope discipline behind planning a startup MVP without overbuilding it.

Have a workflow in mind? The Fractional Stack's AI Workflow Design helps you decide what to automate, where human review belongs, and what to build first.