You are building document processing over scanned PDFs, invoices, contracts, forms. Vision model or OCR pipeline, and how do you make the output trustworthy?

Question

Accepted Answer

For messy real-world documents, my default is a vision-capable model reading rendered pages, because layout is meaning in invoices and forms, which table a number sits in, what label is adjacent, and vision models use that where classic OCR-to-text flattens it away. Traditional OCR keeps a place at high volume on clean consistent documents, where it is much cheaper per page, and a sensible hybrid is OCR plus layout as supplementary input to the model. But the architecture choice is the smaller half; trustworthiness is the job. Extraction is schema-constrained, every field typed, categorical fields as enums, and everything nullable, because a required field the document does not contain is a fabrication machine. Then verification in layers: deterministic validation per field, dates parse, currencies match expected ranges, tax ids checksum; cross-field consistency, line items sum to the subtotal, subtotal plus tax equals total, which catches a large share of misreads because a hallucinated digit rarely keeps the arithmetic consistent; and for the critical fields, amounts, account numbers, a second extraction pass, either re-asking with the crop of the relevant region or a different model, agreement between passes is a strong correctness signal. Confidence routing ties it together: documents that pass validation, cross-checks and agreement flow straight through; failures and disagreements queue for human review with the model output pre-filled, which typically means humans see maybe 10 to 20 percent of volume instead of 100. And it is measured like a product: per-field accuracy on a labelled set of a few hundred documents, straight-through rate, and human-correction rate feeding back as eval cases and, eventually, fine-tuning data. The goal is not a model that is never wrong, it is a system that knows when it might be.

Answer

Why does the totals cross-check catch so many errors?

Digit-level misreads are independent errors, and independent errors rarely preserve arithmetic identities. Consistency constraints are nearly free verification.

Answer

Accuracy is fine in testing but drops for one supplier in production. What is going on?

A template or scan-quality cluster the eval set missed. Slice accuracy per supplier and template, add the failing cluster to the eval set, and consider template-specific prompting or examples.

You are building document processing over scanned PDFs, invoices, contracts, forms. Vision model or OCR pipeline, and how do you make the output trustworthy?

You are building document processing over scanned PDFs, invoices, contracts, forms. Vision model or OCR pipeline, and how do you make the output trustworthy?

The follow-ups they ask next

What the interviewer is listening for

What sinks the answer

If you genuinely do not know

Keep going with multimodal

Knowing the answer is not the same as recalling it under pressure