For decades, document workflows depended on fragile OCR pipelines. A standard setup ran through a sequence of brittle steps: binarize the image, deskew it, extract raw text strings with tools like Tesseract or ABBYY, run coordinate-based layout engines, and hope regular expressions did not break when a supplier moved a VAT number. When a multi-column table collapsed into a single inline string or an engine misread a decimal point as a comma, downstream systems received corrupted data.
Modern intelligent document processing (IDP) addresses this issue by processing layout and text together. Instead of stripping out spatial context, visual-first systems feed raw page images directly into vision-language models (VLMs) and visual vector indexes. At Lidex Soluções Digitais, we build document automation pipelines that replace coordinate templates with typed schema contracts, automated validation gates, and self-hosted inference.
1. Visual-first document ingestion: ColPali and multi-vector retrieval
Standard document retrieval for retrieval-augmented generation (RAG) and downstream extraction assumes text can be separated from its visual presentation. In practice, converting a PDF into plain markdown or text chunks strips away the spatial structure of balance sheets, multi-column reports, and nested key-value tables.
ColPali takes a different approach. Built on PaliGemma, it uses a lightweight VLM to generate multi-vector patch embeddings directly from high-resolution page images. Instead of producing one dense vector for a text chunk, the model maps visual patches from the page into the same semantic space as the query tokens.
Traditional Pipeline:
[Scanned PDF] ──> [OCR Engine] ──> [Text Chunker] ──> [Single Dense Vector] ──> Semantic Loss
ColPali Pipeline:
[Page Image] ──> [Vision Backbone] ──> [Multi-Vector Patch Embeddings] ──> [Late Interaction (MaxSim)]
For retrieval, ColPali uses ColBERT-style late interaction (MaxSim) to find the maximum similarity between each query token and every visual patch on the page. On standard benchmarks like ViDoRe (Visual Document Retrieval), this patch-level retrieval outperforms traditional OCR-then-embed pipelines by 20% to 30% on visually dense documents, including infographics, balance sheets, and multi-column legal filings. Skipping the OCR step prevents extraction errors from entering the pipeline before parsing starts.
2. Enforcing deterministic contracts with Pydantic and Instructor
Conversational responses work well for chat applications, but ERP systems like SAP and Oracle require strict types. An enterprise IDP pipeline cannot rely on a prompt asking the model to return valid JSON; it needs strict schema enforcement at decode time.
To bridge the gap between probabilistic vision models and deterministic databases, modern architectures enforce contract-driven extraction with libraries such as Instructor and Pydantic v2. Schemas are defined as typed Python classes with code-level validation rules instead of free-form prompts.
from pydantic import BaseModel, Field, field_validator
from typing import List
import re
class InvoiceItem(BaseModel):
description: str
quantity: float = Field(..., gt=0)
unit_price: float = Field(..., gt=0)
total_price: float = Field(..., gt=0)
@field_validator("total_price")
@classmethod
def verify_math(cls, v: float, info) -> float:
qty = info.data.get("quantity", 0)
price = info.data.get("unit_price", 0)
if round(qty * price, 2) != round(v, 2):
raise ValueError(f"Calculated line item mismatch: {qty} * {price} != {v}")
return v
class ExtractedInvoice(BaseModel):
invoice_id: str
tax_id: str
items: List[InvoiceItem]
net_amount: float
tax_amount: float
gross_amount: float
@field_validator("tax_id")
@classmethod
def validate_tax_id(cls, v: str) -> str:
clean_id = re.sub(r"\D", "", v)
if not clean_id:
raise ValueError("Tax ID cannot be empty.")
return clean_id
When a model returns a malformed tax identifier or outputs line items that do not add up to the parsed invoice total, Instructor catches the resulting ValidationError. Instead of failing outright or writing bad data to the database, the pipeline feeds the error message back into the model context in an automated retry loop. The model corrects its output using the visual grounding tokens, reaching near 100% schema compliance before any human operator has to intervene.
3. Handling multi-page packets with agentic workflows
Enterprise document sets rarely arrive as tidy single-page files. Mortgage files, insurance packets, and corporate tax packages often contain dozens or hundreds of pages combining IDs, bank statements, and distinct forms into one unindexed PDF.
Processing these bundles requires orchestration engines like Temporal or LangGraph, structured into distinct operational steps:
- Logical segmentation: Beginning-Inside-Outside (BIO) tagging segments the combined PDF into logical units. The model identifies where one document ends and another begins, such as separating a driver's license from an attached bank statement, without manual separator sheets.
- Grounded spatial extraction: Instead of outputting ungrounded strings, the visual extraction layer outputs normalized bounding-box coordinates (
[ymin, xmin, ymax, xmax]) for every field. This creates an audit trail for compliance and lets review interfaces highlight the exact source coordinates on the page. - Downstream integration: After extraction, the orchestrator runs deterministic checks against business logic (such as confirming that line items plus tax match the invoice total) and queries external ERP systems like SAP or Oracle to verify records against vendor master data.
4. Production economics and straight-through processing rates
Character Error Rate (CER) alone does not show how well a system works in production. The useful metric for enterprise ROI is the straight-through processing (STP) rate: the percentage of documents that move from ingestion to core ERP or database posting without manual review.
Traditional template and OCR setups usually plateau at STP rates between 40% and 60%. As soon as a vendor changes an invoice layout, coordinate heuristics break, sending documents to manual review queues.
Modern VLM-based pipelines reach 80% to 95% STP on recurring document types like invoices, bills of lading, and utility bills. Because multimodal models parse visual tokens and spatial hierarchy rather than fixed pixel coordinates, layout variations do not break the extraction flow.
This shift changes the operating numbers:
- Unit costs: Processing an invoice or complex document by hand costs $7.80 to $15.00 when accounting for labor, cycle times, and error corrections. An automated pipeline reduces machine parsing costs to between $0.03 and $0.30 per document. Even with human-in-the-loop exception handling included, median processing costs fall by 60% to 75%.
- Cycle latency: Turnaround time drops significantly. Handling times fall by up to 80%, with average per-invoice processing dropping from 15 minutes manually to under 2 minutes.
5. Dual-gated routing and human-in-the-loop review
A common issue in enterprise deployments is relying on raw model confidence scores alone. Vision-language models can generate incorrect values while assigning them high internal log probabilities.
Production IDP setups manage this by combining statistical probability with deterministic checks in a two-stage routing system:
- Confidence thresholds: Token logit probabilities must clear calibrated statistical thresholds, such as >0.92 for sensitive fields like bank routing numbers.
- Deterministic business rules: Extracted values must pass strict programmatic validation. The system matches tax IDs against ERP vendor master data and checks routing numbers or IBANs against mathematical check-digit algorithms.
If a document fails either the statistical threshold or any business rule, it bypasses automatic posting and moves into an exception queue for human review.
Using the bounding-box coordinates captured during extraction, the review screen displays the original page image with the predicted boxes and flagged errors side by side. Operators can resolve discrepancies without retyping entire forms. Those corrections are logged to build evaluation sets and fine-tuning data, preventing repeated failures on recurring templates.
6. Data sovereignty: running open-weight VLMs on-premise
For companies operating under regulations like Brazil's LGPD, Europe's GDPR, or US HIPAA, sending medical records, identification cards, bank statements, or M&A contracts to third-party SaaS APIs creates compliance risks.
Open-weight vision-language models offer an alternative. Models like Qwen2.5-VL (available in 7B, 32B, and 72B parameter sizes under open licenses), used alongside tools like Docling and olmOCR, let teams keep all document processing inside a private cloud or on-premise Kubernetes cluster.
A self-hosted setup typically pairs open weights with optimized serving infrastructure:
- High-throughput inference: Serving frameworks like vLLM or Triton Inference Server, combined with FP8 quantization and window attention, deliver sub-second per-page extraction on standard enterprise hardware like NVIDIA L40S or H100 GPUs.
- Data-segregated routing: Upstream proxies like LiteLLM or internal routing services direct low-risk documents to cloud endpoints while keeping documents containing personal data on local, isolated compute nodes.
Building production document pipelines
Automated document processing has shifted away from fragile regex rules on raw OCR text. Production setups treat documents as visual files first, extract data using vision-language models, enforce typed contracts through libraries like Pydantic and Instructor, and guard downstream databases with automated validation gates and human review workflows.
Reaching 80% to 95% straight-through processing requires integrating visual inference, deterministic backend services, and regulatory controls. At Lidex, we build custom document pipelines and enterprise integrations that replace manual review with auditable automated systems. Contact our engineering team to evaluate an architecture for your production workflows.