Invoice understanding model & MLOps pipeline (in progress)
Chalaniq
A document-AI system that reads Indian invoices (PDFs, scans and phone photos) and returns validated structured data, comparing rules, a layout model and a fine-tuned vision-language model.
My role: ML engineer: data generation, modelling, validation, serving and MLOps (project in progress)
- PyTorch
- LayoutLMv3
- Qwen2.5-VL (LoRA)
- ONNX Runtime
- MLflow
Overview
Chalaniq turns invoices into clean, validated data: seller and buyer details, GSTINs, invoice number and date, line items, taxes and totals. It is built as a full ML project rather than a single prompt: a synthetic data generator, several extraction approaches compared on the same schema, GST-aware validation, and an MLOps setup with experiment tracking, monitoring and a human-review loop. The working demo already extracts synthetic invoices with both the baseline and the vision-language model, with validation and review; training and benchmarking of the models are in progress.
The challenge
Invoice processing is still largely manual in Indian accounting teams. Layouts vary endlessly, many invoices arrive as poor scans or phone photos, and the data must satisfy strict rules: valid GSTINs, the right split between CGST, SGST and IGST, and totals that add up. Generic OCR tools miss tables and fields, and sending every page to a large vision model is slow and expensive.
Chalaniq aims to extract invoices accurately and cheaply, prove every result with validation checks, and keep improving from human corrections, while being honest about where each approach succeeds or fails.
How it works
- 1PDF / scan / photo
- 2Native text or OCR + boxes
- 3Rules → LayoutLMv3 → Qwen2.5-VL
- 4GST validation
- 5Needs-review flag
- 6Corrections → retraining
Architecture
Documents enter as PDFs, scans or photos. Digital PDFs give native text and coordinates; images go through PaddleOCR or Tesseract. A 2D tokenizer normalises words with their bounding boxes. Extraction is tiered: rule and regex heuristics handle simple, regular layouts; a LayoutLMv3 token classifier tags fields using text, position and page image; and a LoRA-tuned Qwen2.5-VL produces the JSON schema directly for complex layouts. The layout model is exported to ONNX with INT8 quantisation for CPU serving.
Every output passes the same validation engine: GSTIN structure and checksum, seller-state match, intra- vs inter-state tax type, line maths, totals, amount in words and date order, with Indian and Western number formats parsed. Failed checks or low confidence set a needs-review flag. Training data comes from a synthetic generator built on HTML templates with realistic GST logic and Augraphy scan degradation, so no private documents are needed.
The service is an async FastAPI API with a Gradio demo for uploading invoices, viewing extracted fields, line items and validation results, and submitting corrections. MLflow tracks experiments and model versions, Hydra holds configs, Prometheus records latency and errors, and KS-test and PSI checks watch for drift. Corrections from the review panel feed an active-learning dataset for retraining.
What I built
Tiered extraction design
Rule and regex heuristics for simple invoices, a LayoutLMv3 token classifier over OCR text and 2D bounding boxes, and a LoRA-tuned Qwen2.5-VL for complex or unstructured layouts.
Synthetic invoice generator
Realistic, labelled Indian invoices from HTML templates with valid GSTIN structure, GST tax logic and amounts in words, aged with Augraphy scan effects such as blur, folds, stains and perspective.
GST-aware validation
GSTIN format and checksum, seller-state match, CGST/SGST vs IGST, line maths and totals within ₹1, amount in words and date checks, with each failure explained.
Built for CPU serving
The layout model is exported to ONNX with INT8 quantisation, so most invoices can be processed on CPU while the vision-language model handles the hard cases.
Human review and active learning
Low-confidence fields or failed checks raise a needs-review flag; reviewer corrections feed a dataset for the next training round.
MLOps
Async FastAPI service, MLflow tracking and registry, Hydra configs, Prometheus metrics, drift detection (KS test and PSI), Docker, and a Gradio demo.
Engineering decisions
- Generate synthetic invoices rather than using anyone's private documents, and keep a separate real test set only for final scores.
- Compare approaches on one schema and one evaluation, so accuracy, speed and cost can be weighed honestly.
- Route only uncertain documents to the expensive model or a human, using validation results and confidence.
Results
- Working demo: baseline and vision-language-model extraction to the invoice schema, with all validation checks passing on synthetic invoices.
- Human-review panel that sends corrections back for active learning.
- Model training and benchmarking in progress.
- Benchmark results will be published from a held-out real test set once measured.
Stack
- Models
- PyTorch, Hugging Face Transformers, LayoutLMv3, Qwen2.5-VL with LoRA / PEFT
- OCR & data
- PaddleOCR, Tesseract, pypdf, Augraphy, synthetic invoice templates
- Optimisation
- ONNX Runtime, INT8 dynamic quantisation
- Serving
- FastAPI (async), Gradio demo, Docker Compose
- MLOps
- MLflow, Hydra, Prometheus, KS-test / PSI drift detection, active-learning review queue
Related services
Need something like this?
I can build a version of this for your product, your data and your stack.
Start a project