Live Benchmark · Target EMNLP 2026

IndiaFinBench

A reproducible benchmark for LLM performance on Indian financial regulatory text — SEBI circulars and RBI master directions.

ProblemNo benchmark existed to test whether LLMs can be trusted on Indian financial regulation — dense, cross-referenced text where a confident wrong answer is costly.

ApproachBuild 406 expert-annotated questions from real SEBI and RBI documents, then score 12 leading LLMs against a human-expert baseline.

ResultEvery model beat the human baseline; the best, Gemini 2.5 Flash, reached 89.7% versus 69% for human experts.

SignificanceThe first evidence that today's LLMs can outperform expert humans on this specific, high-stakes regulatory domain.

406Expert QA items

12LLMs evaluated

89.7%Top model (Gemini 2.5)

69%Human-expert baseline

The Gap

No public benchmark existed for how LLMs handle Indian financial regulatory text — a domain dense with numerical thresholds, cross-references, and temporal amendments where a confident wrong answer is costly.

Construction

From 192 SEBI/RBI documents I built 406 expert-annotated QA items across four task types — regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning — with three-round inter-annotator agreement. Twelve LLMs are scored closed-book with bootstrap significance and Wilson confidence intervals, plus a hybrid BM25+dense RAG demo.

Outcome

Every model clears the 69% human-expert baseline; Gemini 2.5 Flash leads at 89.7%, with statistically distinct performance tiers. The benchmark and leaderboard are live on HuggingFace for reproducible evaluation.

Methodology Pipeline

Key Result

Gemini 2.5 Flash (top model)89.7%

Human-expert baseline · 69%

All 12 models scored above the human-expert line — Gemini cleared it by 20.7 points

↗ View on HuggingFace