A frontier model that scores 93.1% on the MedQA benchmark (arXiv, 2025) can still invent a drug interaction that does not exist, and in a compliance review that single fabrication is the whole risk. For hospital and pharma compliance work, the right question is rarely "which model is smartest." It is "which model behaves predictably on my documents, keeps patient data inside my walls, and produces an output I can defend to a regulator." On all three counts, a domain-specific or small language model, tuned and hosted for the task, tends to beat a general giant reached over a public API. This post is what we learned building one.
What does "small" and "domain-specific" actually mean here?
A small language model (SLM) is a model with far fewer parameters than a frontier system, small enough to run on modest on-premise hardware or a private cloud tenant. Domain-specific means it has been trained or fine-tuned on the language of a single field: pharmacovigilance narratives, adverse event reports, informed-consent templates, promotional-material rules, ABDM data-handling policies. The two properties travel together. You make a model smaller by narrowing what it needs to know, and you narrow it by pointing it at one domain.
The trade the industry assumes is that smaller means dumber. For open-ended medical reasoning, larger models do lead. Nature reported that small models trained directly on medical textbooks picked up stronger reasoning than their size would predict (npj Digital Medicine, 2025), which is the more useful signal for compliance. Compliance is a bounded task. You are checking a document against a known rulebook rather than writing a differential diagnosis from scratch. A model that has read your rulebook a thousand times will out-perform a bigger model that has read it zero times.
Why not just use the biggest general model?
Three reasons, in the order they hurt.
First, hallucination. Medical LLM hallucination rates have been reported above 60% without grounding, and between 43% and 67% depending on case complexity and whether a mitigation prompt is used (MedRxiv clinical benchmark, cited via SQ Magazine, 2026). Grounded, task-scoped systems do far better: one clinical safety framework measured a 1.47% hallucination rate and a 3.45% omission rate across 12,999 clinician-annotated sentences in note generation (PMC, 2025). The gap between those numbers is the entire argument. Compliance output is read by an auditor, so a plausible-but-false line is worse than a blank. A narrow model constrained to your document and your rules gives fewer places for invention to hide.
Second, data residency. Sending patient records or unpublished pharmacovigilance data to a public API means your most sensitive information leaves your control. Under the DPDP Act 2023, every hospital, lab, and telemedicine platform now carries explicit duties for how it collects, stores, and shares patient information, and ABDM-integrated health data must be stored in India under the Health Data Management Policy (Truecopy DPDP healthcare guide, 2025). A model you host inside your own environment keeps the data where the regulation expects it. SLMs make this practical because they run on-premise or on-device, which reduces the leakage risk of shipping sensitive data to the public cloud (CloverDX, 2025).
Third, cost and predictability. Compliance is high-volume and repetitive: thousands of documents, the same checks each time. For that shape of work, SLMs can cut inference cost by up to 90% versus large cloud models while giving near-instant latency, and a locally deployed model carries a fixed, predictable cost once it is running (Ajith Prabhakar, 2025). Predictable spend matters to a CXO signing off on a system that will run every day for years.
What did we learn building one?
We built a compliance-scoped model to review regulated documents, and a few lessons held across every iteration.
The rulebook is the product. The model is a smaller part of the work than most people expect. Most of the effort went into turning fuzzy policy language into structured, testable checks: what a valid consent clause contains, what claims a promotional piece may make, what an adverse-event narrative must record. Once the rules were structured, a modest model could apply them reliably. Structured prompting alone has been shown to cut hallucinations by roughly a third, with medical AI research reporting a 33% reduction from structured prompts (SQ Magazine, 2026), and grounding the model in an explicit rulebook compounds that.
The model must show its work. A compliance answer of "this document passes" is useless. The output that earns trust points to the exact clause, quotes it, names the rule it satisfies or breaks, and flags what it could not verify. We treat an "I am not certain" from the model as a feature. An honest abstention routes a document to a human, which is exactly the handoff a regulated workflow needs.
Keep a human accountable, by design. The model triages and drafts. A named reviewer signs. This is not a limitation to apologise for. Under NMC and DPDP expectations, accountability sits with a person, and the system should make that person faster while keeping the decision theirs. Reviewer acceptance is the metric we optimise, more than raw benchmark accuracy.
Evaluate on your documents rather than on a leaderboard. Public benchmarks predict very little about performance on your consent forms or your pharmacovigilance backlog. We built a graded test set from real, de-identified documents with known correct answers and measured against that on every change. A model that looks worse on MedQA can be the better compliance reviewer on your corpus, and the only way to know is to test on the corpus.
Where does a small model fit, and where does it not?
Be honest about the boundary. Narrow models are strong when the task is bounded and the rulebook is knowable: consent completeness, promotional-claim checking against approved language, adverse-event narrative structuring, ABDM data-handling review. They are weaker on open-ended clinical judgement, novel reasoning, or anything requiring broad world knowledge the model was never trained on. For those, a larger model has its place.
The design most teams land on is a hybrid: a small compliance-aware model handles the sensitive, repetitive, in-house checks, and a larger model is called only for the harder, less sensitive reasoning (InfoWorld, 2025). Match the model to the task rather than defaulting the whole pipeline to the biggest available system.
This is the thinking behind NextComply AI, our compliance co-pilot for regulated industries, currently in beta and paid POCs. It reviews regulated documents against a structured rulebook, cites the clause behind every finding, and hands anything uncertain to a named reviewer. The point is a system a compliance leader can defend, deployed where the data already lives.
What should a compliance leader ask before buying?
Ask where the data goes and whether it ever leaves your environment. Ask whether the model cites the specific clause behind each finding or only returns a verdict. Ask how the vendor measured accuracy, and insist that they measure on your documents before you sign. Ask who is accountable for a wrong answer and how the human handoff works in practice. Ask what the cost looks like at your real document volume over three years, rather than per API call in a demo. The answers separate a defensible deployment from a demo that will not survive an audit.
Frequently asked questions
Is a small language model less accurate than a large one for compliance?
Not on a bounded task with a known rulebook. Frontier models lead on open-ended medical reasoning, but compliance is a checking task. A model tuned on your rules and grounded in your documents often out-performs a larger general model on your actual corpus. Test both on your own graded document set before deciding.
Does a domain-specific model help with DPDP Act compliance?
It helps mainly through deployment. A small model can run inside your own environment, so patient data and sensitive records stay within your control and, where ABDM applies, within India. That aligns with DPDP 2023 duties around storage and sharing far better than sending records to a public API.
Can a language model be the final decision-maker in a compliance workflow?
No. Under NMC and DPDP expectations, accountability rests with a named person. The model should triage, draft, and cite evidence to make a human reviewer faster while keeping the decision and the sign-off with that reviewer.
How do you stop the model from inventing rules or findings?
Constrain it to a structured rulebook, require it to quote the source clause for every finding, and let it abstain when uncertain so the document routes to a human. Grounding and structured prompting measurably reduce hallucination compared with an open-ended query.
Is it cheaper to run a small model in-house than to call a large model API?
For high-volume, repetitive compliance work it usually is, because a locally hosted model gives a fixed, predictable cost and low latency. The saving grows with volume. Model the total cost at your real document throughput over several years rather than judging by a single API call.