Evaluate an enterprise AI vendor on one question: can they get a working system into production inside your regulatory and operational constraints, and stay accountable for it once it is live. Most cannot. A Gartner survey of 782 infrastructure and operations leaders (November to December 2025) found that only 28% of AI use cases fully succeed and meet ROI expectations, while 20% fail outright. So the demo is close to irrelevant. What matters is whether the vendor can name the failure modes before you hit them, price the cost of running the system, and put their engineers next to your workflow rather than mailing you a model. This checklist gives CXOs the questions that separate the two.
Why most enterprise AI vendor decisions go wrong
The base rate is brutal, and it is worth internalising before any vendor walks into the room. MIT's NANDA initiative, in The GenAI Divide: State of AI in Business 2025, reported that roughly 95% of enterprise generative AI pilots delivered no measurable impact on the P&L. The same study found that buying from specialised vendors and building the partnership succeeded about 67% of the time, while internal builds succeeded roughly a third as often. S&P Global (October 2025) added that the share of companies abandoning most of their AI initiatives rose from 17% to 42% in a single year.
Read those numbers together and the pattern is clear. The technology mostly works in the lab. The failure happens in the gap between a pilot and a production system that survives contact with real users, real data, and real compliance obligations. A good vendor evaluation is a way of stress-testing that gap before you sign, rather than discovering it eighteen months in.
The mistake CXOs make is evaluating the model. Models are close to a commodity now, and they change every quarter. You are buying the engineering discipline, the domain understanding, and the operating commitment that wrap the model. That is what this checklist measures.
What should be in a CXO checklist for evaluating an AI vendor?
Work through these in order. Each one is a filter, and a vendor who fumbles the early ones rarely recovers on the later ones.
1. Can they show a production deployment in a regulated setting, with a named client and a metric? You want a system real users depend on today, rather than a pilot, a hackathon, or a slide. Ask who the client is, what the system does, and what number moved. If every reference is a proof of concept, you are their production experiment. Gartner (2024) predicted that 30% of generative AI projects would be abandoned after the proof of concept stage by the end of 2025, so "we ran a successful POC" is a low bar that most of the market clears and then stalls at.
2. Do they understand your domain before they understand their product? In regulated industries the constraint is rarely the model. It is the clinical protocol, the audit trail, the consent flow, the language mix on the floor. A vendor who opens with their platform architecture before asking how your OPD actually runs is selling you their roadmap. A vendor who asks sharp questions about your workflow is trying to build for it.
3. Who exactly will be in the room during the build? Ask for the names and seniority of the people who will do the work, rather than the org chart of the company. Enterprise AI is a forward deployed activity: engineers sit inside your process, watch it, and instrument it. If the sales engineer is impressive and the delivery team is a rotating pool of juniors you never meet, the quality you saw in the pitch will not survive the handoff.
4. How do they price the running cost rather than the build cost? Token spend, inference, caching strategy, model routing, and the human review loop are the real cost of an agentic system in production, and they recur every month. Ask a vendor to model your cost per transaction at your expected volume. A serious partner has done this maths and will show it to you. A weak one quotes a build fee and goes quiet on operations.
5. What is their compliance posture under DPDP, and can they prove it? Under India's Digital Personal Data Protection Act 2023, the Data Fiduciary stays legally accountable for a breach even when a processor caused it. So your vendor's security practice is your liability. Ask where data is stored, how personal data is minimised and deleted, whether sub-processors are disclosed, and whether they will sign a DPDP-compliant processing agreement with breach-notification and deletion clauses. Vague answers here are disqualifying.
6. How do they handle the human handoff and the failure case? Every accountable system knows when to stop and escalate to a person. Ask what happens when the agent is uncertain, when it hits an edge case, and when it is simply wrong. A vendor who claims full autonomy in a clinical or financial workflow is either inexperienced or selling risk you will own.
7. Do they measure error at each step, or only the final output? Multi-step agentic systems compound small errors. A 95% accurate step, run five times in sequence, drops below 80% end to end. Ask how they instrument each stage, how they catch drift, and how they know the system is degrading before your customers tell you.
8. What does exit look like? Ask who owns the prompts, the fine-tuned weights, the evaluation datasets, and the integration code if you part ways. A partner confident in the relationship writes clean exit terms. Lock-in dressed up as a platform is a cost you pay later.
How is evaluating an AI vendor different from evaluating a software vendor?
Traditional software is deterministic. You test it, it passes, it behaves the same way on Tuesday. An AI system is probabilistic, so it can pass every test in the demo and still produce a wrong answer in production because the input distribution shifted. This changes what you are buying and how you check it.
Three differences matter most. First, the system needs continuous evaluation, so you are buying an ongoing relationship rather than a one-time licence. Second, the data governance stakes are higher, because the system learns from and acts on your most sensitive records, which pulls DPDP and, in health, ABDM and NMC obligations directly into the contract. Third, the value shows up in a changed workflow rather than an installed feature, which means the vendor has to understand operations deeply enough to redesign a process, rather than deep enough to ship a screen.
A vendor who treats an AI engagement like a software licence will hand you a model and disappear. That is precisely the handoff where the 95% pilot-failure rate lives.
What questions expose a weak enterprise AI vendor fast?
Four questions do most of the work in a first meeting.
"Walk me through a deployment that went wrong and what you changed." A practitioner has scar tissue and will tell you about it. A vendor who has never had a project struggle has never shipped anything hard.
"What is my cost per transaction at production volume, and how does it change as we scale?" This forces them off the build fee and onto the economics that actually determine whether the system survives budget review next year.
"Which parts of this should we not automate yet?" A partner optimising for your outcome will name the parts that are not ready. A vendor optimising for contract size will tell you everything is ready now.
"When the model is wrong, who finds out, and how fast?" This tests whether they have built real observability or whether they are hoping the model behaves.
If the answers are confident, specific, and slightly uncomfortable, you are talking to a builder. If they are smooth and reassuring on every point, you are talking to a sales motion.
How Nextdot approaches this
We built Nextdot as a forward deployed engineering practice for regulated industries, which means our engineers sit inside the client's workflow through the build and stay accountable after it is live. Our voice-first CX agents run in production at Narayana Health and Gleneagles and are in build at Fortis Mulund, and we work with pharma and clinical organisations including Mankind Pharma, Wockhardt, and Clove Dental. NextComply AI, our compliance co-pilot for regulated industries, is in beta and paid POCs. We are roughly 30 people, and we would rather scope a hard problem honestly than win a project we cannot land in production. If you are running the checklist above against a shortlist, we are happy to be one of the names on it.
Frequently asked questions
How long should evaluating an enterprise AI vendor take?
Plan for four to eight weeks for a serious enterprise decision. The useful signal comes from a short, paid, scoped engagement on a real slice of your data, rather than from more demos. If a vendor will not do a small paid proof against your actual constraints, that is information.
Should we build in-house or buy from a vendor?
MIT's State of AI in Business 2025 found that vendor partnerships reached production success roughly 67% of the time, about three times the rate of internal builds. For most enterprises the honest answer is a hybrid: buy the engineering and orchestration from a partner while building internal capability to own and govern the system over time.
How do we evaluate an AI vendor on compliance in India?
Ask for their data flow diagram, their sub-processor list, and their willingness to sign a DPDP-compliant processing agreement. Under the DPDP Act 2023 you remain the accountable Data Fiduciary, so their security posture becomes your legal exposure. In healthcare, confirm alignment with ABDM data standards and NMC guidance where the workflow is clinical.
What is the single biggest red flag?
A vendor whose references are all pilots and whose delivery team you never meet. Production experience and named senior engineers on your account are the two hardest things to fake, and the two most predictive of success.
How much should we budget for running the system rather than building it?
Model the recurring cost separately from the build: inference and token spend, caching, model routing, monitoring, and the human review loop. For agentic systems the running cost often rivals the build cost within the first year, so a vendor who cannot quantify it has not run one at scale.