The AI vendor landscape is noisy. Every product claims to be enterprise-grade, secure, and ready to deploy in regulated environments. Most of them are not — or at least, not without significant integration, configuration, and governance work that the sales process will not surface.
This article is a practical guide for procurement, risk, and technology teams evaluating AI vendors in professional or regulated contexts. It is based on the questions that consistently separate capable vendors from ones that create liability.
The problem with most AI vendor assessments
Standard vendor assessments were designed for software with deterministic behaviour. You test it, it either works or it doesn't, and you document what you found. AI systems are different in ways that matter for procurement:
- They can produce outputs that are plausible but wrong, with no error signal
- Their behaviour can change when the underlying model is updated — sometimes without notice
- Their performance on your data may be significantly different from their performance on benchmark data
- The risk is not just that they fail — it is that they fail in ways that are hard to detect
A good vendor assessment for AI has to account for all of this. The questions below are structured around what you actually need to know before committing.
Data and privacy questions
Ask: Is our data used to train or fine-tune your models?
A credible vendor should be able to answer this clearly and in writing. "No, your data is not used for training" should be contractually enforceable, not just a sales assertion. If the answer is vague or qualified, treat it as a yes.
Ask: Where is our data processed and stored?
Residency matters for regulated industries. EU clients need EU processing for most personal data. Healthcare clients may need additional guarantees. "We're hosted on AWS" is not an answer — region, jurisdiction, and sub-processor list matter.
Ask: What happens to our data if we terminate the contract?
Deletion timelines, export capabilities, and what "deletion" actually means (including backups) should all be documented before you sign.
Model and performance questions
Ask: Can you provide evaluation results on data similar to ours?
Benchmark scores are near-useless for regulated use cases. A model that performs well on general knowledge tasks may perform poorly on domain-specific tasks involving legal language, clinical notation, or financial instruments. Ask for evidence of performance in your domain, or build in a paid evaluation period before committing.
Ask: How are model updates communicated, and what is your notice period?
Model behaviour can change with updates. In a governed deployment, you need to know when this happens so you can re-evaluate. A vendor with no change notification process is a governance risk regardless of how good the model is today.
Ask: What is your process when the model produces a harmful or incorrect output?
This reveals how seriously a vendor takes operational responsibility. A good answer includes: a clear reporting channel, a defined SLA for investigation, and a history of how previous incidents were handled. A vague answer is a red flag.
Security and access questions
Ask: What security certifications do you hold?
ISO 27001 and SOC 2 Type II are baseline expectations for enterprise vendors. If neither is held, ask why — some early-stage vendors are in the process, but you need a timeline and should treat the gap as a risk to price into the engagement.
Ask: Do you offer a private deployment option?
For sensitive data, a shared cloud deployment may not be acceptable even with strong contractual protections. Private deployment — whether on your infrastructure or a dedicated tenancy — adds cost but removes a class of risk. Know whether it is available and what it costs before the evaluation progresses.
The questions vendors cannot answer well — and what that tells you
The most informative part of a vendor evaluation is often what they cannot answer. The following questions are routinely poorly answered, and the quality of the response tells you a great deal:
- "What are the known failure modes of this model for our use case?" — A vendor who cannot describe failure modes has not characterised them. That means you will discover them in production.
- "Can you provide a model card?" — Model cards are a standard practice for responsible AI deployment. A vendor without one either does not use the underlying model carefully or is not oriented toward enterprise governance.
- "Who is contractually responsible if the model produces an output that causes harm?" — This question surfaces liability posture. Many AI contracts include broad indemnification carve-outs that put all liability on the customer. You need to know this before you sign.
The vendors worth working with are the ones who answer these questions directly, including the uncomfortable ones. Evasion at the sales stage becomes operational risk after signature.
A note on integration costs
AI vendor assessments tend to focus on the product and underweight the integration. The total cost of deploying an AI system includes: integration engineering, data pipeline work, prompt development and testing, governance documentation, staff training, and ongoing monitoring. These costs are rarely in the vendor's quote and routinely exceed the licence fee in year one.
Build them into the business case before the evaluation is complete. A cheaper vendor with a worse integration story often costs more than a more expensive vendor with mature enterprise tooling.
When to bring in independent assessment
For AI deployments that will inform consequential decisions — clinical, legal, financial, or operational — an independent technical assessment before sign-off is worth the cost. An assessment that maps the vendor against your regulatory environment, reviews the contractual terms for liability exposure, and documents the governance gaps is the kind of artefact that a risk committee or regulator will want to see.
It also frequently changes the outcome of the procurement — either by surfacing a vendor that looks strong but has critical gaps, or by providing the evidence needed to proceed with confidence.