Written by Vendortell - the Contract Performance Management platform. We've been on both sides of the AI-vendor evaluation table - the questions that matter aren't the ones vendors want you to ask.
The words 'AI-powered' now appear in essentially every contract management vendor's marketing. What sits under those words varies enormously - from mature productized AI with measurable accuracy on extraction, to marketing wrapper around commodity APIs, to pattern matching that would have been called 'rules-based' three years ago.
The buyer's job during evaluation is to figure out which one is on the other side of the table. The following ten questions do that.
Question 1 - What model powers the extraction?
Vendortell is the Contract Performance Management platform. Our AI has extracted and matched terms across 10,000+ contract books against live transactional data - with the data hygiene to make it trustworthy at scale.
That's why we can offer specific questions to ask contract-management vendors about AI - because we answer them ourselves, in the same evaluations.
Honest answers: 'GPT-4 via OpenAI API', 'a fine-tuned Llama derivative we run internally', 'proprietary transformer ''we trained on contract data', 'hybrid - pattern matching plus LLM for edge cases'. All of these are legitimate architectures.
Warning signs: vague answers about 'our proprietary AI', unwillingness to name the underlying technology, or claims that the AI 'evolves' without specifics.
The technology is not the differentiator - the honesty about what it is, is.
Question 2 - Where is our data sent during extraction?
If the vendor uses a foundation model API (OpenAI, Anthropic, Google), your contract text goes through that provider. Ask: which provider, in which region, under what data processing agreement, and with what retention.
For EU-headquartered buyers, this matters for GDPR posture. For contracts with confidential pricing, this matters for commercial secrecy. Both concerns are legitimate; both have resolution paths.
Vendors who cannot answer this clearly are exposing you to compliance risk they may not even be aware of.
Question 3 - What is the extraction accuracy on our contracts?
The right measurement is not vendor's demo accuracy - it is accuracy on your contracts. Provide five to ten contracts (redacted if needed) and ask for measured accuracy of the extraction against a human-verified ground truth.
Numbers to expect: 90%+ on standard commercial terms in well-drafted contracts; 70-85% on complex or bespoke language. Vendors claiming 99%+ across all contracts are either over-selling or over-simplifying what they measure.
Question 4 - What happens when the extraction is wrong?
This is the most important question in the list. The answer reveals the workflow design around the AI:
Good answer: 'Low-confidence extractions are flagged for human review before entering the reconciliation pipeline. High-confidence extractions can be reviewed on demand. Every extraction is auditable to the source document.'
Concerning answer: 'Our accuracy is very high so this rarely happens.'
Good AI workflows treat extraction errors as expected and manageable. Sales-first workflows treat them as embarrassing and hide them.
Question 5 - How do you handle amendments and side letters?
Amendments modify the original contract. A well-designed system extracts them, maintains the parent-child link, and produces a 'current effective' view combining base contract plus all amendments.
A poorly designed system extracts amendments as separate contracts, or ignores them entirely, or requires manual linkage - all of which produce reconciliation errors downstream.
Question 6 - What does the model do when your ERP data doesn't match the contract terms?
Mismatches happen constantly - a supplier invoices at a different price than the contract specifies, a purchase order references a superseded rate card, an ERP master record differs from the contract counterparty name.
Ask specifically what the platform does when it detects these. The good answer is 'flags for review, does not auto-book, generates a reconciliation report.' The bad answer is 'trusts the ERP as source of truth' - which defeats the purpose of having a contract layer at all.
Question 7 - How is the model updated?
Language models get updated. So do the templates and pattern libraries that support them. Ask:
When you update the model, does existing extracted data get re-processed, or does it stay under the old model? What is the customer-visible change management for model updates? Is there a way for the customer to lock a model version for stability?
These questions matter because a silent model update can change your accrual calculations mid-year.
Question 8 - What is the audit trail for AI decisions?
External auditors and internal compliance functions increasingly ask about AI-influenced financial outputs. Ask the vendor whether:
- Every extraction can be traced to the source document location
- Model version and confidence score are logged per extraction
- Human overrides are captured with reason codes
- The end-to-end chain from PDF to journal entry is queryable
For anything that touches financial reporting, this is not optional.
Questions 9 and 10 - Cost model and roadmap
9. Cost model. Is AI extraction priced per contract, per document page, per token consumed, or bundled? Understand the marginal cost of scaling.
10. Roadmap and risk. What happens if your vendor's underlying foundation model provider changes terms or prices? What is the vendor's continuity plan? A vendor whose entire product depends on one API is a different risk profile than one running in-house models.
FAQ
Should we prefer vendors with in-house AI over foundation-model wrappers?
How do we test the answers we get?
What's the risk profile of AI in contract management specifically?
How does this connect to Contract Performance Management?
