Buying Guides
How to Evaluate Legal AI Vendors: 12 Questions to Ask Before You Buy
Legal AI procurement has a specific failure mode: the demo is curated, the pilot is unstructured, and six months later the tool is shelfware because it never fit the actual workflow. The fix is to walk into the evaluation with questions the demo cannot dodge.
Here are twelve, grouped into the four areas that decide outcomes: output quality, security, workflow fit, and commercial terms.
Output: questions 1–4
Output questions come first because they are the fastest disqualifiers.
- 1. Can I run it on my own documents during the evaluation? A vendor that resists this is telling you something.
- 2. What format is the output? Native .docx with tracked changes is the standard for transactional work; citation-backed answers are the standard for research.
- 3. Does it work from my precedent and playbook, or only the vendor's forms?
- 4. How does it handle what it doesn't know? Good systems flag uncertainty for attorney review; bad ones guess fluently.
Security: questions 5–7
Run security in parallel with the product evaluation, not after — a failed security review means the product evaluation was wasted time.
- 5. Is client data used to train models — by you or your upstream model providers? The only acceptable answer is a contractual no.
- 6. What is your audit posture? SOC 2 with continuous monitoring is the institutional baseline; ask to see the trust center.
- 7. Where do documents live, who can access them, and what are the retention and deletion guarantees?
Workflow fit: questions 8–10
Tools fail in the seams between themselves and the way lawyers actually work. These questions probe the seams.
- 8. What does the round trip look like? Draft in the tool, edit in Word, return to the tool — does anything break?
- 9. How do attorneys supervise the AI? Look for review gates and audit trails, not just an undo button.
- 10. What happens on day one of a real matter? Ask the vendor to walk your actual matter type end to end, not their standard demo.
Commercial terms: questions 11–12
Pricing structure predicts adoption. Per-seat pricing on a tool only partners use monthly is a very different spend than flat pricing a whole team can share.
- 11. What is the pricing model — per seat, per matter, usage-based, or flat — and what happens at renewal?
- 12. What does a pilot cost, and what are the success criteria? Define them in writing before starting: specific matters, specific outputs, specific reviewers.
Scoring the answers
Weight the disqualifiers first: training on client data, no path to running your own documents, and output that cannot be reviewed as tracked changes each end the conversation regardless of other strengths. Among tools that clear the bar, workflow fit is the best predictor of adoption a year out — the tool that lives where the work already happens beats the tool with the better model but the second browser tab.
Frequently asked questions
How long should a legal AI pilot run?
Four to six weeks with defined success criteria is usually enough: real matters, named reviewing attorneys, and a written definition of what output quality clears the bar. Open-ended pilots select for inertia rather than evidence.
What is the biggest red flag when evaluating legal AI?
A vendor that cannot or will not let you run the tool on your own documents during evaluation. Second is any ambiguity about whether client data trains models. Both are disqualifiers regardless of how strong the demo is.
Should small firms evaluate AI differently than large firms?
The questions are the same; the weights differ. Small firms should weight flat, predictable pricing and fast time-to-value more heavily, since they lack the innovation teams to shepherd long deployments. The security bar does not lower with firm size — your clients' expectations set it, not your headcount.