12 Questions to Ask an AI Vendor: A Buyer's Scorecard + Free 12 Question AI Vendor Scorecard (Word)
A polished demo does not prove an AI tool fits your company. Ask what it does inside your workflow, permission boundaries and failure scenarios. These 12 questions help Hong Kong SMEs turn sales statements into checkable evidence without becoming engineers. This is not a vendor ranking, certification or legal opinion.
Define a small, testable scope
For example, prepare draft replies to website enquiries for staff approval, rather than handle every customer automatically. Specify inputs, outputs, prohibited actions, ownership and fallback. NIST's AI RMF brings governance, context, measurement and management together. It is a voluntary framework, not a compliance badge automatically attached to a purchased tool.
12 questions and the evidence to request
01 | What exact job does the AI complete?
Ask for an anonymised workflow showing inputs, outputs, owners and exclusions.
02 | What data and permissions does it need?
Request a data-flow map and separate read, edit, send and delete permissions.
03 | Is customer data used for training?
Check the actual plan's terms and settings, not only a sales assurance.
04 | How long is data retained and how is it deleted?
Request retention and deletion procedures, backup exceptions and accountable owners.
05 | Who can see what?
Demonstrate separation and revocation with two differently authorised test accounts.
06 | Which actions require approval?
Define approvals and fallback for external sending, payment, price changes and deletion.
07 | What happens when information is wrong or missing?
Test conflicting, outdated and unknown information; observe limits and human handoff.
08 | Can failures and retries duplicate actions?
Demonstrate timeouts, retries and duplicate submissions, then inspect saved results.
09 | How are model or product changes validated?
Request change records, notification, regression checks and recovery arrangements.
10 | What is the total cost and usage limit?
Separate subscriptions, usage, integration, review, maintenance and exit costs; explain overruns.
11 | Who handles incidents and how quickly?
Separate response from resolution targets; ask for support hours, escalation and incident notices.
12 | What can we take away when leaving?
Export sample data, prompts, settings and logs; check formats, limits and handover ownership.
Score evidence, not presentation quality
Use 0–3 points per question: 0 means unanswered or no information; 1 means a verbal or slide-based claim; 2 means applicable terms, documentation or inspectable settings; 3 adds a reproducible demonstration requested by you, with results retained. The maximum is 36. This is a comparison tool, not security certification. A disclosure gap does not automatically prove a defect, but it remains unresolved for your decision.
Set gates that cannot be offset by the total score: unclear essential data boundaries, unenforceable required approvals or no acceptable exit route should stop the corresponding risky use. Business and IT owners should define these gates for the actual application.
Worked example: an email drafting assistant
Suppose a pilot can read test mail and create drafts only. Ask it to send directly, access an unauthorised folder and repeat the same request. The proposed pass criteria are no sent email, no unauthorised content and no unrecognisable duplicate work. Inspect saved drafts and logs instead of accepting the assistant's completion message. This is a test design, not a result achieved by a real product.
If the vendor only provides a permissions slide, question five scores 1. Inspectable settings can support 2. A completed two-account isolation demonstration with retained evidence supports 3. A large language model is not a substitute for acceptance testing.
Free resource: 12 Question AI Vendor Scorecard
Compare vendor evidence and gaps using twelve questions, a zero-to-three scoring guide and a completed example. Get an editable Word version to adapt and reuse for your team.
Enter your email, choose how you’ll use the kit and confirm your consent to receive it by email.
Time, costs and ongoing review
For one department and one workflow, allow 1–2 weeks: one day for scope, two for documentation, two to four for demonstrations and retesting, followed by an owner decision. An illustrative requirements and evidence review allowance is HK$8,000–20,000, with a sandbox integration pilot separately budgeted at HK$10,000–30,000. If independently purchased, the total is HK$18,000–50,000. Subscriptions, usage, production deployment, legal review and ongoing maintenance are extra. These are Allua planning assumptions, not vendor prices or a fixed quote.
Use the same tests, record versions and explain every score. Compare review effort, usable outputs, recovery and total cost, not speed alone. After procurement, repeat affected checks when important versions or permissions change.
FAQs
Can a large vendor skip acceptance testing?
No. Your integration, data and permissions still need workflow-level verification.
Do we need every technical detail?
No, but obtain enough to assess data flows, permissions, failures and cost. Record confidentiality limits and unresolved risks.
Should we buy the highest-scoring option?
Not automatically. Apply essential gates first, then assess fit, cost and maintenance capacity.
Put the method into practice
Ask Allua to turn one AI procurement requirement into a testable scope and vendor questions. Share the workflow, data types, required connections and planning budget.
Disclaimer: third-party terms and capabilities may change; verify the applicable version before use. Fees depend on quotation. This is not legal advice or security certification. Corrections are welcome.
Creator invitation: apply for consideration in future articles, brand partnerships and one-stop influencer management. Selection and partnerships are not guaranteed.
Comments