Blog
12 dated notes on what AI testing vendors state, written from the directory's verified records. Every number in a post is computed from the data at build time.
2026-11-09
AI testing vendors in November 2026: what the verified records show
A snapshot of the directory at launch: which AI product types and testing services vendors state, where they are based, and how the coverage score falls.
2026-11-09
Bias audits for AI systems: what they measure and how they run
What a bias audit of an AI system measures, the NYC Local Law 144 rules, the four-fifths rule, EU AI Act Article 10, fairness metrics and an audit procedure.
2026-11-09
How AI product testing differs from traditional QA
Why fixed asserts do not cover AI products: non-determinism, the oracle problem, evaluation sets, metamorphic testing, data and model risks, and testing after release.
2026-11-09
How security testing of an LLM application works
Direct and indirect prompt injection, data and system prompt leakage, improper output handling and tool abuse, mapped to OWASP and MITRE ATLAS, with a test checklist.
2026-11-09
How to choose a vendor to test an AI product
A procedure for selecting an external AI testing vendor: scope and risks, shortlist filters, RFP questions, proposal review, a paid pilot and contract terms.
2026-11-09
How to evaluate LLM applications before release
Evaluation sets from real use, rubric scoring, LLM judges and their known biases, repeated runs, hallucination rates, version regressions and release gates.
2026-11-09
How to monitor an AI system in production for drift
Data drift and concept drift defined from primary sources, PSI and Kolmogorov-Smirnov checks, sampled grading of LLM output, and the incident-to-regression loop.
2026-11-09
How voice AI is tested: metrics and methods
Word error rate, response delay and turn-taking, barge-in, MOS for synthesised speech, noise and accent coverage, and conversational flow and hand-off tests for voice agents.
2026-11-09
RAG evaluation: testing retrieval and grounding separately
How to score retrieval with recall@k, MRR and nDCG, measure faithfulness, answer relevance and citation accuracy, catch index regressions and build a RAG test set.
2026-11-09
Red teaming is the AI testing service vendors state most often
How often each testing service appears on vendors' AI pages, which company types state red teaming and AI penetration testing, and what the two services cover.
2026-11-09
Testing AI agents: tool calls, trajectories and permissions
How to test AI agents: outcome and trajectory evaluation, tool-call validation, permission and data-access tests, failure injection and repeated runs, with public benchmarks.
2026-11-09
Why some AI testing companies are not in the directory
The rules that keep a company out of the directory, how often each rule applied at launch, and what a vendor can publish to be listed.