How voice AI is tested: metrics and methods

Which parts of a voice agent need separate tests?

A voice agent chains speech recognition, language understanding, a response step, speech synthesis and turn control. Each stage fails in its own way, so each has its own metric. ITU-T Supplement 24 to the P-series lists interaction parameters for spoken dialogue systems, from word error rate to response delay and task success [1]. A test plan that only scores whole calls cannot show which stage caused a failure.

How is word error rate calculated?

Word error rate (WER) compares the recognised text with a human transcript of the same audio. The two word sequences are aligned so that the total error count is as small as possible [2]. WER is then the sum of substitutions, deletions and insertions divided by the number of words in the reference [1], [2]. Supplement 24 defines word accuracy as one minus WER and refers to the NIST sclite algorithm for the alignment [1].

Term	Meaning
S, substitutions	Reference word recognised as a different word
D, deletions	Reference word missing from the output
I, insertions	Output word with no reference word
N, reference words	Hits plus substitutions plus deletions [2]
WER	(S + D + I) / N [1], [2]

Because insertions are counted, WER can exceed 100 percent. Morris et al. give an example in which a system that outputs two wrong words for each input word scores 200 percent [2]. Transcripts must follow one normalisation rule for numbers, casing and punctuation before scoring, or WER measures formatting instead of recognition.

Why can WER hide the failures that matter?

WER weights every word the same. Morris et al. argue that for most applications other than dictation, "a more meaningful performance measure would be given by the proportion of information communicated" [2]. A missed "not" or a wrong account number costs one word in WER and can fail the whole task. Test plans therefore add entity accuracy for the fields the agent acts on, such as dates, amounts, names and identifiers.

How should noise and accent coverage be built?

A public corpus gives a reference point but does not match the product's callers. LibriSpeech offers about 1,000 hours of read English speech, with "clean" test sets and "other" test sets that are "more challenging" [3]. Read speech differs from phone calls, so the main test set should be recorded or sampled from the product's own channel.

Noise is added in controlled steps. The MUSAN corpus contains "a wide assortment of technical and non-technical noises" alongside music and speech in twelve languages, under a Creative Commons license [4]. Mixing such noise into clean test audio at stated signal-to-noise ratios shows how WER changes as conditions get worse. The report should list each noise type and level used.

Accent and dialect coverage is a fairness check as well as an accuracy check. Koenecke et al. tested five commercial speech recognition systems and found an average WER of 0.35 for Black speakers against 0.19 for white speakers [5]. They traced the gap to the acoustic models [5]. Reporting WER per speaker group, rather than one average, is the core of a bias audit for speech input; bias and fairness audit covers the wider method.

bias audit: Measurement of whether a model's outcomes differ across demographic groups.

How are latency and turn-taking measured?

Supplement 24 defines system response delay as the time "from the user stopping speaking to the system starting speaking" [1]. For a voice agent, this delay covers end-of-speech detection, recognition, the response step and the start of synthesis. Each stage should be timed separately so that a slow stage can be found.

Human conversation sets the reference that callers bring. Stivers et al. studied 10 languages and found "a general avoidance of overlapping talk and a minimization of silence between conversational turns" [6]. The mode gap was between 0 and 200 ms in each language, with a cross-language median of 100 ms [6]. ITU-T G.114 adds a network view: below 150 ms one-way delay, most applications have "essentially transparent interactivity", and 400 ms should not be exceeded for general network planning [7].

These figures describe people and networks, not a pass threshold for an AI agent. The test plan should state its own target for response delay at the median and at a high percentile, measured under the expected call load. Performance and latency testing covers load conditions.

How is barge-in tested?

Supplement 24 defines barge-in as "the ability of a human to speak over a system prompt or system output" [1]. It counts a barge-in attempt when "the user intentionally addresses the system while the system is still speaking" [1]. The W3C VoiceXML 2.0 specification describes two behaviours: with speech barge-in, the prompt stops "as soon as speech or DTMF input is detected"; with hotword barge-in, it stops only on "a complete match of an active grammar" [8].

Barge-in tests cover four cases:

• True interruption: the caller speaks over a prompt, and the agent stops and responds to the new input.
• False trigger: background noise, a cough or a short "mm-hm" plays during a prompt, and the agent keeps speaking.
• Protected prompt: a prompt that must be heard in full, which VoiceXML describes for "a warning, legal notice, or advertisement", does not stop [8].
• Recovery: after an interruption, the agent neither resumes nor repeats the cut prompt.

How is synthesis quality rated with MOS?

ITU-T P.800 defines the Absolute Category Rating listening-quality scale: Excellent 5, Good 4, Fair 3, Poor 2, Bad 1 [9]. The mean of these ratings is the mean opinion score, represented by the symbol MOS [9]. A MOS score is therefore a listener panel result, not a property computed from the audio.

MOS score: Mean opinion score: listeners rate synthesised or transmitted speech quality.

P.800 states that every experiment should include reference conditions so that results from different laboratories or times "can be sensibly compared" [9]. It also asks for listeners who have not worked on assessing telephone circuits or on speech coding [9]. A MOS without its reference conditions and panel description cannot be compared with another MOS.

For synthesised speech, ITU-T P.85 notes that "the use of multiple opinion scales improves the description of listeners' perception" [10]. Its scales include listening effort, pronunciation, speaking rate, articulation, voice pleasantness and acceptance [10]. P.85 also has listeners answer questions about the message content before they rate it, so that they attend to the information [10].

How are conversational flow and hand-off tested?

Supplement 24 defines task success by whether the user "has reached his/her goal by the end of a dialogue" [1]. Its labels separate success, success after relaxing constraints and failure caused by the system or by the user [1]. It also counts help requests, time-out prompts, rejected utterances and user attempts to cancel a previous action [1]. ITU-T P.851 adds questionnaires that capture quality as the caller perceives it [11].

Flow tests run scripted scenarios through the live voice channel, including corrections, topic changes and silence. Hand-off tests check three things: the trigger fires when the caller asks for a person or the agent fails repeatedly, the caller reaches a human without repeating the request, and the context passed to the human matches what was said. Conversational flow testing covers intent coverage and multi-turn context in more depth. When the response step uses a language model, its answers can be graded as described in how to evaluate LLM applications.

Which entries state voice AI testing?

17 published entries state testing for voice AI. The table shows how many of them state each related service.

Counts reflect what each vendor's AI page states, as described in the methodology.

Voice AI test plan

1. 01

   Build the reference set

   Record or sample calls from the product's channel, transcribe them under one normalisation rule, and tag speaker group, device and noise.

2. 02

   Score recognition

   Report WER as (S + D + I) / N with sclite-style alignment, plus entity accuracy for fields the agent acts on [2], [1].

3. 03

   Add noise in steps

   Mix technical and non-technical noise into clean audio at stated signal-to-noise ratios and plot WER per level [4].

4. 04

   Split by speaker group

   Report WER per accent, dialect and demographic group, not only the average [5].

5. 05

   Time each stage

   Measure system response delay and its parts at the median and a high percentile under expected load [1].

6. 06

   Test barge-in

   Check true interruptions, noise and backchannels, non-interruptible notices, and behaviour after a cut prompt [1], [8].

7. 07

   Rate the voice

   Run an ACR listening test for MOS with listeners outside speech quality work and with reference conditions; add P.85 scales for synthesised speech [9], [10].

8. 08

   Run scenarios end to end

   Label task success per Supplement 24 and collect caller ratings with a P.851 questionnaire [1], [11].

9. 09

   Test hand-off

   Confirm triggers, transfer without repetition, and accurate context passed to the human agent.

10. 10

    Keep a regression set

    Store failing calls as fixed audio cases and rerun them after every model, prompt or voice change.

Failing calls from production feed the same regression set; monitoring AI in production describes that loop.

Sources

[1] ITU-T P-series Supplement 24, Parameters describing the interaction with spoken dialogue systems, ITU, 2005, https://www.itu.int/rec/T-REC-P.Sup24/en

[2] Morris, A. C., Maier, V., Green, P. "From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition", Interspeech, ISCA, 2004, https://www.isca-archive.org/interspeech_2004/morris04_interspeech.html

[3] LibriSpeech ASR corpus (SLR12), Panayotov, V., Chen, G., Povey, D., Khudanpur, S., OpenSLR, 2015, https://www.openslr.org/12/

[4] Snyder, D., Chen, G., Povey, D. "MUSAN: A Music, Speech, and Noise Corpus", arXiv:1510.08484, 2015, https://arxiv.org/abs/1510.08484

[5] Koenecke, A. et al. "Racial disparities in automated speech recognition", Proceedings of the National Academy of Sciences 117, 2020, https://pmc.ncbi.nlm.nih.gov/articles/PMC7149386/ (DOI 10.1073/pnas.1915768117)

[6] Stivers, T. et al. "Universals and cultural variation in turn-taking in conversation", Proceedings of the National Academy of Sciences 106, 2009, https://pmc.ncbi.nlm.nih.gov/articles/PMC2705608/ (DOI 10.1073/pnas.0903616106)

[7] ITU-T G.114, One-way transmission time, ITU, 2003, https://www.itu.int/rec/T-REC-G.114/en

[8] Voice Extensible Markup Language (VoiceXML) Version 2.0, W3C Recommendation, W3C, 2004, https://www.w3.org/TR/voicexml20/

[9] ITU-T P.800, Methods for subjective determination of transmission quality, ITU, 1996, https://www.itu.int/rec/T-REC-P.800-199608-I/en

[10] ITU-T P.85, A method for subjective performance assessment of the quality of speech voice output devices, ITU, 1994, https://www.itu.int/rec/T-REC-P.85/en

[11] ITU-T P.851, Subjective quality evaluation of telephone services based on spoken dialogue systems, ITU, 2003, https://www.itu.int/rec/T-REC-P.851/en