Predicting relative intelligibility from inter-talker distances in a perceptual similarity space for speech

Researchers have generally assumed that listeners perceive speech compositionally, based on the combined processing of local acoustic–phonetic cues associated with individual linguistic units. Yet, these cue-based approaches have failed to fully account for variation in listeners’ identification of the words produced by a talker (i.e., variation in talker intelligibility). The current study adopts an alternative approach, estimating the perceptual representations used to process speech (the perceptual similarity space) using the machine learning technique of self-supervised learning. We assessed intelligibility of 114 second-language (L2) English talkers and 25 L1 American English talkers through a speech-in-noise experiment (collecting data from ten L1 English listeners per talker, each transcribing 120 sentences). For each sample in a speech recording, we obtained a representation from a self-supervised learning model; the sequence of these representations forms a trajectory in the perceptual similarity space. The holistic distance between trajectories (two speakers’ productions of the same sentence) was analyzed. We found that for L2 talkers, the average distance between the trajectories of an L2 talker and the L1 American English talker group predicts relative intelligibility of a given L2 talker. Crucially, the distance measure predicted relative intelligibility among L2 talkers over and above a set of traditional acoustic–phonetic cues. Additionally, we found that the distance measure accounts for some of the relative intelligibility among L1 talkers. These results provide evidence that relative talker intelligibility is better captured with the perceptual similarity space approach, suggesting it is an appropriate tool to study variability in human speech production and perception.