Speech Generation Speaker Poisoning: Capability Erasure in Zero-Shot Text-to-Speech

Recent zero-shot Text-to-Speech (TTS) systems leverage representations learned from large speech language models to clone previously unseen voices from only a few seconds of audio. This capability raises a broader question in generative AI: can a trained model be prevented from exhibiting a specific capability while preserving its overall utility? We formulate this challenge as \emph{Speech Generation Speaker Poisoning} (SGSP), a capability-erasure task that seeks to prevent a model from synthesizing targeted speaker identities while maintaining performance on all other speakers. Unlike conventional machine unlearning, removing training examples is insufficient because modern zero-shot TTS systems can reconstruct identities through learned speaker representations and strong generalization capabilities. We evaluate both inference-time filtering and parameter-modification approaches across settings involving 1, 15, and 100 forget speakers. To characterize the trade-off between utility and privacy, we introduce an evaluation framework based on AUC analysis and a proposed metric, Forget Set Similarity (FSSIM). Our results demonstrate effective speaker suppression for up to 15 forget speakers while revealing scalability challenges at 100 speakers due to increasing overlap in the learned speaker representation space. Together, our work establishes targeted speaker poisoning as a capability-erasure task for generative speech models and provides methods and evaluation protocols for future research.