Voice Identity Crisis: ChatGPT's Unintended Mimicry During Tests

Imagine speaking to an AI application, and suddenly, instead of its preset voice, it begins mimicking your voice. It sounds unnerving, doesn't it?

Recently, OpenAI revealed that during testing, ChatGPT unexpectedly began speaking in the user's voice during conversations.

This incident was discovered by a member of OpenAI's red team (engaged in adversarial testing to identify issues) during testing. Following this discovery, OpenAI traced the system's processes and found that ChatGPT's advanced voice mode might unintentionally mimic voices in noisy input environments.

Under normal circumstances, ChatGPT's advanced real-time voice mode cannot mimic a user's voice without consent, as there are built-in safety mechanisms. However, the test exposed vulnerabilities within these safety mechanisms.

OpenAI explained that they have a specific system to detect if the model is generating unauthorized audio, meaning the model is only supposed to use preset voices. Additionally, there's an output classifier designed to detect if the model deviates from these preset voices.

In any case, the system should correct any "violations," but the red team's test case showed that AI could somehow bypass these limitations. OpenAI did not disclose how they rectified the issue, but the company emphasized that it has implemented enhanced safety measures to prevent such incidents from happening again.

In theory, completely resolving this issue is challenging, as the model might always find a way to break through the safety boundaries during extended conversations. Patching up might reduce the likelihood of such occurrences, but completely eliminating them could be more difficult.