OpenAI bans Chinese accounts using ChatGPT to aid Social Media surveillance

San Francisco, February 17, 2025 – OpenAI has terminated a group of Chinese ChatGPT accounts that were reportedly used to edit and debug code for an AI-powered social media surveillance tool.

The company said the accounts, part of an operation it dubbed “Peer Review,” attempted to leverage ChatGPT for generating sales pitches for a program designed to monitor anti-Chinese sentiment across platforms such as X, Facebook, YouTube, and Instagram.

According to OpenAI, the network of accounts operated during mainland Chinese business hours, used Chinese-language prompts, and demonstrated a pattern consistent with manual, rather than automated, use. “The operators used our models to proofread claims that their insights had been sent to Chinese embassies abroad, and to intelligence agents monitoring protests in countries including the United States, Germany, and the United Kingdom,” the company stated in a press release.

Ben Nimmo, a principal investigator with OpenAI, said this is the first instance in which the company has uncovered an AI tool being used in this manner. “Threat actors sometimes give us a glimpse of what they are doing in other parts of the internet because of the way they use our AI models,” Nimmo told The New York Times.

The surveillance tool’s code appears to have been derived from an open-source version of one of Meta’s Llama models. Documents indicate that the group also used ChatGPT to generate an end-of-year performance review, in which it claimed responsibility for writing phishing emails on behalf of clients in China.

OpenAI noted that fully assessing the impact of this activity will require input from multiple stakeholders, including operators of open-source models who can provide further insight into the operation.

In a separate action, OpenAI recently banned an account that used ChatGPT to generate social media posts critical of Cai Xia, a Chinese political scientist and dissident living in exile in the United States. The same group was also implicated in generating Spanish-language articles critical of the United States, which were published by mainstream news organizations in Latin America and attributed to either an individual or a Chinese company.

OpenAI’s actions highlight the ongoing challenges of monitoring and regulating the use of AI tools in politically sensitive contexts, particularly as these technologies become increasingly integrated into global information networks.

Copied