OpenAI bans accounts linked to surveillance tool development

Investing.com -- OpenAI, an artificial intelligence startup, has recently banned multiple accounts on its platform, ChatGPT, that appeared to be involved in developing a surveillance tool. The company’s report, published in February 2025, indicates that these accounts are likely linked to China.

The banned accounts were reportedly using OpenAI’s models to generate detailed descriptions of a social media listening tool, which they claimed was used to provide real-time reports about protests in the West to Chinese security services. The accounts were also using OpenAI’s models to debug code that seemed intended to implement such a tool. OpenAI’s policies strictly prohibit the use of AI for communications surveillance or unauthorized monitoring of individuals, including activities by or on behalf of governments that seek to suppress personal freedoms and rights.

The banned accounts were identified based on behavioral patterns and other findings. The accounts were using OpenAI’s models to assist with analyzing documents, generating sales pitches and descriptions of tools for monitoring social media activity, and researching political actors and topics.

The activities of the banned accounts were primarily conducted during mainland Chinese business hours, and the accounts prompted OpenAI’s models in Chinese. They were also using the tools with a volume and variety consistent with manual prompting, rather than automation. In some instances, it appeared that the same account may have been used by multiple operators.

The operators used the model for various tasks, such as researching publicly available information about think tanks in the United States, politicians and government officials in countries including Australia, Cambodia and the United States. They also used the models to read, translate and analyze screenshots of English-language documents, some of which were announcements of Uyghur rights protests in a range of Western cities.

Another activity involved using ChatGPT to generate short to medium-length comments about Chinese dissident organizations, notably the Falun Gong, and about US policies and politics. The operators also used the model to edit and debug code and generate promotional materials for what appeared to be an AI-powered social media listening tool.

One of the main activities of the operation was generating detailed descriptions, consistent with sales pitches, for what they described as the "Qianyue Overseas Public Opinion AI Assistant". According to the descriptions, this tool was designed to ingest and analyze posts and comments from platforms such as X, Facebook (NASDAQ:META), YouTube, Instagram, Telegram and Reddit, and feed the resulting insights to Chinese authorities. However, OpenAI could not independently verify these descriptions.

The impact of this operation is difficult to assess as very little of its activity appeared to be designed for publication on social media or other distribution platforms. Most of the content appeared to be for capability development, such as code debugging, or for internal purposes, such as image analysis and the development of promotional materials. Assessing the impact of this activity would require inputs from multiple stakeholders, including operators of any open-source models who can shed light on this activity.

This article was generated with the support of AI and reviewed by an editor. For more information see our T&C.