AI Deepfakes Are On the Rise in China

Mr. Guo didn’t suspect anything when his friend called him up for a favor. They hopped on a WeChat video call, where everything looked and sounded normal.

Guo’s friend mentioned he was bidding on a project in another city, and needed to pay 4.3 million RMB, around $570,000. The friend asked to use Guo’s company bank account for the transaction, and promised to send him the money first.

When Guo received a screenshot showing that the money had been deposited, he transferred the 4.3 million RMB.

Unfortunately, that recipient was not actually Guo’s friend.

The scammer was a deepfake, using AI to impersonate the friend’s voice and likeness in real time over video call. Guo contacted the police immediately when he realized he’d been scammed, and with the help of the bank, managed to block around 3.3 million RMB from being transferred. The other one million had already been received by the scammers.

Deepfake scams are on the rise in China as AI technology becomes more accessible, and the Beijing Anti-Fraud Center even claimed on WeChat that certain emerging AI scams have nearly a 100% success rate.

“How are we supposed to avoid these scams if we can’t even trust the reality of someone’s face and voice,” wrote one user on social media platform Weibo.

Voice and face cloning have become commonplace, and a simple search on Bilibili, a Chinese streaming site similar to YouTube, yields thousands of tutorials for free and easy ways to clone someone’s likeness. Celebrities and artists are some of the most popular subjects.

“Okay, I’ll use this trick to make money as a livestreamer in the future,” reads one sarcastic comment underneath a video demonstrating how to clone yourself into a ‘Fairy Girl.’

The commenter is referencing the recent phenomenon of livestreamers on Douyin, China’s version of TikTok, who use AI to transform themselves into popular celebrities in an effort to boost sales numbers. The practice has sparked debates about false advertising, and the legality of using someone else’s likeness.

Simliarly, just last month, a content creator on Bilibili cloned the voice of famous Singaporean Mando-pop singer Stefanie Sun, uploading videos of “Sun” singing hit songs from other artists like Faye Wong and Jay Chou. The videos racked up millions of views, but many questioned whether or not they infringed on the artists’ copyrights.

As official mouthpiece Global Times hilariously put it, “Beware: AI-generated singing voices pleasing to the ear.”

On May 23, Sun broke her silence and addressed the fast-rising videos.

In a statement, she expressed a sense of irony about fading into the past, while her AI clone emerges as a hot new commodity. Her long, personal post contemplated the rapid advancement of AI, asserting that it won’t be long before the technology completely eclipses the capabilities of humans.

“The new technology will be able to mass produce everything everyone needs,” she wrote. “No matter how niche, deviant, or unhinged you are, there’s going to be unique content created just for you. You are not special, you are already predictable, and unfortunately also malleable,” said Sun.

The singer referenced the multiverse of Academy Award-winning Everything Everywhere All At Once, but depressingly stated that she “doesn’t think love can save us this time.”

“In the boundless sea of existence, where anything is possible, where nothing matters, I think it will be enough to keep my purity of thought and be exactly who I am,” she wrote.

Cover images via Beijing Anti-Fraud Center, Bilibili