How GenAI Creates New Forms of Online Gender-Based Violence (OGBV)

  • 28/10/2025
  • Category: Commentaries
No Comments
kbgo

Authors: Afra Hanifah Prasastisiwi and Mey Wulandari
Editor: Ayom Mratita Purbandani

When Technology Triggers Digital Harassment

Behind the rapid advancement of technology, artificial intelligence (AI) has now created a new stage for visual deception and identity theft. One increasingly alarming form of misuse is the use of deepfakes to produce pornographic content without the consent of those involved. The term deepfake first appeared in 2017, when a Reddit user synthesized pornographic content using the faces of celebrities.[1] Since then, the misuse of deepfakes has grown rapidly. The 2019 Deeptrace Report recorded that 96% of the 14,600 deepfake videos circulating on the internet were used to create non-consensual pornographic content, with a total of more than 134 million views across four major deepfake pornography websites.[2] By 2023, the number of online deepfake videos had sharply increased to around 95,820 videos. Of all these cases, 99% of the targeted individuals were women.[3]

The misuse of deepfakes has emerged as a global phenomenon. In the United States, 48% of men surveyed admitted to having seen deepfake pornographic content at least once. In the United Kingdom, it is estimated that around 4,000 celebrities have become victims of AI-generated pornography.[4] Meanwhile, in South Korea, about 53% of individuals appearing in deepfake pornographic content are singers and actresses, making them the most frequently targeted.[5] A prominent example is the controversy surrounding the Korean drama The Queen Who Crowns. In this case, the actress’s face was digitally superimposed on another person’s body in a nude scene without her consent.[6] This incident once again underscores that women remain the most vulnerable group to the misuse of digital technologies, particularly those of a sexual and non-consensual nature.

Not only in the form of videos, similar technological misuse is also widely employed to manipulate images. The term non-consensual synthetic intimate imagery (NSII) refers to the practice of creating or editing intimate content that is digitally generated or manipulated to depict real individuals without their consent.[7] UNESCO (2023) states that the emergence of generative AI (GenAI) has in fact expanded the forms of gender-based violence through two pathways: (1) embedded harms, or violence inherent within the system through social biases in training data that cause AI to tend to reproduce images of women in sexualized contexts without consent, and (2) malicious use, or harmful use through open access to AI models that enable the rapid production of manipulative content with minimal control.[8] Upon closer examination, this phenomenon shows that digital violence against women has long existed but has transformed through advances in AI. Such advancement has made the process increasingly fast, easy, and massive, and has even evolved into a large-scale business.

According to Graphika (2023), 34 identified NSII service providers received more than 24 million unique visitors in just one month, with an increase of over 2,000% in promotional links on platforms such as Reddit and X since early 2023. In addition, 52 Telegram groups used to access these services have accumulated at least one million users.[9] Looking more closely at the phenomenon in Indonesia, EngageMedia documented six incidents of non-consensual deepfake pornography during 2022–2023, and this number surged to 23 cases in 2024.[10] One of the most alarming examples is the case of an anonymous group called “Rahasia Mantan” (“Ex’s Secret”), which operated through Telegram and entrapped many victims, particularly university students.[11] The perpetrators ran their operations by offering pornographic content creation services, uploading photos of women (including minors) into publicly accessible generative AI platforms via online forms. These criminal acts are driven not only by personal motives such as hatred and economic gain through extortion (sextortion) but also by a deeper desire to control and silence women in digital public spaces.[12] This form of malicious use demonstrates that genAI production operates within a framework of digital patriarchy, marking a new phase in the industrialization of non-consensual intimate content that is increasingly pervasive and institutionalized.

The Trend of AI Generated Polaroid Photos and the Absence of Consent

If previously the creation of non-consensual intimate content was carried out through certain forums or services, now the free versions of popular GenAI platforms such as Gemini are also capable of producing realistic-style images from simple prompts. This ease of access indicates that the production of non-consensual intimate content no longer requires advanced technical skills but is instead facilitated by GenAI.

Sources: Tribun News, and social media platforms X and TikTok.

The images above illustrate phenomena occurring in Indonesia. In mid-2025, a trend emerged among internet users to create GenAI photos depicting themselves in intimate poses with famous figures, such as celebrities, professional football players, and K-pop idols. This phenomenon has caused discomfort among the targeted public figures, one professional football player from Indonesia’s national team expressed his unease on his official Instagram account regarding edited photos showing him in indecent poses.[13] The debate surrounding this trend went viral on the platform X (formerly Twitter), with one related post reaching over 6.4 million views, receiving more than 8,000 likes, and garnering over 5,000 quotes and replies. These cases affirm that the issue of consent remains a crucial concern often overlooked in the digital era. A person’s image is no longer fully under their control but has become a commodity that can be modified and exploited (even monetized) by the public or industry without clear boundaries. This reinforces the practice of objectification, where the body and visual identity of individuals are treated merely as instruments of visual consumption.[14] Furthermore, NSII has also highlighted how regulations and ethical frameworks remain unprepared to address these consequences, particularly in efforts to protect digital identity.[15]

Although GenAI technology is not always synonymous with negative impacts, the digital ecosystem has conditioned algorithms to become ‘enablers’ of abusive behavior. A concrete example is when searching for information and sources about the trend of polaroid photos on several major search engines (Google, Bing, etc.), what appears is not case reports or analyses of the trend, but rather prompt recommendations that actually encourage internet users to try creating similar content. This situation once again affirms that digital systems indeed facilitate both embedded harms and malicious use in reinforcing non-consensual practices within online spaces.[16]

This situation further underscores the importance of a legal framework that is responsive to technological developments. In Indonesia, regulations concerning deepfake pornography and the dissemination of non-consensual intimate images or videos remain limited. Although Law No. 1/2024 on Electronic Information and Transactions (Articles 27, 28, 45), Law No. 27/2022 on Personal Data Protection (Articles 10, 16, 65, 67), and Law No. 44/2008 on Pornography (Articles 4, 7, 9, 10) contain certain provisions that can be applied, there are still gaps in regulating content production, imposing sanctions on perpetrators, and providing protection for victims of technological misuse.[17] Therefore, it is hoped that spaces for reflection on ethical boundaries, authenticity, and accountability in digital content production will continue to expand.

  1. Diel, A., Lalgi, T., Schröter, I. C., MacDorman, K. F., Teufel, M. & Bäuerle, A. (2024) “Human performance in detecting deepfakes: A systematic review and meta-analysis of 56 papers.” Computers in Human Behavior Reports, 16 (6), p. 100538. doi:10.1016/j.chbr.2024.100538. ↑
  2. Sandoval, M-P., de Almeida Vau, M., Solaas, J. & Rodrigues, L. (2024) ‘Threat of deepfakes to the criminal justice system: a systematic review’, Crime Science, 13, 41. doi:10.1186/s40163-024-00239-1. ↑
  3. Security Hero (2023) 2023 State of Deepfakes: Realities, Threats, and Impact [online] Available at: https://www.securityhero.io/state-of-deepfakes/ ↑
  4. Panda Security (2024) Deepfake pornography explosion [online] Available at: https://www.pandasecurity.com/en/mediacenter/deepfake-pornography-explosion/ ↑
  5. Security Hero (2023) 2023 State of Deepfakes: Realities, Threats, and Impact [online] Available at: https://www.securityhero.io/state-of-deepfakes/ ↑
  6. Asia News Network (2025) ‘Should “deepfakes” be used in visual content?’, Asia News Network. Available at: https://asianews.network/should-deepfakes-be-used-in-visual-content/ ↑
  7. Lakatos, S. (2023) A Revealing Picture: AI-Generated ‘Undressing’ Images Move from Niche Pornography Discussion Forums to a Scaled and Monetized Online Business. New York: Graphika. Available at: https://public-assets.graphika.com/reports/graphika-report-a-revealing-picture.pdf ↑
  8. UNESCO (2023) “Future state: How will generative AI impact technology-facilitated gender-based violence? How will generative AI impact technology-facilitated gender-based violence? from ‘Your opinion doesn’t matter, anyway’: Exposing Technology-Facilitated Gender-Based Violence in an Era of Generative AI” [online] Available at: https://www.jstor.org/stable/resrep70942.7?seq=3 ↑
  9. Lakatos, S. (2023) A Revealing Picture: AI-Generated ‘Undressing’ Images Move from Niche Pornography Discussion Forums to a Scaled and Monetized Online Business. New York: Graphika. Available at: https://public-assets.graphika.com/reports/graphika-report-a-revealing-picture.pdf ↑
  10. EngageMedia (2025) Pemantauan Media 2022-2024: Apa Dampak Pemakaian Akal Imitasi (AI) di Indonesia [online] Available at: https://engagemedia.org/2025/pemantauan-media-2022-2024-apa-dampak-pemakaian-akal-imitasi-ai-di-indonesia/ ↑
  11. VOA Indonesia (2024) Menguak ‘Rahasia Mantan’ [online] Available at: https://www.voaindonesia.com/a/7737960.html ↑
  12. Tempo (2025) Ledakan Pornografi Deepfake di Era AI [online] Available at: https://www.tempo.co/newsletter/ledakan-pornografi-deepfake-di-era-ai-2056243 ↑
  13. Meilina, K. (2025) Komdigi Soal Viral Edit Foto Bareng Artis dan Tak Senonoh: Laporkan [online] Available at: https://katadata.co.id/digital/teknologi/68c960cb1bc52/komdigi-soal-viral-edit-foto-bareng-artis-dan-tak-senonoh-laporkan ↑
  14. Huang, H. (2024) Under the influence of digital media, the sexualization and objectification of idol industry. SHS Web of Conferences, 199, 02024. https://doi.org/10.1051/shsconf/202419902024 ↑
  15. VOI (2025) The Trend Of AI Polaroid with Idol Is Getting More Nyeleleneh, Komdigi Says It Can Be Reported [online] Available at: https://voi.id/en/technology/515029 ↑
  16. UNESCO (2023) “Future state: How will generative AI impact technology-facilitated gender-based violence? How will generative AI impact technology-facilitated gender-based violence? from ‘Your opinion doesn’t matter, anyway’: Exposing Technology-Facilitated Gender-Based Violence in an Era of Generative AI” [online] Available at: https://www.jstor.org/stable/resrep70942.7?seq=3 ↑
  17. EngageMedia (2025) Pemantauan Media 2022-2024: Apa Dampak Pemakaian Akal Imitasi (AI) di Indonesia [online] Available at: https://engagemedia.org/2025/pemantauan-media-2022-2024-apa-dampak-pemakaian-akal-imitasi-ai-di-indonesia/ ↑