{"id":23466,"date":"2023-11-28T05:20:59","date_gmt":"2023-11-28T13:20:59","guid":{"rendered":"http:\/\/www.palada.net\/index.php\/2023\/11\/28\/news-17196\/"},"modified":"2023-11-28T05:20:59","modified_gmt":"2023-11-28T13:20:59","slug":"news-17196","status":"publish","type":"post","link":"http:\/\/www.palada.net\/index.php\/2023\/11\/28\/news-17196\/","title":{"rendered":"Cybercriminals can&#8217;t agree on GPTs"},"content":{"rendered":"<p><strong>Credit to Author: Matt Wixey| Date: Tue, 28 Nov 2023 11:00:13 +0000<\/strong><\/p>\n<div class=\"entry-content lg:prose-lg mx-auto prose max-w-4xl\">\n<p>A significant amount of media coverage followed the news that large language models (LLMs) intended for use by cybercriminals \u2013 including <a href=\"https:\/\/slashnext.com\/blog\/wormgpt-the-generative-ai-tool-cybercriminals-are-using-to-launch-business-email-compromise-attacks\/\">WormGPT<\/a> and <a href=\"https:\/\/abnormalsecurity.com\/blog\/fraudgpt-malicious-generative-ai\">FraudGPT<\/a> \u2013 were available for sale on underground forums. Many commenters expressed fears that such models would enable threat actors to create \u201c<a href=\"https:\/\/www.csoonline.com\/article\/575487\/chatgpt-creates-mutating-malware-that-evades-detection-by-edr.html\">mutating malware<\/a>\u201d and were part of a \u201c<a href=\"https:\/\/www.darkreading.com\/application-security\/chatgpt-jailbreaking-forums-dark-web-communities\">frenzy<\/a>\u201d of related activity in underground forums.<\/p>\n<p>The <a href=\"https:\/\/link.springer.com\/article\/10.1007\/s10676-023-09703-z\">dual-use<\/a> aspect of LLMs is undoubtedly a concern, and there is no doubt that threat actors will seek to leverage them for their own ends. Tools like WormGPT are an early indication of this (although the WormGPT developers have now shut the project down, ostensibly because they grew alarmed at the amount of media attention they received). What\u2019s less clear is how threat actors more generally think about such tools, and what they\u2019re actually using them for beyond a few <a href=\"https:\/\/news.sophos.com\/en-us\/2023\/08\/02\/sha-zhu-pan-scam-uses-ai-chat-to-target-iphone-and-android-users\/\">publicly-reported incidents<\/a>.<\/p>\n<p>Sophos X-Ops decided to investigate LLM-related discussions and opinions on a selection of criminal forums, to get a better understanding of the current state of play, and to explore what the threat actors themselves actually think about the opportunities \u2013 and risks \u2013 posed by LLMs. We trawled through four prominent forums and marketplaces, looking specifically at what threat actors are using LLMs for; their perceptions of them; and their thoughts about tools like WormGPT.<\/p>\n<p>A brief summary of our findings:<\/p>\n<ul>\n<li><strong>We found multiple GPT-derivatives<\/strong> claiming to offer capabilities similar to WormGPT and FraudGPT \u2013 including EvilGPT, DarkGPT, PentesterGPT, and XXXGPT. However, we also noted skepticism about some of these, including allegations that they\u2019re scams (<a href=\"https:\/\/news.sophos.com\/en-us\/2022\/12\/07\/the-scammers-who-scam-scammers-on-cybercrime-forums-part-1\/\">not unheard of on criminal forums<\/a>)<\/li>\n<li><strong>In general, there is a lot of skepticism<\/strong> about tools like ChatGPT \u2013 including arguments that it is overrated, overhyped, redundant, and unsuitable for generating malware<\/li>\n<li><strong>Threat actors also have cybercrime-specific concerns about LLM-generated code<\/strong>, including operational security worries and AV\/EDR detection<\/li>\n<li><strong>A lot of posts focus on jailbreaks<\/strong> (which also appear with regularity on social media and legitimate blogs) and compromised ChatGPT accounts<\/li>\n<li><strong>Real-world applications remain aspirational for the most part<\/strong>, and are generally limited to social engineering attacks, or tangential security-related tasks<\/li>\n<li><strong>We found only a few examples of threat actors using LLMs to generate malware and attack tools<\/strong>, and that was only in a proof-of-concept context<\/li>\n<li><strong>However, others are using it effectively for other work<\/strong>, such as mundane coding tasks<\/li>\n<li><strong>Unsurprisingly, unskilled \u2018script kiddies\u2019 are interested in using GPTs<\/strong> to generate malware, but are \u2013 again unsurprisingly \u2013 often unable to bypass prompt restrictions, or to understand errors in the resulting code<\/li>\n<li><strong>Some threat actors are using LLMs to enhance the forums they frequent<\/strong>, by creating chatbots and auto-responses \u2013 with varying levels of success \u2013 while others are using it to develop redundant or superfluous tools<\/li>\n<li><strong>We also noted examples of AI-related \u2018thought leadership\u2019 <\/strong>on the forums, suggesting that threat actors are wrestling with the same logistical, philosophical, and ethical questions as everyone else when it comes to this technology<\/li>\n<\/ul>\n<p>While writing this article, which is based on our own independent research, we became aware that Trend Micro had recently published <a href=\"https:\/\/www.trendmicro.com\/vinfo\/ie\/security\/news\/cybercrime-and-digital-threats\/hype-vs-reality-ai-in-the-cybercriminal-underground\">their own research on this topic<\/a>. Our research in some areas confirms and validates some of their findings.<\/p>\n<h2>The forums<\/h2>\n<p>We focused on four forums for this research:<\/p>\n<ul>\n<li>Exploit: a prominent Russian-language forum which prioritizes Access-a-a-Service (AaaS) listings, but also enables buying and selling of other illicit content (including malware, data leaks, infostealer logs, and credentials) and broader discussions about various cybercrime topics<\/li>\n<li>XSS: a prominent Russian-language forum. Like Exploit, it\u2019s well-established, and also hosts both a marketplace and wider discussions and initiatives<\/li>\n<li>Breach Forums: Now in its second iteration, this English-language forum replaced <a href=\"https:\/\/www.justice.gov\/opa\/pr\/united-states-leads-seizure-one-world-s-largest-hacker-forums-and-arrests-administrator\">RaidForums after its seizure in 2022<\/a>; the first version of Breach Forums was <a href=\"https:\/\/therecord.media\/breachforums-seized-by-fbi-months-after-arrest-of-alleged-administrator-dark-web-marketplace\">similarly shut down in 2023<\/a>. Breach Forums specializes in data leaks, including databases, credentials, and personal data<\/li>\n<li>Hackforums: a long-running English-language forum which has <a href=\"https:\/\/cyberscoop.com\/inside-hackforums-rebellious-cybercrime-empire\/\">a reputation for being populated by script kiddies<\/a>, although some of its users have previously been linked to high-profile malware and incidents<\/li>\n<\/ul>\n<p>A caveat before we begin: the opinions discussed here cannot be considered as representative of all threat actors\u2019 attitudes and beliefs, and don\u2019t come from qualitative surveys or interviews. Instead, this research should be considered as an exploratory assessment of LLM-related discussions and content as they currently appear on the above forums.<\/p>\n<h2>Digging in<\/h2>\n<p>One of the first things we noticed is that AI is not exactly a hot topic on any of the forums we looked at. On two of the forums, there were fewer than 100 posts on the subject \u2013 but almost 1,000 posts about cryptocurrencies across a comparative period.<\/p>\n<p>While we\u2019d want to do further research before drawing any firm conclusions about this discrepancy, the numbers suggest that there hasn\u2019t been an explosion in LLM-related discussions in the forums \u2013 at least not to the extent that there has been on, say, LinkedIn. That could be because many cybercriminals see generative AI as still being in its infancy (at least compared to cryptocurrencies, which have a real-world relevance to them as an established and relatively mature technology). And, unlike some LinkedIn users, threat actors have little to gain from speculating about the implications of a nascent technology.<\/p>\n<p>Of course, we only looked at the four forums mentioned above, and it\u2019s entirely possible that more active discussions around LLMs are happening in other, less visible channels.<\/p>\n<h2>Let me outta here<\/h2>\n<p><a href=\"https:\/\/www.trendmicro.com\/vinfo\/ie\/security\/news\/cybercrime-and-digital-threats\/hype-vs-reality-ai-in-the-cybercriminal-underground\">As Trend Micro also noted in its report<\/a>, we found that a significant amount of LLM-related posts on the forums focus on jailbreaks \u2013 either those from other sources, or jailbreaks shared by forum members (a \u2018jailbreak\u2019 in this context is a means to trick an LLM into bypassing its own self-censorship when it comes to returning harmful, illegal, or inappropriate responses).<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-952462 size-full\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image1.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"223\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image1.png 1558w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image1.png?resize=300,105 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image1.png?resize=768,268 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image1.png?resize=1024,358 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image1.png?resize=1536,536 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 1: A user shares details of <a href=\"https:\/\/medium.com\/@neonforge\/meet-dan-the-jailbreak-version-of-chatgpt-and-how-to-use-it-ai-unchained-and-unfiltered-f91bfa679024\">the publicly-known \u2018DAN\u2019 jailbreak<\/a><\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952463\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image2.png\" alt=\"A screenshot of a post on a criminal forum, with a screenshot of a ChatGPT window\" width=\"640\" height=\"444\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image2.png 891w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image2.png?resize=300,208 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image2.png?resize=768,533 768w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 2: A Breach Forums user shares details of an unsuccessful jailbreak attempt<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image3.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952464\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image3.png\" alt=\"A screenshot of two posts on a criminal forum\" width=\"640\" height=\"286\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image3.png 1574w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image3.png?resize=300,134 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image3.png?resize=768,344 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image3.png?resize=1024,458 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image3.png?resize=1536,687 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 3: A\u00a0 forum user shares a jailbreak tactic<\/em><\/p>\n<p>While this may appear concerning, jailbreaks are also publicly and widely shared on the internet, including in social media posts; dedicated websites containing collections of jailbreaks; subreddits devoted to the topic; and YouTube videos.<\/p>\n<p>There is an argument that threat actors may \u2013 by dint of their experience and skills \u2013 be in a better position than most to develop novel jailbreaks, but we observed little evidence of this.<\/p>\n<h2>Accounts for sale<\/h2>\n<p>More commonly \u2013 and, unsurprisingly, especially on Breach Forums \u2013 we noted that many of the LLM-related posts were actually compromised ChatGPT accounts for sale.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image4.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952465\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image4.png\" alt=\"A screenshot of a post on a criminal forum offering ChatGPT accounts for sale\" width=\"571\" height=\"539\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image4.png 571w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image4.png?resize=300,283 300w\" sizes=\"auto, (max-width: 571px) 100vw, 571px\" \/><\/a><\/p>\n<p><em>Figure 4: A selection of ChatGPT accounts for sale on Breach Forums<\/em><\/p>\n<p>There\u2019s little of interest to discuss here, only that threat actors are obviously seizing the opportunity to compromise and sell accounts on new platforms. What\u2019s less clear is what the target audience would be for these accounts, and what a buyer would seek to do with a stolen ChatGPT account. Potentially they could access previous queries and obtain sensitive information, or use the access to run their own queries, or check for password reuse.<\/p>\n<h2>Jumping on the \u2018BandwagonGPT\u2019<\/h2>\n<p>Of more interest was our discovery that WormGPT and FraudGPT are not the only players in town \u2013 a discovery which Trend Micro also noted in its report. During our research, we observed eight other models either offered for sale on forums as a service, or developed elsewhere and shared with forum users.<\/p>\n<ol>\n<li>XXXGPT<\/li>\n<li>Evil-GPT<\/li>\n<li>WolfGPT<\/li>\n<li>BlackHatGPT<\/li>\n<li>DarkGPT<\/li>\n<li>HackBot<\/li>\n<li>PentesterGPT<\/li>\n<li>PrivateGPT<\/li>\n<\/ol>\n<p>However, we noted some mixed reactions to these tools. Some users were very keen to trial or purchase them, but many were doubtful about their capabilities and novelty. And some were outright hostile, accusing the tools\u2019 developers of being scammers.<\/p>\n<h3>WormGPT<\/h3>\n<p>WormGPT, released in June 2023, was a private chatbot service purportedly based on <a href=\"https:\/\/www.cerebras.net\/blog\/cerebras-makes-it-easy-to-harness-the-predictive-power-of-gpt-j\">LLM GPT-J 6B<\/a>, and offered as a commercial service on several criminal forums. As with many cybercrime services and tools, its launch was accompanied by a slick promotional campaign, including posters and examples.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image5.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952466\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image5.png\" alt=\"An advert for WormGPT on a criminal forum\" width=\"640\" height=\"295\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image5.png 1399w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image5.png?resize=300,138 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image5.png?resize=768,354 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image5.png?resize=1024,472 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 5: WormGPT advertised by one of its developers in July 2023<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image6.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952467\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image6.png\" alt=\"A screenshot of some of the WormGOT promotional material, showing example queries and responses\" width=\"640\" height=\"177\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image6.png 1492w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image6.png?resize=300,83 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image6.png?resize=768,212 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image6.png?resize=1024,283 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 6: Examples of WormGPT queries and responses, featured in promotional material by its developers<\/em><\/p>\n<p>The extent to which WormGPT facilitated any real-world attacks is unknown. However, the project received a considerable amount of media attention, which perhaps led its developers to first restrict some of the subject matter available to users (including business email compromises and carding), and then to shut down completely in August 2023.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image7.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952468\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image7.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"559\" height=\"339\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image7.png 559w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image7.png?resize=300,182 300w\" sizes=\"auto, (max-width: 559px) 100vw, 559px\" \/><\/a><\/p>\n<p><em>Figure 7: One of the WormGPT developers announces changes to the project in early August<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image8.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952469\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image8.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"315\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image8.png 1565w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image8.png?resize=300,148 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image8.png?resize=768,378 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image8.png?resize=1024,504 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image8.png?resize=1536,756 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 8: The post announcing the closure of the WormGPT project, one day later<\/em><\/p>\n<p>In the announcement marking the end of WormGPT, the developer specifically calls out the media attention they received as a key reason for deciding to end the project. They also note that: \u201cAt the end of the day, WormGPT is nothing more than an unrestricted ChatGPT. Anyone on the internet can employ a well-known jailbreak technique and achieve the same, if not better, results.\u201d<\/p>\n<p>While some users expressed regrets over WormGPT\u2019s closure, others were irritated. One Hackforums user noted that their licence had stopped working, and users on both Hackforums and XSS alleged that the whole thing had been a scam.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image9.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952470\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image9.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"123\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image9.png 1149w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image9.png?resize=300,58 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image9.png?resize=768,148 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image9.png?resize=1024,197 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 9: A Hackforums user alleges that WormGPT was a scam<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image10.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952471\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image10.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"160\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image10.png 896w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image10.png?resize=300,75 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image10.png?resize=768,192 768w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 10: An XSS user makes the same allegation. Note the original comment, which suggests that since the project has received widespread media attention, it is best avoided<\/em><\/p>\n<h3>FraudGPT<\/h3>\n<p>The same accusation has also been levelled at FraudGPT, and others have questioned its stated capabilities. For example, one Hackforums user asked whether the claim that FraudGPT can generate \u201ca range of malware that antivirus software cannot detect\u201d was accurate. A fellow user provided them with an informed opinion:<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image11.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952472\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image11.png\" alt=\"\" width=\"640\" height=\"180\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image11.png 1220w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image11.png?resize=300,84 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image11.png?resize=768,216 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image11.png?resize=1024,288 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 11: A Hackforums user conveys some skepticism about the efficacy of GPTs and LLMs<\/em><\/p>\n<p>This attitude seems to be prevalent when it comes to malicious GPT services, as we\u2019ll see shortly.<\/p>\n<h3>XXXGPT<\/h3>\n<p>The misleadingly-titled XXXGPT was announced on XSS in July 2023. Like WormGPT, it arrived with some fanfare, including promotional posters, and claimed to provide \u201ca revolutionary service that offers personalized bot AI customization\u2026with no censorship or restrictions\u201d for $90 a month.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image12.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952473\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image12.png\" alt=\"A screenshot of a promotional poster for XXXGPT\" width=\"640\" height=\"905\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image12.png 704w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image12.png?resize=212,300 212w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 12: One of several promotional posters for XXXGPT, complete with a spelling mistake (\u2018BART\u2019 instead of \u2018BARD\u2019)<\/em><\/p>\n<p>However, the announcement met with some criticism. One user asked what exactly was being sold, questioning whether it was just a jailbroken prompt.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image13.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952474\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image13.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"394\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image13.png 1138w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image13.png?resize=300,185 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image13.png?resize=768,473 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image13.png?resize=1024,631 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 13: A user queries whether XXXGPT is essentially just a prompt<\/em><\/p>\n<p>Another user, testing the XXXGPT demo, found that it still returned censored responses.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image14.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952475\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image14.png\" alt=\"\" width=\"640\" height=\"164\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image14.png 1349w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image14.png?resize=300,77 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image14.png?resize=768,197 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image14.png?resize=1024,263 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 14: A user is unable to get the XXXGPT demo to generate malware<\/em><\/p>\n<p>The current status of the project is unclear.<\/p>\n<h3>Evil-GPT<\/h3>\n<p>Evil-GPT was announced on Breach Forums in August 2023, advertised explicitly as an alternative to WormGPT at a much lower cost of $10. Unlike WormGPT and XXXGPT, there were no alluring graphics or feature lists, only a screenshot of an example query.<\/p>\n<p>Users responded positively to the announcement, with one noting that while it \u201cis not accurate for blackhat questions nor coding complex malware\u2026[it] could be worth [it] to someone to play around.\u201d<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image15.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952476\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image15.png\" alt=\"\" width=\"640\" height=\"95\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image15.png 1307w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image15.png?resize=300,44 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image15.png?resize=768,113 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image15.png?resize=1024,151 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 15: A Hackforums moderator gives a favourable review of Evil-GPT<\/em><\/p>\n<p>From what was advertised, and from the user reviews, we assess that Evil-GPT is targeting users seeking a \u2018budget-friendly\u2019 option \u2013 perhaps limited in capability compared to some other malicious GPT services, but a \u201ccool toy.\u201d<\/p>\n<h3>Miscellaneous GPT derivatives<\/h3>\n<p>In addition to WormGPT, FraudGPT, XXXGPT, and Evil-GPT, we also observed several derivative services which don\u2019t appear to have received much attention, either positive or negative.<\/p>\n<h4>WolfGPT<\/h4>\n<p>WolfGPT was shared on XSS by a user who claims it is a Python-based tool which can \u201cencrypt malware and create phishing texts\u2026a competitor to WormGPT and ChatGPT.\u201d The tool appears to be a GitHub repository, although there is no documentation for it. <a href=\"https:\/\/www.trendmicro.com\/vinfo\/ie\/security\/news\/cybercrime-and-digital-threats\/hype-vs-reality-ai-in-the-cybercriminal-underground\">In its article<\/a>, Trend Micro notes that WolfGPT was also advertised on a Telegram channel, and that the GitHub code appears to be a Python wrapper for ChatGPT\u2019s AI.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image16.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952477\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image16.png\" alt=\"A screenshot of the WolfGPT GitHub repository\" width=\"640\" height=\"450\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image16.png 928w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image16.png?resize=300,211 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image16.png?resize=768,540 768w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 16: The WolfGPT GitHub repository<\/em><\/p>\n<h4>BlackHatGPT<\/h4>\n<p>This tool, announced on Hackforums, claims to be an uncensored ChatGPT.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image17.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952478\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image17.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"453\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image17.png 848w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image17.png?resize=300,212 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image17.png?resize=768,543 768w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 17: The announcement of BlackHatGPT on Hackforums<\/em><\/p>\n<h4>DarkGPT<\/h4>\n<p>Another project by a Hackforums user, DarkGPT again claims to be an uncensored alternative to ChatGPT. Interestingly, the user claims DarkGPT offers anonymity, although it\u2019s not clear how that is achieved.<\/p>\n<h4>HackBot<\/h4>\n<p>Like WolfGPT, HackBot is a GitHub repository, which a user shared with the Breach Forums community. Unlike some of the other services described above, HackBot does not present itself as an explicitly malicious service, and instead is purportedly aimed at security researchers and penetration testers.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image18.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952479\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image18.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"83\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image18.png 1351w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image18.png?resize=300,39 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image18.png?resize=768,100 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image18.png?resize=1024,133 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 18: A description of the HackBot project on Breach Forums<\/em><\/p>\n<h4>PentesterGPT<\/h4>\n<p>We also observed another security-themed GPT service, PentesterGPT.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image19.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-952480 size-full\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image19.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"184\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image19.png 1296w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image19.png?resize=300,86 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image19.png?resize=768,221 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image19.png?resize=1024,295 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 19: PentesterGPT shared with Breach Forums users<\/em><\/p>\n<h4>PrivateGPT<\/h4>\n<p>We only observed PrivateGPT mentioned briefly on Hackforums, but <a href=\"https:\/\/github.com\/imartinez\/privateGPT\">it claims to be an offline LLM<\/a>. A Hackforums user expressed interest in gathering \u201chacking resources\u201d to use with it. There is no indication that PrivateGPT is intended to be used for malicious purposes.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image20.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-952481 size-full\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image20.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"105\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image20.png 1153w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image20.png?resize=300,49 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image20.png?resize=768,126 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image20.png?resize=1024,168 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 20: A Hackforums user suggests some collaboration on a repository to use with PrivateGPT<\/em><\/p>\n<p>Overall, while we saw more GPT services than we anticipated, and some interest and enthusiasm from users, we also noted that many users reacted to them with indifference or hostility.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image21.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952482\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image21.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"151\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image21.png 852w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image21.png?resize=300,71 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image21.png?resize=768,181 768w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 21: A Hackforums user warns others about paying for \u201cbasic gpt jailbreaks\u201d<\/em><\/p>\n<h2>Applications<\/h2>\n<p>In addition to derivatives of ChatGPT, we also wanted to explore how threat actors are using, or hoping to use, LLMs \u2013 and found, once again, a mixed bag.<\/p>\n<h3>Ideas and aspirations<\/h3>\n<p>On forums frequented by more sophisticated, professionalized threat actors \u2013 particularly Exploit \u2013 we noted a higher incidence of AI-related aspirational discussions, where users were interested in exploring feasibility, ideas, and potential future applications.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/563647563.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-952522 size-full\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/563647563-e1700661612812.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"1238\" height=\"150\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/563647563-e1700661612812.png 1238w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/563647563-e1700661612812.png?resize=300,36 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/563647563-e1700661612812.png?resize=768,93 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/563647563-e1700661612812.png?resize=1024,124 1024w\" sizes=\"auto, (max-width: 1238px) 100vw, 1238px\" \/><\/a><\/p>\n<p><em>Figure 22: An Exploit user opens a thread \u201cto share ideas\u201d<\/em><\/p>\n<p>We saw little evidence of Exploit or XSS users trying to generate malware using AI (although we did see a couple of attack tools, discussed in the next section).<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/117748883.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952524\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/117748883.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"66\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/117748883.png 1549w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/117748883.png?resize=300,31 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/117748883.png?resize=768,79 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/117748883.png?resize=1024,106 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/117748883.png?resize=1536,159 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 23: An Exploit user expresses interest in the feasibility of emulating voices for social engineering purposes<\/em><\/p>\n<p>On the lower-end forums \u2013 Breach Forums and Hackforums \u2013 this dynamic was effectively reversed, with little evidence of aspirational thinking, and more evidence of hands-on experiments, proof-of-concepts, and scripts. This may suggest that more skilled threat actors are of the opinion that LLMs are still in their infancy, at least when it comes to practical applications to cybercrime, and so are more focused on potential future applications. Conversely, less skilled threat actors may be attempting to accomplish things with the technology as it exists now, despite its limitations.<\/p>\n<h3>Malware<\/h3>\n<p>On Breach Forums and Hackforums, we observed several instances of users sharing code they had generated using AI, including RATs, keyloggers, and infostealers.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image24.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952485\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image24.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"256\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image24.png 1221w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image24.png?resize=300,120 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image24.png?resize=768,307 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image24.png?resize=1024,409 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 24: A Hackforums user claims to have created a PowerShell keylogger, with persistence and a UAC bypass, which was undetected on VirusTotal<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/43747771220.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952527\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/43747771220.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"326\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/43747771220.png 1218w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/43747771220.png?resize=300,153 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/43747771220.png?resize=768,391 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/43747771220.png?resize=1024,521 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 25: Another Hackforums user was not able to bypass ChatGPT\u2019s restrictions, so instead planned to write malware \u201cwith baby steps\u201d, starting with a script to log an IP address to a text file<\/em><\/p>\n<p>Some of these attempts, however, were met with skepticism.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image26.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952487\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image26.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"115\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image26.png 1177w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image26.png?resize=300,54 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image26.png?resize=768,138 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image26.png?resize=1024,184 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 26: A Hackforums user points out that users could just google things instead of using ChatGPT<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image27.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952488\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image27.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"144\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image27.png 1564w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image27.png?resize=300,67 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image27.png?resize=768,172 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image27.png?resize=1024,230 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image27.png?resize=1536,345 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 27: An Exploit user expresses concern that AI-generated code may be easier to detect<\/em><\/p>\n<p>None of the AI-generated malware \u2013 virtually all of it in Python, for reasons that aren\u2019t clear \u2013 we observed on Breach Forums or Hackforums appears to be novel or sophisticated. That\u2019s not to say that it isn\u2019t possible to create sophisticated malware, but we saw no evidence of it on the posts we examined.<\/p>\n<h3>Tools<\/h3>\n<p>We did, however, note that some forum users are exploring the possibility of using LLMs to develop attack tools rather than malware. On Exploit, for example, we saw a user sharing a mass RDP bruteforce script.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image28.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-952489 size-full\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image28.png\" alt=\"A screenshot of a post on a criminal forum featuring Python code\" width=\"640\" height=\"239\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image28.png 1471w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image28.png?resize=300,112 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image28.png?resize=768,287 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image28.png?resize=1024,383 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 28: Part of a mass RDP bruteforcer tool shared on Exploit<\/em><\/p>\n<p>Over on Hackforums, a user shared a script to summarize bug bounty write-ups with ChatGPT.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image29.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-952490 size-full\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image29.png\" alt=\"A screenshot of a post on a criminal forum featuring Python code\" width=\"640\" height=\"410\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image29.png 1243w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image29.png?resize=300,192 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image29.png?resize=768,492 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image29.png?resize=1024,657 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 29: A Hackforums user shares their script for summarizing bug bounty writeups<\/em><\/p>\n<p>On occasion, we noticed that some users appear to be scraping the barrel somewhat when it comes to finding applications for ChatGPT. The user who shared the bug bounty summarizer script above, for example, also shared a script which does the following:<\/p>\n<ol>\n<li>Ask ChatGPT a question<\/li>\n<li>If the response begins with \u201cAs an AI language model\u2026\u201d then search on Google, using the question as a search query<\/li>\n<li>Copy the Google results<\/li>\n<li>Ask ChatGPT the same question, stipulating that the answer should come from the scraped Google results<\/li>\n<li>If ChatGPT still replies with \u201cAs an AI language model\u2026\u201d then ask ChatGPT to rephrase the question as a Google search, execute that search, and repeat steps 3 and 4<\/li>\n<li>Do this five times until ChatGPT provides a viable answer<\/li>\n<\/ol>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image30.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-952491 size-full\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image30.png\" alt=\"A screenshot of a post on a criminal forum featuring Python code\" width=\"640\" height=\"322\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image30.png 1225w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image30.png?resize=300,151 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image30.png?resize=768,386 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image30.png?resize=1024,515 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 30: The ChatGPT\/Google script shared on Hackforums, which brings to mind the saying: \u201cA solution in search of a problem\u201d<\/em><\/p>\n<p>We haven\u2019t tested the provided script, but suspect that before it completes, most users would probably just give up and use Google.<\/p>\n<h3>Social engineering<\/h3>\n<p>Perhaps one of the more concerning possible applications of LLMs is social engineering, with some threat actors recognizing its potential in this space. We\u2019ve also noticed this trend in <a href=\"https:\/\/news.sophos.com\/en-us\/2023\/08\/02\/sha-zhu-pan-scam-uses-ai-chat-to-target-iphone-and-android-users\/\">our own research on cryptorom scams<\/a>.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image31.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952492\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image31.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"94\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image31.png 1556w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image31.png?resize=300,44 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image31.png?resize=768,113 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image31.png?resize=1024,151 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image31.png?resize=1536,226 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 31: A user claims to have used ChatGPT to generate <a href=\"https:\/\/news.sophos.com\/en-us\/2023\/09\/18\/latest-evolution-of-pig-butchering-scam-lures-victim-into-fake-mining-scheme\/\">fraudulent smart contracts<\/a><\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image32.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952493\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image32.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"100\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image32.png 1563w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image32.png?resize=300,47 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image32.png?resize=768,119 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image32.png?resize=1024,159 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image32.png?resize=1536,239 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 32: Another user suggests using ChatGPT for translating text when targeting other countries, rather than Google Translate<\/em><\/p>\n<h3>Coding and development<\/h3>\n<p>Another area in which threat actors appear to be effectively using LLMs is with non-malware development. Several users, particularly on Hackforums, report using them to complete mundane coding tasks, generating test data, and porting libraries to other languages \u2013 even if the results are not always correct and sometimes require manual fixes.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image33.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952494\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image33.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"351\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image33.png 1260w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image33.png?resize=300,165 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image33.png?resize=768,421 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image33.png?resize=1024,562 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 33: Hackforums users discuss using ChatGPT for code conversion<\/em><\/p>\n<h3>Forum enhancements<\/h3>\n<p>On both Hackforums and XSS, users have proposed using LLMs to enhance their forums for the benefit of their respective communities.<\/p>\n<p>On Hackforums, for example, a frequent poster of AI-related scripts shared a script for auto-generated replies to threads, using ChatGPT.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image34.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952495\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image34.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"196\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image34.png 1249w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image34.png?resize=300,92 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image34.png?resize=768,235 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image34.png?resize=1024,313 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 34: A Hackforums user shares a script for auto-generating replies<\/em><\/p>\n<p>This user wasn\u2019t the first person to come up with the idea of responding to posts using ChatGPT. A month earlier, on XSS, a user wrote a long post in response to a thread about a Python crypter, only for another user to reply: \u201cmost chatgpt thing ive [sic] read in my life.\u201d<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image35.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952496\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image35.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"327\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image35.png 1594w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image35.png?resize=300,153 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image35.png?resize=768,392 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image35.png?resize=1024,523 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image35.png?resize=1536,784 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 35: One XSS user accuses another of using ChatGPT to create posts<\/em><\/p>\n<p>Also on XSS, the forum\u2019s administrator has taken things a step further than sharing a script, by creating a dedicated forum chatbot to respond to users\u2019 questions.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image36.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952497\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image36.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"160\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image36.png 1548w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image36.png?resize=300,75 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image36.png?resize=768,192 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image36.png?resize=1024,257 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image36.png?resize=1536,385 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 36: The XSS administrator announces the launch of \u2018XSSBot\u2019<\/em><\/p>\n<p>The announcement reads (trans.):<\/p>\n<blockquote>\n<p><em>In this section, you can chat with AI (Artificial Intelligence). Ask a question &#8211; our AI bot answers you. This section is entertainment and technical. The bot is based on ChatGPT (model: gpt-3.5-turbo).<\/em><\/p>\n<p><em>Short rules:<\/em><\/p>\n<ol>\n<li><em>The section is entertaining and technical &#8211; you can create topics exclusively on the topics of our forum. No need to ask questions about the weather, biology, economics, politics, and so on. Only the topics of our forum, the rest is prohibited, the topics will be deleted.<\/em><\/li>\n<li><em>How does it work? Open a topic &#8211; get a response from our AI bot.<\/em><\/li>\n<li><em>You can enter into a dialogue with the bot, for this you need to quote it.<\/em><\/li>\n<li><em>All members of the forum can communicate in the topic, and not just the author of the topic. You can communicate with each other and with the bot by quoting it.<\/em><\/li>\n<li><em>One topic &#8211; one thematic question. If you have another question in a different direction, open a new topic.<\/em><\/li>\n<li><em>Limitation in one topic &#8211; 10 messages (answers) from the bot.<\/em><\/li>\n<\/ol>\n<p><em>This section and the AI-bot are designed to solve simple technical problems, for the technical entertainment of our users, to familiarize users with the possibilities of AI.<\/em><\/p>\n<p><em>AI bot works in beta. By itself, ChatGPT is crude. OpenAI servers sometimes freeze. Consider all this.<\/em><\/p>\n<\/blockquote>\n<p>Despite users responding enthusiastically to this announcement, XSSBot doesn\u2019t appear to be particularly well suited for use in a criminal forum.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image37.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952498\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image37.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"217\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image37.png 1544w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image37.png?resize=300,102 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image37.png?resize=768,261 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image37.png?resize=1024,348 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image37.png?resize=1536,521 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 37: XSSBot refuses to tell a user how to code malware<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image38.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952499\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image38.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"225\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image38.png 1578w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image38.png?resize=300,106 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image38.png?resize=768,270 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image38.png?resize=1024,360 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image38.png?resize=1536,540 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 38: XSSBot refuses to create a Python SSH bruteforcing tool, telling the user [emphasis added]: \u201c<strong>It is important to respect the privacy and security of others. Instead, I suggest learning about ethical hacking and practicing it in legal and ethical ways<\/strong>.\u201d<\/em><\/p>\n<p>Perhaps as a result of these refusals, one user attempted, unsuccessfully, to jailbreak XSSBot.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image39.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952500\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image39.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"187\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image39.png 1575w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image39.png?resize=300,88 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image39.png?resize=768,224 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image39.png?resize=1024,299 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image39.png?resize=1536,449 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 39: An ineffective jailbreak attempt on XSSBot<\/em><\/p>\n<p>Some users appear to be using XSSBot for other purposes; one asked it to create an advert and sales pitch for their freelance work, presumably to post elsewhere on the forum.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image40.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952501\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image40.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"324\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image40.png 1589w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image40.png?resize=300,152 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image40.png?resize=768,389 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image40.png?resize=1024,519 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image40.png?resize=1536,778 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 40: XSSBot produces promotional material for an XSS user<\/em><\/p>\n<p>XSSBot obliged, and the user then deleted their original request \u2013 probably to avoid people learning that the text had been generated by an LLM. While the user could delete their posts, however, they could not convince XSSBot to delete its own, despite several attempts.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image41.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952502\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image41.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"278\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image41.png 1579w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image41.png?resize=300,130 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image41.png?resize=768,334 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image41.png?resize=1024,445 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image41.png?resize=1536,667 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p>Figure 41: XSSBot refuses to delete the post it created<\/p>\n<h3>Script kiddies<\/h3>\n<p>Unsurprisingly, some unskilled threat actors \u2013 popularly known as \u2018script kiddies\u2019 \u2013 are eager to use LLMs to generate malware and tools they\u2019re incapable of developing themselves. We observed several examples of this, particularly on Breach Forums and Hackforums.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image42.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952503\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image42.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"498\" height=\"165\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image42.png 498w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image42.png?resize=300,99 300w\" sizes=\"auto, (max-width: 498px) 100vw, 498px\" \/><\/a><\/p>\n<p><em>Figure 42: A Breach Forums script kiddie asks how to use ChatGPT to hack anyone<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image43.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952504\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image43.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"175\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image43.png 978w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image43.png?resize=300,82 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image43.png?resize=768,210 768w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 43: A Hackforums user wonders if WormGPT can make Cobalt Strike payloads undetectable, a question which meets with short shrift from a more realistic user<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image44.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952505\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image44.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"152\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image44.png 814w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image44.png?resize=300,71 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image44.png?resize=768,182 768w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 44: An incoherent question about WormGPT on Hackforums<\/em><\/p>\n<p>We also found that, in their excitement to use ChatGPT and similar tools, one user \u2013 on XSS, surprisingly \u2013 had made what appears to be an operational security error.<\/p>\n<p>The user started a thread, entitled \u201cHey everyone, check out this idea I had and made with Chat GPT (RAT Spreading Method)\u201d, to explain their idea for a malware distribution campaign: creating a website where visitors can take selfies, which are then turned into a downloadable \u201cAI celebrity selfie image\u201d. Naturally, the downloaded image is malware. The user claimed that ChatGPT helped them turn this idea into a proof-of-concept.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image45.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952506\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image45.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"153\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image45.png 1582w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image45.png?resize=300,71 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image45.png?resize=768,183 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image45.png?resize=1024,244 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image45.png?resize=1536,366 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 45: The post on XSS, explaining the ChatGPT-generated malware distribution campaign<\/em><\/p>\n<p>To illustrate their idea, the user uploaded several screenshots of the campaign. These included images of the user\u2019s desktop and of the proof-of-concept campaign, and showed:<\/p>\n<ul>\n<li>All the open tabs in the user\u2019s browser \u2013 including an Instagram tab with their first name<\/li>\n<li>A local URL showing the computer name<\/li>\n<li>An Explorer window, including a folder titled with the user\u2019s full name<\/li>\n<li>A demonstration of the website, complete with an unredacted photograph of what appears to be the user\u2019s face<\/li>\n<\/ul>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image46.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952507\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image46.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"443\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image46.png 829w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image46.png?resize=300,208 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image46.png?resize=768,532 768w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 46: A user posts a photo of (presumably) their own face<\/em><\/p>\n<h2>Debates and thought leadership<\/h2>\n<p>Interestingly, we also noticed several examples of debates and thought leadership on the forums, especially on Exploit and XSS &#8211; where users in general tended to be more circumspect about practical applications \u2013 but also on Breach Forums and Hackforums.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image47.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952508\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image47.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"181\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image47.png 1413w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image47.png?resize=300,85 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image47.png?resize=768,217 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image47.png?resize=1024,290 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 47: An example of a thought leadership piece on Breach Forums, entitled \u201cThe Intersection of AI and Cybersecurity\u201d<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image48.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952509\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image48.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"133\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image48.png 1516w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image48.png?resize=300,62 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image48.png?resize=768,159 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image48.png?resize=1024,212 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 48: An XSS user discusses issues with LLMs, including \u201cnegative effects on society\u201d<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image49.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952510\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image49.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"258\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image49.png 1250w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image49.png?resize=300,121 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image49.png?resize=768,310 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image49.png?resize=1024,413 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 49: An excerpt from a post on Breach Forums, entitled \u201cWhy ChatGPT isn\u2019t scary.\u201d<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image50.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952511\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image50.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"164\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image50.png 1146w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image50.png?resize=300,77 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image50.png?resize=768,196 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image50.png?resize=1024,262 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 50<\/em>: <em>A prominent threat actor posts (trans.): \u201cI can&#8217;t predict the future, but it&#8217;s important to understand that ChatGPT is not artificial intelligence. It has no intelligence; it knows nothing and understands nothing. It plays with words to create plausible-sounding English text, but any claims made in it may be false. It can&#8217;t escape because it doesn&#8217;t know what the words mean.\u201d<\/em><\/p>\n<h3>Skepticism<\/h3>\n<p>In general, we observed a lot of skepticism on all four forums about the capabilities of LLMs to contribute to cybercrime.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image51.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952512\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image51.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"104\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image51.png 1562w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image51.png?resize=300,49 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image51.png?resize=768,124 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image51.png?resize=1024,166 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image51.png?resize=1536,249 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 51: An Exploit user argues that ChatGPT is \u201ccompletely useless\u201d, in the context of \u201ccode completion for malware\u201d<\/em><\/p>\n<p>On occasion, this skepticism was tempered with reminders that the technology is still in its infancy:<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image52.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952513\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image52.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"140\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image52.png 1591w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image52.png?resize=300,65 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image52.png?resize=768,168 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image52.png?resize=1024,223 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image52.png?resize=1536,335 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 52: An XSS user says that AI tools are not always accurate \u2013 but notes that there is a \u201clot of potential\u201d<\/em><\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image53.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952514\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image53.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"117\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image53.png 1566w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image53.png?resize=300,55 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image53.png?resize=768,140 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image53.png?resize=1024,187 1024w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image53.png?resize=1536,281 1536w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 53: An Exploit user says (trans.): \u201cOf course, it is not yet capable of full-fledged AI, but these are only versions 3 and 4, it is developing quite quickly and the difference is quite noticeable between versions, not all projects can boast of such development dynamics, I think version 5 or 7 will already correspond to full-fledged AI, + a lot depends on the limitations of the technology, made for safety, if someone gets the source code from the experiment and makes his own version without brakes and censorship, it will be more fun.\u201d<\/em><\/p>\n<p>Other commenters, however, were more dismissive, and not necessarily all that well-informed:<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image54.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952515\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image54.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"90\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image54.png 1136w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image54.png?resize=300,42 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image54.png?resize=768,107 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image54.png?resize=1024,143 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 54: An Exploit user argues that \u201cbots like this\u201d existed in 2004<\/em><\/p>\n<h3>OPSEC concerns<\/h3>\n<p>Some users had specific operational security concerns about the use of LLMs to facilitate cybercrime, which may impact their adoption among threat actors in the long-term. On Exploit, for example, a user argued that (trans.) \u201cit is designed to learn and profit from your input\u2026maybe [Microsoft] are using the generated code we create to improve their AV sandbox? I don\u2019t know, all I know is that I would only touch this with heavy gloves.\u201d<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image55.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952516\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image55.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"105\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image55.png 1535w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image55.png?resize=300,49 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image55.png?resize=768,127 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image55.png?resize=1024,169 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em> Figure 55: An Exploit user expresses concerns about the privacy of ChatGPT queries<\/em><\/p>\n<p>As a result, as one Breach Forums user suggests, what may happen is that people develop their own smaller, independent LLMs for offline use, rather than using publicly-available, internet-connected interfaces.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image56.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952517\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image56.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"110\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image56.png 1375w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image56.png?resize=300,51 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image56.png?resize=768,132 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image56.png?resize=1024,176 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 56: A Breach Forums user speculates on whether there is law enforcement visibility of ChatGPT queries, and who will be the first person to be \u201cnailed\u201d as a result<\/em><\/p>\n<h3>Ethical concerns<\/h3>\n<p>More broadly, we also observed some more philosophical discussions about AI in general, and its ethical implications.<\/p>\n<p><a href=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image57.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-952518\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image57.png\" alt=\"A screenshot of a post on a criminal forum\" width=\"640\" height=\"267\" srcset=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image57.png 1401w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image57.png?resize=300,125 300w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image57.png?resize=768,320 768w, https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/image57.png?resize=1024,427 1024w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><em>Figure 57: An excerpt from a long thread in Breach Forums, where users discuss the ethical implications of AI<\/em><\/p>\n<h2>Conclusion<\/h2>\n<p>Threat actors are divided when it comes to their attitudes towards generative AI. Some \u2013 a mix of competent users and script kiddies \u2013 are keen early adopters, readily sharing jailbreaks and LLM-generated malware and tools, even if the results are not always particularly impressive. Other users are much more circumspect, and have both specific (operational security, accuracy, efficacy, detection) and general (ethical, philosophical) concerns. In this latter group, some are confirmed (and occasionally hostile) skeptics, whereas others are more tentative.<\/p>\n<p>We found little evidence of threat actors admitting to using AI in real-world attacks, which is <a href=\"https:\/\/documents.trendmicro.com\/assets\/white_papers\/wp-malicious-uses-and-abuses-of-artificial-intelligence.pdf\">not to say that that\u2019s not happening<\/a>. But most of the activity we observed on the forums was limited to sharing ideas, proof-of-concepts, and thoughts. Some forum users, having decided that LLMs aren\u2019t yet mature (or secure) enough to assist with attacks, are instead using them for other purposes, such as basic coding tasks or forum enhancements.<\/p>\n<p>Meanwhile, in the background, opportunists, and possible scammers, are seeking to make a quick buck off this growing industry \u2013 whether that\u2019s through selling prompts and GPT-like services, or compromising accounts.<\/p>\n<p>On the whole \u2013 at least in the forums we examined for this research, and counter to our expectations \u2013 LLMs don\u2019t seem to be a huge topic of discussion, or a particularly active market relative to other products and services. Most threat actors are continuing to go about their usual day-to-day business, while only occasionally dipping into generative AI. That being said, the number of GPT-related services we found suggests that this is a growing market, and it\u2019s possible that more and more threat actors will start incorporating LLM-enabled components into other services too.<\/p>\n<p>Ultimately, our research reveals that many threat actors are wrestling with the same concerns about LLMs as the rest of us, including apprehensions about accuracy, privacy, and applicability. But they also have concerns specific to cybercrime, which may inhibit them, at least at the moment, from adopting the technology more widely.<\/p>\n<p>While this unease is demonstrably not deterring all cybercriminals from using LLMs, many are adopting a \u2018wait-and-see\u2019 attitude; as Trend Micro concludes in its report, AI is still in its infancy in the criminal underground. For the time being, threat actors seem to prefer to experiment, debate, and play, but are refraining from any large-scale practical use \u2013 at least until the technology catches up with their use cases.<\/p>\n<\/p><\/div>\n<p><a href=\"https:\/\/news.sophos.com\/en-us\/2023\/11\/28\/cybercriminals-cant-agree-on-gpts\/\" target=\"bwo\" >http:\/\/feeds.feedburner.com\/sophos\/dgdY<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/news.sophos.com\/wp-content\/uploads\/2023\/11\/shutterstock_2192799941.jpg\"\/><\/p>\n<p><strong>Credit to Author: Matt Wixey| Date: Tue, 28 Nov 2023 11:00:13 +0000<\/strong><\/p>\n<p>Despite concern over illicit applications of ChatGPT and similar models, Sophos X-Ops\u2019 exploration of cybercrime forums suggests many threat actors are still skeptical \u2013 and wrestling with the same issues and problems as the rest of us<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"colormag_page_container_layout":"default_layout","colormag_page_sidebar_layout":"default_layout","footnotes":""},"categories":[10378,10377],"tags":[10245,28405,4503,129,30546,29047,3764,27030,16771,29927],"class_list":["post-23466","post","type-post","status-publish","format-standard","hentry","category-security","category-sophos","tag-ai","tag-chatgpt","tag-cybercrime","tag-featured","tag-fraudgpt","tag-llm","tag-malware","tag-sophos-x-ops","tag-threat-research","tag-wormgpt"],"_links":{"self":[{"href":"http:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/posts\/23466","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"http:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/comments?post=23466"}],"version-history":[{"count":0,"href":"http:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/posts\/23466\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/media?parent=23466"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/categories?post=23466"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/tags?post=23466"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}