Save time with unbiased, independent feedback on vendor solutions.
Watch weekly bite-sized webinars hosted by IANS Faculty.
ChatGPT is being used just about everywhere, from the floor of the U.S. House of Representatives
to the classroom. This piece details potential ChatGPT misuse scenarios around common vulnerabilities and as well as current AI concerns
for security teams today.
Much has been made of the expansion of AI technology into the general user space. While, in social media, image-based AI such as Craiyon and Midjourney have become popular, the recent release of ChatGPT has made significant waves in the business and IT
fields. This AI model uses a “text chat” style of interface designed to generate text-based content, such as essays, policy documents and code.
The promise of AI used in this way is that it is possible to quickly and easily generate “content,” where the nature of the content aligns with how the AI model was developed. This varies by the type of engine, the initial “training”
material used for the engine, how much time was spent developing the model, and the skill of the engineers working on the system. In general, how well the promise of AI is met depends largely on the quality of the output, given that the speed of any
AI system is going to exceed human capability by several orders of magnitude.
The threat largely comes from human fears, which are often stoked by science fiction stories. The worst and most dramatic AI predictions are unlikely to come true in the near future. Instead, the more likely negative scenario is misuse of AI through misapplication,
as opposed to outright malice or negligence.
ChatGPT has some safeguards in place designed specifically to prevent the most obvious abuses. For example, when the engine is asked to generate a phishing email, it basically says no and explains why it is prevented from doing so:
However, taking just a slightly different approach can provide attackers with significant guidance for customizing phishing attacks. Let’s change the question to: What would a 25-year-old male administrative assistant, with a college degree, living
in Chicago find interesting? ChatGPT is then much more forthcoming. It returns the following:
The results are even more interesting when you attempt to target people’s bosses. Consider the query: What do CEOs of large insurance companies have in common and what interests them? ChatGPT returns the following:
If we then combine these two ideas, it is possible to identify subjects that could be used to hijack the interest of the administrative assistant to get them to click on something they think might advance their career and, with luck, get a direct connection
to that company’s CEO. Consider the following example:
AI-generated phishing, smishing and vishing campaigns have yet to be seen, but if the adoption by malware writers is any indication, it’s not far.
READ: Detect and Prevent Clone Phishing Attacks
The checks against misuse are even weaker when it comes to generating code. There have been reports of unskilled attackers generating malicious code using just the descriptions of the code from malware researchers. Although these initial trials do not
use secure techniques and are likely to be easily caught by existing anti-malware and endpoint detection and response software, obfuscation techniques are trivial, and the tools needed to take ChatGPT output as input to an obfuscation routine are
almost certainly being developed now.
While it is likely that, as the engine matures and more controls are added, this type of code generation will become more difficult—the critical lesson here is that a technology that lowers a barrier to entry, lowers it for everyone, not just the
“good guys.” This is a problem that applies to all arms races but is particularly noticeable with early-stage technologies, such as ChatGPT.
It is useful to consider vulnerabilities from a class-based view. Many times, we see significant industry improvement when entire classes of vulnerabilities are eradicated—such as when direct memory access attacks, like buffer overflows, became
non-issues when languages were released that simply didn’t allow that sort of access. Unfortunately, these issues were replaced by new classes of vulnerabilities, such as incorrect variable typing and improper use of features bundled with the
language, such as network stack abuse.
As ChatGPT evolves, it very likely will eliminate an entire class of vulnerabilities, such as common errors made by entry-level programmers. However, it almost certainly will introduce new classes of vulnerabilities, such as common shortcuts and errors
made by the AI engine. On this topic, time will tell, but it will be particularly interesting when it is possible to play multiple code generation AI systems against one another.
Besides misuse the biggest concern today for businesses is the speed/quality tradeoff when using ChatGPT. Experimentation with AI code and document generation systems should be encouraged because this technology is unlikely to go away, and early experimentation
(not necessarily adoption) usually benefits the organization. However, great care should be taken when directly using the outputs of a system like ChatGPT. To sidestep the thorniest issues:
Although reasonable efforts will be made to ensure the completeness and accuracy of the information contained in our blog posts, no liability can be accepted by IANS or our Faculty members for the results of any actions taken by individuals or firms in
connection with such information, opinions, or advice.
December 7, 2023
By IANS Research
Learn how to create an actionable CISO dashboard with meaningful security metrics using the three C’s principle that supports informed decision-making.
December 5, 2023
By Bryson Bort
As the year draws to a close, IANS Faculty provide their 2024 Cyber Predictions. Watch our video with Bryson Bort for tips on planning your 2024 IT/OT security strategy.
November 30, 2023
CISOs, find guidance on what to focus on within the first 30 days, 6 months and first year of your tenure to ensure a fast, successful start.