May 1, 2023
Author: Reed McGinley-Stempel
As AI continues to transform the digital landscape, organizations must remain vigilant and adapt their application security strategies to address new threats and opportunities. By understanding the implications of AI on identity and access management (IAM) and taking proactive measures, businesses can stay ahead of the curve and protect their digital assets in an increasingly interconnected world.
AI-fueled risks include more sophisticated bot traffic, more believable phishing attacks, and the rise of legitimate AI agents accessing customers’ online accounts on behalf of users. In effect, applications will need to gain confidence in their ability to do the following:
In this article we’ll take a look at the main threats to cybersecurity posed by recent advances in AI, and provide concrete actions companies can take today to protect their applications and customers.
Even if you’ve heard of or even tried ChatGPT, it can be hard to sift through the media chatter to understand the real risks and real promises of this technology. At a CIAM company like Stytch, we’re far less concerned about a possible Hal or Blade Runner scenario, and much more concerned about what happens as the advances of AI become available to human beings already in the business of fraud and hacking.
While there’s quite a bit of nuance to how different kinds of AI tech are improving today, the crux of the issue from a cybersecurity perspective comes down to developments in sophistication and availability.
If you’ve caught the pope sporting some pretty pricey threads or gotten a strange voicemail from a close friend or relative, this larger point may be self-evident. But the past decade has given AI scientists both the time and resources to sharpen their deep learning models, paired with vastly greater computer power. As a result, AI has gotten, well, impressively good. While there are of course areas where AI still has room to improve, the gap between the output a computer and a human can generate has gotten much smaller.
AI technology is more widely and openly available for anyone to try than it ever was before. This is true on two fronts: both that there are more consumer AI products on the market (think ChatGPT) that anyone can try using, and with the advances in this technology there’s been a proliferation of startups and companies that rely on or advertise AI as a core differentiator. As of 2023, the number of AI companies in the world has doubled.
Like with other technological developments, if you see legitimate industry leaders getting excited about a new development, you can be sure hackers and fraudsters are paying attention too.
Let’s look closer at what precisely the risks are, and why they’re so urgent to address.
While a lot of the initial discussion around AI and cybersecurity focused on how hackers may use AI tools to better imitate human behavior, many are also interested in the sheer computing power exposed by legitimate AI applications that make their resources publicly available.
Historically, whenever a resource has been made available online, cybercriminals will seek methods to exploit it if there are sufficient financial incentives. Namely, hackers reverse engineer sites to piggyback on open resources, allowing them to automate and scale the reach of their attacks or fraudulent activity quickly and easily.
Past resources hackers have exploited include:
A Twilio diagram describing the fraud ring involved in SMS toll fraud (aka “sms pumping”).
With the rise of open AI resources over the past couple of years, this last kind of compute resource attack has become especially profitable. In this model, bots piggyback on an application’s AI resources using a man-in-the-middle approach. Attackers then sell access to the API outputs to a third party while leveraging a legitimate application’s API pathways to avoid paying for the AI APIs.
There are four main aspects to AI specifically that make this compute resource exploitation so attractive:
In other words, the same computing power that makes a search engine like You.com impressive or a productivity tool like Tome so helpful for a non-malicious user is equally available and perhaps more attractive for illicit purposes.
If you doubt the temptation of open AI resources, just look into GPT4Free. Just last week, this GitHub project dedicated to reverse engineering sites to piggyback on GPT resources accumulated 15,000+ stars on GitHub in just a few days. If over 15,000 developers saved this repository in just a few days, it’s daunting to imagine how many fraudsters might attempt to use it in the course of the coming months or year.
GPT4Free is an open-source project that reverse engineers LLM APIs in order to provide users with free access to AI capabilities by piggybacking on sites that pay for their usage.
If you find it uncanny how human-like AI can seem, either with audio, visuals, or text, you can be sure those imitative capabilities have also caught the eye of hackers. Namely, AI today accelerates three major “developments” along the phishing attack vector: believability, efficiency, and scale.
Aided by AI, phishing is already becoming much more complex and believable when it comes to the linguistics and visual interfaces generated by attackers to deceive users. Cybersecurity firm Darktrace saw the linguistic complexity jump in phishing emails nearly 20% in Q1 2023. Additionally, 37% of organizations reported deep fake voice fraud attacks and 29% reported deep fake videos being used for phishing.
As the cost of building a fake landing page and login portal plummets ever lower, security experts expect attackers’ sophistication in spinning up convincing sites to continue to rise.
Fraudsters are continuing to chase efficiency and scalability in their attacks. In the case of phishing, this means turning to programmatic ways to bypass MFA defenses.
In addition to generating phishing campaigns and sending them programmatically, attackers are also employing bots more frequently to automate the actual validation of phished 2FA codes (e.g. OTP, TOTP) in real-time. To achieve this, attackers operate low-volume bots that appear normal traffic-wise to applications (after all, there’s a real user on the other end) and submit the stolen codes to the phished application. Doing so eliminates the need for fraudsters to employ humans to perform this action and ensures stolen credentials aren’t stale.
As an example, Microsoft Threat Intelligence researchers recently discovered a phishing kit that is behind more than 1 million malicious emails daily and can bypass multi-factor authentication (MFA) systems using time-based one-time passwords (TOTPs). The kit, which sells for $300 (standard) or $1,000 (VIP), includes advanced features for deploying phishing campaigns and evading anti-phishing defenses. The technique, known as Adversary in the Middle (AitM), places a phishing site between the targeted user and the site they are attempting to log in to, capturing TOTPs in real time. The phishing kit also uses features such as CAPTCHA and URL redirection to bypass automated defenses and block lists.
Diagram courtesy of Microsoft’s Threat Intelligence team and their research on phishing kits
Security experts have long lauded MFA methods like OTP and CAPTCHA is important prophylactics against phishing attacks, so the ability of fraudsters to get around these blockers proves that greater protection will soon be needed on a global scale.
As phishing attacks become both trivial to generate with AI and much more believable, this also changes the cost-benefit analysis behind how attackers choose who to target.
We often mock the errors found in phishing schemes because they seem obvious to the discerning user. However, these typos have traditionally been a feature (not a bug) for attackers – they’re intentional and designed to weed out more intelligent targets so that attackers are more likely to hear back from only the most gullible of victims.
This design choice derives from phishing’s historically manual and human labor-intensive architecture: historically, hackers have needed real people to continue the conversation with potential victims to convince them to forward their password or MFA passcode. With this very real labor constraint, attackers have been forced to find clever ways to increase their probability of success, and focus their efforts only on targets they feel like both offer a high probability of success and offer sufficient reward.
Unfortunately, conversational-level AI like ChatGPT changes this cost benefit analysis drastically. Without human labor as a constraint, we can expect a sharp uptick in the number of phishing attempts and the sophistication of those conversations, as fraudsters may abandon the typo strategy and generally target a much wider audience with far more believable attacks.
The last major change we’re seeing from AI is the most nascent but also potentially the most fundamental shift for every application’s authentication strategies: the emergence of AI agents.
AI agents are designed to browse the web and perform tasks on behalf of users. In terms of benefits, these agents promise to eliminate tedious tasks in users’ personal and professional lives. In some ways, they’re like a more intelligent form of robotic process automation (RPA). But instead of requiring specific instructions, users can provide general directions, and the AI agent can problem-solve to fulfill the request.
This not only increases efficiency for end users but also brings significant advantages for businesses. AI agents can become adept at navigating unfamiliar user interfaces, reduce the customer support load for apps, and as a result, increase the likelihood of consumer or organizational adoption because users don’t have to learn the nooks and crannies of every app’s UI that they want to use. It’s no wonder agents like AutoGPT have been growing in popularity over the past couple of months.
The AutoGPT project is one of the fastest Github projects to 100k+ stars in history, and it has the potential to dramatically change how users and businesses interact with applications.
At the same time, AI agents also introduce an entirely new set of risks for applications to manage. Most consumers and businesses may not want these agents to have unrestricted access to all their application settings due to their autonomous nature. An AI agent could mistakenly delete crucial profile information, sign up for a more expensive subscription than intended, expose sensitive account access data, or take any other number of self-directed actions that harm the user, the application, or both parties. As of today, companies’ authentication and authorization models still have to catch up to these new kinds of divisions of labor.
ChatGPT now has a version with browsing capabilities in alpha access
With the pace of AI advancement showing no signs of slowing down, it may feel daunting to think about the implications for your individual product.
Fortunately, there are already tools out there that can help combat fraudulent use of AI – devs just have to know what capabilities to prioritize. To prepare their application’s security model in the age of artificial intelligence, engineers should consider the following:
To mitigate these risks, organizations should invest in advanced fraud and bot mitigation tools to prevent reverse engineering of AI APIs. Common anti-bot methods like CAPTCHA, rate limiting, and JA3 (a form of TLS fingerprinting) can be valuable in defeating run-of-the-mill bots that have little incentive to invest in evading these protections. But for more sophisticated bot problems like those facing AI endpoints, these common methods are easily defeated. Attackers with any moderate sophistication can defeat these methods:
There are of course many device fingerprinting solutions to choose from, but there are a few differentiators we’ve invested in at Stytch that are particularly effective at mitigating AI-driven attacks:
Whether you invest in Stytch’s product or decide on a competitor, the most important takeaway here is that organizations that prioritize detecting and stopping reverse engineering will be better equipped to protect their AI-powered applications against malicious actors.
The best way to counter these types of account takeover attacks is to invest in phishing-resistant MFA. Unlike one-time and time-based passcodes, phishing-resistant methods like WebAuthn can’t be accidentally forwarded along to an attacker, making it a valuable security measure in the face of AI-generated phishing emails and websites. WebAuthn includes two distinct categories of device-based authentication factors that help confirm a user’s identity:
With the increasing sophistication of phishing attacks, organizations should prioritize integrating phishing-resistant authentication methods. And if your organization isn’t yet ready to adopt to these more advanced MFA methods, the best defense is a fine-grained bot detection layer. In the above diagram, steps 2 and 6 require fraudsters to proxy user inputs from a fake application to the phished application and can be thwarted by investing in a fingerprinting approach (like Stytch’s) that can detect proxied traffic that otherwise appears to be normal.
To address these challenges, companies must adapt their permission and access control models to accommodate these well-intentioned bots. It’s crucial to establish fine-grained role-based access controls (RBAC) to mitigate security concerns arising from autonomous AI conducting tasks unsupervised on behalf of real users. Businesses may need to require human-in-the-loop MFA for certain actions initiated by AI agents or restrict some access entirely to strike the right balance between convenience and security.
Whether you’re building an AI company or reevaluating AI-driven attack vectors that might affect your application, we’d love to chat. Let’s talk about how Stytch can secure your product.