Bot Detection 101: How to Detect (and Beat) Bot Traffic
Bot detection is crucial for protecting websites and apps from malicious activities. With bots accounting for almost half of internet traffic, distinguishing between legitimate users and automated bots can be murky and challenging — but the impact on the security, cost-effectiveness, and ongoing viability of your applications is substantial.
This article explains the top methods you can use to identify and block bots from your websites, apps, APIs, and other online services, ensuring the security and integrity of your applications and their associated data.
Why bot detection, and why now?
Whenever a financial opportunity presents itself, a fraudster will be there to exploit it. This will likely never change no matter how much we advance our machine learning or how well we study the dark web's elusive tricks.
Bots, some of the most technically advanced tools in a fraudster’s disposal, are widely used in fast-evolving attacks on websites, mobile apps, and APIs. As such, the importance of bot detection and the need to block bad bots is greater than ever. Today, developers and organizations must have a strategy for blocking increasingly intelligent and elusive bots in order to safeguard sensitive customer and business data from the hands of fraudsters.
Defending your application from a bot-driven internet
Bot detection is not merely a defensive or reactive strategy; it’s an ongoing practice in maintaining the integrity of web interactions. By identifying patterns and signals that indicate bot activity, businesses and developers can distinguish between actual human users and automated bots.
By some estimates, about half of the nearly 50% of internet bot traffic is malicious. The other half is made up of benign or “good” bot activity, making it tricky for modern technology to discern the subtle clues that distinguish between not just humans and bots, but good and bad bots.
While good bots serve useful purposes (for example, web indexing done by search engine bots), you must protect your customers from malicious bots: data breaches, scraping, spamming, account takeovers, DDoS attacks, carding, and inventory hoarding are just some of the harmful outcomes of unmitigated bot access for your users. Your business also suffers: your reputation can be tarnished if there is an increased risk of fraud caused by using your product, and you may see an increase in infrastructure costs if you are providing services to synthetic users or fall victim to compute abuse.
Bot detection and the rise of AI agents
If they haven’t already, applications will also soon see the arrival of AI agents. These are "good bots" that act autonomously on behalf of real users and may actually spend real money — for example, by automatically booking plane tickets when a user buys conference tickets in another city, or making a regular purchase when a customer's inventory is low.
This will result in a huge increase of bot traffic, and make it even more difficult to identify whether a user is performing legitimate actions or attempting to engage in malicious behavior. This must be factored into application security and bot detection: good bots must be identified as such and treated with the same respect as real users while they perform their programmatic duties.
How do you detect bot traffic on the modern web?
With the increasingly widespread use of privacy tools like VPNs and TOR, identifying the egress point of traffic is not enough to identify a unique visitor or for the purpose of bot detection. Simple detection methods like watching for abnormally high request rates are also well known to bot authors, and they only provide basic protection.
Bots adapt quickly and use multiple techniques to mask their traffic and blend in, including regularly changing their IP address, manipulating request rates, and using JavaScript to spoof properties such as their device attributes, user agent and request headers. To address this, modern bot detection examines various signals across user and device behavior to solve the puzzle where each piece — whether it be a click, a scroll, or an IP address — reveals the bigger picture of who (or what) that visitor is, and flags markers of potential fraud.
For example, traffic from an evasive botnet may be detected due to it occurring at an unusual time or from an unusual location; if your application sees thousands of users arrive from Rendville, Ohio (population 28) at 3 a.m., there's a good chance that it's not coming from legitimate users.
Bot detection techniques and technologies
Successful bot detection is reliant on precise, rapid data ingestion that paints a holistic view of web traffic online. The following bot detection techniques should be considered as central to creating an accurate profile of a user:
IP and request frequency analysis
IP analysis monitors traffic to discern patterns that indicate bots, scrutinizing the origin, frequency, and volume of IP requests to identify potential threats. This detection method can also check if traffic comes from known data centers or IP addresses or ranges associated with bots. Abnormal traffic spikes or repeated access attempts from specific IPs can also be used to identify and halt DDoS attacks. Additionally, IP analysis supports geolocation-based strategies for blocking bots, restricting access from regions frequently tied to malicious activities.
Behavioral analysis
Behavioral analysis is the science of studying user patterns across web and app activity, scrutinizing every click, scroll, and keystroke for telltale signs of bad bots. By examining the cadence of user interactions, this technique can detect bot traffic by discerning the erratic nature of human activity from the more predictable, programmatic nature of bots. As the AI tools behind modern bots continue to improve, behavioral analysis must constantly evolve in its ability to detect the increasingly minute, subtle discrepancies that will define highly capable modern bots.
JavaScript tagging
JavaScript tagging is a behavioral analysis technique that tracks user activities on websites by embedding a snippet of code into web pages. It captures real-time behaviors like mouse movements and keystrokes to provide insights into user interactions and distinguish genuine engagement from automated patterns. When behaviors deviate from typical human interactions (such as unusually fast keystrokes or repetitive mouse movements) JavaScript tagging quickly flags these anomalies as potential bot activity.
CAPTCHA and reCAPTCHA challenges
CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is the friendly, somewhat funky request you may occasionally stumble across that asks you to type a blurry string of characters or words into a text box or simply mark a box stating that you're not a robot. CAPTCHA challenges like these separate human ability from a would-be bot attack or bad bot traffic.
Device fingerprinting
Device fingerprinting (DFP) looks at the unique characteristics of user devices, such as browser type, screen size, operating system, or time zone to unearth subtle clues that betray a bot’s presence. By cataloging the intricate details of a device’s configuration, DFP creates a profile for a user device that is as distinctive as a human fingerprint. Inconsistencies in these profiles can be the red flags that signal a bot’s attempt to infiltrate a website or app under the guise of legitimacy.
Aside: Many DFP solutions available today can be reverse-engineered by attackers, leaving your site exposed. Stytch Device Fingerprinting employs an in-depth strategy that obfuscates the fingerprinting process, making it resistant to tampering and spoofing. This approach ensures robust protection against even the most advanced attackers, combining industry-standard and proprietary signals into stable identifiers, fortified with multiple security layers to thwart reverse-engineering.
Machine learning algorithms
Machine learning (ML) algorithms can analyze far larger volumes of data on user behavior than a single person (or even a team) can. Bot detection tools that utilize ML can track the evolution of bot and human behavior in real-time, identify new behavior patterns that fall outside what is "normal" for your application, and assist with the bot detection process.
Implementing effective bot detection tools
Developers building a bot detection solution fit for modern application ecosystems should ensure that they are:
- Adaptable and robust, capable of scaling with web traffic and evolving as fraudsters’ techniques change
- Flexible to the unique demands of different digital environments
- Seamlessly integrated and customizable
- Invisible, so as to reduce user friction and imitability from fraudsters as much as possible
With the right tools configured to fit the intricacies of their domains, developers can create an ecosystem where bots are reliably detected and neutralized. This supports a secure environment for the business to transact and scale without the fear of multi-faceted attacks from bad bots of all kinds.
Poorly implemented bot detection negatively impacts real users
One of the problems with implementing your own bot detection through custom code or web application firewall (WAF) rules is when it locks real users out as well as the bots.
The broad traffic management rules made available by most WAF platforms are unsuitable for effectively blocking bad bots (and only bad bots). It's like using a hammer where surgical precision is required. Blocking by IP range or broad geographic region or using rigid access control lists (ACLs) puts out fires by quickly blocking access to large swathes of traffic, but this also inevitably blocks real users. These users will then either be turned away from your product or be forced to contact your support channels to resolve the issue.
This is made more difficult by hybrid attacks, in which real people solve CAPTCHAs for bots and then hand the session over to a bot to continue their automated activities. Rather than broad and overreaching WAF rules or ineffective custom code that can't keep up with new bot behaviors, stable identifiers for every device and visitor to your app are required for the best possible bot detection coverage that won't frustrate your customers.
Beating bad bots with Stytch Device Fingerprinting
Businesses and developer teams who need to scale quickly can see huge security and efficiency benefits from third-party bot detection solutions. Stytch Device Fingerprinting (DFP) seamlessly integrates production-grade DFP with your applications, enabled by well documented and flexible APIs and SDKs for real-time bot detection that doesn't impact real users.
Stytch's Device Fingerprinting utilizes every bot detection technique explained above, including IP fingerprinting with intelligent rate limiting, and a sophisticated JavaScript library and Web Assembly binary that collects signal data and hardware and browser information.
This information is evaluated and consolidated to generate highly stable device identifiers that remain consistent in the face of anonymization measures such as incognito browsing, VPNs, and changes in user agent or IP address. These bot detection technologies are supplemented by a supervised ML model, ensuring reliable bot detection that is much more accurate than solutions that rely solely on CAPTCHA/reCAPTCHA.
Additionally, Stytch offers customizable recommendations to allow, block, or challenge device access based on risk assessments. This flexibility is crucial for dynamically responding to potential security threats, ensuring robust security while maintaining user experience. Precise traffic management and tailored security can also be implemented with granular control based on device classification, trust levels, and authorization status.
For more information on how Stytch's Device Fingerprinting can help you block bad bots, check out our docs section or learn how to bootstrap Stytch DFP in just a few minutes.
Need additional help getting started? Reach out to a Stytch authentication expert.
Share
