Bots Never Sleep: How I Deal With Constant Bot Attacks
If you run anything on the public internet, you're not alone on it. The second a site goes live, something starts poking at it. Not a person. A script.
That's a big part of my day job. I look after a pile of our internal sites plus a bunch of customer sites, and the bots never take a day off. Login pages get hammered with stolen passwords. Contact forms fill up with spam. Scrapers crawl every page they can find, sometimes hard enough to slow the site down for real visitors. And there's the constant background noise of scanners hunting for /wp-admin, .env files and old phpMyAdmin installs on servers that have never run any of those.
The numbers back up the feeling. Recent industry reports say more than half of all web traffic is now automated, and around 40% of it comes from bad bots. Put another way, on a typical site, a lot of the "visitors" you're paying to serve aren't people at all, and plenty of them are up to no good.
Why It Got Worse
Two things changed in the last couple of years.
First, AI made bots smarter and cheaper. Writing a scraper that rotates IPs, fakes a real browser and clicks around like a human used to take real effort. Now someone can get a decent one generated in an afternoon. The reports I read put the growth of AI-driven bot attacks at something wild, like a double-digit multiple year over year. Even if you halve that, it's a lot.
Second, AI companies themselves are crawling everything. Training crawlers and AI "agent" traffic now make up a real share of requests on most sites. Some of them follow robots.txt. Some of them... don't. For a small customer site on a modest server, an aggressive crawler can look a lot like a DDoS, even when nobody means any harm.
And APIs are a big target. A good chunk of bot attacks now go after APIs directly instead of the pages, because APIs return clean data and often have weaker protection than the login form in front of them.
This Week's Fire Drill: PHP2Shell
This week was a perfect example. A pair of critical bugs in PHP itself (not some plugin, the actual language runtime) went public, quickly nicknamed "PHP2Shell". They're unauthenticated remote code execution bugs. In plain terms: under the right conditions, someone can run their own code on your server without logging in. They affect a huge range of versions, all the way from ancient PHP 5.4 up to the current 8.5.
The disclosure even got pushed up a day because details leaked, and proof-of-concept exploits showed up almost immediately. Within hours, scanners were already knocking on doors to see who was vulnerable. That's the bot problem in a nutshell. The time between "bug announced" and "bots are scanning the whole internet for it" is now basically zero.
So yeah, we had to rush. A lot of our internal tools and plenty of customer sites run on PHP, so the first job was figuring out what was running where and which versions. The good news: about 95% of our infrastructure was already on supported PHP versions behind our usual edge protection, so for most of it the fix was just "apply the patch release and move on." The remaining 5% was the usual suspects: an older app nobody wanted to touch and a couple of customer sites stuck on unsupported versions that don't get official fixes. Those are getting moved up, and quickly.
The lesson I keep relearning: firewall rules and bot protection buy you time, but for bugs like this there's no complete workaround. Patching is the actual fix. Staying on supported versions is what turns an emergency into a boring afternoon, and boring is exactly what you want from a security incident.
Start at the Edge With Cloudflare
My first rule: stop as much junk as possible before it ever reaches the server. That's why almost every site I manage sits behind Cloudflare (or something like it).
Putting a CDN and reverse proxy in front gets you a lot for very little work:
- Your origin IP stays hidden. Bots can't skip the protection if they can't find the real server. Lock the origin firewall down so it only accepts traffic from Cloudflare's IP ranges, otherwise the hiding doesn't count for much.
- DDoS protection comes built in. Big volumetric floods get absorbed at the edge instead of at your poor little VPS.
- Bot Fight Mode and bot scores. Cloudflare gives each request a guess at how likely it is to be a bot. Even the free tier can challenge the obvious ones.
- AI crawler controls. You can see which AI crawlers are visiting, how often, and block or allow them individually. On a couple of customer sites, just blocking the AI training crawlers cut the request volume noticeably.
It's not magic, and good bots still get through. But it takes care of the bulk of the dumb stuff, and that leaves you more time for the clever stuff.
Firewall Rules That Actually Help
Generic protection is great, but the real wins come from rules that know your site. Cloudflare's WAF custom rules (or Nginx/Caddy rules if you're self-hosting) let you write things like:
# Block scanners looking for stuff we don't run
(http.request.uri.path contains "/wp-login.php") or
(http.request.uri.path contains "/.env") or
(http.request.uri.path contains "/phpmyadmin")
→ Block
# Admin panel only from the office/VPN
(http.request.uri.path starts_with "/admin") and
not (ip.src in $office_ips)
→ Block
# Challenge login attempts from countries we never serve
(http.request.uri.path eq "/login") and
not (ip.geoip.country in {"US" "CA" "GB"})
→ Managed ChallengeA few rules I set up on pretty much every site now:
- Block paths you don't use. If it's not a WordPress site, nobody legit is asking for
/wp-login.php. Blocking those at the edge cuts a surprising amount of log noise. - Lock internal tools to known IPs. Our internal dashboards have no reason to be reachable from the whole planet. Restrict them to the VPN or office IPs, or put them behind something like Cloudflare Access so you have to log in before the app even loads.
- Challenge, don't always block. For anything iffy, like a weird country or a low bot score, a challenge is kinder than a hard block. Humans click through, scripts mostly don't.
- Watch the hosting providers. A lot of bad traffic comes from cloud and VPS IP ranges, not home internet connections. Challenging traffic from data center ASNs on sensitive endpoints like login works well.
One honest warning: geo-blocking and ASN rules are blunt tools. Real customers travel and use VPNs. Use challenges over blocks wherever you can, and keep an eye on what you're catching.
CAPTCHAs (But the Nice Kind)
Nobody likes clicking on traffic lights. Traditional CAPTCHAs annoy real users, and modern bots (and paid solving farms) can often beat them anyway.
That's why I've moved forms over to Cloudflare Turnstile. Most of the time it's invisible: it runs a few quick checks in the browser and the user never sees a puzzle. I use it on this very site's contact form, and it's on login, signup and contact forms on the sites I manage.
The part people forget: always verify the token on the server. The widget only gives you a token. If your backend doesn't send that token to Cloudflare's siteverify API and check the response, a bot can just skip the widget and post straight to your endpoint. I've seen that bug in the wild more than once.
A couple of cheap tricks still work nicely alongside it:
- Honeypot fields. Add a hidden form field that real users never see. If it comes back filled in, it's a bot. It's basically free and catches a lot of lazy spam.
- Timing checks. A human takes a few seconds to fill out a form. A submission that arrives 200ms after the page loaded probably isn't a person.
Rate Limit Everything (Especially APIs)
If I could only pick one defense, it'd probably be rate limiting. It doesn't care whether the client is a bot or a human. It just says "that's enough for now."
Where I put limits:
- Login and password reset. Credential stuffing (trying leaked username/password combos) is all about volume. A handful of attempts per minute per IP, plus a per-account limit, makes it way less useful.
- Forms. My own contact form allows a few submissions per IP per hour. Nobody needs to email me 50 times.
- APIs. Every endpoint gets a limit, usually per API key or per user and not just per IP. Expensive endpoints like search, exports and anything that hits an AI model get much stricter ones.
Per-IP limits alone aren't enough anymore, since serious bots rotate through thousands of residential IPs. So I layer them: per IP, per account, per API key, and sometimes per session. Do it at the edge (Cloudflare rate limiting rules) and in the app as well, so a bot that gets past one still runs into the other. And return a proper 429 Too Many Requests with a Retry-After header so well-behaved clients know to back off.
// Super simple sliding-window limiter. Fine for one server;
// use Redis or your edge provider once you have more than one.
const hits = new Map<string, number[]>();
function isRateLimited(key: string, max = 5, windowMs = 60_000) {
const now = Date.now();
const recent = (hits.get(key) ?? []).filter((t) => now - t < windowMs);
recent.push(now);
hits.set(key, recent);
return recent.length > max;
}The Other Stuff That Quietly Helps
A few more tools I lean on, roughly in the order I'd reach for them:
- MFA everywhere. Even if a bot guesses the right password, MFA stops it. For internal sites this isn't optional, full stop.
- Breached password checks. Reject passwords that already show up in known leaks. It hurts credential stuffing right where it lives.
- Fail2ban or CrowdSec. For self-hosted servers, these watch your logs and ban IPs that misbehave. CrowdSec also shares blocklists across its users, so you can block an attacker it spotted somewhere else before they ever reach you.
- Proof-of-work challenges. Tools like Anubis make the browser solve a small puzzle before the page loads. A human barely notices it, but a scraper hitting thousands of pages ends up paying for every single one. It's become popular with open source projects getting hammered by AI crawlers. It won't stop a determined, targeted bot, though.
robots.txtand AI crawler rules. It's an honor system, so it only works on the polite bots. Still worth having, because a lot of the big crawlers do respect it.- Caching. If a scraper is going to hit your pages anyway, serve them from cache. A bot reading cached HTML at the edge costs you basically nothing.
- Monitoring and alerts. Watch for spikes in failed logins, 4xx/5xx errors and traffic from weird places. A lot of credential stuffing attacks get mistaken for "the site is just slow today."
The honest truth is that there's no single setting that makes bots go away. Every layer catches a different slice: the edge eats the floods, firewall rules kill the scanners, Turnstile handles form spam, rate limits blunt credential stuffing and API abuse, and MFA saves you when everything else misses.
Bot protection is never really done. Bots adapt, so I check the firewall analytics every week or so, tweak rules that are catching too much or too little, and add new ones when some fresh weird pattern shows up. It's a bit like weeding a garden. You don't win once, you just keep at it. But with a few solid layers in place, the noise drops way down, the servers breathe easier, and you can spend your time building stuff instead of cleaning up after scripts.