I’ve been hosting and managing WordPress websites for a long time, and every so often a problem comes along that doesn’t match any of the usual patterns. Earlier this week, every site on one of our servers slowed to a crawl. Some pages went completely blank. Others loaded without their stylesheets, that broken half-rendered look where the content shows up but the design is gone. And the strange part? The server had plenty of resources available.
If you run a server with WordPress sites on it, especially any site using an events calendar plugin, what I found might save you a very frustrating afternoon.
Everything Looked Fine on the Surface, but Problems Lurked Below
The specs on this server are respectable: 8 cores and 12GB of RAM. During the slowdown, the load average was hovering around 3 to 5. On an 8-core machine, that’s not even close to maxed out. Memory was mostly free. Swap was untouched. Disk I/O was quiet.
By every dashboard metric I’d normally check, the server should have been fast. It wasn’t.
I did what most of us do when a server misbehaves and nothing obvious explains it: a graceful reboot. It didn’t help. Within minutes, everything was slow again. That turned out to be an important clue. A reboot clears stuck processes and runaway memory. It does nothing against a problem that walks right back in the front door.
The Suspect That Couldn’t Have Done It
Resource monitoring showed one account consistently at the top of the usage charts. But here’s the thing: that account was limited to just 3 PHP-FPM workers. Three workers on an 8-core machine. How could that possibly drag down an entire server?
That question sent me down the wrong path for a bit, because the intuitive answer is “it can’t.” A worker cap feels like a safety fence. What I eventually understood is that a worker cap limits concurrency, not damage. Three workers pinned at 100% CPU around the clock is still nearly 3 full cores gone. And more importantly, those three workers were generating a flood of heavy database queries.
That’s the cascade: the busy site’s workers hammer MySQL, MySQL slows down for everyone, every other site’s PHP requests start stalling while they wait on their queries, stalled requests pile up and occupy all of Apache’s connection slots, and once those slots are full, even requests for plain static files like CSS get queued or dropped. That’s where the blank pages and missing stylesheets came from, on sites that had absolutely nothing to do with the problem.
Finding the Real Culprit
When I looked at the process list, the story got interesting. The account’s PHP workers each showed over 5 minutes of accumulated CPU time on a server that had only been up for half an hour. A normal PHP request finishes in milliseconds. Workers grinding at 100% for minutes at a time mean something is very wrong: an infinite loop, a stampede of expensive requests, or malware.
So I went to the access logs. First gotcha: on a cPanel server, the regular domain log only shows plain HTTP traffic, which for most modern sites is just redirected to HTTPS. The real traffic lives in the separate -ssl_log file. If you only check the regular log, a site under heavy fire can look almost idle.
The SSL log told a different story. Out of the last 5,000 requests, one single IP accounted for about 3,400 of them, a sustained rate of roughly 200 requests per minute. The user agent?
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +)ClaudeBot, Anthropic’s AI training crawler. And right behind it, OpenAI’s GPTBot doing the same thing. These weren’t malicious actors. They were well-known crawlers politely identifying themselves while accidentally running a denial-of-service attack.
The Infinite Calendar Problem
Here’s where it gets useful for anyone running WordPress. The site being hammered uses The Events Calendar plugin. Look at the kinds of URLs the bots were requesting:
/calendar/list/page/46/?ical=1&tribe-bar-date=2023-11-15
/calendar/list/page/13/?tribe-bar-date=2020-02-26
/?eventDate=2018-10-23&eventDisplay=day&paged=17&post_type=tribe_eventsEvery date, every view type (day, list, month), every page number, every combination of query parameters generates a unique URL. Date pickers go back years and forward forever. The crawlable URL space is effectively infinite, and AI crawlers are thorough. They will happily walk that space all day, every day.
Two properties make this so destructive:
- Every URL is unique, so page caching never helps. Each request is a full WordPress bootstrap.
- Calendar queries are expensive, joining posts and meta tables in ways that make the database work hard for every single hit.
Multiply that by 200 requests per minute, around the clock, and a modest events site becomes a server killer. This isn’t specific to one plugin either. Any faceted URL structure works the same way: layered navigation on WooCommerce stores, filterable directories, search pages with combinable parameters. If your URLs can be recombined infinitely, AI crawlers will find them.
Why I Didn’t Just Block Everything
My first instinct was to block AI crawlers site-wide, and as an emergency measure, that’s exactly what I did. Load dropped within minutes. But it’s worth pausing before making that permanent, because there’s a real tradeoff.
AI assistants increasingly answer people’s questions directly and cite their sources. When someone asks a chatbot about services in your area, you want your site to be in the running for that citation. Block every AI-related user agent and you’ve opted out of that channel entirely. Traditional SEO isn’t affected either way, since Google and Bing crawlers are separate, but AI visibility is becoming its own thing worth protecting.
The crawlers also come in two distinct flavors, and they deserve different treatment:
- Bulk training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider) systematically download everything. These are the ones causing the load.
- Live user-fetch agents (ChatGPT-User, OAI-SearchBot, Claude-User, PerplexityBot) only fire when a real person asks a question and the assistant fetches a page to answer it. Low volume, high value. These are the ones you want to keep.
So I landed on a tiered approach.
The Fix, In Three Tiers
Tier 1: junk scrapers, blocked everywhere. Bots like Bytespider that offer nothing in return and have a reputation for ignoring robots.txt. These get a 403 on every URL.
Tier 2: major AI crawlers, blocked only from the calendar. ClaudeBot, GPTBot, and friends can still read normal pages, so the site stays visible to AI assistants. They just can’t touch the infinite URL space that was causing the damage.
Tier 3: user-fetch agents and search engines, untouched.
The enforcement lives in two places. First, robots.txt politely asks compliant bots to stay away from the faceted URLs and slow down:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?*eventDate=
Disallow: /*?*tribe-bar-date=
Disallow: /*?*eventDisplay=
Disallow: /*?*ical=1
Disallow: /calendar/list/page/
User-agent: ClaudeBot
User-agent: GPTBot
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: Applebot-Extended
Crawl-delay: 10
Disallow: /wp-admin/
Disallow: /*?*eventDate=
Disallow: /*?*tribe-bar-date=
Disallow: /*?*eventDisplay=
Disallow: /*?*ical=1
Disallow: /calendar/list/page/Two details matter here. When a bot matches a named group, it ignores the * group entirely, so you have to repeat your Disallow rules in the named group. And Crawl-delay: 10 caps compliant bots at roughly 6 requests per minute instead of 200. ClaudeBot documents that it honors both.
One more robots.txt tip: make it a real, physical file. WordPress generates a virtual robots.txt through PHP, which means that when your server is overloaded, the very file that could call off the bots starts returning 500 errors. A static file always serves.
Second, .htaccess enforces the rules for bots that don’t ask permission:
ErrorDocument 403 "Forbidden"
<IfModule mod_rewrite.c>
RewriteEngine On
# Tier 1: bulk scrapers with no user-facing value - blocked site-wide
RewriteCond %{HTTP_USER_AGENT} (Bytespider|CCBot|omgili|Diffbot|ImagesiftBot|Timpibot) [NC]
RewriteRule .* - [F,L]
# Tier 2: major AI crawlers - blocked only from calendar URLs
RewriteCond %{HTTP_USER_AGENT} (ClaudeBot|GPTBot|Amazonbot|meta-externalagent|Applebot-Extended) [NC]
RewriteCond %{QUERY_STRING} (eventDate=|tribe-bar-date=|eventDisplay=|ical=1) [NC]
RewriteRule .* - [F,L]
RewriteCond %{HTTP_USER_AGENT} (ClaudeBot|GPTBot|Amazonbot|meta-externalagent|Applebot-Extended) [NC]
RewriteCond %{REQUEST_URI} ^/(calendar|events)(/[^/]+)*/page/ [NC]
RewriteRule .* - [F,L]
</IfModule>A 403 served by Apache costs almost nothing. The request dies before WordPress or the database ever get involved, which is the whole point.
The Trap That Almost Fooled Me
Now for the part I couldn’t find documented anywhere, and the reason this post exists.
After deploying the block rules, I tested them. Bot user agent on a calendar URL: 403.
Except it wasn’t my 403.
When I tested more carefully, using URLs that didn’t exist as real pages, the “blocked” requests were sailing straight through to WordPress. My rules weren’t firing at all in the way I thought. The 403s I was seeing came from the calendar plugin’s own PHP-level bot handling, which still costs a full WordPress bootstrap per request. The load problem would have quietly continued.
Here’s the mechanism, and it’s sneaky. See that first line in the code above, ErrorDocument 403 "Forbidden"? Without it, this happens:
- Your rewrite rule matches and Apache generates a 403.
- To render the error page, Apache makes an internal subrequest for the error document.
- That subrequest has no query string and doesn’t match your original URL, so it falls through to WordPress’s catch-all rewrite rule.
- WordPress boots up, doesn’t recognize the error document URL, and issues a 301 redirect to the homepage.
The result: the bot receives a friendly redirect instead of a 403, PHP runs anyway, and your block accomplishes nothing while appearing to work. Defining the error document inline as a plain string breaks the chain. No subrequest, no WordPress, just seven bytes of “Forbidden” and a closed connection.
The lesson that goes with it: test your blocks with URLs that don’t exist. A real page might be getting 403’d by a security plugin or the theme itself, which proves nothing about your Apache rules. Only an Apache-level block can 403 a URL that WordPress would otherwise 404. That distinction is the difference between a block that saves your server and one that just redecorates the problem.
The Aftermath
Load dropped from around 5 to under 2 within minutes of the final rules going in, and it stayed there. Real visitors never noticed a thing, which is the best kind of fix. The client’s site is still visible to AI assistants for normal content, search rankings are untouched, and the calendar now belongs to actual humans looking for actual events.
A few takeaways if you manage WordPress servers:
- Low load average doesn’t mean healthy. A shared database bottleneck can strangle every site while your CPU graph looks fine.
- A reboot that doesn’t fix it is information. It means the cause is external and recurring, not a stuck process.
- Check the SSL access logs, not just the default ones. That’s where the real traffic is.
- Worker limits cap concurrency, not collateral damage. Three workers can absolutely take down a server if they’re feeding an expensive shared resource.
- Faceted URLs plus AI crawlers is a fire waiting to happen. Calendars, filtered shops, directories. If URL combinations are infinite, someone’s crawler will eventually try to visit all of them.
- Verify blocks at the Apache level with nonexistent URLs, and always pair deny rules with an inline ErrorDocument on WordPress sites.
AI crawler traffic has exploded over the past couple of years, and from what I saw in my logs, it’s not slowing down. If you’ve run into something similar, whether it was a calendar, a store filter, or some other infinite URL generator, I’d like to hear how it showed up for you and what you did about it.
