When the Server Fell Over

The Setup

Both servers run RunCloud, with NGINX in front of PHP-FPM and MariaDB in Docker. The first has 4 CPU cores and 8 GB of memory for about 20 WordPress sites. Every site shares that memory and one database server, so one site that runs out of control can starve all of them.

During the cleanup, RunCloud Hub, RunCache and their Redis object cache came off every site. On one server they had stored and served the attacker’s spam pages, and on the other they overlapped with WP Rocket in ways nobody could reason about. WP Rocket replaced them as the only page cache. That was the right call, and it also left a few hours in which some sites had no working cache at all.

Ten Minutes Down

At 03:19 UTC one morning, Facebook’s crawler, facebookexternalhit, started requesting the homepage of the main brand site about 200 times a minute. On a normal day it sends about 15 requests an hour.

Three things meant every one of those requests ran WordPress in full:

• WP_CACHE was gone from wp-config.php, probably since RunCloud Hub was uninstalled, so WP Rocket’s page cache was not loading.
• The Redis object cache had just been removed as part of the switch to WP Rocket.
• WP Rocket does not serve its cache to the Facebook and WhatsApp crawlers. Even with caching working, they would have reached PHP.

The site’s PHP-FPM pool kept starting workers until it had 50, and together they used up the 8 GB. The kernel’s out-of-memory killer then stopped MariaDB. With the database gone, every site on the server went down.

1. 01
   Facebook's crawler requests one homepage 200 times a minute
2. 02
   No page cache WP_CACHE missing, and the crawler bypasses WP Rocket anyway
3. 03
   The site's PHP pool grows to 50 workers
4. 04
   Memory runs out, and the kernel kills MariaDB
5. 05
   Every site on the server returns errors

Getting It Back

I killed the site’s PHP workers and added a temporary NGINX rule that answered the Facebook crawler with 429 Too Many Requests. MariaDB came back, and so did the sites. I removed the 429 rule eight minutes after the outage started and kept it in quarantine for next time.

Then I got WP Rocket caching again. Deactivating it deletes its per-domain config file, and activating it again through WP-CLI does not always recreate the file, so caching fails without a single error. One command regenerates it:

wp eval 'rocket_generate_config_file();'

The test is to request the homepage twice. The second response should contain WP Rocket’s footer comment, “This website is like a Rocket”. If it does not, the page was not served from cache.

Serving Crawlers from Cache

Three hours later the Facebook crawler was still at about 33 requests a minute, and the load on four cores was about 15. Visitors got the cached homepage in 0.06 seconds. The crawler got a fresh render every time, 3 seconds on average and up to 51 seconds under that load.

A small must-use plugin now lets the Facebook and WhatsApp crawlers receive WP Rocket’s cached pages like any visitor. When a page is not cached yet, only two crawler renders can run at once per site, and further crawler requests get 503 with Retry-After: 60. Well-behaved crawlers come back later, and it keeps a link shared on Facebook from turning into a flood of PHP.

After the change, the crawlers got cached pages in 0.03 to 0.07 seconds. Six parallel crawler requests for a page that can’t be cached led to two renders and four quick 503 answers. Normal visitors were not affected.

A Cap on Every Pool

The deeper problem was that one site could start enough PHP workers to use all the memory on the server. RunCloud sets pm.max_children per site, and the defaults had been left alone:

• The main brand site is capped at 12 workers.
• Two training sites that were allowed 50 workers each are now capped at 5, like every other site.
• On the second server, the PHP 7.4 sites went from 50 to 10.

A capped pool queues requests instead of growing. Under a flood, that site slows down, and the rest of the server keeps running. Since the change, the only out-of-memory kill on the first server is the one from the outage, and the load sits between 1.7 and 2.4.

The Opcache Nobody Sized

The second server had a different problem. Its health report showed nine homepages timing out, the load average was about 210, and the PHP 7.4 container logged thousands of out-of-memory kills an hour. It looked like an attack, but the cause was the opcache, the shared memory where PHP keeps compiled scripts so it does not parse them again on every request. About 20 sites shared the PHP 7.4 container, and its opcache was still at the RunCloud default of 192 MB and 7,963 files. It was full, with a hit rate of 25%. So PHP recompiled most files on every request, which cost CPU and memory, and the leftover spam crawl traffic made it worse.

I raised it to 2,048 MB, 100,000 files and 128 MB for interned strings. The hit rate went to 95%, the load fell from about 210 to 7, and the out-of-memory kills stopped. The first server had the same problem in a milder form: PHP 8.1 was full at a 77% hit rate, and PHP 7.4 at 91%.

Two details made this harder than editing a number:

• The settings live inside the container. A RunCloud rebuild resets them, so the daily report checks the opcache size and alerts when it shrinks.
• The first edit changed nothing. On one server the script edited an ini file that PHP there does not load, and the first report said “opcache ok”. I now check the value in the running PHP, not in the file.

Caches That Fail Without an Error

Most of this week came down to caches that stopped working without saying so:

• Uninstalling one cache plugin can break another. RunCache’s deactivate hook deleted WP Rocket’s advanced-cache.php drop-in and the WP_CACHE constant on a site where both had been installed.
• A leftover cache with nothing to purge it. On one site, RunCloud Hub’s NGINX page cache was still live after its plugin was gone, so visitors could get stale pages.
• A site that answers 200 for junk fills its cache with copies of itself. One site returned its homepage with 200 for scanner probes such as /wp-content/backups/db.sql/. WP Rocket cached each one, about 460 copies of the homepage, which was part of why the disk filled up again within five days of the cleanup.

The daily report now checks that every site serves its pages from cache.

What I Would Change Next

The client asked whether moving the sites to headless WordPress or a static site would make the servers easier to run. My answer was that most of the pain so far came from operations, not from WordPress rendering pages, so I recommended this order:

1. Cache HTML at the edge. With Cloudflare APO or Cache Rules, most visitors and crawlers never reach PHP at all.
2. Give the applications their own servers. A careers site with tens of thousands of applicant files, a store and a learning platform behave like applications. They should not share memory with brochure sites.
3. Pilot a static rebuild. Rebuild one low-risk content site in Astro on Cloudflare Pages, with WordPress kept as a private CMS, and measure how editors cope without a page builder before deciding on the rest.

The headless setup I already run for one brand site follows that pattern. Its weak spot was the same: Cloudflare did not cache its HTML, so every page request still reached the server.

Lessons

Cap every pool
Without a cap, one site under a flood can use all the memory and take the database down with it. With a cap, that site slows down and the others keep running.

Test the cache with a second request
Request the page twice and look for the cache’s own marker. A plugin can be active with its config file missing.

Know which visitors skip the cache
Link-preview crawlers arrive in bursts when a link is shared, and the default setup sent every one of them to PHP.

Size the opcache for the number of sites
A full opcache made a shared PHP container recompile everything, and it looked like a traffic attack.

Verify a setting in the running process
A config file that PHP does not read will report whatever you write into it.

Apart from one memory increase on the second server, none of this needed new hardware. The servers had enough capacity once the caches worked and no single site could take all of it.