{"id":34661,"date":"2025-10-11T14:27:13","date_gmt":"2025-10-11T21:27:13","guid":{"rendered":"https:\/\/www.podfeet.com\/blog\/?p=34661"},"modified":"2025-10-11T19:39:55","modified_gmt":"2025-10-12T02:39:55","slug":"rate-limiting-bots","status":"publish","type":"post","link":"https:\/\/www.podfeet.com\/blog\/2025\/10\/rate-limiting-bots\/","title":{"rendered":"Demon Bots, I Rebuke Thee!"},"content":{"rendered":"<p>Podfeet.com is on a virtual server hosted by DigitalOcean. Bart and I have both been enjoying their services for quite a few years. A few years back, Bart convinced me to rent a second server to host the WordPress database. This setup has been working great.<\/p>\n<p>But then a few months ago, I noticed in the DigitalOcean interface that my server&#8217;s CPU was getting positively <em>hammered<\/em> from time to time. And by hammered, I mean 100% of the CPU was being used for hours at a time.<\/p>\n<p>Bart is super busy these days, so rather than bothering him, I spent some quality time with ChatGPT trying to figure out what was wrong. ChatGPT had me look at various log files and came to some conclusions, but it was during this discussion that I discovered my server was on a deprecated version of Linux.<\/p>\n<p>That&#8217;s when the awesome Tage Bushman stepped in and helped me perform <a href=\"https:\/\/www.podfeet.com\/blog\/2025\/09\/server-migration-2025\/\">The Great Podfeet Server Migration of 2025<\/a>. We debated whether to copy everything over as-is or do a fresh install of WordPress, just in case that might stop the server slamming problem.  In the end, we copied everything over.<\/p>\n<p>Over the last month or so, I kept an eye on the DigitalOcean CPU graph, and the web server wasn&#8217;t getting slammed.  But then I got an email from Marc Loehrwald telling me that podfeet.com was down.  Sure enough, it was, and when I looked at the graphs on DigitalOcean, it had been down for a day and a half! I can&#8217;t believe nobody else told me sooner.<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/graph-from-digital-ocean-showing-server-down-for-a-day-and-a-half.png\" alt=\"Graph from digital ocean showing server down for a day and a half.\"  title=\"graph from digital ocean showing server down for a day and a half.png\" width=\"599 \" height=\"245\"><figcaption style=\"text-align:center\">Only Marc Told Me it was Down!<\/figcaption><\/figure>\n<p>I launched Core Shell and secure shelled into my server, and restarted the nginx webserver, and it came back up. While I fixed the symptom, I didn&#8217;t know why the server went down. A bit after that catastrophic event, I realized that my new server wasn&#8217;t able to automatically keep my WordPress plugins up to date. When I tried to do it manually, I was met with an error telling me I needed to enter my Secure FTP credentials to proceed.<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/WordPress-requesting-FTP-credentials.png\" alt=\"WordPress requesting FTP credentials.\"  title=\"WordPress requesting FTP credentials.png\" width=\"480 \" height=\"559\"><figcaption style=\"text-align:center\">Why does WordPress Need These Credentials?<\/figcaption><\/figure>\n<p>While figuring out why my website went down and why my site keeps getting hammered are pretty important, out-of-date WordPress plugins are possibly the most dangerous thing you can do on the Internet.<\/p>\n<p>I figured these problems might be interesting enough to tickle Tage into another playdate.  Remember when he said he was sad we didn&#8217;t have an excuse to have more playdates? He&#8217;ll rue the day&#8230;<\/p>\n<h2>Tage Saves the Day<\/h2>\n<p>The day Tage and I sat down to tackle these problems, to our great luck, my server just happened to be getting slammed. I say great luck, because I had not yet observed it happening even once since we&#8217;d done the migration to Rocky Linux. This gave us the opportunity to try to fix it, and if successful, we&#8217;d get instant gratification.<\/p>\n<p>Think about it like you take your car into the shop, but when you bring it in, it&#8217;s no longer making that funny noise. The mechanic could try a bunch of stuff, but you&#8217;d never know if it fixed anything. Ideally it makes the horrific noise while you\u2019re there.<\/p>\n<p>We started with two diagnostic steps that we\u2019d used before, and which ChatGPT had suggested.  The first was to use the <code>top<\/code> command, which shows you what processes are using the most CPU. It\u2019s essentially what you see when you use Activity Monitor on the Mac to find processes that are out of control. You can run <code>top<\/code> in the Terminal on a Mac if you\u2019d like to see it for yourself.<\/p>\n<p>During this latest playdate, the <code>top<\/code> command showed us what it had shown us every time, that about a dozen or more processes called <code>php-fpm<\/code> were all using maximum CPU percentages. I don\u2019t entirely understand what <code>php-fpm<\/code> is, but ChatGPT says:<\/p>\n<blockquote><p>\n  PHP-FPM stands for <strong>PHP FastCGI Process Manager<\/strong>. It is a server component that manages pools of PHP worker processes to handle web requests efficiently.\n<\/p><\/blockquote>\n<p>The most information I get out of that is that this has something to do with WordPress.<\/p>\n<p>Another diagnostic step that ChatGPT suggested, and which Tage and I had tried before, was to look at the access log to see what all this processor load was all about.  The access log for my server is called <code>access.podfeet.log<\/code>. The way you look at a log file in real-time is to use the command <code>tail<\/code> with the flag <code>-f<\/code>, which shows you the latest entries as they\u2019re being written to the log file. Pretty simple command, actually.<\/p>\n<p><code>tail -f \/var\/log\/nginx\/access.podfeet.log<\/code><\/p>\n<p>What\u2019s not simple is <em>watching<\/em> this log file being written to real time. As you watch the log file flying by with each line of the file wordwrapping 2-3 times so the screen, it is just covered with gibberish. Somehow, with a super-human skill I\u2019ve never witnessed before, Tage will say, \u201cOh I see the problem&#8230;\u201d  I could discern literally nothing looking at the screen as all this slop flew by.  I don\u2019t have a screenshot of it when this was happening, or better yet a video, but I did capture it when it was a bit calmer after Tage saved the day and fixed the problem.<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/Rocky-showing-results-of-tail-f-gibberish.png\" alt=\"Rocky showing results of tail -f gibberish.\"  title=\"Rocky showing results of tail -f gibberish.png\" width=\"599 \" height=\"530\"><figcaption style=\"text-align:center\">Tailing Log File Looks Like Gibberish to Me<\/figcaption><\/figure>\n<p>Just like the last time we looked at the access log, Tage\u2019s super skills spotted <code>xmlrpc.php<\/code> flying by over and over again. Again I can\u2019t really explain what XML-RPC really is, but I know that MarsEdit requires it for me to be able to push blog posts to Podfeet. ChatGPT\u2019s definition is:<\/p>\n<blockquote><p>\n  XML-RPC is a <strong>remote procedure call protocol<\/strong> that uses <strong>XML to encode data<\/strong> and <strong>HTTP as the transport<\/strong>.\n<\/p><\/blockquote>\n<p>Like I said, I don\u2019t understand it, but I know I need it. It was obvious, though, that this was a hot target for bots to try to break into my server. The last time we tried to solve this, we tried having MarsEdit write xmlrpc.php to a different location on my server, hoping that by obscuring it from the known location for WordPress would confuse the bots. It was unsuccessful.<\/p>\n<p>The other thing that Tage spotted in tailing the access log, which we hadn\u2019t noticed the last time, was that bots were hammering at <code>wp-login.php<\/code>. I didn\u2019t have to ask ChatGPT what that one meant. Clearly, this is a very bad thing.<\/p>\n<h2>Cloudflare for the Win<\/h2>\n<p>At this point in the story, Tage had a genius idea.  I\u2019ve mentioned before that I use a service called Cloudflare to manage my DNS entries for Podfeet.com. Cloudflare\u2019s offering is based on the philosophy of increasing the security and performance of websites. You can use their DNS servers with your router and be pretty sure you\u2019re getting the fastest and most secure connection to the web. But you can also have them do more if you manage any servers.<\/p>\n<p>They have a lot of paid features, but they give a fair bit away for free. Tage thought that maybe I could use Cloudflare to block access to the two PHP services that were getting hammered on my server.<\/p>\n<p>In the Cloudflare interface, under Security in the left sidebar, you\u2019ll find a section for security rules. Not to get Tage\u2019s head all swole up with all these compliments, but somehow, never having seen this interface before, he knew what to choose in dropdowns and what to type in each field. I might have figured it out after spending quality time with Cloudflare\u2019s excellent documentation, but he seemed to know it intuitively.<\/p>\n<p>Under security rules, there was a dropdown to show all rule types, and the option he told me to select was called rate limiting rules. From there, we clicked the button to create a rule.<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/setting-up-rate-limiting-rules-on-Cloudflare.png\" alt=\"Setting up rate limiting rules on Cloudflare.\"  title=\"setting up rate limiting rules on Cloudflare.png\" width=\"599 \" height=\"364\"><figcaption style=\"text-align:center\">Creating a New Rate-Limiting Rule<\/figcaption><\/figure>\n<p>Cloudflare asks you to name the rule. Again with Tage guiding my every move, we chose from the Field dropdown, \u201cURI Path\u201d, and then for Operator chose \u201ccontains\u201d, and finally for Value I typed in xmlrpc.php.  In retrospect, that makes sense. The XML-RPC file they were attacking was defined by a URL of its path.<\/p>\n<p>We ran into a limitation on paid vs. free on Cloudflare when we tried to apply a rate limit. The section was entitled \u201cWhen rate exceeds\u2026\u201d  They have a dropdown for the number of requests, and then how long to block the requests. I could enter a pretty big number for requests, but I could only choose a period of 10 seconds. That wasn\u2019t a big limitation since the bots were hitting that file at least a few times a second, so once every 10 seconds would catch all of it, and would be way more often than I would hit XML-RPC with MarsEdit.<\/p>\n<p>We ran into the same limit when we got to the \u201cThen take action\u2026\u201d section. We chose Block from the dropdown, but we could only block the requests for 10 seconds.<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/Cloudflare-rate-limiting-as-described-in-the-text.png\" alt=\"Cloudflare rate limiting as described in the text.\"  title=\"Cloudflare rate limiting as described in the text.png\" width=\"600 \" height=\"\"><figcaption style=\"text-align:center\">Settings for My Rate-Limiting Rule on XML-RPC<\/figcaption><\/figure>\n<p>We were hopeful, but we also took a look at the options for paid accounts just in case this didn\u2019t work. If I have to pay for this service, you\u2019d get a <em>much<\/em> harder press for the panhandling section of the NosillaCast.<\/p>\n<p>The rate-limiting rule section has the option to add more rules. We chose \u201cor\u201d to add the URI path to wp-login.php as well and saved the rule.<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/rate-limiting-xmlrpc.php-wp-login.php_.png\" alt=\"Rate limiting xmlrpc.php AND wp-login.php.\"  title=\"rate limiting xmlrpc.php &#038; wp-login.php.png\" width=\"600 \" height=\"543\"><figcaption style=\"text-align:center\">Rate Limiting Both xmlrpc and wp-login<\/figcaption><\/figure>\n<p>We held our collective breaths as I frantically and unnecessarily refreshed the DigitalOcean graphs for my CPU (unnecessarily because it auto-refreshes). After a little bit of time, we saw the most glorious thing &#8211; the CPU and load graphs falling off a cliff. CPU went from 100% to 25% or so right at the timestamp when we asked Cloudflare to rate limit those two files.<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/dropped-cpu-usage-6-hours-framed.png\" alt=\"6 hour graph showing CPU dropping 100% to 25% and load dropping to almost nothing\"  title=\"dropped cpu usage 6 hours framed.png\" width=\"600 \" height=\"419\"><figcaption style=\"text-align:center\">Isn&#8217;t That the Most Beautiful Graph?<\/figcaption><\/figure>\n<p>I\u2019m jumping ahead a little bit, but when we finished this call, I thanked Tage profusely, and he said, \u201cThis was all worth it to see that graph.\u201d<\/p>\n<p>Back on the security rules page, we can see my fancy new rate-limiting rule, but it also shows the number of Events blocked in the last 24 hours. As of the time of my writing this up, which was a few days after Tage and I finished our play date, it shows only 117 events blocked with a tiny little graph.<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/Cloudflare-graph-of-Events-blocked-by-rate-limiting-in-the-last-24-hours.png\" alt=\"Cloudflare graph of Events blocked by rate limiting in the last 24 hours.\"  title=\"Cloudflare graph of Events blocked by rate limiting in the last 24 hours.png\" width=\"596 \" height=\"116\"><figcaption style=\"text-align:center\">Tiny Graph Showing Blocked IP Addresses<\/figcaption><\/figure>\n<p>If I select the number of events, I can see the country of origin and the IP address for each event. I suspect that each IP address is sending zillions of requests each. Right after we first set up the rate-limiting rules, we saw 94 requests almost immediately, and then the graph dropped to zero. The vast majority (all but a half dozen) were coming from Singapore. Now that it\u2019s been a few days, the bots seem to be coming from many different countries, including Singapore. Maybe most of the Singapore bots got offended by my rate limiting them and moved on.<\/p>\n<p>While this is most definitely a whack-a-mole effort, after months and months of this, I can\u2019t tell you how great it feels to have this server hammering slowed way down.<\/p>\n<h2>WordPress Plugin Updates &#8211; SELinux is to Blame<\/h2>\n<p>Tage had one more action item on my list, and that was to figure out why WordPress couldn\u2019t run auto-updates to my plugins. Even I couldn\u2019t push the button in the WordPress interface to update them without being hit by a request to supply my FTP credentials.<\/p>\n<p>Again, this was something I couldn\u2019t have solved on my own, and ChatGPT wasn\u2019t even any help. Tage guessed that Security-Enhanced Linux (aka SELinux) was not giving permission to read and write to my WordPress directory. For the full-scale nerds, I dropped the command into the shownotes that fixed the problem.<\/p>\n<pre><code># semanage fcontext -a -t httpd_sys_rw_content_t \"\/srv\/websites\/podfeet\/public_html\/blog(\/.*)?\"\n# restorecon -Rv \/srv\/websites\/podfeet\/public_html\/blog\/\n<\/code><\/pre>\n<p>In future, I plan on always blaming SELinux when things go wrong. Gained a few pounds over the weekend? Probably SELinux. Can\u2019t find my glasses? I bet SELinux moved them. Goofed around too much on TikTok? SELinux made me do it.<\/p>\n<p>As soon as we beat SELinux into submission, I was able to update my WordPress plugins to keep my site safe again.<\/p>\n<p>One More Thing<\/p>\n<p>After I finished writing up this article, I pushed the Send to Blog button in MarsEdit. I had a lot of screenshots attached, but it seemed to take longer than expected. After about a minute, an error popped up saying it couldn\u2019t upload the file. Oh no\u2026 did all of our great work to block bots from attacking XML-RPC block MarsEdit?<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/MarsEdit-cannot-upload-the-file.png\" alt=\"MarsEdit error saying it cannot upload the file.\"  title=\"MarsEdit cannot upload the file.png\" width=\"296 \" height=\"312\"><figcaption style=\"text-align:center\">What Have I Done?<\/figcaption><\/figure>\n<p>I popped back over to Cloudflare, clicked on the little graph showing a big spike in blocks, and opened the first in the list of IP addresses. Sure enough, it was my IP address. It even helpfully told me that the user agent was MarsEdit.<\/p>\n<figure style=\"float: center; margin: 10px\"><img decoding=\"async\" src=\"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/Cloudflare-showing-MarsEdit-getting-blocked-framed.png\" alt=\"Cloudflare showing MarsEdit getting blocked.\"  title=\"Cloudflare showing MarsEdit getting blocked framed.png\" width=\"599 \" height=\"493\"><figcaption style=\"text-align:center\">Confirmed MarsEdit is Blocked by Cloudflare<\/figcaption><\/figure>\n<p>I went into the rules and changed it to 10 requests every 10 seconds, which might be more than I need, AND might start letting bots attack again, but I\u2019m confident I can tweak that number until I can legitimately post to the blog from MarsEdit without any bots tailgating in behind me.<\/p>\n<h2>Bottom Line<\/h2>\n<p>The bottom line is that I know how incredibly lucky I am to be surrounded by a helpful community of nerds to help me when I don\u2019t know how to do things. Tage has proven himself to be a peach of a human being whom I\u2019m happy to call my friend. He never makes me feel stupid or even unknowledgeable when he pulls off these feats of genius, and he seems to genuinely enjoy helping me.<\/p>\n<p>Here\u2019s each of my fingers and toes crossed that we\u2019ve finally banished these demon bots for good.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Podfeet.com is on a virtual server hosted by DigitalOcean. Bart and I have both been enjoying their services for quite a few years. A few years back, Bart convinced me to rent a second server to host the WordPress database. This setup has been working great. But then a few months ago, I noticed in [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":34666,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[147],"tags":[2463,1117,7618,7589,7619,607,7620,7577,3627,216],"class_list":["post-34661","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog-posts","tag-cloudflare","tag-dns","tag-rate-limiting-bots","tag-rocky-linux","tag-selinux","tag-server","tag-tage","tag-tage-bushman","tag-webserver","tag-wordpress"],"jetpack_featured_media_url":"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/10\/dropped-cpu-usage-6-hours-framed-1040x520-1.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts\/34661","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/comments?post=34661"}],"version-history":[{"count":4,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts\/34661\/revisions"}],"predecessor-version":[{"id":34668,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts\/34661\/revisions\/34668"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/media\/34666"}],"wp:attachment":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/media?parent=34661"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/categories?post=34661"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/tags?post=34661"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}