{"id":25442,"date":"2022-03-13T10:44:14","date_gmt":"2022-03-13T17:44:14","guid":{"rendered":"https:\/\/www.podfeet.com\/blog\/?p=25442"},"modified":"2022-03-13T11:53:29","modified_gmt":"2022-03-13T18:53:29","slug":"podfeet-wicked-fast","status":"publish","type":"post","link":"https:\/\/www.podfeet.com\/blog\/2022\/03\/podfeet-wicked-fast\/","title":{"rendered":"Podfeet is Wicked Fast Now!"},"content":{"rendered":"
\"New
New vs Old Web Server CPU Load<\/figcaption><\/figure>\n

I have something VERY exciting to tell you about, and instead of telling a 2500-word story first and saving the punch line, I\u2019m going to tell you the news first<\/em>. I\u2019m still going to tell the 2000-word story but you just won\u2019t have to wait for the dramatic reveal. Here\u2019s the announcement – podfeet.com is WICKED FAST<\/strong>. We have finally solved the problems that were plaguing it, and it\u2019s now faster than I ever thought possible. My hope was that we\u2019d get it down to maybe 2-second page loads, but it\u2019s way way faster than that now. Ok, with the fun part out of the way, how did we fix it? We\u2019ll start with a tiny bit of review.<\/p>\n

A few weeks ago<\/a> I talked about how I was working hard to try to figure out why podfeet was so slow. Podfeet hasn\u2019t been the snappiest website for a long time, but it had really hit a wall in performance, making it nearly unusable. I threw money at the problem first by doubling the number of virtual CPUs, and while that did get the server out of its temporary jam, very quickly the percentage CPU usage started to climb again. We were out of 40-second page load times, but we were still seeing hours at a time where the CPU was just under 100% usage.<\/p>\n

Cloudflare<\/h4>\n

Bart\u2019s next idea, as I explained last time, was to move the database to a dedicated server, and that\u2019s when our hero, William (Bill) Reveal joined the party to figure out how to fix that pesky encoding problem. As I reported, things got faster after we successfully moved the database, but still, we had those pesky 98% high load periods for extended periods of time.<\/p>\n

Bart\u2019s next idea was to move podfeet.com under the protection of Cloudflare<\/a>. They have a free service where they manage the DNS records (those are the records that allow you to type in a URL and it gets converted to the correct IP address.) Cloudflare specialize in protecting those domains under their wing against denial of service attacks. We didn\u2019t think my server was under a DDoS because the traffic numbers didn\u2019t account for that, but we knew it wouldn\u2019t hurt. It didn\u2019t hurt, but it also didn\u2019t help.<\/p>\n

Moving podfeet.com to a VPC<\/h4>\n

At this point, my web server and my database were on separate virtual machines, and both were hosted by DigitalOcean. But Bill noted that the traffic between them was on the open Internet which isn\u2019t recommended from a security standpoint if you can help it.<\/p>\n

In the DigitalOcean service, if you create a new Droplet today (their name for a virtual machine), they put it in what they call a Virtual Private Cloud, or VPC. VPC is a logically isolated network for cloud resources. The database server was created in their VPC, but the original web server was built long enough ago that it was not. Bill suggested that the traffic would be more protected and hopefully the site would be served more quickly if both servers were on the VPC.<\/p>\n

Luckily, it\u2019s very easy to move a droplet into the VPC with DigitalOcean. You simply press a button to create a snapshot of your server, turn off the original server, and turn on the snapshot in the VPC. I bet you\u2019ll be shocked to learn that this method did not work for us. The problem was something called cPanel.<\/p>\n

I’m what my friend Linda would call \u201ca reasonably bright girl\u201d, but I’m definitely still learning when it comes to managing my servers myself. Bart has been slowly but surely teaching me, by having me do all the typing and configuring while he directs. But to be honest, a great deal of it often flies over my head. One of those fly-over-my-head parts was a few years ago when he suggested I use something called cPanel to help me manage my site.<\/p>\n

Until two weeks ago, I thought cPanel was just a nifty interface to get to things like phpMyAdmin to play with my database, add certificates, and update things. Evidently cPanel is much much more, and it was actually part of the structural fabric of my web server. I know that now, because the existence of cPanel in my environment meant it was not possible to put the snapshot of podfeet.com into the VPC.<\/p>\n

Bill and I did a lot of searches about how to extricate cPanel from the server and they all said, \u201cDon\u2019t even try, you must burn it to the ground and start over.\u201d So that\u2019s what we did.<\/p>\n

PHP-CGI vs PHP-FPM<\/h4>\n

Bill was very much in favor of this move for a more fundamental reason. I’ve mentioned before that web servers are often on what\u2019s called the LAMP (or WAMP) stack. The acronym stands for Linux Apache MySQL and PHP. That\u2019s the operating system, the web server software, the database, and the programming language. When Bill and I were done, my web server was no longer on the LAMP stack. I think the changes and debates along the way were really interesting.<\/p>\n

As I said, PHP is the programming language in LAMP, but it turns out there are two versions of PHP, which I didn\u2019t know before. I promise I won\u2019t get too deep into this part (mostly because my understanding is fairly limited) but it may actually be the most significant part of the story.<\/p>\n

PHP is a way to run a server-side script when an HTTP request comes in. The two versions of PHP are PHP-CGI and PHP-FPM. PHP-FPM is the \u201cfast\u201d version of PHP-CGI. As Bill explained it to me, PHP-CGI is better for low-traffic sites but PHP-FPM (FastCGI Process Manager) is significantly more efficient for high-traffic sites. (Learn more at basezap.com\/…<\/a>)<\/p>\n

I had to ask Bill, \u201cIs my site high or low traffic?\u201d I know podfeet.com isn\u2019t the New York Times, but it\u2019s also not Sally\u2019s Left-Handed Wingnut Supply Store either. He did some research into my traffic and declared podfeet.com officially \u201cmedium\u201d. He said I should be congratulated. I was so proud.<\/p>\n

Bill said that if we\u2019re going to rebuild podfeet.com from scratch, we should do it using PHP-FPM. He based his theory on the fact that the only thing that seemed to be churning on my server when these CPU bottlenecks were happening was PHP-CGI. PHP-FPM it is.<\/p>\n

Building podfeet.com From Bare Metal<\/h4>\n

I didn\u2019t really appreciate what we were about to undertake in building podfeet.com from bare metal. While it was a ton of work, it was fascinating and I learned so much from Bill. Like Bart, Bill directed while I did all of the typing. I’m not even going to pretend that I understood everything we did, and I definitely couldn\u2019t replicate 90% of what we did, but I got a better appreciation for what it means to build a server from scratch.<\/p>\n

The first thing you have to do is decide on an operating system. This first decision created an interesting discussion between Bart, Bill, and me. Bart likes CentOS and it\u2019s what he manages with his university work hat on, but if it was all up to Bill he would have gone with Ubuntu. CentOS is more of an enterprise-level OS with lots of security features built in, but being an enterprise OS means it moves more slowly so it\u2019s actually using a fairly old version of PHP. Bart, Bill and I had to discuss which way to go.<\/p>\n

My worry at this point was that I would be caught between the two people doing so much work to help me. Nerds can get into holy wars but that wasn\u2019t going to happen with Bill and Bart. Because they\u2019re both fine, intelligent people with good hearts, we quickly decided to go with Bart\u2019s choice of CentOS because we know he\u2019ll be working with me for the long haul. I suspect Bill will be too, but we had to flip a coin and the toss went to CentOS.<\/p>\n

The second decision was which web server software to go with. I said that the A in LAMP was for Apache, but Apache isn\u2019t the only option. Bart prefers something called NGINX. I remember a hundred years ago he stood up a tiny IRC chat server for me and it used NGINX, but I didn\u2019t really know what that meant back then. Bart sees NGINX as much easier to manage than Apache, but of course Bill was much more comfortable with Apache. The coin toss went to Bart again, and we went with NGINX.<\/p>\n

Remember I said my webserver isn\u2019t on the LAMP stack anymore? The M is gone altogether because MySQL isn\u2019t even running on the web server, as the database is on a separate, dedicated server. That would leave me with LAP, but the A is gone too because I\u2019m not using Apache as the web server software, I\u2019m using NGINX. That means I have an LNP server. Doesn\u2019t quite have the same ring to it, does it? Can\u2019t even pronounce it.<\/p>\n

Another thing I didn\u2019t appreciate before we went into this exercise is what doesn\u2019t exist when you stand up a virtual machine with JUST an operating system. There was nothing there. I mean not even a user account for me! We even had to install tools in order to install tools.<\/p>\n

The good news for you is that I can\u2019t remember everything we did and probably couldn\u2019t explain a lot of it. Once we had NGINX functioning as a web server, it was actually a pretty easy task to copy over all of the WordPress files to the new server and point the Cloudflare DNS at the new server.<\/p>\n

There was another piece to figure out, and it\u2019s not completely working just yet. You know how I tell you to go to podfeet.com\/patreon<\/a> to help support the show, but if you go there it actually takes you to patreon.com? That works because of what\u2019s called a redirect, and in Apache, it\u2019s stored in a text file called .htaccess.<\/p>\n

Donald Burr actually helped me build my .htaccess file years ago, and every time I think of something new for which I want to give you an easy link, I just copy a line Donald wrote and change it for the new redirect. I love these redirects because if at some point there\u2019s a better way to support the show than Patreon, I can keep podfeet.com\/patreon and redirect it to the new service. I also use these for internal links, like podfeet.com\/live actually goes to https:\/\/www.podfeet.com\/blog\/nosillacast-live\/<\/a> which is way too hard to remember.<\/p>\n

But NGINX doesn\u2019t use .htaccess, it uses a .conf file inside the NGINX directory. The syntax is completely<\/em> different, and I don\u2019t understand it very well at all. It was unfamiliar to Bill as well, and we got most of it working but even with his help, we managed to make an infinite loop of redirects within podfeet.com at one point. Since without Patreon and PayPal I don\u2019t get paid, I wasn\u2019t willing to move the server over until we had at least<\/em> those two links functioning.<\/p>\n

Eventually, we got most of the redirects working so we flipped the switch to move to the new, built-from-scratch server.<\/p>\n

How Fast Is It?<\/h4>\n
\"Podfeet
.25 Second Page Loads!<\/figcaption><\/figure>\n

I told you it\u2019s fast now, and I even said wicked fast. I can prove that with real numbers. When all this mess started, I installed a plugin for WordPress called Query Monitor<\/a>. We installed it because it would report on errors and warnings that we hoped would point at a root cause. It didn\u2019t find anything wrong which was disappointing but actually good news. But Query Monitor also has a little display that shows when you\u2019re logged into WordPress to tell you how fast the pages are loading.<\/p>\n

After I doubled the number of CPUs, the page loads went from as bad as 40 seconds down to 6 seconds. On occasion when podfeet.com was in a good mood, it was down to 2 seconds to paint a page. I know that sounds great, but 2 seconds is actually really annoying.<\/p>\n

After Bill and I built the webserver from scratch running NGINX and PHP-FPM with a dedicated database server and using Cloudflare, I am happy to report that most pages come up in a quarter of a second! Seriously, a 10X improvement in speed.<\/p>\n

I find it interesting that we still don’t actually know why the original server was churning the CPUs so hard. We know the database move didn’t fix it and neither did going behind Cloudflare. We’re pretty sure nothing was attacking it from the outside, but something was clearly going on inside the machine. Was it PHP-CGI? Was it Apache? Was it nearly 17 years of cruft and just time to do a nuke and pave? We’ll never know. The one thing we do know is the speed of podfeet.com makes me deliriously happy.<\/p>\n

It made me happy when I created my diagram about the different M1 options and I knew if it got popular people would get it quickly (which helps it get more popular). It also makes both Steve and me positively joyous because we\u2019re the ones who go into the WordPress interface all the time which took FOREVER until now. I love it so much that I just keep going to page after page on my website to watch it blink and load. Of all the comments about how fast the site is now, I think my latest favorite is what Allister Jenks said in Slack: “It’s disconcertingly<\/em> fast.”<\/p>\n

Bill is My Hero<\/h4>\n

Another super fun side effect of this work was that Bill slowly and gently improved my skills using the vi<\/code> editor. He didn\u2019t make me use it but I said I wanted to get better at it. By the end of two weeks of working together, he started noticing that I was getting faster at anticipating what he was going to tell me to do and becoming more natural in using the tools of managing a server.<\/p>\n

I simply can\u2019t thank Bill enough for all of his help. I said he did a lot before, but what he did in the last couple of weeks and the time he spent was beyond generous. He was fun, he taught me many things, we laughed a lot (mostly geek jokes) and we were successful. I told him that Steve and I felt we should pay him for this time, but he didn\u2019t agree. He said that he\u2019s been listening to the NosillaCast for nearly 15 years and that he\u2019s never contributed anything. He was happy that he was finally able to contribute in a meaningful way.<\/p>\n

Bill is my hero.<\/p>\n

Please go to podfeet.com and push all the links and rejoice with us at how fast it is!<\/p>\n","protected":false},"excerpt":{"rendered":"

New vs Old Web Server CPU Load I have something VERY exciting to tell you about, and instead of telling a 2500-word story first and saving the punch line, I\u2019m going to tell you the news first. I\u2019m still going to tell the 2000-word story but you just won\u2019t have to wait for the dramatic […]<\/p>\n","protected":false},"author":1,"featured_media":25448,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[147],"tags":[707,1047,2463,5081,5102,201,5101,607,2041,706,3627],"class_list":["post-25442","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog-posts","tag-apache","tag-centos","tag-cloudflare","tag-digital-ocean","tag-droplet","tag-linux","tag-nginx","tag-server","tag-ubuntu","tag-web-server","tag-webserver"],"jetpack_featured_media_url":"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2022\/03\/New-vs-Old-Web-Server-CPU-Load.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts\/25442","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/comments?post=25442"}],"version-history":[{"count":8,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts\/25442\/revisions"}],"predecessor-version":[{"id":25461,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts\/25442\/revisions\/25461"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/media\/25448"}],"wp:attachment":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/media?parent=25442"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/categories?post=25442"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/tags?post=25442"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}