{"id":34517,"date":"2025-09-14T16:22:17","date_gmt":"2025-09-14T23:22:17","guid":{"rendered":"https:\/\/www.podfeet.com\/blog\/?p=34517"},"modified":"2025-09-14T17:37:14","modified_gmt":"2025-09-15T00:37:14","slug":"server-migration-2025","status":"publish","type":"post","link":"https:\/\/www.podfeet.com\/blog\/2025\/09\/server-migration-2025\/","title":{"rendered":"The Great Podfeet Server Migration of 2025"},"content":{"rendered":"

History<\/h2>\n

I start a lot of my articles with the phrase, \u201cYou may remember\u2026\u201d and this is another one of those times. You may remember back in 2022, when the delightful Bill Reveal helped me rebuild podfeet.com<\/a> from scratch with a different operating system, new web server software, and more. My server was getting hammered, and it was time for a big update. My victory dance post showed how fast the new server was and how, by moving the database to a dedicated server and increasing the capacity of my web server, all was good.<\/p>\n

Well, those were the good old days. A few months back, I was perusing the performance graphs for my web server at my hosting company, DigitalOcean. What I saw concerned me \u2014 periodic episodes lasting an hour or more where my CPU was hitting 100%. I asked around a bit to see if anyone wanted to help me poke into this and got a few suggestions, but nothing particularly actionable.<\/p>\n

Ask AI<\/h2>\n

I hated to lean on Bill again, and Bart is crazy busy these days, so I thought I\u2019d try using AI to help me diagnose what could be happening. I used ChatGPT for this effort, and it was pretty fun. I know enough of what I’m doing to be able to mostly tell when I’m changing something vs. just looking at something, so even knowing how often it returns correct answers, I was pretty confident I could keep from damaging anything.<\/p>\n

For example, the command cat<\/code> is what you use when you want to look at the contents of a file, but vi<\/code> or nano<\/code> would mean going in to actually edit a file. The command tail<\/code> means to watch a log file that’s changing in real-time. Nothing destructive there.<\/p>\n

For the most part, I felt that ChatGPT was very helpful. We looked at a lot of logs, which suggested many possibilities that we systematically eliminated, and I found the experience pretty instructive.<\/p>\n

\"Example
Example from My Conversation with ChatGPT<\/figcaption><\/figure>\n

That is, until at one point when it suggested I create a tiny php file and drop it into some top-level directory. I don\u2019t know how to read or write php, so I had no business following these instructions. But it looked like a little \u201cHello World\u201d kind of thing, so I thought, what harm could it do?<\/p>\n

I tried to open podfeet.com in a browser, and instead got redirected to beds.com! I raced back to ChatGPT and told it what happened, and my little friend rang the 3-alarm fire bell. It yelled \u201cAWOOGA!\u201d and told me I\u2019d been hacked, my whole system was ruined, that I should shut the server down, burn all of my documentation, and maybe join a convent.<\/p>\n

I asked, \u201cCould it maybe be that little file you had me put up there?\u201d And ChatGPT said, \u201cOh, yeah, that could be it. Maybe try deleting that file.\u201d And deleting the file fixed the problem.<\/p>\n

After that, I was a lot<\/em> more cautious when ChatGPT would tell me to do things. Whatever it told me to do, I\u2019d ask it if that was safe or what could go wrong. With that prompt, it often came back with something like, \u201cYeah, that could mess things up. Want to try a less dangerous path?\u201d Sheesh.<\/p>\n

After many, many hours, we ran into a curious problem. ChatGPT asked me to run some commands that I knew were harmless, and yet I couldn’t run them.<\/p>\n

We first came across it when ChatGPT suggested I run the command strace<\/code> to monitor some log files. My server responded with “command not found”. No worries, we’ll just install it. ChatGPT suggested we use the yum<\/code> package manager to install strace<\/code>. Think of a package manager like an app store.<\/p>\n

But when I tried that, I got a surprising result. It turned:
\nCentOS Stream 8 - AppStream<\/code>
\nErrors during downloading metadata for repository 'appstream':<\/code>
\nand
\nCould not resolve host: mirrorlist.centos.org<\/code><\/p>\n

My operating system was CentOS Stream 8, so why couldn\u2019t it see the repository at centos.org? It turns out, unbeknownst to me, that CentOS of all versions had been deprecated more than a year! When they fully deprecated it, they shut down access to all<\/em> of the package managers. There was no way I could install any new commands.<\/p>\n

\"CentOS
CentOS Has Been Dead Since May 2024!<\/figcaption><\/figure>\n

More importantly, I needed to get off this operating system.<\/p>\n

Migrating off CentOS<\/h2>\n

I put out the call to the Programming By Stealth channel in our Slack and didn\u2019t get much of a reaction. I think it\u2019s like when the teacher starts to scan the classroom to look for someone to call on, and everyone ducks down and pretends to be doing something else really important.<\/p>\n

Eventually, Bill Reveal said something along the lines of, \u201cI can help if need be if none of the young bucks step up.\u201d<\/p>\n

Enter Tage Bushman<\/h2>\n

Not too long after that, I interviewed Tage Bushman for Chit Chat Across the Pond<\/a> to learn about his work as a professor at a technical college. After we finished recording, we talked for a long time about all kinds of things. I\u2019m not sure how I worked it into the conversation, but I ended up telling him about my server woes and how I needed to migrate to a supported version of Linux.<\/p>\n

He got very excited and said he\u2019d love to help me. He teaches server admin and networking, and he sounded like he knew what he was doing, so I gladly said yes. We set a play date for a few weeks in the future to start working on the migration.<\/p>\n

As I awaited our scheduled play date, and I was out of the glow of the fun of talking to Tage, I started thinking, \u201cWhat do I really know about this guy?\u201d He\u2019s literally<\/em> some guy I met on the Internet who sounded smart and knowledgeable. What did I really know about this guy? Would he even show up? Does he actually know what he\u2019s talking about?<\/p>\n

Now here\u2019s the funny thing. At the same time that I was doubting the wisdom of going ahead with him, Tage was having his own doubts. This will sound funny as you hear more of the story, but he had this huge fear that he would tell me to do something that ended up completely and irreparably borking up my server. He was more worried than I was.<\/p>\n

Luckily, both of our fears were unfounded. A few days before our play date, Tage sent me a huge<\/em> document with not just an outline of a plan, but step-by-step instructions on how to do it!<\/p>\n

\"Page
This is Just Page One of Tage’s Instructions<\/figcaption><\/figure>\n

I knew right then that I was in good hands. He sent me the document as a PDF, but I immediately converted it to a Markdown document so I could do things like put little checkboxes next to each line item as I completed each step.<\/p>\n

One of the reasons Tage was trepidatious, and he was right to be fearful, was that I hadn\u2019t told him hardly anything about my current setup, nor what operating system I was hoping to migrate to. All he knew was that I was running WordPress on CentOS, and I think he knew I used DigitalOcean for my servers.<\/p>\n

His directions were so good that with a little help from the Internet, I was able to complete the entire first two sections of his document all on my own before we got together.<\/p>\n

DigitalOcean Droplets & Database Servers<\/h2>\n

One of the magical things about technology today is the ability to stand up a virtual machine with literally a click of a button and a credit card. Step one of Tage\u2019s instructions said to create a new droplet, which is what DigitalOcean calls their virtual machines, because they love their little water theme.<\/p>\n

He suggested I use Ubuntu, but since that\u2019s a Debian variant of Linux and I was used to the Red Hat variants, I chose to go with Rocky Linux. To be perfectly honest, I couldn\u2019t tell a Red Hat Linux from a Debian Linux if you threatened to harm Kepler, but I sound all nerdy when I say that, don\u2019t I?<\/p>\n

Luckily, Tage appears to be agnostic when it comes to his \u2018nixes, so he didn\u2019t care which way I went. His instructions to set up the new OS included installing tools using the package manager apt<\/code>, but Rocky Linux uses the package manager dnf<\/code>. It was pretty easy to keep an AI running by my side to do the translations for me between the two package managers. For example, in Debian, Tage asked me to run:<\/p>\n

apt update && apt -y upgrade<\/code><\/p>\n

But with Rocky Linux, the command is:<\/p>\n

dnf check-update && dnf -y upgrade<\/code><\/p>\n

I made the decision to go with Rocky months ago, but it made me particularly happy because I just finished rereading the Andy Weir book, Project Hail Mary<\/a>, in which one of the heroes is named Rocky. If you haven\u2019t read it, drop everything and make that happen. Weir is the author of The Martian, and they\u2019re making Hail Mary into a movie that\u2019s coming out next year!<\/p>\n

Anywho, before getting together with Tage, following his amazing instructions I was able to create the droplet, install the web server nginx, do some configuration stuff, and install some helpful tools that Tage recommended.<\/p>\n

I was very proud of my little self, and Tage appropriately patted me on my little punkin\u2019 head when he saw all of my progress. I stopped at the next section, which was starting to deal with WordPress.<\/p>\n

Which Way WordPress?<\/h2>\n

We had a decision to make here. Since something is hammering my server, and the only information we have is that the top processes are related to WordPress (php-fpm<\/code>), Tage was thinking we should set up WordPress from scratch. Bart, on the other hand, thought just copying the whole WordPress directory over would be fine.<\/p>\n

Remember, Tage had close to no information about how I had set up other folders for documents that show up on my website (for example, my travelogues), and Bart had the luxury of knowing I had more to my site than the root WordPress directory. When Tage and I got on our play date, we went back and forth with the pros and cons.<\/p>\n

In the end, we decided to copy over the WordPress directory and all other files and see how it went, with a contingency plan to build it from scratch at a later date if necessary. For the nerdlings listening, we copied my entire directory from the old droplet to the new one using rsync<\/code> , and it was wicked fast. I think the infrastructure between virtual private cloud services on DigitalOcean is a big part of that, and the rsync<\/code> protocol is super fast too.<\/p>\n

Note Taking<\/h2>\n

Once we were in the thick of it, I created a new Markdown file and took copious notes as we went along. Tage said he\u2019d never seen anyone do such good documentation while learning, and I again felt chuffed at the praise.<\/p>\n

One of the things that made this whole process so joyful was that Tage is a terrific instructor. I know this was his occupation, but not all teachers are good<\/em> teachers. He was great at explaining what we were doing, and telling me what to type (often character-by-character since every space and hyphen means something), and going at a perfect pace for me to keep up. We really worked well together.<\/p>\n

At one point, we wanted to see if my website would render on the new Rocky installation. We hadn\u2019t yet flipped over the DNS to point to Rocky, though. There\u2019s a little file on your Mac called hosts<\/code> , and you can trick it into changing your destination when you type in a URL. You could change podfeet.com to go to lets-talk.ie, if you wanted to by changing the hosts<\/code> file to go to Bart\u2019s server\u2019s IP address instead. But that would be weird. For our first test of the new installation, we pointed podfeet.com to the new IP address for Rocky.<\/p>\n

Unsurprisingly, it didn\u2019t work on the first try. nginx<\/code>, (the web server), returned a permissions error, but at least it was serving up the generic webpage for nginx<\/code>.<\/p>\n

This was probably the first time I saw Tage actually get stuck for a while and not immediately know what to do. He eventually figured out that it was Security-Enhanced Linux enforcing some security policies, and he figured out how to fix it.<\/p>\n

When he finally got the permissions error fixed, we got a new error that said, \u201cError establishing database connection\u201d. That\u2019s the classic error you see when your WordPress and your database aren\u2019t communicating.<\/p>\n

This was possibly my favorite moment of our whole time working together because Tage\u2019s reaction was to throw his arms up in the air to celebrate and say \u201cYay!\u201d \u2026 because it was a different error message and it meant he\u2019d conquered the previous one. That\u2019s a person who loves playing this game.<\/p>\n

\"Tage
Tage Celebrates a New Error<\/figcaption><\/figure>\n

Flipping the Switch<\/h2>\n

One of the scariest (and yet easiest) things to do is to flip the switch on DNS. One wrong move, and when people type in podfeet.com, there will be nothing there. I’ve done it many times but I don’t think I’ll ever have the nerve to do it without someone holding my hand.<\/p>\n

When Tage and I changed what’s called the “A Record” in DNS to switch from the old server I’d named Ghez to the new Rocky server’s IP address, I held my breath. But as you can see in the shownotes, the web traffic dropped to zero on Ghez right around the same time it fired up on Rocky. And nobody knew it had happened but me and Tage.<\/p>\n

\"End
End of Ghez, Beginning of Rocky<\/figcaption><\/figure>\n

Bottom Line<\/h2>\n

The bottom line is that with Tage\u2019s superhuman efforts, Podfeet.com was completely rebuilt and moved between servers \u2026 and you probably didn\u2019t even notice. That, of course, delights me, but what delights me even more is what Tage posted on Mastodon after we were done.<\/p>\n

\n Worked the afternoon helping @podfeet with tech stuff! What an awesome documenter of things. Great experience, would do again. \u2b50\ufe0f\u2b50\ufe0f\u2b50\ufe0f\u2b50\ufe0f\u2b50\ufe0f Also, go check out her podcast at podfeet.com!\n<\/p><\/blockquote>\n

Isn\u2019t he awesome?<\/p>\n

When we realized we were finished, he was actually a little bit sad that we didn\u2019t have any excuse to play any longer. I told him that there\u2019s a good chance that, since we dragged over my entire WordPress installation from the old server, there\u2019s still hope that whatever was ailing it on CentOS came along to the new server. I will probably still end up needing more help.<\/p>\n

Tage looked delighted at the idea as he started musing on what scripts he could write to try to figure out where the problem could be. If he\u2019s good, I may let him do this as his reward.<\/p>\n

I would be remiss if I didn\u2019t give Bart a shoutout at the end here. Just a few weeks before Tage and I got together, Bart moved lets-talk.ie<\/a> to a new server too, and he took copious notes along the way. He gave us several documents ahead of time, and we used the one on how to set up a new Cert and get it to auto-update. The rest of it looked like Klingon to me, so I ignored the other documents.<\/p>\n

However, now that I\u2019ve been through the process with Tage, Bart\u2019s other documents have been very helpful. I don\u2019t understand all of them, but I\u2019ve set up automatic OS updates, I may have successfully set up email sending, and I\u2019ve got the timezone on my server set to Los Angeles.<\/p>\n

Many thanks to Bart and especially to Tage for doing yeoman\u2019s work helping me move my server and for being a delightful human all around.<\/p>\n

One More Thing<\/h2>\n

But that’s not the end of the story. Tage and I forgot to test at least one<\/em> thing. Whether I could write new posts with embedded images on podfeet.com. Two days after we did our victory laps at the success of our adventure, I tried to push this very post up to podfeet.com from MarsEdit … and I got an error saying I couldn’t upload the image!<\/p>\n

I alerted Tage right away, and we started to noodle a couple of things. I won’t drag you through all of the gory details, but there were at least two problems. As I perused my notes from when the glorious Bill Reveal helped me, I found where he had me change a configuration file to allow large file uploads, which matched the advice Tage was giving me. While that definitely had to be done, it didn’t solve the problem.<\/p>\n

I’m not the knowledgeable person here, but I checked file permissions for my uploads folder in the WordPress directory, and it all looked hunky dory to me \u2014 it showed that my user name had read\/write\/execute privileges.<\/p>\n

But writing images to a WordPress directory isn’t actually executed by my user. There’s a tool called php-fpm<\/code> that controls the users, and it was set by default by the author to the user apache<\/code>. Well, I’m not running the Apache web server, I’m running Nginx. We did a few more things to create a group that included my user and nginx<\/code>, and then changed the php-fpm<\/code> config file to match.<\/p>\n

And finally, I was able to bring you this post, exactly 45 minutes before showtime!<\/p>\n

Tage suggested that my initial fears about “some guy on the Internet” helping me were well-founded, but he was there for me to clean things up when I needed him so he’s still a hero in my books.<\/p>\n","protected":false},"excerpt":{"rendered":"

History I start a lot of my articles with the phrase, \u201cYou may remember\u2026\u201d and this is another one of those times. You may remember back in 2022, when the delightful Bill Reveal helped me rebuild podfeet.com from scratch with a different operating system, new web server software, and more. My server was getting hammered, […]<\/p>\n","protected":false},"author":1,"featured_media":34534,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[147],"tags":[1047,4901,7590,201,7589,7591],"class_list":["post-34517","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog-posts","tag-centos","tag-centos-stream","tag-deprecated","tag-linux","tag-rocky-linux","tag-sys-admin"],"jetpack_featured_media_url":"https:\/\/www.podfeet.com\/blog\/wp-content\/uploads\/2025\/09\/tage-celebrates-errror-database-connection-1040x520-1.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts\/34517","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/comments?post=34517"}],"version-history":[{"count":7,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts\/34517\/revisions"}],"predecessor-version":[{"id":34542,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/posts\/34517\/revisions\/34542"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/media\/34534"}],"wp:attachment":[{"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/media?parent=34517"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/categories?post=34517"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.podfeet.com\/blog\/wp-json\/wp\/v2\/tags?post=34517"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}