Nuke and Pave 2021 – All New Enhancements!

If you’ve been following me for years, then you already know that I’m a huge proponent of nuking and paving computers. By nuke and pave, I mean a clean install of the operating system, but the truly nuke version of clean. I try to do this every year “or so” where “or so” can be up to another year while I procrastinate because this is such a daunting task. Every single time I do it I’m really glad I did, but every single time it is definitely difficult to convince myself that it’s worth the trouble.

If you’ve heard me sing this song before, I hope you’ll indulge me in talking about it yet again because I learn new things each time I do a nuke and pave. As always we have to start with the Problem to be Solved™️.

I hear a lot of people talk about how awesome it is that they use Migration Assistant when they get a new machine. They even brag about how seldom they start from scratch because Migration Assistant makes it so easy to carry everything forward for years and years. But I also hear the same people talk about their computer experience being “weird”. They talk about running out of disk space and about their computer being slow. They talk about how much time they spent chasing down strange and annoying bugs.

I experience weird behaviors too, and it seems to reach a fever pitch of annoying about a year after a nuke and pave. It’s rarely a single big bad bug that pushes me over the edge. It’s the million little paper cuts that get to me after a while. Lately, I’ve had truly inconsistent behavior on whether my Apple Watch will unlock my Mac. I’m lucky that I have a Touch ID Mac, but unlock with Apple Watch is faster for me. My fingers are very dry so Touch ID takes 2 or 3 tries and often just gives up. Even when Touch ID is working, the Apple Watch unlocks before I can get to the Touch ID sensor with my finger. When it works it’s glorious, and then it just stops working.

I really wanted Apple Watch unlock to work in the operating system too, because when it is working, then it can also unlock 1Password. It’s truly delightful when it works. And maddening when it doesn’t. Agilebits can’t fix it if it fails because it’s macOS refusing the approval from Apple Watch to 1Password.

A few weeks ago Bart mentioned that he doesn’t use any scanning apps on his iPhone. He simply right-clicks in Finder (or in Notes or lots of other Apple services) and chooses Import from iPhone or iPad ? Scan Documents under his listed phone. This is a feature we got with Continuity a few OSs ago, but I forgot about it. I gave it a try and none of my iDevices were listed. Yes they were on the same WiFi network, yes I had Bluetooth on everywhere, and yes I had Allow Handoff checked in System Preferences ? General. It simply doesn’t work, and I couldn’t find any answers on the Googles as to why.

And remember a while ago I told you about a cool feature when you take screenshots with Apple’s built-in screenshot utility where you can tap a button and annotate the screenshot with your iPad with Pencil? Yeah, that stopped working too.

Then search stopped working in Mail. And Contacts got real weird, where I simply could not enter a new contact because it would save right when I was partway through entering the person’s name. I started running BusyContacts just so I could use my contacts.

I could go on and on with all of the fiddly bits I dealt with day in and day out on my Mac, but I’m sure you get the picture. I had two choices: spend exhausting hours and hours trying to track down the root cause of all these fiddly bits, or burn it to the ground. I’m betting that burn it to the ground actually takes less time than fiddly-bit tracking, but it’s still hard to get your nerve up to do it.

There’s another reason to burn it to the ground and that’s speed. Truly starting from scratch is the least expensive way to speed up your machine.

Before going down any path like this, you have to consider your backup strategy. I use Backblaze for continuous backups and more importantly, I have a full Carbon Copy Cloner bootable backup on an external SSD. That bootable disk is critical to the process. Don’t even think about undertaking this until you have at least two backups. Remember that with one backup, as soon as you nuke your Mac you now have no backups at all because you only have one copy of your data. Ok, public service announcement over, let’s have some fun.

Over-the-Top Install

There are a few methods to fix Mac weirdness that are not truly a nuke and pave. I want to explain why I didn’t choose these paths.

One method that does have merit, is to do what is called an over-the-top install. This method reinstalls macOS, which can be handy if you feel like some system items have gotten borked up. It’s probably the least-invasive method and you do it via macOS Recovery. (Instructions at support.apple.com/…) It’s altogether possible that this method would have cleaned up the paper cuts I described earlier, but I also wanted to upgrade to macOS Big Sur. And again I don’t enjoy chasing down little bugs as much as I do starting from scratch.

Nuke but Pave User Data With Migration Assistant

Another method would be to erase your drive (a full nuke) but then do what I would call a repave, where you use Migration Assistant to put everything back. To me that’s like moving out of your house, putting in new carpeting, and then putting every bit of stuff you had before right back where it was. No throwing anything away, no purging of clothes or tools or computer gear, no Marie Kondo’ing anything at all. That would defeat one of the main benefits of the process. You’d still be bringing back apps you don’t really use anymore, and settings files and caches from apps you did delete, and more.

Migration Assistant does allow you to pick and choose what you want to migrate, so you don’t have to bring everything forward. But if you use it to migrate your data, the checkbox next to the User Library cannot be deselected. My User Library on the 16″ MacBook Pro was up to 177GB by the time I did the nuke and pave. This is one of the problems I want to fix.

Deep in your heart of hearts do you believe the entire 177GB of Library data is necessary? Or is it just possible that some of the 177GB of junk can go? A bit later I’ll walk through how I determined why my Library had grown so huge.

Migration Assistant for Applications Only

Another partial pave option would be to nuke the drive, move the User data by hand, but use Migration Assistant only to migrate Applications. This isn’t the dumbest idea if you are fairly stringent on what apps you load on your Mac. I am not at all stringent, and in fact, have probably added new apps to my Mac just while writing up this process! I suspect that even the most rigorous amongst us have kept old apps around out of some sort of nostalgia, thinking, “I will use this app someday…”. That’s the stuff you’ll have on your Mac for the rest of all time if you don’t do a nuke and pave.

The Nuclear Option

Personally, I’ve found that the most freeing option is to do a full nuke and pave. I like to compare the above options to trying to clean out a closet by only removing what you don’t need. You’ll remove maybe 10% of your clothes and it won’t’ feel like you have any more space than when you started. A nuke and pave of a closet means you take everything out of the closet and methodically only put back what you do use. I do this around once a year on my closet and I probably get rid of more like 30% of my unneeded clothes, and someone else gets the joy of “new to them” clothes. The same thing is true with nuking and paving your Mac.

I hope I’ve made a compelling case for a true nuke and pave, so let’s walk through the process. First, check your backups. Make sure that bootable backup you made truly does boot your Mac because you’ll probably end up needing to boot back into it from time to time for things you’ve forgotten. At least I always do.

Next, you probably want to make a bootable installer of the operating system you hope to install. I say probably because while I did make one it ended up being of no use to me. There used to be an app that would do this for you in a nice GUI interface, but it hasn’t worked for a few operating systems back, so I followed Apple’s instructions to make my bootable installer. It’s pretty simple, you just download the installer from the Mac App Store and run a simple Terminal command to make your thumb drive a bootable installer.

Nuking a Mac is a tricky thing. First of all, you’re terrified, but I’ve accidentally done an over-the-top install because I wasn’t paying attention. You’re smarter than me so I assume you won’t waste time doing it wrong like I have in the past.

This time I followed Apple’s detailed instructions on (support.apple.com/…) on how to erase an Intel or M1-based Mac from macOS Recovery. Once you’re in Recovery there’s an option to erase your disk, and if it’s an APFS-formatted drive you’ll have to choose to erase the entire Volume Group.

Ok, the scariest part is done. There’s no going back now! It’s time to boot to that bootable installer. This is where things went wrong for me. I was able to select my installer thumb drive, but it also needs your WiFi network password. I suspect that’s to check with Apple to verify this is a legit device to receive it, not some janky hackintosh. For some reason, this step would simply not accept my WiFi password.

I backed out and while still in macOS Recovery, I tried to change the Startup disk to the bootable installer thumb drive and I got a baffling message. It said, “The bless tool was unable to set the current boot disk.” I have no clue what that means or why it wouldn’t bless it. You may be hollering into your device that it’s because I have a T2 Mac and didn’t remember to go into Startup Security Utility and allow booting from external media. That’s not it though, because I had enabled booting from external media on this Mac.

When faced with baffling problems, I turn to Research Assistant Steven Goetz for guidance. He suggested that I use Internet Recovery, which is a method that pulls the OS down for you from the Internet as part of the process. I had looked into that before the nuke, and it showed it was going to give me Catalina. I had no other choice, and the great news is that Internet Recovery was now offering me Big Sur.

I pushed the button and went for a long, stress-relieving walk. When I returned, I had a lovely, fresh, pristine copy of macOS Big Sur 11.2 awaiting me. I should mention that later in the day I was prompted that my OS was out of date and did I want to install 11.2.1. They did patch that scary bug in sudo with this one so I guess I’ll allow it.

The Paving Process

Now that I have my nice empty closet, it’s time to start reassembling my system. If you’ve followed me for a while, you know that I’m a huge fan of using mind maps to organize information, and I favor the app iThoughts from Toketaware to make my mind maps.

I started maintaining a mind map in iThoughts many years ago to manage my nuke and pave process, and it’s one of the smartest things I’ve ever done. This mind map gives me a structure to follow to make sure I only install the stuff I need, and in the order of importance of those installations.

First, I categorized the apps into Mission Critical, High Priority, and Low Priority. I need the High Priority apps, but I basically can’t do anything on my Mac without my Mission Critical apps.

For example, I can’t even start to install apps until I have 1Password installed. Many of my apps now sync their data through Dropbox, so Dropbox itself is in the Mission Critical pile. Also Mission Critical is anything I use to create the podcasts, such as MarsEdit, Hindenburg, everything Rogue Amoeba makes, Discord and mimoLive. I can’t do Programming By Stealth without my Git clients so Sourcetree and GitKraken make the list too. If I want to actually deliver you the podcast, then Feeder has to make the top tier.

As I go through my mind map installing apps based on their priority, I change the color of the little bubbles to green so I can see my progress. That’s fun visually but after a while, it gets pretty cluttered, so I move the completed apps into a completed bubble within the Mission Critical topic. Now my reward isn’t the color change, with the completed bubbles collapsed, my mind map actually shrinks and I can concentrate on the apps I still have left.

The right side of the mind map is where I put the apps I want to install. On the left-hand side of the mind map, I documented every single little thing I have to do to configure those apps. As I explain this part of the process, you’ll realize the value of that Migration Assistant I was complaining about.

Configuring every tool is definitely the time-consuming part of the paving process. Perhaps for normal people, it wouldn’t be that big of a task, but if you’re into automation on your Mac, you’ve probably added many more configurations than you would think. Here are a few simple examples.

Most images in my blog posts are floated to the right or left and have a nice explanatory caption centered under the image. I could write out the HTML to do that every single time, but instead, I configured the media markup template built into MarsEdit to do it for me with a little pulldown. It’s not hard to reproduce, and I keep the configuration in my mind map so I can copy and paste it back in on my fresh install. Ideally, MarsEdit would have a way to export these settings and import them back in, but I’ve been working with the developer and even the command-line method he gave me didn’t work this year for some reason.

I do have to say that this year a lot of my apps synced their settings beautifully through iCloud or Dropbox. I was delighted especially that Steve Harris, developer of Feeder, spent the time and energy recently to add iCloud syncing for the servers for me to post the feed. It was glorious to open it on the new installation and realize that I had zero configuration to do.

The mind map I’ve made is a fantastic resource, and it’s also a living document. Each time I do a nuke and pave, many things will have changed. This year I created a category for app installs called “Maybe don’t install”. I created it because I found 9 apps that I don’t use that often or maybe don’t need at all any more, but I considered highly important the last time. If they’re still uninstalled by the next time I do this, I’ll probably delete them from the app list altogether.

Remember I said to make sure that bootable backup disk is really bootable before you start? The reason that’s so important (other than to save you in case of an internal drive failure) is that invariably you’re going to forget a setting that can only be retrieved by booting back into the old install.

This year an enhancement to the process was that I created a Dropbox folder entitled “Nuke and Pave” and I put all of the settings files I exported from my apps in there. One thing I didn’t used to keep track of are the hidden files I use more and more these days, like my RSA keys. Having these keys stored allow me to log into my servers for all the file transfers I do for the content of the shows without having to constantly enter my credentials. I was smart enough to zip the RSA keys up with a password when I put them in Dropbox.

Another major enhancement to my Nuke and Pave document was to create a bubble at the top with the title, “Before Nuke and Pave”. I made the bubble red, and because iThoughts supports Markdown, I was able to give make it a Heading 1 which means the text is HUGE. That’s where I put all of the stuff I forgot to export before the nuking step and had to go back and get by booting to the backup drive. I’m hoping that each time I go through this process I’ll get smarter.

When you embark on this journey yourself, and you create a mind map or an outline or even a plain text document of your process, please heed one important piece of advice. Leave Easter eggs for yourself. I found one of the High Priority tasks said “Get printing working”, and noticed that it had a sub-bubble and when I expanded it, it simply said, “Hahahahaha!” It’s good to make yourself laugh.

Several of the settings have morphed over time as well, so I keep the document up to date as I go, adjusting the text information and also adding screenshots of settings pages for clarity.

Oddly the very last two applications I install on a newly nuked Mac are actually in the list of Mission Critical apps. The two apps are my backup applications Carbon Copy Cloner and Backblaze. If I start those running before I’m 100% sure I’m done with paving, I might miss something and lose it forever. It’s scary to run without a new backup for a week or two, but the risk of losing data seems lower with this path.

How About that Pesky Data?

Speaking of data, when and how does my data come back?

One of the great joys of the Cloud life we lead these days is that a lot of my data came streaming down automatically. I sync Documents and Desktop to iCloud so everything was at my fingertips immediately. iCloud may take a bit to download but I can access anything just by selecting it and it will download. Dropbox holds a lot of other data I need, which is why it took a top spot on installations.

Photos are one of the most precious categories of data I have, but I’ve learned over the years that the slowest way to migrate the original full-resolution versions of my 83,171 photos is to import them from the backup. If I do it that way, iCloud Photo Library has to check every single photo with it’s database online to see if the full res version is actually downloaded. This process takes more than 3 weeks, and yes I’ve verified with Apple that this is inline with their predictions.

Senior Advisor once explained to me that a much faster method is to turn on iCloud Photo Library with optimized images instead, wait the short time for the small versions to download, and only then turn on download originals. Now iCloud Photo Library simply turns on the firehose and shoves them all down because it knows it doesn’t have to check to see if you already have them. If I keep my machine on 24×7 and Photos open, it takes about 3 days instead of 3 weeks. In addition, during those 3 days I have access to all of my photos and can, on demand, open the originals. In the 3-week method, no new photos download that I take with my phone so my Photos library is useless.

After the cloud downloads have done their work, I connect my backup drive and simply drag over anything that’s not in the cloud services. It’s a actually a surprisingly short list. In my Pictures folder, I have a lot of data that’s not in my Photos Library, I have a small smidge of music in my Music folder that I want to preserve outside of Apple Music, and my Videos folder has a lot of files I care about.

In Programming By Stealth when we are working on web apps, we run an app called MAMP that allows us to run a web server on our Macs. (Windows users use WAMP and Linux users use LAMP, see the pattern?) I bring this up because the VERY annoying thing about MAMP is that you have to keep all of your program files inside the MAMP folder inside Applications. How dumb is that? Some of my most critical data is inside my Applications folder!

What Have I Forgotten?

Remember that 174GB Library folder? I figured maybe I’d better take a walk through that to see if any of my apps were keeping their data in the Library and I’d be very sorry and sad to lose it. That has happened before, especially with database programs, and ones I don’t access super often so I don’t think to test right away when I get onto a new install.

It’s not a bad idea to peruse some of the folders, such as Application Support, but I only found things I was certain I didn’t need. What the heck is Propellerhead Software? I have no memory of that and yet a Migration Assistant process would have brought that along to my fresh install of Big Sur. Once I’d poked around in there and not found anything I needed, I thought it might be a good idea to figure out why my Library is so huge. Large files could potentially be of interest.

My tool of choice to look for large files is the free OmniDiskSweeper from the OmniGroup. I like OmniDiskSweeper because I can point at a specific folder and drill into it. With most of the disk space utilities out there you have to start at the top level, and with an 800GB+ Photos library, it dominates the graphics and makes it much harder to find large files.

I took a look at my Library folder using OmniDiskSweeper and found the biggest folder was called ScreenRecordings, and inside was an 82GB screen recording video. It’s a very weird video that’s several minutes long and yet I can see by the clock in the menu bar that time never advances. Had I migrated my user account, and thus my user Library, I would have carried that GIANT file with me into the future and been none the wiser. Ok, I could be wiser and scan my Library with OmniDiskSweeper on a schedule but you know that’s not going to happen.

Next up on the hit parade of giant folders in my Library is 22GB of attachments from Messages. I know it’s possible there’s an adorable photo or video of one of my adorable grandchildren in there, but I try to import the good ones into Photos as I go along so I’m pretty sure I don’t need any of them. I’m blindly going to let them all go.

Things got weirder in the third-largest folder at 12GB. It was in Group Containers and had a bunch of files with goop characters at the beginning but ended in “.Telegram”. I am an avid Telegram user, and in theory Telegram is supposed to keep attachments in the cloud. There are circumstances under which a file does get downloaded locally, but there were 63,932 “partial meta” files in there that I 100% know I do not need. Again this would have been 12GB of lost space on my drive if I’d used even the most minimal version of Migration Assistant.

Also inside the Group Containers was sync data from apps like 1Password, Apple Podcasts, Keep It and Mail. I know that all of these apps will recreate this data when installed so I can ignore these files too.

The Application Support folder in the User Library is an interesting place as well, but again I didn’t need any of it. My clipboard history was 4.5GB, and I suspect that’s because I left a really high limit on my clipboard manager Copy ‘Em. I can adjust that from within Copy ‘Em, and probably should , but since it syncs data through iCloud I know I don’t have to drag this data over.

The lesson here is that if you’re willing to go through and scan your Library to be certain you’re not leaving something behind, you can probably save an enormous amount of disk space if you avoid using Migration Assistant.

So Many Computers!

If you have one computer and you’re doing the Nuke and Pave dance, you’ll probably have to do a fair amount of rebooting back and forth between the old and the new to get this done. I did it using three computers and an iPad. I had my 2016 MacBook Pro booted to my backup drive, the 2019 getting all the nuking and paving and eventually hooked to the big display, and then the M1 Mac mini with my handy 12″ USB-C display being used to look things up and verify configurations.

After a while I did start to descend into madness, grabbing the wrong keyboard or trackpad or mouse as I flipped back and forth. It was a great exercise of the mind and muscle memory, but I had to step away from the keyboards once in a while to refocus and stay sane!

Move Bootable Backup to Network Attached Storage for Safekeeping

One of the last steps is to preserve my bootable backup somewhere so I can wipe the SSD and start using it as a new backup. I have about 4TB of space left on my Synology network attached storage and I knew I could put it there because I did it back in 2019, but I couldn’t remember how I did it! My poor memory is one of the reasons I document so much. All I could remember was that it was a sparse bundle.

I asked Steven Goetz how I did it. He reminded me that Carbon Copy Cloner (and SuperDuper!) can create sparse bundles and he thought it could save that sparse bundle to the Synology. That shook a brain cell loose, and I remembered when I taught the tutorial on Carbon Copy Cloner for ScreenCastsOnline, I explained that you can set the destination of a backup clone to go to any network attached drive. I started that process and let it go. Took a while at about 750Mbits/sec but I now have a copy of my backup so I feel better. If I ever need to use it to boot from, I would need to clone it back to an attached drive, but at least it’s safe and sound.

Bottom Line

Chasing down weird bugs is not repeatable, and it isn’t any fun so you probably just live with the bugs gnawing away at you. I’m sure as I’ve described the nuke and pave process it sounds more daunting than you hoped, but this process is repeatable and predictable. It also doesn’t take nearly as long as you think if you prepare well before you dive in.

I started my Nuke and Pave at 1:47pm on Wednesday, and by Thursday night Steve and I were able to run a successful test of the live show and I was able to record. I’ve been cleaning up bits and pieces since then when I find little stragglers to fix.

And guess what? I can use my Apple Watch to unlock my Mac. I can authenticate in 1Password with my Apple Watch. I can install apps authenticating with my Apple Watch. Search works in email. I can even use my phone as a scanner again! And my Mac is as fast as the day I got it.