iCloud Photo Library is a glorious thing. With a few dollars a month, you can have all of your photos swooshing up and down to the cloud, resident on all of your devices. On each device, you can choose whether to keep the originals or to let Apple use their algorithms to optimize your photo library. This optimization means you’ll never run out of space on your iPhone, iPad or your Mac.
If you choose optimized photos, some images will be stored locally in full resolution and some come down on demand when you tap on them. At all times thumbnails are available to be tapped, and as long as you have an Internet connection your vast library is always available to you. Any edits on one device are magically reflected on all other devices. Life is good.
My Photos library is giant by any measure, with more than 70,000 images taking up over 500GB of space. And yet I have access to all of these photos on my iPhone and iPad, which certainly don’t have 500GB of storage. I can even get to all of my photos by logging into iCloud.com. It really is a wonderful thing.
Except when it’s not. Some problems require logging out of iCloud and back in again to solve, or at the very least turning it off and on again. Maybe Contacts got corrupted. Maybe iCloud Drive isn’t syncing properly. Maybe you just bump the checkbox for iCloud Photo Library in System Preferences. This is when things can go horribly wrong.
Here’s the problem. On the machine which has the full resolution photos, If that connection to iCloud gets disturbed in any way, the iCloud service will check every single image to see if it’s been synced to and from iCloud. With a normal library, this might cause a check of say 1000 photos or even 5000 photos. But when you’ve got 70,000 photos, this turns into a huge problem.
With a 70,000 photo library, and a 100Mbps symmetric FiOS Internet connection, this process of checking every single photo takes around 3 weeks. That means I have to leave my computer on and not asleep 24 hours a day, 7 days a week for 3 weeks till it’s done verifying every file. It doesn’t actually take up a lot of bandwidth because it’s just saying, “Have you got this adorable picture of Allison’s grandson wearing rabbit ears and nothing else?” “Yup, got it.” “Ok, next one…”
That wouldn’t be the worst thing in the world, except that during this 3 week period, no new photos come down to the Mac, because it’s busy with this tedious task. If you’re not used to having your photos happily swoosh from your phone to your Mac, this doesn’t sound like a big deal, but as a huge Photos user, this is the whole point of iCloud Photo Library.
I’ve been putting off doing a nuke and pave for far too long and it’s all because after I drag my Photos library from my backup to fresh install of macOS, it does this three-week dance of agony.
Recently I happened to uncheck the iCloud Photo Library checkbox in System Preferences for some reason. I don’t even remember what possessed me to do it, but suddenly my Mac started rechecking every photo.
Every time I go through this agony, I call Apple again, asking them isn’t there a better way, but they never have an answer. This time when I called, first tier guy Corey told me there was no other way. Corey said that he’s been told to expect the process to take 1-4 hours for every gigabyte of photos. At 1 hour for 500 GB, that’s 500 hours, divided by 24 hours is around 20 days, which is just about 3 weeks. The wisdom he’s been told to impart matches my experience, but it didn’t make me any happier!
At that point, I started asking questions he couldn’t answer, so he transferred me to my new little friend Senior Advisor Frank. I told Frank about my problem and talked about a few other things that have been going on and he came up with a really interesting workaround to my problem. I’m not going to call it a solution because it doesn’t fix the root cause, but it has significant advantages in speed.
Frank didn’t explain what was going to happen so I’ll keep you in suspense as I walk you through the steps he asked me to follow. The first thing he asked me to do (while he was watching on a screen share) was to show him how much free space I had on disk. I have a 2TB disk in my MacBook Pro and I have around 900GB free. I didn’t know why he asked that but it will become clear later.
First, he said to open System Preferences → iCloud → Photos → Options and then to uncheck iCloud Photo Library. This was the madness that started everything going awry but I had nothing to lose right then because it had already started it’s 3-week dance of doom (™️Donald Burr), so I went along with it.
Next he had me hold down the option key while opening Photos. This gives you the option to select a different Library than your main Library, or you can create a new Photos Library. He said to create a new Photos Library and to entertain him, I named this new Library “Photos Library Frank”.
Inside Photos, he had me open Preferences, and on the General tab, he had me select Use as System Photo Library. In Apple Photos, you can have many Libraries, but only one can be the System Photo Library. The System Photo Library is the one that will be synced across all of your devices. It wasn’t yet obvious to me at this point why he was making a completely empty Library be my System Photo Library.
Then he had me go back into System Preferences → iCloud → Photos → Options and re-check the iCloud Photo Library check box.
I’m sure some of you have figured out what Frank did for me. By creating an empty Photos Library and making it the System Photo Library and then turning on iCloud Photo Library, he told my Mac to download all 70,000 photos.
This sounds like an insane idea, but it was actually quite brilliant. I know, I’ve got those same 70,000 photos on my disk in the other library, but it turns out that with a fast enough Internet connection, it’s actually faster to download all 500GB of them again than it is to check all of them one by one.
An important step to remember is the very first thing he did. He turned off iCloud Photo Library at the top level in System Preferences. Had he not done that, things would have started syncing while we were flipping over to the new empty library. That’s crucial and something I wouldn’t have thought to do. I have always controlled my library syncing inside Photos Preferences.
When the process first started, it was positively lightning speed. I figured out why it was downloading all of the thumbnail versions first. That turned out to be awesome for a couple of reasons. First of all, it downloaded the most recent ones first. Remember I couldn’t see any of my recent photos during the 3-week dance of doom? With the thumbnails coming down of the recent ones, I could tap on them and immediately had access.
I monitored the download progress, first by the number of images, and once that stabilized (at 70,836) I started watching how much disk space it chewed up as my metric of progress. When all of the thumbnails were done, it was 6.29GB and that took 15 hours.
I started plotting the disk space used as a function of time in Excel (as one does) so I could plot it and figure out when it would be done. According to my plot estimate, it should have been done in about 58 hours, but unfortunately, during the middle of the night on the third night, my Mac decided to jump onto my test Orbi network that wasn’t actually still connected to my modem so I lost an indeterminate amount of time.
I restarted it the next morning and in all, even with that network hiccup, the entire 70,000 Photo Library came down in around 4 days, instead of 3 weeks! I should note that during the 3 days of the full downloads, my Mac’s Photo Library wasn’t getting new photos, but that’s a lot better than 3 weeks.
Now remember one of the first things Frank did was check my disk space? You’ll realize now the reason why. He had to make sure I had enough space to have an entire duplicate of my Photos Library. We talked about that after they started the new download, and he said he’d have had me simply delete my Photos Library if I hadn’t had the space. Of course, he made sure I had a backup and probably would have insisted on two if I’d had to delete the library before starting.
As I said earlier, this is not a solution, but a clever workaround that helped cut my Photo Library downtime by 80%. I sure wish there was a way to say, “They’re all in sync, I promise!”