iCloud Photo Library is a glorious thing. With a few dollars a month, you can have all of your photos swooshing up and down to the cloud, resident on all of your devices. On each device, you can choose whether to keep the originals or to let Apple use their algorithms to optimize your photo library. This optimization means you’ll never run out of space on your iPhone, iPad or your Mac.
If you choose optimized photos, some images will be stored locally in full resolution and some come down on demand when you tap on them. At all times thumbnails are available to be tapped, and as long as you have an Internet connection your vast library is always available to you. Any edits on one device are magically reflected on all other devices. Life is good.
My Photos library is giant by any measure, with more than 70,000 images taking up over 500GB of space. And yet I have access to all of these photos on my iPhone and iPad, which certainly don’t have 500GB of storage. I can even get to all of my photos by logging into iCloud.com. It really is a wonderful thing.
Except when it’s not. Some problems require logging out of iCloud and back in again to solve, or at the very least turning it off and on again. Maybe Contacts got corrupted. Maybe iCloud Drive isn’t syncing properly. Maybe you just bump the checkbox for iCloud Photo Library in System Preferences. This is when things can go horribly wrong.
Here’s the problem. On the machine which has the full resolution photos, If that connection to iCloud gets disturbed in any way, the iCloud service will check every single image to see if it’s been synced to and from iCloud. With a normal library, this might cause a check of say 1000 photos or even 5000 photos. But when you’ve got 70,000 photos, this turns into a huge problem.
With a 70,000 photo library, and a 100Mbps symmetric FiOS Internet connection, this process of checking every single photo takes around 3 weeks. That means I have to leave my computer on and not asleep 24 hours a day, 7 days a week for 3 weeks till it’s done verifying every file. It doesn’t actually take up a lot of bandwidth because it’s just saying, “Have you got this adorable picture of Allison’s grandson wearing rabbit ears and nothing else?” “Yup, got it.” “Ok, next one…”
That wouldn’t be the worst thing in the world, except that during this 3 week period, no new photos come down to the Mac, because it’s busy with this tedious task. If you’re not used to having your photos happily swoosh from your phone to your Mac, this doesn’t sound like a big deal, but as a huge Photos user, this is the whole point of iCloud Photo Library.
I’ve been putting off doing a nuke and pave for far too long and it’s all because after I drag my Photos library from my backup to fresh install of macOS, it does this three-week dance of agony.
Recently I happened to uncheck the iCloud Photo Library checkbox in System Preferences for some reason. I don’t even remember what possessed me to do it, but suddenly my Mac started rechecking every photo.
Every time I go through this agony, I call Apple again, asking them isn’t there a better way, but they never have an answer. This time when I called, first tier guy Corey told me there was no other way. Corey said that he’s been told to expect the process to take 1-4 hours for every gigabyte of photos. At 1 hour for 500 GB, that’s 500 hours, divided by 24 hours is around 20 days, which is just about 3 weeks. The wisdom he’s been told to impart matches my experience, but it didn’t make me any happier!
At that point, I started asking questions he couldn’t answer, so he transferred me to my new little friend Senior Advisor Frank. I told Frank about my problem and talked about a few other things that have been going on and he came up with a really interesting workaround to my problem. I’m not going to call it a solution because it doesn’t fix the root cause, but it has significant advantages in speed.
Frank didn’t explain what was going to happen so I’ll keep you in suspense as I walk you through the steps he asked me to follow. The first thing he asked me to do (while he was watching on a screen share) was to show him how much free space I had on disk. I have a 2TB disk in my MacBook Pro and I have around 900GB free. I didn’t know why he asked that but it will become clear later.
First, he said to open System Preferences ? iCloud ? Photos ? Options and then to uncheck iCloud Photo Library. This was the madness that started everything going awry but I had nothing to lose right then because it had already started it’s 3-week dance of doom (™️Donald Burr), so I went along with it.
Next he had me hold down the option key while opening Photos. This gives you the option to select a different Library than your main Library, or you can create a new Photos Library. He said to create a new Photos Library and to entertain him, I named this new Library “Photos Library Frank”.
Inside Photos, he had me open Preferences, and on the General tab, he had me select Use as System Photo Library. In Apple Photos, you can have many Libraries, but only one can be the System Photo Library. The System Photo Library is the one that will be synced across all of your devices. It wasn’t yet obvious to me at this point why he was making a completely empty Library be my System Photo Library.
Then he had me go back into System Preferences ? iCloud ? Photos ? Options and re-check the iCloud Photo Library check box.
I’m sure some of you have figured out what Frank did for me. By creating an empty Photos Library and making it the System Photo Library and then turning on iCloud Photo Library, he told my Mac to download all 70,000 photos.
This sounds like an insane idea, but it was actually quite brilliant. I know, I’ve got those same 70,000 photos on my disk in the other library, but it turns out that with a fast enough Internet connection, it’s actually faster to download all 500GB of them again than it is to check all of them one by one.
An important step to remember is the very first thing he did. He turned off iCloud Photo Library at the top level in System Preferences. Had he not done that, things would have started syncing while we were flipping over to the new empty library. That’s crucial and something I wouldn’t have thought to do. I have always controlled my library syncing inside Photos Preferences.
When the process first started, it was positively lightning speed. I figured out why it was downloading all of the thumbnail versions first. That turned out to be awesome for a couple of reasons. First of all, it downloaded the most recent ones first. Remember I couldn’t see any of my recent photos during the 3-week dance of doom? With the thumbnails coming down of the recent ones, I could tap on them and immediately had access.
I monitored the download progress, first by the number of images, and once that stabilized (at 70,836) I started watching how much disk space it chewed up as my metric of progress. When all of the thumbnails were done, it was 6.29GB and that took 15 hours.
I started plotting the disk space used as a function of time in Excel (as one does) so I could plot it and figure out when it would be done. According to my plot estimate, it should have been done in about 58 hours, but unfortunately, during the middle of the night on the third night, my Mac decided to jump onto my test Orbi network that wasn’t actually still connected to my modem so I lost an indeterminate amount of time.
I restarted it the next morning and in all, even with that network hiccup, the entire 70,000 Photo Library came down in around 4 days, instead of 3 weeks! I should note that during the 3 days of the full downloads, my Mac’s Photo Library wasn’t getting new photos, but that’s a lot better than 3 weeks.
Now remember one of the first things Frank did was check my disk space? You’ll realize now the reason why. He had to make sure I had enough space to have an entire duplicate of my Photos Library. We talked about that after they started the new download, and he said he’d have had me simply delete my Photos Library if I hadn’t had the space. Of course, he made sure I had a backup and probably would have insisted on two if I’d had to delete the library before starting.
As I said earlier, this is not a solution, but a clever workaround that helped cut my Photo Library downtime by 80%. I sure wish there was a way to say, “They’re all in sync, I promise!”
Nice article.
I too am in the same boat. 480GB Photo library that I was managing myself. I turned on iCloud Library after months of decision making. Initially it seemed to work. But at some point I ‘needed’ to turn off my iCloud Account which included turning off iCloud photos. Turned iCloud back on and didn’t think twice. Two weeks later I checked my photo library on my Mac and I didn’t see any new photos or videos from the past two weeks.
Noticed that iCloud photo library was “uploading” another 30,324 items. Also noticed that number was not changing after an hour or so.
First thing I did was to kill the Photos Agent in Activity Monitor. Then quit Photos and restarted Photos. Uploading was now working.
But after reading this blog I now understand that it is in fact, checking the sync status of each item.
So yeah, same issue but if I had known the reifications of simply turning off a toggle I would have never attempted to “turn off” my iCloud account.
Let me add this, Apple charges a premium for EVERYTHING they sell and you pay that premium as a consumer. This premium (back in the SJ days) was there to ensure you had a seamless experience and that everything “Just works”.
Apple should be ashamed that they consider 1-4 hours per GB just for sync checking to be acceptable.
Maybe they should consider upgrading the DSL connection to their 286sx servers.
I could keep going but lately Apple seems to be reminding me of my old PC days where I spent hours “maintaining” my machine.
Just make is work, please.
I have experienced the same issue many times and learnt the trick myself. I now maintain two original libraries of 1.4TB for my 70000 images (many in raw format)
1 on my MacBook Pro and external 5tb drives seems to work well
The other on my Mac mini using a 3tb drive seems to need to be killed and re downloaded every few weeks. I had no problems with this until I upgraded to Catalina and then I started getting Apple ID issues on my Mac mini and photos became a nightmare!!
Bummer, Kev! I’m curious, what happens on the Mac mini that makes you need to download again?
I spend a few days trying to fix this problem until I came across you guidance. Thank you!!!!!
I just upgraded to a new Mac, copied over my non-cloud Photos library to a new external hard disk, turned on iCloud Photos and started syncing. I have 50,000+ photos and 1,400 videos. I’m 5 days in and still at “Uploading 29,000 Items.”
I’m very encouraged by the workaround described above, but I want to confirm my understanding of how the two-library structure works:
1) Original Library is the one on my hard disk that I want to copy to iCloud (no longer designated as the System Photo Library)
2) New Library that is designated as System Photo Library — this starts off empty, but becomes the “Master Library” during the iCloud Photos syncing process and downloads a duplicate copy of the Original Library via the iCloud Photos syncing process just like it would any other photos library on any other device associated with my iCloud account?
So when everything is done syncing, do you have to keep both the Original Library and New Library for things to continue to work? Or can you delete the Original Library? (assuming you are sure everything from the Original Library has been uploaded to iCloud Photos?)
Thanks again for sharing this workaround, I would definitely love to cut the synching down to a few days instead of a few weeks!
P.S. In macOS Catalina, iCloud Photos settings are native to the Photos app and not in System Preferences… but I assume the workaround still works?
Hi Hugh – Let’s see if I can answer to your satisfaction (though it’s TERRIFYING to give advice on something so precious!). First of all, please make sure you back up the originals in every way you can including a disk at your Mom’s house, one buried in the back yard and one given to a delivery pigeon, ok?
The basic idea is you’re going to trick iCloud into thinking there are no photos on your Mac, and have it simply _download_ the originals, not sync them. Downloading is fast, syncing is slow because every single one has to be checked individually. After the download, then it can start syncing like it used to.
I don’t want to use the specific terminology because I haven’t executed it in a few years. In my method I had enough disk space to save the original photos and let it download a whole new library, and I did indeed delete the original libraray once I was certain it was all there. In your case, you don’t have the entire library yet on the machine (I presume they’re all on the old machine), so getting rid of this half-done library shouldn’t be a big deal.
I hadn’t noticed macOS Catalina had moved the iCloud Photos settings but the concept should still work. Good luck and let us know how you get on?
Thanks for taking the time to respond!
After thinking more about this since my last post, I’m wondering if my non-cloud stored photos have to be uploaded to iCloud Photos *first* for this workaround to work. I haven’t used iCloud Photos before, so all my 55,000+ photos/videos are only stored locally on my external hard drive. So when you say “download” to the “New System Library”, I don’t see how that will work because the “Original Library” is just sitting on the external hard drive and not connected to iCloud or anything.
I thought maybe I should export all my photos/videos from the “Original Library” in Photos to a folder on my Mac, then upload them to iCloud via the web interface, but then I realized that this approach wouldn’t retain any of my albums, etc.
So I think I’m stuck with the sync/upload from the Original Library, at least for this first time I’m uploading everything to iCloud. After that, then maybe the workaround would be helpful when I’m migrating in the future to a new computer etc.?
If I’m missing anything, please let me know!
OHHHHH! Well then YES you have to upload to iCloud first! The instructions here are for when it’s already IN iCloud Photo Library and you want to bring the originals down. Glad you clarified. iCloud Photo Library is so fabulous, I love having all my photos absolutely everywhere. It will cost you a fair bit of money with a library as big as that, but I sure love it.
Update: my photo library is now in iCloud Photos!!! A couple of things I did in an effort to avoid the uploading process from stalling:
1) Ethernet connection instead of wifi:
I suspected that my wifi router might be throttling/stalling the upload, so I connected my Mac directly to my ethernet router. My upload speed clocks around 20-40 Mbps. However, since I have a new MacBook Air with only 2 USB-C ports, one of which was used for the external hard drive where all my photos are stored, I had to use two adaptors to get both power and ethernet from one USB-C port. I used Apple’s USB-C Digital AV Multiport Adapter, which has USB-C (for power), HDMI (not used in this case but still handy) and a standard USB port, the last of which I then used to attach a USB to Ethernet adaptor.
2) Automator workflow to copy a dummy file to iCloud every 5 seconds:
I had read in other postings that the connection to iCloud can sometimes stall out, so I put together this simple 4-step Automator workflow that would copy a dummy file to my iCloud Drive every 5 seconds:
i. “Get Specific Finder Items”, dragged the dummy file into this box to select it
ii. “Copy Finder Items”, selected the target folder on iCloud Drive and checked the Replacing existing files box
iii. “Pause” for 5 seconds
iv. “Loop” with loop automatically and stop after 1,000 minutes (I had to restart this after it timed out)
Not sure if the Automator workflow helped or not, but I don’t think it hurt. It took 3.5 days to upload 57,000 items (compared to my initial attempt over 5 days after which only 29,000 items had uploaded), and it looks like everything is there including photos albums etc.
At any rate, thanks again for the responses and hopefully this thread will help others.
Hugh – that’s fantastic! Even if the upload a dummy file every 5 seconds thing was voodoo that didn’t help, what a cool nerdy way to do it with Automator. I’m so happy you got it to work. Thanks so much for posting exactly how you did it to help others.
Now are you going to keep originals on the external drive locally I presume or what’s your plan about that?
Thanks! Oh yes, I’ll keep originals on the external drive and have an extra backup, too.
Thanks for sharing! My sync was on its 2nd day which didn’t seems to be advancing any on my macOS Catalina, but then I used the create new phone library method mentioned in the article above and my 7,000 photos downloaded in a minutes! The videos will take a bit longer to download completely but this is better than waiting days to complete.
Awesome, Carlos! So glad this worked for you.
I have a similar problem. I did a complete reinstall of my 2020 MacBook Pro and then uploaded the latest version of Big Sur. Was nice to have a clean setup again with the latest OS. And I thought with everything backed up in iCloud I should be okay. The only thing is that when clicking on my photo library I get the message “updating”. For a couple of days now!
So probably the solution above should work for me as well!
Now I wonder if it is not much quicker to just connect the iPhone (or iPad) with a usb cable and synchronize photos in this way (described in https://support.apple.com/en-in/guide/mac-help/mchl4af095d3/mac “Sync photos between your Mac and iPhone or iPad)
Is that just to simple and do I overlook something???
(In my case almost all photos in the MacBook photo library were taken from my iPhone)
I think I may have stumbled onto a way to prompt Photos to download originals immediately, without the above workarounds.
I have a 600GB Photos library, containing 91,714 photos and 1575 videos. My primary computer is a 2017 iMac running MacOS Ventura 13.01. My originals are stored there, on an offboard SSD.
I wanted to create a standalone copy of the library I could give to my kids. However, their machines are running older OS’s (mostly Big Sur), and a direct copy of the file would not open, because it had been converted for Ventura. My plan was to use an older MacBook Pro running Big Sur, sign it into my iCloud account, and sync photos there, using the “download originals to this computer” option. Because my broadband data capacity is capped at home, I had to start it on a work network. I ran into the reindexing issue described above pretty quickly when I returned home to resume the download on my home network.
Starting over, using your method, wasn’t an option, as I had already downloaded about a third of my directory at work, and didn’t want to start over again from scratch on my home network.
My solution, gleaned from a posting elsewhere, was to create a Smart Album containing ALL of my images and videos (Date created 1900-2022), then select “Play Slideshow”. After watching the spinning beachball for about 5 minutes, the machine starts “Downloading xx of xxxxx items for creating a slideshow”! I had tested this with smaller albums first, and confirmed that the Photos directory did increase in size each time, and that the number of photos remaining to sync in the main library window changed each time to reflect the images downloaded. It’ll still be a while before this completes, but I do thing this is a simple way to prompt Photos to intitate downloading of originals to a remote machine.