I spent quite a lot of yesterday fixing my automated backup system at work.
Most important bit of this post: In recent versions of macOS, if you want a scheduled Bash shell script to write to your disk, you need to explicitly give Bash “Full Disk Access” permissions.
Side note: In most contexts, using a standard backup system like Time Machine or Backblaze would probably address the problem I was trying to solve here. (It’s even possible that the standard backup system at work would solve my problem, but I set up my system before that existed, so I tend to forget about it.) So the specifics of what I was doing are probably something that few people would need to do. But some of the general ideas might still be useful.
Some time ago, sparked by the frequency of losing all my browser tabs on my work computer, I set up shell scripts to make backup copies of various files, such as Chrome’s sessions and tabs files. (Also my Things database, and various other important administrative files.)
I then set up a crontab to run those scripts regularly.
But at some point in the past few months, the cron jobs stopped working.
I couldn’t find any reason for that. When I ran the scripts manually, they worked fine. I checked the crontab syntax and it seemed to be fine. But the backups weren’t happening.
Red herring: time zones
As part of my investigation, I checked on whether the computer’s idea of what time it was matched mine. But I did that using the UNIX
date command, which gave me the time in UTC instead of my local time.
I then spent quite a bit of time trying to figure out why the command line didn’t seem to know my local time zone, even though the GUI did. I thought that maybe cron was going by UTC time instead of local time, and that’s why my tests weren’t working.
Eventually, I just gave up on this whole line of inquiry. I never did figure out what was going on with the time zones.
macOS cron is deprecated?
While I was investigating various things about cron, I discovered that Apple doesn’t recommend using it to schedule things!
It turns out that, as of several years ago, Apple instead recommends using
launchctl. (cron is still supported, but not recommended.) You specify a bunch of parameters in a .plist file (including saying what to do and when to do it), and then you load the .plist file into the launch system, and the launch system runs the code at the specified time(s).
(There’s a third-party GUI tool that does various launchctl things for you, but I figured that was unlikely to be usable on my work computer.)
So I started trying to figure out
launchctl, but then I remembered that Apple now has a GUI for automating stuff.
Automator to the … oops, never mind
So I created an Automator workflow to call a shell script. And I scheduled running that workflow using Calendar. (After a frustrating side trip when the Calendar pick-a-file GUI froze up for no clear reason and I misinterpreted what was going on.)
And then I re-discovered that at work, we can’t run unapproved applications. (I’m oversimplifying here, but close enough to true for this context.) So I couldn’t use Automator for this task.
But in many contexts, I think using Automator to call the backup script and scheduling it using Calendar might be the simplest approach to the problem I was trying to solve.
(Well, simplest approach other than just using a standard backup system.)
Back to launchctl
So I created a launchctl .plist file and used
launchctl load to prepare it to run, and I used
launchctl list to make sure it was loaded, and then I waited for the time that I had specified in the .plist file…
…and nothing happened.
Except that the status-of-last-run in
launchctl list changed from 0 to 1, suggesting that some kind of error had happened on the last run, but of course no indication of what kind of error.
So I spent a very long time trying all sorts of things to get it to work.
One tool that I wish I had known about sooner:
plutil, which (among other things) checks the syntax of .plist files. It turned out that my syntax was fine, but it would have been nice to know that a lot sooner so that I could have skipped spending time manually checking syntax.
Another tool that I wish I had tried sooner: the
StandardErrorPath parameter in the .plist file, which lets you specify a file for the system to log error messages to. When I finally tried that, the resulting error messages were super helpful, and quickly led me to the solution.
Solution: Full Disk Access
It turns out that if you’re calling a Bash script from
launchctl, and that script writes to your disk, you need to explicitly give Bash “Full Disk Access” permissions. (In System Preferences, Security & Privacy preferences, Privacy tab.) Even if you’re calling the script as superuser, it won’t write to disk without Full Disk Access.
(Giving Full Disk Access to a shell is slightly complicated: you need to open /bin in Finder, then drag the shell icon into the right part of the System Preferences window.)
As soon as I gave Full Disk Access to Bash, everything started working.
And it didn’t occur to me until I wrote up this post that that might have been the problem with the cron jobs in the first place. Possibly if I had given Full Disk Access to Bash from the start, the cron jobs would have worked fine.
Some day, I need to learn how to get scheduled shell scripts to provide me with useful error reporting. But today is not that day.