3-2-1 Backup: Setting Up Raspberry Pi for Cloud Sync

Home IT

Jan 22

This article provides a guidance to configure a Raspberry Pi for sending backup data off premises / to cloud. Nowadays, there is a wide selection of cloud providers. Covering specific details of working with all of them is an enormous task, hence it is out of scope of this article. Having said that, Rclone - the tool used for pushing data to cloud and keeping remote copies in sync with local backup data - makes it somewhat provider-agnostic and straightforward to interface with over three dozen of cloud services. The examples below assume Dropbox as a cloud service provider.

Note: This write-up is a part of 3-2-1 Backup series. Please make sure to start with the Introduction.

Prerequisites

Dropbox account. Both Personal and Business plan families will work. If you do not have one, head over to Dropbox to register. Dropbox has a free tier plan limited to 2 GBs of storage capacity. While it most likely won’t be enough for production, it is more than suitable for experimentation.
Note: since we are going to interface with Dropbox via Rclone, installing Dropbox client software on your Mac is not necessary.

Configuring Rclone

Rclone is a command-line program for managing files on cloud storage. I invite you to read more about it on the official website.

Connecting Rclone to Dropbox Using Mac

We will perform Rclone configuration on a Mac. That is because the steps require launching a web browser and granting Rclone the necessary permissions to your Dropbox account. We will copy the resulting rclone.config file from the Mac to Raspberry Pi.

Let’s starting with installing Rclone on a Mac. There are three official methods to choose from:

Open one of the links above and follow the instructions. My preference is Homebrew, in this case

Execute the following from a Terminal window:
```
$ brew install rclone
```
Then, still on your Mac, follow the instructions to configure Rclone for use with Dropbox.

Successful completion of Rclone configurations produces ~/.config/rclone/rclone.conf file with the contents shown below (obviously, your token value will be different):

[remote]

type = dropbox

token = {"access_token":"sl.BFd1BZNUJ6LrICymhfQ5RWqOAmrdOJHzRDEw--UeJXGEzeKAihZAHFS2VrapULgidHR2tX80E13LEhvZSPgtXxJvaTEAv10Br8GorS_heKSYKftobstCuOwyNb2K_NIs0sIbj7jh","token_type":"bearer","refresh_token":"r826QNxnKY4AAxAAAYAC29IfyNJL0XFXOAAcn7vis4xiIE4QoOCETggNPfpWSNTt","expiry":"2022-04-10T18:14:02.026336-07:00"}

Rclone on Raspberry Pi

In the previous part we created a dedicated user called scribe to perform backup related tasks. We will add more responsibilities for this user here, including syncing backup data to cloud. Hence, all the steps after intstallation of Rclone will be performed by scribe.

Ssh to your Raspberry Pi:
```
$ ssh pi@scribe.local
```
First things first, install Rclone:
```
$ sudo apt install rclone -y
```
Switch to scribe user account:
```
$ sudo su scribe
```

Create an empty ~/.config/rclone/rclone.conf file:

$ cd && ( [ -d ~/.config/rclone ] || mkdir -p ~/.config/rclone ) && touch ~/.config/rclone/rclone.conf

Open ~/.config/rclone/rclone.conf in nano text editor:
```
$ nano ~/.config/rclone/rclone.conf
```
Then copy / paste text from your Mac’s ~/.config/rclone/rclone.conf file into the open nano editor window.
Save the changes by pressing ^O (Conrol+O on Mac keyboard) and exit ^X (Control+X).

Testing Rclone

Now we can test that Rclone connects to the remote at Dropbox and can transfer data.

Make sure we are running as user scribe:

$ [ $(whoami) == "scribe" ] || sudo su scribe

Generate a random file of 1 MB size:

$ ( [ -d ~/tmp/rclone-test ] || mkdir -p ~/tmp/rclone-test ) && cd ~/tmp/rclone-test && dd if=/dev/urandom of=1MB.bin bs=1M count=1

Let’s create a directory in Dropbox where we will send the file from the previous step. I call it clone-test:
```
$ rclone mkdir remote:rclone-test  --log-level INFO
```

Finally, let’s send the file to Dropbox:

$ rclone copy "$(pwd)/1MB.bin" "remote:rclone-test" --log-level INFO

3-2-1 Backup. Test Remote Copy with Rclone — Fig 1. Test Remote Copy with Rclone

Fig 1. shows console output of the four commands above. Now login to your Dropbox account and verify that rclone-test directory exists and contains file 1MB.bin.

Automating Sync to Cloud

Scripting Sync to Cloud Actions

Create an empty sync-backups.sh file in scribe’s home directory, under ~/bin, and make it executable:
```
$ ( [ -d ~/bin ] || mkdir ~/bin ) && touch ~/bin/sync-backups.sh && chmod a+x ~/bin/sync-backups.sh
```
Open it in nano:
```
$ nano ~/bin/sync-backups.sh
```
Copy / paste the script contents from the snippet below:

#!/bin/bash

# Global vars
CLOUD="remote:backups-$(hostname)"
BTRFS_SUBVOL="/mnt/store/backups"
SNAPSHOTS_DIR="/mnt/store"
LOGS_DIR="/home/scribe/logs"

DATE=$(date +"%Y-%m-%d_%H%M%S")

# Calculate snapshot filename
SNAPSHOT="$SNAPSHOTS_DIR/snapshot-$DATE"

# Calculate log filename
YEAR=${DATE:0:4}
MONTH=${DATE:5:2}
LOG="$LOGS_DIR/$YEAR/$MONTH/$(basename "$0")-$DATE.log"

echo -e "Current log file is:\n  \"$LOG\""

# Make sure the directory structure for the log file exists
[ ! -d "$LOGS_DIR/$YEAR/$MONTH" ] && mkdir -p "$LOGS_DIR/$YEAR/$MONTH"

# Create a read-only snapshot
echo -e "\n#\n# Step 1. Creating read-only snapshot \"$SNAPSHOT\"\n#\n" 2>&1 | tee -a "$LOG"
btrfs subvolume snapshot -r "$BTRFS_SUBVOL" "$SNAPSHOT" 2>&1 | tee -a "$LOG"
[ ! $? -eq 0 ] && exit 1

# Send to CLOUD
echo -e "\n#\n# Step 2. Sending to cloud. From \"$SNAPSHOT\" to \"$CLOUD\"\n#\n" 2>&1 | tee -a "$LOG"
rclone sync "$SNAPSHOT" "$CLOUD" --log-level INFO 2>&1 | tee -a "$LOG"
[ ! $? -eq 0 ] && exit 1

# Make snapshot writable / deletable
echo -e "\n#\n# Step 3. Removing read-only property from snapshot \"$SNAPSHOT\"\n#\n" 2>&1 | tee -a "$LOG"
btrfs property set -ts "$SNAPSHOT" ro false 2>&1 | tee -a "$LOG"
[ ! $? -eq 0 ] && exit 1

# Delete the snapshot
echo -e "\n#\n# Step 4. Deleting snapshot \"$SNAPSHOT\"\n#\n" 2>&1 | tee -a "$LOG"
btrfs subvolume delete "$SNAPSHOT" 2>&1 | tee -a "$LOG"
[ ! $? -eq 0 ] && exit 1

echo -e "\n#\n# Step 5. Success\n#\n" >> "$LOG"

Save the changes by pressing ^O (Conrol+O) and exit ^X (Control+X).

Testing sync-backups.sh

Let’s make sure that we have some content (aka files) at the backup location on Raspberry Pi:
```
$ cd /mnt/store/backups && dd if=/dev/urandom of=1MB.bin bs=1M count=1 && cd
```
Then run sync-backup.sh:
```
$ ~/bin/sync-backups.sh
```

Fig 2. shows output produced by sync-backup.sh. After running sync-backup.sh your Dropbox account should have backups-scribe directory synchronized with the contents of your Raspberry Pi’s /mnt/store/backups/ Btrfs subvolume.

Two more things to note. First, sync-backup.sh is implemented to delete a snapshot upon succefull execution. So, /mnt/store/ should contain no items starting with snapshot-*. If for any reason you will need to perform manual cleanup, please remember that even though those appear like directories, in reality they are Btrfs subvolumes. Btrfs subvolumes must be manipulated via Btrfs commands: btrfs property set and btrfs subvolume delete only! Use the script above as a cheatsheet. More details can be found the official Btrfs wiki.

Second, sync-backup.sh writes detailed logs under /home/scribe/logs/. The logs are organized into folder hierarchy by year and months for convenience.

Triggering sync-backup.sh from Cron

We will rely on a common Unix / Linux cron job scheduler facility to trigger sync-backup.sh execution every night at 2 AM. Cron knows when and what to run by looking at a specially formatted crontab (cron table) file. For user scribe we want crontab to look like shown below:

MAILTO=""

0 2 * * * cd /home/scribe/bin && ./sync-backups.sh

Crontab file is not meant to be manipulated directly, instead we will use ctonrab utility for that:
```
$ echo -e "MAILTO=\"\"\n\n0 2 * * * cd /home/scribe/bin && ./sync-backups.sh" | crontab -
```
Checking the contents of crontab file is easy as well:
```
$ crontab -l
```

3-2-1 Backup. Scheduling Cron Job for User scribe — Fig 3. Scheduling Cron Job for User **scribe**

Summary

We have achieved two major milestones: the Raspberry Pi is ready to accept Time Machine backups from Macs on the local network. Additionally, every night at 2 AM local time, the Raspberry Pi executes sync-backup.sh script and pushes changed files from /mnt/store/backups to Dropbox.

The last remaining task is making sure Time Machine does not run backups at the time of the Raspberry Pi grabbing a snapshot of data for cloud sync.

Next part: Tuning Time Machine on Mac Clients

Keep it Safe. Back Up!

BackupRaspberry PimacOSTime Machine3-2-1 Backup

Oleg Ovechkin https://ovechkin.xyz