Automated Tarball Backups via Rclone
Goal: Automatically compress select directories and upload tarball to remote storage via rclone.
Sharing my small script that I finalized, which may come in handy for others. Prior to doing automated backups, I typically backed up important data manually, as needed. This has come to bite me in the ass on a few occasions with not having the latest backups or not having what I really needed in my latest backup.
The gist of the script is that you make two files, the backup-include.txt list and the backup-exclude.txt list. Everything thatās included in the include list (one directory per line) is compressed and added to the tarball. Everything thatās included in the exclude list is blocked from being added to the tarball. An entry to the exclusion list is only necessary if the parent directory is in the included list, but you donāt really want everything thatās in that directory to be backed up. For example, if I have ā/home/mason/hostballs-filesā as a line in the inclusion file, but I donāt want to back up Jarlandās cat videos, then Iād add ā/home/mason/hostballs-files/jars-kitty-vidsā to the exclusion file. Thus, the entirety of the hostballs-files directory will be in the tarball, with the exception of Jarlandās cat collection.
How to Setup/Run
- Install rclone on your server
- Configure an rclone remote endpoint (
rclone config)
2a. (recommended, but optional) Set up an encrypted endpoint using the previously added remote. Be sure to either record and save your password and salt strings or save a copy of the generated rclone.conf file - Create a backups directory (not within one of the directories to be included in the backups) (
sudo mkdir /backups) - Save the script below as backup.sh (
sudo vi /backups/backup.sh) - (recommended, but optional) Move the rclone.conf file into your backup directory (
sudo mv ~/.config/rclone/rclone.conf /backups/rclone.conf) - Create and add whichever directories are to be included/excluded to the respective files.
6a.sudo touch /backups/backup-include.txt /backups/backup-exclude.txt
6b. Add all directories to be included in the backup to backup-include.txt
6c. Add all directories to be excluded (whoās parent directory is within the inclusion list) from the backup to backup-exclude.txt - Edit the script with your information (path to rclone config file, rclone remote name, backup path within remote)
- Change permissions so only root has access to your rclone conf file and can run backups (
sudo chown -R root:root /backups/ && sudo chmod -R 700 /backups/) - (recommended, but optional) Give the script a test run to make sure everything runs correctly and check remote storage to make sure the backup was transferred sucessfully (
sudo su -c 'cd /backups && ./backup.sh) - If successful, add to root crontab using desired backup interval (ex. to run every day at midnight, add
0 0 * * * cd /backups && ./backup.sh)
The Script
#!/bin/bash
# Automated Tarball Backups via Rclone by Mason, 13 Nov 19
# Tested on Ubuntu 18.04, GNU tar v1.29
# Note: rclone needs to be installed and remote endpoint configured
HOSTNAME=`hostname` # replace with desired hostname, if not set
RCLONE_CONF="/backups/rclone.conf" # path to rclone config file
RCLONE_REMOTE="my-remote-crypt" # rclone remote name to store backup
RCLONE_REMOTE_PATH="/backups/$HOSTNAME/" # path to store tarball on remote
NAME=$HOSTNAME-backup-`date +%d%b%y`.tar.gz
EXCLUDE_STRING=( )
# make sure include and exclude files exist
touch backup-include.txt backup-exclude.txt
# build "--exclude" options using list of directories to ignore
# (only needed if parent dir of the excluded dir is being included)
while IFS="" read -r p || [ -n "$p" ]
do
EXCLUDE_STRING=( "${EXCLUDE_STRING[@]}" --exclude "$p" )
done < backup-exclude.txt
# captures error code to prevent some non-error warnings from
# killing the script prematurely (i.e. "file changed as we read it")
set +e
tar -cvzf $NAME "${EXCLUDE_STRING[@]}" -T backup-include.txt
exitcode=$?
if [ "$exitcode" != "1" ] && [ "$exitcode" != "0" ]; then
exit $exitcode
fi
set -e
# upload backup to rclone remote and remove file
rclone --config=$RCLONE_CONF copy $NAME $RCLONE_REMOTE:$RCLONE_REMOTE_PATH
rm $NAME
It has built-in support for cloud storage and it supports deduplication. Itās made my life easier 
