# Automated Backups via Rclone (Cloud Storage: GDrive, S3, Dropbox, FTP, +More)

**URL:** <https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406>\
**Category:** Wiki\
**Created:** [November 14, 2019, 1:34pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406 "2019-11-14T13:34:56Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mason](https://hostballs.com/user_avatar/hostballs.com/mason/32/66_2.png) [@Mason](https://hostballs.com/u/Mason)\
**Post date:** [November 14, 2019, 1:34pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/1 "2019-11-14T13:34:56Z")

</div>

**Automated Tarball Backups via Rclone**

**Goal** : Automatically compress select directories and upload tarball to remote storage via rclone.

Sharing my small script that I finalized, which may come in handy for others. Prior to doing automated backups, I typically backed up important data manually, as needed. This has come to bite me in the ass on a few occasions with not having the latest backups or not having what I really needed in my latest backup.

The gist of the script is that you make two files, the _backup-include.txt_ list and the _backup-exclude.txt_ list. Everything that’s included in the include list (one directory per line) is compressed and added to the tarball. Everything that’s included in the exclude list is blocked from being added to the tarball. An entry to the exclusion list is only necessary if the parent directory is in the included list, but you don’t really want everything that’s in that directory to be backed up. For example, if I have “/home/mason/hostballs-files” as a line in the inclusion file, but I don’t want to back up Jarland’s cat videos, then I’d add “/home/mason/hostballs-files/jars-kitty-vids” to the exclusion file. Thus, the entirety of the hostballs-files directory will be in the tarball, with the exception of Jarland’s cat collection.

**How to Setup/Run**

1. Install [rclone](https://rclone.org/) on your server
2. Configure an rclone remote endpoint (`rclone config`)  
2a. (recommended, but optional) Set up an [encrypted endpoint](https://rclone.org/crypt/) using the previously added remote. Be sure to either record and save your password and salt strings or save a copy of the generated rclone.conf file
3. Create a backups directory (not within one of the directories to be included in the backups) (`sudo mkdir /backups`)
4. Save the script below as _backup.sh_ (`sudo vi /backups/backup.sh`)
5. (recommended, but optional) Move the _rclone.conf_ file into your backup directory (`sudo mv ~/.config/rclone/rclone.conf /backups/rclone.conf`)
6. Create and add whichever directories are to be included/excluded to the respective files.  
6a. `sudo touch /backups/backup-include.txt /backups/backup-exclude.txt`  
6b. Add all directories to be included in the backup to _backup-include.txt_  
6c. Add all directories to be excluded (who’s parent directory is within the inclusion list) from the backup to _backup-exclude.txt_
7. Edit the script with your information (path to rclone config file, rclone remote name, backup path within remote)
8. Change permissions so only root has access to your rclone conf file and can run backups (`sudo chown -R root:root /backups/ && sudo chmod -R 700 /backups/`)
9. (recommended, but optional) Give the script a test run to make sure everything runs correctly and check remote storage to make sure the backup was transferred sucessfully (`sudo su -c 'cd /backups && ./backup.sh`)
10. If successful, add to root crontab using desired backup interval (ex. to run every day at midnight, add `0 0 * * * cd /backups && ./backup.sh`)

**The Script**

```sh
#!/bin/bash
# Automated Tarball Backups via Rclone by Mason, 13 Nov 19
# Tested on Ubuntu 18.04, GNU tar v1.29
# Note: rclone needs to be installed and remote endpoint configured

HOSTNAME=`hostname` # replace with desired hostname, if not set
RCLONE_CONF="/backups/rclone.conf" # path to rclone config file
RCLONE_REMOTE="my-remote-crypt" # rclone remote name to store backup
RCLONE_REMOTE_PATH="/backups/$HOSTNAME/" # path to store tarball on remote
NAME=$HOSTNAME-backup-`date +%d%b%y`.tar.gz
EXCLUDE_STRING=( )

# make sure include and exclude files exist
touch backup-include.txt backup-exclude.txt

# build "--exclude" options using list of directories to ignore
# (only needed if parent dir of the excluded dir is being included)
while IFS="" read -r p || [-n "$p"]
do
        EXCLUDE_STRING=( "${EXCLUDE_STRING[@]}" --exclude "$p" )
done < backup-exclude.txt

# captures error code to prevent some non-error warnings from
# killing the script prematurely (i.e. "file changed as we read it")
set +e
tar -cvzf $NAME "${EXCLUDE_STRING[@]}" -T backup-include.txt
exitcode=$?
if ["$exitcode" != "1"] && ["$exitcode" != "0"]; then
        exit $exitcode
fi
set -e

# upload backup to rclone remote and remove file
rclone --config=$RCLONE_CONF copy $NAME $RCLONE_REMOTE:$RCLONE_REMOTE_PATH
rm $NAME

```

---

<div class="post-metadata">

**Author:** ![turbo](https://hostballs.com/letter_avatar/turbo/32/5_5575768a8748004e209b776fc1b2916d.png) [@turbo](https://hostballs.com/u/turbo)\
**Post date:** [November 14, 2019, 3:29pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/2 "2019-11-14T15:29:22Z")

</div>

You posted this right as I was about to start backing up one of my servers lol

---

<div class="post-metadata">

**Author:** ![Mason](https://hostballs.com/user_avatar/hostballs.com/mason/32/66_2.png) [@Mason](https://hostballs.com/u/Mason)\
**Post date:** [November 14, 2019, 3:39pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/3 "2019-11-14T15:39:28Z")

</div>

Hehe perfect timing, then! Hope it helps, give me a shout if you have any questions.

It’d also be trivial to swap out rclone with some other method of transferring the backup (such as, rsync after setting up SSH certs to keep things automated), if rclone isn’t your cup of tea 🙂

---

<div class="post-metadata">

**Author:** ![Wolveix](https://hostballs.com/user_avatar/hostballs.com/wolveix/32/19370_2.png) [@Wolveix](https://hostballs.com/u/Wolveix)\
**Post date:** [November 14, 2019, 3:45pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/4 "2019-11-14T15:45:12Z")

</div>

Nice writeup @Mason 🙂 I used to use a script similar to this, then I discovered [Duplicati](https://duplicati.readthedocs.io/en/latest/) 😛 It has built-in support for cloud storage and it supports deduplication. It’s made my life easier 😅

---

<div class="post-metadata">

**Author:** ![WSS](https://hostballs.com/letter_avatar/wss/32/5_5575768a8748004e209b776fc1b2916d.png) [@WSS](https://hostballs.com/u/WSS)\
**Post date:** [November 15, 2019, 4:57am UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/5 "2019-11-15T04:57:17Z")

</div>

It’s like bourne threw up a little bit in my mouth.

---

<div class="post-metadata">

**Author:** ![flips](https://hostballs.com/user_avatar/hostballs.com/flips/32/5647_2.png) [@flips](https://hostballs.com/u/flips)\
**Post date:** [November 15, 2019, 9:53am UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/6 "2019-11-15T09:53:40Z")

</div>

Looks nice. What does `rclone` bring to the table? (I’ve mostly used `rsync` …)

Disclaimer: Yep, I was to lazy/busy to Google it right now 😇

---

<div class="post-metadata">

**Author:** ![Mason](https://hostballs.com/user_avatar/hostballs.com/mason/32/66_2.png) [@Mason](https://hostballs.com/u/Mason)\
**Post date:** [November 15, 2019, 12:26pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/7 "2019-11-15T12:26:28Z")

</div>

rclone is self-described as “the rsync for cloud storage”. Best way to view it is as rsync, but with a wrapper enabling it to talk/sync with a ton of cloud storage options/protocols (i.e. 1fichier, amazon drive/s3, b2, box, dropbox, ftp, gdrive, mega, own/nextcloud, wasabi, webdav, and [a bunch more](https://rclone.org)).

As I stated earlier, though, if you keep backups local/just move them to other machines, then the rclone line can be swapped out in favor of rsync within the script.

---

<div class="post-metadata">

**Author:** ![Mason](https://hostballs.com/user_avatar/hostballs.com/mason/32/66_2.png) [@Mason](https://hostballs.com/u/Mason)\
**Post date:** [November 15, 2019, 12:28pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/8 "2019-11-15T12:28:22Z")

</div>

Heh. Only “bourne again” shellers would understand.

---

<div class="post-metadata">

**Author:** ![WSS](https://hostballs.com/letter_avatar/wss/32/5_5575768a8748004e209b776fc1b2916d.png) [@WSS](https://hostballs.com/u/WSS)\
**Post date:** [November 15, 2019, 6:22pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/9 "2019-11-15T18:22:17Z")

</div>

I actually prefer zsh, but I use bash these days just because it’s always there. 🤷‍♂️

---

<div class="post-metadata">

**Author:** ![anon40039896](https://hostballs.com/letter_avatar/anon40039896/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon40039896](https://hostballs.com/u/anon40039896)\
**Post date:** [November 15, 2019, 6:49pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/10 "2019-11-15T18:49:33Z")

</div>

> [@Mason](#):
>
> Only “bourne again” shellers would understand.

The saying “Bourne Again” was born after a couple of sequels of Jason Bourne.

---

<div class="post-metadata">

**Author:** ![angstrom](https://hostballs.com/user_avatar/hostballs.com/angstrom/32/1204_2.png) [@angstrom](https://hostballs.com/u/angstrom)\
**Post date:** [November 15, 2019, 7:36pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/11 "2019-11-15T19:36:34Z")

</div>

> [@anon40039896](#):
>
> The saying “Bourne Again” was born after a couple of sequals of Jason Bourne.

I think that @Mason was alluding to the fact that “bash” was originally an abbreviation for “Bourne again shell”. In this case, the Bourne was Stephen, not Jason.

---

<div class="post-metadata">

**Author:** ![Mason](https://hostballs.com/user_avatar/hostballs.com/mason/32/66_2.png) [@Mason](https://hostballs.com/u/Mason)\
**Post date:** [November 15, 2019, 7:58pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/12 "2019-11-15T19:58:10Z")

</div>

Yeah, I was trying to make a ‘Born Again’ Christian joke (but using bash’s “Bourne”), but it didn’t come out too well 😛

---

<div class="post-metadata">

**Author:** ![angstrom](https://hostballs.com/user_avatar/hostballs.com/angstrom/32/1204_2.png) [@angstrom](https://hostballs.com/u/angstrom)\
**Post date:** [November 15, 2019, 8:20pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/13 "2019-11-15T20:20:51Z")

</div>

> [@Mason](#):
>
> Yeah, I was trying to make a ‘Born Again’ Christian joke (but using bash’s “Bourne”), but it didn’t come out too well 😛

Indeed, it was a play on (the Christian) “Born again” that was at the origin of bash as the “Bourne again shell”. 🙂

---

<div class="post-metadata">

**Author:** ![anon40039896](https://hostballs.com/letter_avatar/anon40039896/32/5_5575768a8748004e209b776fc1b2916d.png) [@anon40039896](https://hostballs.com/u/anon40039896)\
**Post date:** [November 15, 2019, 8:22pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/14 "2019-11-15T20:22:52Z")

</div>

> [@angstrom](#):
>
> the Bourne was Stephen, not Jason

Seriously… you didn’t get my joke

---

<div class="post-metadata">

**Author:** ![angstrom](https://hostballs.com/user_avatar/hostballs.com/angstrom/32/1204_2.png) [@angstrom](https://hostballs.com/u/angstrom)\
**Post date:** [November 15, 2019, 8:28pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/15 "2019-11-15T20:28:40Z")

</div>

> [@anon40039896](#):
>
> Seriously… you didn’t get my joke

But I did get it! J. Bourne, J. Bourne sequel, J. Bourne sequel sequel, …

I was just pointing out that “Bourne again” was born much earlier.

(I thought that you had missed @Mason’s reference, but perhaps you hadn’t.)

---

<div class="post-metadata">

**Author:** ![Razza](https://hostballs.com/user_avatar/hostballs.com/razza/32/602_2.png) [@Razza](https://hostballs.com/u/Razza)\
**Post date:** [November 15, 2019, 8:29pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/16 "2019-11-15T20:29:50Z")

</div>

The script is helpful for automatic backup, I prefer using something like Borg since it support encryption and deduplication.

---

<div class="post-metadata">

**Author:** ![thagoat](https://hostballs.com/user_avatar/hostballs.com/thagoat/32/4431_2.png) [@thagoat](https://hostballs.com/u/thagoat)\
**Post date:** [November 15, 2019, 9:45pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/17 "2019-11-15T21:45:26Z")

</div>

+1 for zsh. Nice script btw @Mason

---

<div class="post-metadata">

**Author:** ![SERIESN](https://hostballs.com/user_avatar/hostballs.com/seriesn/32/10978_2.png) [@SERIESN](https://hostballs.com/u/SERIESN)\
**Post date:** [November 16, 2019, 12:52pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/18 "2019-11-16T12:52:26Z")

</div>

Every day, I fall in love with your scripts. Again and again.

---

<div class="post-metadata">

**Author:** ![aaronstuder](https://hostballs.com/user_avatar/hostballs.com/aaronstuder/32/5080_2.png) [@aaronstuder](https://hostballs.com/u/aaronstuder)\
**Post date:** [November 20, 2019, 8:22pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/19 "2019-11-20T20:22:19Z")

</div>

Would this work on Proxmox?

---

<div class="post-metadata">

**Author:** ![Mason](https://hostballs.com/user_avatar/hostballs.com/mason/32/66_2.png) [@Mason](https://hostballs.com/u/Mason)\
**Post date:** [November 20, 2019, 8:50pm UTC](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406/20 "2019-11-20T20:50:02Z")

</div>

Yes, however, it may need some tweaking depending on what you want to do.

I’m backing up a couple of my proxmox hosts with the following list of directories in the _backup-include.txt_ file. This is where most of the configuration files and proxmox setup is located.

```auto
/home/me
/etc/pve
/var/lib/pve-cluster
/var/lib/pve-firewall
/var/lib/pve-manager
/etc/dhcp
/etc/nginx/sites-available

```

If you also wish to backup VMs running on your proxmox host, then I’d suggest adding some lines to the end of the script to backup the VMs you care about (i.e. `vzdump {vm_id} --mode snapshot`). And then use rclone to sync/upload the backups to your remote destination (i.e. `rclone --config=$RCLONE_CONF copy /var/lib/vz/dump/* $RCLONE_REMOTE:$RCLONE_REMOTE_PATH/vms/`).

[Next page](https://hostballs.com/t/automated-backups-via-rclone-cloud-storage-gdrive-s3-dropbox-ftp-more/3406.md?page=2)