Backup Rollback

Jarland · January 9, 2019, 6:12am

Hey friends,

I just had to roll the site back 5 hours. From what I can tell, maintenance on the hypervisor led to a corrupted file system on the main instance. I tried to recover it, but it was too far gone. I restored the latest backup and we’re running clean again

Jarland

Andrei · January 9, 2019, 9:30am

Setup a database slave for just such cases, you’ll have an up to date backup ready to go at any moment. I’d imagine a few bucks instance should be able to take it.

redbamboohouse · January 9, 2019, 10:11am

As someone who’s always been bad with databases in general, I’m listening on how would you implement it

Am I correct to assume that you’re referring to something like a live/real-time slave database?

Andrei · January 9, 2019, 10:17am

Yep, a regular good ol’ slave database, that gets replicated in real time from the master database.

But just as with RAID1, this should not be confused with a backup solution on its own. It will only come in handy for situations such as this one, where the master suffers some sort of hardware issue, so you have an identical copy of the data ready to go.

Mason · January 9, 2019, 12:11pm

Jarland · January 9, 2019, 3:57pm

I’ll admit I’m not quite as good with postgres as I am with mysql. Added layer of frustration is that discourse is packaged into docker containers now in a very tightly controlled setup. It’s actually a great setup and I’m quite thankful for the ease of management, but it makes redundancy a much larger task.

Mason · January 9, 2019, 4:25pm

Looks like Discourse has the db slave capability built in

github.com

discourse/discourse/blob/main/config/discourse_defaults.conf#L56


      
          # username accessing database
          db_username = discourse
          
          
# password used to access the db
          db_password =
          
          
# Disallow prepared statements
          # see: https://github.com/rails/rails/issues/21992
          db_prepared_statements = false
          
          
# host address for db replica server
          db_replica_host =
          
          
# port running replica db server, defaults to 5432 if not set
          db_replica_port =
          
          
db_advisory_locks = true
          
          
# hostname running the forum
          hostname = "www.example.com"

Unsure if this is modifiable after everything is already installed and initialized or if it can be added after-the-fact.

Neoon · January 9, 2019, 4:37pm

Galera is good stuff.

Andrei · January 9, 2019, 8:48pm

An alternative could be R1Soft continuous data protection. It would be a bit more expensive, considering R1Soft has been a b*tch lately and keeps increasing the license prices… but all in all, I can’t see HB incremental backups lasting more than maybe a few seconds each, you could virtually back it up every 10 minutes.

FHR · January 10, 2019, 12:01pm

Plain old simple Linux tar can do incremental backups as well.
tar --listed-incremental is an awesome thing.