Feedback required for backup strategy on multiple low/medium traffic BOA managed sites

Summary

I'm working on a backup strategy for our SaaS distro that runs on BOA provisioned cloud vps nodes and I'm looking for feedback. I'll move all of this to a wiki page if it becomes useful and provide the scripts that are currently working for us, as it will be useful for most folk running multiple sites on a single Drupal core.

Use Case

Small to medium web development firm/Drupal hosting provider/Hosted SaaS Drupal distro with x number of BOA managed servers running multiple sites. The site owners are not involved in maintenance and system administration. Drupal core/distros are under version control and do not need to be backed up.

Requirements

1. automated
2. use Amazon S3 or other cheap providers for storage
3. maximum use of resources, read: low cost and small footprint
4. easy to restore/migrate by user with root access
5. encrypted back up of databases for each site and keeps 7 days of backups
6. backs up each of the sites in the /sites directory
7. easy to provision on multiple BOA managed servers

Strategy and [reasoning] for each requirement

1. Automation via a bash wrapper script run by cron. [Fairly easy to write, small footprint, immediately available on BOA servers]
2. S3 bucket for data storage. [Cheap, readily available tools like s3cmd, cheaper reduced redundancy option available, possibility of automatically providing access to backups for site owners (see unsolved below) via Aegir client UI and custom module.]
3. Back up databases and files incrementally. 7 days worth of databases and exact copy of the /sites/example.com directory. [For smaller Drupal sites, 7 days of db backups is sufficient. Small sites may have a large /files directory. By syncing a copy of the /sites/example.com directory elsewhere, minimal space is used for the backup as compared to several archived backups used in most backup strategies.]
4. Amazon IAM permissions, duplicity, GPG (Gnu Privacy Guard) key and s3cmd for restoring a backup. [These freely available and easy to use tools make it easy to restore to the same BOA managed server or another BOA managed server in minimal time.]
5. Drush, Duplicity and GPG for backing up databases. [Drush provides optimal database dumps in a single command for multi-sites. Duplicity encrypts and incrementally backs up the local database backup folder to S3.]
6. s3cmd provides an easy to use mechanism for syncing a local directory tree to S3 and vice versa. [For smaller sites, having one exact copy of the site stored off site with another service provider is sufficient. Most available backup solutions archive the entire /sites/example.com directory and keep several of these archives in storage. This method uses up too much disk space and bandwidth when the /files directory starts to grow in size.
7. wrapper bash scripts are used for restore and backups. [Cron is used to call the backup script daily, easy to provision on additional BOA servers, alias restore commands are easier to use than the full duplicity and s3cmd commands. eg. restore-site-db example.com local_path/example.com && restore example.com/ local_path/example.com/]

Current Status

We have the above solution running on our cloud vps nodes. Databases are backed up to /home/o1.ftp/drush-backups via drush @sites sql-dump command. 7 days of backups are stored locally and duplicity is used to encrypt the backups then store them in a proximal S3 bucket (one bucket for each vps node that exist in different data centres/continents). s3cmd syncs the /data/disk/o1/static/ourdistro/sites folder (ignoring the ../sites/all/* which we keep under version control) and deletes remote files when they no longer exist locally. The restore workflow is easy enough for anyone on our team to handle.

This is costing us around $2-$3 a month to backup a VPS node @ ~10 sites per node

Unsolved

1. BOA has an efficient script that backs up databases daily at 04:15 to /data/disk/arch/sql. The script also optimizes the db tables, truncates cache tables before backing up the db, restarts redis and fixes permissions. This means databases are backed up twice.
   • Should we disable part of this script?
   • Why is redis restarted?
   • Why are cache tables truncated?
   • Is it really good practice to optimize tables so often?
   • Should we rather tell duplicity to backup /data/disk/arch/sql and get rid of our present way of calling drush to backup dbs?
2. A way to provide a secure download of x number of db backups via the Aegir UI
3. Automated IAM user and bucket for every BOA vps provisioned and automatic configuration of the backup strategy. I have to manually create the IAM user and bucket then configure the backup script variables after provisioning a new VPS. Beyond the scope of this page, but ideas/clues welcome from anyone who's achieved this.