Postgresql: How to take incremental backup with wal-e

Question

I am new to postgresql. I've connected to my demo database with psycopg2 and inserted data with python faker. I want to take incremental backup with the tool WAL-E and want to see how it actually works. But the few tutorials aren't helping that much as they're not for naive users.

Can someone help explaining the steps simply how to backup data with WAL-E? That would be a great help ! I'm using psql 10.4 in Ubuntu 18.04.

Thanks

Answer 1

Install Python & Dependencies

apt-get install python-dev python-virtualenv libevent-dev pv lzop daemontools
sudo apt-get install -y python3-pip
sudo apt-get install build-essential libssl-dev libffi-dev
sudo apt-get install -y python3-venv

Create & Activate Virtual Environment

Sudo python3 -m venv /opt/wal-e
source /opt/wal-e/bin/activate

Install Wal-e

sudo /opt/wal-e/bin/pip install wal-e[aws]

Setup Environment Variables

umask u=rwx,g=rx,o=
mkdir -p /etc/wal-e.d/env
sudo -i -u root
echo "Your AWS Secret Key" > /etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
echo "Your AWS Access Key" > /etc/wal-e.d/env/AWS_ACCESS_KEY_ID
echo 'Your AWS Region > /etc/wal-e.d/env/AWS_REGION
echo 's3://some-bucket/directory/or/whatever' > /etc/wal-e.d/env/WALE_S3_PREFIX
sudo chown -R root:postgres /etc/wal-e.d

Change Postgres Config to Activate Archiving

sudo nano /etc/postgresql/10/main/postgresql.conf
wal_level = replica
archive_mode = on
archive_command = '/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-push %p'
archive_timeout = 60
sudo systemctl restart postgresql

Test Wal-e is Working

sudo -i -u postgres
/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e backup-list

Now Push Your First Base Backup to S3

/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e backup-push /var/lib/postgresql/10/main

Now Set the Cron For Regular Backup

sudo -i -u postgres 

mkdir logs
touch backup.log
touch deletebackup.log

crontab -e

00 00 * * * /usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e backup-push /var/lib/postgresql/10/main > ~/logs/backup.log 2>&1
00 00 * * * /usr/bin/envdir /etc/wal-e.d/env  /opt/wal-e/bin/wal-e delete --confirm retain 1 > ~/logs/deletebackup.log 2>&1