How do I do backup and (point-in-time) recovery for etcd?

Question

I understand that etcd stores its data in some kind of distributed log.

Are there tools to back up these logs and do recovery from them? Does that also support point-in-time recovery (and how far back can we go)?

Extra points: do these logs also contain audit trails (i.e. who updated the data, assuming client authentication is in place)?

Answer 1

I think this may answer your question: https://labs.consol.de/kubernetes/2018/05/25/kubeadm-backup.html

It does automated backup of etcd as a Cronjob, so if you have it backup once an hour, that'd be like point in time recovery?

You could also use Heptio ARK on a schedule to do similar I believe. https://www.youtube.com/watch?v=qRPNuT080Hk

Answer 2

One approach is to read all existing keys into a file, then use that file by reading in and adding all keys to restore. There is an open source solution that takes that approach here https://github.com/fanhattan/etcd-backup. I don't think audit trails are included.

Answer 3

If you want to take etcd backup in JSON format that you can easily restore to any cluster, you can use this simple backup and restore utility: simple-etcd-backup-restore

This utility makes the backup and restore process easier and faster. You can also compress/decompress the backup file and upload/download it on/from AWS S3 bucket. The size of backup files also gets smaller. Data in the backup files stays readable as well.