PGBACKREST: Local backup works but remote backup to S3 FAILS

Question

I am setting up pgbackrest to save wal archives to S3. I have two machines. When I used to take backup on the localhost, I am able to save it to S3. However, when I initiate backup from 'backup server' it fails.

Backup server config:

[global]
repo1-path=/var/lib/pgbackrest
repo1-retention-full=1
process-max=2
log-level-console=info
log-level-file=debug
start-fast=y
#stop-auto=y

repo2-path=/repo_my1
repo2-type=s3
repo2-s3-endpoint=s3.amazonaws.com
repo2-s3-bucket=<bucket>
repo2-s3-verify-tls=n
repo2-s3-key=<key>
repo2-s3-key-secret=<secret>
repo2-s3-region=us-west-2
repo2-retention-full=1

[db-primary]
pg1-path=/var/lib/pgsql/11/data
pg1-host=db-primary-host
pg1-host-user=postgres

[global:archive-push]
compress-level=3

---

Primary server (postgres server)

[global]
repo1-host=backup-host
repo1-host-user=backrest
process-max=2
log-level-console=info
log-level-file=debug

[db-primary]
pg1-path=/var/lib/pgsql/11/data


#sudo -iu backrest pgbackrest --stanza=demo --log-level-console=info check

error:

2022-05-30 16:43:52.432 P00   INFO: check repo2 configuration (primary)
2022-05-30 16:43:54.494 P00   INFO: check repo1 archive for WAL (primary)
ERROR: [082]: WAL segment 00000001000000000000002B was not archived before the 60000ms timeout
HINT: check the archive_command to ensure that all options are correct (especially --stanza).
HINT: check the PostgreSQL server log for errors.
HINT: run the 'start' command if the stanza was previously stopped.
2022-05-30 16:44:54.802 P00   INFO: check command end: aborted with exception [082]

Answer 1

I setup pgbackrest and postgres as docker . when run command "pgbackrest --stanza=main check" on container pgbackrest encontered same error .

pgbackrest --stanza=main check
2023-03-24 14:06:52.203 P00   INFO: check command begin 2.44: --exec-id=330-9de8d587 --log-level-console=info --log-level-file=debug --pg1-host=postgres_container --pg1-path=/data/postgres --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest --stanza=main
2023-03-24 14:06:52.992 P00   INFO: check repo1 configuration (primary)
2023-03-24 14:06:53.194 P00   INFO: check repo1 archive for WAL (primary)
ERROR: [082]: WAL segment 00000001000000000000001A was not archived before the 60000ms timeout
       HINT: check the archive_command to ensure that all options are correct (especially --stanza).
       HINT: check the PostgreSQL server log for errors.
       HINT: run the 'start' command if the stanza was previously stopped.
2023-03-24 14:07:53.216 P00   INFO: check command end: aborted with exception [082]

when inspect logs container postgres . I saw logs

postgres_container | ERROR: [103]: unable to find a valid repository:
postgres_container |        repo1: [UnknownError] remote-0 process on 'pgbackrest' terminated unexpectedly [255]: pgbackrest@pgbackrest: Permission denied (publickey,password).
postgres_container | 2023-03-24 14:09:58.376 UTC [113] LOG:  archive command failed with exit code 103
postgres_container | 2023-03-24 14:09:58.376 UTC [113] DETAIL:  The failed archive command was: pgbackrest --stanza=main archive-push pg_wal/000000010000000000000001

then login to container postgresql . and in file config pgbackrest.conf in part [global] add line backup-user=postgres