Second node of PXC dont join to cluster

1.3k views Asked by At

Im trying to setup PXC with 3 nodes, 1st node was bootstrapped succesfully but when trying to start second node it cant get SST. Logfile from 2nd node:

2020-10-07T13:17:16.480905Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address 'xxx.xxx.xxx.xxx:4444' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib/mysql/plugin/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '154359' --mysqld-version '8.0.20-11.1'  --binlog 'mysql-bin' )
2020-10-07T13:17:17.203042Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2020-10-07T13:17:17.642030Z 1 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|xxx.xxx.xxx.xxx:4444/xtrabackup_sst//1
2020-10-07T13:17:17.642609Z 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2020-10-07T13:17:17.643130Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2020-10-07T13:17:17.643718Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
2020-10-07T13:17:17.644031Z 1 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2020-10-07T13:17:17.644344Z 1 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 613f9455-07f0-11eb-9e01-139f2b6b4973
2020-10-07T13:17:17.644667Z 1 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 57, STRv: 3
2020-10-07T13:17:17.645190Z 1 [Note] [MY-000000] [Galera] IST receiver addr using ssl://xxx.xxx.xxx.xxx:4568
2020-10-07T13:17:17.645589Z 1 [Note] [MY-000000] [Galera] IST receiver using ssl
2020-10-07T13:17:17.646458Z 1 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-57, listening at: ssl://xxx.xxx.xxx.xxx:4568
2020-10-07T13:17:17.647629Z 0 [Warning] [MY-000000] [Galera] Member 1.0 (engine2) requested state transfer from 'engine3', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2020-10-07T13:17:17.648009Z 1 [Note] [MY-000000] [Galera] Requesting state transfer failed: -11(Resource temporarily unavailable). Will keep retrying every 1 second(s)
2020-10-07T13:17:18.651866Z 0 [Warning] [MY-000000] [Galera] Member 1.0 (engine2) requested state transfer from 'engine3', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2020-10-07T13:17:18.969089Z 0 [Note] [MY-000000] [Galera] (6b19a1b7, 'ssl://0.0.0.0:4567') turning message relay requesting off
2020-10-07T13:17:19.654067Z 0 [Warning] [MY-000000] [Galera] Member 1.0 (engine2) requested state transfer from 'engine3', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2020-10-07T13:18:57.356706Z 0 [Note] [MY-000000] [WSREP-SST] pigz: skipping: <stdin> empty
2020-10-07T13:18:57.360327Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** 
2020-10-07T13:18:57.363359Z 0 [ERROR] [MY-000000] [WSREP-SST] Possible timeout in receving first data from donor in gtid/keyring stage
2020-10-07T13:18:57.363393Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1108
2020-10-07T13:18:57.363412Z 0 [ERROR] [MY-000000] [WSREP-SST] ****************************************************** 
2020-10-07T13:18:57.363430Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32
2020-10-07T13:18:57.384013Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address 'xxx.xxx.xxx.xxx:4444' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib/mysql/plugin/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '154359' --mysqld-version '8.0.20-11.1'  --binlog 'mysql-bin' : 32 (Broken pipe)
2020-10-07T13:18:57.384488Z 0 [ERROR] [MY-000000] [WSREP] Failed to read uuid:seqno from joiner script.
2020-10-07T13:18:57.384541Z 0 [ERROR] [MY-000000] [WSREP] SST script aborted with error 32 (Broken pipe)
2020-10-07T13:18:57.385257Z 3 [Note] [MY-000000] [Galera] Processing SST received
2020-10-07T13:18:57.385338Z 3 [Note] [MY-000000] [Galera] SST request was cancelled
2020-10-07T13:18:57.385387Z 3 [ERROR] [MY-000000] [Galera] State transfer request failed unrecoverably: 32 (Broken pipe). Most likely it is due to inability to communicate with the cluster primary component. Restart required.
2020-10-07T13:18:57.385421Z 3 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()
2020-10-07T13:18:57.385453Z 3 [Note] [MY-000000] [Galera] Closing send monitor...
2020-10-07T13:18:57.385484Z 3 [Note] [MY-000000] [Galera] Closed send monitor.
2020-10-07T13:18:57.385519Z 3 [Note] [MY-000000] [Galera] gcomm: terminating thread
2020-10-07T13:18:57.385733Z 3 [Note] [MY-000000] [Galera] gcomm: joining thread
2020-10-07T13:18:57.385762Z 3 [Note] [MY-000000] [Galera] gcomm: closing backend
2020-10-07T13:18:57.945476Z 1 [ERROR] [MY-000000] [Galera] Requesting state transfer failed: -77(File descriptor in bad state)
2020-10-07T13:18:57.945563Z 1 [ERROR] [MY-000000] [Galera] State transfer request failed unrecoverably: 77 (File descriptor in bad state). Most likely it is due to inability to communicate with the cluster primary component. Restart required.
1

There are 1 answers

0
utdrmac On

A couple things could be causing this. 1) Make sure ports 4444, 4567, and 4568 are open between all nodes. 2) Make sure you have copied the SSL certificates from node1 over to node 2 BEFORE starting node2. Please read the PXC documentation on setting up new nodes.