One of my node was down and it was also the donor for the node A, so from node A config I removed the down node as donor. This is so that node A should not have to wait for the node to come up.
As the node that originally went down was on slow storage and slow network connection I decided to remove it. As I removed it and restarted the cluster even the node A didn't come up, on restart it started reporting error:
● mariadb.service - MariaDB 10.1.47 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2020-12-13 22:06:50 GMT; 12h ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 32653 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 17439 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_STA
Process: 17436 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 17432 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 32653 (code=exited, status=1/FAILURE)
CPU: 801ms
Dec 13 22:06:50 rockpi sh[17439]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/error-creating-innodb.html
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [ERROR] Plugin 'InnoDB' init function returned error.
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [Note] Plugin 'FEEDBACK' is disabled.
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [ERROR] Unknown/unsupported storage engine: innodb
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [ERROR] Aborting'
Dec 13 22:06:50 rockpi systemd[1]: mariadb.service: Control process exited, code=exited status=1
Dec 13 22:06:50 rockpi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 13 22:06:50 rockpi systemd[1]: Failed to start MariaDB 10.1.47 database server.
Why it started reporting Unknown/unsupported storage engine innodb now when it was working all this time?
Incidentally, I used another third node to bootstrap (start Galera_new_cluster).