Multi Source Replication MySQL 5.6 to 5.7 GTID Auto Position Issues

Question

I have 3 master servers, different DBs, I am trying to replicate into a single server. I am having a hard time getting them setup and current. I have Duplicate Entry errors on all 3 Channels. Skipping them manually is painful to say the least. Is there a way to auto sync to the correct position? I was under the impression that this was easy as pie with GTID.

I used:

Dump:

mysqldump --databases profiles --single-transaction --triggers --routines --host=10.10.10.10 --port=3306 --user=user --password=pass > ~/dump.sql

Initialize:

CHANGE MASTER TO MASTER_HOST="10.10.10.10", MASTER_PORT=3306, MASTER_USER="user", MASTER_PASSWORD="pass", MASTER_AUTO_POSITION=1 FOR CHANNEL "channel1";

Master My.cnf:

gtid_mode       = ON
enforce_gtid_consistency        = true
log_bin = /var/log/mysql/bin_log.index
log_slave_updates       = true
server-id       = 2061

Slave My.cnf:

  [client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock

[mysqld_safe]
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
nice            = 0

[mysqld]
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
server-id       = 10001
explicit_defaults_for_timestamp

gtid_mode=ON
enforce_gtid_consistency=true
log_bin=/var/log/mysql/bin_log.index
log_slave_updates=true
master_info_repository=TABLE
relay_log_info_repository=TABLE


# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address   = 127.0.0.1

log-error       = /var/log/mysql/error.log

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

Am I missing something? Any help is appreciated.

Answer 1

The problem was that GLOBAL.GTID_PURGED had only registered one of my master databases, which sets the proper position from which to continue the replication process. The other databases were starting from the beginning of time essentially. Thus the improper positioning and the Duplicate record errors I was receiving.

So this was my solution:

MySQL Multi-Source GTID Replication Guide by ME:

Re/Install MySQL Server:

After securing the current data I performed a clean install of MySQL Server 5.7.7-rc onto the slave server. Any MySQL Server can be completely removed using the following:

$ sudo apt-get --purge remove mysql-client mysql-server mysql-common
$ sudo apt-get autoremove

and then selecting YES at the prompt to remove the "Data" directory (This will permanently delete all of your databases, configurations, etc.). If you have any custom configurations, now is the time to backup your /etc/mysql/my.cnf file.

Reinstall MySQL Server 5.7 for Ubuntu 14-lts

$ sudo apt-get install mysql-server-5.7

If you don't have 5.7 on your system use this guide

Backup Live Master Databases:

I created a current MySQL Dump of all 3 Live Master databases. Each of my databases have a different name e.g. db01, db02, db03 and they are being saved directly onto the slave server.

In my case each database is on its own server, so I ran this a few times changing the ip and database, and filename.

$ mysqldump -u username -p -h 10.10.10.10 --skip-lock-tables --single-transaction --triggers --routines --databases db01 > ~/dumpDB01.sql

Once complete, you will need the GTID_PURGED data from each dump and save it for later:

$ grep PURGED ~/dumpDB01.sql
SET @@GLOBAL.GTID_PURGED='d23dceda-08a4-11e5-85e4-005056a2431f:1-10073';

You will need this entire string: d23dceda-08a4-11e5-85e4-005056a2431f:1-10073

MySQL Slave Server Configuration:

Now I decided to completely configure mysql before I ever imported any data and I will explain why shortly.

Edit my.cnf:

sudo vi /etc/mysql/my.cnf

gtid_mode                             =ON
enforce_gtid_consistency              =true
log_bin                               =/var/log/mysql/bin_log.index
log_slave_updates                     =true
master_info_repository                =TABLE
relay_log_info_repository             =TABLE
server-id                             =1001

# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address   = 127.0.0.1

You will need to save the configuration file and create the bin_log.index file, otherwise the server will not start.

$ sudo touch /var/log/mysql/bin_log.index
$ sudo chown mysql:mysql /var/log/mysql/bin_log.index
$ sudo service mysql restart

Import / Configure Data:

While importing the dumps, the first one will register its GTID_PURGED automatically like this:

SET @@GLOBAL.GTID_PURGED='d23dceda-08a4-11e5-85e4-005056a2431f:1-10073';

Once this happens your GTID_EXECUTED will be set with the same data, and each subsequent import will produce errors like this:

ERROR 1840 (HY000) at line 24: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty.

These errors can be ignored on import with $ mysql -u username -p -f < ~/dumpDB02.sql as we are going to modify the GTID_PURGED manually after the import process. If you have already forced the imports and have seen this error you can clear both GTID variables by executing mysql > RESET MASTER;

From the mysql console run the following:

mysql> RESET MASTER; 

You will need all 3 GTIDs from the dumps and comma separate them in the assignment below.

mysql> SET @@GLOBAL.GTID_PURGED='d23dceda-08a4-11e5-85e4-005056a2431f:1-10073,d23dceda-08a4-11e5-85e4-005056a2431f:1-10073,d23dceda-08a4-11e5-85e4-005056a2431f:1-10073';

You can now initialize and start replication:

For each database (in my case I changed the ip and channel is a string of your choice. You will use this channel to access the slave data):

mysql> CHANGE MASTER TO MASTER_HOST="10.10.10.10", MASTER_PORT=3306, MASTER_USER="username", MASTER_PASSWORD="password", MASTER_AUTO_POSITION=1 FOR CHANNEL "db01";

Then start each slave:

mysql> START SLAVE FOR CHANNEL "db01";
mysql> SHOW SLAVE STATUS FOR CHANNEL "db01"\G

and success!

I have all of the data, no errors, and its now up to date with the Master Servers