MySQL Master-Master Replication

To set up a dual master replication with MySQL, only some modification must be done, it's very simple and there is a lot of howto's on the topic. This is my contribution with a basic setup and I've tried to gather common commands to maintain such solution.

The big picture :

The first step is to configure both mysql using the file my.cnf (usualy on /etc in GNU/Linux) :

on matisse (192.168.14.13), under the mysqld section we add the following :

server-id=1
master-host=192.168.14.3
master-user=repl
master-password=repl
master-port=3306
log-bin
binlog-do-db=replication
replicate-do-db=replication
auto_increment_increment      = 10
auto_increment_offset         = 1

and on picasso (192.168.14.3), under the same section :

server-id=2
master-host=192.168.14.13
master-user=repl
master-password=repl
master-port=3306
log-bin
binlog-do-db=replication
replicate-do-db=replication
auto_increment_increment      = 10
auto_increment_offset         = 2

explanations of these settings :

• server-id : this is an integer id helping to identify the server (must be unique in your replication farm!)
• master-host : specifies the ip/hostname of the MySQL acting as master for the current server
• master-user : specifies the user used to make the connection to the master
• master-password : is the user's password
• master-port : specifies on which port the master is listening
• log-bin : needed to start the binary logging process
• binlog-do-db : specifies on which databases the binary logging must be active (only those databases will be in the binary log)
• replicate-do-db : which database must be replicated by the server as slave.
• auto_increment_increment : controls the increment between successive AUTO_INCREMENT values.
• auto_increment_offset : determines the starting point for AUTO_INCREMENT column values.

The last two options are needed to prevent key collisions when using AUTO_INCREMENT columns, otherwise multiple masters may attempt to use the same AUTO_INCREMENT value when inserting rows.

optionally two extra options can be added :

• show-slave-auth-info : Display slave usernames and passwords in the output of SHOW SLAVE HOSTS on the master server.
• slave_compressed_protocol={0|1} : this option enable the use of compression for the slave/master protocol if both the slave and the master support it. The default is 0 (no compression). This accelerates the replication process

Creating the replication user on both nodes :

on miro :

mysql> grant replication slave, replication client on *.* to repl@"matisse" identified by "repl";
Query OK, 0 rows affected (0.00 sec)

mysql> show grants for repl@"matisse"; 
+--+
| Grants for repl@matisse                                                                                          |
+--+
| GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl'@'matisse' IDENTIFIED BY PASSWORD '5ec3db8f603fcb03' | 
+--+
1 row in set (0.00 sec)

on matisse :

mysql> grant replication slave, replication client on *.* to repl@"miro" identified by "repl";
Query OK, 0 rows affected (0.00 sec)

mysql> show grants for repl@"miro";
+---+
| Grants for repl@miro                                                                                          |
+---+
| GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl'@'miro' IDENTIFIED BY PASSWORD '5ec3db8f603fcb03' | 
+---+
1 row in set (0.00 sec)

Create the database (in this example the db is not existing) :

on the master (matisse)

mysql> create database replication;
Query OK, 1 row affected (0.05 sec)

mysql> show master status;
++++--+
| File               | Position | Binlog_Do_DB | Binlog_Ignore_DB |
++++--+
| matisse-bin.000004 |       98 | replication  |                  | 
++++--+
1 row in set (0.00 sec)

mysql> show slave status\G
*************************** 1. row ***************************
             Slave_IO_State: Connecting to master
                Master_Host: 10.0.0.2
                Master_User: repl
                Master_Port: 3306
              Connect_Retry: 16
            Master_Log_File: localhost-bin.000001
        Read_Master_Log_Pos: 310
             Relay_Log_File: matisse-relay-bin.000006
              Relay_Log_Pos: 98
      Relay_Master_Log_File: localhost-bin.000001
           Slave_IO_Running: No
          Slave_SQL_Running: Yes
            Replicate_Do_DB: replication
        Replicate_Ignore_DB: 
         Replicate_Do_Table: 
     Replicate_Ignore_Table: 
    Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
                 Last_Errno: 0
                 Last_Error: 
               Skip_Counter: 0
        Exec_Master_Log_Pos: 310
            Relay_Log_Space: 98
            Until_Condition: None
             Until_Log_File: 
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File: 
         Master_SSL_CA_Path: 
            Master_SSL_Cert: 
          Master_SSL_Cipher: 
             Master_SSL_Key: 
      Seconds_Behind_Master: NULL
1 row in set (0.00 sec)

on the slave :

mysql> show master status;
+++++
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+++++
| localhost-bin.000001 | 98 | replication | |
+++++
1 row in set (0.00 sec)

mysql> show master status\G

*************************** 1. row ***************************

           File: localhost-bin.000001
       Position: 98
   Binlog_Do_DB: replication

Binlog_Ignore_DB:
1 row in set (0.00 sec)

mysql> show slave status\G
*************************** 1. row ***************************
             Slave_IO_State: Waiting for master to send event
                Master_Host: 192.168.14.13
                Master_User: repl
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: matisse-bin.000004
        Read_Master_Log_Pos: 98
             Relay_Log_File: localhost-relay-bin.000006
              Relay_Log_Pos: 237
      Relay_Master_Log_File: matisse-bin.000004
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB: replication
        Replicate_Ignore_DB: 
         Replicate_Do_Table: 
     Replicate_Ignore_Table: 
    Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
                 Last_Errno: 0
                 Last_Error: 
               Skip_Counter: 0
        Exec_Master_Log_Pos: 98
            Relay_Log_Space: 237
            Until_Condition: None
             Until_Log_File: 
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File: 
         Master_SSL_CA_Path: 
            Master_SSL_Cert: 
          Master_SSL_Cipher: 
             Master_SSL_Key: 
      Seconds_Behind_Master: 0
1 row in set (0.00 sec)

Then, we create a table on the master (matisse) :

mysql> create table users (
    -> id int(4) auto_increment not null primary key,
    -> name varchar(15));
Query OK, 0 rows affected (0.01 sec)

on the slave (miro) :

mysql> use replication;
Database changed
mysql> show tables;
+---+
| Tables_in_replication |
+---+
| users                 | 
+---+
1 row in set (0.00 sec)

we can also check this in the status :

on the master :

mysql> show master status;
++++--+
| File               | Position | Binlog_Do_DB | Binlog_Ignore_DB |
++++--+
| matisse-bin.000004 |      258 | replication  |                  | 
++++--+
1 row in set (0.00 sec)

and on the slave :

mysql> show slave status\G
*************************** 1. row ***************************
             Slave_IO_State: Waiting for master to send event
                Master_Host: 192.168.14.13
                Master_User: repl
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: matisse-bin.000004
        Read_Master_Log_Pos: 258
             Relay_Log_File: localhost-relay-bin.000006
              Relay_Log_Pos: 397
      Relay_Master_Log_File: matisse-bin.000004
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB: replication
        Replicate_Ignore_DB: 
         Replicate_Do_Table: 
     Replicate_Ignore_Table: 
    Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
                 Last_Errno: 0
                 Last_Error: 
               Skip_Counter: 0
        Exec_Master_Log_Pos: 258
            Relay_Log_Space: 397
            Until_Condition: None
             Until_Log_File: 
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File: 
         Master_SSL_CA_Path: 
            Master_SSL_Cert: 
          Master_SSL_Cipher: 
             Master_SSL_Key: 
      Seconds_Behind_Master: 0
1 row in set (0.00 sec)

And now we can create records. The first scenario is the following :

1) add 2 records on the master
2) add 2 records on the slave

on the master :

mysql> insert into users values (0,"fred"),(0,"vanne");
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

on the slave :

mysql> select * from users;
++---+
| id | name  |
++---+
|  1 | fred  | 
| 11 | vanne | 
++---+
2 rows in set (0.00 sec)

mysql> insert into users values (0,"raskas"),(0,"stintel");
Query OK, 2 rows affected (0.00 sec) 
Records: 2  Duplicates: 0  Warnings: 0

on the master :

mysql> select * from users;
++-+
| id | name    |
++-+
|  1 | fred    | 
| 11 | vanne   | 
| 12 | raskas  | 
| 22 | stintel | 
++-+

Good ! Our first master-master replication is working !

Now let stop the slave and add 2 extar records on the master :

mysql> slave stop;
Query OK, 0 rows affected (0.00 sec

on the master :

mysql> insert into users values (0,"sdog"),(0,"jacke");
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

and on the slave :

mysql> select * from users;
++-+
| id | name    |
++-+
|  1 | fred    | 
| 11 | vanne   | 
| 12 | raskas  | 
| 22 | stintel | 
++-+
4 rows in set (0.00 sec)

let's try to add some data on the slave too (the databases are not synchronized, but matisse is still slave of miro) :

on miro (the slacve):

mysql> insert into users values (0,"jozzel"),(0,"pleemans");
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

mysql> select * from users;
++--+
| id | name     |
++--+
|  1 | fred     | 
| 11 | vanne    | 
| 12 | raskas   | 
| 22 | stintel  | 
| 32 | jozzel   | 
| 42 | pleemans | 
++--+
6 rows in set (0.00 sec)

on matisse (the master) :

mysql> select * from users;
++--+
| id | name     |
++--+
|  1 | fred     | 
| 11 | vanne    | 
| 12 | raskas   | 
| 22 | stintel  | 
| 31 | sdog     | 
| 41 | jacke    | 
| 32 | jozzel   | 
| 42 | pleemans | 
++--+
8 rows in set (0.00 sec)

now let's start slave again on miro :

mysql> slave start;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from users;
++--+
| id | name     |
++--+
|  1 | fred     | 
| 11 | vanne    | 
| 12 | raskas   | 
| 22 | stintel  | 
| 32 | jozzel   | 
| 42 | pleemans | 
| 31 | sdog     | 
| 41 | jacke    | 
++--+
8 rows in set (0.00 sec)

Ok this is great an works like expected.

For our last test we gonna add some new records on both nodes but previously we gonna stop both slave process :

on matisse :

mysql> insert into users values (0,"alain"),(0,"tux");
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

mysql> select * from users;
++--+
| id | name     |
++--+
|  1 | fred     | 
| 11 | vanne    | 
| 12 | raskas   | 
| 22 | stintel  | 
| 31 | sdog     | 
| 41 | jacke    | 
| 32 | jozzel   | 
| 42 | pleemans | 
| 51 | alain    | 
| 61 | tux      | 
++--+
10 rows in set (0.00 sec)

on miro :
(I will even remove a record)

mysql> insert into users values (0,"billou"),(0,"Linus");
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

mysql> delete from users where id=22;
Query OK, 1 row affected (0.02 sec)

mysql> select * from users;
++--+
| id | name     |
++--+
|  1 | fred     | 
| 11 | vanne    | 
| 12 | raskas   | 
| 32 | jozzel   | 
| 42 | pleemans | 
| 31 | sdog     | 
| 41 | jacke    | 
| 52 | billou   | 
| 62 | Linus    | 
++--+
9 rows in set (0.00 sec)

let's restart both slaves :

from matisse :

mysql> select * from users;
++--+
| id | name     |
++--+
|  1 | fred     | 
| 11 | vanne    | 
| 12 | raskas   | 
| 31 | sdog     | 
| 41 | jacke    | 
| 32 | jozzel   | 
| 42 | pleemans | 
| 51 | alain    | 
| 61 | tux      | 
| 52 | billou   | 
| 62 | Linus    | 
++--+
11 rows in set (0.00 sec)

from miro :

mysql> select * from users;
++--+
| id | name     |
++--+
|  1 | fred     | 
| 11 | vanne    | 
| 12 | raskas   | 
| 51 | alain    | 
| 32 | jozzel   | 
| 42 | pleemans | 
| 31 | sdog     | 
| 41 | jacke    | 
| 52 | billou   | 
| 62 | Linus    | 
| 61 | tux      | 
++--+
11 rows in set (0.00 sec)

and what happens if we try do delete the same record when both slaves are stopped ?

on both :

mysql> slave stop;
Query OK, 0 rows affected (0.00 sec)

mysql> delete from users where id =52;
Query OK, 1 row affected (0.00 sec)

mysql> slave start;
Query OK, 0 rows affected (0.01 sec)

mysql> select * from users;
++--+
| id | name     |
++--+
|  1 | fred     | 
| 11 | vanne    | 
| 12 | raskas   | 
| 31 | sdog     | 
| 41 | jacke    | 
| 32 | jozzel   | 
| 42 | pleemans | 
| 51 | alain    | 
| 61 | tux      | 
| 62 | Linus    | 
++--+
10 rows in set (0.00 sec)

mysql> show slave status\G
*************************** 1. row ***************************
             Slave_IO_State: Waiting for master to send event
                Master_Host: 192.168.14.3
                Master_User: repl
                Master_Port: 3306
              Connect_Retry: 16
            Master_Log_File: localhost-bin.000002
        Read_Master_Log_Pos: 851
             Relay_Log_File: matisse-relay-bin.000009
              Relay_Log_Pos: 344
      Relay_Master_Log_File: localhost-bin.000002
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB: replication
        Replicate_Ignore_DB: 
         Replicate_Do_Table: 
     Replicate_Ignore_Table: 
    Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
                 Last_Errno: 0
                 Last_Error: 
               Skip_Counter: 0
        Exec_Master_Log_Pos: 851
            Relay_Log_Space: 344
            Until_Condition: None
             Until_Log_File: 
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File: 
         Master_SSL_CA_Path: 
            Master_SSL_Cert: 
          Master_SSL_Cipher: 
             Master_SSL_Key: 
      Seconds_Behind_Master: 0
1 row in set (0.00 sec)

cool ! no errors and everything seems to be as it should be.

Some maintenance :

it's also nice to know some maintenance commands.
the first dba replication action will be to manage the binary logs that could increase very quickly.

to see the logs use the following command :

mysql> show master logs;
++---+
| Log_name           | File_size |
++---+
| matisse-bin.000001 |       381 | 
| matisse-bin.000002 |       117 | 
| matisse-bin.000003 |       219 | 
| matisse-bin.000004 |       427 | 
| matisse-bin.000005 |       739 | 
++---+
5 rows in set (0.00 sec)

Then after having done a backup (long silence), you can purge then since a date (could be a day in the past or even now :

mysql> purge master logs before now();
Query OK, 0 rows affected (0.00 sec)

mysql> show master logs;
++---+
| Log_name           | File_size |
++---+
| matisse-bin.000005 |       739 | 
++---+
1 row in set (0.00 sec)

or

mysql> show master logs;
++-+
| Log_name             | File_size |
++-+
| localhost-bin.000001 |       271 | 
| localhost-bin.000002 |       851 | 
++-+
2 rows in set (0.00 sec)

mysql> purge master logs to 'localhost-bin.000002';
Query OK, 0 rows affected (0.00 sec)

mysql> show master logs;
++-+
| Log_name             | File_size |
++-+
| localhost-bin.000002 |       851 | 
++-+
1 row in set (0.00 sec)

To reset the slave and the master you should use :

reset master;

reset slave;

Next time I'll discuss about a case study on migrating replicated data on none synchronized tables.