Tag Archives: high availability

Topology refactoring using classic MySQL replication commands - version 2

This one in not very different than Topology refactoring using classic MySQL replication commands - version 1, actually it's the other way around.

Context:

  • Initial state:
    dbfoo1(M) - dbfoo2(S)
              \ dbfoo3(S)
  • Desired state:
    dbfoo1(M) - dbfoo2(MS) - dbrfoo3(S)
  • Let's get it done:

    1. stop slave on dbfoo3
    2. MariaDB [(none)]> stop slave;
    3. check the "freezing point" on dbfoo2 and note down the output from 'show' commands
    4. MariaDB [(none)]> stop slave; show master status\Gshow slave status\Gstart slave;
      
      *************************** 1. row ***************************
                  File: bin.009642
              Position: 91547140
          Binlog_Do_DB: 
      Binlog_Ignore_DB: 
      1 row in set (0.00 sec)
      
      *************************** 1. row ***************************
                     Slave_IO_State: 
                        Master_Host: dbfoo1
                        Master_User: user
                        Master_Port: 3306
                      Connect_Retry: 60
                    Master_Log_File: bin.293567
                Read_Master_Log_Pos: 90061272
                     Relay_Log_File: relay.028914
                      Relay_Log_Pos: 90061553
              Relay_Master_Log_File: bin.293567
                   Slave_IO_Running: No
                  Slave_SQL_Running: No
                    Replicate_Do_DB: 
                Replicate_Ignore_DB: 
                 Replicate_Do_Table: 
             Replicate_Ignore_Table: 
            Replicate_Wild_Do_Table: 
        Replicate_Wild_Ignore_Table: 
                         Last_Errno: 0
                         Last_Error: 
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 90061272
                    Relay_Log_Space: 90062173
                    Until_Condition: None
                     Until_Log_File: 
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File: 
                 Master_SSL_CA_Path: 
                    Master_SSL_Cert: 
                  Master_SSL_Cipher: 
                     Master_SSL_Key: 
              Seconds_Behind_Master: NULL
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 0
                      Last_IO_Error: 
                     Last_SQL_Errno: 0
                     Last_SQL_Error: 
        Replicate_Ignore_Server_Ids: 
                   Master_Server_Id: 270
                     Master_SSL_Crl: 
                 Master_SSL_Crlpath: 
                         Using_Gtid: No
                        Gtid_IO_Pos: 
      1 row in set (0.00 sec)
      
      Query OK, 0 rows affected (0.00 sec)
    5. make sure dbfoo3 has the same data as dbfoo2 and use Relay_Master_Log_File and Exec_Master_Log_Pos from show master status
    6. MariaDB [(none)]> start slave until master_log_file='bin.293567', master_log_pos=90061272;
    7. point dbfoo3 to dbfoo2 and start the slave
    8. MariaDB [(none)]> stop slave;
      MariaDB [(none)]> change master to master_host='foo2', master_log_file='bin.009642', master_log_pos=91547140;
      MariaDB [(none)]> start slave;

Topology refactoring using classic MySQL replication commands - version 1

A week ago I wrote a post about promoting a mysql slave as master. Today I will write topology refactoring using classic replication commands.

Context:

  • Initial state:
    dbfoo1(M) - dbfoo2(MS) - dbrfoo3(S)
  • Desired state:
    dbfoo1(M) - dbfoo2(S)
              \ dbfoo3(S)

Let's get to work:

  1. stop slave on dbfoo3
  2. MariaDB [(none)]> stop slave;
  3. check the "freezing point" on dbfoo2 and note down the output from 'show' commands
  4. MariaDB [(none)]> stop slave; show master status\Gshow slave status\Gstart slave;
    
    *************************** 1. row ***************************
                File: bin.009642
            Position: 91547140
        Binlog_Do_DB: 
    Binlog_Ignore_DB: 
    1 row in set (0.00 sec)
    
    *************************** 1. row ***************************
                   Slave_IO_State: 
                      Master_Host: dbfoo1
                      Master_User: user
                      Master_Port: 3306
                    Connect_Retry: 60
                  Master_Log_File: bin.293567
              Read_Master_Log_Pos: 90061272
                   Relay_Log_File: relay.028914
                    Relay_Log_Pos: 90061553
            Relay_Master_Log_File: bin.293567
                 Slave_IO_Running: No
                Slave_SQL_Running: No
                  Replicate_Do_DB: 
              Replicate_Ignore_DB: 
               Replicate_Do_Table: 
           Replicate_Ignore_Table: 
          Replicate_Wild_Do_Table: 
      Replicate_Wild_Ignore_Table: 
                       Last_Errno: 0
                       Last_Error: 
                     Skip_Counter: 0
              Exec_Master_Log_Pos: 90061272
                  Relay_Log_Space: 90062173
                  Until_Condition: None
                   Until_Log_File: 
                    Until_Log_Pos: 0
               Master_SSL_Allowed: No
               Master_SSL_CA_File: 
               Master_SSL_CA_Path: 
                  Master_SSL_Cert: 
                Master_SSL_Cipher: 
                   Master_SSL_Key: 
            Seconds_Behind_Master: NULL
    Master_SSL_Verify_Server_Cert: No
                    Last_IO_Errno: 0
                    Last_IO_Error: 
                   Last_SQL_Errno: 0
                   Last_SQL_Error: 
      Replicate_Ignore_Server_Ids: 
                 Master_Server_Id: 270
                   Master_SSL_Crl: 
               Master_SSL_Crlpath: 
                       Using_Gtid: No
                      Gtid_IO_Pos: 
    1 row in set (0.00 sec)
    
    Query OK, 0 rows affected (0.00 sec)
  5. make sure dbfoo3 has the same data as dbfoo2 and use File and Position from show master status
  6. MariaDB [(none)]> start slave until master_log_file='bin.009642', master_log_pos=91547140;
  7. point dbfoo3 to dbfoo1 and start the slave
  8. MariaDB [(none)]> stop slave;
    MariaDB [(none)]> change master to master_host='foo1', master_log_file='bin.293567', master_log_pos=90061272;
    MariaDB [(none)]> start slave;

mysql: Promoting a slave to master

I assume you you have a high availability (master - slave topology) setup. I will be brief and technical on this one :)

Context:

  • dbfoo1 is master and dbfoo2 is slave - dbfoo1(M) - dbfoo2(S)
  • dbfoo2 has to be promoted to master - dbfoo2(M) - dbfoo1(S)
  • note that in some cases the slave requires a innodb buffer pool warm-up before being promoted to master

Let's get to work:

  • remove the VIP from dbfoo1 (/etc/network/interfaces)
  • add the VIP to dbfoo2 (/etc/network/interfaces)
  • take down the VIP on dbfoo1

[codesyntax lang="bash"]

dbfoo1# ip a d dev eth0 x.x.x.x/y

[/codesyntax]

  • dbfoo2 should not be lagging behind dbfoo1
dbfoo2> show slave status\G
Relay_Log_Space: 0
Seconds_Behind_Master: 0
  • check the "freezing point" on dbfoo1 and note down the output from 'show' commands (this is optional but good to have)
dbfoo1> flush logs;
dbfoo1> show master status\G
dbfoo1> show slave status\G
  • check the "freezing point" on dbfoo1 and note down the output from 'show' commands
dbfoo2> slave stop;
dbfoo2> flush logs;
dbfoo2> show master status\G  <--- from here use File: and Position: to attach dbfoo1 (two steps below)
dbfoo2> show slave status\G
MariaDB [(none)]> show master status\Gshow slave status\G
*************************** 1. row ***************************
            File: bin.005662
        Position: 385
    Binlog_Do_DB: 
Binlog_Ignore_DB: 
1 row in set (0.00 sec)

*************************** 1. row ***************************
               Slave_IO_State: 
                  Master_Host: dbfoo1
                  Master_User: user
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: bin.006034
          Read_Master_Log_Pos: 245
               Relay_Log_File: relay.017538
                Relay_Log_Pos: 523
        Relay_Master_Log_File: bin.006034
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 245
              Relay_Log_Space: 1128
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 310
1 row in set (0.00 sec)

MariaDB [(none)]>
  • finish the flip - promote dbfoo2 as master by adding the VIP on it

[codesyntax lang="bash"]

dbfoo2# ip a a dev eth0 x.x.x.x/y

[/codesyntax]

  • connect dbfoo2  to the new master
dbfoo1> reset slave all;
dbfoo1> change master to master_host='dbfoo2', master_user='user', master_password='password', master_log_file='bin.005662', master_log_pos=385;
dbfoo1> slave start;
  • done!