MySQL5.7一主两从MHA手工故障切换
数据库架构：一主两从
成都创新互联公司拥有10余年成都网站建设工作经验,为各大企业提供做网站、成都网站建设服务，对于网页设计、PC网站建设（电脑版网站建设）、成都APP应用开发、wap网站建设（手机版网站建设）、程序开发、网站优化（SEO优化）、微网站、国际域名空间等，凭借多年来在互联网的打拼，我们在互联网网站建设行业积累了很多网站制作、网站设计、网络营销经验，集策划、开发、设计、营销、管理等网站化运作于一体，具备承接各种规模类型的网站建设项目的能力。
master：192.168.8.57
slave1：192.168.8.58
slave2：192.168.8.59
manager：192.168.8.60
MHA工具包：
mha4MySQL-manager-0.58.tar.gz
mha4mysql-node-0.58.tar.gz
一、修改master_ip_online_change内容
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 |
#!/usr/bin/env perl
# Copyright (C) 2011 DeNA Co.,Ltd.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
use
strict;
use
warnings FATAL =>
'all'
;
use
Getopt::Long
qw(:config pass_through)
;
use
Pod::Usage;
use
MHA::MasterMonitor;
use
MHA::MasterFailover;
use
MHA::MasterRotate;
use
MHA::ManagerConst;
my
$master_state
=
""
;
my
$help
;
my
$version
;
$| = 1;
GetOptions(
'help'
=> \
$help
,
'version'
=> \
$version
,
'master_state=s'
=> \
$master_state
);
my
$exit_code
= 1;
if
(
$version
) {
print
"masterha_master_switch version $MHA::ManagerConst::VERSION.\n"
;
exit
0;
}
if
(
$help
) {
pod2usage(0);
}
if
(
$master_state
eq
"dead"
) {
$exit_code
= MHA::MasterFailover::main(
@ARGV
);
}
elsif
(
$master_state
eq
"alive"
) {
$exit_code
= MHA::MasterRotate::main(
@ARGV
);
}
else
{
pod2usage(1);
}
exit
$exit_code
;
# ############################################################################
# Documentation
# ############################################################################
=pod
=head1 NAME
masterha_master_switch - Switching MySQL master server to one of other slave servers
=head1 SYNOPSIS
# For master failover
masterha_master_switch --master_state=dead --global_conf=/etc/masterha_default.cnf --conf=/usr/local/masterha/conf/app1.cnf --dead_master_host=host1
# For online master switch
masterha_master_switch --master_state=alive --global_conf=/etc/masterha_default.cnf --conf=/usr/local/masterha/conf/app1.cnf
See online reference (http://code.google.com/p/mysql-master-ha/wiki/masterha_master_switch) for details.
=head1 DESCRIPTION
See online reference (http://code.google.com/p/mysql-master-ha/wiki/masterha_master_switch) for details.
[root@manager MHA]#
[root@manager MHA]#
[root@manager MHA]# cat /usr/local/bin/master_ip_online_change
#!/usr/bin/env perl
# Copyright (C) 2011 DeNA Co.,Ltd.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
## Note: This is a sample script and is not complete. Modify the script based on your environment.
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
use MHA::DBHelper;
use MHA::NodeUtil;
use Time::HiRes qw( sleep gettimeofday tv_interval );
use Data::Dumper;
my $_tstart;
my $_running_interval = 0.1;
my (
$command, $orig_master_is_new_slave, $orig_master_host,
$orig_master_ip, $orig_master_port, $orig_master_user,
$orig_master_password, $orig_master_ssh_user, $new_master_host,
$new_master_ip, $new_master_port, $new_master_user,
$new_master_password, $new_master_ssh_user,
);
GetOptions(
'command=s' => \$command,
'orig_master_is_new_slave' => \$orig_master_is_new_slave,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'orig_master_user=s' => \$orig_master_user,
'orig_master_password=s' => \$orig_master_password,
'orig_master_ssh_user=s' => \$orig_master_ssh_user,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
'new_master_user=s' => \$new_master_user,
'new_master_password=s' => \$new_master_password,
'new_master_ssh_user=s' => \$new_master_ssh_user,
);
exit &main();
sub current_time_us {
my ( $sec, $microsec ) = gettimeofday();
my $curdate = localtime($sec);
return $curdate . " " . sprintf( "%06d", $microsec );
}
sub sleep_until {
my $elapsed = tv_interval($_tstart);
if ( $_running_interval > $elapsed ) {
sleep( $_running_interval - $elapsed );
}
}
sub get_threads_util {
my $dbh = shift;
my $my_connection_id = shift;
my $running_time_threshold = shift;
my $type = shift;
$running_time_threshold = 0 unless ($running_time_threshold);
$type = 0 unless ($type);
my @threads;
my $sth = $dbh->prepare("SHOW PROCESSLIST");
$sth->execute();
while ( my $ref = $sth->fetchrow_hashref() ) {
my $id = $ref->{Id};
my $user = $ref->{User};
my $host = $ref->{Host};
my $command = $ref->{Command};
my $state = $ref->{State};
my $query_time = $ref->{Time};
my $info = $ref->{Info};
$info =~ s/^\s*(.*?)\s*$/$1/ if defined($info);
next if ( $my_connection_id == $id );
next if ( defined($query_time) && $query_time < $running_time_threshold );
next if ( defined($command) && $command eq "Binlog Dump" );
next if ( defined($user) && $user eq "system user" );
next
if ( defined($command)
&& $command eq "Sleep"
&& defined($query_time)
&& $query_time >= 1 );
if ( $type >= 1 ) {
next if ( defined($command) && $command eq "Sleep" );
next if ( defined($command) && $command eq "Connect" );
}
if ( $type >= 2 ) {
next if ( defined($info) && $info =~ m/^select/i );
next if ( defined($info) && $info =~ m/^show/i );
}
push @threads, $ref;
}
return @threads;
}
sub main {
if ( $command eq "stop" ) {
## Gracefully killing connections on the current master
# 1. Set read_only= 1 on the new master
# 2. DROP USER so that no app user can establish new connections
# 3. Set read_only= 1 on the current master
# 4. Kill current queries
# * Any database access failure will result in script die.
my $exit_code = 1;
eval {
## Setting read_only=1 on the new master (to avoid accident)
my $new_master_handler = new MHA::DBHelper();
# args: hostname, port, user, password, raise_error(die_on_error)_or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );
print current_time_us() . " Set read_only on the new master.. ";
$new_master_handler->enable_read_only();
if ( $new_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}
$new_master_handler->disconnect();
# Connecting to the orig master, die if any database error happens
my $orig_master_handler = new MHA::DBHelper();
$orig_master_handler->connect( $orig_master_ip, $orig_master_port,
$orig_master_user, $orig_master_password, 1 );
## Drop application user so that nobody can connect. Disabling per-session binlog beforehand
$orig_master_handler->disable_log_bin_local();
print current_time_us() . " Drpping app user on the orig master..\n";
#FIXME_xxx_drop_app_user($orig_master_handler);
## Waiting for N * 100 milliseconds so that current connections can exit
my $time_until_read_only = 15;
$_tstart = [gettimeofday];
my @threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_read_only > 0 && $#threads >= 0 ) {
if ( $time_until_read_only % 5 == 0 ) {
printf
"%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_read_only * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = [gettimeofday];
$time_until_read_only--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}
## Setting read_only=1 on the current master so that nobody(except SUPER) can write
print current_time_us() . " Set read_only=1 on the orig master.. ";
$orig_master_handler->enable_read_only();
if ( $orig_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}
## Waiting for M * 100 milliseconds so that current update queries can complete
my $time_until_kill_threads = 5;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {
if ( $time_until_kill_threads % 5 == 0 ) {
printf
"%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_kill_threads * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = [gettimeofday];
$time_until_kill_threads--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}
## Terminating all threads
print current_time_us() . " Killing all application threads..\n";
$orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 );
print current_time_us() . " done.\n";
$orig_master_handler->enable_log_bin_local();
$orig_master_handler->disconnect();
## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
## Activating master ip on the new master
# 1. Create app user with write privileges
# 2. Moving backup script if needed
# 3. Register new master's ip to the catalog database
# We don't return error even though activating updatable accounts/ip failed so that we don't interrupt slaves' recovery.
# If exit code is 0 or 10, MHA does not abort
my $exit_code = 10;
eval {
my $new_master_handler = new MHA::DBHelper();
# args: hostname, port, user, password, raise_error_or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );
## Set read_only=0 on the new master
$new_master_handler->disable_log_bin_local();
print current_time_us() . " Set read_only=0 on the new master.\n";
$new_master_handler->disable_read_only();
## Creating an app user on the new master
print current_time_us() . " Creating app user on the new master..\n";
FIXME_xxx_create_app_user($new_master_handler);
$new_master_handler->enable_log_bin_local();
$new_master_handler->disconnect();
## Update master ip on the catalog database, etc
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
# do nothing
exit 0;
}
else {
&usage();
exit 1;
}
}
sub usage {
print
"Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
die;
}
|
二、停止MHA监控程序
1 |
masterha_stop --conf=
/etc/masterha/app1
.cnf
|
三、手工停止主库MySQL进程，模拟故障发生
mysqladmin -uroot -pmysql shutdown
四、手工故障切换
masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=dead --dead_master_host=192.168.8.57 --dead_master_port=3306 --new_master_host=192.168.8.58 --new_master_port=3306 --ignore_last_failover
--dead_master_ip=is not set. Using 192.168.8.57. Fri Oct 26 16:18:05 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Fri Oct 26 16:18:05 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Fri Oct 26 16:18:05 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Fri Oct 26 16:18:05 2018 - [info] MHA::MasterFailover version 0.58. Fri Oct 26 16:18:05 2018 - [info] Starting master failover. Fri Oct 26 16:18:05 2018 - [info] Fri Oct 26 16:18:05 2018 - [info] * Phase 1: Configuration Check Phase.. Fri Oct 26 16:18:05 2018 - [info] Fri Oct 26 16:18:07 2018 - [info] GTID failover mode = 1 Fri Oct 26 16:18:07 2018 - [info] Dead Servers: Fri Oct 26 16:18:07 2018 - [info] 192.168.8.57(192.168.8.57:3306) Fri Oct 26 16:18:07 2018 - [info] Checking master reachability via MySQL(double check)... Fri Oct 26 16:18:07 2018 - [info] ok. Fri Oct 26 16:18:07 2018 - [info] Alive Servers: Fri Oct 26 16:18:07 2018 - [info] 192.168.8.58(192.168.8.58:3306) Fri Oct 26 16:18:07 2018 - [info] 192.168.8.59(192.168.8.59:3306) Fri Oct 26 16:18:07 2018 - [info] Alive Slaves: Fri Oct 26 16:18:07 2018 - [info] 192.168.8.58(192.168.8.58:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled Fri Oct 26 16:18:07 2018 - [info] GTID ON Fri Oct 26 16:18:07 2018 - [info] Replicating from 192.168.8.57(192.168.8.57:3306) Fri Oct 26 16:18:07 2018 - [info] 192.168.8.59(192.168.8.59:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled Fri Oct 26 16:18:07 2018 - [info] GTID ON Fri Oct 26 16:18:07 2018 - [info] Replicating from 192.168.8.57(192.168.8.57:3306) Master 192.168.8.57(192.168.8.57:3306) is dead. Proceed? (yes/NO): yes Fri Oct 26 16:18:14 2018 - [info] Starting GTID based failover. Fri Oct 26 16:18:14 2018 - [info] Fri Oct 26 16:18:14 2018 - [info] ** Phase 1: Configuration Check Phase completed. Fri Oct 26 16:18:14 2018 - [info] Fri Oct 26 16:18:14 2018 - [info] * Phase 2: Dead Master Shutdown Phase.. Fri Oct 26 16:18:14 2018 - [info] Fri Oct 26 16:18:14 2018 - [info] HealthCheck: SSH to 192.168.8.57 is reachable. Fri Oct 26 16:18:15 2018 - [info] Forcing shutdown so that applications never connect to the current master.. Fri Oct 26 16:18:15 2018 - [info] Executing master IP deactivation script: Fri Oct 26 16:18:15 2018 - [info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.8.57 --orig_master_ip=192.168.8.57 --orig_master_port=3306 --command=stopssh --ssh_user=root Fri Oct 26 16:18:15 2018 - [info] done. Fri Oct 26 16:18:15 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. Fri Oct 26 16:18:15 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed. Fri Oct 26 16:18:15 2018 - [info] Fri Oct 26 16:18:15 2018 - [info] * Phase 3: Master Recovery Phase.. Fri Oct 26 16:18:15 2018 - [info] Fri Oct 26 16:18:15 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase.. Fri Oct 26 16:18:15 2018 - [info] Fri Oct 26 16:18:15 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000012:359 Fri Oct 26 16:18:15 2018 - [info] Retrieved Gtid Set: a92f70a4-d5ea-11e8-af28-080027c0450d:10 Fri Oct 26 16:18:15 2018 - [info] Latest slaves (Slaves that received relay log files to the latest): Fri Oct 26 16:18:15 2018 - [info] 192.168.8.58(192.168.8.58:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled Fri Oct 26 16:18:15 2018 - [info] GTID ON Fri Oct 26 16:18:15 2018 - [info] Replicating from 192.168.8.57(192.168.8.57:3306) Fri Oct 26 16:18:15 2018 - [info] 192.168.8.59(192.168.8.59:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled Fri Oct 26 16:18:15 2018 - [info] GTID ON Fri Oct 26 16:18:15 2018 - [info] Replicating from 192.168.8.57(192.168.8.57:3306) Fri Oct 26 16:18:15 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000012:359 Fri Oct 26 16:18:15 2018 - [info] Retrieved Gtid Set: a92f70a4-d5ea-11e8-af28-080027c0450d:10 Fri Oct 26 16:18:15 2018 - [info] Oldest slaves: Fri Oct 26 16:18:15 2018 - [info] 192.168.8.58(192.168.8.58:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled Fri Oct 26 16:18:15 2018 - [info] GTID ON Fri Oct 26 16:18:15 2018 - [info] Replicating from 192.168.8.57(192.168.8.57:3306) Fri Oct 26 16:18:15 2018 - [info] 192.168.8.59(192.168.8.59:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled Fri Oct 26 16:18:15 2018 - [info] GTID ON Fri Oct 26 16:18:15 2018 - [info] Replicating from 192.168.8.57(192.168.8.57:3306) Fri Oct 26 16:18:15 2018 - [info] Fri Oct 26 16:18:15 2018 - [info] * Phase 3.3: Determining New Master Phase.. Fri Oct 26 16:18:15 2018 - [info] Fri Oct 26 16:18:15 2018 - [info] 192.168.8.58 can be new master. Fri Oct 26 16:18:15 2018 - [info] New master is 192.168.8.58(192.168.8.58:3306) Fri Oct 26 16:18:15 2018 - [info] Starting master failover.. Fri Oct 26 16:18
新闻标题：MySQL5.7一主两从MHA手工故障切换
本文来源：http://shouzuofang.com/article/gcdgsd.html