The following guide can be used to set up a two node active/standby cluster running MariaDB:
A few assumptions are made; you will need to adjust these assumptions for your own environment.
Debian 11/Ubuntu 22.04 is being used for the installation. If using a different distribution or release there may be different configuration files and/or package names.
The two nodes have two NICs. One NIC will be used for the primary network interface while the other is dedicated to Corosync and DRBD.
A virtual IP will be used to expose the MariaDB service to clients; this virtual IP will be bound to the eth0
interface of the current cluster master.
node1.example.com
:
eth0
- 192.0.2.201/29
: Primary NIC (Internet/LAN)eth1
- 192.0.2.249/29
: Secondary NIC (Corosync/DRBD)node2.example.com
:
eth0
- 192.0.2.202/29
: Primary NIC (Internet/LAN)eth1
- 192.0.2.250/29
: Secondary NIC (Corosync/DRBD)A virtual IP, 192.0.2.206/29
, will be bound to the eth0
adapter of the current cluster master. Do not add the IP to the adapter manually. MariaDB clients will be connecting to this IP address.
The two nodes have two disks for use with DRBD:
/dev/sdb
: A 1GB disk used for DRBD metadata/dev/sdc
: A 30GB disk used as the backing DRBD deviceBoth devices are unpartioned.
While not required, the metadata disk is recommended as it makes future modifications to the DRBD volume size easier. If you are using a larger DRBD disk your metadata disk size may need to be adjusted appropriately.
The DRBD volume will be /dev/drbd0
with a resource name of mariadb
. On the active server, the DRBD volume will be mounted to /sql
.
A secret key will be defined as some-secret-key
; this should be changed to a more secure one.
The MariaDB data directory (datadir
in config) will be set to /sql/data
. The binary logs and slave relay logs (if relevant) directory will be set to /sql/logs
.
It is assumed that the MariaDB server package has already been installed and that there isn't any existing MariaDB data. If there is existing MariaDB data it can be dumped before continuing and then restored once complete.
STONITH will be set to disabled. This is not suitable for production; you should be using some form of fencing to prevent split brain. Configuration of STONITH is out of scope for this guide.
The following packages need to be installed:
apt -y install \
corosync \
pacemaker pacemaker-cli-utils \
crmsh \
drbd-utils \
resource-agents
The following steps should be performed on BOTH DRBD notes unless otherwise noted.
First tell the main /etc/drbd.conf
configuration file to load additional configurations from /etc/drbd.d
:
# This configuration file will include configurations from /etc/drbd.d
# Include common configuration
include "drbd.d/common.conf";
# Include resources/devices
include "drbd.d/*.res";
Ensure the /etc/drbd.d
directory exists:
mkdir /etc/drbd.d
Create a common configuration file in /etc/drbd.d/common.conf
that specifies global configuration:
# Global config
global {
usage-count no;
udev-always-use-vnr;
}
common {
protocol C;
}
Create the mariadb
resource configuration in the file /etc/drbd.d/mariadb.res
. This file will contain information such as the backing/metadata devices for the DRBD volume and the peer hostnames/IP addresses.
Note that the peer hostnames must match the FQDN for the node otherwise you will get an error as the configuration will not be able to be matched.
# Define the MariaDB DRBD resource
resource mariadb {
# Secret configuration
net {
cram-hmac-alg sha1;
shared-secret "some-secret-key";
after-sb-0pri discard-younger-primary;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
}
# Tweak the syncing settings
disk {
c-fill-target 10M;
c-max-rate 700M;
c-plan-ahead 7;
c-min-rate 4M;
}
# Per host configurations
# Ensure that the hostnames and IP addresses are correct!
on node1.example.com {
device /dev/drbd0;
meta-disk /dev/sdb;
disk /dev/sdc;
address 192.0.2.249:7788;
}
on node2.example.com {
device /dev/drbd0;
meta-disk /dev/sdb;
disk /dev/sdc;
address 192.0.2.250:7788;
}
}
The DRBD metadata disk and volumes now need to be initialized.
On both nodes, create the metadata:
drbdadm create-md mariadb
On one of the nodes (doesn't matter which one), configure it to become primary and act as the sync source:
drbdadm up mariadb
drbdadm primary mariadb/0 --force
On the other node, bring the resource up:
drbdadm up mariadb
The volume should start synchronisation. The procfs file /proc/drbd
can be used to monitor its progress from either node:
root@node1:~# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 98E710E58B3041F3046305B
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:170132 nr:0 dw:0 dr:173096 al:8 bm:0 lo:2 pe:4 ua:3 ap:0 ep:1 wo:f oos:31288584
[>....................] sync'ed: 0.6% (30552/30720)M
finish: 0:12:21 speed: 42,172 (42,172) K/sec
root@node1:~# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 98E710E58B3041F3046305B
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:31330408 nr:0 dw:135568 dr:31329192 al:14 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Once both disk states (the ds
value above) show as UpToDate
the volume has been synchronised.
You may create the file system on the /dev/drbd0
device now:
mkfs.ext4 /dev/drbd0
The Corosync configuration needs to be applied to both nodes.
On one of the nodes, generate the authentication key:
corosync-keygen
The key will then need to be copied to the other node (note it is a binary file):
rsync /etc/corosync/authkey node2.example.com:/etc/corosync/authkey
The authentication key must match on both nodes.
On both nodes, create the Corosync configuration file, /etc/corosync/corosync.conf
:
totem {
version: 2
ip_version: ipv4-6
cluster_name: mariadb
link_mode: passive
interface {
linknumber: 1
}
interface {
linknumber: 2
}
secauth: on
}
logging {
fileline: off
to_stderr: no
to_logfile: no
to_syslog: yes
syslog_facility: daemon
debug: off
timestamp: on
logger_subsys {
subsys: QUORUM
debug: off
}
}
quorum {
provider: corosync_votequorum
two_node: 1
expected_votes: 2
}
nodelist {
node {
name: node1
nodeid: 1
quorum_votes: 1
ring0_addr: 192.0.2.201
ring1_addr: 192.0.2.249
}
node {
name: node2
nodeid: 2
quorum_votes: 1
ring0_addr: 192.0.2.202
ring1_addr: 192.0.2.250
}
}
The configuration defines two rings for Corosync for redundancy. If you want to only use a single ring, remove the ring1_addr
values and remove the totem interface definition for linknumber: 2
.
Once configured on both nodes, restart the Corosync service:
systemctl restart corosync
Both nodes should come online in a few seconds if configured correctly. On either of the nodes this can be verified with crmsh status
:
root@node1:~# crm status
Cluster Summary:
* Stack: corosync
* Current DC: node2 (version 2.1.2-ada5c3b36e2) - partition with quorum
* Last updated: Sat May 6 13:44:55 2023
* Last change: Fri May 5 21:45:07 2023 by hacluster via crmd on node1
* 2 nodes configured
* 0 resource instances configured
Node List:
* Online: [ node1 node2 ]
Full List of Resources:
* No resources
If the nodes are not healthy review the syslogs/journal logging for Corosync and fix the issue before continuing.
Some basic setup needs to be performed for MariaDB and the file system that resides on DRBD.
Before performing the following steps, ensure the MariaDB server service is stopped on both nodes.
Prevent the MariaDB service from starting on boot:
systemctl disable mariadb
As an additional precaution, add a systemd override file that ensures a file exists before starting the service. This will prevent accidental starts of MariaDB if using systemctl
manually or when upgrading the packages. To create the override file:
mkdir /etc/systemd/system/mariadb.service.d
cat << EOF > /etc/systemd/system/mariadb.service.d/fileexists.conf
[Unit]
# Require that the specified file exists before starting MariaDB
ConditionPathExists=/sql/drbd_mounted
EOF
systemctl daemon-reload
The above configuration will check that the file /sql/drbd_mounted
exists before managing the MariaDB service.
On both nodes, create the /sql
path:
mkdir /sql
This path is where the DRBD volume will be mounted to.
The following steps must only be performed on the current DRBD primary server. Check the current DRBD primary server (use cat /proc/drbd
) before continuing to make sure you are on the correct node.
Mount /dev/drbd0
to /sql
:
mount /dev/drbd0 /sql
Create the data and log directories for MariaDB:
mkdir /sql/data /sql/logs
Ensure the directories are owned by the MariaDB user and set permissions:
chown mysql:mysql /sql/data /sql/logs
chmod 0750 /sql/data /sql/logs
Create the "magic" file that will be checked to ensure the DRBD file system is mounted:
touch /sql/drbd_mounted
The MariaDB server configuration needs to be modified to use the new data/log directories (otherwise MariaDB will be storing data files in /var/lib/mysql
). The configuration on both nodes must match otherwise failover will not work.
The following configuration options must be set at the minimum:
[mysql]
# Storage locations
datadir = /sql/data
# Enable file per table storage for InnoDB
innodb_file_per_table
While not required, it is recommended to split the error log files so that each node logs to its own error log. This can be done by using a drop in config file:
cat << EOF > /etc/mysql/mariadb.conf.d/error-log.cnf
[mysql]
# Set error log file for this node
log-error = /sql/data/$(hostname -f)-error.log
EOF
To enable binary logging the following options should be set:
[mysql]
# Enable binary logging
log_bin = /sql/logs/master.bin
log_bin_index = /sql/logs/master.index
log_bin_compress = 1
log_slave_updates = 1
If using replication and the slave relay log is required, ensure the relay log is set to the correct path:
[mysql]
# Configure slave relay log paths
relay_log = /sql/logs/slave-relay.log
relay_log_index = /sql/logs/slave-relay.index
relay_log_info_file = /sql/logs/slave-relay.info
Once the configuration changes have been made, the default MariaDB databases can be initialized. On the node that has the DRBD disk mounted to /sql
run the following command:
mysql_install_db
Fix up any broken permissions:
chown -R mysql:mysql /sql/data /sql/logs
The DRBD disk should now be unmounted:
umount /sql
The cluster resources now need to be configured. This configuration is what will manage the DRBD file system mount, MariaDB service and the virtual IP address that clients connect to.
The configuration only needs to be applied to one of the nodes, it will be synchonized automatically on commit to both nodes. You can create/modify the configuration from either node.
Enter the cluster resource manager shell configuration:
crm configure
Configure the resources:
# Disable STONITH; not for production!
property stonith-enabled=false
# Set options required to make a two node cluster work
property no-quorum-policy=ignore
property stop-all-resources=false
# Configure resource stickiness
rsc_defaults resource-stickiness=100
# Add the resources to be managed
## DRBD resource to control primary/secondary status
primitive r_drbd_disk ocf:linbit:drbd \
params drbd_resource="mariadb" \
op start interval="0" timeout="240" \
op stop interval="0" timeout="120s" \
op monitor interval="17s" role="Master" timeout="120s" \
op monitor interval="16s" role="Slave" timeout="119s"
## File system located on DRBD disk to be mounted to /sql
primitive r_drbd_fs ocf:heartbeat:Filesystem \
params device="/dev/drbd/by-res/mariadb/0" directory="/sql" fstype="ext4"
## The mariadb service
primitive r_mariadb systemd:mariadb \
op monitor interval="10s" \
meta target-role=Started
## The virtual IP address that MariaDB clients connect to
primitive r_ip_ipv4 ocf:heartbeat:IPaddr2 \
params ip="192.0.2.206" cidr_netmask="29" \
op monitor interval="10s" \
meta migration-threshold="2" failure-timeout="60s"
## If you would like to also add a virtual IPv6 address, uncomment the following lines and set the relevant IP
# primitive r_ip_ipv6 ocf:heartbeat:IPaddr2 \
# params ip="2001:db8:a::a" cidr_netmask="64" \
# op monitor interval="10s" \
# meta migration-threshold="2" failure-timeout="60s"
# Create a master/slave relationship for the DRBD disk
# This is to ensure that the disk is only running on a single node
ms ms_drbd r_drbd_disk \
meta master-max="1" master-node-max="1" \
clone-max="2" clone-node-max="1" \
notify="true"
# Create a group named g_mariadb containing the MariaDB service and virtual IP address
# If IPv6 is used (and the r_ip_ipv6 resource exists), ensure the r_ip_ipv6 resource is included
# As an example the group should look like this:
# group g_mariadb r_mariadb r_ip_ipv4 r_ip_ipv6
group g_mariadb r_mariadb r_ip_ipv4
# Set up the colocation options to make sure all services are bound to the master
colocation c_drbd_fs inf: r_drbd_fs ms_drbd:Master
colocation c_mariadb inf: g_mariadb ms_drbd:Master
# Set the ordering to ensure the DRBD file system is mounted before MariaDB is started and the virtual IP bound
order o_drbd_mount inf: ms_drbd:promote r_drbd_fs:start
order o_mariadb_start inf: r_drbd_fs g_mariadb
The configuration can now be committed to make it live and start the resources:
commit
You can then exit the shell and check the current cluster status to ensure the resources have started:
crm resource status
crm status
The cluster is now ready to use.
Reboot the cluster master node and ensure services fail over correctly.