Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revisionLast revisionBoth sides next revision | ||
| cloudappliance/twonodeha [2020_07_02 16:56] – created eric | cloudappliance:twonodeha [2023_06_22 05:41] – external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ===== File Fabric HA - Two Node Install with Automatic Failover ===== | ===== File Fabric HA - Two Node Install with Automatic Failover ===== | ||
| - | == last updated | + | == last updated |
| ==== Disclaimer ==== | ==== Disclaimer ==== | ||
| Line 12: | Line 12: | ||
| If the network is partitioned there is a chance that both masters can become active at the same time(split brain) and you will have corrupt data. In case of a split brain you will have to restore the data from a backup. | If the network is partitioned there is a chance that both masters can become active at the same time(split brain) and you will have corrupt data. In case of a split brain you will have to restore the data from a backup. | ||
| - | If you are deploying across multiple data centers or network is not reliable please follow [[cloudappliance: | + | If you are deploying across multiple data centers or network is not reliable please follow [[:cloudappliance/ |
| </ | </ | ||
| ==== Introduction ==== | ==== Introduction ==== | ||
| Line 33: | Line 33: | ||
| == Initial State == | == Initial State == | ||
| - | This guide assumes you set up the four appliance following the instructions in the Appliance Installation guide [[cloudappliance: | + | This guide assumes you set up the four appliance following the instructions in the Appliance Installation guide [[cloudappliance: |
| Keepalived will need two additional Virtual IPs, one for the VIP for the database, and one for memcache. This will mean that you can change the primary DB server without changing the configuration on the application servers. | Keepalived will need two additional Virtual IPs, one for the VIP for the database, and one for memcache. This will mean that you can change the primary DB server without changing the configuration on the application servers. | ||
| Line 88: | Line 88: | ||
| === iptables for dbservers === | === iptables for dbservers === | ||
| - | On both sme01 and sme02, you must update iptables to allow incoming connections to mariadb, do the following. | + | On both smesql01 |
| - | As root: | + | As root we will edit the iptables file to add the following lines: |
| + | / | ||
| < | < | ||
| - | iptables-save > / | + | ... |
| - | ipt_line=`iptables | + | -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 3306 -j ACCEPT |
| - | insert_line=`expr $ipt_line | + | -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 11211 -j ACCEPT |
| - | iptables | + | ... |
| + | COMMIT | ||
| + | </ | ||
| + | Then issue a restart: | ||
| - | iptables | + | < |
| - | + | systemctl restart | |
| - | iptables-save > / | + | systemctl restart docker |
| </ | </ | ||
| Line 109: | Line 113: | ||
| < | < | ||
| - | cp /etc/sysconfig/memcached | + | cd /var/www/smestorage/containers/memcached/ |
| - | cat / | + | sed -i 's/ |
| - | systemctl restart memcached | + | docker-compose down |
| + | docker-compose up -d | ||
| </ | </ | ||
| Line 186: | Line 191: | ||
| === Configure Database Replication === | === Configure Database Replication === | ||
| + | |||
| + | <WRAP center round tip 100%> | ||
| + | See [[/ | ||
| + | </ | ||
| == Create Replication User == | == Create Replication User == | ||
| Line 330: | Line 339: | ||
| We will also need to allow multicast traffic (244.0.0.0) through our iptables firewall, as well as adjust selinux to allow keepalived to operate. Finally, we will add a user (keepalived_script) which keepalived will use. | We will also need to allow multicast traffic (244.0.0.0) through our iptables firewall, as well as adjust selinux to allow keepalived to operate. Finally, we will add a user (keepalived_script) which keepalived will use. | ||
| + | |||
| + | / | ||
| < | < | ||
| - | iptables -I INPUT -i eth0 -d 224.0.0.0/8 -p vrrp -j ACCEPT | + | ... |
| - | iptables-save > / | + | -A RH-Firewall-1-INPUT |
| - | semanage permissive -a keepalived_t | + | ... |
| - | groupadd keepalived_script | + | COMMIT |
| - | useradd -s / | + | |
| </ | </ | ||
| + | Then issue a restart: | ||
| + | |||
| + | < | ||
| + | systemctl restart iptables | ||
| + | systemctl restart docker | ||
| + | </ | ||
| + | |||
| Now we will create a new keepalived configuration file for our two VIPs. | Now we will create a new keepalived configuration file for our two VIPs. | ||
| Line 391: | Line 408: | ||
| vrrp_instance MEMCACHE { | vrrp_instance MEMCACHE { | ||
| - | state MASTER | + | state BACKUP |
| interface eth0 | interface eth0 | ||
| virtual_router_id 61 | virtual_router_id 61 | ||
| Line 582: | Line 599: | ||
| Restart both servers, so on both sme01 and sme02, execute a reboot. | Restart both servers, so on both sme01 and sme02, execute a reboot. | ||
| + | |||
| + | <WRAP center round important 100%> | ||
| + | There must always be exactly one cron node per File Fabric cluster. If the node that was acting as the cron node ceases to be available as part of the cluster, another node must be made the cron node. | ||
| + | </ | ||
| ==== Appendix ==== | ==== Appendix ==== | ||
| Line 630: | Line 651: | ||
| To force a failover from the MASTER to the BACKUP, you can simply restart the keepalived service. Due to the no automatic failback (// | To force a failover from the MASTER to the BACKUP, you can simply restart the keepalived service. Due to the no automatic failback (// | ||
| - | < | + | |
| - | systemctl restart keepalived | + | |
| - | </ | + | |
| === Force Failback === | === Force Failback === | ||
| Line 642: | Line 661: | ||
| **Before you fail back, you need to ensure that all replication is up to date. Look for the Seconds_Behind_Master attribute in the mysql "SHOW SLAVE STATUS\G" | **Before you fail back, you need to ensure that all replication is up to date. Look for the Seconds_Behind_Master attribute in the mysql "SHOW SLAVE STATUS\G" | ||
| - | < | + | |
| - | systemctl restart keepalived | + | |
| - | </ | + | |