Differences
This shows you the differences between two versions of the page.
Next revision | Previous revision | ||
cloudappliance/highavailabilitysetup [2018_04_06 18:38] – external edit 127.0.0.1 | cloudappliance:highavailabilitysetup [2024_01_12 18:13] (current) – removed steven | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ===== SME HA Setup "2 x 2" ===== | ||
- | ==== Introduction ==== | ||
- | The SME Cloud Control appliance as shipped is configured for deployment on a single virtual machine. However, a common deployment scenario for production deployments are redundant web frontends in front of a master-slave MySQL deployment. | ||
- | |||
- | By deploying multiple web frontends and a Master-Slave database your SME Cloud will increase availability and number of concurrent users. | ||
- | |||
- | === Scope === | ||
- | |||
- | This guide details how to deploy SME with multiple web frontends and a Master-Slave database. | ||
- | |||
- | The SME Deep Search Appliance is also not covered in this guide. | ||
- | |||
- | {{: | ||
- | |||
- | ==== Part I ==== | ||
- | === Assumptions === | ||
- | This guide assumes you have working knowledge and an understanding of Linux operating systems, databases, etc. If any questions come up, please contact your account manager or SME support. | ||
- | |||
- | For this guide we are using the following hostnames: smeweb01, smeweb02, smesql01, smesql02 and smesql-vip, you are of course free to select your own names that matches your naming schema. | ||
- | |||
- | In addition, you should have DNS configured and verified for the above 5 DNS records and ip addresses, as well as opened up any internal firewalls that can restrict necessary traffic between the systems | ||
- | |||
- | == Initial State == | ||
- | This guide assumes you set up the four appliance following the instructions in the Appliance Installation guide [[cloudappliance/ | ||
- | For easy failover of the application in case of DB failover, you must assign a virtual IP address for the database servers. This will mean that you can change the primary DB server without changing the configuration on the application servers. There are countless ways in linux to configure automatic failover of the VIP, but this guide will configure the DB servers requiring manual failover. This is often preferred in our scenario of master - slave replication. | ||
- | |||
- | === Preparation === | ||
- | |||
- | Before you start, please be sure to collect / prepare the necessary information. | ||
- | * 4 SME Appliances deployed | ||
- | * SME linux root password | ||
- | * SME linux smeconfiguser password | ||
- | * 5 IP addresses on your LAN, 2 for application servers, 2 for database servers and a virtual IP (VIP) for the master DB server. | ||
- | * 5 DNS names configured for the IPs. | ||
- | * New DB user for app server and password, in this guide we use smestoreremote. | ||
- | * New DB user for DB server replication, | ||
- | |||
- | == Linux Login == | ||
- | For Linux command line operations, you must run the commands shown in this document as the root user unless otherwise specified. However, for security reasons you cannot connect with ssh to the machine directly. Instead, you should ssh to the box using smeconfiguser and then su to root: | ||
- | < | ||
- | # ssh smeconfiguser@smeweb01 | ||
- | </ | ||
- | |||
- | Enter the smeconfiguser password at the prompt. Once logged in, elevate your privileges to root. | ||
- | < | ||
- | # su - | ||
- | </ | ||
- | |||
- | Enter the root password at the prompt | ||
- | |||
- | The ' | ||
- | |||
- | === Database Login === | ||
- | For database operations, to login to the smestorage database, you do not need a password. As root from the command line, type: | ||
- | |||
- | < | ||
- | # mysql smestorage | ||
- | </ | ||
- | |||
- | **Note**: This command only works if you are root, smeconfiguser does not have privilege to run this command. | ||
- | |||
- | === Stop services === | ||
- | Before we start the configuration, | ||
- | |||
- | < | ||
- | # systemctl stop httpd | ||
- | # systemctl stop cloudftp | ||
- | # systemctl stop vsftpd | ||
- | </ | ||
- | |||
- | |||
- | |||
- | ==== Part II ==== | ||
- | === Configuring the Database Servers === | ||
- | You must perform these steps to create a specialized database server from the standard SME appliance distribution. In this guide we also run memcached as a part of the DB server. | ||
- | |||
- | === Restrict external access | ||
- | The database server does not serve web pages and does not need to be accessible from outside WAN. The only traffic you need to allow is TCP port 3306 and 11211 between the four SME appliances. | ||
- | |||
- | === Disable unnecessary services === | ||
- | The DB server will only be used for memcache and database services, so the following services, are unnecessary and can be disabled. The below commands should be executed on both smesql01 and smesql02 as the //root// user. | ||
- | |||
- | < | ||
- | # systemctl disable httpd | ||
- | # systemctl disable cloudftp | ||
- | # systemctl disable vsftpd | ||
- | </ | ||
- | |||
- | == crontab == | ||
- | You must also disable some jobs in crontab, these should only run on one application server. Again, as root on smesql01 and smesql02: | ||
- | < | ||
- | # crontab -e -u smestorage | ||
- | </ | ||
- | |||
- | Place a "#" | ||
- | |||
- | {{:: | ||
- | |||
- | === iptables for dbservers === | ||
- | |||
- | On both smesql01 and smesql02, you must update iptables to allow incoming connections to mariadb, do the following. | ||
- | |||
- | As root: | ||
- | |||
- | < | ||
- | # systemctl stop iptables | ||
- | # vi / | ||
- | </ | ||
- | |||
- | Comment out lines 16 through 21, theses are port numbers 21, 80, 8080, 443, 990 and 2200. | ||
- | |||
- | Then add the following two lines after line 21: | ||
- | |||
- | < | ||
- | -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 11211 -j ACCEPT | ||
- | -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 3306 -j ACCEPT | ||
- | </ | ||
- | |||
- | Note: Port 3306 is the database port, 11211 is the port for memcached. If the traffic from the SME webservers will go through a firewall on your LAN, you have to ensure these two ports are also allowed there. | ||
- | |||
- | The file should now look like this: | ||
- | |||
- | {{:: | ||
- | |||
- | Restart the firewall: | ||
- | < | ||
- | # systemctl start iptables | ||
- | </ | ||
- | |||
- | === Memcached configuration === | ||
- | By default, the appliance memcached service listens only for connections from localhost, in order to share a memcache between the two application servers, we must change that. For redundancy we will do this on both smesql01 and smesql02. | ||
- | |||
- | To allow other machines to connect, edit / | ||
- | < | ||
- | |||
- | Change the line | ||
- | OPTIONS=" | ||
- | |||
- | to | ||
- | |||
- | OPTIONS=" | ||
- | |||
- | Making the file look like this | ||
- | |||
- | {{:: | ||
- | |||
- | then restart the memcached service | ||
- | < | ||
- | |||
- | === MySQL configuration for HA === | ||
- | == Master MySQL Database == | ||
- | The settings for MySQL are stored in / | ||
- | As root on the **primary sql server, smesql01 only:** | ||
- | < | ||
- | |||
- | Paste the following: | ||
- | < | ||
- | [mysqld] | ||
- | # Allow remote addresses to connect to this server. | ||
- | bind-address = 0.0.0.0 | ||
- | # Avoid sysdate() ambiguity. | ||
- | sysdate-is-now=1 | ||
- | |||
- | # | ||
- | # | ||
- | # server-id is 1 for the master. (The slave has server-id=2.) | ||
- | server-id=1 | ||
- | # Transaction logs on the master are written to | ||
- | # / | ||
- | log-bin = / | ||
- | log-bin-index = / | ||
- | expire_logs_days = 10 | ||
- | max_binlog_size = 100M | ||
- | binlog_format=row | ||
- | |||
- | # | ||
- | # | ||
- | </ | ||
- | |||
- | Then restart the database | ||
- | < | ||
- | |||
- | == Active VIP == | ||
- | To configure the active VIP address on the master server, create a virtual interface configuration file: | ||
- | < | ||
- | |||
- | Paste the following content in the newly created file, adjusting for your IP and netmask, etc. | ||
- | |||
- | < | ||
- | # VIP ip configuration | ||
- | DEVICE=" | ||
- | BOOTPROTO=none | ||
- | IPADDR=172.17.2.110 | ||
- | NETMASK=255.255.255.0 | ||
- | USERCTL=YES | ||
- | ONPARENT=YES | ||
- | ONBOOT=NO | ||
- | #EOF | ||
- | </ | ||
- | |||
- | Once saved, execute: | ||
- | < | ||
- | |||
- | From another host, now try to ping smesql-vip or the IP address. | ||
- | |||
- | To disable the VIP, or before running ifup on the slave, execute | ||
- | < | ||
- | |||
- | == Slave MySQL Database == | ||
- | The settings for MySQL are stored in / | ||
- | |||
- | As root on the **secondary sql server, smesql02 only:** | ||
- | < | ||
- | # vi / | ||
- | </ | ||
- | |||
- | Paste the following: | ||
- | < | ||
- | [mysqld] | ||
- | # Allow remote addresses to connect to this server. | ||
- | bind-address = 0.0.0.0 | ||
- | # Avoid sysdate() ambiguity. | ||
- | sysdate-is-now=1 | ||
- | |||
- | # | ||
- | # | ||
- | #Server-id is 2 for the slave. (The master has server-id=1.) | ||
- | server-id=2 | ||
- | # | ||
- | #slave in the / | ||
- | relay-log = / | ||
- | relay-log-index = / | ||
- | # | ||
- | # | ||
- | </ | ||
- | |||
- | Then restart the database | ||
- | |||
- | < | ||
- | # systemctl restart mariadb | ||
- | </ | ||
- | |||
- | == Passive VIP == | ||
- | To configure the passive VIP address on the slave server, create a virtual interface configuration file: | ||
- | |||
- | < | ||
- | # vi / | ||
- | </ | ||
- | |||
- | Paste the following content in the newly created file, adjusting for your IP and netmask, etc. | ||
- | |||
- | < | ||
- | # VIP ip configuration | ||
- | DEVICE=" | ||
- | BOOTPROTO=none | ||
- | IPADDR=172.17.2.110 | ||
- | NETMASK=255.255.255.0 | ||
- | USERCTL=YES | ||
- | ONPARENT=NO | ||
- | ONBOOT=NO | ||
- | #EOF | ||
- | </ | ||
- | Once saved, **if the VIP is diabled on the master**, execute: | ||
- | |||
- | < | ||
- | # /sbin/ifup eth0:0 | ||
- | </ | ||
- | |||
- | From another host, now try to ping smesql-vip or the IP address. | ||
- | |||
- | To disable the VIP, or before running ifup on the master, execute | ||
- | < | ||
- | # / | ||
- | </ | ||
- | |||
- | === Configuring database replication === | ||
- | Using database replication allows you to quickly continue service if a database goes down. | ||
- | |||
- | In MySQL replication, | ||
- | |||
- | === Starting replication | ||
- | You will need to synchronize the replication between the master and slave servers when | ||
- | * You start the database servers using replication for the first time. | ||
- | * You are starting the servers with replication enabled after a period of running without replication. | ||
- | * The master did not shut down cleanly (and its saved replication information may be incorrect). | ||
- | |||
- | === Master actions === | ||
- | == Create replication user account == | ||
- | A replication account must be added to the Master MySQL server to allow the slave to pull the bin logs. Once the user is created the database is locked to prevent new writes and the current database position is listed. | ||
- | |||
- | On the Master DB server smesql01: | ||
- | |||
- | Log into mysql and execute the following commands to create a user called %%repl_user%%. | ||
- | |||
- | < | ||
- | CREATE USER repl_user IDENTIFIED BY '< | ||
- | GRANT REPLICATION SLAVE ON *.* TO ' | ||
- | FLUSH PRIVILEGES; | ||
- | FLUSH TABLES WITH READ LOCK; | ||
- | SHOW MASTER STATUS; | ||
- | </ | ||
- | |||
- | **NOTE**: Record the < | ||
- | |||
- | Each command should succeed without error. | ||
- | |||
- | {{:: | ||
- | |||
- | === Replicated DB and Start Slave === | ||
- | |||
- | The database on the Master Server is now locked and will not accept any writes. | ||
- | |||
- | This section is executed from the smesql02 server only as the root user. | ||
- | From the command prompt replicate the database from smesql01 to smesql02. | ||
- | |||
- | < | ||
- | # ssh smeconfiguser@< | ||
- | </ | ||
- | |||
- | The password for smeconfiguser will need to be supplied. | ||
- | |||
- | < | ||
- | # mysql | ||
- | </ | ||
- | |||
- | **NOTE**: The section below needs modification before it can be executed in the mysql cli. It can be easier and less error prone if you copy/paste it in to a notepad application, | ||
- | < | ||
- | RESET SLAVE; | ||
- | |||
- | CHANGE MASTER TO MASTER_HOST='< | ||
- | MASTER_USER=' | ||
- | MASTER_PASSWORD='< | ||
- | MASTER_LOG_FILE=' | ||
- | MASTER_LOG_POS=< | ||
- | |||
- | START SLAVE; | ||
- | </ | ||
- | |||
- | Note: I have highlighted the values in <> above that you will have to customize for your environment. | ||
- | |||
- | This command gives the slave the information it needs to start replication from the master. repl_user is the username created on the master server. < | ||
- | |||
- | Note that START SLAVE command is sticky: the DB server will start up in slave mode on subsequent restarts | ||
- | |||
- | == Verify the setup == | ||
- | To verify that replication is correctly configured, you can now test if the slave is configured by running this command on the slave server | ||
- | |||
- | From the MySQL cli: | ||
- | < | ||
- | SHOW SLAVE STATUS\G | ||
- | </ | ||
- | |||
- | In your output look for the following values: %%Slave_IO_State%%, | ||
- | |||
- | {{:: | ||
- | |||
- | == Slave doesn' | ||
- | If there is a problem with starting the slave, the Slave_IO_State column will remain blank - you will never see it go to " | ||
- | |||
- | == Unlock Master and create User == | ||
- | On the Master server the database can now be unlocked allowing writes. | ||
- | < | ||
- | UNLOCK TABLES; | ||
- | CREATE USER ' | ||
- | GRANT ALL PRIVILEGES ON *.* TO ' | ||
- | FLUSH PRIVILEGES; | ||
- | </ | ||
- | |||
- | **Note**: | ||
- | |||
- | == Verifing Replication == | ||
- | The user created in the previous step show now be visible in the Slave database. | ||
- | |||
- | < | ||
- | SELECT User FROM mysql.user; | ||
- | |||
- | +----------------+ | ||
- | | User | | ||
- | +----------------+ | ||
- | | smestoreremote | | ||
- | | root | | ||
- | | root | | ||
- | | | | ||
- | | root | | ||
- | | smestore | ||
- | | | | ||
- | +----------------+ | ||
- | 7 rows in set (0.00 sec) | ||
- | |||
- | </ | ||
- | |||
- | Notice that the smestoreremote user has been replicated to the Slave server. | ||
- | |||
- | MySQL can now be exited on both servers with the command: | ||
- | < | ||
- | |||
- | ==== Part III ==== | ||
- | === Configure the application servers === | ||
- | In this section, you change the application server' | ||
- | **These steps are repeated on both smeweb01 and smeweb02.** | ||
- | |||
- | Make a backup of the configuration file: | ||
- | < | ||
- | # cd / | ||
- | # cp config.inc.php / | ||
- | </ | ||
- | |||
- | Open the configuration file to edit the settings. | ||
- | |||
- | < | ||
- | # cd / | ||
- | # nano config.inc.php | ||
- | </ | ||
- | |||
- | We have to update 5 variables in the file. They are all close to the top under "DB settings", | ||
- | < | ||
- | |||
- | In addition add a newline to define a remote memcached instance | ||
- | < | ||
- | |||
- | Further we need to update the $dbuser and $password to match the < | ||
- | |||
- | **Do not change $dbname.** | ||
- | |||
- | After the update of this is what my file looks like: | ||
- | |||
- | {{:: | ||
- | |||
- | == Disable unused services == | ||
- | As the mariadb and memcached services are not used on the application server, you should disable them: | ||
- | |||
- | These steps are repeated on both smeweb01 and smeweb02. | ||
- | |||
- | == mariadb == | ||
- | < | ||
- | # systemctl disable mariadb | ||
- | # systemctl stop mariadb | ||
- | </ | ||
- | |||
- | == memcached == | ||
- | < | ||
- | # systemctl disable memcached | ||
- | # systemctl stop memcached | ||
- | </ | ||
- | |||
- | == crontab == | ||
- | You must also disable crontab on the smeweb02 server, crontab should only run from one application server. | ||
- | **As root on smeweb02 only:** | ||
- | < | ||
- | # crontab -e -u smestorage | ||
- | </ | ||
- | |||
- | Place a # in front of the seven jobs listed in the crontab schedule. For a screenshot, refer to the DB section above. | ||
- | |||
- | Restart both servers, so on both smeweb01 and smeweb02, execute a reboot. | ||
- | |||
- | |||
- | ==== Appendix ==== | ||
- | The following section covers additional topics not necessary to establish HA on a clean pair of new servers. | ||
- | |||
- | === Custom Branding === | ||
- | When setting up multiple application servers, the branding information is saved at / | ||
- | |||
- | If branding is changed you should copy this directory manually to other application servers. | ||
- | |||
- | On smeweb01, as root: | ||
- | < | ||
- | # cd / | ||
- | # tar czf branding.tgz / | ||
- | # scp branding.tgz smeconfiguser@smeweb02: | ||
- | </ | ||
- | |||
- | |||
- | On smeweb02, as root: | ||
- | < | ||
- | # cd / | ||
- | # mv / | ||
- | # tar xzf branding.tgz | ||
- | </ | ||
- | |||
- | === Applying Updates === | ||
- | Each Application server will need to be updated separately when a new version is available. | ||
- | |||
- | == Startup / Shutdown of HA Systems == | ||
- | In the event the environment needs to be brought down for a maintenance cycle, shutdown servers in the following order: | ||
- | - WebServers | ||
- | - Master MySQL | ||
- | - Slave MySQL | ||
- | |||
- | For startup, reverse the order above. | ||
- | |||
- | == Database Backup == | ||
- | One advantage of using replication is that you can perform a backup of the database from the slave at a point in time without stopping the master - so the application is still providing service to users. | ||
- | |||
- | You do the backup from the slave as follows: | ||
- | |||
- | * stop the slave (systemctl stop mariadb) | ||
- | * backup the database from the slave - see the MySQL documentation Using | ||
- | * Replication for Backups for procedures. | ||
- | * restart the slave (systemctl start mariadb). | ||
- | |||
- | Once you restart the slave, it will automatically catch up with the master database status from the point where it left off. | ||
- | |||
- | == Failing over to the MySQL Slave == | ||
- | In the event that the Master MySQL fails or gets corrupted promoting the Slave to a Standalone MySQL server and moving the VIP is the fastest way to restore functionality to SME. | ||
- | |||
- | If the old Master server is still connected to the network it is important to disable the VIP IP address so that it doesn' | ||
- | |||
- | On the Master server: | ||
- | |||
- | Bring down the VIP | ||
- | < | ||
- | # / | ||
- | </ | ||
- | |||
- | Edit ifcfg-eth0: | ||
- | < | ||
- | # vi / | ||
- | </ | ||
- | |||
- | Switch the line ONPARENT from YES to NO | ||
- | |||
- | < | ||
- | # VIP ip configuration | ||
- | DEVICE=" | ||
- | BOOTPROTO=none | ||
- | IPADDR=172.17.2.110 | ||
- | NETMASK=255.255.255.0 | ||
- | USERCTL=YES | ||
- | ONPARENT=NO | ||
- | ONBOOT=NO | ||
- | #EOF | ||
- | </ | ||
- | |||
- | On the Slave Server: | ||
- | |||
- | Stop the MySQL replication | ||
- | < | ||
- | # mysql | ||
- | STOP SLAVE; | ||
- | |||
- | exit | ||
- | </ | ||
- | |||
- | Bring up the VIP | ||
- | |||
- | < | ||
- | # /sbin/ifup eth0:0 | ||
- | </ | ||
- | |||
- | Edit ifcfg-eth0: | ||
- | < | ||
- | # vi / | ||
- | </ | ||
- | |||
- | Switch the line ONPARENT from NO to YES | ||
- | < | ||
- | # VIP ip configuration | ||
- | DEVICE=" | ||
- | BOOTPROTO=none | ||
- | IPADDR=172.17.2.110 | ||
- | NETMASK=255.255.255.0 | ||
- | USERCTL=YES | ||
- | ONPARENT=NO | ||
- | ONBOOT=NO | ||
- | #EOF | ||
- | </ | ||
- | |||
- | Verify that the IP address has moved by running the command "ip addr" on both the Master and Slave servers. | ||
- | {{:: | ||
- | |||
- | === Failback to Master === | ||
- | Once the Master MySQL server is repaired the Master-Slave relationship can be re-established. | ||
- | |||
- | ** On the WebServers: | ||
- | |||
- | The webservers must either be shutdown during this process or the following commands must be run on each. | ||
- | |||
- | < | ||
- | # systemctl stop httpd | ||
- | # systemctl stop crond | ||
- | </ | ||
- | |||
- | **On the Slave Server: ** | ||
- | |||
- | Login and become root with "su -". Disable the VIP and enter MySQL | ||
- | < | ||
- | # su - | ||
- | # / | ||
- | # mysql | ||
- | </ | ||
- | |||
- | Lock the tables and ensure there are no other processes still committing data. If any processes show other than the one listed below ensure the webservers are shutdown or disabled from the previous step. | ||
- | |||
- | < | ||
- | FLUSH TABLES WITH READ LOCK; | ||
- | SHOW FULL PROCESSLIST; | ||
- | |||
- | +-----+------+-----------+------+---------+------+-------+-----------------------+----------+ | ||
- | | Id | User | Host | db | Command | Time | State | Info | Progress | | ||
- | +-----+------+-----------+------+---------+------+-------+-----------------------+----------+ | ||
- | | 215 | root | localhost | NULL | Query | ||
- | +-----+------+-----------+------+---------+------+-------+-----------------------+----------+ | ||
- | 1 row in set (0.00 sec) | ||
- | </ | ||
- | |||
- | ** On the Master Server:** | ||
- | |||
- | Login to smesql01 and become root. Then copy the current database from the Slave Server while the Slave tables are locked for READ. Substitute the IP address or FQDN of < | ||
- | |||
- | < | ||
- | # su - | ||
- | # ssh smeconfiguser@< | ||
- | |||
- | # mysql | ||
- | </ | ||
- | |||
- | In mysql enter the following commands to reset the Master server. | ||
- | |||
- | < | ||
- | RESET MASTER: | ||
- | SHOW MASTER STATUS; | ||
- | |||
- | +--------------------------+----------+--------------+------------------+ | ||
- | | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | | ||
- | +--------------------------+----------+--------------+------------------+ | ||
- | | master-tx-log-bin.000001 | | ||
- | +--------------------------+----------+--------------+------------------+ | ||
- | 1 row in set (0.00 sec) | ||
- | |||
- | </ | ||
- | |||
- | ** On the Slave Server:** | ||
- | < | ||
- | UNLOCK TABLES; | ||
- | RESET SLAVE; | ||
- | |||
- | CHANGE MASTER TO MASTER_HOST='< | ||
- | MASTER_USER=' | ||
- | MASTER_PASSWORD='< | ||
- | MASTER_LOG_FILE=' | ||
- | MASTER_LOG_POS=< | ||
- | |||
- | START SLAVE; | ||
- | </ | ||
- | Note: I have highlighted the values with <> above that you will have to set for your environment. | ||
- | |||
- | This command gives the slave the information it needs to restart replication from the master. // | ||
- | |||
- | Note that START SLAVE command is sticky: the DB server will start up in slave mode on subsequent restarts. | ||
- | |||
- | To verify that replication is correctly running again with this command on the slave server | ||
- | < | ||
- | SHOW SLAVE STATUS\G | ||
- | </ | ||
- | |||
- | In the output look for the following values: Slave_IO_State, | ||
- | |||
- | {{:: | ||
- | |||
- | Now that replication is running again exit mysql on both servers and set the VIP to run on the Master server again. |