In the event that all Foldr cluster members power down unexpectedly, this can leave the database in a state that will not start automatically. This is expected behaviour and is designed to prevent data loss in the event that cluster nodes are out of sync. For the same reason, a cluster of Foldr appliances should not be set to install updates automatically at the same time. If automatic updates are used, it is best practice to stagger updates at different times accordingly.
Should you experience a sudden shutdown event (such as a power issue) and find that the database cluster does not start gracefully when the systems are restarted, you can verify that the database status by running the following command on the Foldr console. Do this on each of the cluster members:
systemctl status mariadb
Recovery Steps
To ensure that you start the database using the most recent version of your data, you should first determine which appliance should be considered the ‘master’. The master is the appliance that last committed a transaction to it’s database (i.e. it holds the most recent version of the data) and as such should be started first, the other cluster members can then rejoin the cluster and any database changes will be synchronised between them.
1. Check the version of the data held on each cluster member by running the following command
db-version
This will display the data revision on each system, and the appliance with the highest version should be considered the master. If all values are the same, the database was fully synchronised when the shutdown event occurred and any appliance can be considered the master in this scenario.
2. Make the relevant Foldr appliance the master and start the database by running the following command.
db-bootstrap
The database should start automatically, this can be verified by running the following command
systemctl status mariadb
3. Reboot the remaining cluster members – Do not restart the master appliance.
4. Once all cluster members have rebooted fully, log into Foldr Settings on one of the cluster nodes and check that the cluster size (wsrep_cluster_size) reports the expected value within Foldr Setting >> Infrastructure >> Cluster
