Cluster Cheat Sheet
Config files
- /etc/corosync/corosync.conf – config file for corosync cluster membership and quorum
- /var/lib/pacemaker/crm/cib.xml – config file for cluster nodes and resources
Log files
- /var/log/cluster/corosync.log
- /var/log/pacemaker.log
- /var/log/pcsd/pcsd.log
- /var/log/messages – used for some other services including crmd and pengine etc.
Resources and Resource Groups
A cluster resource refers to any object or service which is managed by the Pacemaker cluster.
A number of different resources are defined by Pacemaker:
| Primitive | this is the basic resource managed by the cluster | |
| Clone | a resource which can run on multiple nodes simultaneously | |
| MultiStake or Master/Slave | a resource in which one instance serves as master and the other as slave. A common example of this is DRBD | |
| Resource Group | this is a set of primitives or clone which is used to group resources together for easier admin | |
| Resource Classes | OCF or Open Cluster Framework | this is the most commonly used resource class for Pacemaker clusters |
| Service | used for implementing systemd, upstart, and lsb commands | |
| Systemd | used for systemd commands | |
| Fencing | used for Stonith fencing resources | |
| Nagios | used for Nagios plugins | |
| LSB or Linux Standard Base | these are for the older Linux init script operations. Now deprecated | |
| Resource stickiness | this refers to running a resource on the same cluster node even after some problem occurs with the node which is later rectified. This is advised since migrating resources to other nodes should generally be avoided | |
Constraints
A set of rules that sets out how resources or resource groups should be started.
Constraint Types:
| Location | defines on which node a resource should run – or not run, if the priority is set to minus -INFINITY |
| Colocation | defines which resources should be started together – or not started together in the case of -INFINITY |
| Order | defines in which order resources should be started. This is to allow for pre-conditional services to be started first |
Resource Order Priority Scores
These are used with the constraint types above.
The priority score can be set to a value between -1,000,000 (-INFINITY = the event will never happen) right up to INFINITY (1,000,000 = the event must happen).
Any negative priority score will prevent the resource from running.
Cluster Admin Commands
On RedHat Pacemaker Clusters, the pcs command is used to manage the cluster. pcs stands for “Pacemaker Configuration System”:
pcs status – View cluster status.
pcs config – View and manage cluster configuration. pcs cluster – Configure cluster options and nodes. pcs resource – Manage cluster resources. pcs stonith – Manage fence devices. pcs constraint – Manage resource constraints. pcs property – Manage pacemaker properties. pcs node – Manage cluster nodes. pcs quorum – Manage cluster quorum settings. pcs alert – Manage pacemaker alerts. pcs pcsd – Manage pcs daemon. pcs acl – Manage pacemaker access control lists.
Installation
To install packages:
yum install pcs -y
yum install fence-agents-all -y echo CHANGE_ME | passwd –stdin hacluster systemctl start pcsd systemctl enable pcsd
Authenticate new nodes
pcs cluster auth \
node1.example.com node2.example.com node3.example.com Username: hacluster Password: node1.example.com: Authorized node2.example.com: Authorized node3.example.com: Authorized
create and start a new cluster
pcs cluster setup <option> <member> …
eg
pcs cluster setup –start –enable –name mycluster \
node1.example.com node2.example.com node3.example.com
To enable cluster services to start on reboot:
pcs cluster enable –all
To enable cluster service on a specific node[s]:
pcs cluster enable [–all] [node] […]
To disable cluster services on a node[s]:
pcs cluster disable [–all] [node] […]
To display cluster status:
pcs status pcs config pcs cluster status pcs quorum status pcs resource show crm_verify -L -V crm_mon – this is used as equivalent for the crmsh/crmd version of Pacemaker
To delete a cluster:
pcs cluster destroy <cluster>
To start/stop a cluster:
pcs cluster start –all pcs cluster stop –all
To start/stop a cluster node:
pcs cluster start <node> pcs cluster stop <node>
To carry out mantainance on a specific node:
pcs cluster standby <node>
Then to restore the node to the cluster service:
pcs cluster unstandby <node>
To switch a node to standby mode:
pcs cluster standby <node1>
To restore a node from standby mode:
pcs cluster unstandby <node1>
To set a cluster property
pcs property set <property>=<value>
To disable stonith fencing: NOTE: you should usually not do this on a live production cluster!
pcs property set stonith-enabled=false
To reenable the stonith fencing:
pcs property set stonith-enabled=true
To configure firewalling for the cluster:
firewall-cmd –permanent –add-service=high-availability
firewall-cmd –reload
To add a node to the cluster:
check hacluster user and password
systemctl status pcsd
Then on an active node:
pcs cluster auth node4.example.com pcs cluster node add node4.example.com
Then, on the new node:
pcs cluster start pcs cluster enable
To display the xml configuration
pcs cluster cib
To display current cluster status:
pcs status
To manage cluster resources:
pcs resource <tab>
To enable, disable and relocate resource groups:
pcs resource move <resource>
or alternatively with:
pcs resource relocate <resource>
to locate the resource back to its original node:
pcs resource clear <resource>
pcs contraint <type> <option>
To create a new resource:
pcs resource create <resource_name> <resource_type> <resource_options>
To create new resources, reference the appropriate resource agents or RAs.
To list ocf resource types (example below with ocf:heartbeat)
pcs resource list heartbeat ocf:heartbeat:IPaddr2 ocf:heartbeat:LVM ocf:heartbeat:Filesystem ocf:heartbeat:oracle ocf:heartbeat:apache options detail of a resource type or agent: pcs resource describe <resource_type> pcs resource describe ocf:heartbeat:IPaddr2 pcs resource create vip_cluster ocf:heartbeat:IPaddr2 ip=192.168.125.10 –group myservices pcs resource create apache-ip ocf:heartbeat:IPaddr2 ip=192.168.125.20 cidr_netmask=24
To display a resource:
pcs resource show
Cluster Troubleshooting
Logging functions:
journalctl
tail -f /var/log/messages tail -f /var/log/cluster/corosync.log
Debug information commands:
pcs resource debug-start <resource> pcs resource debug-stop <resource> pcs resource debug-monitor <resource> pcs resource failcount show <resource>
To update a resource after modification:
pcs resource update <resource> <options>
To reset the failcount:
pcs resource cleanup <resource>
To remove a resource from a node:
pcs resource move <resource> [ <node> ]
To start a resource or a resource group:
pcs resource enable <resource>
To stop a resource or resource group:
pcs resource disable <resource>
To create a resource group and add a new resource:
pcs resource create <resource_name> <resource_type> <resource_options> –group <group>
To delete a resource:
pcs resource delete <resource>
To add a resource to a group:
pcs resource group add <group> <resource> pcs resource group list pcs resource list
To add a constraint to a resource group:
pcs constraint colocation add apache-group with ftp-group -100000 pcs constraint order apache-group then ftp-group
To reset a constraint for a resource or a resource group:
pcs resource clear <resource>
To list resource agent (RA) classes:
pcs resource standards
To list available RAs:
pcs resource agents ocf | service | stonith
To list specific resource agents of a specific RA provider:
pcs resource agents ocf:pacemaker
To list RA information:
pcs resource describe RA pcs resource describe ocf:heartbeat:RA
To create a resource:
pcs resource create ClusterIP IPaddr2 ip=192.168.100.125 cidr_netmask=24 params ip=192.168.125.100 cidr_netmask=32 op monitor interval=60s
To delete a resource:
pcs resource delete resourceid
To display a resource (example with ClusterIP):
pcs resource show ClusterIP
To start a resource:
pcs resource enable ClusterIP
To stop a resource:
pcs resource disable ClusterIP
To remove a resource:
pcs resource delete ClusterIP
To modify a resource:
pcs resource update ClusterIP clusterip_hash=sourceip
To delete parameters for a resource (resource specific, here for ClusterIP):
pcs resource update ClusterIP ip=192.168.100.25
To list the current resource defaults:
pcs resource rsc default
To set resource defaults:
pcs resource rsc defaults resource-stickiness=100
To list current operation defaults:
pcs resource op defaults
To set operation defaults:
pcs resource op defaults timeout=240s
To set colocation:
pcs constraint colocation add ClusterIP with WebSite INFINITY
To set colocation with roles:
pcs constraint colocation add Started AnotherIP with Master WebSite INFINITY
To set constraint ordering:
pcs constraint order ClusterIP then WebSite
To display constraint list:
pcs constraint list –full
To show a resource failure count:
pcs resource failcount show RA
To reset a resource failure count:
pcs resource failcount reset RA
To create a resource clone:
pcs resource clone ClusterIP globally-unique=true clone-max=2 clone-node-max=2
To manage a resource:
pcs resource manage RA
To unmanage a resource:
pcs resource unmanage RA
Fencing (Stonith)
ipmitool -H rh7-node1-irmc -U admin -P password power on fence_ipmilan –ip=rh7-node1-irmc.localdomain –username=admin –password=password –action=status Status: ON pcs stonith pcs stonith describe fence_ipmilan pcs stonith create ipmi-fencing1 fence_ipmilan \ pcmk_host_list=”rh7-node1.localdomain” \ ipaddr=192.168.100.125 \ login=admin passwd=password \ op monitor interval=60s pcs property set stonith-enabled=true pcs stonith fence pcmk-2 stonith_admin –reboot pcmk-2
To display fencing resources:
pcs stonith show
To display Stonith RA information:
pcs stonith describe fence_ipmilan
To list available fencing agents:
pcs stonith list
To add a filter to list available resource agents for Stonith:
pcs stonith list <string>
To setup properties for Stonith:
pcs property set no-quorum-policy=ignore pcs property set stonith-action=poweroff # default is reboot
To create a fencing device:
pcs stonith create stonith-rsa-node1 fence_rsa action=off ipaddr=”node1_rsa” login=<user> passwd=<pass> pcmk_host_list=node1 secure=true
To display fencing devices:
pcs stonith show
To fence a node off from the rest of the cluster:
pcs stonith fence <node>
To modify a fencing device:
pcs stonith update stonithid [options]
To display fencing device options:
pcs stonith describe <stonith_ra>
To delete a fencing device:
pcs stonith delete stonithd
rodolfo