[root@octopus ~]# stasher
Ready, EOF to exit.
> admin
Connected to objrepo.example.com, node octopus.objrepo.example.com.
Maximum 20 objects, 64 Mb aggregate object size, per transaction.
Maximum 10 concurrent subscriptions.
octopus> editcluster addnode monster set host=monster.example.com
octopus:
HOST=octopus.example.com
monster:
HOST=monster.example.com
octopus> savecluster
octopus> status
Cluster:
Status: master: octopus.objrepo.example.com (uuid S85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS), 1 slaves, timestamp 2012-03-17 16:15:56 -0400
This node:
HOST=octopus.example.com
Node: monster.objrepo.example.com: connected
HOST=monster.example.com
... long output deleted ...
master(octopus.objrepo.example.com, uuidS85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS) (thread 0x00007f90fb383700):
Quorum: full: true, majority: true
Distributor object installed: true
Transactions received by this node:
Transactions being commited:
Slave monster.objrepo.example.com:
Connected, synchronized: true
Synchronization metadata not present
Transactions received by this node:
monster.objrepo.example.com (thread 0x00007f90d9ffb700):
Encrypted connection: false
Peer's last reported status: master: octopus.objrepo.example.com (uuid S85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS), 1 slaves, timestamp 2012-03-17 16:15:56 -0400
This node's status: master: octopus.objrepo.example.com (uuid S85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS), 1 slaves, timestamp 2012-03-17 16:15:56 -0400
Outstanding ping requests:
Connected with controller: octopus.objrepo.example.com (S85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS)
... long output deleted ...
octopus> [EOF, CTRL-D]
[root@octopus ~]#
The final step of joining a new node to the cluster occurs by making an administrative connection to one of the existing nodes in the cluster (it can be any existing node, as long as it's an administrative connection; it's not necessary to make an administrative connection on the cluster's current master, as in this example).
The same editcluster command defines the new node in the repository. The output from editcluster confirms the definition, and savecluster saves it, and makes it effective.
This example defines a new node called “monster” on the machine “monster.example.com”. Once the new node gets defined, all the nodes establish connections with the new node. This is confirmed by the status command. status is very wordy, and the key things to look for confirmations are:
The number of slave nodes reported in “Status”, under “Cluster”. This example reports one slave, and including the master that makes it two nodes in the cluster, all present and accounted for. The list with established connections follows, which should list the newly-connected node.
The “master” section reports the individual status
of this, master node. Its “quorum” status reports the
quorum's status. A true “full”
quorum indicates that the master has established connections to
all defined nodes in the cluster. The “full” status
shows false if the master is not connected and
synchronized with any defined cluster node.
A true “majority”
quorum indicates that the master is connected and synchronized with
the majority of the nodes in the cluster.
false indicates that it's not, and there is no
quorum. Without a quorum, objects cannot be retrieved or placed
into the object repository cluster.
A connection to the newly-added node should also be reported in the “master” section, as shown in this example. For a very large existing object repository, it may take some time for the existing contents of the object repository cluster to get replicated to the newly-added slave, and the slave may be reported as connected, but not synchronized, for a little while. A full quorum will not be reported until the slave has synchronized its copy of the object repository.
In addition to the “master” section, there's a status section for each connected peer. The “monster.objrepo.example.com” section shows the peer's last reported status. In this example, “monster” reports that its current master is “octopus”, this node. During normal operations, all peers should report a consistent status.
And here's the status from the newly-added node's view:
[root@monster ~]# stasher
Ready, EOF to exit.
> admin
Connected to objrepo.example.com, node monster.objrepo.example.com.
Maximum 20 objects, 64 Mb aggregate object size, per transaction.
Maximum 10 concurrent subscriptions.
monster> status
Cluster:
Status: master: octopus.objrepo.example.com (uuid S85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS), 1 slaves, timestamp 2012-03-17 16:15:56 -0400
This node:
HOST=monster.example.com
Node: octopus.objrepo.example.com: connected
HOST=octopus.example.com
Connection attempts in progress:
... long output deleted ...
octopus.objrepo.example.com (thread 0x00007f140affd700):
Encrypted connection: false
Peer's last reported status: master: octopus.objrepo.example.com (uuid S85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS), 1 slaves, timestamp 2012-03-17 16:15:56 -0400
This node's status: master: octopus.objrepo.example.com (uuid S85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS), 1 slaves, timestamp 2012-03-17 16:15:56 -0400
Outstanding ping requests:
Connected with controller: octopus.objrepo.example.com (S85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS)
Disconnection mcguffin installed
Connected to a slave controller:
... long output deleted ...
slave(octopus.objrepo.example.com, uuidS85HVpjTVrQqA000_F1aJm00001vhGO00318n4AS) (thread 0x00007f140a7fc700):
Connection object installed: true
Destination repo copy object installed: false
Receiver object installed: true
Synchronized with master: true
Distributor object installed: true
monster> [EOF, CTRL-D]
[root@monster ~]#
On the “monster” slave, its status is slightly different. It reports its master, in the “Cluster” section, is “octopus”, another node. So, instead of a “master” status section, it has one called “slave”, where it reports its synchronization status.
The slave node also has a separate status section for each peer that it's connected to. This example shows status section for its connection with the master, “octopus”.
Note that it now reports non-default connection limits (20 objects per transaction, 64MB total aggregate size). The non-default connection limits were set when the cluster was initially configured. After the new node joined the cluster, the cluster's configuration parameters took effect.
Congratulations, you have a working object repository cluster.