Synopsis
As a part of this project we would be upgrading the salt infrastructure in all our managed cloud platforms (mcp). Currently we have mixed versions of salt across board and one of the deliverables of this project is to have consistency in all mcps. Please look below for the list of deliverables of this project.Deliverables
- Upgrade all salt-master and salt-minions to version 2015.5.0-1.el6.
- Remove salt-master from xops servers.
- MOM in N1 should be able to manage all the minions across globe. MOMServer
- Multi-master architecture. Pointing all individual masters to super master in NA1.
- Monitor SALT. Host level and service level monitoring.
MCP Salt Topology
Build MOMServer
MOMServer is the name of the server that is going to act as the Master of all Masters. System engineer should be able to manage all minions across globe using the MOMServer. Our MOMServer would be sitting in NA1 management VLAN.Implementation plan
Spin a "Red Hat Enterprise Linux Server release 6.6 (Santiago)" server in the management VLAN in NA1.Run through the below steps to install the MOMServer.
# Enables the EPEL repo for salt installation.rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpmyum -y install salt-master |
order_masters: True >>> # Edit /etc/salt/masterservice salt-master start |
Upgrade existing Masters and Minions
Implementation plan
Below two steps on each server master would upgrade salt to the latest version available on Salt repo. The version as of 15-June-2015 available on Salt EPEL repo is 2015.5.0-1.el6ssh <salt-master_ip>yum update salt-masterCopy the contents of Salt Minion upgrade state from below to YAML file /srv/salt/upgrade-salt.sls on the master server.salt '*' state.sls upgrade-salt |
Icon
Salt project is a part of EPEL and some python dependencies are kept in RHEL6 optional channel. Please ensure rhel-x86_64-server-optional-6 and EPEL repo is enabled on the server.
rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm spacewalk-channel -a -c rhel-x86_64-server-optional-6
Salt Minion upgrade formula
Salt formula to update the salt minions is as below:
saltstack-copr: pkgrepo.managed: - humanname: Copr repo for salt owned by saltstack - baseurl: http://copr-be.cloud.fedoraproject.org/results/saltstack/salt/epel-{{ salt['grains.get']('osmajorrelease', 'unknown') }}-$basearch/ - gpgkey: https://copr-be.cloud.fedoraproject.org/results/saltstack/salt/pubkey.gpg - gpgcheck: 1 - skip_if_unavailable: True - enabled: 1update_salt: pkg: - latest - name: salt-minion - order: last cmd: - wait - name: echo service salt-minion restart | at now + 1 minute - watch: - pkg: update_salt |
Test cases
Execute the below command from primary master to ensure salt version has been updated across board.salt '*' grains.items | grep -A 1 saltversion |
Enable MOMServer functionality - Syndic
The Syndic interface is a special pass through minion, it is run on a master and connects to another master, then the master that the Syndic minion is listening to can control the minions attached to the master running the syndic.We would be using the default syndic configuration. There is no need to make any changes in the master configuration. All we need is that syndic demon/service should be installed. To install the service please use the command below on all the master servers.
Implementation plan
Below steps would help you enable the MOMServer functionality using sycdicssh <primary_master>yum install salt-syndic echo "10.18.71.11 masterofmaster" >> /etc/hostsEdit the /etc/salt/master file and uncomment the below lines:#syndic_master: masterofmaster#syndic_master_port: 4506#syndic_pidfile: /var/run/salt-syndic.pid#syndic_log_file: /var/log/salt/syndic.logservice salt-syndic service |
Icon
Please note that server IP (10.18.71.11) in above code is just an example. Please use the actual IP address of the MOMServer
Test cases
The below command if executed from MOMServer should return true from all the minions connected to the primary master serversalt '*' test.ping |
Multi-Master with fail over capabilities
Implementation plan
DADServerssh <DADServer>echo "master_sign_pubkey: True" >> /etc/salt/masterecho "master_sign_key_name: DADServer_master_sign" >> /etc/salt/masterservice salt-master restartsalt-cp '<primary_masters>' /etc/salt/pki/master/DADServer_master_sign.p* /etc/salt/pki/master/salt-cp '<minions>' /etc/salt/pki/master/DADServer_master_sign.pub /etc/salt/pki/minion/ |
Icon
Please note that the above restart command would generate two file DADServer_master_sign.pem and DADServer_master_sign.pub under /etc/salt/pki/master on the super master server. These files are the signing key-pair and would be used in all the master servers in various DCs. DO NOT copy DADServer_master_sign.pem to any of the minions.
Primary Master
ssh <primary_master>echo "master_sign_pubkey: True" >> /etc/salt/masterecho "master_sign_key_name: DADServer_master_sign" >> /etc/salt/masterservice salt-master start |
Minions
ssh <minions>Edit the /etc/salt/minion and make the below configuration changes.master: - salt - masterofmastermaster_type: failovermaster_alive_interval: 30verify_master_pubkey_sign: Truemaster_sign_key_name: MOMServer_master_sign |
Icon
Restart the master on super master server.Please do not configure master_shuffle: True in the minion file. we do not want to shuffle the selection of master. Primary should always be in the local data center and fail over should happen in NA1. Also make sure 'salt' and 'dadserver' are pointing to the correct IP address in all minions
service salt-master start |
Test cases
Below the results of the test.ping under different scenarios from two masters and a minion configured for multi-master failover capabilities.
Sno.
|
Super Master
|
Primary Master
|
Test Remarks from Super Master
|
Test Remarks from Primary Master
|
Test Results
|
|---|---|---|---|---|---|
| 1 | is kept stopped | is running | As expected not working. | test.ping works. | Passed |
| 2 | is started | is kept running | Keys are not accepted, hence not working | test.ping works. | Passed |
| 3 | is kept running | is stopped | test.ping works. | As expected not working. | Passed |
| 4 | is kept running | is started | test.ping works. | As expected not working. | Passed |
| 5 | is stopped | is kept running | As expected not working. | test.ping works. | Passed |
| 6 | is started | is kept running | As expected not working. | test.ping works. | Passed |
Service Monitoring
What to monitor and how to monitor?Status Tracking
We are using Google Sheet to track the progress of this project.References
Sno.
|
Topic
|
URL
|
|---|---|---|
| 1. | Multi-Master-PKI Tutorial With Fail over. | https://docs.saltstack.com/en/latest/topics/tutorials/multimaster_pki.html |
| 2. | Salt Syndic | https://docs.saltstack.com/en/latest/topics/topology/syndic.html |
| 3. | EPEL 5 | https://dl.fedoraproject.org/pub/epel/epel-release-latest-5.noarch.rpm |
| 4. | EPEL 6 | https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm |
| 5. | Copr repo for salt owned by saltstack | https://copr.fedoraproject.org/coprs/saltstack/salt-el5/repo/epel-5/saltstack-salt-el5-epel-5.repo |