Salt Infrastructure upgrade

7:29 PM

Synopsis

As a part of this project we would be upgrading the salt infrastructure in all our managed cloud platforms (mcp). Currently we have mixed versions of salt across board and one of the deliverables of this project is to have consistency in all mcps. Please look below for the list of deliverables of this project.

Deliverables

  1. Upgrade all salt-master and salt-minions to version 2015.5.0-1.el6.
  2. Remove salt-master from xops servers.
  3. MOM in N1 should be able to manage all the minions across globe. MOMServer
  4. Multi-master architecture. Pointing all individual masters to super master in NA1.
  5. Monitor SALT. Host level and service level monitoring.

MCP Salt Topology

Build MOMServer

MOMServer is the name of the server that is going to act as the Master of all Masters. System engineer should be able to manage all minions across globe using the MOMServer. Our MOMServer would be sitting in NA1 management VLAN.

Implementation plan

Spin a "Red Hat Enterprise Linux Server release 6.6 (Santiago)" server in the management VLAN in NA1.
Run through the below steps to install the MOMServer.
# Enables the EPEL repo for salt installation.
rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
yum -y install salt-master
Make the configuration change in /etc/salt/master and restart the service.
order_masters: True    >>> # Edit /etc/salt/master
service salt-master start

Upgrade existing Masters and Minions

Implementation plan

Below two steps on each server master would upgrade salt to the latest version available on Salt repo. The version as of 15-June-2015 available on Salt EPEL repo is 2015.5.0-1.el6
ssh <salt-master_ip>
yum update salt-master
Copy the contents of Salt Minion upgrade state from below to YAML file /srv/salt/upgrade-salt.sls on the master server.
salt '*' state.sls upgrade-salt
Icon 
Salt project is a part of EPEL and some python dependencies are kept in RHEL6 optional channel. 
Please ensure rhel-x86_64-server-optional-6 and EPEL repo is enabled on the server.
rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
spacewalk-channel -a -c rhel-x86_64-server-optional-6

Salt Minion upgrade formula
Salt formula to update the salt minions is as below:
saltstack-copr:
  pkgrepo.managed:
    - humanname: Copr repo for salt owned by saltstack
    - baseurl: http://copr-be.cloud.fedoraproject.org/results/saltstack/salt/epel-{{ salt['grains.get']('osmajorrelease', 'unknown') }}-$basearch/
    - gpgkey: https://copr-be.cloud.fedoraproject.org/results/saltstack/salt/pubkey.gpg
    - gpgcheck: 1
    - skip_if_unavailable: True
    - enabled: 1
 
update_salt:
  pkg:
    - latest
    - name: salt-minion
    - order: last
  cmd:
    - wait
    - name: echo service salt-minion restart | at now + 1 minute
    - watch:
      - pkg: update_salt

Test cases

Execute the below command from primary master to ensure salt version has been updated across board.
salt '*' grains.items | grep -A 1 saltversion

Enable MOMServer functionality - Syndic

The Syndic interface is a special pass through minion, it is run on a master and connects to another master, then the master that the Syndic minion is listening to can control the minions attached to the master running the syndic.
We would be using the default syndic configuration. There is no need to make any changes in the master configuration. All we need is that syndic demon/service should be installed. To install the service please use the command below on all the master servers.

Implementation plan

Below steps would help you enable the MOMServer functionality using sycdic
ssh <primary_master>
yum install salt-syndic
echo "10.18.71.11           masterofmaster" >> /etc/hosts
 
Edit the /etc/salt/master file and uncomment the below lines:
#syndic_master: masterofmaster
#syndic_master_port: 4506
#syndic_pidfile: /var/run/salt-syndic.pid
#syndic_log_file: /var/log/salt/syndic.log
 
service salt-syndic service
Icon
Please note that server IP (10.18.71.11) in above code is just an example. Please use the actual IP address of the MOMServer

Test cases

The below command if executed from MOMServer should return true from all the minions connected to the primary master server
salt '*' test.ping

Multi-Master with fail over capabilities

Implementation plan

DADServer
ssh <DADServer>
echo "master_sign_pubkey: True" >> /etc/salt/master
echo "master_sign_key_name: DADServer_master_sign" >> /etc/salt/master
service salt-master restart
salt-cp '<primary_masters>' /etc/salt/pki/master/DADServer_master_sign.p* /etc/salt/pki/master/
salt-cp '<minions>' /etc/salt/pki/master/DADServer_master_sign.pub /etc/salt/pki/minion/
Icon
Please note that the above restart command would generate two file DADServer_master_sign.pem and DADServer_master_sign.pub under /etc/salt/pki/master on the super master server. These files are the signing key-pair and would be used in all the master servers in various DCs. DO NOT copy DADServer_master_sign.pem to any of the minions.

 Primary Master
ssh <primary_master>
echo "master_sign_pubkey: True" >> /etc/salt/master
echo "master_sign_key_name: DADServer_master_sign" >> /etc/salt/master
service salt-master start

 Minions
ssh <minions>
Edit the /etc/salt/minion and make the below configuration changes.
master:
    - salt
    - masterofmaster
 
master_type: failover
master_alive_interval: 30
verify_master_pubkey_sign: True
master_sign_key_name: MOMServer_master_sign
Icon 
Please do not configure master_shuffle: True in the minion file. we do not want to shuffle the selection of master. 
Primary should always be in the local data center and fail over should happen in NA1.

Also make sure 'salt' and 'dadserver' are pointing to the correct IP address in all minions
Restart the master on super master server.
service salt-master start

Test cases

Below the results of the test.ping under different scenarios from two masters and a minion configured for multi-master failover capabilities.
Sno.
Super Master
Primary Master
Test Remarks from Super Master
Test Remarks from Primary Master
Test Results
1is kept stoppedis runningAs expected not working.test.ping works.Passed
2is startedis kept runningKeys are not accepted, hence not workingtest.ping works.Passed
3is kept runningis stoppedtest.ping works.As expected not working.Passed
4is kept runningis startedtest.ping works.As expected not working.Passed
5is stoppedis kept runningAs expected not working.test.ping works.Passed
6is startedis kept runningAs expected not working.test.ping works.Passed

Service Monitoring

What to monitor and how to monitor?

Status Tracking

We are using Google Sheet to track the progress of this project.

References

Sno.
Topic
URL
1.
Multi-Master-PKI Tutorial With Fail over.
https://docs.saltstack.com/en/latest/topics/tutorials/multimaster_pki.html
2.Salt Syndichttps://docs.saltstack.com/en/latest/topics/topology/syndic.html
3.EPEL 5https://dl.fedoraproject.org/pub/epel/epel-release-latest-5.noarch.rpm
4.EPEL 6https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
5.
Copr repo for salt owned by saltstack
https://copr.fedoraproject.org/coprs/saltstack/salt-el5/repo/epel-5/saltstack-salt-el5-epel-5.repo

Share This Story

Tags: , ,

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor Aenean massa.

You Might Also Like

0 comments

Contact Form

Name

Email *

Message *

Translate

Wikipedia

Search results