2017-08-07

salt advanced topic

salt高级话题

salt job管理

// job概念
slat每次运行任务时都会将作业任务发布到pub-sub总线,minion会对任务做出响应.为区分不同任务,salt-master每次发布一个任务都会为该任务创建一个jobid,不同的jobid对应一个独立的操作任务,master端默认会缓存24小时内的所有job详细操作

master 缓存目录: /var/cache/salt/master/jobs
minion 缓存目录: /var/cache/salt/minion/proc,创建以jobid为名的文件,文件内容是对本次任务的缓存,任务执行完后文件会被删除.使用"strings /var/cache/salt/minion/proc/2017080912124243524"进行查看缓存文件内容

例如:
在master上长时间执行一个命令:
[root@saltserver ~]# salt '*' cmd.run 'sleep 1000;echo hello'
在minion上查看:
[root@saltminion ~]# ls /var/cache/salt/minion/proc/20170807162006823661
/var/cache/salt/minion/proc/20170807162006823661
[root@saltminion ~]# strings /var/cache/salt/minion/proc/20170807162006823661
tgt_type
glob
20170807162006823661
user
root
sleep 1000;echo hello
cmd.run


// 管理job
通过 salt-run 命令来管理job,这种方式其实是通过runner系统对job进行管理;另一种管理job的方式是通过 saltutil模块

例如: 执行过程中 中断
[root@saltserver ~]# salt '*' cmd.run 'sleep 1000;echo hello'
^C
Exiting gracefully on Ctrl-c
This job's jid is: 20170807162218636420
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
salt-run jobs.lookup_jid 20170807162218636420

综上,我们看到jobid是20170807162218636420
在minion上查看:
[root@saltminion ~]# ls /var/cache/salt/minion/proc/20170807162218636420
/var/cache/salt/minion/proc/20170807162218636420

通过saltutil进行查看:
[root@saltserver ~]# salt '*' saltutil.find_job 20170807162218636420
192.168.13.187:
    ----------
    arg:
        - sleep 1000;echo hello
    fun:
        cmd.run
    jid:
        20170807162218636420
    pid:
        30823
    ret:
    tgt:
        *
    tgt_type:
        glob
    user:
        root

通过kill指令杀死job:
[root@saltserver ~]# salt '*' saltutil.kill_job 20170807162218636420    
192.168.13.187:
    Signal 9 sent to job 20170807162218636420 at pid 30823

查看master上cache的所有job:
[root@saltserver ~]# salt '*' saltutil.runner jobs.list_jobs

runner系统

前面通过salt master通过执行模块管理salt minion,除此之外,salt还为我们提供了runners系统,runner和salt的模块类似,不同点是salt的模块在minion上执行,runner在sat master上执行,而salt master可以在任意minion执行任务.
所以,通过runner可以控制不同minion的执行顺序,这一点是state模块无法做到的.通过salt-run可以调用salt的runner功能

//minion管理
探测主机是否存活:
[root@saltserver ~]# salt '*' test.ping
192.168.13.187:
    True
通过runner也可以达到同样效果,不同的是runner不需要匹配目标minion:
[root@saltserver ~]# salt-run manage.up
- 192.168.13.187
这条命令实质上是runner调用master并对所有minion进行test.ping,不同的是runner只在master上执行,没有和任何minion进行通信

[root@saltserver ~]# salt-run manage.down
这条命令也是通过master对所有minion调用test.ping函数,然后和已经注册的minion做比较,找出现在已经不能连接的minion,然后进行输出

//job管理
查看所有和job管理相关的命令:
[root@saltserver ~]# salt-run -d |grep job

列出24小时内所有job:
[root@saltserver ~]# salt-run jobs.list_jobs

查看某一个特定的job:
[root@saltserver ~]# salt-run jobs.list_job 20170807162218636420

//orchestrate
runner可以控制不同minion的执行顺序,runner有一项功能用于管理多台minion的状态:orchestrate runner.
例如:
[root@saltserver ~]# vim /srv/salt/orch.sls
mysql_mod:
  salt.state:
    - tgt: 'minion-one'
    - tgt_type: list
    - sls: mysql
web_mod:
  salt.state:
    - tgt: 'minion-two'
    - tgt_type: list
    - sls: httpd
    - require:
      - salt: mysql_mod

执行:
[root@saltserver ~]# salt-run state.orchestrate orch

综上,通过这样的模块顺序控制使得minion-one必须先安装mysql后,才能让minion-two再安装httpd

event系统和reactor系统

//event系统
event是salt通信的核心,每一个event可以认为是salt的一个执行单元,比job要宽泛,job可以看成一种特殊的event

例如: 写一个脚本用于监听master的event
* 编写脚本:
[root@saltserver ~]# vim eventlisten.py
#-*- coding:utf-8 -*-
"""
This script is a generic tool to test event output
"""

# import python libs
from __future__ import absolute_import,print_function
import optparse
import pprint
import time
import os

# import salt libs
import salt.utils.event

# import 3rd-party libs
import salt.ext.six as six

def parse():
    """
    Parse the script command line inputs
    """
    parser = optparse.OptionParser()
    parser.add_option(
        '-s',
        '--sock-dir',
        dest='sock_dir',
        default='/var/run/salt',
        help=('statically define the directory holding the salt unix sockets for communication')
    )
    parser.add_option(
        '-n',
        '--node',
        dest='node',
        default='master',
        help=('state if this listener will attach to a master or a minion daemon,pass "master" or "minion"')
    )
    parser.add_option(
        '-f',
        '--func_count',
        default='',
        help=('Return a count of the number of minions which hava replied to a job with a ggiven func.')
    )
    parser.add_option(
        '-i',
        '--id',
        default='',
        help=('if connection to a live master or minion,pass in the id')
    )
    parser.add_option(
        '-t',
        '--transport',
        default='zeromq',
        help=('Transport to use. Default:\'zeromq\'')
    )

    options, args = parser.parse_args()

    opts = {}

    for k,v in six.iteritems(options.__dict__):
        if v is not None:
            opts[k] = v

    opts['sock_dir'] = os.path.join(opts['sock_dir'],opts['node'])

    if 'minion' in options.node:
        if args:
            opts['id'] = args[0]
            return opts
        if options.id:
            opts['id'] = options.id
        else:
            opts['id'] = options.node
    return opts

def check_access_and_print_warning(sock_dir):
    """
    check if this user is able to access the socket directory and print a warning if not
    """
    if (os.access(sock_dir,os.R_OK) and
        os.access(sock_dir,os.W_OK) and
        os.access(sock_dir,os.X_OK)):
        return
    else:
        print('Warning: Events will not be reported "(not able to access {0})".format(sock_dir)')


def listen(opts):
    """
    attach to the pub socket and grab messges
    """
    event = salt.utils.event.get_event(
        opts['node'],
        sock_dir = opts['sock_dir'],
        transport = opts['transport'],
        opts = opts,
        listen = True
    )
    check_access_and_print_warning(opts['sock_dir'])
    print(event.puburi)
    jid_counter = 0
    found_minions = []
    while True:
        ret = event.get_event(full=True)
        if ret is None:
            continue
        if opts['func_count']:
            data = ret.get('data',False)
            if data:
                if 'id' in six.iterkeys(data) and data.get('id',False) not in found_minions:
                    if data['fun'] == opts['func_count']:
                        jid_counter += 1
                        found_minions.append(data['id'])
                        print('Reply received from [{0}]. Total replies now:[{1}].'.format(ret['data']['id'],jid_counter))
                    continue
        else:
            print('Event fired at {0}'.format(time.asctime()))
            print('*' * 25)
            print('Tag: {0}'.format(ret['tag']))
            print('Data:')
            pprint.pprint(ret['data'])

if __name__ == '__main__':
    opts = parse()
    listen(opts)

* 在salt-master上执行脚本:
[root@saltserver ~]# python eventlisten.py
/var/run/salt/master/master_event_pub.ipc

* 在打开一个salt-master终端进行执行:
[root@saltserver ~]# salt '*' test.ping
192.168.13.187:
    True

* 在salt-master上查看输出信息:
[root@saltserver ~]# python eventlisten.py
/var/run/salt/master/master_event_pub.ipc
Event fired at Tue Aug  8 10:06:47 2017
*************************
Tag: 20170808100647714897
Data:
{'_stamp': '2017-08-08T02:06:47.715514', 'minions': ['192.168.13.187']}
Event fired at Tue Aug  8 10:06:47 2017
*************************
Tag: salt/job/20170808100647714897/new
Data:
{'_stamp': '2017-08-08T02:06:47.716171',
 'arg': [],
 'fun': 'test.ping',
 'jid': '20170808100647714897',
 'minions': ['192.168.13.187'],
 'tgt': '*',
 'tgt_type': 'glob',
 'user': 'root'}
Event fired at Tue Aug  8 10:06:47 2017
*************************
Tag: salt/job/20170808100647714897/ret/192.168.13.187
Data:
{'_stamp': '2017-08-08T02:06:47.797479',
 'cmd': '_return',
 'fun': 'test.ping',
 'fun_args': [],
 'id': '192.168.13.187',
 'jid': '20170808100647714897',
 'retcode': 0,
 'return': True,
 'success': True}

 综上,event输出包含两部分:
 部分一: tag,类似于一个url
        salt/job/20170808100647714897/new
        这条tag从左到右说明了:类型,任务ID,任务标识
 部分二: 数据字典,包含:时间戳,调用函数,jid,minion信息



//salt为我们提供了event.send模块可自定义event,例如:
* 在master端执行
[root@saltserver ~]# python eventlisten.py
/var/run/salt/master/master_event_pub.ipc
* 在minion端执行
[root@saltminion ~]# salt-call  event.send myco/mytag foo=Foo bar=Bar
local:
    True
* 在master端查看
[root@saltserver ~]# python eventlisten.py         
/var/run/salt/master/master_event_pub.ipc
Event fired at Tue Aug  8 10:21:41 2017
*************************
Tag: salt/auth
Data:
{'_stamp': '2017-08-08T02:21:41.187922',
 'act': 'accept',
 'id': '192.168.13.187',
 'pub': '-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnZHnTfGzMpKiNqPmwM1Q\nrgnRGSrA8eNghWRBECq+raT3qG99xvY7sjP1at8Ul/fnEMCC/K1W4CJz/6TN2lG/\n1jVh77zk9W/PbABGv5i00ZpmxZBtgNywRj4wL9IPPDmalSqm13HnjnVQ7+1I4MK7\naQDmkihGWynSh5U/CGLpApWNowcGw2Xui0d3jdrhGjngvmFuwooCLHlXDOXE3kLY\n1/5oE1+NpM4+6Hbg09XQ6wXtAnM7zNW1GFBfuNTh9ZymkicUso8uqZXCH2dUrf+e\n8VViFb01jBOBgThXm4hmqyF+NfbrjlFmcfSMJ73pJBC1VvmfQ4+F9oSksJGVufkl\n7QIDAQAB\n-----END PUBLIC KEY-----',
 'result': True}
Event fired at Tue Aug  8 10:21:41 2017
*************************
Tag: minion/refresh/192.168.13.187
Data:
{'Minion data cache refresh': '192.168.13.187',
 '_stamp': '2017-08-08T02:21:41.311715'}
Event fired at Tue Aug  8 10:21:41 2017
*************************
Tag: myco/mytag
Data:
{'_stamp': '2017-08-08T02:21:41.396306',
 'cmd': '_minion_event',
 'data': {'__pub_fun': 'event.send',
          '__pub_jid': '20170808102141379062',
          '__pub_pid': 31706,
          '__pub_tgt': 'salt-call',
          'bar': 'Bar',
          'foo': 'Foo'},
 'id': '192.168.13.187',
 'tag': 'myco/mytag'}
Event fired at Tue Aug  8 10:21:41 2017
*************************
Tag: salt/job/20170808102141404168/ret/192.168.13.187
Data:
{'_stamp': '2017-08-08T02:21:41.405667',
 'arg': ['myco/mytag', 'foo=Foo', 'bar=Bar'],
 'cmd': '_return',
 'fun': 'event.send',
 'fun_args': ['myco/mytag', 'foo=Foo', 'bar=Bar'],
 'id': '192.168.13.187',
 'jid': '20170808102141404168',
 'retcode': 0,
 'return': True,
 'tgt': '192.168.13.187',
 'tgt_type': 'glob'}

综上,我们看到新产生的event的tag正是我们自己触发的Event.我们既然可以获取这么多的Event,如果能对这些event做定制化的相应会使salt更加灵活强大. salt为我们设计了Reactor系统,可以对event做出相应


//reactor系统
reactor分两部分来配置:
部分一: 在master配置文件中,定义哪些event触发哪些reactor文件
部分二: reactor的SLS配置文件,定义对事件作出相应时采取哪些操作

例如:(修改master配置)
* 修改master配置文件
[root@saltserver ~]# vim /etc/salt/master
reactor:
  - 'myco/*':
    - salt://reactor.sls

* 重启salt-master
[root@saltserver ~]# pkill salt-master
[root@saltserver ~]# salt-master -c /etc/salt -d

* 定义reactor.sls文件
[root@saltserver ~]# vim /srv/salt/reactor.sls
my_first_reactor:
  local.cmd.run:
    - tgt: '192.168.13.187'
    - arg:
      - 'touch /tmp/my_first_reactor'

* 在master上执行
[root@saltserver ~]# python eventlisten.py
/var/run/salt/master/master_event_pub.ipc

* 在minion上执行
[root@saltminion ~]# salt-call  event.send myco/mytag foo=Foo bar=Bar
local:
    True

* 在master上观察输出结果
[root@saltserver ~]# python eventlisten.py
/var/run/salt/master/master_event_pub.ipc
Event fired at Tue Aug  8 10:32:51 2017
*************************
Tag: salt/auth
Data:
{'_stamp': '2017-08-08T02:32:51.543849',
 'act': 'accept',
 'id': '192.168.13.187',
 'pub': '-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnZHnTfGzMpKiNqPmwM1Q\nrgnRGSrA8eNghWRBECq+raT3qG99xvY7sjP1at8Ul/fnEMCC/K1W4CJz/6TN2lG/\n1jVh77zk9W/PbABGv5i00ZpmxZBtgNywRj4wL9IPPDmalSqm13HnjnVQ7+1I4MK7\naQDmkihGWynSh5U/CGLpApWNowcGw2Xui0d3jdrhGjngvmFuwooCLHlXDOXE3kLY\n1/5oE1+NpM4+6Hbg09XQ6wXtAnM7zNW1GFBfuNTh9ZymkicUso8uqZXCH2dUrf+e\n8VViFb01jBOBgThXm4hmqyF+NfbrjlFmcfSMJ73pJBC1VvmfQ4+F9oSksJGVufkl\n7QIDAQAB\n-----END PUBLIC KEY-----',
 'result': True}
Event fired at Tue Aug  8 10:32:51 2017
*************************
Tag: minion/refresh/192.168.13.187
Data:
{'Minion data cache refresh': '192.168.13.187',
 '_stamp': '2017-08-08T02:32:51.669152'}
Event fired at Tue Aug  8 10:32:51 2017
*************************
Tag: myco/mytag
Data:
{'_stamp': '2017-08-08T02:32:51.755789',
 'cmd': '_minion_event',
 'data': {'__pub_fun': 'event.send',
          '__pub_jid': '20170808103251736557',
          '__pub_pid': 31742,
          '__pub_tgt': 'salt-call',
          'bar': 'Bar',
          'foo': 'Foo'},
 'id': '192.168.13.187',
 'tag': 'myco/mytag'}
Event fired at Tue Aug  8 10:32:51 2017
*************************
Tag: salt/job/20170808103251765140/ret/192.168.13.187
Data:
{'_stamp': '2017-08-08T02:32:51.768086',
 'arg': ['myco/mytag', 'foo=Foo', 'bar=Bar'],
 'cmd': '_return',
 'fun': 'event.send',
 'fun_args': ['myco/mytag', 'foo=Foo', 'bar=Bar'],
 'id': '192.168.13.187',
 'jid': '20170808103251765140',
 'retcode': 0,
 'return': True,
 'tgt': '192.168.13.187',
 'tgt_type': 'glob'}
Event fired at Tue Aug  8 10:32:51 2017
*************************
Tag: 20170808103251827538
Data:
{'_stamp': '2017-08-08T02:32:51.829254', 'minions': ['192.168.13.187']}
Event fired at Tue Aug  8 10:32:51 2017
*************************
Tag: salt/job/20170808103251827538/new
Data:
{'_stamp': '2017-08-08T02:32:51.829749',
 'arg': ['touch /tmp/my_first_reactor'],
 'fun': 'cmd.run',
 'jid': '20170808103251827538',
 'minions': ['192.168.13.187'],
 'tgt': '192.168.13.187',
 'tgt_type': 'glob',
 'user': 'root'}
Event fired at Tue Aug  8 10:32:51 2017
*************************
Tag: salt/auth
Data:
{'_stamp': '2017-08-08T02:32:51.890407',
 'act': 'accept',
 'id': '192.168.13.187',
 'pub': '-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnZHnTfGzMpKiNqPmwM1Q\nrgnRGSrA8eNghWRBECq+raT3qG99xvY7sjP1at8Ul/fnEMCC/K1W4CJz/6TN2lG/\n1jVh77zk9W/PbABGv5i00ZpmxZBtgNywRj4wL9IPPDmalSqm13HnjnVQ7+1I4MK7\naQDmkihGWynSh5U/CGLpApWNowcGw2Xui0d3jdrhGjngvmFuwooCLHlXDOXE3kLY\n1/5oE1+NpM4+6Hbg09XQ6wXtAnM7zNW1GFBfuNTh9ZymkicUso8uqZXCH2dUrf+e\n8VViFb01jBOBgThXm4hmqyF+NfbrjlFmcfSMJ73pJBC1VvmfQ4+F9oSksJGVufkl\n7QIDAQAB\n-----END PUBLIC KEY-----',
 'result': True}
Event fired at Tue Aug  8 10:32:51 2017
*************************
Tag: salt/job/20170808103251827538/ret/192.168.13.187
Data:
{'_stamp': '2017-08-08T02:32:51.984504',
 'cmd': '_return',
 'fun': 'cmd.run',
 'fun_args': ['touch /tmp/my_first_reactor'],
 'id': '192.168.13.187',
 'jid': '20170808103251827538',
 'retcode': 0,
 'return': '',
 'success': True}

 * 在minion上查看
 [root@saltminion ~]# ls -1 /tmp/my_first_reactor
/tmp/my_first_reactor

salt api

salt 提供了大量的 python API 来供第三方程序调用,这些python API可以让我们将Salt强大的功能通过几行代码嵌入到自己的脚本和程序中来完成复杂的任务

// salt配置文件信息获取
>>> import salt.config
>>> master_opts = salt.config.client_config('/etc/salt/master')
>>> master_opts

>>> minion_opts = salt.config.client_config('/etc/salt/minion')
>>> minion_opts

// salt loader API
通过Loader API我们加载salt的执行模块,还可以获取salt的grains等数据信息
>>> import salt.config
>>> import salt.loader
>>> __opts__ = salt.config.minion_config('/etc/salt/minion')
>>> __grains__ = salt.loader.grains(__opts__)
>>> __grains__

加载执行模块:
>>> import salt.config
>>> import salt.loader
>>> __opts__ = salt.config.minion_config('/etc/salt/minion')
>>> __salt__ = salt.loader.minion_mods(__opts__)
>>> __salt__['cmd.run']("ls /")

//LocalClient
LocalClient是最常用的Salt API之一,通过LocalClient的cmd等方法我们可以实现salt执行远程模块的功能
* 用local.cmd批量执行远程模块
>>> import salt.client
>>> local = salt.client.LocalClient()
>>> local.cmd('*','test.fib','10')
* 用local.cmd_async批量异步执行远程模块
>>> import salt.client
>>> local = salt.client.LocalClient()
>>> local.cmd_async('*','test.sleep',[30])
'20170808104645529446'
[root@saltserver ~]# salt-run  jobs.list_job 20170808104645529446
Arguments:
    - 30
Function:
    test.sleep
Minions:
    - 192.168.13.187
Result:
    ----------
    192.168.13.187:
        ----------
        return:
            True
StartTime:
    2017, Aug 08 10:46:45.529446
Target:
    *
Target-type:
    glob
User:
    root
jid:
    20170808104645529446

smallasa

I known what i do !

salt advanced topic

salt高级话题