ELK-日志监控ElastAlert

ElastAlert简介

ElastAlert是一个简单的框架，用于从Elasticsearch中的数据中发出异常，尖峰或其他感兴趣的模式的警报。ElastAlert适用于所有版本的Elasticsearch。在Yelp，我们使用Elasticsearch，Logstash和Kibana来管理不断增长的数据和日志。Kibana非常适合可视化和查询数据，但是我们很快意识到，它需要一个辅助工具来警告数据不一致。为此，创建了ElastAlert。如果您有近乎实时的数据写入Elasticsearch，并且想要在数据与特定模式匹配时收到警报，则ElastAlert是为您提供的工具。如果您可以在Kibana中看到它，则ElastAlert可以对其发出警报。

总结来说Elast Alert可以对定时对ES中的日志进行过滤关键字，通过多种方式将匹配的信息发送给相关人员

Elast Alert所需组件关系

Elast Alert简称EA

环境

CentOS 7.2
Python 3.6
pip 3.6

Python

EA从0.2.0版本已经不再支持Python2.x版本了，0.2.0版本以后的EA必须使用Python3.6之后的版本

PIP

pip为python的包管理工具，不同版本的pip版本对应不同的python版本，低版本pip会找不到高版本python所需安装包的版本

Python和PIP总结

此次安装我是使用的系统是CentOS7.2 默认python环境是2.7，安装了python2.7版本的pip，但最新版本的EA已经不支持python2.7了，所以在系统上安装了python3.6版本自动安装了pip的3.6版，

需要注意：

  1. python2.7的lib在/usr/lib/python下面，而自己安装的python3.6的lib在/usr/local/python36/lib下面在安装的时候，可能会发生找不到lib文件的情况，就是这种导致的
  2. 不同版本的pip会将软件包安装至相对应的lib下，比如pip会将软件包安装到python2.7中，而pip3.6才会将软件包安装至python3.6中，如果使用错误的pip安装会导致python找不到对应的软件包
  3. 在运行EA的时候一定要选对python版本，比如，依赖包都安装到python3.6上，而你使用python运行EA就会报错

Elast Alert的部署

Python的安装
  • CentOS7.2上安装Python3.6，多个python版本共存
  • 安装依赖包,因为此次是编译安装python3.6，如果没有在编译前安装依赖包会导致，编译安装后的python找不到软件包，而编译后再安装依赖包，会发现依赖包已经安装可python还是无法找到依赖包

    yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel

  • 下载python3.6的gz包

    wget https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz

  • 编译安装python3.6

    ./configure –prefix=/usr/local/python36
    make
    make install

  • 将python3.6加入环境变量

PIP的安装

此次我安装完Python3.6后自动生成了pip3.6，所以此次没有进行如下步骤，但是我在Python2.7上安装的时候进行了下面的步骤

1、先安装setup-tools在https://pypi.python.org/pypi/setuptools下载安装包，可以复制下载链接使用wget命令下载。 

 下载下来后，解压安装：wget https://pypi.python.org/packages/45/29/8814bf414e7cd1031e1a3c8a4169218376e284ea2553cc0822a6ea1c2d78/setuptools-36.6.0.zip#md5=74663b15117d9a2cc5295d76011e6fd1
unzip setuptools-36.6.0.zip
cd setuptools-36.6.0
python setup.py install 

2、安装pip下载地址：https://pypi.python.org/pypi/pip，和上面一样下载下来，这里可以选择tgz格式的安装包。

 wget https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9
tar -zxvf pip-9.0.1.tar.gz
cd pip-9.0.1
python setup.py install

EA的安装
pip安装EA

pip install elastalert 安装EA，
需要注意pip和python的对应关系以及pip所需的依赖包
pip uninstall moudle 卸载模块
pip list 查看已经安装的模块
pip install moudle>=3.1 可以指定安装的模块的版本

python安装EA

git clone https://github.com/Yelp/elastalert.git 通过git clone 最新版本的EA
python3.6 install -r requirements.txt 通过新安装的python3.6安装EA的依赖包
python3.6 setup.py install 安装EA
安装完成后会出现四个命令

EA的配置

cd /app/.elastalert 到git clone 的EA目录下
cp config.yaml.example config.yaml

config.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: /app/elastalert/example_rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: esip

# The Elasticsearch port
es_port: 9200

# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1

# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
writeback_alias: elastalert_alerts

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
#logging:
#  version: 1
#  incremental: false
#  disable_existing_loggers: false
#  formatters:
#    logline:
#      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
#
#    handlers:
#      console:
#        class: logging.StreamHandler
#        formatter: logline
#        level: DEBUG
#        stream: ext://sys.stderr
#
#      file:
#        class : logging.FileHandler
#        formatter: logline
#        level: DEBUG
#        filename: elastalert.log
#
#    loggers:
#      elastalert:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch.trace:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      '':  # root logger
#        level: WARN
#          handlers:
#            - console
#            - file
#        propagate: false
  • rules_folder：ElastAlert将加载规则配置文件的地方，它将尝试加载文件夹中的每个.yaml文件(这里是加* es_rules目录下)。
  • run_every：ElastAlert查询Elasticsearch的频率。
  • buffer_time：是查询窗口的大小，从每个查询运行的时间向后延伸。对于其中use_count_query或*
  • use_terms_query设置为true的规则，此值将被忽略。
  • es_host：是Elasticsearch集群的地址，ElastAlert将存储有关其状态、查询运行、警报和错误的数据。每个规则也可以设置不同的elasticsearch主机进行查询。
  • es_port：Elasticsearch对应的端口。
  • use_ssl： （可选的）是否使用TLS;连接到es_host；设置为True或False。
  • verify_certs： （可选的）是否验证TLS证书; 设置为True或False，默认是True。
  • client_cert： （可选的）PEM证书的路径。
  • client_key： （可选的） 作为客户端密钥使用的私钥文件的路径。
  • ca_certs： （可选的） 用于验证SSL连接的CA证书的路径。
  • es_username： （可选的） 用于连接Elasticsearch的basic-auth用户名。
  • es_password： （可选的） 用于连接Elasticsearch的密码。
  • es_url_prefix： （可选的） Elasticsearch端点的URL前缀。
  • es_send_get_body_as： （可选的） 查询Elasticsearch方法- GET，POST或- source，默认是GET。
  • writeback_index：是ElastAlert将存储数据的索引名称。
  • alert_time_limit： 是失败警报的重试窗口。

报警规则配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
# Alert when the rate of events exceeds a threshold

# (Optional)
# Elasticsearch host
es_host: IP

# (Optional)
# Elasticsearch port
es_port: 9200

# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: false

# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# (Required)
# Rule name, must be unique
name: rulename

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: any

# (Required)
# Index to search, wildcard supported
index: logstash-server-logback-*

# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 1

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
  hours: 1

# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- query:
    query_string:
      query: "(host:VM_13_*_centos)  AND ( exception OR error )"

# (Required)
# The alert is use when a match is found
alert:
- "email"

alert_text: "error information"
smtp_host: smtp.exmail.qq.com
smtp_port: 25
smtp_auth_file: /app/elastalert/example_rules/smtp_auth_file.yaml
email_reply_to: yourmail@qq.com
from_addr: yourmail@qq.com

# (required, email specific)
# a list of email addresses to send alerts to
email:
- "tomail1@qq.com"
- "tomail2@qq.com"

邮件配置文件(smtp_auth_file.yaml)

1
2
user: “邮箱地址"
password: "客户端授权码"

有些邮箱需要设置授权码来认证，即不允许直接通过邮箱登陆密码来发送邮件，授权码只能发送邮件 不能登陆，授权码一般通过网页登陆邮箱后在设置界面-安全选项

启动命令

1
nohup python3.6 -m elastalert.elastalert --config /app/elastalert/config.yaml --verbose --rule /app/elastalert/example_rules/rulename.yaml  >> /app/elastert/elastalert.log 2>&1 &

相关网址

Elastic官网
ElastAler的github地址
PIP下载地址

ELK
#ELK #ElastAlert #log
ELK-日志监控ElastAlert
https://imwang77.github.io/2020/05/10/ELK_日志监控_ElastAlert/
作者
imwang77
发布于
2020年5月10日
更新于
2024年1月3日
许可协议