ELK-日志监控ElastAlert_python2.7版本

简介

• 之前已经写过elastalert的安装过程了，但这次安装过程中碰到的问题挺多的，特此记录一下
• 系统：Centos:7.6
• Python:2.7和3.6并存
• elastalert 0.1.20
• 安装pip #pip2.7 pip3.6
• requirements.txt 中软件的版本问题
• 安装完成后更改脚本中的执行python版本
• dingtalk插件安装

安装

pip的安装

需要注意pip不同的名称对应不同版本的python pip2.7 pip3.6

- curl -O https://bootstrap.pypa.io/pip/2.7/get-pip.py
- python get-pip.py
- python -m pip install --upgrade "pip < 21.0"

elastalert

因为dingtalk插件依赖elastalert 0.1.20版本，所以本次安装的是elastalert 0.1.20版本

dinttalk–github

wget https://github.com/Yelp/elastalert/archive/refs/tags/v0.1.20.tar.gz
tar xf v0.1.20.tar.gz
mv elastalert-0.1.20 elastalert
#把依赖文件中的软件版本>=改为== 因为过高的版本和python2.7不兼容，pip2.7安装的时候也会失败
# cat /usr/local/elastalert/requirements.txt 
    aws-requests-auth==0.3.0
    blist==1.3.6
    boto3==1.4.4
    configparser==3.5.0
    croniter==0.3.16
    elasticsearch
    envparse==0.2.0
    exotel==0.1.3
    jira==1.0.10
    jsonschema==2.6.0
    mock==2.0.0
    PyStaticConfiguration==0.10.3
    python-dateutil==2.6.0
    PyYAML==3.12
    requests==2.0.0
    simplejson==3.10.0
    stomp.py==4.1.17
    texttable==0.8.8
    twilio==6.0.0
#安装依赖，在这些依赖软件中还可能依赖其他的软件，所以在执行pip安装的命令时候出现报错 要根据报错提示去安装相应的依赖
pip2.7 install "setuptools==23.0.0"
pip2.7 install requirements.txt
python2.7 setup.py install

pip2.7安装UNKNOW软件

因为setuptools版本问题和软件包python2和3之间的支持问题会导致pip2.7安装依赖包成功了 但提示UNKNOW名称，在执行python2.7 setup.py install的时候检测不到新版本的软件包，只检测到旧的并且还提示软件包版本过低，这种情况就要去手动修改安装后的名称了

#python2.7软件依赖库地址： /usr/lib/python2.7/site-packages/
 cd /usr/lib/python2.7/site-packages/
 mv UNKNOWN.egg-info /usr/lib/python2.7/site-packages/PyJWT-1.7.0.dist-info
 mv UNKNOWN-0.0.0-py2.7.egg PyJWT-2.0.0-py2.7.egg
 mv UNKNOWN-0.0.0-py2.7.egg-info PyJWT-2.0.0-py2.7.egg-info
 #并且把easy-install.pth 中的UNKOWN软件包改掉
 cat /usr/lib/python2.7/site-packages/easy-install.pth 
    ./cloud_init-17.1-py2.7.egg
    ./requests-2.25.1-py2.7.egg
    ./elastalert-0.1.20-py2.7.egg
    ./simplejson-3.17.2-py2.7-linux-x86_64.egg
    ./PyJWT-2.0.0-py2.7.egg

dingtalk插件

dingtalk插件的github

安装完成后更改脚本中的执行python版本

#将下面生成的几个脚本第一行指向改为python或者python2.7
 ls /usr/local/bin/elastalert
elastalert                   elastalert-create-index      elastalert-rule-from-kibana  elastalert-test-rule         

head -1 /usr/local/bin/elastalert
#!/usr/bin/python

安装完成后执行报错，根据提示 去安装对应的包

[root /]# elastalert -h 
Traceback (most recent call last):
  File "/usr/local/bin/elastalert", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 637, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 650, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jsonschema>=2.6.0' distribution was not found and is required by elastalert
[root /]# pip install "jsonschema==2.6.0"
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Looking in indexes: http://mirrors.aliyun.com/pypi/simple/
Collecting jsonschema==2.6.0
  Downloading http://mirrors.aliyun.com/pypi/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: functools32; python_version == "2.7" in /usr/lib/python2.7/site-packages (from jsonschema==2.6.0) (3.2.3.post2)
Installing collected packages: jsonschema
  Attempting uninstall: jsonschema
    Found existing installation: jsonschema 2.0.0
    Uninstalling jsonschema-2.0.0:
      Successfully uninstalled jsonschema-2.0.0
Successfully installed jsonschema-2.6.0

日志告警配置及优化

rules/h.com.yml

es_host: 12.20.1.7
es_port: 9200
es_username: prod
es_password: cZ

name: h.com HTTPCode  Elastalert Rule
type: frequency
index:  nginx-access-*
num_events: 30
timeframe:
  minutes: 15
realert:
  minutes: 60
#只检查匹配数量，不下载相关内容
#use_count_query: true
#doc_type: _doc

filter:
- query:
    #query_string对应lucene语法，而不是KQL，需要特别注意
    query_string:
      query: "domain: h.com AND response: [499 TO 599]"

alert_subject: "h.com 499&5xx {} @{}"
alert_subject_args:
- name
- "@timestamp"

alert_text_type: alert_text_only
alert_text: "
HTTP状态码监控告警:     \n
> 请求域名      : {}    \n
> 响应状态码    : {}    \n
> 请求路径举例  : {}    \n
> 日志索引      : {}    \n
> 日志地址      : http://log \n
近15分钟共出现{}次(499 or 5xx)类HTTP状态码,请尽快处理, 谢谢!!"

alert_text_args:
- domain
- response
- request
- fields.log_source
#num_hits命中数值不准确，num_matches更准确
- num_matches


alert:
- "elastalert_modules.dingtalk_alert.DingTalkAlerter" 


dingtalk_msgtype: text

dingtalk_webhooks:
- https://oapi.dingtalk.com/robot/send?access_token=

dingtalk_at_mobiles:
- 18

luence语法
elastalert文档

测试rule方法
elastalert-test-rule –config config.yaml h.com.yml