ELK_日志监控_ElastAlertServer

简介

公司新建了一套ELK7.6.2的，之前对一个项目Elast Alert做了个日志监控报警，新版本的ELK集群用老版本的Elast Alert不支持，所以要将EA升级，想到之前看到kibana集成EA，这次就想用起来。

EA的官方版本支持到ES7.5版本，下面我所使用的EA是非官方版本

安装

因为ELK集群已经搭建完成，并且在正常使用了，所以本篇只记录EAServer的安装调试以及Kibana plugin的集成。

kafka 2.6.0
zookeeper 3.6.2
filebeat 7.6.2
logstash 7.6.2
elasticSearch 7.6.2
kibana 7.6.2
bitsensor/elastalert:3.0.0-beta.0

下面会补充一段看到的ELK EA全docker版本的配置

EAServer的安装

EA镜像构建时候一直报错，就直接使用的别人构建好的镜像

1
2
3
4
git clone https://github.com/bitsensor/elastalert.git
cd elastalert
docker pull bitsensor/elastalert:3.0.0-beta.0


将config/elastalert.yaml 中的配置修改成自己ES集群的地址即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# grep -v "^#" config/elastalert.yaml 
es_host: localhost
es_port: 9200
rules_folder: rules
run_every:
  seconds: 5

buffer_time:
  minutes: 1
writeback_index: elastalert_status
alert_time_limit:
  days: 2
  
  
启动docker:
 docker run -d -p 3030:3030 -p 3333:3333     -v `pwd`/config/elastalert.yaml:/opt/elastalert/config.yaml     -v `pwd`/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml     -v `pwd`/config/config.json:/opt/elastalert-server/config/config.json     -v `pwd`/rules:/opt/elastalert/rules     -v `pwd`/rule_templates:/opt/elastalert/rule_templates     --net="host"     --name elastalert imageid 
 
检查容器启动是否正常，查看容器日志是否有报错
#dokcer ps 
#docker logs -f 容器ID
我的有一些警告信息
02:35:45.423Z ERROR elastalert-server:
    ProcessController:  WARNING:apscheduler.scheduler:Execution of job "ElastAlerter.handle_rule_execution (trigger: interval[0:00:05], next run at: 2020-11-13 02:35:45 UTC)" skipped: maximum number of running instances reached (1)
    
02:36:30.146Z ERROR elastalert-server:
    ProcessController:  WARNING:apscheduler.scheduler:Execution of job "ElastAlerter.handle_rule_execution (trigger: interval[0:00:05], next run at: 2020-11-13 02:36:30 UTC)" skipped: maximum number of running instances reached (1)
    
02:36:31.198Z ERROR elastalert-server:
    ProcessController:  WARNING:apscheduler.scheduler:Execution of job "ElastAlerter.handle_rule_execution (trigger: interval[0:00:05], next run at: 2020-11-13 02:36:31 UTC)" skipped: maximum number of running instances reached (1)

Kibana Plugin安装

官方版本支持到7.5，之后的版本我使用的一位开发人员维护的版本
下载地址：https://github.com/nsano-rururu/elastalert-kibana-plugin/releases
官方版本：https://github.com/bitsensor/elastalert-kibana-plugin/releases/


下载完后通过命令行安装：

1
2
3
4
5
6
7
#安装kibana
/app/kibana/bin/kibana-plugin install file:///opt/elastalert-kibana-plugin-1.2.0-7.6.2.zip 
# 查看kibana已安装的插件
/app/kibana/bin/kibana-plugin list
# 重启kibana:
/app/kibana/node/bin/node /app/kibana/src/cli


看到这个界面安装就算完成了，下面是告警规则的配置，可通过web界面配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
es_host: 127.0.0.1

es_port: 9200
name: credit_rule
type: any
index: logstash-server-*
num_events: 1
timeframe:
  minutes: 1

filter:
- query:
    query_string:
      query: "message:exception"

alert:
- "email"

alert_text: "error information"
smtp_host: smtp.exmail.qq.com
smtp_port: 25
smtp_auth_file: /opt/elastalert/rule_templates/smtp_auth_file.yaml
email_reply_to: elastalert@qq.com
from_addr: elastalert@qq.com

email:
- "oper@qq.com"

全部组件docker版本

下面内容来自nsano-rururu发表于https://github.com/bitsensor/elastalert/issues/132

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
Elasticsearch 7.4.2
Kibana 7.4.2
kibana-elastalert-plugin
bitsensor/elastalert:3.0.0-beta.0

/home/user/docker-wk
|--docker-compose.yml
|--Dockerfiles
|  |--Dockerfile-elastalert
|  |--Dockerfile-kibana
|
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|
|--kibana
|  |--config
|  |  |--kibana.yml
|
|--elastalert
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--config.json
|  |  |--elastalert-test.yaml
|  |  |--elastalert.yaml
|  |--rule_templates
|  |--rules
docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.4.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms128m -Xmx256m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile-kibana
    image: kibana:7.4.2
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile-elastalert
    image: elastalert:3.0.0-beta.0
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
      - kibana
    restart: always
    volumes:
      - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./elastalert/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml
      - ./elastalert/config/config.json:/opt/elastalert-server/config/config.json
      - ./elastalert/rules:/opt/elastalert/rules
      - ./elastalert/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s
es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

# elastalert-kibana-plugin
elastalert-kibana-plugin.serverHost: elastalert
elastalert-kibana-plugin.serverPort: 3030
Dockerfiles/Dockerfile-kibana

FROM docker.elastic.co/kibana/kibana:7.4.2

USER root


RUN /usr/share/kibana/bin/kibana-plugin install https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.4.2.zip --allow-root

USER kibana
Dockerfiles/Dockerfile-elastalert

FROM bitsensor/elastalert:3.0.0-beta.0

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]
elastalert/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2
elastalert/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start
elastalert/config/config.json

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "writeback_index": "elastalert_status"
}
elastalert/config/elastalert-test.yml

# NOTE: This config is used when testing a rule

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2
elastalert/config/elastalert.yml

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2
chmod 777 es/data
chmod 777 elastalert/rules
chmod 777 elastalert/rule_templates
docker-compose up -d
[CORP\sano@a-ngft53r34ong docker-wk]$ docker logs elastalert -f
Giving Elasticsearch at  time to start...
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!

> @bitsensor/elastalert@3.0.0-beta.0 start /opt/elastalert-server
> sh ./scripts/start.sh

14:45:00.311Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
14:45:00.320Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
14:45:00.370Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
14:45:00.431Z  INFO elastalert-server: Router:  Listening for GET request on /.
14:45:00.432Z  INFO elastalert-server: Router:  Listening for GET request on /status.
14:45:00.433Z  INFO elastalert-server: Router:  Listening for GET request on /status/control/:action.
14:45:00.434Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
14:45:00.434Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
14:45:00.439Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id.
14:45:00.440Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id.
14:45:00.441Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id.
14:45:00.442Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
14:45:00.442Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id.
14:45:00.443Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id.
14:45:00.444Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id.
14:45:00.445Z  INFO elastalert-server: Router:  Listening for POST request on /test.
14:45:00.446Z  INFO elastalert-server: Router:  Listening for GET request on /config.
14:45:00.446Z  INFO elastalert-server: Router:  Listening for POST request on /config.
14:45:00.447Z  INFO elastalert-server: Router:  Listening for POST request on /download.
14:45:00.448Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/:type.
14:45:00.449Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
14:45:00.449Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
14:45:00.454Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
14:45:00.454Z  INFO elastalert-server: ProcessController:  Creating index
14:45:06.677Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.4.2
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index elastalert_status created
    Done!
    
14:45:06.677Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
14:45:06.677Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
14:45:06.689Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 215)
14:45:06.694Z  INFO elastalert-server: Server:  Server listening on port 3030
14:45:06.695Z  INFO elastalert-server: Server:  Websocket listening on port 3333
14:45:06.696Z  INFO elastalert-server: Server:  Server started
14:45:08.640Z  INFO elastalert-server:
    ProcessController:  0 rules loaded
    
14:45:24.774Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
14:45:55.364Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
14:46:25.876Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
14:46:56.393Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
14:47:26.901Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
14:47:57.316Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
14:48:27.759Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
^C
[CORP\sano@a-ngft53r34ong docker-wk]$ docker ps

CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                   PORTS                                            NAMES
f7b1f2624b4f        elastalert:3.0.0-beta.0                               "/usr/local/bin/elas…"   4 minutes ago       Up 4 minutes (healthy)   0.0.0.0:3030->3030/tcp, 0.0.0.0:3333->3333/tcp   elastalert
5959a6777f42        kibana:7.4.2                                          "/usr/local/bin/dumb…"   4 minutes ago       Up 4 minutes (healthy)   0.0.0.0:5601->5601/tcp                           kibana
112613b8f089        docker.elastic.co/elasticsearch/elasticsearch:7.4.2   "/usr/local/bin/dock…"   4 minutes ago       Up 4 m

相关网站

问题解决及Dockers安装步骤
ElastAlert
非官方版ElastAlert
ElastAlertForKibanaPlugin\

Naoyuki Sano

ELK
#ELK #ElastAlert #log #Docker #kibana-plugin
ELK_日志监控_ElastAlertServer
https://imwang77.github.io/2020/11/13/ELK_日志监控_ElastAlertServer/
作者
imwang77
发布于
2020年11月13日
更新于
2024年1月3日
许可协议