filebeat_无法收集日志

简介

收集doris的日志，安装file beat后，配置文件按照之前的配置后，发现日志并没有输出到logstash.起初怀疑日志格式不匹配，logstash没有过滤出日志，经过不断尝试 排错 发现多行匹配的问题。

filebeat 配置文件

- type: log
  enabled: true
  paths:
    - /data/doris/be/log/be.INFO
    - /data/doris/fe/log/fe.log
  fields:
    log_source: prod-doris
    type: trd
##  multiline:
##    pattern: '^\[\S+:\S+:\d{2,}] '
 ##   negate: true
 ##   match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["ESip:port"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

可以看到multiline的配置已经被注释掉了，它的作用是在日志报错中将报错信息(java error exception at这种)合并到前一行，不进行拆分。

没有输出到logstash的file beat日志

2021-06-29T16:46:52.981+0800    INFO    [publisher]     pipeline/module.go:97   Beat name: doris3
2021-06-29T16:46:52.982+0800    WARN    beater/filebeat.go:152  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-29T16:46:52.982+0800    INFO    instance/beat.go:422    filebeat start running.
2021-06-29T16:46:52.982+0800    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2021-06-29T16:46:52.982+0800    INFO    registrar/registrar.go:145      Loading registrar data from filebeat/data/registry/filebeat/data.json
2021-06-29T16:46:52.982+0800    INFO    registrar/registrar.go:152      States Loaded from registrar: 1
2021-06-29T16:46:52.982+0800    WARN    beater/filebeat.go:368  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-29T16:46:52.982+0800    INFO    crawler/crawler.go:72   Loading Inputs: 1
2021-06-29T16:46:52.983+0800    INFO    log/input.go:152        Configured paths: [/opt/doris/be/log/be.INFO /opt/doris/fe/log/fe.log]
2021-06-29T16:46:52.983+0800    INFO    input/input.go:114      Starting input of type: log; ID: 11960684066008455970 
2021-06-29T16:46:52.983+0800    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2021-06-29T16:46:52.983+0800    INFO    cfgfile/reload.go:171   Config reloader started
2021-06-29T16:46:52.983+0800    INFO    cfgfile/reload.go:226   Loading of config files completed.
2021-06-29T16:46:52.983+0800    INFO    log/harvester.go:251    Harvester started for file: /fe.log
2021-06-29T16:46:55.977+0800    INFO    add_cloud_metadata/add_cloud_metadata.go:87     add_cloud_metadata: hosting provider type not detected.
2021-06-29T16:47:22.985+0800    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":37}},"total":{"ticks":110,"time":{"ms":117},"value":110},"user":{"ticks":80,"time":{"ms":80}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":7},"info":{"ephemeral_id":"70-d422-bf61-9b07870b8521","uptime":{"ms":30039}},"memstats":{"gc_next":7816288,"memory_alloc":4155600,"memory_total":12330080,"rss":26935296},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":16},"load":{"1":0.6,"15":0.98,"5":0.89,"norm":{"1":0.0375,"15":0.0613,"5":0.0556}}}}}}

可以在日志中看到 没有链接到logstash 就输出的日志内容无变化，30S后进行下一次检查

成功的日志

2021-06-29T16:51:11.157+0800    INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.4.1
2021-06-29T16:51:11.158+0800    INFO    [publisher]     pipeline/module.go:97   Beat name: doris3
2021-06-29T16:51:11.158+0800    WARN    beater/filebeat.go:152  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-29T16:51:11.158+0800    INFO    instance/beat.go:422    filebeat start running.
2021-06-29T16:51:11.158+0800    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2021-06-29T16:51:11.158+0800    INFO    registrar/registrar.go:145      Loading registrar data from filebeat/data/registry/filebeat/data.json
2021-06-29T16:51:11.158+0800    INFO    registrar/registrar.go:152      States Loaded from registrar: 1
2021-06-29T16:51:11.158+0800    WARN    beater/filebeat.go:368  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-29T16:51:11.158+0800    INFO    crawler/crawler.go:72   Loading Inputs: 1
2021-06-29T16:51:11.159+0800    INFO    log/input.go:152        Configured paths: [doris/be/log/be.INFO /opt/doris/fe/log/fe.log]
2021-06-29T16:51:11.159+0800    INFO    input/input.go:114      Starting input of type: log; ID: 7001747164310636462 
2021-06-29T16:51:11.159+0800    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2021-06-29T16:51:11.159+0800    INFO    cfgfile/reload.go:171   Config reloader started
2021-06-29T16:51:11.159+0800    INFO    cfgfile/reload.go:226   Loading of config files completed.
2021-06-29T16:51:11.159+0800    INFO    log/harvester.go:251    Harvester started for file: doris/fe/log/fe.log
2021-06-29T16:51:14.154+0800    INFO    add_cloud_metadata/add_cloud_metadata.go:87     add_cloud_metadata: hosting provider type not detected.
2021-06-29T16:51:14.225+0800    INFO    pipeline/output.go:95   Connecting to backoff(async(tcp://ip:port))
2021-06-29T16:51:14.244+0800    INFO    pipeline/output.go:105  Connection to backoff(async(tcp://ip:port)) established