filebeat上传日志到ES，日志转换失败，记录新方法

因为nginx日志字段有所变化导致filebeat 上传日志信息ES默认分词识别失败处理方法思路## 解析只能在ES只能进行 ## Filebeat 只发送原始日志不解析没有兼容性问题 ## 所有解析在 ES 中完成使用成熟的 grok 和 user_agent 处理器步骤1在 Kibana Dev Tools 中创建 Pipeline# 将你所需要的字段全部重新手动定义一下 方式一 PUT _ingest/pipeline/nginx-access-parser-diquzhan { description: Parse nginx access log using dissect (performance optimized), processors: [ { dissect: { field: message, pattern: %{x_forwarded_for} - %{remote_user} [%{time_local}] \%{method} %{url} HTTP/%{version}\ %{status} %{bytes} \%{referrer}\ static%{static_hit} cookie\%{cookie}\ fetch_dest%{fetch_dest} \%{user_agent_original}\ \%{client_ip}\ } }, { rename: { field: x_forwarded_for, target_field: source.ip, ignore_failure: true } }, { rename: { field: method, target_field: http.request.method, ignore_failure: true } }, { rename: { field: url, target_field: url.original, ignore_failure: true } }, { rename: { field: version, target_field: http.version, ignore_failure: true } }, { rename: { field: status, target_field: http.response.status_code, ignore_failure: true } }, { rename: { field: bytes, target_field: http.response.body.bytes, ignore_failure: true } }, { rename: { field: referrer, target_field: http.request.referrer, ignore_failure: true } }, { rename: { field: user_agent_original, target_field: user_agent.original, ignore_failure: true } }, { rename: { field: client_ip, target_field: client.ip, ignore_failure: true } }, { user_agent: { field: user_agent.original, target_field: user_agent, ignore_failure: true } }, { date: { field: time_local, target_field: timestamp, formats: [ dd/MMM/yyyy:HH:mm:ss Z ], timezone: Asia/Shanghai } }, { remove: { field: [ message, remote_user, time_local, static_hit, cookie, fetch_dest ], ignore_missing: true } } ] } 方式二 PUT _ingest/pipeline/nginx-access-parser { description: Parse custom nginx access log, processors: [ { grok: { field: message, patterns: [ %{IPORHOST:x_forwarded_for} - %{USER:remote_user} \\[%{HTTPDATE:time_local}\\] \%{WORD:method} %{DATA:url} HTTP/%{NUMBER:version}\ %{NUMBER:status} %{NUMBER:bytes} \%{DATA:referrer}\ static%{DATA:static_hit} cookie\%{DATA:cookie}\ fetch_dest%{DATA:fetch_dest} \%{DATA:user_agent_original}\ \%{DATA:client_ip}\ ] } }, { rename: { field: x_forwarded_for, target_field: source.ip, ignore_failure: true } }, { rename: { field: method, target_field: http.request.method, ignore_failure: true } }, { rename: { field: url, target_field: url.original, ignore_failure: true } }, { rename: { field: version, target_field: http.version, ignore_failure: true } }, { rename: { field: status, target_field: http.response.status_code, ignore_failure: true } }, { rename: { field: bytes, target_field: http.response.body.bytes, ignore_failure: true } }, { rename: { field: referrer, target_field: http.request.referrer, ignore_failure: true } }, { rename: { field: user_agent_original, target_field: user_agent.original, ignore_failure: true } }, { rename: { field: client_ip, target_field: client.ip, ignore_failure: true } }, { user_agent: { field: user_agent.original, target_field: user_agent, ignore_failure: true } }, { date: { field: time_local, target_field: timestamp, formats: [dd/MMM/yyyy:HH:mm:ss Z], timezone: Asia/Shanghai } }, { remove: { field: [message, remote_user, time_local, static_hit, cookie, fetch_dest], ignore_missing: true } } ] }这两个有本质区别绝对不能混用 对比方式一dissect方式二grok处理器dissectgrok工作原理按位置切分正则匹配Cookie 处理不受内容影响超长 Cookie 会导致性能雪崩丢包风险无高生产环境✅ 安全❌ 不能用步骤2Filebeat 配置# dit_filebeat.yml (自定义名称) path.data: /usr/local/filebeat/data-dit filebeat.inputs: - type: filestream enabled: true paths: - /var/log/nginx/access-www.log output.elasticsearch: hosts: [http://192.168.0.200:9200] index: nginx-access pipeline: nginx-access-parser setup.template.name: nginx-access-template setup.template.pattern: nginx-access setup.template.settings: index.number_of_replicas: 0 logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7 permissions: 0640步骤三# 测试 Filebeat 配置 /usr/local/filebeat/filebeat test config -c /usr/local/filebeat/dit_filebeat.yml Config OK删除旧索引可忽略在 Kibana Dev Tools 中执行DELETE nginx-access步骤四# 重启 pkill -9 -f dit_filebeat.yml /usr/local/filebeat/filebeat -c /usr/local/filebeat/dit_filebeat.yml # 查看日志 ll -ltrh /var/log/filebeat/filebeat-20260* -rw-r-----. 1 root root 11M 7月 7 15:29 /var/log/filebeat/filebeat-20260707-25.ndjson -rw-r-----. 1 root root 9.1M 7月 7 15:31 /var/log/filebeat/filebeat-20260707-26.ndjson -rw-r-----. 1 root root 6.4K 7月 7 15:32 /var/log/filebeat/filebeat-20260707-28.ndjson -rw-r-----. 1 root root 28K 7月 7 15:34 /var/log/filebeat/filebeat-20260707-27.ndjson -rw-r-----. 1 root root 28K 7月 7 15:36 /var/log/filebeat/filebeat-20260707-29.ndjson -rw-r-----. 1 root root 6.4K 7月 7 15:37 /var/log/filebeat/filebeat-20260707-31.ndjson -rw-r-----. 1 root root 31K 7月 7 15:40 /var/log/filebeat/filebeat-20260707-30.ndjson -rw-r-----. 1 root root 14K 7月 7 15:40 /var/log/filebeat/filebeat-20260707-32.ndjson tail -f /var/log/filebeat/filebeat-20260707-32.ndjson {log.level:info,timestamp:2026-07-07T15:41:14.1760800,log.logger:monitoring,log.origin:{function:github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot, file.name:log/log.go,file.line:192},message:Non-zero metrics in the last 30s,service.name:filebeat,monitoring:{metrics:{beat:{cgroup:{cpu:{id:session-209051.scope},memory:{id:session-209051. scope,mem:{usage:{bytes:1594318848}}}},cpu:{system:{ticks:240,time:{ms:240}},total:{ticks:1480,time:{ms:1480},value:1480},user:{ticks:1240,time:{ms:1240}}},handles:{limit:{hard:655350,soft:655350},open:11},info:{ephemeral_id: 03b62737-2108-41ff-a02e-785ddf846ca4,name:filebeat,uptime:{ms:30347},version:9.0.1}, memstats:{gc_next:53029370,memory_alloc:28552272,memory_sys:82665752,memory_total:142040184,rss:178577408},runtime:{goroutines:33}},filebeat:{harvester:{open_files:1,running:1,started:1}},libbeat:{config:{module:{running:0}},output: {events:{acked:7923,active:0,batches:5,total:7923},read:{bytes:4379,errors:4},type:elasticsearch,write:{bytes:1950702,latency:{histogram:{count:5,max:480,mean:410.4,median:403, min:370,p75:447,p95:480,p99:480,p999:480,stddev:37.918860742379906}}}},pipeline:{clients:1,events: {active:866,published:8789,retry:1600,total:8789},queue:{acked:7923,added:{bytes:13352235,events:8789},consumed:{bytes:12066047,events:7923},filled:{bytes:1286188,events:866,pct:0.270625},max_bytes:0,max_events:3200,removed: {bytes:12066047,events:7923}}}},registrar:{states:{current:0}},system:{cpu:{cores:80},load:{1:1.33,15:1.3,5:1.35,norm:{1:0.0166,15:0.0163,5:0.0169}}}},ecs.version:1.6.0}}一切正常可以看出访问状态次数了。