庚子夜半漏下三刻，众微机突发雪崩！余施大华胄日志天网，救大匠于九死一生

近岁余总督诸省大算每念及往昔跨云汉AWS御集群、收罗流言日志之役至今犹惊心动魄。昔者居方寸机房构筑 ELK 塔台不过如庭院戏沙自娱自乐耳。及至转战天朝上国之云端AWS御 ECS Fargate 与 EKS 战阵面对百千瞬息生灭之“神荚”Pod与“差事”Task始知旧法胶柱鼓瑟决计难行。忆昔某夜漏下三刻三军忽溃。研发大匠趋步来告急索行军日志。余登临台阁控制台视之则神荚因贪墨资粮OOM早已灰飞烟灭片甲不留欲哭无泪莫过如此。今日不坐论泰西之玄理唯倾囊相授直言 ECS 与 EKS 多神荚日志收罗之变法实战。此策历经百战涉险坑无数皆乃碧血换得之真知。势逼其变何故弃旧因循初登云汉之同僚常陷一误区不论青红皂白先于 EKS 营寨中强塞一 Logstash 臃肿之物或以“戴曼护卫”DaemonSet强行钩连宿主机之栈道。然于今之云原生变局中此法必致全军覆没Fargate 御风无形无迹可寻彼乃无碑之神兵Serverless底层躯壳EC2隐匿于虚空欲寻/var/log/pods之栈道无门可入。资粮内讧同室操戈尝有恶战业务洪流猝至收罗日志之卒Agent暴食机杼CPU竟将身侧核心业务之神荚驱逐出境。此正所谓“逐麋鹿而丧其家”本末倒置。聚散无常收罗莫及军阵暴涨收罗未及置办军阵暴缩而存诸内存之残墨随风蒸发。由是余等痛下决心尽毁旧制重整乾坤。其核心要奥唯“解耦”二字而已。ECS Fargate 阵线FireLens 乃天子赤子若尔等所御者乃 ECS Fargate听余一言切莫于业务神舟内自造漏卮以导日志亦休得折腾 Sidecar 共用卷之繁文缛节。云汉官家特为 ECS 赐造神器名曰FireLens。此物本是 Fluent Bit 或 Fluentd 之锦衣外袍与 ECS 差事之法度契合得天衣无缝。余等彼时擢用 Fluent Bit 为三军基石。何不用 Fluentd盖因 Fluentd 乃 Ruby 蛮语所制吞噬内存之状极其骇人。而 Fluent Bit 乃 C 语言淬炼精悍敏捷居万马奔腾之势所耗资粮不过十数兆省下者皆乃库银真金。核心画卷与避坑秘录于 ECS 差事法度Task Definition中安设 FireLens实乃于业务神舟之侧垂挂一掌灯法船。然此处隐一巨坑。诸多同僚按图索骥事毕却见 CloudWatch 或 S3 幽冥之海中寂然无声死活不见日志。按部班排查良久始知乃awslogs与firelens二驭者互搏气血冲撞。且看余等当年精简之法度图谱此即筑宇之墨线JSON{ containerDefinitions: [ { name: log_router, image: amazon/aws-for-fluent-bit:stable, essential: true, firelensConfiguration: { type: fluentbit, options: { enable-ecs-log-metadata: true } }, logConfiguration: { logDriver: awslogs, options: { awslogs-group: firelens-container-errors, awslogs-region: ap-southeast-1, awslogs-stream-prefix: firelens } } }, { name: app-service, image: your-app-image:latest, essential: true, logConfiguration: { logDriver: awsfirelens, options: { Name: cloudwatch, region: ap-southeast-1, log_group_name: /aws/ecs/app-cluster, log_stream_name: app-logs-$(ecs_task_id), auto_create_group: true } } } ] }诸君拭目业务神舟app-service之logDriver必得遥尊awsfirelens。且看那log_stream_name尾缀赫然缀以$(ecs_task_id)变数此乃绝妙之笔若无此变数诸路神荚之墨迹尽皆涌入同一溪流。洪流至时CloudWatch 阀门API 限制必将厉声断流致日志溃散殆尽。EKSK8s多神荚阵线Fluent Bit 与 IRSA 之铁血杀伐及至 EKS 疆场变阵更为繁复亦最易使人目眩神迷。余等今日所奉之王道架构乃是Fluent Bit (戴曼护卫) - Amazon Kinesis Data Firehose - Amazon S3 / OpenSearch。何故居中横插一尊 Firehose 巨鼎直入 OpenSearch 或 S3 岂不痛快非也。诸君听真若贵司业务日产数武TB之墨迹直教数百神荚之 Fluent Bit 纷乱叩击 S3 龙门则 S3 问卷之资API 费用翌月定教诸君卷席除名。Firehose 之功在于聚沙成塔、御压缓冲积满五兆5MB或候六十息方倾泻入 S3 库房既省帑银又显儒雅。权柄警示切莫乞灵于群节点之 IAM 旧袍于安设 Fluent Bit 之前须棒喝一安全隐患。余阅寰宇诸多公司之 EKS 营寨皆将染指 CloudWatch 或 S3 之权柄径直加诸 EKS 节点底层 EC2之 IAM 袍服上。此举无异于将阖里之锁钥高悬于辕门。寨中但凡有一心怀鬼胎之行军卒皆可透过乾坤地址Metadata窃得此袍进而将贵司 S3 宝库焚毁一空。必得奉行IRSA (IAM Roles for Service Accounts)。将 IAM 权柄法袍遥绑定于 K8s 之“役事账房”ServiceAccount之身。简而言之布阵唯三步莫嫌繁复此乃保命之圭臬于 AWS IAM 秘阁中勒石立盟特许往 CloudWatch/Firehose 录入墨迹。铸一 IAM 权柄法袍遥信尔等 EKS 营寨之 OIDC 执符者。于 K8s 之中辟一“役事账房”钤印法袍之流派Annotation。YAMLapiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit-sa namespace: logging annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/EKS-FluentBit-LogRole如是Fluent Bit 游走之时独奉此特定法袍之金牌临时 Token与云汉钦差交涉稳若泰山。Fluent Bit 贝叶经ConfigMap如何运笔方不至崩裂尔后乃是全军之重器Fluent Bit 之法度。诸多同僚尝陷于日志残缺、多行墨迹如 Java 之 Exception 绝命堆栈被斩为寸断之苦境。日日听闻研发大匠怨声载道“此恶报缺首断足如何按图索骥”此乃“句读编排器”Parser未得其法耳。余特将线上海战所悬之贝叶经核心定规录于下方尤以规训多行墨迹、斥退无用微恙检查健康检查之法度为要YAMLapiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: logging data: fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 50MB Skip_Long_Lines On Refresh_Interval 10 [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Keep_Log Off K8S-Logging.Parser On K8S-Logging.Exclude On # 要义屏退 Nginx 聒噪之微恙检查还府库以清宁空间 [FILTER] Name grep Match kube.* Exclude log kube-probe|HealthCheck [OUTPUT] Name cloudwatch_logs Match kube.* region ap-southeast-1 log_group_name /eks/prod-cluster-logs log_stream_prefix k8s- auto_create_group true其中隐匿三般细节DB /var/log/flb_kube.db必得置一 SQLite 账本以录寻墨之位Position。如若不然Fluent Bit 但凡惊变重启必将旧墨重抄一遍致案牍连篇研发势必与尔论剑。Mem_Buf_Limit 50MB乃是对纳墨闸口强行限流。若下游云汉府库微有凝滞Fluent Bit 必拼死将墨汁纳于腹中内存若无此限制彼必因腹胀OOM而遭 K8s 天雷殛之。终章总结与行军碎碎念演练诸般阵法终得血泪教训数条墨迹留存之策Retention新辟之 CloudWatch 墨池公家默认“万寿无疆”。昔有一愚卒未察于生产大阵中驰骋三月数百神荚日夜狂喷墨汁及至月底觇视账单CloudWatch 存墨之资竟僭越 EC2 铁骑几近革职。切记将其拘于七日或十四日之限不常用者借生命周期沉降至 S3 寒冰冷灶中归档。多行句读Multiline倘系 Java 或 Python 之属速于 INPUT 或 FILTER 阵线加持multiline.parser。否则那长蛇般的绝命堆栈化作数十条残笔呈递非但面目可憎亦将索引天书撑爆。大抵云原生之御史皆如是也。风平浪静之下暗礁密布。天下安有万全之阵唯适宜自家兵刃与军费预算者方为上策。今夕姑且言尽于此。诸君于折腾 AWS ECS 或 EKS 日志之时若遇权柄胶着、或墨迹延宕等诡异妖变愿乞言于评论区吾辈当秉烛共研之。诸君若觉此文击中痛处或助尔等避开疆场之暗枪切莫吝惜双指赞、在看、转发三连走一波生产一线之玄机吾辈下期再行切磋。