SkyWalking OAP 查询链路追踪Query Tracing调试指南定位指标、Trace、拓扑与日志查询性能瓶颈【免费下载链接】skywalkingAPM, Application Performance Monitoring System项目地址: https://gitcode.com/gh_mirrors/sky/skywalkingSkyWalking OAP 后端自带一套「查询链路追踪Query Tracing」能力可以在一次查询请求MQE 指标、Trace、Zipkin、Topology 拓扑、Log 日志中自动记录 OAP 内部的执行调用链每个环节的耗时、错误信息甚至底层存储Elasticsearch / BanyanDB的真实请求与响应内容。本文基于 query-tracing.md 展开结合 debugging-query-plugin 与 server-core 源码完整讲解如何通过 HTTP REST 接口与 GraphQL 两种方式开启查询追踪、解读 DebuggingTrace 结构并给出可直接复制的 curl 与 GraphQL 示例帮助你快速诊断 SkyWalking 后端自身的查询性能问题。一、什么是 OAP Query TracingSkyWalking OAPObservability Analysis Platform对外提供指标metrics、Trace、日志log、拓扑topology等查询能力这些查询最终会落到存储层Elasticsearch、BanyanDB、MySQL 等执行。当用户反馈「查询很慢」或「某条查询不符合预期」时仅看最终结果很难定位瓶颈——究竟是 MQE 语法解析慢、聚合计算慢还是存储层响应慢、返回数据过大Query Tracing 正是为此设计它把一次 OAP 查询请求的内部执行过程按照类似分布式追踪的 Span 树结构记录下来并随查询结果一起返回给调用方。从源码看这一机制的载体是DebuggingTraceContextDebuggingTraceContext.java每个查询线程通过ThreadLocalDebuggingTraceContext TRACE_CONTEXT保存上下文createSpan(operation)创建并压栈一个DebuggingSpan记录System.nanoTime()级别的起止时间纳秒stopSpan(span)负责计算耗时并弹栈天然形成父子层级stopTrace()收尾生成整条 trace 的duration。public DebuggingSpan createSpan(String operation) { DebuggingSpan span new DebuggingSpan(spanIdGenerator, operation); if (debug) { span.setStartTime(System.nanoTime()); DebuggingSpan parentSpan spanStack.isEmpty() ? null : spanStack.peek(); if (parentSpan ! null) { span.setParentSpanId(parentSpan.getSpanId()); } else { span.setParentSpanId(-1); } spanStack.push(span); execTrace.addSpan(span); } return span; }这意味着只有开启 debug 的查询才会产生追踪开销createSpan/stopSpan内部有if (debug)守卫日常查询不会受任何影响。1.1 Trace 结构一次查询追踪由根 Trace 与若干 Span 组成字段定义见 DebuggingTrace.java字段说明traceId本次追踪的唯一 IDUUID.randomUUID()生成condition本次查询的完整条件描述表达式、实体、时间范围等startTime追踪开始时间纳秒endTime追踪结束时间纳秒duration追踪总耗时纳秒spans本次追踪包含的所有 Span1.2 Span 结构字段说明spanIdSpan 的唯一 IDparentSpanId父 Span ID根 Span 为 -1operationSpan 的操作名如 MQE query、MQE syntax analysisstartTimeSpan 开始时间纳秒相对时间依赖具体实现与环境endTimeSpan 结束时间纳秒durationSpan 耗时纳秒msgSpan 附加信息可包含请求条件、数据库响应、BanyanDB 内部 trace 的 TagserrorSpan 出错时的错误信息二、通过 HTTP REST API 调试Query Tracing 服务内置于 OAP 的 rest server所有调试接口统一走 HTTP GEThttp://{core restHost}:{core restPort}/debugging/query/...restHost/restPort 即 OAP 的 HTTP 监听地址与端口默认0.0.0.0:12800可在application.yml的core.restPort配置。所有调试接口都由 DebuggingHTTPHandler.java 实现它内部直接复用了 GraphQL 层的MetricsExpressionQuery、TraceQuery、TopologyQuery、LogQuery以及 Zipkin 的ZipkinQueryHandler因此返回内容与 GraphQL 查询等价并额外以YAML 格式输出调试追踪信息通过transToYAMLString序列化。这也说明 Query Tracing 不是另起炉灶的「假查询」而是对真实查询链路的旁路观测。另外该插件还提供GET /debugging/config/dump用于导出启动配置serverStatusService.dumpBootingConfigurations与查询追踪配合可用于排查配置类问题。2.1 追踪 MQE 执行/debugging/query/mqeURLhttp://{core restHost}:{core restPort}/debugging/query/mqe?{parameters}参数字段说明必填dumpDBRsp是否把数据库响应 dump 到 span 的 msg 中支持 Elasticsearch 与 BanyanDB否默认 falseexpressionMQE 查询表达式是startTime查询开始时间是endTime查询结束时间是step查询步长SECOND/MINUTE/HOUR/DAY 等是service服务名是serviceLayer服务层名如 GENERAL、MESH是serviceInstance服务实例名否endpoint端点名否process进程名否destService目标服务名否destServiceLayer目标服务层名否destServiceInstance目标服务实例名否destEndpoint目标端点名否destProcess目标进程名否时间与 step 参数遵循 Duration 格式start/end为yyyy-MM-dd HHmm或yyyy-MM-dd等格式step为枚举值。示例追踪一条 MQE 表达式avg(service_sla)查询 2024-07-03 一天、步长 DAY、服务mock_a_service、服务层GENERAL并开启dumpDBRsptruecurl -X GET http://127.0.0.1:12800/debugging/query/mqe?dumpDBRsptrueexpressionavg(service_sla)startTime2024-07-03endTime2024-07-03stepDAYservicemock_a_serviceserviceLayerGENERAL响应同时包含查询结果与debuggingTrace信息type: SINGLE_VALUE results: - metric: labels: [] values: - id: null value: 10000 traceID: null doubleValue: 10000.0 emptyValue: false error: null debuggingTrace: traceId: 4f972417-c543-4f7d-a3f1-f5e694cfeb2b condition: Expression: avg(service_sla), Entity: Entity(scopenull, serviceNamemock_a_service,\ \ normaltrue, serviceInstanceNamenull, endpointNamenull, processNamenull,\ \ destServiceNamenull, destNormalnull, destServiceInstanceNamenull, destEndpointNamenull,\ \ destProcessNamenull), Duration: Duration(start2024-07-03, end2024-07-03,\ \ stepDAY) startTime: 115828803080350 endTime: 115828877400237 duration: 74319887 rootSpan: spanId: 0 parentSpanId: -1 operation: MQE query startTime: 115828803110686 endTime: 115828877396756 duration: 74286070 msg: null error: null childSpans: - spanId: 1 parentSpanId: 0 operation: MQE syntax analysis startTime: 115828803699331 endTime: 115828805015745 duration: 1316414 msg: null error: null childSpans: [] - spanId: 2 parentSpanId: 0 operation: MQE Aggregation OP: avg(service_sla) startTime: 115828805052267 endTime: 115828876877134 duration: 71824867 msg: null error: null childSpans: - spanId: 3 parentSpanId: 2 operation: MQE Metric OP: service_sla startTime: 115828805209453 endTime: 115828875634953 duration: 70425500 msg: null error: null childSpans: ...从上面这棵 Span 树可以清晰看出耗时分布整个查询耗时约 74.3ms其中 MQE 语法分析仅约 1.3ms聚合算子avg(service_sla)占了约 71.8ms而真正的瓶颈是底层指标读取MQE Metric OP: service_sla约 70.4ms——问题直指存储层而非表达式解析。这些 Span 的来源可以从源码得到印证MQEVisitorBase.java 中每类 MQE 操作Binary OP、Aggregation OP、Mathematical OP、TopN OP、Logical OP、Trend OP、Sort OP 等都会调用traceContext.createSpan(...)MetricsQueryService.java 则在readMetricsValues/readLabeledMetricsValues处创建Query Service: readMetricsValues等 Span。BanyanDB 原生存储下的内部执行 trace注意如果使用 SkyWalking 原生存储 BanyanDBdebuggingTrace会进一步包含 BanyanDB 内部的执行追踪信息例如 measure 查询、index scan 的具体计划此时msg中携带了大量底层细节... childSpans: - spanId: 7 parentSpanId: 6 operation: BanyanDB: measure-grpc startTime: 1720059017222584700 endTime: 1720059017223492400 duration: 907700 msg: [Tag(keyrequest, value{\groups\:[\measure-default\], \\ name\:\service_sla_day\, \timeRange\:{\begin\:\2024-07-02T16:00:00Z\\ , \end\:\2024-07-03T16:00:00Z\}, \criteria\:{\condition\\ :{\name\:\entity_id\, \op\:\BINARY_OP_EQ\, \value\:{\\ str\:{\value\:\bW9ja19hX3NlcnZpY2U.1\}}}}, \tagProjection\\ :{\tagFamilies\:[{\name\:\storage-only\, \tags\:[\entity_id\\ ]}]}, \fieldProjection\:{\names\:[\percentage\]}, \trace\\ :true})] error: null childSpans: - spanId: 8 parentSpanId: 7 operation: BanyanDB:>curl -X GET http://127.0.0.1:12800/debugging/query/trace/queryBasicTraces?startTime2024-06-26%200900endTime2024-06-26%200915stepMINUTEservicemock_a_serviceserviceLayerGENERALserviceInstancemock_a_service_instancetraceStateALLqueryOrderBY_DURATIONpageNum1pageSize15tagshttp.status_code%3D404%2Chttp.method%3Dget响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪traces: ... debuggingTrace: ...注意tags参数在源码中是按逗号切分、再按等号切分组装成Tag列表的因此 URL 中需要将,编码为%2C、编码为%3D。queryBasicTraces的 Span 由 TraceQueryService.java 中的Query Service: queryBasicTraces创建。queryTraceURLhttp://{core restHost}:{core restPort}/debugging/query/trace/queryTrace?{parameters}参数字段说明必填traceId要查询的 Trace ID是示例curl -X GET http://127.0.0.1:12800/debugging/query/trace/queryTrace?traceId8211a1d1-de0f-4485-8766-c88866a8f034响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪spans: ... debuggingTrace: ...2.3 追踪 Zipkin Trace 查询Zipkin API /api/v2/tracesURLhttp://{core restHost}:{core restPort}/debugging/query/zipkin/api/v2/traces?{parameters}参数字段说明必填serviceName服务名否remoteServiceName远端服务名否spanNameSpan 名否annotationQuery注解查询否minDurationTrace 最小耗时否maxDurationTrace 最大耗时否endTs查询结束时间戳默认当前时间戳否lookback回看窗口默认8640000024 小时否limit返回数量上限默认10否所有参数与 Zipkin 原生 API/api/v2/traces保持一致。示例curl -X GET http://127.0.0.1:12800/debugging/query/zipkin/api/v2/traces?serviceNamefrontend响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪traces: ... debuggingTrace: .../api/v2/trace/{traceId}URLhttp://{core restHost}:{core restPort}/debugging/query/zipkin/api/v2/trace?{parameters}参数字段说明必填traceId要查询的 Trace ID是示例curl -X GET http://127.0.0.1:12800/debugging/query/zipkin/api/v2/trace?traceIdfcb10b060c6b2492响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪spans: ... debuggingTrace: ...从源码可以看到Zipkin 调试接口的实现方式略有不同它先在DebuggingTraceContext中手动设置 condition再调用真实的ZipkinQueryHandler基于默认的 ZipkinQueryConfig执行查询最后把内嵌的 exec trace 一并返回见 DebuggingHTTPHandler.java 中queryZipkinTraces与getZipkinTraceById的try/finally结构——finally中stopTrace()并清理TRACE_CONTEXT保证线程上下文不会泄漏。2.4 追踪拓扑查询getGlobalTopologyURLhttp://{core restHost}:{core restPort}/debugging/query/topology/getGlobalTopology?{parameters}参数字段说明必填startTime查询开始时间是endTime查询结束时间是step查询步长是serviceLayer服务层名否示例curl -X GET http://127.0.0.1:12800/debugging/query/topology/getGlobalTopology?startTime2024-07-03endTime2024-07-03stepDAYserviceLayerGENERAL响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪nodes: ... calls: ... debuggingTrace: ...getServicesTopologyURLhttp://{core restHost}:{core restPort}/debugging/query/topology/getServicesTopology?{parameters}参数字段说明必填startTime查询开始时间是endTime查询结束时间是step查询步长是serviceLayer服务层名是services服务名列表逗号分隔如mock_a_service, mock_b_service是示例curl -X GET http://127.0.0.1:12800/debugging/query/topology/getServicesTopology?startTime2024-07-03endTime2024-07-03stepDAYserviceLayerGENERALservicesmock_a_service%2Cmock_b_service响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪nodes: ... calls: ... debuggingTrace: ...getServiceInstanceTopologyURLhttp://{core restHost}:{core restPort}/debugging/query/topology/getServiceInstanceTopology?{parameters}参数字段说明必填startTime查询开始时间是endTime查询结束时间是step查询步长是clientService客户端服务名是serverService服务端服务名是clientServiceLayer客户端服务层名是serverServiceLayer服务端服务层名是示例curl -X GET http://127.0.0.1:12800/debugging/query/topology/getServiceInstanceTopology?startTime2024-07-03endTime2024-07-03stepDAYclientServicemock_a_serviceserverServicemock_b_serviceclientServiceLayerGENERALserverServiceLayerGENERAL响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪nodes: ... calls: ... debuggingTrace: ...getEndpointDependenciesURLhttp://{core restHost}:{core restPort}/debugging/query/topology/getEndpointDependencies?{parameters}参数字段说明必填startTime查询开始时间是endTime查询结束时间是step查询步长是service服务名是serviceLayer服务层名是endpoint端点名是示例curl -X GET http://127.0.0.1:12800/debugging/query/topology/getEndpointDependencies?startTime2024-07-03endTime2024-07-03stepDAYservicemock_a_serviceserviceLayerGENERALendpoint%2Fdubbox-case%2Fcase%2Fdubbox-rest%2F404-test响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪nodes: ... calls: ... debuggingTrace: ...getProcessTopologyURLhttp://{core restHost}:{core restPort}/debugging/query/topology/getProcessTopology?{parameters}参数字段说明必填startTime查询开始时间是endTime查询结束时间是step查询步长是service服务名是serviceLayer服务层名是instance实例名是示例curl -X GET http://127.0.0.1:12800/debugging/query/topology/getProcessTopology?startTime2024-07-03endTime2024-07-03stepDAYservicemock_a_serviceserviceLayerGENERALinstancemock_a_service_instance响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪nodes: ... calls: ... debuggingTrace: ...拓扑类查询的 Span 埋点分布在各拓扑构建器中例如 TopologyQueryService.java 中的Query Service: getGlobalTopology、Query Service: getServiceInstanceTopology等以及 ServiceTopologyBuilder.java 中的Build service topology、ServiceInstanceTopologyBuilder.java 的Build service instance topology、EndpointTopologyBuilder.java 的Build endpoint topology、ProcessTopologyBuilder.java 的Build process topology。借此可以区分「拓扑计算耗时」与「底层存储查询耗时」。2.5 追踪日志查询queryLogsURLhttp://{core restHost}:{core restPort}/debugging/query/log/queryLogs?{parameters}参数字段说明必填startTime查询开始时间是除非 traceId 非空endTime查询结束时间是除非 traceId 非空step查询步长是除非 traceId 非空service服务名否需配合 serviceLayerserviceLayer服务层名否serviceInstance服务实例名否需配合 serviceendpoint端点名否需配合 servicetraceIdTrace ID否segmentIdSegment ID否需配合 traceIdspanIdSpan ID否需配合 traceIdqueryOrder查询结果排序ASC、DES默认DES否tags日志标签过滤格式key1value1,key2value2否pageNum结果页码是pageSize每页大小是keywordsOfContent日志内容关键词keyword1,keyword2否excludingKeywordsOfContent排除关键词keyword1,keyword2否示例curl -X GET http://127.0.0.1:12800/debugging/query/log/queryLogs?servicee2e-service-providerserviceLayerGENERALstartTime2024-07-09endTime2024-07-09stepDAYpageNum1pageSize15queryOrderASCtagslevel%3DINFO响应包含查询结果与debuggingTrace信息结构同 MQE 查询追踪logs: ... debuggingTrace: ...从 DebuggingHTTPHandler.java 的queryLogs实现可以看到一个细节当traceId为空时必须提供startTime/endTime/step否则直接返回错误提示字符串按traceId查询时走TraceScopeCondition可叠加segmentId/spanId精确定位日志。日志查询的 Span 由 LogQueryService.java 中的Query Service: queryLogs创建。三、通过 GraphQL 调试除了 REST 调试接口Query Tracing 也内置于 GraphQL API 中。所有 GraphQL 查询接口都新增了一个debug: Boolean参数设为true即可开启查询追踪并在返回类型中追加debuggingTrace: DebuggingTrace字段。GraphQL 相关类型定义可参考 query-protocol.md 与 query-graphql-plugin 源码。3.1 追踪 MQE 执行Metrics V3 APIsBundle APIMetrics V3 APIsextend type Query { ... # Param, if debug is true will enable the query tracing and return DebuggingTrace in the ExpressionResult. # Param, if dumpDBRsp is true the database response will dump into the DebuggingTrace span message. execExpression(expression: String!, entity: Entity!, duration: Duration!, debug: Boolean, dumpDBRsp: Boolean): ExpressionResult! }type ExpressionResult { ... debuggingTrace: DebuggingTrace }示例通过 GraphQL 查询指标并开启追踪。GraphQL 端点地址为http://127.0.0.1:12800/graphql{ execExpression(expression: avg(service_sla), entity: {serviceName: mock_a_service, normal: true}, duration: {start: 2024-07-03, end: 2024-07-03, step: DAY}, debug: true, dumpDBRsp: true) { type error results { metric { labels { key value } } values { id value traceID } } debuggingTrace { traceId condition startTime endTime duration spans { spanId parentSpanId operation startTime endTime duration msg error } } } }响应包含查询结果与debuggingTrace信息{ data: { execExpression: { type: SINGLE_VALUE, error: null, results: [ { metric: { labels: [] }, values: [ { id: null, value: 10000, traceID: null } ] } ], debuggingTrace: { traceId: 3116ffe3-ee9c-4047-9f22-c135c237aad5, condition: Expression: avg(service_sla), Entity: Entity(scopenull, serviceNamemock_a_service, normaltrue, serviceInstanceNamenull, endpointNamenull, processNamenull, destServiceNamenull, destNormalnull, destServiceInstanceNamenull, destEndpointNamenull, destProcessNamenull), Duration: Duration(start2024-07-03, end2024-07-03, stepDAY), startTime: 117259274324665, endTime: 117259279847720, duration: 5523055, spans: [ { spanId: 0, parentSpanId: -1, operation: MQE query, startTime: 117259274328719, endTime: 117259279846559, duration: 5517840, msg: null, error: null }, { spanId: 1, parentSpanId: 0, operation: MQE syntax analysis, startTime: 117259274333084, endTime: 117259274420159, duration: 87075, msg: null, error: null }, { spanId: 2, parentSpanId: 0, operation: MQE Aggregation OP: avg(service_sla), startTime: 117259274433533, endTime: 117259279812549, duration: 5379016, msg: null, error: null }, ... ] } } } }注意与 REST 接口相同若使用 BanyanDB 存储debuggingTrace.spans中会包含 BanyanDB 内部执行 trace如 measure-grpc 请求、IndexScan 执行计划等... { spanId: 7, parentSpanId: 6, operation: BanyanDB: measure-grpc, startTime: 1720060447687765300, endTime: 1720060447688830200, duration: 1064900, msg: [Tag(keyrequest, value{\groups\:[\measure-default\], \name\:\service_sla_day\, \timeRange\:{\begin\:\2024-07-02T16:00:00Z\, \end\:\2024-07-03T16:00:00Z\}, \criteria\:{\condition\:{\name\:\entity_id\, \op\:\BINARY_OP_EQ\, \value\:{\str\:{\value\:\bW9ja19hX3NlcnZpY2U.1\}}}}, \tagProjection\:{\tagFamilies\:[{\name\:\storage-only\, \tags\:[\entity_id\]}]}, \fieldProjection\:{\names\:[\percentage\]}, \trace\:true})], error: null }, { spanId: 8, parentSpanId: 7, operation: BanyanDB:># Param, if debug is true will enable the query tracing and return DebuggingTrace in the result. extend type Query { # Search segment list with given conditions queryBasicTraces(condition: TraceQueryCondition, debug: Boolean): TraceBrief # Read the specific trace ID with given trace ID queryTrace(traceId: ID!, debug: Boolean): Trace ... }# The list of traces type TraceBrief { ... #For OAP internal query debugging debuggingTrace: DebuggingTrace } # The trace represents a distributed trace, includes all segments and spans. type Trace { ... #For OAP internal query debugging debuggingTrace: DebuggingTrace }用法与 MQE 查询追踪一致按 GraphQL 协议与语法查询结果并把debug参数置为true即可在返回中取得debuggingTrace信息。3.3 追踪拓扑查询GraphQLBundle APITopology# Param, if debug is true will enable the query tracing and return DebuggingTrace in the result. extend type Query { # Query the global topology # When layer is specified, the topology of this layer would be queried getGlobalTopology(duration: Duration!, layer: String, debug: Boolean): Topology # Query the topology, based on the given service getServiceTopology(serviceId: ID!, duration: Duration!, debug: Boolean): Topology # Query the topology, based on the given services. # #getServiceTopology could be replaced by this. getServicesTopology(serviceIds: [ID!]!, duration: Duration!, debug: Boolean): Topology # Query the instance topology, based on the given clientServiceId and serverServiceId getServiceInstanceTopology(clientServiceId: ID!, serverServiceId: ID!, duration: Duration!, debug: Boolean): ServiceInstanceTopology ... # v2 of getEndpointTopology getEndpointDependencies(endpointId: ID!, duration: Duration!, debug: Boolean): EndpointTopology # Query the topology, based on the given instance getProcessTopology(serviceInstanceId: ID!, duration: Duration!, debug: Boolean): ProcessTopology }# The overview topology of the whole application cluster or services, type Topology { nodes: [Node!]! calls: [Call!]! debuggingTrace: DebuggingTrace } # The instance topology based on the given serviceIds type ServiceInstanceTopology { nodes: [ServiceInstanceNode!]! calls: [Call!]! debuggingTrace: DebuggingTrace } # The endpoint topology type EndpointTopology { nodes: [EndpointNode!]! calls: [Call!]! debuggingTrace: DebuggingTrace } # The process topology type ProcessTopology { nodes: [ProcessNode!]! calls: [Call!]! debuggingTrace: DebuggingTrace }用法与 MQE 查询追踪一致按 GraphQL 协议与语法查询结果并把debug参数置为true即可在返回中取得debuggingTrace信息。3.4 追踪日志查询GraphQLBundle APILogextend type Query { ... queryLogs(condition: LogQueryCondition, debug: Boolean): Logs ... }type Logs { # When this field is not empty, frontend should display it in UI errorReason: String logs: [Log!]! debuggingTrace: DebuggingTrace }用法与 MQE 查询追踪一致按 GraphQL 协议与语法查询结果并把debug参数置为true即可在返回中取得debuggingTrace信息。四、底层实现原理与使用建议4.1 埋点链路一览从源码来看Query Tracing 的埋点覆盖了 OAP 查询的完整调用链MQE 表达式层MQEVisitorBase.java 为每个 MQE 操作Binary OP、Aggregation OP、Mathematical OP、TopN OP、Logical OP、Trend OP、Sort 系列 OP 等创建 Span查询服务层MetricsQueryService.java、TraceQueryService.java、LogQueryService.java、TopologyQueryService.java、AggregationQueryService.java 等在方法入口/出口创建Query Service: ...与sortMetrics等 Span拓扑构建层ServiceTopologyBuilder.java 等各 Builder 创建Build xxx topologySpan存储层若使用 BanyanDB其内部 trace 通过createSpanForTransform挂接到 OAP 的 Span 树中直接展示存储侧的 gRPC 请求、IndexScan 计划与执行细节。4.2 实践建议只在需要时开启 debugdebug/dumpDBRsp参数默认关闭生产环境默认不产生额外开销排查问题时再针对单条查询开启。优先观察根因层级阅读返回的 Span 树时先看duration最大的子树再对照operation判断瓶颈落在 MQE 计算、拓扑构建还是存储查询结合error字段定位失败环节。善用dumpDBRsptrue当怀疑存储返回数据异常时开启该参数可将 Elasticsearch / BanyanDB 的真实响应 dump 到 Span 的msg中直接核对存储层行为。区分两种调试入口REST 调试接口/debugging/query/...适合命令行快速验证与脚本化GraphQL 的debug参数适合在业务集成、UI 二次开发场景中随查询一起返回追踪信息。五、相关资源Query Tracing 官方文档Debugging HTTP Handler 实现调试模块配置含敏感信息脱敏关键词DebuggingTrace 数据结构DebuggingTraceContext 上下文实现MQE 运行时 Span 埋点GraphQL 查询协议含 Duration 格式BanyanDB 存储接入文档【免费下载链接】skywalkingAPM, Application Performance Monitoring System项目地址: https://gitcode.com/gh_mirrors/sky/skywalking创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考