Agent Platform 评估 SDK 模式实战从单轮评测到托管 Agent 评估的 8 大代码范式【免费下载链接】skillsAgent Skills for Google products and technologies项目地址: https://gitcode.com/GitHub_Trending/skills29/skills本篇技术指南围绕 Google Cloud Agent Platform GenAI 评估 SDKagentplatform的常用编码模式展开系统梳理了单轮评测、多轮 Agent 轨迹评测、冷启动合成数据生成、自定义 LLM 裁判、代码执行度量、成对模型对比与结果解析等 8 大实战范式并覆盖初始化、错误处理等关键环节。读完本文你将能够基于 sdk_patterns.md 所定义的 API 形态独立搭建从数据集构造到评估结果落盘的完整评测流程并接入 Quality Flywheel质量飞轮的迭代优化闭环。前置准备SDK 初始化与依赖安装安装依赖评估脚本依赖agentplatform基于google-cloud-aiplatform[evaluation]、google-genai、pandas和requests。仓库 SKILL.md 建议不要创建虚拟环境——空虚拟环境会隐藏环境中已装好的包导致冗余安装。正确做法是先探测再补装缺失的包python3 -c import vertexai, google.genai, pandas, requests \ || pip install google-cloud-aiplatform[evaluation]1.163.0 google-genai1.0.0注意版本号约束必须加引号未加引号时 bash 会把1.163.0当作重定向静默写出一个空文件而非约束安装版本。客户端初始化import agentplatform from agentplatform import types from google.genai import types as genai_types client agentplatform.Client(project{PROJECT_ID}, location{LOCATION})初始化前需确认环境变量GOOGLE_CLOUD_PROJECT与GOOGLE_CLOUD_LOCATION已配置对于 Gemini 3 模型应使用locationglobal。新版本的 Gemini 模型通常也建议locationglobal。一个极易踩坑的关键约定EvalCase的prompt/reference/response值都是Content 对象而非纯字符串。必须使用genai_types.UserContent(str)/genai_types.ModelContent(str)包装——它们会把字符串包进一个 Part 并设置正确的角色。直接传字符串会触发pydantic.ValidationError快速失败这是构造数据集时最常见的错误之一详见 dataset_schema.md 的 Core Types 说明。SDK 入口的正确形态评估相关操作统一挂在client.evals下client.evals.run_inference(model..., src...) client.evals.evaluate(dataset..., metrics...) client.evals.generate_conversation_scenarios(...)两个看起来合理但不可用的导入形态来自 SKILL.mdfrom agentplatform.types import evals——会抛ModuleNotFoundError。types是模块而非包应使用from agentplatform import typesfrom vertexai.evaluation import PointwiseMetric, EvalTask——这是已被取代的旧 SDK其类参数不同如PointwiseMetric没有system_instruction按旧 API 写的代码会以TypeError而非导入错误失败。Pattern 1单轮评测——最简单的起点单轮评测适用于 QA、摘要等 prompt/response 对评估。构造EvaluationDataset并在metrics中同时混用预定义 rubric 度量与计算型度量dataset types.EvaluationDataset(eval_cases[ types.EvalCase( promptgenai_types.UserContent(What causes rain?), responses[types.ResponseCandidate( responsegenai_types.ModelContent( Rain is caused by water evaporating...))], referencetypes.ResponseCandidate( responsegenai_types.ModelContent( Rain forms when water vapor condenses...)), ), ]) result client.evals.evaluate( datasetdataset, metrics[ types.RubricMetric.GENERAL_QUALITY, types.Metric(namerouge_l_sum), ], )注意三点细节responses是复数列表不存在单数的response参数。由于EvalCase设置了extraallow误写response不会报错而是被静默存储、永不读取候选响应按缺失计分——这是典型的静默失败陷阱reference是ResponseCandidate对象而非字符串types.Metric(namerouge_l_sum)走的是确定性计算路径无需 LLM 裁判。如果觉得直接构造EvalCase过于冗长易错可以改用 pandas DataFrame 形式转换器会自动用字符串列包装成 Content 对象推荐import pandas as pd from agentplatform import types df pd.DataFrame({ prompt: [What is 22?, Capital of France?], response: [4, Paris], reference: [4, Paris], }) dataset types.EvaluationDataset(eval_dataset_dfdf)这一推荐路径在 dataset_schema.md 中有完整说明。Pattern 2多轮 Agent 评测——带工具调用的完整轨迹要评估一个完整的多轮 Agent 对话轨迹含工具调用需要使用AgentData类型层级AgentData→ConversationTurn→AgentEvent。每个AgentEvent通过author区分发言者content按角色区分消息类型agent_data types.evals.AgentData( agents{ my_agent: types.evals.AgentConfig( agent_idmy_agent, instructionYou are a helpful assistant., tools[genai_types.Tool(function_declarations[ genai_types.FunctionDeclaration( namesearch, descriptionSearch the web, parametersgenai_types.Schema( typeOBJECT, properties{query: genai_types.Schema(typeSTRING)}, ), ), ])], ), }, turns[ types.evals.ConversationTurn(turn_index0, events[ types.evals.AgentEvent( authoruser, contentgenai_types.Content(roleuser, parts[genai_types.Part(textFind me the weather in NYC)]), ), types.evals.AgentEvent( authormy_agent, contentgenai_types.Content(rolemodel, parts[genai_types.Part(function_callgenai_types.FunctionCall( namesearch, args{query: NYC weather}))]), ), types.evals.AgentEvent( authormy_agent, contentgenai_types.Content(roletool, parts[genai_types.Part(function_responsegenai_types.FunctionResponse( namesearch, response{result: 72F, sunny}))]), ), types.evals.AgentEvent( authormy_agent, contentgenai_types.Content(rolemodel, parts[genai_types.Part(textIts 72F and sunny in NYC.)]), ), ]), ], ) result client.evals.evaluate( datasettypes.EvaluationDataset(eval_cases[ types.EvalCase(agent_dataagent_data), ]), metrics[ types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY, types.RubricMetric.MULTI_TURN_TASK_SUCCESS, ], )轨迹构造的硬性约定仓库中的 validate_dataset.py 脚本会对这些约束做自动化校验其校验逻辑揭示了以下规则角色必须是user/model/tool使用roleassistant会被判为错误Agent Platform 约定用modelturn_index必须是从 0 开始连续递增的序号非顺序索引会触发 WARNING工具响应必须用genai_types.FunctionResponse包装且parts不能为空每个EvalCase只能使用prompt单轮或agent_data多轮二选一混用会被标记。另外若你的轨迹来自 ADKAgent Development Kit会话导出无需手写转换逻辑直接使用仓库脚本python scripts/parse_adk_traces.py --input session.json --output dataset.json详见 parse_adk_traces.py。Pattern 3冷启动合成数据生成——没有评估数据时怎么办当评估数据集为空、无从谈起评测时可以用生成场景 → 模拟推理 → 评估三步走的冷启动流程# Step 1: Generate scenarios scenarios client.evals.generate_conversation_scenarios( agents{ agent: types.evals.AgentConfig( agent_idagent, instructionYou are a customer support agent for an airline., ), }, root_agent_idagent, user_scenario_generation_configtypes.evals.UserScenarioGenerationConfig( user_scenario_count10, simulation_instructionSimulate customers with flight booking issues., environment_dataFlights available: NYC-LAX, NYC-SFO. Cancellation policy: free within 24h., model_namegemini-2.5-flash, ), ) # Step 2: Run inference with user simulation dataset_with_responses client.evals.run_inference( agentmy_agent, # Your callable agent srcscenarios, config{ user_simulator_config: { model_name: gemini-2.5-flash, max_turn: 5, }, }, ) # Step 3: Evaluate result client.evals.evaluate( datasetdataset_with_responses, metrics[types.RubricMetric.MULTI_TURN_GENERAL_QUALITY, types.RubricMetric.SAFETY], )冷启动 API 的三个易错点SKILL.md 明确标注了该 API 的坑参数名是agent或agent_info不是agentsPattern 2 构造数据集时才是agents且config为必填配置类全名是types.evals.UserScenarioGenerationConfig不是types.UserScenarioGenerationConfig必须设置user_scenario_count取值范围 1–100。它默认是None客户端会照单全收然后服务端以400 INVALID_ARGUMENT拒绝调用注意count是另一个独立字段不能替代它。在推理阶段run_inference对不同类型的被评测对象形态不同SKILL.md Stage 2Agent 评测传modelagent_callable包装用户 ADK Agent/App 的可调用对象模型评测直接传模型 ID如modelgemini-2.5-flash合成场景让模拟器驱动加user_simulator_configUserSimulatorConfig(max_turn10)DataFrame 也可直接作为src无需包EvalCase。Pattern 4用 MetricPromptBuilder 构建自定义 LLM 裁判领域特定评估如医疗、金融、客服质量没有现成的内置度量时可以用LLMMetricMetricPromptBuilder构建结构化评分裁判。与手写prompt_template字符串相比MetricPromptBuilder更适合复杂评分卡metric types.LLMMetric( namedomain_expertise, prompt_templatetypes.MetricPromptBuilder( metric_definitionEvaluates domain expertise in the response., criteria{ Accuracy: Claims are factually correct for the domain, Depth: Response shows understanding beyond surface level, Actionability: Advice is specific and actionable, }, rating_scores{ 1: Incorrect or misleading information, 2: Partially correct but superficial, 3: Correct and shows reasonable understanding, 4: Accurate with good depth, 5: Expert-level accuracy, depth, and actionability, }, ), judge_modelgemini-2.5-flash, judge_model_sampling_count3, )使用 LLM 裁判的硬性规则criteria和rating_scores两者都必填只提供一个会抛ValidationError: Both criteria and rating_scores are required to construct the LLM-based metric prompt template text见 metric_registry.md必须显式设置judge_model。它默认是None此时每个评测用例都会以400 INVALID_ARGUMENT: Error parsing JSON失败judge_model_sampling_count默认 1、上限 32值过高会导致评测超时见 failure_patterns.md。若只是基于固定 prompt 模板而非结构化评分卡也可以直接使用types.LLMMetric(name..., prompt_template..., judge_model...)形态甚至支持从 YAML/JSON 文件加载types.LLMMetric.load(path/to/metric_config.yaml)。Pattern 5CodeExecutionMetric 结构化校验——超越文本对比当需要程序化校验如 JSON 结构、正则、字段完整性时用CodeExecutionMetric在远程沙箱中执行 Python 函数。函数必须命名为evaluate(instance: dict) - dict返回{score: ..., explanation: ...}# Validate JSON output structure json_validator types.CodeExecutionMetric( namejson_structure_check, custom_function import json def evaluate(instance: dict) - dict: try: data json.loads(instance.get(response, )) required_keys {name, status, result} missing required_keys - set(data.keys()) if missing: return {score: 0.0, explanation: fMissing keys: {missing}} return {score: 1.0, explanation: All required keys present} except json.JSONDecodeError as e: return {score: 0.0, explanation: fInvalid JSON: {e}} , )本地函数与远程沙箱的选择metric_registry.md 区分了两种自定义代码度量本地自定义函数types.Metric(name..., custom_functioncallable)客户端执行、迭代最快、无 API 调用但以调用进程的权限运行只可用于可信代码远程沙箱执行CodeExecutionMetric的custom_function字符串在 Agent Platform 沙箱中运行适合不可信代码。SDK 内部的度量处理器按固定顺序分派Handler Dispatch Order先匹配CodeExecutionMetric的字符串函数再匹配本地Callable、注册度量资源名、计算型度量bleu/rouge_1等、翻译度量comet/metricx、API 预定义度量最后才是LLMMetric的prompt_template。理解这个顺序有助于排查自定义度量没生效的问题。注意在 Agent 轨迹数据集的instance字典里顶层标准字段是agent_data结构化的 turns/events 对象最终回复嵌在其中扁平占位符{response}只在 DataFrame 评测路径下可用。自定义函数应始终用.get()带默认值见 failure_patterns.md 的 KeyError 故障说明。Pattern 6成对模型对比——用 calculate_win_rates 计算胜率对比两个模型时SDK 没有PairwiseMetric类。正确做法是对同一数据集分别跑两个模型的评估再用calculate_win_rates()计算胜率# Same dataset, two different model responses dataset_a types.EvaluationDataset(eval_cases[ types.EvalCase(promptgenai_types.UserContent(Explain quantum computing), responses[types.ResponseCandidate( responsegenai_types.ModelContent( Model A response...))]), ]) dataset_b types.EvaluationDataset(eval_cases[ types.EvalCase(promptgenai_types.UserContent(Explain quantum computing), responses[types.ResponseCandidate( responsegenai_types.ModelContent( Model B response...))]), ]) result_a client.evals.evaluate(datasetdataset_a, metrics[types.RubricMetric.GENERAL_QUALITY]) result_b client.evals.evaluate(datasetdataset_b, metrics[types.RubricMetric.GENERAL_QUALITY]) # Compare from agentplatform._genai._evals_metric_handlers import calculate_win_rates win_rates calculate_win_rates(result_a, result_b)该方法与 metric_registry.md 中 Pairwise Comparison 一节的用法一致。注意calculate_win_rates来自 SDK 内部模块_evals_metric_handlers这是文档中唯一出现的内部导入路径其余场景应尽量使用公开 API。Pattern 7结果解析——从摘要到逐条 rubric 判定evaluate()返回的结果对象有清晰的层级可按三种粒度解析result client.evals.evaluate(datasetdataset, metricsmetrics) # Interactive HTML report (recommended) result.show() # Summary level for summary in result.summary_metrics: print(f{summary.metric_name}: mean{summary.mean_score}, pass_rate{summary.pass_rate}) # Per-case level for case in result.eval_case_results: for candidate in case.response_candidate_results: for metric_name, metric_result in candidate.metric_results.items(): print(f {metric_name}: score{metric_result.score}) print(f explanation: {metric_result.explanation}) # Rubric verdicts (for rubric-based metrics) if metric_result.rubric_verdicts: for v in metric_result.rubric_verdicts: print(f rubric {v.evaluated_rubric.rubric_id}: f{PASS if v.verdict else FAIL} - {v.reasoning})持久化结果JSON HTML 双份落盘结果字段路径很深eval_case_results[].response_candidate_results建议落盘保存供后续分析与对比。仓库 SKILL.md Stage 3 给出了标准做法——JSON机器可读、可 diff与 HTML人类可读、可分享各存一份import datetime from pathlib import Path from agentplatform._genai import _evals_visualization out_dir Path(artifacts/grade_results) out_dir.mkdir(parentsTrue, exist_okTrue) ts datetime.datetime.now().strftime(%Y%m%d_%H%M%S) # fallbackstr, or a DataFrame-backed dataset raises PydanticSerializationError. result_json result.model_dump_json(fallbackstr) (out_dir / fresults_{ts}.json).write_text(result_json) html _evals_visualization.get_evaluation_html(result_json) (out_dir / fresults_{ts}.html).write_text(str(html))之后可以用仓库配套脚本快速渲染与筛选inspect_results.pypython scripts/inspect_results.py --result result.json # 摘要 逐用例分数 python scripts/inspect_results.py --result result.json --failing-only # 只看失败用例 (score 1.0) python scripts/inspect_results.py --result result.json --metric multi_turn_task_success python scripts/inspect_results.py --result result.json --save-html report.html在迭代优化阶段用 compare_results.py 对比修复前后两个结果文件确认目标指标提升且无回归python scripts/compare_results.py --baseline baseline.json --candidate candidate.json python scripts/compare_results.py -b baseline.json -c candidate.json --threshold 0.05 --json该脚本按metric_name对齐两文件的summary_metrics输出均值与通过率的增量任一指标下降超过阈值默认 0.0即任何下降都算回归时退出码为 1。Pattern 8托管 Agent 评估Gemini Agents API对于使用 Managed Agents API 构建的 Agent可以只凭其资源名完成生成场景 → 推理 → 评估的完整工作流无需自己包装可调用对象import agentplatform from agentplatform import types client agentplatform.Client(projectPROJECT_ID, locationglobal) AGENT_RESOURCE projects/PROJECT_ID/locations/global/agents/AGENT_ID # Step 1: Generate conversation scenarios from the agents configuration. scenarios client.evals.generate_conversation_scenarios( agentAGENT_RESOURCE, config{ user_scenario_count: 5, simulation_instruction: Create agent scenarios, }, ) scenarios.show() # Step 2: Run inference, execute the agent against each scenario. inference_results client.evals.run_inference( agentAGENT_RESOURCE, srcscenarios, config{user_simulator_config: {max_turn: 3}}, ) inference_results.show() # Step 3: Evaluate the conversation traces. result client.evals.evaluate( datasetinference_results, metrics[types.RubricMetric.MULTI_TURN_TASK_SUCCESS], agentAGENT_RESOURCE, ) result.show()注意这里generate_conversation_scenarios与run_inference的agent参数直接接受资源名形如projects/.../locations/global/agents/...并且location固定为global。评估已记录的交互Interactions API如果交互已通过 Interactions API 记录可以跳过推理直接评估避免重复运行interactions_dataset types.EvaluationDataset( eval_cases[ types.EvalCase( interactions_data_sourcetypes.InteractionsDataSource( interactionprojects/PROJECT_ID/locations/global/interactions/INTERACTION_ID, gemini_agent_configtypes.GeminiAgentConfig( gemini_agentAGENT_RESOURCE, ), ), ), ] ) result client.evals.evaluate( datasetinteractions_dataset, metrics[types.RubricMetric.MULTI_TURN_TASK_SUCCESS], agentAGENT_RESOURCE, ) result.show()错误处理区分权限、参数与配额问题evaluate()调用失败时根据异常类型快速定位问题类别try: result client.evals.evaluate(datasetdataset, metricsmetrics) except Exception as e: error_type type(e).__name__ if PermissionDenied in error_type: print(Check: GCP project permissions, API enabled, billing active) elif InvalidArgument in error_type: print(Check: dataset format, metric compatibility with data type) elif ResourceExhausted in error_type: print(Check: API quota, reduce dataset size or add delay) else: raise结合 failure_patterns.md 的故障排查清单这三类错误的常见根因分别是PermissionDeniedGCP 项目权限不足、API 未启用、结算未激活InvalidArgument数据集格式错误如prompt传了字符串、度量与数据类型不匹配如对单轮数据用了multi_turn_*度量、judge_model未设置ResourceExhaustedAPI 配额耗尽可减小数据集或增加延迟。另需留意is_infra_error: true的结构性失败配额、超时、端点不可用与评测超时问题——超时通常源于数据集过大、自定义度量代码过慢或judge_model_sampling_count过高可分批评测并降低采样数。接入 Quality Flywheel五个阶段如何落地以上模式是 agent-platform-eval-flywheel 这套 Skill 的技术底座。在实战中它们服务于质量飞轮的五个阶段准备数据Prepare Data按 Pattern 1/2 构造数据集EvalCase、DataFrame 或agent_data或用 Pattern 3 冷启动生成提交前用scripts/validate_dataset.py校验运行推理Run Inference用run_inference填充响应已有完整轨迹如生产日志回放时跳过评分Grade必做按 Pattern 1/4/5 选择度量并执行evaluate()按 Pattern 7 落盘 JSON HTML度量选择参考 metric_registry.md 的分类Agent 度量优先multi_turn_task_success/multi_turn_trajectory_quality/multi_turn_tool_use_quality静态 rubric 关注hallucination/grounding/safety分析失败Analyze Failures用 Pattern 7 解析 rubric verdicts配合scripts/inspect_results.py --failing-only定位失败用例同一指标 10 失败时可用 Error Analysis 服务client.evals.generate_loss_clusters聚类失败主题优化迭代Optimize Iterate针对失败指标修复 Agent 或提示词重跑后用scripts/compare_results.py确认目标提升且无回归。每个失败用例通常需要 5–10 次迭代。需要特别提醒的是评测结果必须来自真实的result对象绝不可编造分数无法产出证据SDK 调用失败、结果截断、度量不支持时应当明确说明而不是掩盖缺口——这是该 Skill 反复强调的Proving your work原则。延伸阅读sdk_patterns.md本文所依据的模式定义原文dataset_schema.mdEvaluationDataset/EvalCase/AgentData完整类型层级与字段说明metric_registry.md全部预定义、计算型、翻译、多模态与自定义度量的目录与选择指南failure_patterns.md常见失败模式到根因与修复方案的映射SKILL.mdQuality Flywheel 完整方法论与安全确认层级validate_dataset.py评测数据集结构校验脚本parse_adk_traces.pyADK 会话轨迹转规范数据集脚本inspect_results.py结果摘要与逐用例分数渲染脚本compare_results.py基线 vs 候选结果对比与回归检测脚本【免费下载链接】skillsAgent Skills for Google products and technologies项目地址: https://gitcode.com/GitHub_Trending/skills29/skills创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Agent Platform 评估 SDK 模式实战：从单轮评测到托管 Agent 评估的 8 大代码范式
Agent Platform 评估 SDK 模式实战从单轮评测到托管 Agent 评估的 8 大代码范式【免费下载链接】skillsAgent Skills for Google products and technologies项目地址: https://gitcode.com/GitHub_Trending/skills29/skills本篇技术指南围绕 Google Cloud Agent Platform GenAI 评估 SDKagentplatform的常用编码模式展开系统梳理了单轮评测、多轮 Agent 轨迹评测、冷启动合成数据生成、自定义 LLM 裁判、代码执行度量、成对模型对比与结果解析等 8 大实战范式并覆盖初始化、错误处理等关键环节。读完本文你将能够基于 sdk_patterns.md 所定义的 API 形态独立搭建从数据集构造到评估结果落盘的完整评测流程并接入 Quality Flywheel质量飞轮的迭代优化闭环。前置准备SDK 初始化与依赖安装安装依赖评估脚本依赖agentplatform基于google-cloud-aiplatform[evaluation]、google-genai、pandas和requests。仓库 SKILL.md 建议不要创建虚拟环境——空虚拟环境会隐藏环境中已装好的包导致冗余安装。正确做法是先探测再补装缺失的包python3 -c import vertexai, google.genai, pandas, requests \ || pip install google-cloud-aiplatform[evaluation]1.163.0 google-genai1.0.0注意版本号约束必须加引号未加引号时 bash 会把1.163.0当作重定向静默写出一个空文件而非约束安装版本。客户端初始化import agentplatform from agentplatform import types from google.genai import types as genai_types client agentplatform.Client(project{PROJECT_ID}, location{LOCATION})初始化前需确认环境变量GOOGLE_CLOUD_PROJECT与GOOGLE_CLOUD_LOCATION已配置对于 Gemini 3 模型应使用locationglobal。新版本的 Gemini 模型通常也建议locationglobal。一个极易踩坑的关键约定EvalCase的prompt/reference/response值都是Content 对象而非纯字符串。必须使用genai_types.UserContent(str)/genai_types.ModelContent(str)包装——它们会把字符串包进一个 Part 并设置正确的角色。直接传字符串会触发pydantic.ValidationError快速失败这是构造数据集时最常见的错误之一详见 dataset_schema.md 的 Core Types 说明。SDK 入口的正确形态评估相关操作统一挂在client.evals下client.evals.run_inference(model..., src...) client.evals.evaluate(dataset..., metrics...) client.evals.generate_conversation_scenarios(...)两个看起来合理但不可用的导入形态来自 SKILL.mdfrom agentplatform.types import evals——会抛ModuleNotFoundError。types是模块而非包应使用from agentplatform import typesfrom vertexai.evaluation import PointwiseMetric, EvalTask——这是已被取代的旧 SDK其类参数不同如PointwiseMetric没有system_instruction按旧 API 写的代码会以TypeError而非导入错误失败。Pattern 1单轮评测——最简单的起点单轮评测适用于 QA、摘要等 prompt/response 对评估。构造EvaluationDataset并在metrics中同时混用预定义 rubric 度量与计算型度量dataset types.EvaluationDataset(eval_cases[ types.EvalCase( promptgenai_types.UserContent(What causes rain?), responses[types.ResponseCandidate( responsegenai_types.ModelContent( Rain is caused by water evaporating...))], referencetypes.ResponseCandidate( responsegenai_types.ModelContent( Rain forms when water vapor condenses...)), ), ]) result client.evals.evaluate( datasetdataset, metrics[ types.RubricMetric.GENERAL_QUALITY, types.Metric(namerouge_l_sum), ], )注意三点细节responses是复数列表不存在单数的response参数。由于EvalCase设置了extraallow误写response不会报错而是被静默存储、永不读取候选响应按缺失计分——这是典型的静默失败陷阱reference是ResponseCandidate对象而非字符串types.Metric(namerouge_l_sum)走的是确定性计算路径无需 LLM 裁判。如果觉得直接构造EvalCase过于冗长易错可以改用 pandas DataFrame 形式转换器会自动用字符串列包装成 Content 对象推荐import pandas as pd from agentplatform import types df pd.DataFrame({ prompt: [What is 22?, Capital of France?], response: [4, Paris], reference: [4, Paris], }) dataset types.EvaluationDataset(eval_dataset_dfdf)这一推荐路径在 dataset_schema.md 中有完整说明。Pattern 2多轮 Agent 评测——带工具调用的完整轨迹要评估一个完整的多轮 Agent 对话轨迹含工具调用需要使用AgentData类型层级AgentData→ConversationTurn→AgentEvent。每个AgentEvent通过author区分发言者content按角色区分消息类型agent_data types.evals.AgentData( agents{ my_agent: types.evals.AgentConfig( agent_idmy_agent, instructionYou are a helpful assistant., tools[genai_types.Tool(function_declarations[ genai_types.FunctionDeclaration( namesearch, descriptionSearch the web, parametersgenai_types.Schema( typeOBJECT, properties{query: genai_types.Schema(typeSTRING)}, ), ), ])], ), }, turns[ types.evals.ConversationTurn(turn_index0, events[ types.evals.AgentEvent( authoruser, contentgenai_types.Content(roleuser, parts[genai_types.Part(textFind me the weather in NYC)]), ), types.evals.AgentEvent( authormy_agent, contentgenai_types.Content(rolemodel, parts[genai_types.Part(function_callgenai_types.FunctionCall( namesearch, args{query: NYC weather}))]), ), types.evals.AgentEvent( authormy_agent, contentgenai_types.Content(roletool, parts[genai_types.Part(function_responsegenai_types.FunctionResponse( namesearch, response{result: 72F, sunny}))]), ), types.evals.AgentEvent( authormy_agent, contentgenai_types.Content(rolemodel, parts[genai_types.Part(textIts 72F and sunny in NYC.)]), ), ]), ], ) result client.evals.evaluate( datasettypes.EvaluationDataset(eval_cases[ types.EvalCase(agent_dataagent_data), ]), metrics[ types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY, types.RubricMetric.MULTI_TURN_TASK_SUCCESS, ], )轨迹构造的硬性约定仓库中的 validate_dataset.py 脚本会对这些约束做自动化校验其校验逻辑揭示了以下规则角色必须是user/model/tool使用roleassistant会被判为错误Agent Platform 约定用modelturn_index必须是从 0 开始连续递增的序号非顺序索引会触发 WARNING工具响应必须用genai_types.FunctionResponse包装且parts不能为空每个EvalCase只能使用prompt单轮或agent_data多轮二选一混用会被标记。另外若你的轨迹来自 ADKAgent Development Kit会话导出无需手写转换逻辑直接使用仓库脚本python scripts/parse_adk_traces.py --input session.json --output dataset.json详见 parse_adk_traces.py。Pattern 3冷启动合成数据生成——没有评估数据时怎么办当评估数据集为空、无从谈起评测时可以用生成场景 → 模拟推理 → 评估三步走的冷启动流程# Step 1: Generate scenarios scenarios client.evals.generate_conversation_scenarios( agents{ agent: types.evals.AgentConfig( agent_idagent, instructionYou are a customer support agent for an airline., ), }, root_agent_idagent, user_scenario_generation_configtypes.evals.UserScenarioGenerationConfig( user_scenario_count10, simulation_instructionSimulate customers with flight booking issues., environment_dataFlights available: NYC-LAX, NYC-SFO. Cancellation policy: free within 24h., model_namegemini-2.5-flash, ), ) # Step 2: Run inference with user simulation dataset_with_responses client.evals.run_inference( agentmy_agent, # Your callable agent srcscenarios, config{ user_simulator_config: { model_name: gemini-2.5-flash, max_turn: 5, }, }, ) # Step 3: Evaluate result client.evals.evaluate( datasetdataset_with_responses, metrics[types.RubricMetric.MULTI_TURN_GENERAL_QUALITY, types.RubricMetric.SAFETY], )冷启动 API 的三个易错点SKILL.md 明确标注了该 API 的坑参数名是agent或agent_info不是agentsPattern 2 构造数据集时才是agents且config为必填配置类全名是types.evals.UserScenarioGenerationConfig不是types.UserScenarioGenerationConfig必须设置user_scenario_count取值范围 1–100。它默认是None客户端会照单全收然后服务端以400 INVALID_ARGUMENT拒绝调用注意count是另一个独立字段不能替代它。在推理阶段run_inference对不同类型的被评测对象形态不同SKILL.md Stage 2Agent 评测传modelagent_callable包装用户 ADK Agent/App 的可调用对象模型评测直接传模型 ID如modelgemini-2.5-flash合成场景让模拟器驱动加user_simulator_configUserSimulatorConfig(max_turn10)DataFrame 也可直接作为src无需包EvalCase。Pattern 4用 MetricPromptBuilder 构建自定义 LLM 裁判领域特定评估如医疗、金融、客服质量没有现成的内置度量时可以用LLMMetricMetricPromptBuilder构建结构化评分裁判。与手写prompt_template字符串相比MetricPromptBuilder更适合复杂评分卡metric types.LLMMetric( namedomain_expertise, prompt_templatetypes.MetricPromptBuilder( metric_definitionEvaluates domain expertise in the response., criteria{ Accuracy: Claims are factually correct for the domain, Depth: Response shows understanding beyond surface level, Actionability: Advice is specific and actionable, }, rating_scores{ 1: Incorrect or misleading information, 2: Partially correct but superficial, 3: Correct and shows reasonable understanding, 4: Accurate with good depth, 5: Expert-level accuracy, depth, and actionability, }, ), judge_modelgemini-2.5-flash, judge_model_sampling_count3, )使用 LLM 裁判的硬性规则criteria和rating_scores两者都必填只提供一个会抛ValidationError: Both criteria and rating_scores are required to construct the LLM-based metric prompt template text见 metric_registry.md必须显式设置judge_model。它默认是None此时每个评测用例都会以400 INVALID_ARGUMENT: Error parsing JSON失败judge_model_sampling_count默认 1、上限 32值过高会导致评测超时见 failure_patterns.md。若只是基于固定 prompt 模板而非结构化评分卡也可以直接使用types.LLMMetric(name..., prompt_template..., judge_model...)形态甚至支持从 YAML/JSON 文件加载types.LLMMetric.load(path/to/metric_config.yaml)。Pattern 5CodeExecutionMetric 结构化校验——超越文本对比当需要程序化校验如 JSON 结构、正则、字段完整性时用CodeExecutionMetric在远程沙箱中执行 Python 函数。函数必须命名为evaluate(instance: dict) - dict返回{score: ..., explanation: ...}# Validate JSON output structure json_validator types.CodeExecutionMetric( namejson_structure_check, custom_function import json def evaluate(instance: dict) - dict: try: data json.loads(instance.get(response, )) required_keys {name, status, result} missing required_keys - set(data.keys()) if missing: return {score: 0.0, explanation: fMissing keys: {missing}} return {score: 1.0, explanation: All required keys present} except json.JSONDecodeError as e: return {score: 0.0, explanation: fInvalid JSON: {e}} , )本地函数与远程沙箱的选择metric_registry.md 区分了两种自定义代码度量本地自定义函数types.Metric(name..., custom_functioncallable)客户端执行、迭代最快、无 API 调用但以调用进程的权限运行只可用于可信代码远程沙箱执行CodeExecutionMetric的custom_function字符串在 Agent Platform 沙箱中运行适合不可信代码。SDK 内部的度量处理器按固定顺序分派Handler Dispatch Order先匹配CodeExecutionMetric的字符串函数再匹配本地Callable、注册度量资源名、计算型度量bleu/rouge_1等、翻译度量comet/metricx、API 预定义度量最后才是LLMMetric的prompt_template。理解这个顺序有助于排查自定义度量没生效的问题。注意在 Agent 轨迹数据集的instance字典里顶层标准字段是agent_data结构化的 turns/events 对象最终回复嵌在其中扁平占位符{response}只在 DataFrame 评测路径下可用。自定义函数应始终用.get()带默认值见 failure_patterns.md 的 KeyError 故障说明。Pattern 6成对模型对比——用 calculate_win_rates 计算胜率对比两个模型时SDK 没有PairwiseMetric类。正确做法是对同一数据集分别跑两个模型的评估再用calculate_win_rates()计算胜率# Same dataset, two different model responses dataset_a types.EvaluationDataset(eval_cases[ types.EvalCase(promptgenai_types.UserContent(Explain quantum computing), responses[types.ResponseCandidate( responsegenai_types.ModelContent( Model A response...))]), ]) dataset_b types.EvaluationDataset(eval_cases[ types.EvalCase(promptgenai_types.UserContent(Explain quantum computing), responses[types.ResponseCandidate( responsegenai_types.ModelContent( Model B response...))]), ]) result_a client.evals.evaluate(datasetdataset_a, metrics[types.RubricMetric.GENERAL_QUALITY]) result_b client.evals.evaluate(datasetdataset_b, metrics[types.RubricMetric.GENERAL_QUALITY]) # Compare from agentplatform._genai._evals_metric_handlers import calculate_win_rates win_rates calculate_win_rates(result_a, result_b)该方法与 metric_registry.md 中 Pairwise Comparison 一节的用法一致。注意calculate_win_rates来自 SDK 内部模块_evals_metric_handlers这是文档中唯一出现的内部导入路径其余场景应尽量使用公开 API。Pattern 7结果解析——从摘要到逐条 rubric 判定evaluate()返回的结果对象有清晰的层级可按三种粒度解析result client.evals.evaluate(datasetdataset, metricsmetrics) # Interactive HTML report (recommended) result.show() # Summary level for summary in result.summary_metrics: print(f{summary.metric_name}: mean{summary.mean_score}, pass_rate{summary.pass_rate}) # Per-case level for case in result.eval_case_results: for candidate in case.response_candidate_results: for metric_name, metric_result in candidate.metric_results.items(): print(f {metric_name}: score{metric_result.score}) print(f explanation: {metric_result.explanation}) # Rubric verdicts (for rubric-based metrics) if metric_result.rubric_verdicts: for v in metric_result.rubric_verdicts: print(f rubric {v.evaluated_rubric.rubric_id}: f{PASS if v.verdict else FAIL} - {v.reasoning})持久化结果JSON HTML 双份落盘结果字段路径很深eval_case_results[].response_candidate_results建议落盘保存供后续分析与对比。仓库 SKILL.md Stage 3 给出了标准做法——JSON机器可读、可 diff与 HTML人类可读、可分享各存一份import datetime from pathlib import Path from agentplatform._genai import _evals_visualization out_dir Path(artifacts/grade_results) out_dir.mkdir(parentsTrue, exist_okTrue) ts datetime.datetime.now().strftime(%Y%m%d_%H%M%S) # fallbackstr, or a DataFrame-backed dataset raises PydanticSerializationError. result_json result.model_dump_json(fallbackstr) (out_dir / fresults_{ts}.json).write_text(result_json) html _evals_visualization.get_evaluation_html(result_json) (out_dir / fresults_{ts}.html).write_text(str(html))之后可以用仓库配套脚本快速渲染与筛选inspect_results.pypython scripts/inspect_results.py --result result.json # 摘要 逐用例分数 python scripts/inspect_results.py --result result.json --failing-only # 只看失败用例 (score 1.0) python scripts/inspect_results.py --result result.json --metric multi_turn_task_success python scripts/inspect_results.py --result result.json --save-html report.html在迭代优化阶段用 compare_results.py 对比修复前后两个结果文件确认目标指标提升且无回归python scripts/compare_results.py --baseline baseline.json --candidate candidate.json python scripts/compare_results.py -b baseline.json -c candidate.json --threshold 0.05 --json该脚本按metric_name对齐两文件的summary_metrics输出均值与通过率的增量任一指标下降超过阈值默认 0.0即任何下降都算回归时退出码为 1。Pattern 8托管 Agent 评估Gemini Agents API对于使用 Managed Agents API 构建的 Agent可以只凭其资源名完成生成场景 → 推理 → 评估的完整工作流无需自己包装可调用对象import agentplatform from agentplatform import types client agentplatform.Client(projectPROJECT_ID, locationglobal) AGENT_RESOURCE projects/PROJECT_ID/locations/global/agents/AGENT_ID # Step 1: Generate conversation scenarios from the agents configuration. scenarios client.evals.generate_conversation_scenarios( agentAGENT_RESOURCE, config{ user_scenario_count: 5, simulation_instruction: Create agent scenarios, }, ) scenarios.show() # Step 2: Run inference, execute the agent against each scenario. inference_results client.evals.run_inference( agentAGENT_RESOURCE, srcscenarios, config{user_simulator_config: {max_turn: 3}}, ) inference_results.show() # Step 3: Evaluate the conversation traces. result client.evals.evaluate( datasetinference_results, metrics[types.RubricMetric.MULTI_TURN_TASK_SUCCESS], agentAGENT_RESOURCE, ) result.show()注意这里generate_conversation_scenarios与run_inference的agent参数直接接受资源名形如projects/.../locations/global/agents/...并且location固定为global。评估已记录的交互Interactions API如果交互已通过 Interactions API 记录可以跳过推理直接评估避免重复运行interactions_dataset types.EvaluationDataset( eval_cases[ types.EvalCase( interactions_data_sourcetypes.InteractionsDataSource( interactionprojects/PROJECT_ID/locations/global/interactions/INTERACTION_ID, gemini_agent_configtypes.GeminiAgentConfig( gemini_agentAGENT_RESOURCE, ), ), ), ] ) result client.evals.evaluate( datasetinteractions_dataset, metrics[types.RubricMetric.MULTI_TURN_TASK_SUCCESS], agentAGENT_RESOURCE, ) result.show()错误处理区分权限、参数与配额问题evaluate()调用失败时根据异常类型快速定位问题类别try: result client.evals.evaluate(datasetdataset, metricsmetrics) except Exception as e: error_type type(e).__name__ if PermissionDenied in error_type: print(Check: GCP project permissions, API enabled, billing active) elif InvalidArgument in error_type: print(Check: dataset format, metric compatibility with data type) elif ResourceExhausted in error_type: print(Check: API quota, reduce dataset size or add delay) else: raise结合 failure_patterns.md 的故障排查清单这三类错误的常见根因分别是PermissionDeniedGCP 项目权限不足、API 未启用、结算未激活InvalidArgument数据集格式错误如prompt传了字符串、度量与数据类型不匹配如对单轮数据用了multi_turn_*度量、judge_model未设置ResourceExhaustedAPI 配额耗尽可减小数据集或增加延迟。另需留意is_infra_error: true的结构性失败配额、超时、端点不可用与评测超时问题——超时通常源于数据集过大、自定义度量代码过慢或judge_model_sampling_count过高可分批评测并降低采样数。接入 Quality Flywheel五个阶段如何落地以上模式是 agent-platform-eval-flywheel 这套 Skill 的技术底座。在实战中它们服务于质量飞轮的五个阶段准备数据Prepare Data按 Pattern 1/2 构造数据集EvalCase、DataFrame 或agent_data或用 Pattern 3 冷启动生成提交前用scripts/validate_dataset.py校验运行推理Run Inference用run_inference填充响应已有完整轨迹如生产日志回放时跳过评分Grade必做按 Pattern 1/4/5 选择度量并执行evaluate()按 Pattern 7 落盘 JSON HTML度量选择参考 metric_registry.md 的分类Agent 度量优先multi_turn_task_success/multi_turn_trajectory_quality/multi_turn_tool_use_quality静态 rubric 关注hallucination/grounding/safety分析失败Analyze Failures用 Pattern 7 解析 rubric verdicts配合scripts/inspect_results.py --failing-only定位失败用例同一指标 10 失败时可用 Error Analysis 服务client.evals.generate_loss_clusters聚类失败主题优化迭代Optimize Iterate针对失败指标修复 Agent 或提示词重跑后用scripts/compare_results.py确认目标提升且无回归。每个失败用例通常需要 5–10 次迭代。需要特别提醒的是评测结果必须来自真实的result对象绝不可编造分数无法产出证据SDK 调用失败、结果截断、度量不支持时应当明确说明而不是掩盖缺口——这是该 Skill 反复强调的Proving your work原则。延伸阅读sdk_patterns.md本文所依据的模式定义原文dataset_schema.mdEvaluationDataset/EvalCase/AgentData完整类型层级与字段说明metric_registry.md全部预定义、计算型、翻译、多模态与自定义度量的目录与选择指南failure_patterns.md常见失败模式到根因与修复方案的映射SKILL.mdQuality Flywheel 完整方法论与安全确认层级validate_dataset.py评测数据集结构校验脚本parse_adk_traces.pyADK 会话轨迹转规范数据集脚本inspect_results.py结果摘要与逐用例分数渲染脚本compare_results.py基线 vs 候选结果对比与回归检测脚本【免费下载链接】skillsAgent Skills for Google products and technologies项目地址: https://gitcode.com/GitHub_Trending/skills29/skills创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考