DeepSeek/deepseek-harness
1
0 Fork 0
mirror of https://github.com/deepseek-ai/deepseek-harness.git synced 2026-10-27 01:05:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(ptc): refresh ordinary snapshots without extra retention

Browse Source
This commit is contained in:
Tianyi Cui committed 2026-10-07 03:53:15 +08:00
1 parent 9b9496b692
commit aeb166c42e
59 files changed
+411 -6046
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

python/development.i18n.yaml
+2 -2
View File
@@ -9,8 +9,8 @@
en: 8e99640e4936332d
zh: 729d5de1e0ad2869
/python-contributor-workflows/validate-the-sdk:
en: e3909cfeda73622a
zh: b93001f829ea9c61
en: a944dff6ee37346f
zh: fc1565da859ae6c7
/python-contributor-workflows/run-against-node-source:
en: 4d30c74632781443
zh: 13191cb05f99dd31
python/development.md
+3 -5
View File
@@ -34,13 +34,11 @@ uv run --project python/sdk python scripts/smoke-python-runtime.py \
--scenario sdk-minimal --exe dist-exe/deepseek-harness-sdk-runtime-macos-arm64
```
SDK scenarios compare committed expected output under `scripts/snapshots/python-sdk-single-exe/`. `minimal/model-visible.json` pins the Linux/macOS `sdk-minimal` profile's assembled system prompts, advertised tool schemas, and model-visible messages; `minimal/win-x64/model-visible.json` pins its PowerShell counterpart. A plugin that contributes an unintended system section or user message therefore fails the job, and every message the profile emits is compared. `advanced-native/` pins direct tool calls, child agents, and workflows; `advanced-ptc/` pins program calls and nested denials. Both compare SDK results and session logs across every target. `restart-native/` launches two complete SDK runtime processes against one persistence root and snapshots their isolated model histories, high-level results, and separate durable logs across every target. `sdk-minimal-in-history` reuses the persistent-shell and editor scenario with a section that changes after the first successful shell call. `minimal-in-history/prompt-history.json` pins both prompt versions, the unchanged leading prompt in later requests, appended SDK system-message events, and `request/context.systemPromptUpdate`; tool schemas stay fixed and the editor file is checked independently. `sdk-dynamic-tools` pins native tool additions and removals in `dynamic-tools/tool-history.json`, including historical header references and agreement between Python SDK results, notifications, and persisted events. Rerun the owning scenario with `--update-snapshots` and review that diff before committing it.
SDK scenarios compare committed expected output under `scripts/snapshots/python-sdk-single-exe/`. `minimal/model-visible.json` pins the Linux/macOS `sdk-minimal` profile's assembled system prompts, advertised tool schemas, and model-visible messages; `minimal/win-x64/model-visible.json` pins its PowerShell counterpart. A plugin that contributes an unintended system section or user message therefore fails the job, and every message the profile emits is compared. `advanced/` pins direct tool calls, child agents, and workflows; `advanced-ptc/` pins program calls and nested denials. Both compare SDK results and session logs across every target. `restart/` launches two complete SDK runtime processes against one persistence root and snapshots their isolated model histories, high-level results, and separate durable logs across every target. `sdk-minimal-in-history` reuses the persistent-shell and editor scenario with a section that changes after the first successful shell call. `minimal-in-history/prompt-history.json` pins both prompt versions, the unchanged leading prompt in later requests, appended SDK system-message events, and `request/context.systemPromptUpdate`; tool schemas stay fixed and the editor file is checked independently. `sdk-dynamic-tools` pins native tool additions and removals in `dynamic-tools/tool-history.json`, including historical header references and agreement between Python SDK results, notifications, and persisted events. Rerun the owning scenario with `--update-snapshots` and review that diff before committing it.
The retained `advanced/`, `restart/`, and `scheduler-recovery/` directories keep immutable historical output. Read-only comparison tests cover all three; executable scenarios use `advanced-native/`, `advanced-ptc/`, `restart-native/`, and `scheduler-recovery-native/`.
`scheduler-recovery/` records a failed tool turn and a subsequent completed turn. It preserves the original `UNKNOWN` turn error, the first tool's completed result, and `TOOL_OUTCOME_UNKNOWN` / `TOOL_NOT_STARTED` results for the remaining calls. `sdk-snapshot` and `all` include this case; `--scenario sdk-recovery` runs it alone.
`scheduler-recovery-native/` records a failed tool turn and a subsequent completed turn. It preserves the original `UNKNOWN` turn error, the first tool's completed result, and `TOOL_OUTCOME_UNKNOWN` / `TOOL_NOT_STARTED` results for the remaining calls. `sdk-snapshot` and `all` include this case; `--scenario sdk-recovery` runs it alone.
The `advanced-native`, `advanced-ptc`, and `restart-native` comparisons match native delivery qualifiers against each input's own Session generation, including SDK events and notifications. Captured and other delivery generations retain their numeric values. Each side must use one generation across its Session roles, and fresh logs must identify the current writer. Comparison never rewrites committed recordings.
The `advanced`, `advanced-ptc`, and `restart` comparisons match native delivery qualifiers against each input's own Session generation, including SDK events and notifications. Captured and other delivery generations retain their numeric values. Each side must use one generation across its Session roles, and fresh logs must identify the current writer. Without `--update-snapshots`, comparison writes no files. Refresh writes current-writer outputs and preserves earlier Session generations.
Trusted pull requests and master pushes also run `--scenario sdk-live --installed-wheel` on each selected native target. That scenario performs two tool-using turns against `https://api.deepseek.com`: it checks the created file immediately, replaces its content with a host-only random challenge, and requires the second turn to copy the changed content into a fresh receipt without modifying the source. Both turns must complete with a standalone sentinel acknowledgement and a model-requested tool call; the acknowledgement may precede or follow explanation, and external byte comparisons check the files. Missing repository secrets fail instead of self-skipping. Fork and Dependabot pull requests run the complete keyless installed-wheel path but receive no key.
python/development.zh.md
+3 -5
View File
@@ -34,13 +34,11 @@ uv run --project python/sdk python scripts/smoke-python-runtime.py \
--scenario sdk-minimal --exe dist-exe/deepseek-harness-sdk-runtime-macos-arm64
```
SDK 场景会比对 `scripts/snapshots/python-sdk-single-exe/` 下已提交的期望输出。`minimal/model-visible.json` 固定 Linux／macOS `sdk-minimal` profile 所组装的系统提示词、对外公布的工具 schema 与模型可见消息；`minimal/win-x64/model-visible.json` 固定对应的 PowerShell 版本。因此，插件一旦贡献出计划外的系统分段或 user 消息，该任务即失败，且该 profile 发出的每条消息都会参与比对。`advanced-native/` 固定直接工具调用、子 agent 和 workflow；`advanced-ptc/` 固定程序调用与嵌套拒绝。两者都跨所有目标比对 SDK 结果及会话日志。`restart-native/` 针对同一持久化根目录启动两个完整 SDK 运行时进程，并跨所有目标固定其彼此隔离的模型历史、高层结果与独立持久日志。`sdk-minimal-in-history` 复用持久 shell 与编辑器场景，并在首次 shell 调用成功后更改一个分段。`minimal-in-history/prompt-history.json` 固定两个提示词版本、后续请求中不变的首条提示词、追加的 SDK system-message 事件及 `request/context.systemPromptUpdate`；工具 schema 保持不变，并独立检查编辑器创建的文件。`sdk-dynamic-tools` 在 `dynamic-tools/tool-history.json` 中固定原生工具的新增与移除，包括历史 header 引用，以及 Python SDK 结果、通知与持久事件的一致性。重新运行对应场景时加上 `--update-snapshots`，并在提交前审阅该差异。
SDK 场景会比对 `scripts/snapshots/python-sdk-single-exe/` 下已提交的期望输出。`minimal/model-visible.json` 固定 Linux／macOS `sdk-minimal` profile 所组装的系统提示词、对外公布的工具 schema 与模型可见消息；`minimal/win-x64/model-visible.json` 固定对应的 PowerShell 版本。因此，插件一旦贡献出计划外的系统分段或 user 消息，该任务即失败，且该 profile 发出的每条消息都会参与比对。`advanced/` 固定直接工具调用、子 agent 和 workflow；`advanced-ptc/` 固定程序调用与嵌套拒绝。两者都跨所有目标比对 SDK 结果及会话日志。`restart/` 针对同一持久化根目录启动两个完整 SDK 运行时进程，并跨所有目标固定其彼此隔离的模型历史、高层结果与独立持久日志。`sdk-minimal-in-history` 复用持久 shell 与编辑器场景，并在首次 shell 调用成功后更改一个分段。`minimal-in-history/prompt-history.json` 固定两个提示词版本、后续请求中不变的首条提示词、追加的 SDK system-message 事件及 `request/context.systemPromptUpdate`；工具 schema 保持不变，并独立检查编辑器创建的文件。`sdk-dynamic-tools` 在 `dynamic-tools/tool-history.json` 中固定原生工具的新增与移除，包括历史 header 引用，以及 Python SDK 结果、通知与持久事件的一致性。重新运行对应场景时加上 `--update-snapshots`，并在提交前审阅该差异。
保留的 `advanced/`、`restart/` 与 `scheduler-recovery/` 目录保存不可变的历史输出。只读比较测试覆盖这三个目录；可执行场景使用 `advanced-native/`、`advanced-ptc/`、`restart-native/` 与 `scheduler-recovery-native/`。
`scheduler-recovery/` 记录一个失败的工具轮次及后续完成的轮次。它保留原始的 `UNKNOWN` 轮次错误、第一个工具已完成的结果，以及其余调用的 `TOOL_OUTCOME_UNKNOWN` / `TOOL_NOT_STARTED` 结果。`sdk-snapshot` 和 `all` 包含此场景；`--scenario sdk-recovery` 单独运行它。
`scheduler-recovery-native/` 记录一个失败的工具轮次及后续完成的轮次。它保留原始的 `UNKNOWN` 轮次错误、第一个工具已完成的结果，以及其余调用的 `TOOL_OUTCOME_UNKNOWN` / `TOOL_NOT_STARTED` 结果。`sdk-snapshot` 和 `all` 包含此场景；`--scenario sdk-recovery` 单独运行它。
`advanced-native`、`advanced-ptc` 与 `restart-native` 的比较按每份输入自身的 Session 代际匹配原生 delivery 限定值，包括 SDK 事件与通知。捕获的代际及其他 delivery 代际保留数值。每一侧的 Session 角色必须使用同一个代际，新日志必须标识当前写入器。比较绝不重写已提交的记录。
`advanced`、`advanced-ptc` 与 `restart` 的比较按每份输入自身的 Session 代际匹配原生 delivery 限定值，包括 SDK 事件与通知。捕获的代际及其他 delivery 代际保留数值。每一侧的 Session 角色必须使用同一个代际，新日志必须标识当前写入器。未使用 `--update-snapshots` 时，比较不会写入文件。刷新只写入当前 writer 的输出，并保留更早的 Session 代际。
可信拉取请求与 master 推送还会在各自选定的原生目标上运行 `--scenario sdk-live --installed-wheel`。该场景面向 `https://api.deepseek.com` 执行两个使用工具的轮次：立即检查已创建文件，将其内容替换为仅宿主知道的随机挑战值，并要求第二轮将变更后的内容复制到全新的回执文件，且不修改源文件。两个轮次都必须完成、返回独占一行的哨兵确认并由模型请求调用工具；确认可以位于解释之前或之后，文件通过外部逐字节比较验证。仓库密钥缺失时失败，而不是自行 skip。Fork 与 Dependabot 拉取请求会运行完整的 keyless 安装后 wheel 路径，但不会获得密钥。
python/sdk/tests/test_smoke_model.py
+4 -13
View File
@@ -643,19 +643,10 @@ def test_native_writer_comparison_rejects_mixed_expected_role_generations(tmp_pa
SMOKE["compare_snapshot_files"](fresh, False, tmp_path, tuple(golden), native_writer_output=True)
@pytest.mark.parametrize(("scenario", "filenames"), [
("advanced", SMOKE["ADVANCED_SNAPSHOT_FILENAMES"]),
("advanced-native", SMOKE["ADVANCED_SNAPSHOT_FILENAMES"]),
("advanced-ptc", ("result.json", "session.v4.jsonl")),
("restart", SMOKE["RESTART_SNAPSHOT_FILENAMES"]),
("restart-native", SMOKE["RESTART_SNAPSHOT_FILENAMES"]),
("scheduler-recovery", SMOKE["RECOVERY_SNAPSHOT_FILENAMES"]),
("scheduler-recovery-native", SMOKE["RECOVERY_SNAPSHOT_FILENAMES"]),
])
def test_committed_python_native_goldens_compare_current_output_without_rewriting(
scenario: str, filenames: tuple[str, ...],
) -> None:
directory = ROOT / "scripts" / "snapshots" / "python-sdk-single-exe" / scenario
@pytest.mark.parametrize("scenario", ["ADVANCED", "RESTART"])
def test_committed_python_native_goldens_compare_current_output_without_rewriting(scenario: str) -> None:
directory = SMOKE[f"{scenario}_SNAPSHOT_DIRECTORY"]
filenames = SMOKE[f"{scenario}_SNAPSHOT_FILENAMES"]
before = {path.name: path.read_bytes() for path in directory.iterdir() if path.is_file()}
selected = SMOKE["selected_snapshot_session_files"](directory)
first = next(iter(selected.values()))