Merge pull request #6354 from deepseek-harness/agent/web-fetch-fakeip-469be4db70d0

fix(web)!: let web_fetch reach non-public destinations
This commit is contained in:
Turtle authored and GitHub committed 2026-10-20 14:13:24 +08:00
commit c3e9b28845
52 files changed
+255 -806

No files matched your search

@@ -24,8 +24,8 @@
en: ec3703b7e7c6b23d
zh: 325755e6e90092da
/agent-note-web-capability-seam-stable-tools-over-multiple-providers/fetch-request-and-result-schema:
en: f734fdc57a6546c9
zh: 027c28cbc584a8c2
en: 6dba3b9a51c28b6d
zh: 58a93f6ba2d269ea
/agent-note-web-capability-seam-stable-tools-over-multiple-providers/tool-consumer-behavior:
en: 62127883453dd79c
zh: 01503a8177a0ddab
@@ -55,19 +55,16 @@
zh: 64032ea6ab61b836
/agent-note-web-capability-seam-stable-tools-over-multiple-providers/alternatives-considered/mirror-claude-code-s-url-prompt-webfetch-shape:
en: 56a080de39e96056
zh: ae4d070aa7657503
/agent-note-web-capability-seam-stable-tools-over-multiple-providers/alternatives-considered/validate-dns-and-then-call-an-ordinary-fetch:
en: aa6c1e7e6374a875
zh: 045c8af6cd44c3e1
/agent-note-web-capability-seam-stable-tools-over-multiple-providers/alternatives-considered/block-private-looking-hostname-strings-without-pinning-resolved-addresses:
en: 3d906f3056733479
zh: d652723eb5372007
zh: 937336a73de70518
/agent-note-web-capability-seam-stable-tools-over-multiple-providers/alternatives-considered/restrict-fetch-to-public-addresses:
en: af3014c0de7ff66f
zh: 509040c9183327fe
/agent-note-web-capability-seam-stable-tools-over-multiple-providers/alternatives-considered/require-per-call-approval-before-public-fetches:
en: f8d1fc8e9c9659cf
zh: da5792520e35cba0
en: 3ea94ffb142d144a
zh: 89645f68412f6769
/agent-note-web-capability-seam-stable-tools-over-multiple-providers/consequences:
en: 12fd20ba54818215
zh: c1ca000b13666701
en: 9774448fd6b21c41
zh: 8411c590ad70d5c4
/agent-note-web-capability-seam-stable-tools-over-multiple-providers/deferred-work:
en: 48b7b8e7310cd56b
zh: d406cd6a8a649e58
@@ -202,7 +202,7 @@ Full page retrieval remains the job of `web_fetch(url)`. Search snippets are dis
## Fetch request and result schema
The `web_fetch` implementation is an anonymous public HTTP(S) fetch provider, `http`. It fetches bytes from a concrete URL, resolves and pins public destinations, applies the transport hygiene below, decodes textual content, and returns only the minimal model-useful result: final URL, status code, body, and truncation. It carries no browser cookies, editor credentials, git credentials, internal auth tokens, or implicit access to private services.
The `web_fetch` implementation is an anonymous HTTP(S) fetch provider, `http`. It fetches bytes from a concrete URL, applies the transport hygiene below, decodes textual content, and returns only the minimal model-useful result: final URL, status code, body, and truncation. It carries no browser cookies, editor credentials, git credentials, or internal auth tokens. It does not restrict destination addresses.
The seam request stays smaller than OpenCode's model-facing tool:
@@ -238,14 +238,12 @@ The provider owns safe resource retrieval: URL validation, HTTP transport, redir
The fetch provider's resource controls:
- Only `http:` and `https:` URLs are accepted; credentials in URLs are rejected.
- A literal address or the complete result of one hostname lookup must contain only globally reachable unicast IPv4 or IPv6 destinations. IPv6 resolution also discovers the active DNS64 prefix and rejects NAT64 addresses that translate to non-public IPv4. Loopback, private, link-local, carrier-grade NAT, multicast, reserved, transition, translation, and private IPv4-mapped IPv6 addresses are rejected.
- The request retains that validated address set in an Undici lookup callback instead of resolving the hostname again. The original hostname remains the HTTP Host and TLS SNI value, while DNS rebinding cannot replace the connection destination after validation.
- Maximum URL length, response byte cap, decoded body character cap, timeout, and redirect hop cap are enforced.
- Abort signals propagate through network fetches and expensive decoding.
- Only same-origin redirects are followed automatically; each followed hop performs a fresh public-address lookup and pins its own connection. A cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call and fresh public-address validation. (Claude Code's WebFetch uses this same model — it does not auto-follow a cross-host redirect; it returns the redirect target to the model for a fresh call.)
- Only same-origin redirects are followed automatically; each followed hop repeats URL validation. A cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call. (Claude Code's WebFetch uses this same model — it does not auto-follow a cross-host redirect; it returns the redirect target to the model for a fresh call.)
- Requests carry an explicit product user agent rather than silently impersonating a browser.
The provider rejects an entire DNS answer set when any address is not public instead of silently filtering the unsafe members. This fail-closed rule prevents connection-family selection or fallback from reaching an address that did not satisfy the public-network policy.
Requests use the process-wide dispatcher, so the installed outbound proxy policy and the system resolver decide the connection. Loopback, private-network, and cloud metadata destinations are reachable.
## Tool consumer behavior
@@ -313,17 +311,13 @@ Rejected for the first version. Those providers often return extracted or summar
Rejected for the seam. `prompt` turns fetch into LLM summarization and couples public-web retrieval to a model provider. The harness seam should fetch and decode deterministically; `dsh-tool-web` can later offer summaries as a presentation mode without making `ctx.web` depend on `ctx.llm`.
### Validate DNS and then call an ordinary fetch
### Restrict fetch to public addresses
Rejected because an ordinary fetch resolves the hostname again when it opens the connection. An attacker can return a public address during validation and a private address during the second lookup. Passing the validated answer set through the connection's lookup callback closes that rebinding interval while preserving hostname-based HTTP and TLS behavior.
### Block private-looking hostname strings without pinning resolved addresses
Rejected because hostname syntax does not establish the connection destination: an arbitrary public-looking name can resolve to loopback, a private range, or a cloud metadata address. Address classification belongs after resolution, and every address available to connection fallback must pass it.
Rejected. Resolving each hostname, rejecting any non-public answer, and pinning the connection to the validated set does not confine the model, which reaches the same networks through mounted shell tools, and it refuses every page on hosts whose DNS answers with non-public addresses, such as TUN-mode proxies with fake-IP DNS. Deployments that must keep the model away from private destinations need network confinement, a `tools/pre-execute` policy, or no fetch tool.
### Require per-call approval before public fetches
Rejected for the shipped presets. Public-address validation blocks SSRF destinations, while per-call confirmation would interrupt ordinary browsing without controlling public data egress reliably: a model can reach the same public network through mounted shell tools. Deployments that require a dedicated confirmation step can add a `tools/pre-execute` policy or disable `web_fetch`.
Rejected for the shipped presets. Per-call confirmation would interrupt ordinary browsing without controlling data egress reliably: a model can reach the same networks through mounted shell tools. Deployments that require a dedicated confirmation step can add a `tools/pre-execute` policy or disable `web_fetch`.
## Consequences
@@ -335,7 +329,7 @@ Rejected for the shipped presets. Public-address validation blocks SSRF destinat
**Provider state can change after startup.** A tool can be visible in the request assembled at step start and lose its provider before execution. The execution path resolves again and fails with a structured error.
**Fetch is a network boundary, not just a read-only tool.** Public-address validation and connection pinning prevent `web_fetch` from reaching non-public destinations, but a model can still disclose data through a public URL and fetched text remains untrusted model input. The shipped `cordis`, `code`, and `standard` presets expose `web_fetch` in every sandbox and approval mode without per-call confirmation.
**Fetch is a network boundary, not just a read-only tool.** `web_fetch` reaches any destination the host can, including loopback and private-network services, a model can disclose data through a URL, and fetched text remains untrusted model input. The shipped `cordis`, `code`, and `standard` presets expose `web_fetch` in every sandbox and approval mode without per-call confirmation.
**Large web content can damage context quality.** Providers enforce byte/character caps and report `truncated`; `tool-web` formats bounded model output with clear continuation or follow-up guidance.
@@ -202,7 +202,7 @@ Exa 搜索将提供方扁平 `results[]` 的每一项映射为 `WebSearchSource`
## Fetch 请求与结果 schema
`web_fetch` 的实现是一个匿名公开 HTTP(S) fetch 提供方 `http`。它从具体 URL 获取字节,解析并固定公开目的地址,应用下述传输卫生措施,解码文本内容,并仅返回最小的模型可用结果:最终 URL、状态码、正文和截断标志。它不携带浏览器 cookie、编辑器凭证、git 凭证、内部认证令牌,也不隐式访问私有服务。
`web_fetch` 的实现是一个匿名 HTTP(S) fetch 提供方 `http`。它从具体 URL 获取字节,应用下述传输卫生措施,解码文本内容,并仅返回最小的模型可用结果:最终 URL、状态码、正文和截断标志。它不携带浏览器 cookie、编辑器凭证、git 凭证或内部认证令牌。它不限制目的地址。
seam 请求比 OpenCode 的面向模型工具更小:
@@ -238,14 +238,12 @@ export type WebFetchBody =
fetch 提供方的资源控制:
- 仅接受 `http:` 和 `https:` URL;拒绝 URL 中的凭证。
- 字面 IP 地址或 hostname 一次解析得到的完整结果只能包含全球可达的单播 IPv4 或 IPv6 目的地址。IPv6 解析还会发现当前 DNS64 前缀,并拒绝转换到非公开 IPv4 的 NAT64 地址。loopback、私有、link-local、运营商级 NAT、多播、保留、过渡、转换和映射到私有 IPv4 的 IPv6 地址都会被拒绝。
- 请求通过 Undici lookup 回调保留这一组已验证地址,不会再次解析 hostname。原 hostname 仍作为 HTTP Host 与 TLS SNI 值,而 DNS rebinding 无法在验证后替换连接目的地址。
- 强制执行最大 URL 长度、响应字节上限、解码正文字符上限、超时和重定向跳数上限。
- Abort 信号传播到网络获取和高开销解码。
- 仅自动跟随同源重定向;每个跟随的跳转都会重新解析公开地址,并把自己的连接固定到解析结果。跨源重定向以 `WEB_REDIRECT_BLOCKED` 失败,要求一次新的工具调用和新的公开地址校验。(Claude Code 的 WebFetch 使用同样的模型——它不自动跟随跨主机重定向,而是将重定向目标返回给模型以发起新调用。)
- 仅自动跟随同源重定向;每个跟随的跳转都会重复 URL 校验。跨源重定向以 `WEB_REDIRECT_BLOCKED` 失败,要求一次新的工具调用。(Claude Code 的 WebFetch 使用同样的模型——它不自动跟随跨主机重定向,而是将重定向目标返回给模型以发起新调用。)
- 请求携带显式的产品 User-Agent,而非静默伪装浏览器。
只要 DNS 完整解析结果中存在任一非公开地址,提供方就会拒绝整个结果,而不是静默过滤不安全成员。该 fail-closed 规则可防止连接的地址族选择或回退触及未满足公开网络策略的地址。
请求使用进程级 dispatcher,因此由已安装的出站代理策略与系统解析器决定连接。loopback、内网与云 metadata 目的地址都可访问。
## 工具消费方行为
@@ -313,17 +311,14 @@ fetch 提供方的资源控制:
在 seam 层面否决。`prompt` 将 fetch 变成 LLM 摘要,并将公开 web 获取耦合到模型提供方。harness seam 应当确定性地获取和解码;`dsh-tool-web` 日后可以将摘要作为展示模式提供,而无需让 `ctx.web` 依赖 `ctx.llm`。
### 验证 DNS 后调用普通 fetch
<a id="restrict-fetch-to-public-addresses"></a>
### 只允许抓取公开地址
否决,因为普通 fetch 在打开连接时会再次解析 hostname。攻击者可以在验证时返回公开地址,在第二次解析时返回私有地址。把已验证解析结果通过连接的 lookup 回调传入,可以在保留基于 hostname 的 HTTP 与 TLS 行为的同时关闭这一 rebinding 时间窗口。
### 只阻断看起来像私网的 hostname 字符串,不固定解析地址
否决,因为 hostname 语法无法确定连接目的地址:任意看似公开的名称都可能解析到 loopback、私有网段或云 metadata 地址。地址分类必须在解析后执行,连接回退可使用的每个地址都必须通过校验。
否决。解析每个 hostname、拒绝任何非公开解析结果并把连接固定到已验证地址,并不能约束模型,因为模型可以通过已挂载的 shell 工具访问相同网络;同时在 DNS 返回非公开地址的主机上(例如使用 fake-IP DNS 的 TUN 模式代理),这会拒绝所有页面。需要阻止模型访问私有目的地址的部署应使用网络隔离、`tools/pre-execute` 策略,或不挂载抓取工具。
### 在公开抓取前要求逐次审批
已交付的 preset 不采用这一方案。公开地址校验会阻断 SSRF 目的地址,而逐次确认会打断普通浏览,却不能可靠控制公开数据出站:模型可以通过已挂载的 shell 工具访问同一公开网络。要求专门确认步骤的部署可以添加 `tools/pre-execute` 策略或禁用 `web_fetch`。
已交付的 preset 不采用这一方案。逐次确认会打断普通浏览,却不能可靠控制数据出站:模型可以通过已挂载的 shell 工具访问相同网络。要求专门确认步骤的部署可以添加 `tools/pre-execute` 策略或禁用 `web_fetch`。
## 后果
@@ -335,7 +330,7 @@ fetch 提供方的资源控制:
**提供方状态可能在启动后变化。** 一个工具可能在步骤开始时组装的请求中可见,但在执行前失去其提供方。执行路径重新解析并以结构化错误失败。
**Fetch 是网络边界,不仅仅是只读工具。** 公开地址校验与连接固定可防止 `web_fetch` 触达非公开目的地址,但模型仍可通过公开 URL 泄露数据,抓取文本也仍是不受信任的模型输入。已交付的 `cordis`、`code` 与 `standard` preset 会在所有 sandbox 和审批模式下暴露 `web_fetch`,无需逐次确认。
**Fetch 是网络边界,不仅仅是只读工具。** `web_fetch` 可以访问主机能访问的任何目的地址,包括 loopback 与内网服务;模型可以通过 URL 泄露数据,抓取文本也仍是不受信任的模型输入。已交付的 `cordis`、`code` 与 `standard` preset 会在所有 sandbox 和审批模式下暴露 `web_fetch`,无需逐次确认。
**大量 web 内容可能损害上下文质量。** 提供方强制执行字节/字符上限并报告 `truncated`;`tool-web` 格式化有界的模型输出,附带清晰的继续或后续引导。
@@ -9,14 +9,14 @@
en: 6379f0a2522f4c46
zh: c45ffeb5089be274
/agent-note-one-outbound-proxy-policy-installed-before-anything-can-request/decision:
en: 92526818ff4870c2
zh: 8c54ff3f4941d5ad
en: 6435c336ab53ca0f
zh: 57070799a7061f85
/agent-note-one-outbound-proxy-policy-installed-before-anything-can-request/alternatives-considered:
en: df6c078f239a11f5
zh: 914da26836aa373b
/agent-note-one-outbound-proxy-policy-installed-before-anything-can-request/consequences:
en: 6a4acbab915f1bd1
zh: 2c8ffd62ba74ff96
en: ff1898a64e5db79c
zh: 0a77d02385fb3506
/agent-note-one-outbound-proxy-policy-installed-before-anything-can-request/testing:
en: 0d7eaa666c94a5cd
zh: 210fa098f154d137
en: fddb0be6abbd809e
zh: 88843f0f3ea60555
@@ -36,7 +36,7 @@ This keeps `proxyForUrl()` and the dispatcher answering from one set of values.
**Rejection is quiet, and never reroutes the refused scheme.** A slot the user filled and this package refused keeps that scheme direct rather than falling through to `ALL_PROXY` or the HTTP proxy, so the diagnostic and the route agree. A SOCKS URL, an unparseable string, or an unsupported scheme is reported on stderr and skipped — the variable may have been exported for other tools, and a typo in it must not stop the agent from starting. The environment is the only source, so no configuration surface exists where `AGENTS.md`'s fail-loud rule would apply instead.
**Through a proxy, `web_fetch` stops resolving and pinning.** The provider validates a public address set and pins the connection to it. Through a proxy there is nothing to pin — the proxy performs the origin's DNS — and a pinned direct connection would bypass the proxy entirely. So a proxied hop skips resolution, and configuring a proxy is a statement that the proxy is trusted with destination selection. A hop the policy bypasses, which includes every loopback and every `NO_PROXY` entry, takes the resolved-and-pinned path unchanged. Kimi Code and Claude Code reached this same conclusion independently.
**`web_fetch` uses the installed dispatcher.** The provider sends every request through the process-wide dispatcher, so a proxied hop goes through the proxy, which performs the origin's DNS, and a hop the policy bypasses, which includes every loopback and every `NO_PROXY` entry, connects directly. The provider does not restrict destination addresses ([web capability seam](2026-06-24-web-capability-seam.md#restrict-fetch-to-public-addresses)).
The URL-level policy is untouched: `http(s)` only, no embedded credentials, the length cap, and the cross-origin redirect refusal all still apply on every hop.
@@ -54,7 +54,7 @@ Weighed against that, telemetry is the one outbound channel whose loss costs the
**Every call site carries an egress test.** Each owning package's `egress.spec.ts` drives its actual transport through a fake proxy and checks the observed route. These tests cover search backends, pi-ai discovery, MCP over HTTP, child Node processes, worker threads and telemetry's direct-route exception. They detect dependency transport changes that a static call-site check cannot observe.
**A gate keeps the defect from returning.** `verify-no-bare-dispatcher` parses the TypeScript AST — `scripts/AGENTS.md` requires syntax-aware discovery, and a line-wise regex missed both the `{ dispatcher }` shorthand this repository already uses and a `new Alias(...)` behind a renamed import. It rejects an undici agent construction and an explicit `dispatcher` option outside the owning package. `proxyRouteFor(url)` is the sanctioned replacement, and the one call site that genuinely owns its transport — `web-fetch-http`, pinning a request to addresses it validated — says so with a `proxy-exempt:` comment. The rule exists because `web-fetch-http`'s original `new Agent` was entirely reasonable when it was written — proxying simply did not exist yet, and nothing would have caught it.
**A gate keeps the defect from returning.** `verify-no-bare-dispatcher` parses the TypeScript AST — `scripts/AGENTS.md` requires syntax-aware discovery, and a line-wise regex missed both the `{ dispatcher }` shorthand this repository already uses and a `new Alias(...)` behind a renamed import. It rejects an undici agent construction and an explicit `dispatcher` option outside the owning package. `proxyRouteFor(url)` is the sanctioned replacement, and a call site that genuinely owns its transport says so with a `proxy-exempt:` comment. The rule exists because `web-fetch-http`'s original `new Agent` was entirely reasonable when it was written — proxying simply did not exist yet, and nothing would have caught it.
## Alternatives considered
@@ -78,8 +78,6 @@ A user who exports `HTTPS_PROXY`, or writes it into `$DSH_HOME/.env`, is proxied
Because the operating system's settings are not read, the user-facing documentation is now load-bearing rather than supplementary: a user who only toggled "system proxy" in a proxy application gets nothing and no diagnostic. `docs/user/guide/network-proxy.md` therefore states which variables to export and why a browser is proxied when a terminal is not — the three-mechanism confusion is the single most common report, and it is not specific to this harness.
`web_fetch`'s safety story now has two shapes, and its README says so: direct hops keep address validation and pinning, proxied hops delegate destination selection to a proxy the operator configured. This is the one outward-facing security promise the change alters.
Reaching Node's built-in `fetch` from a userland undici depends on both writing the legacy `Symbol.for('undici.globalDispatcher.1')` slot. That is an implicit cross-version coupling rather than a contract — corepack#834 records it breaking — so `tests/install.spec.ts` drives a real request through a loopback proxy. A version bump that breaks the coupling fails there instead of in the field.
The suite is hermetic against the developer's own environment: every Vitest configuration runs `scripts/test-proxy-environment.ts`, which clears all eight proxy names in both casings before any test, and `install.spec.ts` restores the machine's values around each case that sets its own. It has to. An exported lowercase `all_proxy` decided a test's outcome during development, because resolution reads lowercase first.
@@ -88,7 +86,7 @@ The suite is hermetic against the developer's own environment: every Vitest conf
`packages/util/http-proxy` holds 84 tests at 100% per-file coverage. Resolution covers precedence, the `ALL_PROXY` fallback, blank-shadowing, the SOCKS and malformed diagnostics, and the HTTPS-only environment that leaves `http:` direct; routing covers the whole loopback range structurally, and bypass matching covers suffixes, ports, both IPv6 spellings, and the CIDR entry that deliberately does not match. Installation drives a real loopback proxy and asserts the absolute-form request arrives, that a bypassed target does not, and that disposal restores the dispatcher, the policy, and the environment. Every case installs through `installProxyFromEnvironment`, so no test can assert a policy object a real launch could not produce.
`packages/web/web-fetch-http/tests/proxy.spec.ts` asserts the decision that matters most: under a proxy the public-address resolver is never called, while a bypassed hop still calls it exactly once, and the cross-origin redirect refusal survives on the proxied path.
`packages/web/web-fetch-http/tests/proxy.spec.ts` asserts that a proxied hop reaches the proxy, a bypassed hop connects directly, and the cross-origin redirect refusal survives on the proxied path.
`verify-no-bare-dispatcher.spec.ts` proves the gate rejects the exact shape this package was introduced to fix, accepts `proxyRouteFor`, accepts an annotated exemption, and passes on the current tree.
@@ -36,7 +36,7 @@ Node 内置的 `fetch` 会忽略 `HTTP_PROXY` 与 `HTTPS_PROXY`。开发者运
**拒绝是静默的,且绝不为被拒协议改道。** 用户填写而被本包拒绝的槽位,会让该协议保持直连,而不是继续回退到 `ALL_PROXY` 或 HTTP 代理,从而让诊断与实际路由一致。SOCKS URL、无法解析的字符串或不受支持的协议,会在 stderr 上报告并跳过——该变量可能是为其他工具导出的,它的笔误不应阻止 agent 启动。环境是唯一来源,因此不存在一个本应适用 `AGENTS.md` 「配置错误必须响」规则的配置面。
**经由代理时,`web_fetch` 不再解析与固定地址。** 该提供方会校验一组公网地址并把连接固定到其上。经由代理时没有可固定的对象——origin 的 DNS 由代理执行——而固定后的直连会彻底绕开代理。因此代理转发的一跳跳过解析,配置代理即表示信任该代理进行目的地选择。被策略绕过的一跳,包括每一个 loopback 与每一条 `NO_PROXY` 条目,仍走原有的解析并固定路径。Kimi Code 与 Claude Code 各自独立得出了同一结论。
**`web_fetch` 使用已安装的 dispatcher。** 该提供方的每个请求都经过进程级 dispatcher,因此代理转发的一跳经过代理,由代理执行 origin 的 DNS;被策略绕过的一跳,包括每一个 loopback 与每一条 `NO_PROXY` 条目,则直接连接。该提供方不限制目的地址([web capability seam](2026-06-24-web-capability-seam.zh.md#restrict-fetch-to-public-addresses))。
URL 层策略未受影响:仅 `http(s)`、禁止内嵌凭据、长度上限与跨域重定向拒绝在每一跳上依然生效。
@@ -54,7 +54,7 @@ URL 层策略未受影响:仅 `http(s)`、禁止内嵌凭据、长度上限与
**每个出网点都配有出网测试。** 各所属包中的 `egress.spec.ts` 通过假代理驱动实际传输,并检查观察到的路由。这些测试覆盖搜索后端、pi-ai 发现、走 HTTP 的 MCP、子 Node 进程、worker 线程和遥测的直连例外。它们可发现静态调用点检查无法观察到的依赖传输变化。
**用门禁防止该缺陷复现。** `verify-no-bare-dispatcher` 解析 TypeScript AST——`scripts/AGENTS.md` 要求 source-ownership 门禁使用语法感知发现,而逐行正则漏掉了本仓库已在使用的 `{ dispatcher }` 简写,以及重命名导入后的 `new Alias(...)`。它在所属包之外拒绝 undici agent 构造与显式 `dispatcher` 选项。`proxyRouteFor(url)` 是受支持的替代;唯一一处确实自有传输的调用点——`web-fetch-http`,它把请求钉在已校验的地址上——用 `proxy-exempt:` 注释说明。这条规则之所以存在,是因为 `web-fetch-http` 里原本那行 `new Agent` 在写下时完全合理——那时根本还没有代理这回事,也没有任何机制会拦下它。
**用门禁防止该缺陷复现。** `verify-no-bare-dispatcher` 解析 TypeScript AST——`scripts/AGENTS.md` 要求 source-ownership 门禁使用语法感知发现,而逐行正则漏掉了本仓库已在使用的 `{ dispatcher }` 简写,以及重命名导入后的 `new Alias(...)`。它在所属包之外拒绝 undici agent 构造与显式 `dispatcher` 选项。`proxyRouteFor(url)` 是受支持的替代;确实自有传输的调用点用 `proxy-exempt:` 注释说明。这条规则之所以存在,是因为 `web-fetch-http` 里原本那行 `new Agent` 在写下时完全合理——那时根本还没有代理这回事,也没有任何机制会拦下它。
## Alternatives considered
@@ -78,8 +78,6 @@ URL 层策略未受影响:仅 `http(s)`、禁止内嵌凭据、长度上限与
由于不读取操作系统设置,面向用户的文档从补充材料变成了承重件:仅在代理软件里拨了「系统代理」开关的用户什么也得不到,且没有诊断。因此 `docs/user/guide/network-proxy.md` 说明了要导出哪些变量,以及为什么浏览器走代理而终端不走——这个「三套机制」的困惑是最常见的报障,且并非本 Harness 特有。
`web_fetch` 的安全叙述现在有两种形态,其 README 已如实说明:直连的一跳保留地址校验与固定,代理转发的一跳把目的地选择交给运维方配置的代理。这是本次变更唯一改动的对外安全承诺。
userland undici 能触及 Node 内置的 `fetch`,依赖于两者都会写入 legacy 的 `Symbol.for('undici.globalDispatcher.1')` 槽位。那是跨版本的隐式耦合而非约定——corepack#834 记录了它失效的实例——因此 `tests/install.spec.ts` 会驱动一次真实请求穿过 loopback 代理。破坏该耦合的版本升级会在那里失败,而不是流到线上。
测试套件对开发者自身的环境免疫:每份 Vitest 配置都会先运行 `scripts/test-proxy-environment.ts`,在任何测试之前清除全部八个代理变量名的两种大小写形式;`install.spec.ts` 则在每个自行设值的用例前后还原本机的值。这是必需的。开发过程中,一个已导出的小写 `all_proxy` 曾决定了某个测试的结果,因为解析优先读取小写。
@@ -88,7 +86,7 @@ userland undici 能触及 Node 内置的 `fetch`,依赖于两者都会写入 l
`packages/util/http-proxy` 有 84 个测试,per-file 覆盖率 100%。解析覆盖优先级、`ALL_PROXY` 兜底、空值遮蔽、SOCKS 与畸形值诊断,以及只设 https 变量时 `http:` 保持直连;路由以结构化方式覆盖整个 loopback 网段,绕过匹配覆盖后缀、端口、两种 IPv6 写法,以及刻意不匹配的 CIDR 条目。安装驱动一个真实的 loopback 代理,断言绝对形式的请求确实抵达、被绕过的目标不抵达,且 dispose 会还原 dispatcher、策略与环境。所有用例一律经 `installProxyFromEnvironment` 安装,因此没有测试能断言一次真实启动无法产生的策略对象。
`packages/web/web-fetch-http/tests/proxy.spec.ts` 断言了最关键的那个决定:经由代理时公网地址解析器完全不被调用,而被绕过的一跳仍恰好调用一次,且跨域重定向拒绝在代理路径上依然成立。
`packages/web/web-fetch-http/tests/proxy.spec.ts` 断言代理转发的一跳抵达代理、被绕过的一跳直接连接,且跨域重定向拒绝在代理路径上依然成立。
`verify-no-bare-dispatcher.spec.ts` 证明该门禁能拒掉本包所要修复的那种写法、接受 `proxyRouteFor`、接受带注释的豁免,并在当前代码树上通过。
@@ -18,8 +18,8 @@
en: ad7b3e4fbe528cf2
zh: 8ab8779bfdaaa4fa
/agent-note-direct-deepseek-search-and-fetch-apis/decision/fetch-metadata-and-ownership:
en: 8051417b08739ed0
zh: c11d35a346014436
en: 7566e400f9ab3a51
zh: 118b2e84d0030480
/agent-note-direct-deepseek-search-and-fetch-apis/decision/default-migration:
en: d5a8e36b545f1c04
zh: d741fd0789401c62
@@ -32,7 +32,7 @@ Direct retrieval emits no auxiliary model-request event because Harness makes no
Fetch sends a URL and returns Markdown without an extra Harness summarization request. The API supplies neither a final redirected URL nor a target HTTP status, and the shared result has no title field. The provider keeps the requested URL, returns the Markdown body, and reports HTTP 200 for a successful response: the Fetch API answers 404 when the URL is unavailable, blocked by site policy, or returns an error, or when no Markdown can be extracted. On 2026-10-16 an unresolvable host returned HTTP 500 instead, so every non-success status is a retrieval failure rather than a specific target diagnosis. The shared fetch result and its consumers stay unchanged.
The HTTP provider retains its own public-address validation, connection pinning, and resource limits. Selecting DeepSeek delegates page retrieval and target access policy to that service; Harness rejects redirects of the API request and retains cancellation and bounded tool output. The reported HTTP 200 therefore means the service retrieved the page, not that Harness observed the target response. The guide's Fetch 403 means crawler-policy rejection, while 404 means no usable page content.
The HTTP provider retains its own URL validation and resource limits. Selecting DeepSeek delegates page retrieval and target access policy to that service; Harness rejects redirects of the API request and retains cancellation and bounded tool output. The reported HTTP 200 therefore means the service retrieved the page, not that Harness observed the target response. The guide's Fetch 403 means crawler-policy rejection, while 404 means no usable page content.
### Default migration
@@ -32,7 +32,7 @@ Status: implemented
Fetch 发送 URL,返回 Markdown,无需 Harness 额外发起摘要请求。API 既不提供重定向后的最终 URL,也不提供目标 HTTP 状态,而共享结果没有标题字段。提供方保留请求 URL,返回 Markdown 正文,并对成功响应报告 HTTP 200:URL 不可用、被站点策略拦截、目标站点返回错误或无法提取 Markdown 时,Fetch API 会返回 404。2026-10-16 无法解析的主机实际返回 HTTP 500,因此任何非成功状态都视为抓取失败,而不是对目标的具体诊断。共享抓取结果及其消费方保持不变。
HTTP 提供方继续负责自身的公开地址校验、连接固定和资源限制。选择 DeepSeek 会将页面抓取与目标访问策略交给该服务;Harness 拒绝 API 请求重定向,并保留取消与工具输出限额。因此报告的 HTTP 200 表示服务已获取页面,而不是 Harness 观察到了目标响应。指南中的 Fetch 403 表示抓取器策略拒绝,404 则表示没有可用的页面内容。
HTTP 提供方继续负责自身的 URL 校验和资源限制。选择 DeepSeek 会将页面抓取与目标访问策略交给该服务;Harness 拒绝 API 请求重定向,并保留取消与工具输出限额。因此报告的 HTTP 200 表示服务已获取页面,而不是 Harness 观察到了目标响应。指南中的 Fetch 403 表示抓取器策略拒绝,404 则表示没有可用的页面内容。
### 默认值迁移
+2 -2
View File
@@ -24,8 +24,8 @@
en: 850434e8ca162221
zh: 4893613f97a21eb8
/dsh-cli-behavior-reference/shared-deployment-behavior:
en: 6a7abc9339e8f37f
zh: 133adb834c8e195d
en: f40edcf903a51f3f
zh: 8a9a915451047c8b
/dsh-cli-behavior-reference/source-execution:
en: 05c2b23790f7b296
zh: 51f809020fb230a1
+1 -1
View File
@@ -134,7 +134,7 @@ New sessions in base-backed profiles default to the `workspace-write` permission
## Shared deployment behavior
The base bundle mounts the native DeepSeek adapter, settings and credential providers, stable `web_search` and `web_fetch`, the public-only HTTP fetch provider, default-on DeepSeek session-log upload, and feedback-gated OTel upload for all users. Provider credentials resolve from the inherited environment, `$DSH_HOME/.credentials.yaml`, the invoking directory's `.env`, then `$DSH_HOME/.env`; the managed document is never materialized into `process.env`, while both `.env` files are ordinary launch environment layers. Search uses `DEEPSEEK_API_KEY` and accepts `DEEPSEEK_SEARCH_BASE_URL`. Enabled fetch calls run in every sandbox and approval mode without per-call confirmation; the provider rejects non-public destinations before connecting. The Web app disables the base tool row and exposes the same tools through its `cordis`, `ptc`, and `standard` agent presets.
The base bundle mounts the native DeepSeek adapter, settings and credential providers, stable `web_search` and `web_fetch`, the HTTP fetch provider, default-on DeepSeek session-log upload, and feedback-gated OTel upload for all users. Provider credentials resolve from the inherited environment, `$DSH_HOME/.credentials.yaml`, the invoking directory's `.env`, then `$DSH_HOME/.env`; the managed document is never materialized into `process.env`, while both `.env` files are ordinary launch environment layers. Search uses `DEEPSEEK_API_KEY` and accepts `DEEPSEEK_SEARCH_BASE_URL`. Enabled fetch calls run in every sandbox and approval mode without per-call confirmation and can reach loopback and private-network hosts. The Web app disables the base tool row and exposes the same tools through its `cordis`, `ptc`, and `standard` agent presets.
Feedback is recorded in the Session log without starting model work. The [DeepSeek session-log contributor](../../../packages/session/session-log-deepseek/README.md) sends complete unaccepted log suffixes with subsequent DeepSeek requests by default, including requests sent through configured gateways; turn off **Settings → General → Upload Session Log when using the official model API** in Web, or set its `enabled` configuration to `false` to opt out. [OTel session upload](../../../packages/session/session-telemetry-otel/README.md) applies to all users and providers, including `deepseek-official`, without requiring a request header. The base defaults to `FEEDBACK_ONLY`: new own text feedback, message ratings, edits, and withdrawals release the complete canonical prefix through that event, including stored context; later records wait for the next explicit feedback. Inherited parent feedback does not authorize a fork. Requests, restoration, mount, and HMR do not trigger capture. SDK batching may finish an authorized upload without further interaction or model work. `DSH_TELEMETRY_MODE=DISABLED` disables OTel delivery; `FULL` is rejected, and any non-empty `DSH_TELEMETRY_DISABLED` disables its row. `DSH_TELEMETRY_OTLP_URL` selects the collector. Handoff is best-effort, not collector acceptance; no durable outbox or retry guarantee is provided. These OTel settings do not enable or disable the DeepSeek contribution. Neither path changes model input, but exports can include message text, tool arguments and results, and workspace paths.
+1 -1
View File
@@ -136,7 +136,7 @@ dsh web --help
## 共享部署行为
基础组合包挂载原生 DeepSeek 适配器、settings 与凭据提供方、稳定的 `web_search` 和 `web_fetch`、仅限公网的 HTTP fetch 提供方,默认开启的 DeepSeek 会话日志上传,以及面向所有用户的反馈门控 OTel 上传。提供方凭据依次从继承环境、`$DSH_HOME/.credentials.yaml`、调用目录的 `.env` 和 `$DSH_HOME/.env` 解析;受管文档从不物化进 `process.env`,而两个 `.env` 文件都是普通启动环境层。搜索使用 `DEEPSEEK_API_KEY` 并接受 `DEEPSEEK_SEARCH_BASE_URL`。已启用的抓取调用会在所有 sandbox 与审批模式下执行,无需逐次确认;提供方会在连接前拒绝非公开目的地址。Web app 会禁用 base 工具配置项,再通过 `cordis`、`ptc` 与 `standard` agent preset 暴露相同工具。
基础组合包挂载原生 DeepSeek 适配器、settings 与凭据提供方、稳定的 `web_search` 和 `web_fetch`、HTTP fetch 提供方,默认开启的 DeepSeek 会话日志上传,以及面向所有用户的反馈门控 OTel 上传。提供方凭据依次从继承环境、`$DSH_HOME/.credentials.yaml`、调用目录的 `.env` 和 `$DSH_HOME/.env` 解析;受管文档从不物化进 `process.env`,而两个 `.env` 文件都是普通启动环境层。搜索使用 `DEEPSEEK_API_KEY` 并接受 `DEEPSEEK_SEARCH_BASE_URL`。已启用的抓取调用会在所有 sandbox 与审批模式下执行,无需逐次确认,并且可以访问 loopback 与内网主机。Web app 会禁用 base 工具配置项,再通过 `cordis`、`ptc` 与 `standard` agent preset 暴露相同工具。
反馈记录在会话日志中,不会启动模型工作。[DeepSeek 会话日志贡献器](../../../packages/session/session-log-deepseek/README.zh.md)默认随后续 DeepSeek 请求发送尚未确认接收的完整日志后缀,包括经已配置网关发送的请求;可在 Web 的**设置 → 通用 → 在使用官方模型 API 时上传 Session Log**中关闭上传,或将其 `enabled` 配置设为 `false`。[OTel 会话上传](../../../packages/session/session-telemetry-otel/README.zh.md)适用于所有用户和提供方,包括 `deepseek-official`,无需请求头。基础配置默认使用 `FEEDBACK_ONLY`:新的自身文本反馈、消息评分、编辑与撤回会释放截至该事件的完整规范日志前缀,包含存储的上下文;后续记录等待下一次显式反馈。继承的父级反馈不构成 fork 的授权。请求、恢复、挂载和 HMR 不触发捕获。SDK 批处理可完成已授权上传,无需进一步交互或模型工作。`DSH_TELEMETRY_MODE=DISABLED` 禁止 OTel 投递;`FULL` 被拒绝,任何非空的 `DSH_TELEMETRY_DISABLED` 都会禁用其配置行。`DSH_TELEMETRY_OTLP_URL` 选择采集端。交接尽力而为,不代表采集端接受;不提供持久化 outbox 或重试保证。这些 OTel 设置不会开启或关闭 DeepSeek 贡献。两条路径都不改变模型输入,但导出可能包含消息文本、工具参数和结果,以及工作区路径。
+2 -2
View File
@@ -468,8 +468,8 @@ function assertLeanChildRecord(agent: Agent): void {
* gaps: the `cordis_*` toolset executes model-written JavaScript that no
* sandbox row confines, `mcp_*` servers spawn outside `ctx.shell`, and `ralph`
* runs unsupervised rounds whose completion is a worker self-report.
* `web_fetch` is present because public-address enforcement and one-shot
* approval now confine its model-selected request target. The composition
* `web_fetch` is present because it adds no network reach beyond the
* mounted shell tools. The composition
* Agent Note owns the rationale and its sources.
*/
const EXPECTED_TOOLS = [
+1 -2
View File
@@ -489,7 +489,6 @@ flowchart TD
pkg_remote_mock --> pkg_typert_protocol
pkg_typert_loader --> pkg_typert_registry
pkg_skill_office --> pkg_skill
pkg_web_fetch_http --> pkg_http_proxy
pkg_web_fetch_http --> pkg_timeout
pkg_web_fetch_http --> pkg_web
pkg_web_search_exa --> pkg_launch_environment
@@ -1600,7 +1599,7 @@ flowchart TD
| [`remote-mock`](../packages/test-support/remote-mock) | `test-support` | [`typert-protocol`](../packages/typert/protocol) |
| [`typert-loader`](../packages/typert/loader) | `typert` | [`typert-registry`](../packages/typert/registry) |
| [`skill-office`](../packages/skill/skill-office) | `skill` | [`skill`](../packages/skill/skill) |
| [`web-fetch-http`](../packages/web/web-fetch-http) | `web` | [`http-proxy`](../packages/util/http-proxy), [`timeout`](../packages/util/timeout), [`web`](../packages/web/web) |
| [`web-fetch-http`](../packages/web/web-fetch-http) | `web` | [`timeout`](../packages/util/timeout), [`web`](../packages/web/web) |
| [`web-search-exa`](../packages/web/web-search-exa) | `web` | [`launch-environment`](../packages/util/launch-environment), [`web`](../packages/web/web) |
| [`web-search-perplexity`](../packages/web/web-search-perplexity) | `web` | [`launch-environment`](../packages/util/launch-environment), [`web`](../packages/web/web) |
| [`spill`](../packages/spill/spill) | `spill` | [`brand`](../packages/util/brand), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) |
+1 -2
View File
@@ -491,7 +491,6 @@ flowchart TD
pkg_remote_mock --> pkg_typert_protocol
pkg_typert_loader --> pkg_typert_registry
pkg_skill_office --> pkg_skill
pkg_web_fetch_http --> pkg_http_proxy
pkg_web_fetch_http --> pkg_timeout
pkg_web_fetch_http --> pkg_web
pkg_web_search_exa --> pkg_launch_environment
@@ -1602,7 +1601,7 @@ flowchart TD
| [`remote-mock`](../packages/test-support/remote-mock) | `test-support` | [`typert-protocol`](../packages/typert/protocol) |
| [`typert-loader`](../packages/typert/loader) | `typert` | [`typert-registry`](../packages/typert/registry) |
| [`skill-office`](../packages/skill/skill-office) | `skill` | [`skill`](../packages/skill/skill) |
| [`web-fetch-http`](../packages/web/web-fetch-http) | `web` | [`http-proxy`](../packages/util/http-proxy), [`timeout`](../packages/util/timeout), [`web`](../packages/web/web) |
| [`web-fetch-http`](../packages/web/web-fetch-http) | `web` | [`timeout`](../packages/util/timeout), [`web`](../packages/web/web) |
| [`web-search-exa`](../packages/web/web-search-exa) | `web` | [`launch-environment`](../packages/util/launch-environment), [`web`](../packages/web/web) |
| [`web-search-perplexity`](../packages/web/web-search-perplexity) | `web` | [`launch-environment`](../packages/util/launch-environment), [`web`](../packages/web/web) |
| [`spill`](../packages/spill/spill) | `spill` | [`brand`](../packages/util/brand), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) |
+4 -4
View File
@@ -18,11 +18,11 @@
en: da78eb96d684ad47
zh: 4b726cc04c92ef13
/web-access/fetch-network-policy:
en: 261fce9249be8b2b
zh: bf6abf77679aca6d
en: 6851874be5841d90
zh: dc085bf3b82d77df
/web-access/errors:
en: b07c04f462b4a0d9
zh: de5f09cd2fb0324c
/web-access/the-service:
en: c81fdcdfd6e22861
zh: ac906c9148942041
en: c9e7dc85c08e64e1
zh: 277c52aac58f42dc
+2 -2
View File
@@ -131,7 +131,7 @@ Selection never depends on registration, config, or HMR order: a capability has
The shipped Cordis, Code, and Standard presets expose `web_fetch` in every sandbox and approval mode without per-call confirmation. File sandbox presets do not govern Web network access. A deployment that needs confirmation must add a `tools/pre-execute` policy or disable fetch.
The HTTP provider resolves each actual request, rejects non-public answers including private IPv4 reached through the active DNS64 prefix, pins the validated address set, and repeats enforcement for each same-origin redirect. A cross-origin redirect requires a new tool call and fresh public-address validation. These checks prevent SSRF access to non-public destinations but do not stop a model from sending data to a public URL.
The HTTP provider does not restrict destination addresses: `web_fetch` reaches loopback, private-network, and cloud metadata hosts like any other host, and a model can send data to any URL it can reach. A cross-origin redirect requires a new tool call. A deployment that must keep the model away from private destinations needs network confinement, a `tools/pre-execute` policy, or no fetch tool.
## Errors
@@ -139,7 +139,7 @@ The HTTP provider resolves each actual request, rejects non-public answers inclu
## The service
`WebRuntime` registers search and fetch providers, rejects duplicate ids with `WEB_DUPLICATE_PROVIDER`, and resolves providers at execution time with structured selection errors. The local fetch backend accepts only HTTP(S), rejects credentials, resolves each hostname once, rejects any answer set containing a non-public IPv4 or IPv6 destination or an active-prefix NAT64 translation to non-public IPv4, pins the request connection to the validated addresses, repeats those checks for every same-origin redirect hop, caps redirects, bytes, characters, and time, and decodes the body; the tool owns presentation.
`WebRuntime` registers search and fetch providers, rejects duplicate ids with `WEB_DUPLICATE_PROVIDER`, and resolves providers at execution time with structured selection errors. The local fetch backend accepts only HTTP(S), rejects credentials, sends each request through the installed proxy policy, repeats URL validation for every same-origin redirect hop, caps redirects, bytes, characters, and time, and decodes the body; the tool owns presentation.
<!-- BEGIN GENERATED cordis-surface (gen-cordis-catalog.ts) — do not edit between markers -->
+2 -2
View File
@@ -131,7 +131,7 @@ type WebFetchBody =
已交付的 Cordis、Code 与 Standard preset 会在所有 sandbox 和审批模式下暴露 `web_fetch`,无需逐次确认。文件 sandbox preset 不管辖 Web 网络访问。需要确认步骤的部署必须添加 `tools/pre-execute` 策略或禁用抓取。
HTTP 提供方会解析每个实际请求,拒绝包括通过当前 DNS64 前缀抵达私有 IPv4 在内的非公开结果,固定已验证的地址集合,并在每次同源重定向时重复强制执行。跨源重定向需要新的工具调用和新的公开地址校验。这些检查会阻止通过 SSRF 访问非公开目的地址,但不会阻止模型把数据发送到公开 URL。
HTTP 提供方不限制目的地址:`web_fetch` 会像访问其他主机一样访问 loopback、内网与云元数据主机,模型也可以把数据发送到它能访问的任何 URL。跨源重定向需要新的工具调用。需要阻止模型访问私有目的地址的部署必须使用网络隔离、`tools/pre-execute` 策略,或不挂载抓取工具。
## 错误
@@ -139,7 +139,7 @@ HTTP 提供方会解析每个实际请求,拒绝包括通过当前 DNS64 前
## 服务
`WebRuntime` 注册搜索与抓取提供方,以 `WEB_DUPLICATE_PROVIDER` 拒绝重复 id,并在执行时以结构化的选择错误解析提供方。本地抓取后端仅接受 HTTP(S)、拒绝凭证、对每个 hostname 只解析一次、拒绝包含任一非公开 IPv4/IPv6 目的地址或经当前前缀转换到非公开 IPv4 的 NAT64 地址的解析结果、把请求连接固定到已验证地址、对每一次同源重定向跳转重复这些校验、限制重定向次数、字节数、字符数和时间,并解码正文;展示由工具负责。
`WebRuntime` 注册搜索与抓取提供方,以 `WEB_DUPLICATE_PROVIDER` 拒绝重复 id,并在执行时以结构化的选择错误解析提供方。本地抓取后端仅接受 HTTP(S)、拒绝凭证、经已安装的代理策略发送每个请求、对每一次同源重定向跳转重复 URL 校验、限制重定向次数、字节数、字符数和时间,并解码正文;展示由工具负责。
<!-- BEGIN GENERATED cordis-surface (gen-cordis-catalog.ts) — do not edit between markers -->
@@ -0,0 +1,16 @@
# Bilingual-pair consistency record for guide.md (docs/i18n/README.md): per heading
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/upgrade-guide/v0.2.1-rc.2/web-fetch-private-destinations/guide.md
/:
en: 8d967f90aee1902e
zh: 30735df66cafe05d
/web-fetch-reaches-private-destinations:
en: 8c157d5137d0a507
zh: 27cef187efdb1323
/web-fetch-reaches-private-destinations/change:
en: bcbce3e213b81afc
zh: dda264a679117c36
/web-fetch-reaches-private-destinations/migration:
en: e076bdd8bd132ee8
zh: 26d5da33ccc8e37c
@@ -0,0 +1,21 @@
---
kind: upgrade-guide
description: "The HTTP fetch provider no longer rejects non-public destinations, and HttpFetchProvider no longer accepts an address resolver."
---
# `web_fetch` reaches private destinations
English | [中文](guide.zh.md)
## Change
`@deepseek-ai/dsh-web-fetch-http` used to resolve each hostname, reject any loopback, private, link-local, or reserved answer with `WEB_BLOCKED_URL`, and pin the connection to the validated addresses. It now sends every request through the installed proxy policy and the system resolver, so `web_fetch` reaches loopback, private-network, and cloud metadata hosts. Fetches behind TUN-mode proxies with fake-IP DNS work without configuration.
The package no longer exports `HttpFetchResolver`, and the `HttpFetchProvider` constructor takes only `HttpFetchLimits`.
Deployments that relied on the provider to keep the model away from private services are affected, as is code that passed a resolver to `HttpFetchProvider`.
## Migration
1. If the model must not reach private destinations, confine the process's network access, add a `tools/pre-execute` policy that rejects those `web_fetch` URLs, or remove `@deepseek-ai/dsh-tool-web`'s fetch tool from the composition.
2. Replace `new HttpFetchProvider(limits, resolveAddresses)` with `new HttpFetchProvider(limits)`. To direct a test request to a local server, put the server's address in the URL instead of overriding resolution.
3. Confirm the migration with `pnpm run typecheck`; no reference to `HttpFetchResolver` remains.
@@ -0,0 +1,21 @@
---
kind: upgrade-guide
description: "HTTP 抓取提供方不再拒绝非公开目标,HttpFetchProvider 也不再接受地址解析器。"
---
# `web_fetch` 可以访问私有目标
[English](guide.md) | 中文
## 变更
`@deepseek-ai/dsh-web-fetch-http` 以前会解析每个主机名,以 `WEB_BLOCKED_URL` 拒绝任何 loopback、私有、link-local 或保留地址,并把连接固定到已校验地址。现在它通过已安装的代理策略和系统解析器发送每个请求,因此 `web_fetch` 可以访问 loopback、内网与云元数据主机。使用 fake-IP DNS 的 TUN 模式代理之后,抓取无需配置即可工作。
本包不再导出 `HttpFetchResolver`,`HttpFetchProvider` 构造函数只接受 `HttpFetchLimits`。
依赖该提供方阻止模型访问私有服务的部署会受影响;向 `HttpFetchProvider` 传入解析器的代码也会受影响。
## 迁移
1. 如果模型不得访问私有目标,请限制进程的网络访问、添加拒绝这些 `web_fetch` URL 的 `tools/pre-execute` 策略,或从组合中移除 `@deepseek-ai/dsh-tool-web` 的抓取工具。
2. 把 `new HttpFetchProvider(limits, resolveAddresses)` 替换为 `new HttpFetchProvider(limits)`。如需把测试请求指向本地服务器,请把服务器地址写进 URL,而不是覆盖解析。
3. 用 `pnpm run typecheck` 确认迁移完成,且不再引用 `HttpFetchResolver`。
+2 -2
View File
@@ -21,8 +21,8 @@
en: 0f971f7ac7c09139
zh: ceb866da681c709c
/deepseek-ai-dsh-base/use-this-package/what-you-get:
en: 5bfb7755e987862c
zh: 1f0e9ea7a826a509
en: 2ef2653d55107eec
zh: 4936c0fff94b1984
/deepseek-ai-dsh-base/use-this-package/shell-tools-per-platform:
en: 2dce12f82d6ee955
zh: d4599eb899a3cd88
+1 -1
View File
@@ -47,7 +47,7 @@ Run `dsh --profile my-profile "your task"` and you get a working agent with mode
### What you get
Out of the box, every profile built on this core provides: a DeepSeek model connection (the provider and model are configurable, and you can enable extra providers from your settings), the full tool set — file editing, shell commands, web search, public HTTP(S) fetch, subagents, task and goal tracking — durable sessions that survive restarts, and the default permission policy that confines file writes to your workspace and asks before risky actions. Web fetch runs without per-call approval; its provider rejects non-public destinations. Feedback stays in the Session log. [OTel session upload](../../session/session-telemetry-otel/README.md) defaults to `FEEDBACK_ONLY` for all users, including `deepseek-official`: new text feedback, message ratings, edits, and withdrawals release the complete canonical prefix through that event, including context. Later records wait for the next explicit feedback; sending an authorized batch needs no further interaction or model call. `DISABLED` prevents OTel capture. The default-on [DeepSeek session-log contributor](../../session/session-log-deepseek/README.md) remains a separate request path.
Out of the box, every profile built on this core provides: a DeepSeek model connection (the provider and model are configurable, and you can enable extra providers from your settings), the full tool set — file editing, shell commands, web search, HTTP(S) fetch, subagents, task and goal tracking — durable sessions that survive restarts, and the default permission policy that confines file writes to your workspace and asks before risky actions. Web fetch runs without per-call approval and can reach loopback and private-network hosts. Feedback stays in the Session log. [OTel session upload](../../session/session-telemetry-otel/README.md) defaults to `FEEDBACK_ONLY` for all users, including `deepseek-official`: new text feedback, message ratings, edits, and withdrawals release the complete canonical prefix through that event, including context. Later records wait for the next explicit feedback; sending an authorized batch needs no further interaction or model call. `DISABLED` prevents OTel capture. The default-on [DeepSeek session-log contributor](../../session/session-log-deepseek/README.md) remains a separate request path.
Both `subagent` and `subagent_fork` create managed children and return a child id immediately. Use `send_message` to steer active children or continue their conversations after completion. [Delegation-tool behavior](../../subagent/tool-subagent/README.md) describes result delivery and backend capabilities.
+1 -1
View File
@@ -47,7 +47,7 @@ kind: "package-bundle"
### 你得到什么
开箱即用,基于本核心构建的每个 profile 都提供:DeepSeek 模型连接(提供方与模型可配置,你还可以在设置中启用额外提供方)、完整工具集——文件编辑、shell 命令、web 搜索、公开 HTTP(S) 抓取、subagent、任务与目标跟踪——可跨重启存活的持久会话,以及默认权限策略:把文件写入限制在工作区内,危险操作前征询许可。Web 抓取无需逐次审批,其提供方会拒绝非公开目的地址。反馈保存在会话日志中。[OTel 会话上传](../../session/session-telemetry-otel/README.zh.md)对所有用户默认使用 `FEEDBACK_ONLY`,包括 `deepseek-official`:新的文本反馈、消息评分、编辑与撤回会释放截至该事件的完整规范会话日志前缀,包含上下文。后续记录等待下一次显式反馈;发送已授权批次无需进一步交互或模型调用。`DISABLED` 阻止 OTel 捕获。默认开启的 [DeepSeek 会话日志贡献器](../../session/session-log-deepseek/README.zh.md)仍是独立的请求路径。
开箱即用,基于本核心构建的每个 profile 都提供:DeepSeek 模型连接(提供方与模型可配置,你还可以在设置中启用额外提供方)、完整工具集——文件编辑、shell 命令、web 搜索、HTTP(S) 抓取、subagent、任务与目标跟踪——可跨重启存活的持久会话,以及默认权限策略:把文件写入限制在工作区内,危险操作前征询许可。Web 抓取无需逐次审批,并且可以访问 loopback 与内网主机。反馈保存在会话日志中。[OTel 会话上传](../../session/session-telemetry-otel/README.zh.md)对所有用户默认使用 `FEEDBACK_ONLY`,包括 `deepseek-official`:新的文本反馈、消息评分、编辑与撤回会释放截至该事件的完整规范会话日志前缀,包含上下文。后续记录等待下一次显式反馈;发送已授权批次无需进一步交互或模型调用。`DISABLED` 阻止 OTel 捕获。默认开启的 [DeepSeek 会话日志贡献器](../../session/session-log-deepseek/README.zh.md)仍是独立的请求路径。
`subagent` 与 `subagent_fork` 都创建受管理的子任务,并立即返回 child id。使用 `send_message` 可引导活跃子任务,或在完成后继续其对话。[委派工具行为](../../subagent/tool-subagent/README.zh.md)说明了结果交付和后端能力。
@@ -500,7 +500,7 @@ Every package below exports a Cordis plugin that a bundle patch can name in a Lo
| `@deepseek-ai/dsh-tool-web` | yes | Model-facing web tools (web_search, web_fetch) over the DeepSeek Harness web capability seam (ctx.web) |
| `@deepseek-ai/dsh-web` | yes | Abstract web access capability seam (ctx.web) for the DeepSeek Harness — search/fetch provider registry, registration-order-independent selection, request/result vocabulary, and the WebError taxonomy |
| `@deepseek-ai/dsh-web-deepseek-api` | yes | DeepSeek Search and Fetch API providers for the DeepSeek Harness web capability seam (ctx.web) |
| `@deepseek-ai/dsh-web-fetch-http` | yes | Anonymous public HTTP(S) fetch provider for the DeepSeek Harness web capability seam (ctx.web) |
| `@deepseek-ai/dsh-web-fetch-http` | yes | Anonymous HTTP(S) fetch provider for the DeepSeek Harness web capability seam (ctx.web) |
| `@deepseek-ai/dsh-web-search-deepseek` | yes | DeepSeek-backed search provider (native web_search via the Anthropic-compatible API) for the DeepSeek Harness web capability seam (ctx.web) |
| `@deepseek-ai/dsh-web-search-exa` | yes | Exa-backed search provider for the DeepSeek Harness web capability seam (ctx.web) |
| `@deepseek-ai/dsh-web-search-perplexity` | yes | Perplexity-backed search provider for the DeepSeek Harness web capability seam (ctx.web) |
+2 -2
View File
@@ -18,8 +18,8 @@
en: bc67087b571aa603
zh: 264ac9bb95c8efe8
/deepseek-ai-dsh-http-proxy/use-this-package/writing-a-new-outbound-call:
en: 782b24f91ee8ce71
zh: af4b75a0d5a98fdc
en: 6271c8077268fd74
zh: 53e949034a3a1f0b
/deepseek-ai-dsh-http-proxy/use-this-package/what-the-policy-reads:
en: a1f74dd3a34ed97e
zh: d9440a0666caccb6
+1 -1
View File
@@ -41,7 +41,7 @@ Plain `fetch()` is proxied, and so is any SDK that reaches `globalThis.fetch`
`proxyRouteFor` answers with the transport that answer assumed, not just the answer: its proxied arm carries the dispatcher already routing by this policy. A caller that read the policy and then built its own transport could have an unmount land between the two and send the request somewhere its branch never cleared.
Constructing `new Agent(...)` and passing it as `dispatcher` overrides the global one and silently bypasses the proxy. `verify-no-bare-dispatcher` rejects that outside this package. One call site legitimately owns its transport — `web-fetch-http` pins a request to addresses it validated, which is per-request state a process-wide dispatcher cannot hold — and says so with a `proxy-exempt:` comment on the line.
Constructing `new Agent(...)` and passing it as `dispatcher` overrides the global one and silently bypasses the proxy. `verify-no-bare-dispatcher` rejects that outside this package. A call site that legitimately owns its transport, because it carries per-request state a process-wide dispatcher cannot hold, says so with a `proxy-exempt:` comment on the line.
That gate cannot see inside an SDK, so each outbound call site carries an `egress.spec.ts` that drives its actual transport through a fake proxy and checks the observed route. Every new outbound call site must include that transport test. Telemetry asserts its direct-route exception. These tests detect dependency changes that alter routing without changing the call site.
+1 -1
View File
@@ -41,7 +41,7 @@ kind: "package-reference"
`proxyRouteFor` 给出的不只是答案,还有该答案所假定的传输:走代理的那一支携带着此刻正按该策略路由的 dispatcher。若调用方先读策略、再自建传输,卸载就可能落在两次读取之间,把请求发往其分支从未放行的去处。
构造 `new Agent(...)` 再作为 `dispatcher` 传入会覆盖全局 dispatcher,从而静默绕开代理。`verify-no-bare-dispatcher` 会在本包之外拒绝该写法。有一处调用点确实自有传输——`web-fetch-http` 会把请求钉在它已校验过的地址上,而这是进程级 dispatcher 无法承载的单次请求状态——它在该行用 `proxy-exempt:` 注释说明。
构造 `new Agent(...)` 再作为 `dispatcher` 传入会覆盖全局 dispatcher,从而静默绕开代理。`verify-no-bare-dispatcher` 会在本包之外拒绝该写法。确实自有传输的调用点(因为它携带进程级 dispatcher 无法承载的单次请求状态)要在该行用 `proxy-exempt:` 注释说明。
该门禁看不进 SDK 内部,因此每个出网点都配有 `egress.spec.ts`,通过假代理驱动实际传输并检查观察到的路由。每个新的出网点必须包含该传输测试。遥测断言其直连例外。这些测试可发现调用点不变但依赖变更改变路由的情况。
+4 -4
View File
@@ -15,11 +15,11 @@
en: 1eb35b015a6c423f
zh: a8a3dceee2c9e943
/web-web-access-capability-family/packages:
en: a1989bf59f589d94
zh: 4bd5c0d8e0fbe4e4
en: 89e39c77421cf5b6
zh: 0aedc38d2fb8be39
/web-web-access-capability-family/related-documentation:
en: be5d968ca450364b
zh: 813606f36b569d56
en: 38eff8a93331b70b
zh: d60ad34847eb2e29
/web-web-access-capability-family/dev-note:
en: 5de1d6304e00c58f
zh: 8d8225361cfa29e5
+2 -2
View File
@@ -31,7 +31,7 @@ Six packages play the web roles; the subsystem reference owns the exhaustive voc
| [`web-search-perplexity/`](web-search-perplexity/README.md) | Searches the web through Perplexity | registers on `ctx.web` |
| [`web-search-deepseek/`](web-search-deepseek/README.md) | Searches through DeepSeek Messages | registers on `ctx.web` |
| [`web-deepseek-api/`](web-deepseek-api/README.md) | Searches and fetches pages through the DeepSeek Search and Fetch APIs | registers on `ctx.web` |
| [`web-fetch-http/`](web-fetch-http/README.md) | Fetches public HTTP(S) pages anonymously | registers on `ctx.web` |
| [`web-fetch-http/`](web-fetch-http/README.md) | Fetches HTTP(S) pages anonymously | registers on `ctx.web` |
| [`tool-web/`](tool-web/README.md) | Exposes `web_search` and `web_fetch` to the model | registers on `ctx.tools` |
-----
@@ -41,7 +41,7 @@ Six packages play the web roles; the subsystem reference owns the exhaustive voc
Start with the subsystem reference for the shared vocabulary, then the design decision behind the single provider-selection service.
- [Web subsystem](../../docs/subsystems/web.md) — the search/fetch requests and results, provider availability, `WebError`, and public-address enforcement.
- [Web subsystem](../../docs/subsystems/web.md) — the search/fetch requests and results, provider availability, `WebError`, and the fetch network policy.
- [Web capability seam decision](../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md) — why search and fetch share one provider-selection service.
<a id="dev-note"></a>
+2 -2
View File
@@ -31,7 +31,7 @@ kind: "package-group"
| [`web-search-perplexity/`](web-search-perplexity/README.zh.md) | 通过 Perplexity 搜索 web | 注册到 `ctx.web` |
| [`web-search-deepseek/`](web-search-deepseek/README.zh.md) | 通过 DeepSeek Messages 搜索网页 | 注册到 `ctx.web` |
| [`web-deepseek-api/`](web-deepseek-api/README.zh.md) | 通过 DeepSeek Search 与 Fetch API 搜索和抓取网页 | 注册到 `ctx.web` |
| [`web-fetch-http/`](web-fetch-http/README.zh.md) | 匿名抓取公共 HTTP(S) 页面 | 注册到 `ctx.web` |
| [`web-fetch-http/`](web-fetch-http/README.zh.md) | 匿名抓取 HTTP(S) 页面 | 注册到 `ctx.web` |
| [`tool-web/`](tool-web/README.zh.md) | 向模型公开 `web_search` 与 `web_fetch` | 注册到 `ctx.tools` |
-----
@@ -41,7 +41,7 @@ kind: "package-group"
先从子系统参考文档了解共享词汇,再看单一提供方选择服务背后的设计决策。
- [web 子系统](../../docs/subsystems/web.zh.md)——搜索与抓取的请求和结果、提供方可用性、`WebError` 与公开地址强制规则。
- [web 子系统](../../docs/subsystems/web.zh.md)——搜索与抓取的请求和结果、提供方可用性、`WebError` 与抓取网络策略。
- [web 能力 seam 决策](../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.zh.md)——搜索与抓取为何共用一项提供方选择服务。
<a id="dev-note"></a>
+2 -2
View File
@@ -141,8 +141,8 @@
en: f5d5f97a153f7c70
zh: f681f89596f500f5
/deepseek-ai-dsh-tool-web/known-limitations-and-deferred-work:
en: aa23b6943fc9870c
zh: 180d50b1161e7eef
en: 7cd86ef1db2f6743
zh: b4cf681add3837d6
/deepseek-ai-dsh-tool-web/known-limitations-and-deferred-work/dev-note:
en: 7f775cc79dbce94b
zh: 12b8f68f9e7c48ca
+1 -1
View File
@@ -251,7 +251,7 @@ These limits define when the tools are incomplete or need deployment cooperation
- **There is no batch-wide native-search counter** — `searchMaxQueries` bounds `ctx.web.search` calls, but a provider may perform several native searches inside each call; for example a model-backed provider configured with `maxUses` can permit up to `searchMaxQueries × maxUses` native searches, and `searchMaxResults` limits only the combined sources returned to the caller. Deployments control cost through these independent consumer and provider settings because the service does not know provider-internal search units.
- **HTML→markdown conversion omits inputs it cannot safely represent** — [turndown](https://github.com/mixmark-io/turndown) converts at most `fetchMaxOutputChars` source characters through a real DOM. A 512-level nesting guard and conversion exceptions produce a fixed omission marker instead of raw HTML; table `colspan` remains unsupported because GFM has no spanning-cell representation ([archived dependency decision](../../../.agents/notes/archived/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md)).
- **The model-facing API is minimal by design, with promotions deferred** — `max_results` stays a config bound (not a model argument), and `web_fetch` takes only `url` (no `format`/`prompt`/LLM-summarization mode); both are named later steps in [the seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md).
- **Public fetches do not request approval** — the shipped `cordis`, `code`, and `standard` presets expose `web_fetch` in every sandbox and approval mode. The HTTP provider blocks non-public destinations, but a model can send data to a public URL. Deployments that need per-call confirmation must add a `tools/pre-execute` policy or disable fetch.
- **Fetches do not request approval** — the shipped `cordis`, `code`, and `standard` presets expose `web_fetch` in every sandbox and approval mode. The HTTP provider does not restrict destination addresses, so a model can reach loopback and private-network hosts and send data to any reachable URL. Deployments that need per-call confirmation must add a `tools/pre-execute` policy or disable fetch.
<a id="dev-note"></a>
### Dev Note
+1 -1
View File
@@ -251,7 +251,7 @@ schema 校验会在执行前拒绝缺失或非数组的 `queries` 字段以及
- **没有覆盖整个批次的原生搜索计数器**:`searchMaxQueries` 限制 `ctx.web.search` 调用数,但提供方可以在每次调用内执行多次原生搜索;例如,配置了 `maxUses` 的以模型为后端的提供方最多可以执行 `searchMaxQueries × maxUses` 次原生搜索,`searchMaxResults` 只限制返回给调用方的组合来源。部署通过这些独立的消费方与提供方设置控制成本,因为服务不知道提供方内部的搜索计量单位。
- **HTML→markdown 转换会省略无法安全表示的输入**——[turndown](https://github.com/mixmark-io/turndown) 会通过真实 DOM 转换至多 `fetchMaxOutputChars` 个源字符。512 层嵌套守卫与转换异常会产生固定省略标记,而不是返回原始 HTML;表格 `colspan` 仍不受支持,因为 GFM 无法表示跨列单元格([已归档的依赖决策](../../../.agents/notes/archived/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md))。
- **面向模型的接口有意保持精简,后续扩展暂缓**:`max_results` 保持为配置上限(不是模型参数),`web_fetch` 只接受 `url`(没有 `format`/`prompt`/LLM(大语言模型)摘要模式);两项都列为 [seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.zh.md) 中的后续步骤。
- **公开抓取不请求审批**——随产品交付的 `cordis`、`code` 与 `standard` preset 在所有 sandbox 和审批模式下公开 `web_fetch`。HTTP 提供方会阻止非公开目标,但模型仍可向公开 URL 发送数据。需要逐次确认的部署必须添加 `tools/pre-execute` 策略或禁用抓取。
- **抓取不请求审批**——随产品交付的 `cordis`、`code` 与 `standard` preset 在所有 sandbox 和审批模式下公开 `web_fetch`。HTTP 提供方不限制目标地址,因此模型可以访问 loopback 与内网主机,并向任何可访问的 URL 发送数据。需要逐次确认的部署必须添加 `tools/pre-execute` 策略或禁用抓取。
<a id="dev-note"></a>
### 开发备注
@@ -2,8 +2,7 @@
* Integration: the real fetch backend (`dsh-web-fetch-http`) + a real search provider
* (`dsh-web-search-exa`) + the real seam (`dsh-web`) + the model tool (`dsh-tool-web`) + the
* tool-call timeout policy (`dsh-tool-call-timeout-policy`), exercised through `ctx.tools.execute()` —
* nothing bypasses the tool registry. Fetch verifies world effects against loopback HTTP with
* public-address resolution replaced by the fixture address; search uses the real Exa provider
* nothing bypasses the tool registry. Fetch verifies world effects against loopback HTTP; search uses the real Exa provider
* with only its network boundary stubbed.
*/
@@ -19,7 +18,6 @@ import * as WebFetchLocal from '@deepseek-ai/dsh-web-fetch-http'
import * as WebSearchExa from '@deepseek-ai/dsh-web-search-exa'
import * as ToolWeb from '@deepseek-ai/dsh-tool-web'
import * as TimeoutPolicy from '@deepseek-ai/dsh-tool-call-timeout-policy'
import { publicHttpNetwork } from '../../web-fetch-http/src/network.ts'
const testToolSignal = new AbortController().signal
@@ -32,7 +30,6 @@ let ctx: Context
let fiber: Awaited<ReturnType<Context['plugin']>>
beforeEach(async () => {
vi.spyOn(publicHttpNetwork, 'resolve').mockResolvedValue([{ address: '127.0.0.1', family: 4 }])
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/html' }); res.end('<h1>Hello</h1><p>World</p>') }
server = createServer((req, res) => { handler(req, res) })
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
@@ -27,7 +27,6 @@ import * as WebFetchLocal from '@deepseek-ai/dsh-web-fetch-http'
import LocalSpillStore from '@deepseek-ai/dsh-spill-local'
import * as SpillPolicy from '@deepseek-ai/dsh-spill-policy'
import * as ToolWeb from '@deepseek-ai/dsh-tool-web'
import { publicHttpNetwork } from '../../web-fetch-http/src/network.ts'
type Handler = (req: IncomingMessage, res: ServerResponse) => void
@@ -41,7 +40,6 @@ const BODY = 'X'.repeat(4000) // formatted result is well over the policy cap
const MAX_INLINE_TOKENS = 250 // leaves room for a head/tail preview beside the notice
beforeEach(async () => {
vi.spyOn(publicHttpNetwork, 'resolve').mockResolvedValue([{ address: '127.0.0.1', family: 4 }])
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end(BODY) }
server = createServer((req, res) => { handler(req, res) })
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
+16 -16
View File
@@ -3,14 +3,14 @@
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/web/web-fetch-http/README.md
/:
en: a55f23f2363fdbd6
zh: b1fd7328e041ec52
en: dba7f5089d194048
zh: 956905ee19f6f362
/deepseek-ai-dsh-web-fetch-http:
en: 2e839bfc4c365701
zh: 62dd5239ae93471c
/deepseek-ai-dsh-web-fetch-http/summary:
en: 88157c32f36635d5
zh: a149d576df0349bc
en: afb0ecfdd563bced
zh: 734f74bf506e34e2
/deepseek-ai-dsh-web-fetch-http/table-of-contents:
en: d152484eb41ac6b4
zh: 09388d293f9be9cb
@@ -18,8 +18,8 @@
en: 630b12516d53e905
zh: eed124cfe9c8dc1f
/deepseek-ai-dsh-web-fetch-http/use-this-package/when-to-choose-it:
en: eb52e77279c21d6f
zh: 0560e89caa377bfc
en: acb7517f571c38b8
zh: 8ee9da18953ab65e
/deepseek-ai-dsh-web-fetch-http/use-this-package/minimal-configuration:
en: e4534189d5230861
zh: 5905731d7b5d9c60
@@ -27,8 +27,8 @@
en: 2d18cdcc9b53dd0f
zh: 3bc6ca7c5933ee5d
/deepseek-ai-dsh-web-fetch-http/use-this-package/transport-behavior:
en: c432147daf2877cf
zh: c14d507dd66e124a
en: 584054eb465ae615
zh: c421860ba79bfffc
/deepseek-ai-dsh-web-fetch-http/use-this-package/failures-and-recovery:
en: 35407f106dc65deb
zh: 02118072fba68b8f
@@ -36,14 +36,14 @@
en: 9842c69ad3636bb3
zh: 3cdc3f816cbdd0cb
/deepseek-ai-dsh-web-fetch-http/understand-the-implementation/design-philosophy:
en: a8d806f0706f02e2
zh: d922732a5b7e88bd
en: 422128dac5eebdde
zh: 25ba5f49b305a056
/deepseek-ai-dsh-web-fetch-http/understand-the-implementation/source-map:
en: 023ca0c9a9ef4968
zh: 74a5785a493f5111
en: 1f355ec2726ff61b
zh: 076a0c259c94f22a
/deepseek-ai-dsh-web-fetch-http/understand-the-implementation/read-path:
en: 749c0ffeac145807
zh: 1468821a33a9518a
en: f13cd16555fe39a5
zh: 8f48cb659f2c272c
/deepseek-ai-dsh-web-fetch-http/further-exploration:
en: eeaaf0b455673a1c
zh: 0874f9f44ff9a220
@@ -54,8 +54,8 @@
en: 40294c1c31b74824
zh: d75d48f485447a74
/deepseek-ai-dsh-web-fetch-http/known-limitations-and-deferred-work:
en: 4d935fd8f466a234
zh: c398eb7aa737f5ea
en: aa351dc3edaa6355
zh: 5bb556857b389df7
/deepseek-ai-dsh-web-fetch-http/known-limitations-and-deferred-work/dev-note:
en: e67f633dc6fe7773
zh: 3bb0c20c93f16f98
+8 -8
View File
@@ -1,5 +1,5 @@
---
description: "The anonymous public HTTP(S) fetch backend for ctx.web: how deployments mount bounded, safe URL retrieval with same-origin redirects and text-only decoding."
description: "The anonymous HTTP(S) fetch backend for ctx.web: how deployments mount bounded URL retrieval with same-origin redirects and text-only decoding."
kind: "package-reference"
---
@@ -9,7 +9,7 @@ English | [中文](README.zh.md)
## Summary
With `dsh-web-fetch-http`, the harness can fetch public HTTP(S) pages through the web service (`ctx.web`) and get their status code plus bounded, decoded content without sending credentials. Choose it when a composition needs safe retrieval with URL validation, public-address resolution, connection pinning, same-origin redirects, byte and character caps, and an explicit product `User-Agent`. It returns non-2xx responses as results rather than errors, and rejects non-public destinations, binary data, and unsupported content types. The model-facing `web_fetch` tool lives in `dsh-tool-web`, which renders this provider's bodies.
With `dsh-web-fetch-http`, the harness can fetch HTTP(S) pages through the web service (`ctx.web`) and get their status code plus bounded, decoded content without sending credentials. Choose it when a composition needs retrieval with URL validation, same-origin redirects, byte and character caps, and an explicit product `User-Agent`. It returns non-2xx responses as results rather than errors, and rejects binary data and unsupported content types. It does not restrict destination addresses, so it reaches loopback and private-network hosts. The model-facing `web_fetch` tool lives in `dsh-tool-web`, which renders this provider's bodies.
## Table of Contents
@@ -29,7 +29,7 @@ Mount the provider in a composition that already loads the web service; it regis
### When to choose it
Choose this backend when a deployment must fetch public pages with bounded output and safe transport: no credentials are sent, every resolved address must be public, each connection is pinned to the validated answer set, redirects cannot escape the origin, and every response is capped.
Choose this backend when a deployment must fetch pages with bounded output: no credentials are sent, redirects cannot escape the origin, and every response is capped.
### Minimal configuration
@@ -61,7 +61,7 @@ const page = await ctx.web.fetch({ url: 'https://example.com' })
### Transport behavior
The provider keeps requests anonymous and bounded: it accepts only `http:` and `https:` URLs without embedded credentials and rejects URLs over 2,048 characters. It resolves each hostname once, rejects the complete result if any IPv4 or IPv6 address is not public unicast, and pins the connection to that validated set. IPv6 checks discover the active DNS64 prefix and reject translations to non-public IPv4. Each same-origin redirect repeats resolution and pinning; cross-origin redirects fail and require a fresh call. The provider also enforces byte, character, hop, and time caps, rejects unsupported content types, and sends an explicit product `User-Agent`.
The provider keeps requests anonymous and bounded: it accepts only `http:` and `https:` URLs without embedded credentials and rejects URLs over 2,048 characters. Requests go through the process-wide dispatcher, so the installed proxy policy applies and the system resolver chooses the address; any destination address is allowed. Same-origin redirects are followed; cross-origin redirects fail and require a fresh call. The provider also enforces byte, character, hop, and time caps, rejects unsupported content types, and sends an explicit product `User-Agent`.
### Failures and recovery
@@ -81,7 +81,7 @@ This section explains the design decisions behind the provider; the observable b
The package is built on one separation and one layered timeout:
- **Safe retrieval vs. presentation.** This provider owns URL validation, public-address enforcement, connection pinning, HTTP transport, redirect policy, caps, charset decoding, and binary rejection; `dsh-tool-web` owns HTML→markdown and truncation formatting. A non-2xx response is data, not failure.
- **Retrieval vs. presentation.** This provider owns URL validation, HTTP transport, redirect policy, caps, charset decoding, and binary rejection; `dsh-tool-web` owns HTML→markdown and truncation formatting. A non-2xx response is data, not failure.
- **Two timeout layers.** The provider's `timeoutMs` is a resource backstop for direct `ctx.web.fetch()` callers; the model-facing tool-call budget belongs to `dsh-tool-call-timeout-policy`, which arms `exec.signal`. When the outer deadline fires first the provider reports `WEB_ABORTED` and the policy replaces it with `TOOL_TIMEOUT`; `WEB_FETCH_TIMEOUT` therefore identifies a direct service caller whose provider budget elapsed.
### Source map
@@ -89,13 +89,12 @@ The package is built on one separation and one layered timeout:
| File | Role |
|---|---|
| [`src/index.ts`](src/index.ts) | Plugin entry: config schema, limit validation, provider registration |
| [`src/provider.ts`](src/provider.ts) | The `HttpFetchProvider`: pinned transport, redirect following, capped reads, charset decoding |
| [`src/network.ts`](src/network.ts) | Public-address resolution, DNS64 discovery, and connection pinning |
| [`src/provider.ts`](src/provider.ts) | The `HttpFetchProvider`: transport, redirect following, capped reads, charset decoding |
| [`src/policy.ts`](src/policy.ts) | URL validation, same-origin checks, content-type classification, charset parsing |
### Read path
A fetch validates the URL, resolves the hostname once, rejects the complete answer set when any address is not public, and pins the connection to the accepted addresses. It repeats that check for each same-origin redirect; a cross-origin redirect or non-public target fails before response bytes are accepted. The final response is classified by `Content-Type`, decoded from its declared charset, and read under the byte cap; the decoded text is then truncated to the character cap.
A fetch validates the URL and sends the request. It repeats URL validation for each same-origin redirect; a cross-origin redirect fails before response bytes are accepted. The final response is classified by `Content-Type`, decoded from its declared charset, and read under the byte cap; the decoded text is then truncated to the character cap.
</details>
@@ -131,6 +130,7 @@ No direct invalidation; the named consumer owns any request-prefix changes.
These limits define when the provider is unsafe or a poor fit. They are current package constraints.
- **Private destinations are reachable** — loopback, private-network, and cloud metadata addresses are fetched like any other host. A deployment that must keep the model away from them needs a `tools/pre-execute` policy, network confinement, or no fetch tool.
- **Only textual content decodes** — html/xhtml and `text/*` plus JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work.
- **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `<meta charset>` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back.
+8 -8
View File
@@ -1,5 +1,5 @@
---
description: "ctx.web 的匿名公共 HTTP(S) 抓取后端:部署方如何挂载有界、安全的 URL 抓取,含同源重定向与仅文本解码。"
description: "ctx.web 的匿名 HTTP(S) 抓取后端:部署方如何挂载有界的 URL 抓取,含同源重定向与仅文本解码。"
kind: "package-reference"
---
@@ -9,7 +9,7 @@ kind: "package-reference"
## 概述
有了 `dsh-web-fetch-http`,harness 可以通过 web 服务(`ctx.web`)抓取公共 HTTP(S) 页面,并在不发送凭据的情况下获得状态码与有界、解码后的内容。当组合需要 URL 校验、公开地址解析、连接固定、仅同源重定向、字节和字符上限及显式产品 `User-Agent` 时选择它。它把非 2xx 响应作为结果而非错误返回,并拒绝非公开目标、二进制数据与不受支持的内容类型。面向模型的 `web_fetch` 工具位于 `dsh-tool-web`,由它渲染本提供方的正文。
有了 `dsh-web-fetch-http`,harness 可以通过 web 服务(`ctx.web`)抓取 HTTP(S) 页面,并在不发送凭据的情况下获得状态码与有界、解码后的内容。当组合需要 URL 校验、仅同源重定向、字节和字符上限及显式产品 `User-Agent` 时选择它。它把非 2xx 响应作为结果而非错误返回,并拒绝二进制数据与不受支持的内容类型。它不限制目标地址,因此可以访问 loopback 与内网主机。面向模型的 `web_fetch` 工具位于 `dsh-tool-web`,由它渲染本提供方的正文。
## 目录
@@ -29,7 +29,7 @@ kind: "package-reference"
### 何时选择
当部署必须以有界输出和安全传输抓取公共页面时选择此后端:不发送凭据,每个已解析地址必须是公共地址,每次连接都固定到已校验的地址集合,重定向无法逃出源站,每个响应都有上限。
当部署必须以有界输出抓取页面时选择此后端:不发送凭据,重定向无法逃出源站,每个响应都有上限。
### 最小配置
@@ -61,7 +61,7 @@ const page = await ctx.web.fetch({ url: 'https://example.com' })
### 传输行为
提供方保持请求匿名且有界:只接受不含内嵌凭据且不超过 2,048 个字符的 `http:` 与 `https:` URL。它只解析一次主机名;只要结果中有任何 IPv4 或 IPv6 地址不是公共单播地址,就拒绝整个结果,并把连接固定到已校验的地址集合。IPv6 检查会发现活动 DNS64 前缀,并拒绝指向非公开 IPv4 的转换地址。每次同源重定向都会重复解析与固定;跨源重定向会失败并要求重新调用。提供方还强制执行字节、字符、跳数和时间上限,拒绝不支持的内容类型,并发送显式产品 `User-Agent`。
提供方保持请求匿名且有界:只接受不含内嵌凭据且不超过 2,048 个字符的 `http:` 与 `https:` URL。请求经过进程级 dispatcher,因此已安装的代理策略生效,目标地址由系统解析器选择;任何目标地址都被允许。同源重定向会被跟随;跨源重定向会失败并要求重新调用。提供方还强制执行字节、字符、跳数和时间上限,拒绝不支持的内容类型,并发送显式产品 `User-Agent`。
### 失败与恢复
@@ -81,7 +81,7 @@ const page = await ctx.web.fetch({ url: 'https://example.com' })
本包基于一项职责分离和一套分层超时机制:
- **安全获取与呈现分离。** 本提供方拥有 URL 校验、公开地址强制规则、连接固定、HTTP 传输、重定向策略、上限、charset 解码与二进制拒绝;`dsh-tool-web` 拥有 HTML→markdown 与截断格式化。非 2xx 响应是数据,不是失败。
- **获取与呈现分离。** 本提供方拥有 URL 校验、HTTP 传输、重定向策略、上限、charset 解码与二进制拒绝;`dsh-tool-web` 拥有 HTML→markdown 与截断格式化。非 2xx 响应是数据,不是失败。
- **两层超时。** 提供方的 `timeoutMs` 是直接 `ctx.web.fetch()` 调用方的资源兜底;面向模型的工具调用预算属于 `dsh-tool-call-timeout-policy`,由它触发 `exec.signal`。外层截止期限先到时,提供方报告 `WEB_ABORTED`,策略再以 `TOOL_TIMEOUT` 替换;因此 `WEB_FETCH_TIMEOUT` 标识的是提供方预算耗尽的直接服务调用方。
### 源码地图
@@ -89,13 +89,12 @@ const page = await ctx.web.fetch({ url: 'https://example.com' })
| 文件 | 职责 |
|---|---|
| [`src/index.ts`](src/index.ts) | 插件入口:配置 schema、上限验证、提供方注册 |
| [`src/provider.ts`](src/provider.ts) | `HttpFetchProvider`:固定连接、重定向跟随、有界读取、charset 解码 |
| [`src/network.ts`](src/network.ts) | 公开地址解析、DNS64 发现与连接固定 |
| [`src/provider.ts`](src/provider.ts) | `HttpFetchProvider`:传输、重定向跟随、有界读取、charset 解码 |
| [`src/policy.ts`](src/policy.ts) | URL 校验、同源检查、内容类型分类、charset 解析 |
### 读取路径
抓取先校验 URL,只解析一次主机名,结果中只要有非公开地址就拒绝,并把连接固定到已接受地址。每次同源重定向都重复该检查;跨源重定向或非公开目标在接收响应字节前失败。最终响应按 `Content-Type` 分类、依声明的 charset 解码,并在字节上限内读取;解码后的文本再截断到字符上限。
抓取先校验 URL,再发送请求。每次同源重定向都重复 URL 校验;跨源重定向在接收响应字节前失败。最终响应按 `Content-Type` 分类、依声明的 charset 解码,并在字节上限内读取;解码后的文本再截断到字符上限。
</details>
@@ -131,6 +130,7 @@ const page = await ctx.web.fetch({ url: 'https://example.com' })
这些限制说明提供方何时不安全或不合适。它们是当前包约束。
- **可以访问私有目标**——loopback、内网与云元数据地址与其他主机一样会被抓取。需要阻止模型访问这些地址的部署必须添加 `tools/pre-execute` 策略、网络隔离,或不挂载抓取工具。
- **只解码文本内容**——包括 html/xhtml 与 `text/*` 加 JSON/XML 家族;缺少 `Content-Type` 或任何二进制类型都会抛出 `WEB_UNSUPPORTED_CONTENT_TYPE`,可提取文本的 PDF 解码属于明确的延期工作。
- **charset 只来自 `Content-Type` 标头**(默认 UTF-8)——HTML `<meta charset>` 声明会被忽略;声明但无法识别的 charset 标签会抛出异常,而非回退。
+2 -4
View File
@@ -1,6 +1,6 @@
{
"name": "@deepseek-ai/dsh-web-fetch-http",
"description": "Anonymous public HTTP(S) fetch provider for the DeepSeek Harness web capability seam (ctx.web)",
"description": "Anonymous HTTP(S) fetch provider for the DeepSeek Harness web capability seam (ctx.web)",
"version": "0.2.1-rc.2",
"publishConfig": {
"access": "public"
@@ -31,12 +31,10 @@
"peerDependencies": {
"@deepseek-ai/cordis": "workspace:~",
"@deepseek-ai/dsh-timeout": "workspace:*",
"@deepseek-ai/dsh-web": "workspace:*",
"@deepseek-ai/dsh-http-proxy": "workspace:*"
"@deepseek-ai/dsh-web": "workspace:*"
},
"dependencies": {
"@deepseek-ai/schemastery": "workspace:~",
"ipaddr.js": "^2.5.0",
"undici": "8.11.2"
},
"devDependencies": {
+2 -2
View File
@@ -1,5 +1,5 @@
/**
* Anonymous public HTTP(S) `WebFetchProvider` plugin. It contributes to the
* Anonymous HTTP(S) `WebFetchProvider` plugin. It contributes to the
* `ctx.web` registry without owning the service.
*
* @module @deepseek-ai/dsh-web-fetch-http
@@ -17,7 +17,7 @@ export {
LOCAL_FETCH_PROVIDER_ID,
HttpFetchProvider,
} from './provider.ts'
export type { HttpFetchLimits, HttpFetchResolver } from './provider.ts'
export type { HttpFetchLimits } from './provider.ts'
/** Default `User-Agent`: an explicit product agent, never a browser disguise. */
export const DEFAULT_USER_AGENT = 'deepseek-harness/0.0.1 (+https://github.com/deepseek-ai)'
-301
View File
@@ -1,301 +0,0 @@
/**
* Public-network resolution and address-pinned HTTP transport for `web-fetch-http`.
* One DNS answer set is validated before Undici receives it through a custom lookup,
* so the connection cannot resolve the hostname again to a private address.
*
* @module @deepseek-ai/dsh-web-fetch-http/network
*/
import { lookup as systemLookup } from 'node:dns/promises'
import type { LookupAddress, LookupOptions } from 'node:dns'
import { isIP } from 'node:net'
import type { Dispatcher, Response } from 'undici'
import ipaddr from 'ipaddr.js'
import { WebError } from '@deepseek-ai/dsh-web'
/** One address resolved and retained for the subsequent pinned connection. */
export interface PublicAddress {
/** Canonical textual IPv4 or IPv6 address. */
readonly address: string
/** Address family accepted by Node's connection lookup callback. */
readonly family: 4 | 6
}
/** The result of one address-pinned request; closing releases its private pool. */
export interface PinnedResponse {
/** HTTP response whose body remains readable until `close()` is called. */
readonly response: Response
/** Release the request's dispatcher after the response body is consumed or cancelled. */
close(): Promise<void>
}
/** Resolver signature used to test public-address policy without process DNS changes. */
export type AddressResolver = (hostname: string, options: { all: true; order: 'verbatim' }) => Promise<LookupAddress[]>
/** RFC 6052 prefix lengths that may carry an IPv4 destination through NAT64. */
const RFC6052_PREFIX_LENGTHS = [32, 40, 48, 56, 64, 96] as const
const IPV4ONLY_DISCOVERY_HOST = 'ipv4only.arpa'
const IPV4ONLY_SENTINELS = new Set(['192.0.0.170', '192.0.0.171'])
interface Nat64Prefix {
readonly bytes: readonly number[]
readonly length: typeof RFC6052_PREFIX_LENGTHS[number]
}
/**
* Return whether an address is globally reachable unicast. IPv4-mapped IPv6 is
* classified by its embedded IPv4 address; transition and translation prefixes
* remain blocked because their eventual IPv4 destination cannot be pinned here.
*
* @param input - textual IPv4 or IPv6 address.
* @returns true only for a public unicast destination.
*/
export function isPublicIpAddress(input: string): boolean {
let parsed: ipaddr.IPv4 | ipaddr.IPv6
try {
parsed = ipaddr.parse(stripIpv6Brackets(input))
} catch {
return false
}
if (parsed instanceof ipaddr.IPv4) return parsed.range() === 'unicast'
if (parsed.isIPv4MappedAddress()) return parsed.toIPv4Address().range() === 'unicast'
return parsed.range() === 'unicast'
}
/**
* Resolve a hostname once and reject the complete answer set if any destination
* is not public. The returned addresses are the only ones the transport may use.
*
* @param hostname - URL hostname, including brackets when it is an IPv6 literal.
* @param signal - aborts the wait for system resolution; an in-flight OS lookup may finish unused.
* @param resolver - lookup implementation, overridden only by focused tests.
* @returns the validated, non-empty address set.
*/
export async function resolvePublicAddresses(
hostname: string,
signal: AbortSignal,
resolver: AddressResolver = systemLookup,
): Promise<PublicAddress[]> {
const unbracketed = stripIpv6Brackets(hostname)
const literalFamily = isIP(unbracketed)
const resolved = literalFamily === 0
? await raceWithSignal(resolver(unbracketed, { all: true, order: 'verbatim' }), signal)
: [{ address: unbracketed, family: literalFamily }]
if (resolved.length === 0) {
throw new WebError(`hostname "${hostname}" resolved to no addresses`, 'WEB_PROVIDER_ERROR')
}
const hasIpv6 = resolved.some(entry => entry.family === 6 && isIP(entry.address) === 6)
const nat64Prefixes = hasIpv6
? await discoverNat64Prefixes(signal, resolver)
: []
const addresses: PublicAddress[] = []
for (const entry of resolved) {
if ((entry.family !== 4 && entry.family !== 6) || isIP(entry.address) !== entry.family) {
throw new WebError(`hostname "${hostname}" resolved to an invalid IP address`, 'WEB_PROVIDER_ERROR')
}
if (!isPublicIpAddress(entry.address)) {
throw new WebError(`URL hostname "${hostname}" resolves to a non-public IP address`, 'WEB_BLOCKED_URL')
}
const translatedIpv4 = translatedIpv4Address(entry.address, nat64Prefixes)
if (translatedIpv4 !== undefined && !isPublicIpAddress(translatedIpv4)) {
throw new WebError(`URL hostname "${hostname}" resolves through NAT64 to a non-public IPv4 address`, 'WEB_BLOCKED_URL')
}
addresses.push({ address: entry.address, family: entry.family })
}
return addresses
}
/** Discover the active DNS64 prefix set using RFC 7050's reserved hostname. */
async function discoverNat64Prefixes(signal: AbortSignal, resolver: AddressResolver): Promise<Nat64Prefix[]> {
const discovered = await raceWithSignal(
resolver(IPV4ONLY_DISCOVERY_HOST, { all: true, order: 'verbatim' }),
signal,
)
const prefixes: Nat64Prefix[] = []
const seen = new Set<string>()
for (const entry of discovered) {
if (entry.family !== 6 || isIP(entry.address) !== 6) continue
const bytes = ipaddr.parse(entry.address).toByteArray()
for (const length of RFC6052_PREFIX_LENGTHS) {
const embedded = embeddedIpv4Address(bytes, length)
if (embedded === undefined || !IPV4ONLY_SENTINELS.has(embedded)) continue
const prefixBytes = bytes.slice(0, length / 8)
const key = `${String(length)}:${prefixBytes.join('.')}`
if (seen.has(key)) continue
seen.add(key)
prefixes.push({ bytes: prefixBytes, length })
}
}
return prefixes
}
/** Return the RFC 6052-embedded IPv4 address when an IPv6 address matches a discovered prefix. */
function translatedIpv4Address(input: string, prefixes: readonly Nat64Prefix[]): string | undefined {
if (isIP(input) !== 6) return undefined
const bytes = ipaddr.parse(input).toByteArray()
for (const prefix of prefixes) {
if (!prefix.bytes.every((byte, index) => bytes[index] === byte)) continue
const embedded = embeddedIpv4Address(bytes, prefix.length)
if (embedded !== undefined) return embedded
}
return undefined
}
/** Extract one IPv4 address from an RFC 6052 IPv6 layout. */
function embeddedIpv4Address(bytes: readonly number[], prefixLength: Nat64Prefix['length']): string | undefined {
if (prefixLength === 96) return bytes.slice(12, 16).join('.')
if (bytes[8] !== 0) return undefined
const prefixBytes = prefixLength / 8
const beforeReservedOctet = 8 - prefixBytes
const ipv4 = [
...bytes.slice(prefixBytes, prefixBytes + beforeReservedOctet),
...bytes.slice(9, 9 + 4 - beforeReservedOctet),
]
return ipv4.join('.')
}
/**
* Whether a hostname is an IP literal that {@link resolvePublicAddresses} would refuse.
*
* A proxied hop skips those checks because the proxy resolves the origin, but a literal needs no
* resolution: the address is already stated, and handing it to a proxy running on this machine
* would reach exactly the loopback or private service the checks exist to keep out of reach.
*
* @param hostname - a URL's hostname, bracketed or not.
* @returns true when the host is a literal address no request may be sent to.
*/
export function isNonPublicIpLiteral(hostname: string): boolean {
const unbracketed = stripIpv6Brackets(hostname)
return isIP(unbracketed) !== 0 && !isPublicIpAddress(unbracketed)
}
/**
* Fetch through an agent whose lookup callback returns only the already validated address set. The
* URL hostname remains intact for HTTP Host and TLS SNI.
*
* The agent is this request's own because the address set is: pinning is how this package refuses a
* DNS answer that changes between validation and connection, and it may not apply process-wide —
* an operator-configured MCP server or model endpoint on loopback is a supported destination, and
* only the URLs this tool fetches are the model's to choose.
*
* @param url - validated HTTP(S) URL the policy does not route through a proxy.
* @param addresses - public addresses returned by {@link resolvePublicAddresses}.
* @param headers - request headers.
* @param signal - request and body-read cancellation signal.
* @returns a response plus the disposer its consumer must call.
*/
export async function requestPinned(
url: URL,
addresses: readonly PublicAddress[],
headers: Record<string, string>,
signal: AbortSignal,
): Promise<PinnedResponse> {
// Keep the Node-only transport out of browser-worker startup. The preview can load the provider
// and fail loud at its DNS stub without evaluating Undici; a real request resolves it here.
const { Agent, fetch } = await import('undici')
// Reached only where `proxyRouteFor` reported no proxy for this URL, and the pinned lookup this
// agent carries is per-request state the process-wide dispatcher cannot hold.
// proxy-exempt: pinning one request's validated addresses, on a URL the policy routes directly.
const dispatcher = new Agent({ autoSelectFamily: true, connect: { lookup: createPinnedLookup(addresses) } })
try {
// proxy-exempt: the agent above, whose lifetime is this one request.
const response = await fetch(url, { method: 'GET', redirect: 'manual', headers, signal, dispatcher })
return { response, close: async () => { await dispatcher.close() } }
} catch (error: unknown) {
await dispatcher.close()
throw error
}
}
/**
* Fetch through the dispatcher the proxy policy already installed, letting the proxy resolve the
* origin.
*
* No address set is pinned because none exists to pin: the proxy performs the lookup, and a
* connection pinned to a locally resolved address would reach the origin directly and defeat the
* proxy. The dispatcher is the process-wide one, so hops share its connection pool and no caller
* closes it.
*
* @param dispatcher - the route's dispatcher, from `proxyRouteFor`.
* @param url - validated HTTP(S) URL the policy routes through a proxy.
* @param headers - request headers.
* @param signal - request and body-read cancellation signal.
* @returns a response plus a disposer that releases nothing, so both paths close alike.
*/
export async function requestVia(
dispatcher: Dispatcher,
url: URL,
headers: Record<string, string>,
signal: AbortSignal,
): Promise<PinnedResponse> {
const { fetch } = await import('undici')
// proxy-exempt: the dispatcher is the installed policy's own, handed over by `proxyRouteFor`.
const response = await fetch(url, { method: 'GET', redirect: 'manual', headers, signal, dispatcher })
return { response, close: () => Promise.resolve() }
}
/** Production network operations kept as an object so provider tests can replace resolution only. */
export const publicHttpNetwork = {
resolve: resolvePublicAddresses,
request: requestPinned,
requestVia,
}
type LookupCallback = (
error: NodeJS.ErrnoException | null,
address: string | LookupAddress[],
family?: number,
) => void
/**
* Build the connector lookup that serves a fixed validated answer set.
*
* @param addresses - public addresses retained from the preceding resolution.
* @returns a Node-compatible lookup callback that performs no network resolution.
*/
export function createPinnedLookup(addresses: readonly PublicAddress[]): (
hostname: string,
options: LookupOptions,
callback: LookupCallback,
) => void {
return (hostname: string, options: LookupOptions, callback: LookupCallback): void => {
const family = typeof options.family === 'number'
? options.family
: options.family === 'IPv4' ? 4 : options.family === 'IPv6' ? 6 : 0
const eligible = family === 0 ? addresses : addresses.filter(address => address.family === family)
const selected = eligible[0]
if (selected === undefined) {
const error = Object.assign(new Error(`no validated address for ${hostname} in family ${family}`), {
code: 'ENOTFOUND',
hostname,
})
callback(error, options.all === true ? [] : '', family)
return
}
if (options.all === true) {
callback(null, eligible.map(address => ({ ...address })))
return
}
callback(null, selected.address, selected.family)
}
}
/** Race a non-cancellable OS lookup without letting it delay tool cancellation. */
function raceWithSignal<T>(promise: Promise<T>, signal: AbortSignal): Promise<T> {
const abortError = () => new Error('web fetch aborted during hostname resolution', { cause: signal.reason })
if (signal.aborted) return Promise.reject(abortError())
return new Promise<T>((resolve, reject) => {
const abort = () => { reject(abortError()) }
signal.addEventListener('abort', abort, { once: true })
promise.then(resolve, reject).finally(() => { signal.removeEventListener('abort', abort) })
})
}
/** WHATWG URL retains brackets around IPv6 hostnames; IP parsers do not. */
function stripIpv6Brackets(hostname: string): string {
return hostname.startsWith('[') && hostname.endsWith(']') ? hostname.slice(1, -1) : hostname
}
+2 -3
View File
@@ -8,7 +8,7 @@
import { WebError } from '@deepseek-ai/dsh-web'
/** Maximum accepted request URL length enforced by the public fetch provider. */
/** Maximum accepted request URL length enforced by the fetch provider. */
export const WEB_FETCH_MAX_URL_LENGTH = 2048
/** The body kinds this provider decodes. */
@@ -55,8 +55,7 @@ export function validateFetchUrl(input: string): URL {
/**
* Two URLs are same-origin when scheme, hostname, and port match. A redirect
* that crosses origins is refused so each new origin requires a fresh tool call
* and public-address validation.
* that crosses origins is refused so each new origin requires a fresh tool call.
*
* @param a - one of the two URLs to compare.
* @param b - the other URL to compare.
+47 -78
View File
@@ -1,7 +1,7 @@
/**
* Safe HTTP(S) retrieval for `ctx.web`: validates and pins public IP destinations, follows
* only same-origin redirects, enforces time and size limits, classifies and decodes text,
* and leaves presentation to `@deepseek-ai/dsh-tool-web`. Requests carry no browser cookies
* Bounded HTTP(S) retrieval for `ctx.web`: validates URLs, follows only same-origin redirects,
* enforces time and size limits, classifies and decodes text, and leaves presentation to
* `@deepseek-ai/dsh-tool-web`. Requests carry no browser cookies
* or ambient credentials.
* @module @deepseek-ai/dsh-web-fetch-http/provider
*/
@@ -10,9 +10,6 @@ import { WebError } from '@deepseek-ai/dsh-web'
import type { WebFetchBody, WebFetchProvider, WebFetchRequest, WebFetchResult } from '@deepseek-ai/dsh-web'
import { deadline, timeoutOf } from '@deepseek-ai/dsh-timeout'
import type { Response } from 'undici'
import { proxyRouteFor } from '@deepseek-ai/dsh-http-proxy'
import { isNonPublicIpLiteral, publicHttpNetwork } from './network.ts'
import type { PublicAddress } from './network.ts'
import { classifyContentType, decoderForCharset, isSameOrigin, parseCharset, validateFetchUrl } from './policy.ts'
/** Resolved provider limits (the plugin's schemastery Config supplies defaults). */
@@ -29,26 +26,17 @@ export interface HttpFetchLimits {
userAgent: string
}
/** Resolve one hostname to an already policy-validated address set. */
export type HttpFetchResolver = (hostname: string, signal: AbortSignal) => Promise<PublicAddress[]>
/** Stable id this provider registers under. */
export const LOCAL_FETCH_PROVIDER_ID = 'http'
/** The anonymous public HTTP(S) fetch provider. */
/** The anonymous HTTP(S) fetch provider. */
export class HttpFetchProvider implements WebFetchProvider {
readonly id = LOCAL_FETCH_PROVIDER_ID
/**
* @param limits - resolved transport and response limits.
* @param resolveAddresses - resolver that rejects non-public destinations before returning.
*/
constructor(
private readonly limits: HttpFetchLimits,
private readonly resolveAddresses: HttpFetchResolver = publicHttpNetwork.resolve,
) {}
/** @param limits - resolved transport and response limits. */
constructor(private readonly limits: HttpFetchLimits) {}
/** No credentials to check — an anonymous public fetcher is always usable. */
/** No credentials to check — an anonymous fetcher is always usable. */
available(): boolean {
return true
}
@@ -68,77 +56,58 @@ export class HttpFetchProvider implements WebFetchProvider {
let redirectsFollowed = 0
for (;;) {
const request = await this.requestOnce(currentUrl, signal)
const { response } = request
try {
if (isRedirectStatus(response.status)) {
// Enforce the redirect budget before resolving or validating the next hop.
if (redirectsFollowed >= this.limits.maxRedirects) {
await response.body?.cancel()
throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, 'WEB_REDIRECT_BLOCKED')
}
const location = response.headers.get('location')
if (location === null) {
// A redirect status with no Location is not a usable resource. Cancel
// the (possibly streaming) body before throwing so no socket leaks.
await response.body?.cancel()
throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, 'WEB_PROVIDER_ERROR')
}
const target = resolveRedirect(location, currentUrl)
// Re-validate the target against the same transport hygiene a direct request gets: a
// redirect must not be a back door to a credentialed, non-http(s), or over-long URL
// that validateFetchUrl would reject.
let validatedTarget: URL
try {
validatedTarget = validateFetchUrl(target.toString())
if (!isSameOrigin(validatedTarget, currentUrl)) {
throw new WebError(
`cross-origin redirect to ${validatedTarget.origin} is not followed automatically; retry against that URL directly`,
'WEB_REDIRECT_BLOCKED',
)
}
} catch (error: unknown) {
await response.body?.cancel()
throw error
}
const response = await this.requestOnce(currentUrl, signal)
if (isRedirectStatus(response.status)) {
// Enforce the redirect budget before validating the next hop.
if (redirectsFollowed >= this.limits.maxRedirects) {
await response.body?.cancel()
currentUrl = validatedTarget
redirectsFollowed++
continue
throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, 'WEB_REDIRECT_BLOCKED')
}
return await this.readBody(response, currentUrl, signal)
} finally {
await request.close()
const location = response.headers.get('location')
if (location === null) {
// A redirect status with no Location is not a usable resource. Cancel
// the (possibly streaming) body before throwing so no socket leaks.
await response.body?.cancel()
throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, 'WEB_PROVIDER_ERROR')
}
const target = resolveRedirect(location, currentUrl)
// Re-validate the target against the same transport hygiene a direct request gets: a
// redirect must not be a back door to a credentialed, non-http(s), or over-long URL
// that validateFetchUrl would reject.
let validatedTarget: URL
try {
validatedTarget = validateFetchUrl(target.toString())
if (!isSameOrigin(validatedTarget, currentUrl)) {
throw new WebError(
`cross-origin redirect to ${validatedTarget.origin} is not followed automatically; retry against that URL directly`,
'WEB_REDIRECT_BLOCKED',
)
}
} catch (error: unknown) {
await response.body?.cancel()
throw error
}
await response.body?.cancel()
currentUrl = validatedTarget
redirectsFollowed++
continue
}
return await this.readBody(response, currentUrl, signal)
}
}
private async requestOnce(url: URL, signal: AbortSignal) {
/** Send one GET through the process-wide dispatcher, which applies the installed proxy policy. */
private async requestOnce(url: URL, signal: AbortSignal): Promise<Response> {
const headers = {
'user-agent': this.limits.userAgent,
'accept': 'text/html,application/xhtml+xml,text/*;q=0.9,application/json;q=0.8',
}
try {
// A proxied hop skips public-address resolution and pinning: the proxy performs the origin's
// DNS, so there is no local address to validate, and pinning one would connect directly and
// bypass the proxy. A hop the policy bypasses — every loopback and every `NO_PROXY` entry —
// still takes the resolved-and-pinned path unchanged.
//
// One route decides both the branch and the dispatcher, so a mount or disposal between two
// reads cannot return a direct, unpinned agent for a URL this branch cleared as proxied.
//
// An IP literal the address checks would refuse never takes it. The proxy would resolve
// nothing — the address is already stated — so the shortcut would spend the checks for
// nothing and let a proxy on this machine reach the very service they keep out of reach.
const route = proxyRouteFor(url)
if (route.proxied && !isNonPublicIpLiteral(url.hostname)) {
return await publicHttpNetwork.requestVia(route.dispatcher, url, headers, signal)
}
const addresses = await this.resolveAddresses(url.hostname, signal)
return await publicHttpNetwork.request(url, addresses, headers, signal)
// Keep the Node-only transport out of browser-worker startup; a real request loads it here.
const { fetch } = await import('undici')
return await fetch(url, { method: 'GET', redirect: 'manual', headers, signal })
} catch (error: unknown) {
if (error instanceof WebError) throw error
throw translateAbortOrNetwork(error, signal)
}
}
@@ -4,9 +4,8 @@ import { AddressInfo } from 'node:net'
import { Context } from '@deepseek-ai/cordis'
import WebRuntime from '@deepseek-ai/dsh-web'
import { HttpFetchProvider, LOCAL_FETCH_PROVIDER_ID } from '@deepseek-ai/dsh-web-fetch-http'
import type { HttpFetchLimits, HttpFetchResolver } from '@deepseek-ai/dsh-web-fetch-http'
import type { HttpFetchLimits } from '@deepseek-ai/dsh-web-fetch-http'
import * as fetchPlugin from '@deepseek-ai/dsh-web-fetch-http'
import { createPinnedLookup, isPublicIpAddress, publicHttpNetwork, requestPinned, resolvePublicAddresses } from '../src/network.ts'
import {
classifyContentType,
decoderForCharset,
@@ -30,7 +29,6 @@ type Handler = (req: IncomingMessage, res: ServerResponse) => void
let server: Server
let base: string
let handler: Handler
let restoreResolution: () => void
beforeEach(async () => {
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('default') }
@@ -38,8 +36,6 @@ beforeEach(async () => {
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
const { port } = server.address() as AddressInfo
base = `http://127.0.0.1:${port}`
const spy = vi.spyOn(publicHttpNetwork, 'resolve').mockResolvedValue([{ address: '127.0.0.1', family: 4 }])
restoreResolution = () => { spy.mockRestore() }
})
afterEach(async () => {
@@ -94,168 +90,6 @@ describe('policy helpers', () => {
})
})
describe('public-network policy', () => {
it('accepts only globally reachable unicast addresses', () => {
for (const address of ['8.8.8.8', '2001:4860:4860::8888', '::ffff:8.8.8.8']) {
expect(isPublicIpAddress(address), address).toBe(true)
}
for (const address of [
'0.0.0.0',
'10.0.0.1',
'100.64.0.1',
'127.0.0.1',
'169.254.169.254',
'192.0.2.1',
'224.0.0.1',
'255.255.255.255',
'::',
'::1',
'fe80::1',
'fc00::1',
'ff02::1',
'::ffff:127.0.0.1',
'64:ff9b::808:808',
'not-an-ip',
]) {
expect(isPublicIpAddress(address), address).toBe(false)
}
})
it('retains one fully public DNS answer set', async () => {
const resolver = vi.fn(async () => [
{ address: '8.8.4.4', family: 4 },
{ address: '2001:4860:4860::8888', family: 6 },
])
await expect(resolvePublicAddresses('example.test', new AbortController().signal, resolver))
.resolves.toEqual([
{ address: '8.8.4.4', family: 4 },
{ address: '2001:4860:4860::8888', family: 6 },
])
})
it('rejects the whole DNS answer set when one address is not public', async () => {
const resolver = vi.fn(async () => [
{ address: '8.8.8.8', family: 4 },
{ address: '127.0.0.1', family: 4 },
])
await expect(resolvePublicAddresses('rebinding.test', new AbortController().signal, resolver))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
})
it('rejects empty and invalid resolver results', async () => {
await expect(resolvePublicAddresses('empty.test', new AbortController().signal, async () => []))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
await expect(resolvePublicAddresses('family.test', new AbortController().signal, async () => [{ address: '8.8.8.8', family: 0 }]))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
await expect(resolvePublicAddresses('mismatch.test', new AbortController().signal, async () => [{ address: '::1', family: 4 }]))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
})
it('validates bracketed IPv6 literals after checking for an active DNS64 prefix', async () => {
const resolver = vi.fn(async () => [{ address: '192.0.0.170', family: 4 }])
await expect(resolvePublicAddresses('[2001:4860:4860::8888]', new AbortController().signal, resolver))
.resolves.toEqual([{ address: '2001:4860:4860::8888', family: 6 }])
expect(resolver).toHaveBeenCalledWith('ipv4only.arpa', { all: true, order: 'verbatim' })
})
it('rejects a network-specific NAT64 address that translates to private IPv4', async () => {
const resolver = vi.fn(async (hostname: string) => hostname === 'ipv4only.arpa'
? [{ address: '2001:4860:64:64::c000:aa', family: 6 }]
: [{ address: '2001:4860:64:64::7f00:1', family: 6 }])
await expect(resolvePublicAddresses('nat64.test', new AbortController().signal, resolver))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
})
it('accepts a network-specific NAT64 address that translates to public IPv4', async () => {
const resolver = vi.fn(async (hostname: string) => hostname === 'ipv4only.arpa'
? [{ address: '2001:4860:64:64::c000:aa', family: 6 }]
: [{ address: '2001:4860:64:64::808:808', family: 6 }])
await expect(resolvePublicAddresses('nat64.test', new AbortController().signal, resolver))
.resolves.toEqual([{ address: '2001:4860:64:64::808:808', family: 6 }])
})
it('deduplicates discovered prefixes and ignores addresses outside their translation layout', async () => {
const resolver = vi.fn(async (hostname: string) => hostname === 'ipv4only.arpa'
? [
{ address: '2001:4860:64:64::c000:aa', family: 6 },
{ address: '2001:4860:64:64::c000:ab', family: 6 },
{ address: '2001:4860:64:64:c0:0:aa00:0', family: 6 },
]
: [
{ address: '2001:4860:65:64::808:808', family: 6 },
{ address: '2001:4860:64:64:100::1', family: 6 },
])
await expect(resolvePublicAddresses('native-v6.test', new AbortController().signal, resolver))
.resolves.toEqual([
{ address: '2001:4860:65:64::808:808', family: 6 },
{ address: '2001:4860:64:64:100::1', family: 6 },
])
})
it('stops waiting for DNS when the request is aborted', async () => {
let finish!: (value: never[]) => void
const resolver = vi.fn(() => new Promise<never[]>((resolve) => { finish = resolve }))
const controller = new AbortController()
const pending = resolvePublicAddresses('slow.test', controller.signal, resolver)
controller.abort(new Error('stop'))
await expect(pending).rejects.toThrow('web fetch aborted during hostname resolution')
finish([])
const alreadyAborted = new AbortController()
alreadyAborted.abort(new Error('already stopped'))
await expect(resolvePublicAddresses('slow.test', alreadyAborted.signal, resolver))
.rejects.toThrow('web fetch aborted during hostname resolution')
})
it('propagates resolver failures', async () => {
await expect(resolvePublicAddresses('broken.test', new AbortController().signal, async () => { throw new Error('dns failed') }))
.rejects.toThrow('dns failed')
})
it('serves only the retained addresses through the connector lookup', async () => {
const lookup = createPinnedLookup([
{ address: '8.8.8.8', family: 4 },
{ address: '2001:4860:4860::8888', family: 6 },
])
const call = (options: Parameters<typeof lookup>[1]) => new Promise<{
error: NodeJS.ErrnoException | null
address: string | import('node:dns').LookupAddress[]
family: number | undefined
}>((resolve) => {
lookup('fixed.test', options, (error, address, family) => { resolve({ error, address, family }) })
})
await expect(call({ all: true })).resolves.toMatchObject({
error: null,
address: [{ address: '8.8.8.8', family: 4 }, { address: '2001:4860:4860::8888', family: 6 }],
})
await expect(call({ family: 4 })).resolves.toMatchObject({ error: null, address: '8.8.8.8', family: 4 })
await expect(call({ family: 'IPv6' })).resolves.toMatchObject({ error: null, address: '2001:4860:4860::8888', family: 6 })
await expect(call({ family: 'IPv4' })).resolves.toMatchObject({ error: null, address: '8.8.8.8', family: 4 })
await expect(call({ family: 7 })).resolves.toMatchObject({ error: { code: 'ENOTFOUND' }, address: '', family: 7 })
await expect(call({ family: 7, all: true })).resolves.toMatchObject({ error: { code: 'ENOTFOUND' }, address: [], family: 7 })
})
it('pins the connection to the validated address without resolving the URL hostname again', async () => {
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('pinned') }
const { port } = server.address() as AddressInfo
const request = await requestPinned(
new URL(`http://does-not-resolve.invalid:${port}/`),
[{ address: '127.0.0.1', family: 4 }],
{},
new AbortController().signal,
)
try {
await expect(request.response.text()).resolves.toBe('pinned')
} finally {
await request.close()
}
})
})
describe('HttpFetchProvider success', () => {
it('fetches a text body', async () => {
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('hello world') }
@@ -272,14 +106,6 @@ describe('HttpFetchProvider success', () => {
expect(result.body).toEqual({ kind: 'html', content: '<h1>hi</h1>' })
})
it('uses an explicitly injected validated-address resolver', async () => {
const resolveAddresses = vi.fn<HttpFetchResolver>(async () => [{ address: '127.0.0.1', family: 4 }])
const result = await new HttpFetchProvider(limits, resolveAddresses).fetch({ url: base })
expect(result.statusCode).toBe(200)
expect(resolveAddresses).toHaveBeenCalledWith('127.0.0.1', expect.any(AbortSignal))
expect(publicHttpNetwork.resolve).not.toHaveBeenCalled()
})
it('sends the configured user agent', async () => {
let seen: string | undefined
handler = (req, res) => { seen = req.headers['user-agent']; res.writeHead(200, { 'content-type': 'text/plain' }); res.end('ok') }
@@ -460,10 +286,10 @@ describe('HttpFetchProvider redirects', () => {
})
describe('HttpFetchProvider invalid URLs and abort', () => {
it('blocks a loopback destination before opening a connection', async () => {
restoreResolution()
await expect(provider().fetch({ url: base }))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
it.each(['localhost', '127.0.0.1'])('fetches a non-public destination at %s', async (host) => {
const { port } = server.address() as AddressInfo
const result = await provider().fetch({ url: `http://${host}:${port}/` })
expect(result.body).toEqual({ kind: 'text', content: 'default' })
})
it('rejects a non-http scheme before any network access', async () => {
@@ -520,49 +346,36 @@ describe('HttpFetchProvider invalid URLs and abort', () => {
})
describe('HttpFetchProvider body cancellation on error paths', () => {
/** A fake Response whose body.cancel is observable. */
type FakeInit = { status: number; headers: Record<string, string>; location?: string }
function fakeResponse(init: FakeInit): { response: Response; cancelled: () => boolean } {
let cancelled = false
const headers = new Headers(init.headers)
if (init.location !== undefined) headers.set('location', init.location)
const response = {
status: init.status,
headers,
body: { cancel: () => { cancelled = true; return Promise.resolve() } },
} as unknown as Response
return { response, cancelled: () => cancelled }
}
function stubRequest(response: Response): void {
vi.spyOn(publicHttpNetwork, 'request').mockResolvedValue({
response: response as never,
close: async () => {},
/** Answer with headers and an unfinished body; resolves when the client closes the socket. */
function respondWithOpenBody(status: number, headers: Record<string, string>): Promise<void> {
return new Promise((resolve) => {
handler = (req, res) => {
req.socket.once('close', () => { resolve() })
res.writeHead(status, headers)
res.write('unfinished')
}
})
}
it('cancels the body when a cross-origin redirect is blocked', async () => {
const { response, cancelled } = fakeResponse({ status: 302, headers: {}, location: 'https://elsewhere.test/' })
stubRequest(response)
await expect(provider().fetch({ url: 'http://127.0.0.1:9/' }))
const closed = respondWithOpenBody(302, { location: 'https://elsewhere.test/' })
await expect(provider().fetch({ url: base }))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' }))
expect(cancelled()).toBe(true)
await closed
})
it('cancels the body when an unsupported charset is rejected', async () => {
const { response, cancelled } = fakeResponse({ status: 200, headers: { 'content-type': 'text/plain; charset=not-a-charset' } })
stubRequest(response)
await expect(provider().fetch({ url: 'http://127.0.0.1:9/' }))
const closed = respondWithOpenBody(200, { 'content-type': 'text/plain; charset=not-a-charset' })
await expect(provider().fetch({ url: base }))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
expect(cancelled()).toBe(true)
await closed
})
it('cancels the body when a redirect has no Location header', async () => {
const { response, cancelled } = fakeResponse({ status: 302, headers: {} })
stubRequest(response)
await expect(provider().fetch({ url: 'http://127.0.0.1:9/' }))
const closed = respondWithOpenBody(302, {})
await expect(provider().fetch({ url: base }))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
expect(cancelled()).toBe(true)
await closed
})
})
@@ -1,10 +1,9 @@
import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'
import type { AddressInfo } from 'node:net'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { installProxyFromEnvironment } from '@deepseek-ai/dsh-http-proxy'
import { HttpFetchProvider } from '@deepseek-ai/dsh-web-fetch-http'
import type { HttpFetchLimits } from '@deepseek-ai/dsh-web-fetch-http'
import { isNonPublicIpLiteral, publicHttpNetwork } from '../src/network.ts'
const limits: HttpFetchLimits = {
maxResponseBytes: 5_000_000,
@@ -23,8 +22,7 @@ let originUrl: string
/**
* The target for every assertion about a tunnelled hop. Loopback cannot serve: no policy routes
* this machine through a proxy. The host never resolves — the proxy answers the absolute-form
* request — which is also what makes the skipped resolver observable.
* this machine through a proxy. The host never resolves; the proxy answers the absolute-form request.
*/
const proxyTarget = 'http://origin.test/page'
let disposeProxy: (() => Promise<void>) | undefined
@@ -55,7 +53,6 @@ beforeEach(async () => {
afterEach(async () => {
await disposeProxy?.()
disposeProxy = undefined
vi.restoreAllMocks()
await Promise.all([
new Promise<void>((resolve) => { proxy.close(() => { resolve() }) }),
new Promise<void>((resolve) => { origin.close(() => { resolve() }) }),
@@ -72,68 +69,23 @@ async function installProxy(): Promise<() => Promise<void>> {
}
describe('fetching through a proxy', () => {
it('tunnels the request and never resolves a public address for it', async () => {
const resolve = vi.spyOn(publicHttpNetwork, 'resolve')
it('tunnels the request through the installed proxy', async () => {
disposeProxy = await installProxy()
const result = await new HttpFetchProvider(limits).fetch({ url: proxyTarget })
expect(result.body.content).toBe('via-proxy')
expect(proxied).toEqual([proxyTarget])
// Through a proxy the origin's DNS happens proxy-side, so the resolver that rejects non-public
// destinations is not consulted at all.
expect(resolve).not.toHaveBeenCalled()
})
it('keeps resolving and pinning a hop the policy does not proxy', async () => {
const resolve = vi.spyOn(publicHttpNetwork, 'resolve')
.mockResolvedValue([{ address: '127.0.0.1', family: 4 }])
// No bypass entry needed: a resolved policy never routes loopback through a proxy, which is
// exactly the case this asserts still resolves and pins.
it('connects directly to a host the policy does not proxy', async () => {
// A resolved policy never routes loopback through a proxy.
disposeProxy = await installProxy()
const result = await new HttpFetchProvider(limits).fetch({ url: originUrl })
expect(result.body.content).toBe('direct')
expect(proxied).toEqual([])
expect(resolve).toHaveBeenCalledOnce()
})
it('resolves and pins when no proxy is installed', async () => {
const resolve = vi.spyOn(publicHttpNetwork, 'resolve')
.mockResolvedValue([{ address: '127.0.0.1', family: 4 }])
const result = await new HttpFetchProvider(limits).fetch({ url: originUrl })
expect(result.body.content).toBe('direct')
expect(resolve).toHaveBeenCalledOnce()
})
it.each(['10.0.0.5', '169.254.169.254', '127.0.0.2'])(
'refuses %s instead of letting the proxy reach it for us',
async (host) => {
const resolve = vi.spyOn(publicHttpNetwork, 'resolve')
disposeProxy = await installProxy()
// The proxied path exists because a proxy resolves the origin; a literal needs no resolution,
// so taking it would spend the address checks for nothing and hand a proxy on this machine
// the private or loopback destination those checks exist to refuse. The hop therefore takes
// the validated path instead, where the existing refusal already covers it.
await expect(new HttpFetchProvider(limits).fetch({ url: `http://${host}:8080/` }))
.rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
expect(proxied).toEqual([])
expect(resolve).toHaveBeenCalledOnce()
},
)
it('reads an IPv4-mapped literal as non-public without asking the network', () => {
// Driven through the predicate rather than a fetch: an IPv6 literal sends `resolvePublicAddresses`
// looking for a NAT64 prefix before it refuses anything, and that is a real DNS query. The three
// IPv4 cases above already prove the branch end to end without one.
expect(isNonPublicIpLiteral('[::ffff:7f00:1]')).toBe(true)
expect(isNonPublicIpLiteral('[::1]')).toBe(true)
expect(isNonPublicIpLiteral('[::ffff:808:808]')).toBe(false)
expect(isNonPublicIpLiteral('example.com')).toBe(false)
})
it('still refuses a cross-origin redirect on the proxied path', async () => {
@@ -22,9 +22,6 @@
},
{
"path": "../web"
},
{
"path": "../../util/http-proxy"
}
]
}
-3
View File
@@ -15589,9 +15589,6 @@ importers:
'@deepseek-ai/schemastery':
specifier: link:../../../vendor/schemastery
version: link:../../../vendor/schemastery
ipaddr.js:
specifier: ^2.5.0
version: 2.5.0
undici:
specifier: 8.11.2
version: 8.11.2
-3
View File
@@ -2059,9 +2059,6 @@
"packages/web/tool-web/tests/spill.spec.ts": {
"ecf3420d023d28e1cbc687d438bb66ee49d55a4ae4cb0828dd0f59b926363903": 1
},
"packages/web/web-fetch-http/tests/fetch-http.spec.ts": {
"366f13ff1ff2522ad824a1d4333085e2c771c0b7ca22783aa41ca2e23233b259": 1
},
"packages/web/web-search-deepseek/tests/deepseek.spec.ts": {
"37c9617ff75d6a383f58dd63f939a88803e4ff5e1676bfe7e1c0cc340fc65014": 5,
"598014dcfe77101781e3c6a02709097e3b0151a335e909c44719fac08fdea433": 2
+1 -1
View File
@@ -1,7 +1,7 @@
import { describe, expect, it } from 'vitest'
import { findDispatcherViolations, scanRepository, DISPATCHER_OWNER } from './verify-no-bare-dispatcher.ts'
const FILE = 'packages/web/web-fetch-http/src/network.ts'
const FILE = 'packages/web/web-fetch-http/src/provider.ts'
function reasons(source: string, file = FILE): string[] {
return findDispatcherViolations(file, source).map(violation => violation.what)
+2 -3
View File
@@ -8,9 +8,8 @@
* agent silently bypassed every proxy.
*
* `proxyRouteFor(url)` from that package is the sanctioned way to ask where one request goes and to
* get the transport that answer assumed. A call site that genuinely owns its transport — because it
* carries per-request state the process-wide dispatcher cannot, as `web-fetch-http`'s address
* pinning does — says so with the marker below.
* get the transport that answer assumed. A call site that genuinely owns its transport, because it
* carries per-request state the process-wide dispatcher cannot, says so with the marker below.
*
* Discovery is syntax-aware, as `scripts/AGENTS.md` requires: a line-wise regex misses the
* `{ dispatcher }` shorthand and a `new Alias(...)` whose import renamed `Agent`, and both bypass the
@@ -1,7 +1,7 @@
/**
* Deterministic HTTP provider for the web-fetch snapshot scenario: a small
* HTML page (headings, named entities, a GFM table, nested formatting) on a
* OS-assigned loopback port behind the real address-pinned transport. Recording
* OS-assigned loopback port behind the real HTTP transport. Recording
* and replay therefore exercise fetch and markdown rendering without external
* network while retaining the recorded request URL.
*/
@@ -45,12 +45,7 @@ export async function apply(ctx) {
let transportUrl
let startupError
const resolveAddresses = async (hostname) => {
if (hostname !== 'public.test') throw new Error(`unexpected snapshot hostname: ${hostname}`)
return [{ address: '127.0.0.1', family: 4 }]
}
const provider = new HttpFetchProvider(LIMITS, resolveAddresses)
const provider = new HttpFetchProvider(LIMITS)
const unregister = ctx.web.registerFetchProvider({
id: provider.id,
available: () => provider.available(),
@@ -76,6 +71,7 @@ export async function apply(ctx) {
},
onListening: (address) => {
transportUrl = new URL(RECORDED_URL)
transportUrl.hostname = '127.0.0.1'
transportUrl.port = String(address.port)
readiness.resolve(undefined)
},