mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-10-26 16:55:08 +00:00
feat(cua): add optional macOS setup with existing plugin interfaces
This commit is contained in:
creatixchu committed
1 parent
3a39d2b132
commit
1954e948cb
138 files changed
+4796
-492
No files matched your search
+4
-4
@@ -9,11 +9,11 @@
|
||||
en: 2ba6c43b20a3d347
|
||||
zh: 5647179ddaa38ad9
|
||||
/agent-note-ship-experimental-capabilities-as-optional-bundles/decision:
|
||||
en: 6c8583bdefc5e36c
|
||||
zh: 0928b4f2876bbfe4
|
||||
en: 69df6602f92a3e09
|
||||
zh: 2e2ed04760086f4c
|
||||
/agent-note-ship-experimental-capabilities-as-optional-bundles/alternatives-considered:
|
||||
en: d6f40bf02df37ab7
|
||||
zh: e5b85ce9d41e8754
|
||||
/agent-note-ship-experimental-capabilities-as-optional-bundles/consequences:
|
||||
en: 38d27583533cd028
|
||||
zh: 9bf11f5489c1aa43
|
||||
en: 3064da835e892958
|
||||
zh: bfcf967b6acc6f85
|
||||
+5
-1
@@ -12,12 +12,14 @@ The Web plugin page offered two optional bundles, Agent Teams and voice input. A
|
||||
|
||||
`OPTIONAL_BUNDLES` owns the installation's default-off selections, including maintained optional capabilities and experimental capabilities. Every bundle declares `icon` and exports `./locale/*.json` with `meta.title` and `meta.description`; the Official group displays that metadata. A bundle's presentation does not determine its [experimental status](../../../../packages/experimental/README.md#status). The isolation check validates admission and still rejects experimental dependencies of nonexperimental bundles.
|
||||
|
||||
An optional bundle's dependencies are downloaded with every `dsh` installation. [Reasoning translation](2026-10-05-anonymous-reasoning-translation.md) uses three experimental packages without adding third-party runtime dependencies. Admission therefore considers installation cost and whether a switch supplies a usable composition without an additional configuration form. Browser and computer providers remain explicit compositions because they add substantial runtimes or require an external executable. Stagehand requires model credentials; Python PTC replaces a core provider and conflicts with TypeScript workflow consumers. Demonstrative hooks, webhooks, and mods stay outside the list. [Session Inspector](../feature/2026-09-24-session-inspector.md) owns the Inspector's distinct activation behavior.
|
||||
An optional bundle's dependencies are downloaded with every `dsh` installation. [Reasoning translation](2026-10-05-anonymous-reasoning-translation.md) uses three experimental packages without adding third-party runtime dependencies. Admission therefore considers installation cost and whether a switch supplies a usable composition without an additional configuration form. External browser and Cua Driver providers remain explicit compositions because they add substantial runtimes or require an external executable. Stagehand requires model credentials; Python PTC replaces a core provider and conflicts with TypeScript workflow consumers. Demonstrative hooks, webhooks, and mods stay outside the list. [Session Inspector](../feature/2026-09-24-session-inspector.md) owns the Inspector's distinct activation behavior.
|
||||
|
||||
Bundles insert ordinary Host rows; optional model tools register in the global tool layer, so every preset, including minimal, receives them. Plugin management, profile composition, and reload need no bundle-specific behavior. Search, Ralph, and terminals supply their own isolated services in a Host group. Search opens one in-memory index on demand; the Host search policy stays unchanged. Badge skills and title providers act on the Host. Switching a bundle registers or removes its tools for live Agents, and later user patch layers retain precedence. A replacement title provider may register once the previous provider's disposal starts; the previous provider's late results never commit.
|
||||
|
||||
Claude Code and Codex install on demand through the separate `ON_DEMAND_BUNDLES` catalog; the [on-demand native-bundle decision](2026-10-05-official-on-demand-bundles.md) owns the cost, version, and offline-discovery trade-offs.
|
||||
|
||||
Computer Use is a shipped optional bundle in the experimental group. Its repository-owned native component is prepared when the provider loads, without an external Cua Driver runtime.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Ship every provider as an optional bundle.** Composes and displays correctly, but grows every install by provider runtimes for a capability most installations never switch on; voice input's `sherpa-onnx-node` is the one accepted precedent.
|
||||
@@ -31,3 +33,5 @@ Claude Code and Codex install on demand through the separate `ON_DEMAND_BUNDLES`
|
||||
## Consequences
|
||||
|
||||
Official discovery does not imply experimental status or a different plugin lifecycle. Switching a bundle off keeps its installed files; failed activation retains the existing saved-selection behavior. Bundle rows are ordinary Host rows with their usual switches. Individual and combined composition tests cover the Host rows and every preset's tool catalog; live switching and recorded Sessions cover activation and model-visible behavior.
|
||||
|
||||
Computer Use keeps its permission service and native provider on the Host and registers tools in Agent scopes whose bound preset is `ptc`. Existing Agent events and `createScope()` maintain current, new and resumed Agents. Preset declarations remain unchanged. Only the explicit detail-page action changes the new-session default through the existing settings form; the activation switch does not configure PTC execution.
|
||||
+5
-1
@@ -12,12 +12,14 @@ Web 插件页只提供两个可选 bundle：Agent Teams 与语音输入。Auto r
|
||||
|
||||
`OPTIONAL_BUNDLES` 管理安装随附且默认关闭的选项，包括维护中的可选能力与实验性能力。每个 bundle 声明 `icon`，并导出带 `meta.title` 与 `meta.description` 的 `./locale/*.json`；官方分组显示这些元数据。Bundle 的展示方式不决定其[实验性状态](../../../../packages/experimental/README.zh.md#status)。隔离检查校验准入，并继续拒绝非实验 bundle 的实验性依赖。
|
||||
|
||||
可选 bundle 的依赖随每次 `dsh` 安装下载。[思考过程翻译](2026-10-05-anonymous-reasoning-translation.zh.md)使用三个实验包，不增加第三方运行时依赖。因此准入需要考虑安装成本，以及开关能否在不增加配置表单的情况下提供可用组合。浏览器与电脑提供方仍使用显式组合，因为它们会增加较大的运行时或要求外部可执行文件。Stagehand 需要模型凭据；Python PTC 替换核心提供方，并与 TypeScript workflow 消费方冲突。演示性钩子、webhook 与模组不进入名单。[Session Inspector](../feature/2026-09-24-session-inspector.zh.md) 管理 Inspector 特有的激活行为。
|
||||
可选 bundle 的依赖随每次 `dsh` 安装下载。[思考过程翻译](2026-10-05-anonymous-reasoning-translation.zh.md)使用三个实验包，不增加第三方运行时依赖。因此准入需要考虑安装成本，以及开关能否在不增加配置表单的情况下提供可用组合。外部浏览器与 Cua Driver 提供方仍使用显式组合，因为它们会增加较大的运行时或要求外部可执行文件。Stagehand 需要模型凭据；Python PTC 替换核心提供方，并与 TypeScript workflow 消费方冲突。演示性钩子、webhook 与模组不进入名单。[Session Inspector](../feature/2026-09-24-session-inspector.zh.md) 管理 Inspector 特有的激活行为。
|
||||
|
||||
Bundle 插入普通 Host 行；可选模型工具注册到全局工具层，因此包括 minimal 在内的所有预设都会获得它们。插件管理、profile 组合与重载无需针对 bundle 的特殊行为。搜索、Ralph 与终端在 Host 组内提供自己的隔离服务。搜索按需打开一个内存索引；Host 搜索策略保持不变。徽章技能与标题提供方作用于 Host。切换 bundle 会为运行中的 Agent 注册或移除其工具，后续用户 patch 层继续优先。前一个标题提供方开始清理后，替代提供方即可注册；前一个提供方的迟到结果不会提交。
|
||||
|
||||
Claude Code 与 Codex 通过独立的 `ON_DEMAND_BUNDLES` 目录按需安装，其成本、版本与离线发现取舍由[按需原生 bundle 决策](2026-10-05-official-on-demand-bundles.zh.md)管理。
|
||||
|
||||
Computer Use 是 experimental 包组内随安装提供的可选组合包。提供方加载时准备仓库自有的本机组件，无需外部 Cua Driver 运行时。
|
||||
|
||||
## 考虑过的替代方案
|
||||
|
||||
**把所有提供方都做成可选 bundle。** 组合与展示都正确，但会为多数安装从不启用的能力把提供方运行时塞进每次安装；语音输入的 `sherpa-onnx-node` 是唯一被接受的先例。
|
||||
@@ -31,3 +33,5 @@ Claude Code 与 Codex 通过独立的 `ON_DEMAND_BUNDLES` 目录按需安装，
|
||||
## 后果
|
||||
|
||||
官方发现不表示实验性状态，也不引入不同的插件生命周期。关闭 bundle 保留已安装文件；激活失败保留现有的已保存选择行为。Bundle 行是普通 Host 行，使用常规开关。独立与组合测试覆盖 Host 行及每个预设的工具目录；实时切换与已记录 Session 覆盖激活和模型可见行为。
|
||||
|
||||
Computer Use 将权限服务和原生提供方保留在 Host，在已绑定 `ptc` preset 的 Agent 作用域中注册工具。现有 Agent 事件和 `createScope()` 维护当前、新建和恢复的 Agent，preset 声明保持不变。只有详情页的显式操作通过现有设置表单修改新会话默认模式，启用开关不配置 PTC 执行方式。
|
||||
@@ -9,11 +9,11 @@
|
||||
en: b32a7ce6483b9b48
|
||||
zh: ec0938ad032b99d8
|
||||
/agent-note-application-consent-for-native-computer-use/decision:
|
||||
en: 92247ff599c1103f
|
||||
zh: 26845420a9711c83
|
||||
en: 22b3e4b770d2517f
|
||||
zh: 2250b31a5a50758a
|
||||
/agent-note-application-consent-for-native-computer-use/alternatives-considered:
|
||||
en: 4693dcd217c4fedb
|
||||
zh: 1dc9bf9067314006
|
||||
en: 80cd5fd93a42d937
|
||||
zh: 2a0e241b1b8c2e1d
|
||||
/agent-note-application-consent-for-native-computer-use/consequences:
|
||||
en: 0146a69ab2086edc
|
||||
zh: 4257bc1ab4432c3f
|
||||
en: 4bda3366af65250a
|
||||
zh: 7c83561b4db07284
|
||||
@@ -10,29 +10,38 @@ macOS privileges authorize an execution host, not a user's choice of target appl
|
||||
|
||||
## Decision
|
||||
|
||||
The CUA executor owns application consent for every object-API and native-tool dispatch. It uses the existing User Questions service for human decisions and the provider's existing profile configuration for saved grants. General tool approval and operating-system privileges remain separate requirements. Arbitrary persistent PTC code and separately authorized shell tools retain their own execution authority.
|
||||
The CUA executor owns application consent for every object-API and native-tool dispatch. It uses the existing User Questions service for human decisions and the provider's existing profile configuration for saved grants. The existing question UI presents the offered labels; the executor rejects custom text or answers that do not select exactly one offered label. Permission-page icons and names are presentation metadata, excluded from saved identities and model results. General tool approval and operating-system privileges remain separate requirements. Arbitrary persistent PTC code and separately authorized shell tools retain their own execution authority.
|
||||
|
||||
A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive task access only, tied to a process instance and Agent connection lifetime. Labels do not establish identity.
|
||||
System-permission buttons call the native helper on the DSH host and remain available to remote browsers. The [usage documentation](../../../../packages/experimental/client-ui-cua/README.md#use-this-package) explains where to complete authorization; the page has no persistent remote-access notice.
|
||||
|
||||
A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive session access only, tied to a process instance. Labels do not establish identity.
|
||||
|
||||
Cold application selection prepares an executable identity and handle without launching. The same consent decision guards the private launch continuation, and the helper checks the executable again before launch and binds the resulting process afterward. Signed selections retain the normal designated-requirement identity; an unverifiable executable uses a content identity for this pending selection and cannot create a saved grant. Preparing metadata before asking avoids starting an application the user refuses.
|
||||
|
||||
Selections of the same application within one task are serialized, so a pending question does not produce competing decisions. Saved-grant additions read the current configuration inside one mutation queue. Refusals prevent another prompt within the task; an explicitly restored saved grant takes precedence at dispatch.
|
||||
Session decisions are kept in memory by Session ID and application identity. Agent disposal and helper replacement release native handles without discarding these decisions; provider disposal or DSH restart clears them. This scope covers follow-up turns without relying on the model to decide when a user intent has ended.
|
||||
|
||||
Removing a saved grant invalidates Agent connections retaining the application's handles and cancels active operations; Agent/provider disposal awaits native connection cleanup. Reauthorization uses a fresh connection, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator.
|
||||
Selections of the same application within one session are serialized, so a pending question does not produce competing decisions. Saved-grant additions and revocations share one mutation queue and read the current grant list inside it. Provider shutdown cancels writes still waiting for HMR admission; admitted writes settle before disposal completes. Refusals prevent another prompt until the user permits asking again; a restored saved grant takes precedence in both dispatch and the permission-page projection.
|
||||
|
||||
Revocation invalidates the Agent connections that retain the application's handles, then awaits native cleanup. Reauthorization uses a fresh connection, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Trust model instructions or hide controls.** Retained objects and direct tool calls can still reach native operations. The executor must enforce the decision at dispatch.
|
||||
|
||||
**Persist application names or unsigned paths.** A replacement can reuse both. Durable grants require a verifiable signing identity; task-only access supports unsigned development applications without that promise.
|
||||
**Persist application names or unsigned paths.** A replacement can reuse both. Durable grants require a verifiable signing identity; session access supports unsigned development applications without that promise.
|
||||
|
||||
**Add an approval database or Session control stream.** Profile settings already own user configuration, and User Questions already owns interactive answers.
|
||||
**Add an approval database or Session control stream.** Profile settings already own user configuration, and User Questions already owns interactive answers. The permission UI observes the provider and its owning settings namespace without another authorization store or event channel.
|
||||
|
||||
**Disable permission buttons based on the browser address.** A user may operate the DSH host through screen sharing and complete authorization there. Port forwarding and proxies make the address an unreliable indication of whether the browser and DSH run on the same computer.
|
||||
|
||||
**Use the nearest GUI ancestor as the permission target.** A source-built Desktop app can sit between Terminal and the helper while macOS still attributes Accessibility and Screen Recording to Terminal. Process responsibility supplies the system’s attribution; ancestry alone does not. The optional SPI adds compatibility risk, so lookup failure omits the application instead of guessing.
|
||||
|
||||
**Keep the native context after revocation.** Its application lookup can return an earlier handle and retain window snapshots. Closing the Agent connection discards its native context through existing transport cleanup; preserving unrelated handles would require a native per-application retirement protocol.
|
||||
|
||||
## Consequences
|
||||
|
||||
Revoking one application also invalidates other application handles on the same Agent connection. Those applications require fresh selection; saved grants remain reusable, while task grants expire. Other tasks with independent connections remain unaffected unless they also hold access to the revoked application.
|
||||
Refusing or revoking one application also invalidates other application handles on the same Agent connection. Those applications require fresh selection; session and saved grants remain reusable. Other sessions with independent connections remain unaffected unless they also hold access to the revoked application.
|
||||
|
||||
Explicit enablement may start the shared helper for a read-only permission probe. The helper captures the application identified by macOS process responsibility before readiness releases the launcher, retaining its name and icon after reparenting. The optional native query is dynamically resolved; unavailable attribution uses generic system-prompt guidance. The macOS prompt remains authoritative. Enabling does not request macOS authorization or inspect application content.
|
||||
|
||||
Application decisions and failure results are exercised through policy regressions and the recorded CUA scenario. Desktop binary distribution, signing, and packaged-host permissions remain outside the source-launch workflow.
|
||||
@@ -10,29 +10,38 @@ macOS 权限授权执行宿主，不能表达用户允许操作哪些目标应
|
||||
|
||||
## 决策
|
||||
|
||||
CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。人工决定复用 User Questions 服务，长期许可保存在 provider 的现有 profile 配置中。通用工具审批和操作系统权限仍是独立要求。任意 persistent PTC 代码和另行授权的 shell 工具保留各自的执行权限。
|
||||
CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。人工决定复用 User Questions 服务，长期许可保存在 provider 的现有 profile 配置中。通用工具审批和操作系统权限仍是独立要求。任意 persistent PTC 代码和另行授权的 shell 工具保留各自的执行权限。现有问答界面展示授权选项；执行器拒绝自由文本或未准确选中一个所给选项的回答。权限页图标和名称仅用于展示，不进入保存的身份信息或模型结果。
|
||||
|
||||
长期身份由规范化可执行文件路径和经过验证的签名指定要求组成，以 SHA-256 摘要表示。路径阻止许可跟随应用副本，签名要求区分替换程序，并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得任务授权，许可绑定进程实例和 Agent 连接生命周期。显示名称不用于确定身份。
|
||||
系统权限按钮调用运行 DSH 的电脑上的原生 helper，远程访问时仍可使用。授权地点写在[使用说明](../../../../packages/experimental/client-ui-cua/README.zh.md#use-this-package)中，页面不常驻远程访问提示。
|
||||
|
||||
长期身份由规范化可执行文件路径和经过验证的签名指定要求组成，以 SHA-256 摘要表示。路径阻止许可跟随应用副本，签名要求区分替换程序，并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得会话授权，许可绑定进程实例。显示名称不用于确定身份。
|
||||
|
||||
选择尚未运行的应用时，先准备可执行文件身份和句柄，不立即启动。同一授权决定保护内部启动续接，helper 在启动前再次核对可执行文件，并在启动后绑定实际进程。已签名应用沿用签名指定要求身份；签名不可验证的程序使用本次待启动选择的内容身份，不能生成长期许可。先准备元数据再询问，可以避免启动用户拒绝的应用。
|
||||
|
||||
同一任务对同一应用的选择串行执行，避免未完成的问题产生相互冲突的决定。长期许可的新增在同一写入队列内读取当前配置。拒绝会阻止任务内的再次询问；用户显式恢复的长期许可在执行检查时优先于旧拒绝记录。
|
||||
会话决定按 Session ID 和应用身份保存在内存中。Agent 销毁或 helper 重建会释放原生句柄，但保留授权决定；提供方卸载或 DSH 重启后清除。会话范围可以覆盖用户的后续指令，无须由模型判断一次用户意图何时结束。
|
||||
|
||||
删除长期许可会使保留目标应用句柄的 Agent 连接失效，并取消活动操作；Agent 或提供方销毁时等待对应的原生连接清理。重新授权使用新连接，因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。
|
||||
同一会话对同一应用的选择串行执行，避免未完成的问题产生相互冲突的决定。长期许可的新增和撤销共用写入队列，并在队列内读取当前许可列表。提供方关闭时取消尚在等待 HMR 调度的写入，已开始的写入会在销毁完成前结束。拒绝会阻止再次询问，直到用户允许重询；重新添加的长期许可在执行检查和权限页中均优先于旧拒绝记录。
|
||||
|
||||
撤销使保留目标应用句柄的 Agent 连接失效，并等待原生清理完成。重新授权使用新连接，因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。
|
||||
|
||||
## 考虑过的替代方案
|
||||
|
||||
**依赖模型指引或隐藏控件。** 保留的对象和直接工具调用仍能到达原生操作。执行器必须在派发时检查用户决定。
|
||||
|
||||
**保存应用名称或未签名路径。** 替换程序可以复用二者。长期许可需要可验证的签名身份；任务授权支持未签名的开发应用，但不承诺跨进程复用。
|
||||
**保存应用名称或未签名路径。** 替换程序可以复用二者。长期许可需要可验证的签名身份；会话授权支持未签名的开发应用，但不承诺跨进程复用。
|
||||
|
||||
**新增审批数据库或 Session 控制流。** Profile settings 已负责用户配置，User Questions 已负责交互回答。
|
||||
**新增审批数据库或 Session 控制流。** Profile settings 已负责用户配置，User Questions 已负责交互回答。权限界面查询 provider，并监听其配置 namespace，无须另一份授权存储或事件通道。
|
||||
|
||||
**根据浏览器访问地址禁用权限按钮。** 用户可能通过屏幕共享操作运行 DSH 的电脑，并在该电脑上完成授权。端口转发和代理使访问地址无法可靠判断浏览器与 DSH 是否运行在同一台电脑上。
|
||||
|
||||
**把最近的 GUI 祖先当作授权对象。** 源码构建的 Desktop 可以位于 Terminal 和 helper 之间，但 macOS 仍将辅助功能和屏幕录制归属于 Terminal。责任进程记录提供系统认定的归属，仅凭父进程链无法确定。可选内部接口有兼容性风险，因此查询失败时省略应用信息，不再猜测。
|
||||
|
||||
**撤销后保留原生状态。** 原生应用查找可能返回旧句柄并保留窗口快照。关闭 Agent 连接可以通过现有传输清理释放其原生状态；保留其他应用的句柄则需要新增按应用销毁状态的原生协议。
|
||||
|
||||
## 影响
|
||||
|
||||
撤销一个应用也会使同一 Agent 连接中其他应用的句柄失效。这些应用需要重新选择；长期许可仍可复用，任务许可随之失效。其他任务使用独立连接，除非也持有被撤销应用的许可，否则不受影响。
|
||||
拒绝或撤销一个应用也会使同一 Agent 连接中其他应用的句柄失效。这些应用需要重新选择；会话许可和长期许可仍可复用。其他会话使用独立连接，除非也持有被撤销应用的许可，否则不受影响。
|
||||
|
||||
显式启用可以启动共享 helper 进行只读权限检查。helper 在发布就绪状态、让启动器退出之前，记录 macOS 责任进程对应的应用，因此父进程改变后仍能提供应用名称和图标。可选的本机查询通过动态加载解析，无法取得归属时显示通用系统弹窗指引。授权归属仍以 macOS 提示为准。启用不会申请 macOS 授权或检查应用内容。
|
||||
|
||||
授权决定和失败结果由策略回归测试及 CUA 录制会话验证。桌面安装包的二进制分发、签名和打包宿主权限不属于源码启动流程。
|
||||
@@ -105,6 +105,7 @@
|
||||
"@deepseek-ai/dsh-experimental-terminal-bundle": "workspace:*",
|
||||
"@deepseek-ai/dsh-experimental-session-search": "workspace:*",
|
||||
"@deepseek-ai/dsh-experimental-tool-worktree": "workspace:*",
|
||||
"@deepseek-ai/dsh-cua-bundle": "workspace:*",
|
||||
"@deepseek-ai/dsh-experimental-plugin-market-bundle": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
/** Optional CUA uses existing plugin activation without changing the shared PTC runtime. */
|
||||
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { chromium, type Browser } from 'playwright'
|
||||
import { readProfileManifest } from '@deepseek-ai/dsh-app-boot'
|
||||
import { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type {} from '@deepseek-ai/dsh-ptc-runtime'
|
||||
import type {} from '@deepseek-ai/dsh-agent-preset-registry'
|
||||
import { expect, it } from 'vitest'
|
||||
import { launchWebScaffold, type WebScaffold } from './scaffold.ts'
|
||||
import { connectFreshWorkspaceZh, ZH_BROWSER_LOCALE } from './support.ts'
|
||||
|
||||
it.each([undefined, 'minimal'] as const)('adds CUA to PTC sessions while preserving default %s and runtime configuration', async (selectedDefault) => {
|
||||
const directory = await mkdtemp(join(tmpdir(), 'cua-optin-web-'))
|
||||
let ownedScaffold: WebScaffold | undefined
|
||||
let browser: Browser | undefined
|
||||
try {
|
||||
const patch = join(directory, 'native-fixture.yml')
|
||||
// The executable cannot answer native permission queries, so setup must offer retry in the detail page.
|
||||
await writeFile(patch, JSON.stringify([{ id: 'cua', config: { helperPath: process.execPath } }]))
|
||||
const scaffold = ownedScaffold = await launchWebScaffold({ harnessHome: join(directory, 'home'), extraOverlayPath: [
|
||||
fileURLToPath(new URL('./pin-browse-picker.overlay.yml', import.meta.url)), patch,
|
||||
] })
|
||||
const initialDefault = selectedDefault ?? 'standard'
|
||||
if (selectedDefault !== undefined) await scaffold.ctx.settings.update('agent-preset-registry', { selectedDefault })
|
||||
browser = await chromium.launch()
|
||||
const page = await browser.newPage({ viewport: { width: 1680, height: 1000 }, locale: ZH_BROWSER_LOCALE })
|
||||
await page.goto(scaffold.authenticatedUrl)
|
||||
await connectFreshWorkspaceZh(page, scaffold.workspaceCwd)
|
||||
const originalAgent = scaffold.ctx.agents.list()[0]!
|
||||
const ptcAgent = (await scaffold.ctx.agents.create({ sessionId: SessionId('cua-existing-ptc'),
|
||||
setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx, 'ptc') },
|
||||
})).agent
|
||||
const originalRuntime = scaffold.ctx.ptcRuntime
|
||||
expect(originalRuntime.programFormat).toBe('function-body')
|
||||
const settingsPath = join(scaffold.harnessHome, 'profiles/scaffold/cordis.patch.yml')
|
||||
const before = await readFile(settingsPath, 'utf8')
|
||||
const selected = () => readProfileManifest('scaffold', join(scaffold.harnessHome, 'profiles/scaffold')).dsh?.profile?.bundles
|
||||
await page.getByRole('navigation', { name: '全局面板' }).getByRole('button', { name: '插件', exact: true }).click()
|
||||
const card = page.locator('[data-plugin-package="@deepseek-ai/dsh-cua-bundle"]')
|
||||
const toggle = card.getByRole('switch', { name: '启用 Computer Use', exact: true })
|
||||
expect(await toggle.getAttribute('aria-checked')).toBe('false')
|
||||
expect(await card.textContent()).toContain('请切换到 PTC 模式')
|
||||
await toggle.click()
|
||||
await expect.poll(() => toggle.getAttribute('aria-checked')).toBe('true')
|
||||
const setup = page.getByRole('dialog', { name: '使用 Computer Use 前需要设置', exact: true })
|
||||
await setup.waitFor()
|
||||
expect(await setup.getByText('Computer Use 插件已开启，但暂时无法检测系统权限。请前往插件详情页查看原因并重试。', { exact: true }).count()).toBe(1)
|
||||
await setup.getByRole('button', { name: '稍后', exact: true }).click()
|
||||
expect(selected()).toContain('@deepseek-ai/dsh-cua-bundle')
|
||||
expect(await readFile(settingsPath, 'utf8')).toBe(before)
|
||||
expect(scaffold.ctx.ptcRuntime).toBe(originalRuntime)
|
||||
expect(scaffold.ctx.agentPresets.defaultId).toBe(initialDefault)
|
||||
expect(scaffold.ctx.agents.get(originalAgent.id)).toBe(originalAgent)
|
||||
expect(scaffold.ctx.tools.schemas(originalAgent).some(tool => tool.name === 'cua')).toBe(false)
|
||||
expect(scaffold.ctx.tools.schemas(ptcAgent).some(tool => tool.name === 'cua')).toBe(true)
|
||||
expect((await scaffold.ctx.systemPrompt.assemble({ scope: ptcAgent })).tools.map(tool => tool.name)).toEqual(['run_code'])
|
||||
const fresh = await scaffold.ctx.agents.create({ sessionId: SessionId('cua-new-ptc'),
|
||||
setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx, 'ptc') },
|
||||
})
|
||||
expect(scaffold.ctx.tools.schemas(fresh.agent).some(tool => tool.name === 'cua')).toBe(true)
|
||||
await toggle.click()
|
||||
await expect.poll(() => toggle.getAttribute('aria-checked')).toBe('false')
|
||||
expect(scaffold.ctx.get('computerUse')).toBeUndefined()
|
||||
for (const agent of [ptcAgent, fresh.agent]) expect(scaffold.ctx.tools.schemas(agent).some(tool => tool.name === 'cua')).toBe(false)
|
||||
expect(scaffold.ctx.ptcRuntime).toBe(originalRuntime)
|
||||
expect(await readFile(settingsPath, 'utf8')).toBe(before)
|
||||
await card.getByRole('button', { name: '查看 Computer Use', exact: true }).click()
|
||||
const detailToggle = page.locator('[data-plugin-panel]').getByRole('switch', { name: '启用 Computer Use', exact: true })
|
||||
await detailToggle.click()
|
||||
await expect.poll(() => detailToggle.getAttribute('aria-checked')).toBe('true')
|
||||
const panel = page.locator('[data-cua-settings]')
|
||||
await panel.waitFor()
|
||||
const defaultSetting = panel.getByRole('group', { name: '新会话默认模式', exact: true })
|
||||
await expect.poll(() => defaultSetting.getByRole('button', { name: '设为 PTC', exact: true }).isEnabled()).toBe(true)
|
||||
await defaultSetting.getByRole('button', { name: '设为 PTC', exact: true }).click()
|
||||
await expect.poll(() => scaffold.ctx.agentPresets.defaultId).toBe('ptc')
|
||||
expect(scaffold.ctx.agentPresets.composedPreset(originalAgent.ctx)).toBe(initialDefault)
|
||||
expect(scaffold.ctx.ptcRuntime).toBe(originalRuntime)
|
||||
const confirmed = await scaffold.ctx.agents.create({ sessionId: SessionId('cua-confirmed-default'),
|
||||
setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx) },
|
||||
})
|
||||
expect(scaffold.ctx.tools.schemas(confirmed.agent).some(tool => tool.name === 'cua')).toBe(true)
|
||||
const saved = await readFile(settingsPath, 'utf8')
|
||||
expect(saved).toContain('selectedDefault: ptc')
|
||||
expect(saved).not.toContain('executionMode:')
|
||||
await detailToggle.click()
|
||||
await expect.poll(() => detailToggle.getAttribute('aria-checked')).toBe('false')
|
||||
expect(await readFile(settingsPath, 'utf8')).toBe(saved)
|
||||
expect(scaffold.ctx.ptcRuntime).toBe(originalRuntime)
|
||||
} finally {
|
||||
try { await browser?.close() } finally {
|
||||
try { await ownedScaffold?.close() } finally { await rm(directory, { recursive: true, force: true }) }
|
||||
}
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,139 @@
|
||||
/** Built permission UI talks to the real provider; the isolated helper substitutes only macOS privilege observations. */
|
||||
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { ToolCallId } from '@deepseek-ai/dsh-llm'
|
||||
import type {} from '@deepseek-ai/dsh-user-approval'
|
||||
import type {} from '@deepseek-ai/dsh-agent-preset-registry'
|
||||
import { chromium, type Browser } from 'playwright'
|
||||
import { expect, it, onTestFailed, onTestFinished } from 'vitest'
|
||||
import { captureStableAria, compareOrRefreshGolden, launchWebScaffold, webSnapshotMode, watchConsole, type WebScaffold } from './scaffold.ts'
|
||||
import { connectFreshWorkspace, newEnglishPage, saveFailureShot } from './support.ts'
|
||||
import { installNativeFixture, startFixtureHost } from '../../../packages/experimental/cua-macos/tests/native-fixture.ts'
|
||||
|
||||
const bundle = fileURLToPath(new URL('../../../packages/experimental/cua-bundle', import.meta.url))
|
||||
const expected = fileURLToPath(new URL('./expected/cua-permissions/missing.expected.md', import.meta.url))
|
||||
|
||||
// System-permission onboarding belongs to the macOS-only bundle; rendering and policy units remain portable.
|
||||
it.skipIf(process.platform !== 'darwin')('keeps system requests pending until fresh observation and persists application revocation', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-cua-permissions-web-'))
|
||||
const resources: { scaffold?: WebScaffold; browser?: Browser; closeFixture?: () => Promise<void> } = {}
|
||||
onTestFinished(async () => {
|
||||
try { await resources.browser?.close() } finally {
|
||||
try { await resources.scaffold?.close() } finally {
|
||||
try { await resources.closeFixture?.() } finally { await rm(root, { recursive: true, force: true }) }
|
||||
}
|
||||
}
|
||||
})
|
||||
const helper = await installNativeFixture(root)
|
||||
resources.closeFixture = await startFixtureHost(helper)
|
||||
const state = join(root, 'permissions.json'), audit = join(root, 'helper-events.jsonl')
|
||||
const launchApplication = { name: 'Visual Studio Code.app', path: '/Applications/Visual Studio Code.app',
|
||||
icon: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAACXBIWXMAAAPoAAAD6AG1e1JrAAAADElEQVQImWNgZGIGAAAOAAeCcsnOAAAAAElFTkSuQmCC' }
|
||||
await writeFile(state, JSON.stringify({ accessibility: false, screenRecording: false, launchApplication }))
|
||||
await writeFile(join(root, 'observe-helper'), '')
|
||||
const fixtureBundle = join(root, 'fixture-bundle')
|
||||
await mkdir(fixtureBundle)
|
||||
await writeFile(join(fixtureBundle, 'package.json'), JSON.stringify({ name: 'cua-permission-fixture', version: '1.0.0',
|
||||
dsh: { bundle: { patch: 'cordis.patch.yml' } } }))
|
||||
await writeFile(join(fixtureBundle, 'cordis.patch.yml'), `- id: cua\n config: ${JSON.stringify({ helperPath: helper })}\n`)
|
||||
const scaffold = await launchWebScaffold({
|
||||
profile: { packages: [{ dir: bundle, enabled: true }, { dir: fixtureBundle, enabled: true }] },
|
||||
})
|
||||
resources.scaffold = scaffold
|
||||
await scaffold.ctx.settings.mutate('cua', [{ op: 'set', path: ['allowedApps'],
|
||||
value: [{ id: 'a'.repeat(64), name: 'Fixture', executablePath: '/Applications/Fixture.app' }] }])
|
||||
const browser = await chromium.launch()
|
||||
resources.browser = browser
|
||||
const page = await newEnglishPage(browser), tripwire = watchConsole(page)
|
||||
onTestFailed(() => saveFailureShot(page, 'web-e2e-cua-permissions'))
|
||||
await page.goto(scaffold.authenticatedUrl)
|
||||
await connectFreshWorkspace(page, scaffold.workspaceCwd)
|
||||
const agent = scaffold.ctx.agents.list()[0]!
|
||||
await scaffold.ctx.agentPresets.select(agent, 'ptc')
|
||||
scaffold.ctx.approval.setPolicy(agent, 'never')
|
||||
await scaffold.ctx.settings.mutate('cua', [{ op: 'set', path: ['allowedApps'], value: [] }])
|
||||
// Application consent remains independent of the session's tool-approval policy.
|
||||
const grant = scaffold.ctx.tools.execute({ agent, signal: new AbortController().signal,
|
||||
callId: ToolCallId('cua-app-approval'), name: 'run_code',
|
||||
arguments: { code: 'return await tools.cua({ method: "getApp", params: { selector: "Fixture" } })',
|
||||
description: 'Request application access' } })
|
||||
const approval = page.locator('[data-question-key]')
|
||||
await Promise.race([approval.waitFor(), grant.then((result) => {
|
||||
throw new Error(`Application selection completed before consent: ${JSON.stringify(result)}`)
|
||||
})])
|
||||
expect(await approval.textContent()).toContain('Fixture')
|
||||
await approval.getByRole('radio', { name: 'Always allow', exact: true }).click()
|
||||
await approval.getByRole('button', { name: 'Submit', exact: true }).click()
|
||||
expect((await grant).isError).toBe(false)
|
||||
await approval.waitFor({ state: 'hidden' })
|
||||
await page.getByRole('button', { name: 'Plugins', exact: true }).click()
|
||||
const card = page.locator('[data-plugin-package="@deepseek-ai/dsh-cua-bundle"]')
|
||||
expect(await card.getByRole('button', { name: /Accessibility permission|Screen Recording permission/ }).count()).toBe(0)
|
||||
await page.setViewportSize({ width: 1680, height: 600 })
|
||||
const plugins = page.locator('[data-plugin-panel]')
|
||||
const openDetails = page.getByRole('button', { name: 'View Computer Use', exact: true })
|
||||
await openDetails.scrollIntoViewIfNeeded()
|
||||
expect(await plugins.evaluate(element => element.scrollTop)).toBeGreaterThan(0)
|
||||
await openDetails.click()
|
||||
const panel = page.locator('[data-cua-settings]')
|
||||
const accessibility = panel.getByRole('group', { name: 'Accessibility', exact: true })
|
||||
await accessibility.getByText('Not allowed', { exact: true }).waitFor()
|
||||
expect(await panel.getByRole('region', { name: 'Authorize Computer Use', exact: true }).getByRole('group').count()).toBe(3)
|
||||
expect(await panel.getByRole('region', { name: 'Manage applications Computer Use can access', exact: true }).count()).toBe(1)
|
||||
await expect.poll(() => panel.locator('li img').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBe(1)
|
||||
expect(await panel.getByText('/fixture/app', { exact: true }).count()).toBe(0)
|
||||
expect(await accessibility.getByText('Visual Studio Code.app', { exact: true }).count()).toBe(1)
|
||||
expect(await accessibility.textContent()).toContain('In System Settings, allow')
|
||||
expect(await accessibility.textContent()).toContain('to read interface controls, click, and type.')
|
||||
expect(await panel.getByRole('group', { name: 'Screen Recording', exact: true }).getByText('Visual Studio Code.app', { exact: true }).count()).toBe(1)
|
||||
await expect.poll(() => accessibility.locator('img').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBe(1)
|
||||
await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false')
|
||||
await compareOrRefreshGolden(expected, await captureStableAria(page, '[data-cua-settings]', scaffold.workspaceCwd), webSnapshotMode())
|
||||
try {
|
||||
for (const width of [1680, 390]) {
|
||||
await page.setViewportSize({ width, height: 1000 })
|
||||
for (const colorScheme of ['light', 'dark'] as const) {
|
||||
await page.emulateMedia({ colorScheme })
|
||||
await expect.poll(() => page.evaluate(() => document.body.hasAttribute('data-ds-dark-theme'))).toBe(colorScheme === 'dark')
|
||||
expect(await panel.evaluate(element => element.scrollWidth <= element.clientWidth)).toBe(true)
|
||||
if (webSnapshotMode() === 'refresh') {
|
||||
const directory = fileURLToPath(new URL('../../../.artifacts', import.meta.url))
|
||||
await mkdir(directory, { recursive: true })
|
||||
const path = join(directory, `cua-permission-design-${process.pid}-${width}-${colorScheme}.png`)
|
||||
await panel.screenshot({ path, animations: 'disabled' })
|
||||
console.log(`CUA permission design: ${path}`)
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await page.emulateMedia({ colorScheme: null })
|
||||
await page.setViewportSize({ width: 1680, height: 1000 })
|
||||
}
|
||||
await accessibility.getByRole('button', { name: 'Request permission', exact: true }).click()
|
||||
await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false')
|
||||
expect(await accessibility.getByText('Not allowed', { exact: true }).count()).toBe(1)
|
||||
await writeFile(state, JSON.stringify({ accessibility: true, screenRecording: false, launchApplication }))
|
||||
// Headless pages retain document focus; model returning from System Settings at the window-event boundary.
|
||||
await page.evaluate(() => { window.dispatchEvent(new Event('focus')) })
|
||||
await accessibility.getByText('Allowed', { exact: true }).waitFor()
|
||||
const screen = panel.getByRole('group', { name: 'Screen Recording', exact: true })
|
||||
await screen.getByRole('button', { name: 'Open System Settings', exact: true }).click()
|
||||
await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false')
|
||||
await panel.getByRole('button', { name: 'Revoke', exact: true }).click()
|
||||
await panel.getByRole('heading', { name: 'Always-allowed applications', exact: true }).waitFor({ state: 'hidden' })
|
||||
await panel.getByRole('heading', { name: 'Applications not allowed', exact: true }).waitFor()
|
||||
await panel.getByRole('button', { name: 'Allow asking again', exact: true }).click()
|
||||
await panel.getByText('Fixture', { exact: true }).waitFor({ state: 'hidden' })
|
||||
const saved = await readFile(join(scaffold.harnessHome, 'profiles/scaffold/cordis.patch.yml'), 'utf8')
|
||||
expect(saved).toContain('allowedApps: []')
|
||||
const events = (await readFile(audit, 'utf8')).trim().split('\n').map(line => JSON.parse(line) as { event: string; method: string; params: object })
|
||||
.filter(record => record.event === 'call').map(({ method, params }) => ({ method, params }))
|
||||
expect(events).toContainEqual({ method: 'requestPermission', params: { permission: 'accessibility' } })
|
||||
expect(events.filter(event => event.method === 'openPermissionSettings')).toEqual([
|
||||
{ method: 'openPermissionSettings', params: { permission: 'screenRecording' } },
|
||||
])
|
||||
expect(events.every(event => ['permissionStatus', 'requestPermission', 'openPermissionSettings', 'applicationIcons', 'getApp'].includes(event.method))).toBe(true)
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
}, 60_000)
|
||||
@@ -0,0 +1,26 @@
|
||||
- region "Authorize Computer Use":
|
||||
- heading "Authorize Computer Use" [level=3]
|
||||
- group "Default mode for new sessions":
|
||||
- strong: Default mode for new sessions
|
||||
- paragraph: Computer Use is available only in PTC mode.
|
||||
- button "Set to PTC"
|
||||
- group "Accessibility":
|
||||
- strong: Accessibility
|
||||
- paragraph: In System Settings, allow Visual Studio Code.app to read interface controls, click, and type.
|
||||
- text: Not allowed
|
||||
- button "Request permission"
|
||||
- button "Open System Settings"
|
||||
- group "Screen Recording":
|
||||
- strong: Screen Recording
|
||||
- paragraph: In System Settings, allow Visual Studio Code.app to capture application windows.
|
||||
- text: Not allowed
|
||||
- button "Request permission"
|
||||
- button "Open System Settings"
|
||||
- paragraph: Enable the application in System Settings. Follow the macOS prompt if it names another application.
|
||||
- region "Manage applications Computer Use can access":
|
||||
- heading "Manage applications Computer Use can access" [level=3]
|
||||
- heading "Always-allowed applications" [level=4]
|
||||
- list:
|
||||
- listitem:
|
||||
- strong: Fixture
|
||||
- button "Revoke"
|
||||
@@ -38,6 +38,10 @@
|
||||
- button "查看 开发者工具": 开发者工具
|
||||
- text: 实验性 查看调试会话原始数据、聊天消息分组数据，以及调试 NodeJS 后端
|
||||
- switch "启用 开发者工具"
|
||||
- listitem:
|
||||
- button "查看 Computer Use": Computer Use
|
||||
- text: 实验性 目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。
|
||||
- switch "启用 Computer Use"
|
||||
- heading "已安装插件" [level=3]
|
||||
- list:
|
||||
- listitem:
|
||||
|
||||
@@ -38,6 +38,10 @@
|
||||
- button "查看 开发者工具": 开发者工具
|
||||
- text: 实验性 查看调试会话原始数据、聊天消息分组数据，以及调试 NodeJS 后端
|
||||
- switch "启用 开发者工具"
|
||||
- listitem:
|
||||
- button "查看 Computer Use": Computer Use
|
||||
- text: 实验性 目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。
|
||||
- switch "启用 Computer Use"
|
||||
- heading "已安装插件" [level=3]
|
||||
- list:
|
||||
- listitem:
|
||||
|
||||
@@ -38,6 +38,10 @@
|
||||
- button "查看 开发者工具": 开发者工具
|
||||
- text: 实验性 查看调试会话原始数据、聊天消息分组数据，以及调试 NodeJS 后端
|
||||
- switch "启用 开发者工具"
|
||||
- listitem:
|
||||
- button "查看 Computer Use": Computer Use
|
||||
- text: 实验性 目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。
|
||||
- switch "启用 Computer Use"
|
||||
- heading "已安装插件" [level=3]
|
||||
- list:
|
||||
- listitem:
|
||||
|
||||
@@ -6,7 +6,7 @@ Focus at 500ms:
|
||||
- tooltip "刷新"
|
||||
failure: pending
|
||||
- button "刷新" [disabled]
|
||||
{"busy":"true","spinners":1,"cards":11,"skeletons":0,"alerts":[],"retry":0,"toasts":[]}
|
||||
{"busy":"true","spinners":1,"cards":12,"skeletons":0,"alerts":[],"retry":0,"toasts":[]}
|
||||
failure: settled
|
||||
- button "刷新"
|
||||
{"busy":"false","spinners":0,"cards":11,"skeletons":0,"alerts":[],"retry":0,"toasts":["刷新失败，请重试"]}
|
||||
{"busy":"false","spinners":0,"cards":12,"skeletons":0,"alerts":[],"retry":0,"toasts":["刷新失败，请重试"]}
|
||||
@@ -681,15 +681,23 @@ async function toggleOptionalBundles(page: Page): Promise<void> {
|
||||
}
|
||||
})
|
||||
expect(result.optional.map(entry => entry.name).sort()).toEqual([...OPTIONAL_BUNDLES].sort())
|
||||
expect(result.optional.filter(entry => entry.enabled || entry.availability !== 'installation' || entry.error !== null)).toEqual([])
|
||||
expect(result.optional.filter(entry => entry.enabled || entry.availability !== 'installation')).toEqual([])
|
||||
expect(result.optional.filter(entry => entry.error !== null)).toEqual([
|
||||
expect.objectContaining({ name: '@deepseek-ai/dsh-cua-bundle', error: { code: 'unsupported-platform' } }),
|
||||
])
|
||||
// Metadata resolves through the Worker's Node internal modules: every optional bundle shows its title and icon.
|
||||
expect(result.optional.filter(entry => entry.metaError !== null || entry.title !== 'object' || entry.icon !== 'string')).toEqual([])
|
||||
expect(result.inventoryErrors).toEqual([])
|
||||
console.info(`preview boot: optional bundle toggles ${JSON.stringify(result.toggles.map(({ name, enable, disable, error }) => ({ name, enable, disable, error })))}`)
|
||||
expect(result.toggles.filter(toggle => toggle.enable !== 'applied' || toggle.disable !== 'applied')
|
||||
expect(result.toggles.find(toggle => toggle.name === '@deepseek-ai/dsh-cua-bundle')).toMatchObject({
|
||||
enable: 'failed', disable: 'applied', error: { code: 'unsupported-platform' },
|
||||
changed: 0, inactive: [], restored: true,
|
||||
})
|
||||
const supported = result.toggles.filter(toggle => toggle.name !== '@deepseek-ai/dsh-cua-bundle')
|
||||
expect(supported.filter(toggle => toggle.enable !== 'applied' || toggle.disable !== 'applied')
|
||||
.map(({ name, enable, disable }) => ({ name, enable, disable }))).toEqual([])
|
||||
// Each bundle changes Loader rows, every enabled row it changes runs, and switching it off restores the prior rows.
|
||||
expect(result.toggles.filter(toggle => toggle.changed === 0 || toggle.inactive.length > 0 || !toggle.restored)
|
||||
expect(supported.filter(toggle => toggle.changed === 0 || toggle.inactive.length > 0 || !toggle.restored)
|
||||
.map(({ name, changed, inactive, restored }) => ({ name, changed, inactive, restored }))).toEqual([])
|
||||
const pluginsWith = (bundle: string): Array<{ moduleName: string; enabled: boolean; fiberPhase: string | null }> =>
|
||||
result.toggles.find(toggle => toggle.name === bundle)?.plugins ?? []
|
||||
|
||||
@@ -82,6 +82,9 @@
|
||||
"tests/plugin-config.e2e.ts",
|
||||
"tests/input-plugin-extensions.e2e.ts",
|
||||
"tests/plugin-manager.e2e.ts",
|
||||
"tests/cua-optin.e2e.ts",
|
||||
"tests/bundle-platform.e2e.ts",
|
||||
"tests/cua-permissions.e2e.ts",
|
||||
"tests/schedule-header.e2e.ts",
|
||||
"tests/worktree-opt-in.e2e.ts",
|
||||
"tests/session-inspector.e2e.ts",
|
||||
|
||||
@@ -6,5 +6,5 @@
|
||||
en: 41d772192075e133
|
||||
zh: 5ef3da69b4de1d8a
|
||||
/capability-seams-and-core-services:
|
||||
en: f209321ab0da4f90
|
||||
zh: 4af5c135eec51ce1
|
||||
en: b360360db9a3f6a4
|
||||
zh: d14808d61000bdff
|
||||
@@ -52,6 +52,9 @@ flowchart LR
|
||||
svc_computerUse["ctx.computerUse<br/>Computer-use provider registration"]
|
||||
pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"]
|
||||
pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"]
|
||||
pkg_cua["cua"]
|
||||
svc_cuaAccess["ctx.cuaAccess<br/>Computer Use permission controls"]
|
||||
pkg_client_ui_cua["client-ui-cua"]
|
||||
pkg_office_to_pdf["office-to-pdf"]
|
||||
svc_officeToPdf["ctx.officeToPdf<br/>Office to PDF conversion"]
|
||||
pkg_client_ui_sidebar_documentpreview["client-ui-sidebar-documentpreview"]
|
||||
@@ -334,6 +337,7 @@ flowchart LR
|
||||
pkg_cordis_host_runner --> svc_dynamicCordisRunner
|
||||
pkg_credentials --> svc_credentials
|
||||
pkg_credentials_local --> svc_credentials
|
||||
pkg_cua --> svc_cuaAccess
|
||||
pkg_deepseek_account --> svc_deepseekAccount
|
||||
pkg_deepseek_account_platform --> svc_deepseekAccount
|
||||
pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions
|
||||
@@ -484,6 +488,7 @@ flowchart LR
|
||||
svc_credentials --> pkg_api_settings_controller
|
||||
svc_credentials --> pkg_llm_deepseek
|
||||
svc_credentials --> pkg_llm_pi_ai
|
||||
svc_cuaAccess --> pkg_client_ui_cua
|
||||
svc_deepseekAccount --> pkg_api_account_controller
|
||||
svc_deepseekAccount --> pkg_llm_deepseek
|
||||
svc_deepseekLlmApiExtensions --> pkg_llm_deepseek
|
||||
@@ -628,6 +633,7 @@ flowchart LR
|
||||
| `ctx.mcpResources` | `seam` | [`mcp-resources`](../packages/mcp/mcp-resources) | [`mcp-client`](../packages/mcp/mcp-client) | [`mcp-resources`](../packages/mcp/mcp-resources) | - | Connection-owned providers serve shared resource tools in the calling agent scope. |
|
||||
| `ctx.browserUse` | `seam` | [`browser-use`](../packages/browser-use/browser-use) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | - | One provider-owned name per service instance. Providers own their tools and browser resources per live Session; the shared service has no browser operation API. |
|
||||
| `ctx.computerUse` | `seam` | [`computer-use`](../packages/computer-use/computer-use) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | - | One provider-owned name per service instance. Each provider also owns its model tools; the service has no common action API, runtime selection, or Session workflow lock. |
|
||||
| `ctx.cuaAccess` | `core` | [`cua`](../packages/experimental/cua) | - | [`client-ui-cua`](../packages/experimental/client-ui-cua) | - | The native provider owns application consent and cancellation; the optional Client page queries and changes those decisions through its Remote. |
|
||||
| `ctx.officeToPdf` | `core` | [`office-to-pdf`](../packages/document/office-to-pdf) | - | [`client-ui-sidebar-documentpreview`](../packages/client/ui-sidebar-documentpreview) | - | Authorized Office bytes are converted on the Host using the declared native target engine, or Node WASM when no native target is declared. |
|
||||
| `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`api-session-controller`](../packages/api/session-controller), [`tool-fs`](../packages/fs/tool-fs), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-deepseek`](../packages/llm/llm-deepseek) | - | The host commits accepted images before session events; provider adapters resolve authorized durable references into provider-native content. |
|
||||
| `ctx.fileUploads` | `core` | [`client-file-upload`](../packages/client/file-upload) | - | [`api-session-controller`](../packages/api/session-controller) | - | Owns streaming intake, durable storage, and staged receipt lifetime; the Session controller binds receipts to accepted submissions. |
|
||||
|
||||
@@ -54,6 +54,9 @@ flowchart LR
|
||||
svc_computerUse["ctx.computerUse<br/>Computer-use provider registration"]
|
||||
pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"]
|
||||
pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"]
|
||||
pkg_cua["cua"]
|
||||
svc_cuaAccess["ctx.cuaAccess<br/>Computer Use permission controls"]
|
||||
pkg_client_ui_cua["client-ui-cua"]
|
||||
pkg_office_to_pdf["office-to-pdf"]
|
||||
svc_officeToPdf["ctx.officeToPdf<br/>Office to PDF conversion"]
|
||||
pkg_client_ui_sidebar_documentpreview["client-ui-sidebar-documentpreview"]
|
||||
@@ -336,6 +339,7 @@ flowchart LR
|
||||
pkg_cordis_host_runner --> svc_dynamicCordisRunner
|
||||
pkg_credentials --> svc_credentials
|
||||
pkg_credentials_local --> svc_credentials
|
||||
pkg_cua --> svc_cuaAccess
|
||||
pkg_deepseek_account --> svc_deepseekAccount
|
||||
pkg_deepseek_account_platform --> svc_deepseekAccount
|
||||
pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions
|
||||
@@ -486,6 +490,7 @@ flowchart LR
|
||||
svc_credentials --> pkg_api_settings_controller
|
||||
svc_credentials --> pkg_llm_deepseek
|
||||
svc_credentials --> pkg_llm_pi_ai
|
||||
svc_cuaAccess --> pkg_client_ui_cua
|
||||
svc_deepseekAccount --> pkg_api_account_controller
|
||||
svc_deepseekAccount --> pkg_llm_deepseek
|
||||
svc_deepseekLlmApiExtensions --> pkg_llm_deepseek
|
||||
@@ -630,6 +635,7 @@ flowchart LR
|
||||
| `ctx.mcpResources` | `seam` | [`mcp-resources`](../packages/mcp/mcp-resources) | [`mcp-client`](../packages/mcp/mcp-client) | [`mcp-resources`](../packages/mcp/mcp-resources) | - | 连接所有者提供的操作在调用 agent 的作用域内服务于共享资源工具。 |
|
||||
| `ctx.browserUse` | `seam` | [`browser-use`](../packages/browser-use/browser-use) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | - | 每个服务实例注册一个提供方拥有的名称。提供方按实时 Session 拥有自己的工具与浏览器资源；共享服务不提供浏览器操作 API。 |
|
||||
| `ctx.computerUse` | `seam` | [`computer-use`](../packages/computer-use/computer-use) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | - | 每个服务实例只注册一个提供方自定的名称。各提供方也拥有自己的模型工具；服务不提供通用操作 API、运行时选择或 Session 流程锁。 |
|
||||
| `ctx.cuaAccess` | `core` | [`cua`](../packages/experimental/cua) | - | [`client-ui-cua`](../packages/experimental/client-ui-cua) | - | 原生 provider 负责应用授权和取消操作；可选的 Client 权限页面通过 Remote 查询和修改授权。 |
|
||||
| `ctx.officeToPdf` | `core` | [`office-to-pdf`](../packages/document/office-to-pdf) | - | [`client-ui-sidebar-documentpreview`](../packages/client/ui-sidebar-documentpreview) | - | 已授权的 Office 字节在宿主上使用已声明的原生目标引擎转换；未声明原生目标时使用 Node WASM。 |
|
||||
| `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`api-session-controller`](../packages/api/session-controller), [`tool-fs`](../packages/fs/tool-fs), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-deepseek`](../packages/llm/llm-deepseek) | - | 宿主会在会话事件之前提交已接受的图片；提供方适配器将已授权的持久引用解析为提供方原生内容。 |
|
||||
| `ctx.fileUploads` | `core` | [`client-file-upload`](../packages/client/file-upload) | - | [`api-session-controller`](../packages/api/session-controller) | - | 负责流式接收、持久存储和暂存回执生命周期；Session Controller 将回执绑定到已接受的提交。 |
|
||||
|
||||
@@ -2482,7 +2482,7 @@ export interface PlanModeConfig {
|
||||
## `@deepseek-ai/dsh-plugin-manager`
|
||||
|
||||
- `inject`: `loader` · `profileContext`
|
||||
- `source`: [`packages/boot/plugin-manager/src/index.ts:42`](../packages/boot/plugin-manager/src/index.ts)
|
||||
- `source`: [`packages/boot/plugin-manager/src/index.ts:43`](../packages/boot/plugin-manager/src/index.ts)
|
||||
|
||||
```ts config-catalog
|
||||
/** The pnpm executable, registries, and limits for diagnostics, lookups and connection checks. */
|
||||
@@ -4684,6 +4684,7 @@ These load from a `cordis.yml` entry with no `config:` block; they declare no co
|
||||
| `@deepseek-ai/dsh-client-ui-commands` | — | [`packages/client/ui-commands/src/index.ts`](../packages/client/ui-commands/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-conversation` | — | [`packages/client/ui-conversation/src/index.ts`](../packages/client/ui-conversation/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-cordis` | — | [`packages/extensions/ui-cordis/src/index.ts`](../packages/extensions/ui-cordis/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-cua` | — | [`packages/experimental/client-ui-cua/src/index.ts`](../packages/experimental/client-ui-cua/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-deepseek-model-label-1017` | — | [`packages/client/ui-deepseek-model-label-1017/src/index.ts`](../packages/client/ui-deepseek-model-label-1017/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-deliverables` | `systemPrompt` · `connection` · `sessionQuery` · `sessionController` · `workspaceFiles` · `fs` · `sandboxPolicy` · `workspaceChanges` | [`packages/client/ui-deliverables/src/index.ts`](../packages/client/ui-deliverables/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-directory-picker-browse` | — | [`packages/client/ui-directory-picker-browse/src/index.ts`](../packages/client/ui-directory-picker-browse/src/index.ts) |
|
||||
@@ -4821,6 +4822,7 @@ Imported as libraries by other packages; a `cordis.yml` cannot load them.
|
||||
| `@deepseek-ai/dsh-client-web` | — | [`packages/client/web/src/index.ts`](../packages/client/web/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-cmdline` | — | [`packages/boot/cmdline/src/index.ts`](../packages/boot/cmdline/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-cua-bundle` | — | [`packages/experimental/cua-bundle/src/index.ts`](../packages/experimental/cua-bundle/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-deque` | — | [`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-experimental-agent-team-profile` | — | [`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-experimental-badge-skill-bundle` | — | [`packages/experimental/badge-skill-bundle/src/index.ts`](../packages/experimental/badge-skill-bundle/src/index.ts) |
|
||||
|
||||
@@ -2484,7 +2484,7 @@ export interface PlanModeConfig {
|
||||
## `@deepseek-ai/dsh-plugin-manager`
|
||||
|
||||
- `inject`: `loader` · `profileContext`
|
||||
- `source`: [`packages/boot/plugin-manager/src/index.ts:42`](../packages/boot/plugin-manager/src/index.ts)
|
||||
- `source`: [`packages/boot/plugin-manager/src/index.ts:43`](../packages/boot/plugin-manager/src/index.ts)
|
||||
|
||||
```ts config-catalog
|
||||
/** The pnpm executable, registries, and limits for diagnostics, lookups and connection checks. */
|
||||
@@ -4686,6 +4686,7 @@ export interface Config {
|
||||
| `@deepseek-ai/dsh-client-ui-commands` | — | [`packages/client/ui-commands/src/index.ts`](../packages/client/ui-commands/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-conversation` | — | [`packages/client/ui-conversation/src/index.ts`](../packages/client/ui-conversation/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-cordis` | — | [`packages/extensions/ui-cordis/src/index.ts`](../packages/extensions/ui-cordis/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-cua` | — | [`packages/experimental/client-ui-cua/src/index.ts`](../packages/experimental/client-ui-cua/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-deepseek-model-label-1017` | — | [`packages/client/ui-deepseek-model-label-1017/src/index.ts`](../packages/client/ui-deepseek-model-label-1017/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-deliverables` | `systemPrompt` · `connection` · `sessionQuery` · `sessionController` · `workspaceFiles` · `fs` · `sandboxPolicy` · `workspaceChanges` | [`packages/client/ui-deliverables/src/index.ts`](../packages/client/ui-deliverables/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-client-ui-directory-picker-browse` | — | [`packages/client/ui-directory-picker-browse/src/index.ts`](../packages/client/ui-directory-picker-browse/src/index.ts) |
|
||||
@@ -4823,6 +4824,7 @@ export interface Config {
|
||||
| `@deepseek-ai/dsh-client-web` | — | [`packages/client/web/src/index.ts`](../packages/client/web/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-cmdline` | — | [`packages/boot/cmdline/src/index.ts`](../packages/boot/cmdline/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-cua-bundle` | — | [`packages/experimental/cua-bundle/src/index.ts`](../packages/experimental/cua-bundle/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-deque` | — | [`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-experimental-agent-team-profile` | — | [`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts) |
|
||||
| `@deepseek-ai/dsh-experimental-badge-skill-bundle` | — | [`packages/experimental/badge-skill-bundle/src/index.ts`](../packages/experimental/badge-skill-bundle/src/index.ts) |
|
||||
|
||||
@@ -9,9 +9,9 @@ This matrix shows which packages dispatch each harness-owned event and which pac
|
||||
| Event | Mode | Declared in | Dispatchers | Listeners |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `agent-loop/config-start-failed` | `emit` | [`packages/core/agent-loop/src/index.ts:249`](../packages/core/agent-loop/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`events.dispatch`) | - |
|
||||
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:69`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | `remotes` |
|
||||
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:69`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | [`cua`](../packages/experimental/cua), `remotes` |
|
||||
| `agent/assistant-stream` | `emit` | [`packages/core/agent/src/runtime-types.ts:363`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`headless`](../packages/bundle/headless), `session-controller` |
|
||||
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
|
||||
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`cua`](../packages/experimental/cua), [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
|
||||
| `agent/disposed` | `emit` | [`packages/core/agent/src/runtime-types.ts:270`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`events.dispatch`) | [`agent-loop`](../packages/core/agent-loop), `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal-round-driver`](../packages/goal/goal-round-driver), `session-controller`, [`subagent`](../packages/subagent/subagent), `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
|
||||
| `agent/error` | `emit` | [`packages/core/agent/src/runtime-types.ts:393`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`deepseek-account`](../packages/credentials/deepseek-account), [`goal-round-driver`](../packages/goal/goal-round-driver), [`headless`](../packages/bundle/headless), `server`, `session-controller`, [`session-telemetry`](../packages/session/session-telemetry), [`subagent`](../packages/subagent/subagent) |
|
||||
| `agent/inbox/claimed` | `emit` | [`packages/core/agent/src/runtime-types.ts:299`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`goal-round-driver`](../packages/goal/goal-round-driver), [`subagent`](../packages/subagent/subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`user-questions`](../packages/interaction/user-questions) |
|
||||
|
||||
@@ -11,9 +11,9 @@
|
||||
| Event | Mode | Declared in | Dispatchers | Listeners |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `agent-loop/config-start-failed` | `emit` | [`packages/core/agent-loop/src/index.ts:249`](../packages/core/agent-loop/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`events.dispatch`) | - |
|
||||
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:69`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | `remotes` |
|
||||
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:69`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | [`cua`](../packages/experimental/cua), `remotes` |
|
||||
| `agent/assistant-stream` | `emit` | [`packages/core/agent/src/runtime-types.ts:363`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`headless`](../packages/bundle/headless), `session-controller` |
|
||||
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
|
||||
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`cua`](../packages/experimental/cua), [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
|
||||
| `agent/disposed` | `emit` | [`packages/core/agent/src/runtime-types.ts:270`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`events.dispatch`) | [`agent-loop`](../packages/core/agent-loop), `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal-round-driver`](../packages/goal/goal-round-driver), `session-controller`, [`subagent`](../packages/subagent/subagent), `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
|
||||
| `agent/error` | `emit` | [`packages/core/agent/src/runtime-types.ts:393`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`deepseek-account`](../packages/credentials/deepseek-account), [`goal-round-driver`](../packages/goal/goal-round-driver), [`headless`](../packages/bundle/headless), `server`, `session-controller`, [`session-telemetry`](../packages/session/session-telemetry), [`subagent`](../packages/subagent/subagent) |
|
||||
| `agent/inbox/claimed` | `emit` | [`packages/core/agent/src/runtime-types.ts:299`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`goal-round-driver`](../packages/goal/goal-round-driver), [`subagent`](../packages/subagent/subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`user-questions`](../packages/interaction/user-questions) |
|
||||
|
||||
+15
-9
@@ -246,7 +246,9 @@ flowchart TD
|
||||
pkg_office_to_pdf["office-to-pdf"]
|
||||
end
|
||||
subgraph group_experimental["packages/experimental"]
|
||||
pkg_client_ui_cua["client-ui-cua"]
|
||||
pkg_cua["cua"]
|
||||
pkg_cua_bundle["cua-bundle"]
|
||||
pkg_experimental_agent_team["experimental-agent-team"]
|
||||
pkg_experimental_agent_team_profile["experimental-agent-team-profile"]
|
||||
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
|
||||
@@ -905,14 +907,6 @@ flowchart TD
|
||||
pkg_workspace_changes --> pkg_session
|
||||
pkg_workspace_changes --> pkg_subprocess
|
||||
pkg_workspace_changes --> pkg_tools
|
||||
pkg_cua --> pkg_agent
|
||||
pkg_cua --> pkg_attachment
|
||||
pkg_cua --> pkg_computer_use
|
||||
pkg_cua --> pkg_llm
|
||||
pkg_cua --> pkg_subprocess
|
||||
pkg_cua --> pkg_system_prompt
|
||||
pkg_cua --> pkg_tools
|
||||
pkg_cua --> pkg_user_questions
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_agent
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_browser_use
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_system_prompt
|
||||
@@ -1076,6 +1070,16 @@ flowchart TD
|
||||
pkg_api_settings_controller --> pkg_settings
|
||||
pkg_api_settings_controller --> pkg_typert_protocol
|
||||
pkg_web_app --> pkg_shell_env
|
||||
pkg_cua --> pkg_agent
|
||||
pkg_cua --> pkg_agent_preset_registry
|
||||
pkg_cua --> pkg_attachment
|
||||
pkg_cua --> pkg_computer_use
|
||||
pkg_cua --> pkg_llm
|
||||
pkg_cua --> pkg_scope
|
||||
pkg_cua --> pkg_subprocess
|
||||
pkg_cua --> pkg_system_prompt
|
||||
pkg_cua --> pkg_tools
|
||||
pkg_cua --> pkg_user_questions
|
||||
pkg_experimental_auto_review --> pkg_agent
|
||||
pkg_experimental_auto_review --> pkg_agent_instructions
|
||||
pkg_experimental_auto_review --> pkg_llm
|
||||
@@ -1540,6 +1544,8 @@ flowchart TD
|
||||
| [`client-web`](../packages/client/web) | `client` | — |
|
||||
| [`credentials`](../packages/credentials/credentials) | `credentials` | — |
|
||||
| [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — |
|
||||
| [`client-ui-cua`](../packages/experimental/client-ui-cua) | `experimental` | — |
|
||||
| [`cua-bundle`](../packages/experimental/cua-bundle) | `experimental` | — |
|
||||
| [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — |
|
||||
| [`experimental-badge-skill-bundle`](../packages/experimental/badge-skill-bundle) | `experimental` | — |
|
||||
| [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | — |
|
||||
@@ -1704,7 +1710,6 @@ flowchart TD
|
||||
| [`file-reference-local`](../packages/context/file-reference-local) | `context` | [`agent`](../packages/core/agent), [`file-reference`](../packages/context/file-reference), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`tool-present`](../packages/deliverables/tool-present) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`session-projection`](../packages/session/session-projection), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) |
|
||||
| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) |
|
||||
| [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |
|
||||
@@ -1737,6 +1742,7 @@ flowchart TD
|
||||
| [`session-query`](../packages/session-query/session-query) | `session-query` | [`brand`](../packages/util/brand), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`session-persistence`](../packages/session/session-persistence), [`session-projection`](../packages/session/session-projection), [`session-projection-cache`](../packages/session/session-projection-cache), [`session-title`](../packages/session/session-title), [`tool-todo`](../packages/todo/tool-todo) |
|
||||
| [`api-settings-controller`](../packages/api/settings-controller) | `api` | [`agent-preset-registry`](../packages/preset/agent-preset-registry), [`credentials`](../packages/credentials/credentials), [`native-command`](../packages/util/native-command), [`session`](../packages/core/session), [`settings`](../packages/settings/settings), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`web-app`](../packages/bundle/web-app) | `bundle` | [`shell-env`](../packages/shell/shell-env) |
|
||||
| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`agent-preset-registry`](../packages/preset/agent-preset-registry), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) |
|
||||
| [`experimental-auto-review`](../packages/experimental/auto-review) | `experimental` | [`agent`](../packages/core/agent), [`agent-instructions`](../packages/context/agent-instructions), [`llm`](../packages/llm/llm), [`permission-presets`](../packages/interaction/permission-presets), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`webhook`](../packages/experimental/webhook) | `experimental` | [`agent`](../packages/core/agent), [`agent-default-model`](../packages/core/agent-default-model), [`agent-preset-registry`](../packages/preset/agent-preset-registry), [`llm`](../packages/llm/llm), [`permission-presets`](../packages/interaction/permission-presets), [`session`](../packages/core/session), [`session-title`](../packages/session/session-title), [`workspace`](../packages/workspace/workspace) |
|
||||
| [`tool-cordis`](../packages/extensions/tool-cordis) | `extensions` | [`agent`](../packages/core/agent), [`app-boot`](../packages/boot/app-boot), [`cordis-host-runner`](../packages/extensions/cordis-host-runner), [`tools`](../packages/core/tools) |
|
||||
|
||||
+15
-9
@@ -248,7 +248,9 @@ flowchart TD
|
||||
pkg_office_to_pdf["office-to-pdf"]
|
||||
end
|
||||
subgraph group_experimental["packages/experimental"]
|
||||
pkg_client_ui_cua["client-ui-cua"]
|
||||
pkg_cua["cua"]
|
||||
pkg_cua_bundle["cua-bundle"]
|
||||
pkg_experimental_agent_team["experimental-agent-team"]
|
||||
pkg_experimental_agent_team_profile["experimental-agent-team-profile"]
|
||||
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
|
||||
@@ -907,14 +909,6 @@ flowchart TD
|
||||
pkg_workspace_changes --> pkg_session
|
||||
pkg_workspace_changes --> pkg_subprocess
|
||||
pkg_workspace_changes --> pkg_tools
|
||||
pkg_cua --> pkg_agent
|
||||
pkg_cua --> pkg_attachment
|
||||
pkg_cua --> pkg_computer_use
|
||||
pkg_cua --> pkg_llm
|
||||
pkg_cua --> pkg_subprocess
|
||||
pkg_cua --> pkg_system_prompt
|
||||
pkg_cua --> pkg_tools
|
||||
pkg_cua --> pkg_user_questions
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_agent
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_browser_use
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_system_prompt
|
||||
@@ -1078,6 +1072,16 @@ flowchart TD
|
||||
pkg_api_settings_controller --> pkg_settings
|
||||
pkg_api_settings_controller --> pkg_typert_protocol
|
||||
pkg_web_app --> pkg_shell_env
|
||||
pkg_cua --> pkg_agent
|
||||
pkg_cua --> pkg_agent_preset_registry
|
||||
pkg_cua --> pkg_attachment
|
||||
pkg_cua --> pkg_computer_use
|
||||
pkg_cua --> pkg_llm
|
||||
pkg_cua --> pkg_scope
|
||||
pkg_cua --> pkg_subprocess
|
||||
pkg_cua --> pkg_system_prompt
|
||||
pkg_cua --> pkg_tools
|
||||
pkg_cua --> pkg_user_questions
|
||||
pkg_experimental_auto_review --> pkg_agent
|
||||
pkg_experimental_auto_review --> pkg_agent_instructions
|
||||
pkg_experimental_auto_review --> pkg_llm
|
||||
@@ -1542,6 +1546,8 @@ flowchart TD
|
||||
| [`client-web`](../packages/client/web) | `client` | — |
|
||||
| [`credentials`](../packages/credentials/credentials) | `credentials` | — |
|
||||
| [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — |
|
||||
| [`client-ui-cua`](../packages/experimental/client-ui-cua) | `experimental` | — |
|
||||
| [`cua-bundle`](../packages/experimental/cua-bundle) | `experimental` | — |
|
||||
| [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — |
|
||||
| [`experimental-badge-skill-bundle`](../packages/experimental/badge-skill-bundle) | `experimental` | — |
|
||||
| [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | — |
|
||||
@@ -1706,7 +1712,6 @@ flowchart TD
|
||||
| [`file-reference-local`](../packages/context/file-reference-local) | `context` | [`agent`](../packages/core/agent), [`file-reference`](../packages/context/file-reference), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`tool-present`](../packages/deliverables/tool-present) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`session-projection`](../packages/session/session-projection), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) |
|
||||
| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) |
|
||||
| [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |
|
||||
@@ -1739,6 +1744,7 @@ flowchart TD
|
||||
| [`session-query`](../packages/session-query/session-query) | `session-query` | [`brand`](../packages/util/brand), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`session-persistence`](../packages/session/session-persistence), [`session-projection`](../packages/session/session-projection), [`session-projection-cache`](../packages/session/session-projection-cache), [`session-title`](../packages/session/session-title), [`tool-todo`](../packages/todo/tool-todo) |
|
||||
| [`api-settings-controller`](../packages/api/settings-controller) | `api` | [`agent-preset-registry`](../packages/preset/agent-preset-registry), [`credentials`](../packages/credentials/credentials), [`native-command`](../packages/util/native-command), [`session`](../packages/core/session), [`settings`](../packages/settings/settings), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`web-app`](../packages/bundle/web-app) | `bundle` | [`shell-env`](../packages/shell/shell-env) |
|
||||
| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`agent-preset-registry`](../packages/preset/agent-preset-registry), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) |
|
||||
| [`experimental-auto-review`](../packages/experimental/auto-review) | `experimental` | [`agent`](../packages/core/agent), [`agent-instructions`](../packages/context/agent-instructions), [`llm`](../packages/llm/llm), [`permission-presets`](../packages/interaction/permission-presets), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
|
||||
| [`webhook`](../packages/experimental/webhook) | `experimental` | [`agent`](../packages/core/agent), [`agent-default-model`](../packages/core/agent-default-model), [`agent-preset-registry`](../packages/preset/agent-preset-registry), [`llm`](../packages/llm/llm), [`permission-presets`](../packages/interaction/permission-presets), [`session`](../packages/core/session), [`session-title`](../packages/session/session-title), [`workspace`](../packages/workspace/workspace) |
|
||||
| [`tool-cordis`](../packages/extensions/tool-cordis) | `extensions` | [`agent`](../packages/core/agent), [`app-boot`](../packages/boot/app-boot), [`cordis-host-runner`](../packages/extensions/cordis-host-runner), [`tools`](../packages/core/tools) |
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
# After editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/subsystems/computer-use.md
|
||||
/computer-use:
|
||||
en: a82a6d1ec3fa9724
|
||||
zh: 17483872a54d6899
|
||||
en: 6f40d55bef2d7753
|
||||
zh: 0fc4eb79afb3cd73
|
||||
/computer-use/choose-a-provider:
|
||||
en: fa2e0d7b41c0450b
|
||||
zh: af9a799711f901e6
|
||||
|
||||
@@ -4,9 +4,19 @@ English | [中文](computer-use.zh.md)
|
||||
|
||||
Computer use lets a model observe and operate the local desktop through a configured provider. The shared DSH capability is called **computer use**; **Cua Driver** names the upstream implementation.
|
||||
|
||||
[Computer Use](../../packages/experimental/cua-bundle/README.md) provides an optional experimental switch for macOS CUA permission setup.
|
||||
|
||||
The [`macOS CUA provider`](../../packages/experimental/cua-macos/README.md) uses the [shared CUA objects and executor](../../packages/experimental/cua/README.md), which enforce application consent through User Questions and profile configuration.
|
||||
|
||||
The CUA provider owns `ctx.cuaAccess` for system-permission status, explicit system requests, and application revocation. Its [permission page](../../packages/experimental/client-ui-cua/README.md) projects these controls; app/window operations enforce the provider's consent policy before dispatch.
|
||||
|
||||
| Permission type | Meaning |
|
||||
|---|---|
|
||||
| `CuaApplicationId`, `CuaApplicationGrant` | An executable/signature identity and its displayed saved decision. |
|
||||
| `CuaSystemPermission` | Accessibility or Screen Recording, selected by the human. |
|
||||
| `CuaPermissionProcess`, `CuaSystemPermissionState` | Native privilege checks, observed helper/parent processes, and the application identified by macOS process responsibility at helper startup. Application metadata is absent if the optional system query or application lookup is unavailable. |
|
||||
| `CuaApplicationState` | Saved grants, refused applications, and whether persistence is available. Session-only grants remain internal to the provider. System observations and application icons are queried separately so the page can display each result as it arrives. |
|
||||
|
||||
## Choose a provider
|
||||
|
||||
Mount [`dsh-computer-use`](../../packages/computer-use/computer-use/README.md) and one provider in the same composition. Both Cua Driver providers are experimental public npm packages and require explicit activation.
|
||||
@@ -56,4 +66,60 @@ register(name: ComputerUseProviderName): () => Promise<void>
|
||||
```
|
||||
|
||||
Source: [`packages/computer-use/computer-use/src/index.ts`](../../packages/computer-use/computer-use/src/index.ts)
|
||||
|
||||
<a id="ctxcuaaccess--cuaaccess"></a>
|
||||
|
||||
### `ctx.cuaAccess` — `CuaAccess`
|
||||
|
||||
Browser controls for provider-owned native permission and application decisions.
|
||||
|
||||
```ts cordis-catalog
|
||||
/**
|
||||
* Read application decisions without waiting for the native helper.
|
||||
* @returns Saved grants and refused applications with their persistence owner.
|
||||
*/
|
||||
@Remote applications(): CuaApplicationState
|
||||
|
||||
/**
|
||||
* Inspect system privileges independently of application decisions and icons.
|
||||
* @returns Fresh native preflight and launch metadata.
|
||||
*/
|
||||
@Remote async systemStatus(): Promise<CuaSystemPermissionState>
|
||||
|
||||
/**
|
||||
* Load presentation icons separately from application decisions.
|
||||
* @returns Optional native PNG icons keyed by executable path for saved and blocked applications.
|
||||
*/
|
||||
@Remote async applicationIcons(): Promise<Record<string, string>>
|
||||
|
||||
/**
|
||||
* Request a system privilege explicitly selected by the user.
|
||||
* @param permission macOS privilege to request.
|
||||
* @returns Preflight state after requesting; a false value never implies consent.
|
||||
*/
|
||||
@Remote async requestPermission(permission: CuaSystemPermission): Promise<CuaSystemPermissionState>
|
||||
|
||||
/**
|
||||
* Open the macOS privacy pane selected by the user.
|
||||
* @param permission Corresponding system privilege.
|
||||
* @returns Completion after macOS accepted the open request.
|
||||
*/
|
||||
@Remote async openSettings(permission: CuaSystemPermission): Promise<void>
|
||||
|
||||
/**
|
||||
* Remove saved and session access and join affected operations.
|
||||
* @param id Application identity displayed on the permission page.
|
||||
* @returns Completion after persistence and cancellation; dispatched input cannot be undone.
|
||||
*/
|
||||
@Remote revoke(id: CuaApplicationId): Promise<void>
|
||||
|
||||
/**
|
||||
* Clear this application's refusals across sessions so each session may ask for access again.
|
||||
* @param id Refused or revoked application displayed on the permission page.
|
||||
* @returns Completion after clearing refusals; no permission is granted and no action is replayed.
|
||||
*/
|
||||
@Remote async retryApproval(id: CuaApplicationId): Promise<void>
|
||||
```
|
||||
|
||||
Source: [`packages/experimental/cua/src/access.ts`](../../packages/experimental/cua/src/access.ts)
|
||||
<!-- END GENERATED cordis-surface -->
|
||||
@@ -4,9 +4,19 @@
|
||||
|
||||
计算机操作让模型通过配置的提供方观察并操作本地桌面。DSH 的共享能力称为 **computer use（计算机操作）**；**Cua Driver** 是上游实现的名称。
|
||||
|
||||
[Computer Use](../../packages/experimental/cua-bundle/README.zh.md) 提供 macOS CUA 及权限设置 的可选实验性开关。
|
||||
|
||||
[macOS CUA 提供方](../../packages/experimental/cua-macos/README.zh.md)使用[共享 CUA 对象与执行器](../../packages/experimental/cua/README.zh.md)，通过 User Questions 和配置档设置执行应用授权。
|
||||
|
||||
CUA provider 通过 `ctx.cuaAccess` 提供系统权限状态、用户主动申请和应用授权撤销。[权限页面](../../packages/experimental/client-ui-cua/README.zh.md)投影这些控件；app/window 操作在分派前执行 provider 的授权检查。
|
||||
|
||||
| 权限类型 | 含义 |
|
||||
|---|---|
|
||||
| `CuaApplicationId`、`CuaApplicationGrant` | 可执行文件和签名身份，以及显示给用户的已保存决定。 |
|
||||
| `CuaSystemPermission` | 用户选择申请的辅助功能或屏幕录制权限。 |
|
||||
| `CuaPermissionProcess`、`CuaSystemPermissionState` | 观测到的 helper、父进程和原生权限状态，以及 helper 启动时由 macOS 责任进程记录识别出的应用。可选系统查询或应用查找不可用时，省略应用信息。 |
|
||||
| `CuaApplicationState` | 持久授权、拒绝记录，以及能否持久保存。会话授权只保留在 provider 内部。系统权限和应用图标分别查询，页面在各项结果返回后分别显示。 |
|
||||
|
||||
## 选择提供方
|
||||
|
||||
在同一组合中挂载 [`dsh-computer-use`](../../packages/computer-use/computer-use/README.zh.md) 和一个提供方。两个 Cua Driver 提供方都是公开发布到 npm 的实验性包，均需显式启用。
|
||||
@@ -56,4 +66,60 @@ register(name: ComputerUseProviderName): () => Promise<void>
|
||||
```
|
||||
|
||||
Source: [`packages/computer-use/computer-use/src/index.ts`](../../packages/computer-use/computer-use/src/index.ts)
|
||||
|
||||
<a id="ctxcuaaccess--cuaaccess"></a>
|
||||
|
||||
### `ctx.cuaAccess` — `CuaAccess`
|
||||
|
||||
Browser controls for provider-owned native permission and application decisions.
|
||||
|
||||
```ts cordis-catalog
|
||||
/**
|
||||
* Read application decisions without waiting for the native helper.
|
||||
* @returns Saved grants and refused applications with their persistence owner.
|
||||
*/
|
||||
@Remote applications(): CuaApplicationState
|
||||
|
||||
/**
|
||||
* Inspect system privileges independently of application decisions and icons.
|
||||
* @returns Fresh native preflight and launch metadata.
|
||||
*/
|
||||
@Remote async systemStatus(): Promise<CuaSystemPermissionState>
|
||||
|
||||
/**
|
||||
* Load presentation icons separately from application decisions.
|
||||
* @returns Optional native PNG icons keyed by executable path for saved and blocked applications.
|
||||
*/
|
||||
@Remote async applicationIcons(): Promise<Record<string, string>>
|
||||
|
||||
/**
|
||||
* Request a system privilege explicitly selected by the user.
|
||||
* @param permission macOS privilege to request.
|
||||
* @returns Preflight state after requesting; a false value never implies consent.
|
||||
*/
|
||||
@Remote async requestPermission(permission: CuaSystemPermission): Promise<CuaSystemPermissionState>
|
||||
|
||||
/**
|
||||
* Open the macOS privacy pane selected by the user.
|
||||
* @param permission Corresponding system privilege.
|
||||
* @returns Completion after macOS accepted the open request.
|
||||
*/
|
||||
@Remote async openSettings(permission: CuaSystemPermission): Promise<void>
|
||||
|
||||
/**
|
||||
* Remove saved and session access and join affected operations.
|
||||
* @param id Application identity displayed on the permission page.
|
||||
* @returns Completion after persistence and cancellation; dispatched input cannot be undone.
|
||||
*/
|
||||
@Remote revoke(id: CuaApplicationId): Promise<void>
|
||||
|
||||
/**
|
||||
* Clear this application's refusals across sessions so each session may ask for access again.
|
||||
* @param id Refused or revoked application displayed on the permission page.
|
||||
* @returns Completion after clearing refusals; no permission is granted and no action is replayed.
|
||||
*/
|
||||
@Remote async retryApproval(id: CuaApplicationId): Promise<void>
|
||||
```
|
||||
|
||||
Source: [`packages/experimental/cua/src/access.ts`](../../packages/experimental/cua/src/access.ts)
|
||||
<!-- END GENERATED cordis-surface -->
|
||||
@@ -0,0 +1,16 @@
|
||||
# Bilingual-pair consistency record for guide.md (docs/i18n/README.md): per heading
|
||||
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
|
||||
# After editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/upgrade-guide/v0.2.1-alpha.2/cua-preset-scope/guide.md
|
||||
/:
|
||||
en: 36d6b79b48c08dd5
|
||||
zh: 814caece159e9b0e
|
||||
/cua-defaults-to-ptc-session-scopes:
|
||||
en: 1d6eee8020ebed3b
|
||||
zh: d14279f527983c3c
|
||||
/cua-defaults-to-ptc-session-scopes/change:
|
||||
en: dac96a29e3751249
|
||||
zh: d48cb030f43c07e4
|
||||
/cua-defaults-to-ptc-session-scopes/migration:
|
||||
en: f3d82fe45e41c712
|
||||
zh: e55e424f5a2c096a
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
kind: upgrade-guide
|
||||
description: "Standalone native CUA compositions must explicitly select global tool registration."
|
||||
---
|
||||
|
||||
# CUA defaults to PTC session scopes
|
||||
|
||||
English | [中文](guide.zh.md)
|
||||
|
||||
## Change
|
||||
|
||||
The native CUA provider defaults `agentPreset` to `ptc`. It registers tools for existing and new PTC Agents, including when a saved application-grants configuration omits the new field. Custom compositions without an Agent preset registry no longer receive global CUA tools by default.
|
||||
|
||||
## Migration
|
||||
|
||||
1. For the standard Computer Use switch, keep the default; existing application grants remain usable.
|
||||
2. For a standalone composition that needs global tools, add `agentPreset: false` under the native CUA entry’s `config` in its profile patch or overlay. Preserve its other fields, including `allowedApps` and `helperPath`.
|
||||
3. Restart or let HMR apply the edit, then confirm that the intended Agent exposes `cua` in native mode or through the PTC SDK.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
kind: upgrade-guide
|
||||
description: "独立原生 CUA 组合需要显式选择全局工具注册。"
|
||||
---
|
||||
|
||||
# CUA 默认使用 PTC 会话作用域
|
||||
|
||||
[English](guide.md) | 中文
|
||||
|
||||
## 变更
|
||||
|
||||
原生 CUA 提供方的 `agentPreset` 默认为 `ptc`。工具注册到已有和新建 PTC Agent，即使保存的应用授权配置没有这个新字段也会生效。不包含 Agent preset 注册表的自定义组合不再默认获得全局 CUA 工具。
|
||||
|
||||
## 迁移
|
||||
|
||||
1. 使用标准 Computer Use 开关时保持默认值，已有应用授权仍可使用。
|
||||
2. 独立组合若需要全局工具，在 profile patch 或 overlay 的原生 CUA 条目 `config` 下添加 `agentPreset: false`，保留 `allowedApps`、`helperPath` 等其他字段。
|
||||
3. 重启或等待 HMR 应用后，确认目标 Agent 能在原生模式或 PTC SDK 中使用 `cua`。
|
||||
@@ -290,6 +290,7 @@
|
||||
"@deepseek-ai/dsh-working-directory": "workspace:*",
|
||||
"@deepseek-ai/dsh-experimental-worktree": "workspace:*",
|
||||
"@deepseek-ai/dsh-experimental-tool-worktree": "workspace:*",
|
||||
"@deepseek-ai/dsh-cua-bundle": "workspace:*",
|
||||
"@deepseek-ai/dsh-experimental-subagent-oar": "workspace:*",
|
||||
"@deepseek-ai/dsh-experimental-tool-subagent-oar": "workspace:*",
|
||||
"@deepseek-ai/dsh-experimental-cua-macos": "workspace:*"
|
||||
|
||||
@@ -221,6 +221,7 @@ export const DEFAULT_PROFILE_BUNDLES: readonly string[] = ['@deepseek-ai/dsh-bas
|
||||
* [admission](../../../../.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md)).
|
||||
*/
|
||||
export const OPTIONAL_BUNDLES: readonly string[] = [
|
||||
'@deepseek-ai/dsh-cua-bundle',
|
||||
'@deepseek-ai/dsh-experimental-session-search',
|
||||
'@deepseek-ai/dsh-experimental-ralph-bundle',
|
||||
'@deepseek-ai/dsh-experimental-terminal-bundle',
|
||||
|
||||
@@ -9,8 +9,8 @@
|
||||
en: 77ea90624a004773
|
||||
zh: fe6dbf8414a5aff0
|
||||
/packages-computer-use/summary:
|
||||
en: 3820da56a601d3f1
|
||||
zh: 358e65f8a7cf7446
|
||||
en: 7e5fcf9bb48c9677
|
||||
zh: abee2be288115045
|
||||
/packages-computer-use/table-of-contents:
|
||||
en: 1eb35b015a6c423f
|
||||
zh: a8a3dceee2c9e943
|
||||
@@ -18,8 +18,8 @@
|
||||
en: 9aa86e6451f60c3d
|
||||
zh: c5e6816e90f2d74b
|
||||
/packages-computer-use/related-documentation:
|
||||
en: ad08d3417a0d4630
|
||||
zh: cfe8e03033c2fede
|
||||
en: 58198f9f17e6120f
|
||||
zh: 188fa0ec5c4678f3
|
||||
/packages-computer-use/dev-note:
|
||||
en: b75c8458928b72b9
|
||||
zh: 003ba4e17b4b5269
|
||||
@@ -9,7 +9,7 @@ English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
Computer-use providers let models observe and operate a desktop. This group owns exclusive provider registration. Each provider owns its operations, tools, and platform requirements; the experimental Cua Driver providers live in the experimental group.
|
||||
Computer-use providers let models observe and operate a desktop. This group contains the shared registry that prevents multiple providers from being enabled in one composition. DSH's [CUA implementation](../experimental/cua/README.md), platform providers, and opt-in bundle currently live in `experimental`, alongside the Cua Driver adapters. The CUA package documents the platform requirements for promotion to this group.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
@@ -32,7 +32,8 @@ Choose one provider and mount the shared registration service.
|
||||
## Related documentation
|
||||
|
||||
- [Computer use](../../docs/subsystems/computer-use.md) — capability ownership and provider choices.
|
||||
- [Experimental packages](../experimental/README.md) — Cua Driver MCP and native providers.
|
||||
- [Computer Use bundle](../experimental/cua-bundle/README.md) — enable native CUA and access its permission controls.
|
||||
- [Experimental packages](../experimental/README.md) — native CUA packages and Cua Driver adapters.
|
||||
|
||||
<a id="dev-note"></a>
|
||||
## Dev Note
|
||||
|
||||
@@ -9,7 +9,7 @@ kind: "package-group"
|
||||
|
||||
## 概述
|
||||
|
||||
计算机操作提供方让模型观察并操作桌面。本组负责提供方的独占注册。各提供方拥有自己的操作、工具和平台要求；实验性 Cua Driver 提供方位于 experimental 组。
|
||||
计算机操作提供方让模型观察并操作桌面。本组包含共享注册器，防止同一组合同时启用多个提供方。DSH 的 [CUA 实现](../experimental/cua/README.zh.md)、平台提供方和启用组合包目前位于 `experimental`，Cua Driver 适配包也位于该组。CUA 包文档说明了迁入本组所需的平台支持条件。
|
||||
|
||||
## 目录
|
||||
|
||||
@@ -32,7 +32,8 @@ kind: "package-group"
|
||||
## 相关文档
|
||||
|
||||
- [计算机操作](../../docs/subsystems/computer-use.zh.md) — 能力归属和提供方选择。
|
||||
- [实验性包](../experimental/README.zh.md) — Cua Driver MCP 和原生提供方。
|
||||
- [Computer Use 组合包](../experimental/cua-bundle/README.zh.md)：启用原生 CUA 并使用权限管理功能。
|
||||
- [实验性包](../experimental/README.zh.md)：原生 CUA 包和 Cua Driver 适配包。
|
||||
|
||||
<a id="dev-note"></a>
|
||||
## 开发备注
|
||||
|
||||
@@ -9,8 +9,8 @@
|
||||
en: 2efee0c5f0d577bc
|
||||
zh: fef4b81bfc121052
|
||||
/deepseek-ai-dsh-computer-use/summary:
|
||||
en: 4de047849e68701f
|
||||
zh: 90a32c0e279b4b05
|
||||
en: 4cb331c550852b23
|
||||
zh: d638995ab250397b
|
||||
/deepseek-ai-dsh-computer-use/table-of-contents:
|
||||
en: d152484eb41ac6b4
|
||||
zh: 405b30feaf48c087
|
||||
@@ -21,8 +21,8 @@
|
||||
en: 181679887837591a
|
||||
zh: cc2e96766ce0aa63
|
||||
/deepseek-ai-dsh-computer-use/further-exploration:
|
||||
en: b1bdbcbc150036b3
|
||||
zh: 1d356e0522587abc
|
||||
en: 987295d677466c04
|
||||
zh: 9d7b40a04786490e
|
||||
/deepseek-ai-dsh-computer-use/model-experience:
|
||||
en: 8780c250dfaefaaa
|
||||
zh: 1c4b3598a131fb65
|
||||
|
||||
@@ -9,7 +9,7 @@ English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
A deployment can enable one computer-use provider at a time. Loading another provider fails with the registered provider name. Each provider supplies its own tools and desktop operations. This package adds no model-visible tools and does not coordinate concurrent Sessions.
|
||||
A deployment can enable one computer-use provider at a time. Loading another provider fails with the registered provider name. Both DSH's native CUA and the Cua Driver adapters use this registry. Each provider supplies its own tools and desktop operations. This package adds no model-visible tools and does not coordinate concurrent Sessions.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
@@ -53,6 +53,9 @@ One private name owns the slot. Cordis effects remove contributions when their p
|
||||
## Further Exploration
|
||||
|
||||
- [Computer use](../../../docs/subsystems/computer-use.md) — provider selection and shared-desktop limits.
|
||||
- [Computer Use bundle](../../experimental/cua-bundle/README.md) — enable DSH's native CUA and permission controls.
|
||||
- [Shared CUA library](../../experimental/cua/README.md) — application/window objects, tool execution, and application consent.
|
||||
- [macOS CUA provider](../../experimental/cua-macos/README.md) — DSH's Swift implementation and native helper lifecycle.
|
||||
- [Cua Driver MCP provider](../../experimental/computer-use-cua-driver-mcp/README.md) — use an installed driver.
|
||||
- [Cua Driver native provider](../../experimental/computer-use-cua-driver-native/README.md) — use the npm runtime.
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ kind: "package-reference"
|
||||
|
||||
## 概述
|
||||
|
||||
部署每次可启用一个计算机操作提供方。加载另一个提供方时会失败，并报告已注册的提供方名称。各提供方提供自己的工具和桌面操作。本包不添加模型可见工具，也不协调并发 Session。
|
||||
部署每次可启用一个计算机操作提供方。加载另一个提供方时会失败，并报告已注册的提供方名称。DSH 原生 CUA 和 Cua Driver 适配包都使用此注册器。各提供方提供自己的工具和桌面操作。本包不添加模型可见工具，也不协调并发 Session。
|
||||
|
||||
## 目录
|
||||
|
||||
@@ -53,6 +53,9 @@ kind: "package-reference"
|
||||
## 延伸阅读
|
||||
|
||||
- [计算机操作](../../../docs/subsystems/computer-use.zh.md) — 提供方选择和共享桌面限制。
|
||||
- [Computer Use 组合包](../../experimental/cua-bundle/README.zh.md)：启用 DSH 原生 CUA 和权限管理功能。
|
||||
- [共享 CUA 库](../../experimental/cua/README.zh.md)：应用与窗口对象、工具执行和应用授权。
|
||||
- [macOS CUA 提供方](../../experimental/cua-macos/README.zh.md)：DSH 的 Swift 实现和原生辅助进程生命周期。
|
||||
- [Cua Driver MCP 提供方](../../experimental/computer-use-cua-driver-mcp/README.zh.md) — 使用已安装的驱动。
|
||||
- [Cua Driver 原生提供方](../../experimental/computer-use-cua-driver-native/README.zh.md) — 使用 npm 运行时。
|
||||
|
||||
|
||||
@@ -18,8 +18,8 @@
|
||||
en: b55aeb927737b90d
|
||||
zh: 0b215fcb5c2c7908
|
||||
/packages-experimental/packages:
|
||||
en: 372b182dbf0f662b
|
||||
zh: 7a8031c506394785
|
||||
en: e12eeae4540a53e0
|
||||
zh: 44f65caca730252d
|
||||
/packages-experimental/related-documentation:
|
||||
en: 1cb9d3448d44e48d
|
||||
zh: cd326d70a746b896
|
||||
|
||||
@@ -71,6 +71,8 @@ Promotion requires a named product role, documented public behavior and limitati
|
||||
| [`ptc-runtime-python`](ptc-runtime-python/README.md) | CPython subprocess backend for the PTC execution seam | `ctx.ptcRuntime` |
|
||||
| [CUA](cua/README.md) | Shared application/window objects, consent, and logged execution. | Library |
|
||||
| [macOS CUA](cua-macos/README.md) | Swift native execution and platform preparation. | `ctx.computerUse` |
|
||||
| [`cua-bundle`](cua-bundle/README.md) | Optional Computer Use switch with persistent PTC | — |
|
||||
| [`client-ui-cua`](client-ui-cua/README.md) | Computer Use permission setup and application access controls | — |
|
||||
| [`computer-use-cua-driver-mcp`](computer-use-cua-driver-mcp/README.md) | Use an installed Cua Driver through MCP | `ctx.computerUse` |
|
||||
| [`computer-use-cua-driver-native`](computer-use-cua-driver-native/README.md) | Embed the Cua Driver native npm runtime | `ctx.computerUse` |
|
||||
| [`browser-use-playwright-mcp`](browser-use-playwright-mcp/README.md) | Playwright browser tools over MCP | `ctx.browserUse` |
|
||||
|
||||
@@ -71,6 +71,8 @@ kind: "package-group"
|
||||
| [`ptc-runtime-python`](ptc-runtime-python/README.zh.md) | PTC 执行 seam 的 CPython 子进程后端 | `ctx.ptcRuntime` |
|
||||
| [CUA](cua/README.zh.md) | 共享应用与窗口对象、授权和已记录的执行。 | 库 |
|
||||
| [macOS CUA](cua-macos/README.zh.md) | Swift 原生执行与平台组件准备。 | `ctx.computerUse` |
|
||||
| [`cua-bundle`](cua-bundle/README.zh.md) | 默认关闭的 Computer Use 开关与 persistent PTC 组合 | — |
|
||||
| [`client-ui-cua`](client-ui-cua/README.zh.md) | Computer Use 权限设置与应用授权管理 | — |
|
||||
| [`computer-use-cua-driver-mcp`](computer-use-cua-driver-mcp/README.zh.md) | 通过 MCP 使用已安装的 Cua Driver | `ctx.computerUse` |
|
||||
| [`computer-use-cua-driver-native`](computer-use-cua-driver-native/README.zh.md) | 嵌入 Cua Driver 原生 npm 运行时 | `ctx.computerUse` |
|
||||
| [`browser-use-playwright-mcp`](browser-use-playwright-mcp/README.zh.md) | 通过 MCP 提供 Playwright 浏览器工具 | `ctx.browserUse` |
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# Bilingual-pair consistency record for README.md (docs/i18n/README.md): per heading
|
||||
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
|
||||
# After editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/experimental/client-ui-cua/README.md
|
||||
/:
|
||||
en: e5ff656e81db5fc8
|
||||
zh: 43a8e818159176b3
|
||||
/deepseek-ai-dsh-client-ui-cua:
|
||||
en: 43859860b79a3757
|
||||
zh: e8e0829c36613320
|
||||
/deepseek-ai-dsh-client-ui-cua/summary:
|
||||
en: 4e62e511b5d14986
|
||||
zh: 1fce04dbad7515f7
|
||||
/deepseek-ai-dsh-client-ui-cua/table-of-contents:
|
||||
en: 6658ab477751154b
|
||||
zh: 9b371809a6fe38dc
|
||||
/deepseek-ai-dsh-client-ui-cua/use-this-package:
|
||||
en: d366fc19d6806a9f
|
||||
zh: 0f6b02f19139c3f6
|
||||
/deepseek-ai-dsh-client-ui-cua/understand-the-implementation:
|
||||
en: 7e661a9d00f98068
|
||||
zh: a0508c9b1d48cbb7
|
||||
/deepseek-ai-dsh-client-ui-cua/further-exploration:
|
||||
en: f0fde9f736ade61a
|
||||
zh: 43a534a21bcc1c13
|
||||
/deepseek-ai-dsh-client-ui-cua/model-experience:
|
||||
en: 7cfbaa62f70c1916
|
||||
zh: 384ec816a50fc7c5
|
||||
/deepseek-ai-dsh-client-ui-cua/model-experience/kv-cache-effect:
|
||||
en: f04fc6e6f503960f
|
||||
zh: f080408ae6729fee
|
||||
/deepseek-ai-dsh-client-ui-cua/known-limitations-and-deferred-work:
|
||||
en: c032612d923e1367
|
||||
zh: eb77d474e21ca5d9
|
||||
/deepseek-ai-dsh-client-ui-cua/known-limitations-and-deferred-work/dev-note:
|
||||
en: b836f01bd0bceb4a
|
||||
zh: a6ea1361c4a48df0
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
description: "Check macOS Computer Use permissions and revoke application access."
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-client-ui-cua
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
Open Computer Use in Plugins to check Accessibility and Screen Recording, request missing system permissions, and revoke application access. Explicit activation probes permissions and may start a dedicated native helper. Missing privileges show setup guidance; a failed probe directs the user to diagnostics and retry in the details page. System prompts still require the user's decision.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Use this package](#use-this-package)
|
||||
- [Understand the implementation](#understand-the-implementation)
|
||||
- [Further Exploration](#further-exploration)
|
||||
- [Model Experience](#model-experience)
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
- [Dev Note](#dev-note)
|
||||
|
||||
The permission-page lists show native icons and application names. Paths remain available on hover. Icons are presentation metadata and are never saved with grants. Application access uses the existing User Questions interface, with session access, refusal and, for verifiable applications with writable settings, persistent access.
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## Use this package
|
||||
|
||||
The [Computer Use bundle](../cua-bundle/README.md) mounts this page with its native provider. **Authorize Computer Use** groups the new-session default, Accessibility, and Screen Recording inside one border, with dividers between the rows. Each system-permission row shows the system-reported responsible application’s `.app` bundle filename and icon together in a bordered label. **Manage applications Computer Use can access** places the application lists inside a separate bordered group. Choose **Request permission**, find that application in the macOS permission list, and enable its switch; if macOS names another application, follow that prompt. With no detected source, the page directs the user to the application named by macOS. **Open System Settings** opens the corresponding privacy pane. The page refreshes on entry, when the window regains focus, and after permission actions. A failed query offers **Retry**. Follow any macOS quit-and-reopen prompt and restart DSH as needed.
|
||||
|
||||
System-permission operations affect the computer running DSH. When accessing DSH remotely, complete authorization on that computer.
|
||||
|
||||
Enabling from the plugin list shows setup guidance when system permissions need attention, with **Later** and **Open settings** actions. Enabling from the detail page displays the permission controls directly. The detail page provides **Set to PTC** under **Default mode for new sessions**; only this action saves the default. Existing sessions keep their preset.
|
||||
|
||||
The page lists always-allowed applications and refusals from any session. A refusal in one session does not cancel another session’s grant or prevent a new session from asking. Session-only grants remain effective within their sessions and are omitted from this page. **Revoke** removes the selected application's access and cancels affected work. Input already delivered to an application cannot be undone; observe the application before any further action. **Allow asking again** clears that application’s refusal records across all sessions; the next application selection still requires a new decision.
|
||||
|
||||
This plugin has no configurable fields.
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## Understand the implementation
|
||||
|
||||
<details>
|
||||
<summary>Implementation internals</summary>
|
||||
|
||||
The optional browser contribution mounts the native provider's permission Remote and reuses the Plugins activation and configuration slots. The three setup rows remain visible while unknown fields use placeholders. The default-mode form, system checks, application decisions, and application icons fill in as each query completes. Application names and actions do not wait for icons; icon positions retain their space. User actions run in order without waiting for pending observation queries. Disposal waits for both actions and queries to settle before releasing the separately owned Remote mount. Query errors stay with the affected section, and activation guidance depends only on system checks. Late results after disposal are discarded. One controller projects queries into the page and setup dialog; reconnects and updates to the Host-reported settings namespace refresh it. Operation notices live in the shell overlay, so closing the page does not remove them. The provider owns permission checks, persisted decisions, and cancellation; the browser owns no authorization state.
|
||||
|
||||
The detail page reads the effective default (`selectedDefault ?? default`) from the existing `configForms` service. Its button writes only `selectedDefault` through the form's revision-checked settings path. Loading, read-only and unavailable settings cannot submit a change; a failed write remains beside the setting for retry.
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## Further Exploration
|
||||
|
||||
- [Native provider](../cua/README.md): executable identity and operation checks.
|
||||
- [Plugin manager](../../client/ui-plugin-manager/README.md): activation and page slots.
|
||||
- [UI primitives](../../client/ui-primitives/README.md): shared modal, buttons, and toast.
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## Model Experience
|
||||
|
||||
None, as this package registers browser controls and no model-facing tools or prompt sections.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; the native provider owns model-visible guidance and operation results.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
This CUA-specific UI remains experimental while Windows support is in development; promotion returns it to the `client` group as `ui-cua`.
|
||||
|
||||
- macOS may require restarting the launching application after a system permission change. A pending request is not an authorization grant.
|
||||
- Application attribution depends on an optional native system query and available application metadata. If either is unavailable, the page uses generic system-prompt guidance. Packaged Desktop distribution and permission attribution remain deferred.
|
||||
- No invariant companion is published: this UI projects the provider's query and stores no independent authorization decisions.
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### Dev Note
|
||||
|
||||
<details>
|
||||
<summary>Working context for maintainers</summary>
|
||||
|
||||
None.
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
description: "检查 macOS Computer Use 权限并撤销应用访问权限。"
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-client-ui-cua
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
## 概述
|
||||
|
||||
在插件页打开 Computer Use，可以检查辅助功能和屏幕录制权限、申请缺失的系统权限，并撤销应用授权。用户显式启用时会检查权限，可能启动专用原生 helper。缺少权限时显示设置引导，查询失败时引导用户到详情页查看原因并重试。系统授权仍由用户决定。
|
||||
|
||||
## 目录
|
||||
|
||||
- [使用本包](#use-this-package)
|
||||
- [理解实现](#understand-the-implementation)
|
||||
- [延伸阅读](#further-exploration)
|
||||
- [模型体验](#model-experience)
|
||||
- [已知限制与后续工作](#known-limitations-and-deferred-work)
|
||||
- [开发备注](#dev-note)
|
||||
|
||||
权限页列表显示本机图标与应用名称，悬停可查看路径。图标仅用于展示，不随授权保存。应用授权沿用现有 User Questions 界面，提供本会话允许和拒绝；应用签名可验证且配置可写时，还提供始终允许。
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## 使用本包
|
||||
|
||||
[Computer Use 组合](../cua-bundle/README.zh.md)将本页与原生 provider 一起挂载。“给 Computer Use 授权”把新会话默认模式、辅助功能和屏幕录制放在同一个边框内，三排之间使用分隔线。两条系统权限要求在能识别责任应用时，把该应用的 `.app` 包名称和图标放在同一个标签中。“管理 Computer Use 可访问的应用”使用独立边框包住应用列表。点击“申请权限”，在 macOS 权限列表中找到该应用并开启对应开关；若 macOS 显示其他名称，以系统提示为准。来源无法识别时，页面引导用户查找系统提示的应用。“打开系统设置”会打开对应的隐私设置页。进入详情页、返回窗口或完成授权操作后会自动刷新；查询失败时可点击“重试”。若系统要求退出并重新打开应用，请按提示操作并重新启动 DSH。
|
||||
|
||||
系统权限操作作用于运行 DSH 的电脑。远程访问时，请在该电脑上完成授权。
|
||||
|
||||
从插件列表启用后，系统权限尚未就绪时会显示设置引导，提供“稍后”和“前往设置”。从详情页启用后直接显示权限控件。详情页的“新会话默认模式”提供“设为 PTC”按钮，仅点击此按钮才保存默认值。已有会话保留原 preset。
|
||||
|
||||
页面展示“始终允许的应用”和“不允许访问的应用”。后者汇总任一会话的拒绝记录，不取消其他会话已有的授权，也不阻止新会话询问授权。会话授权仍在对应会话中生效，详情页不展示这类应用。“撤销”会移除对应应用的访问许可并取消相关操作。已发送到应用的输入无法撤回，继续操作前需重新观察应用。“允许再次询问”清除该应用在所有会话中的拒绝记录；下次选择应用时仍需用户重新决定。
|
||||
|
||||
本插件没有配置字段。
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## 理解实现
|
||||
|
||||
<details>
|
||||
<summary>实现细节</summary>
|
||||
|
||||
可选浏览器插件挂载原生 provider 的权限 Remote，复用插件页的启用引导和配置插槽。三排使用条件始终可见，未加载的字段保留占位。默认模式、系统权限、应用列表和应用图标分别查询，结果返回后各自显示。应用名称和操作按钮不等待图标，图标位置保留空位。用户操作按顺序执行，不等待尚未完成的查询；卸载时等待操作和查询全部结束，再释放独立持有的 Remote 挂载。查询错误只显示在对应部分，启用引导只等待系统权限结果。卸载后的迟到结果会被丢弃。一个控制器把查询结果投影到页面和设置弹窗；重新连接或 Host 返回的配置 namespace 更新时刷新状态。操作提示挂载到 shell overlay，关闭页面后仍可显示。provider 负责权限检查、持久授权和取消；浏览器不保存授权决定。
|
||||
|
||||
详情页通过现有 `configForms` 服务读取实际默认值（`selectedDefault ?? default`）。按钮仅写入 `selectedDefault`，沿用表单带版本检查的设置写入流程。设置仍在加载、只读或不可用时不能提交；写入失败时在设置旁显示错误，供用户重试。
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## 延伸阅读
|
||||
|
||||
- [原生 provider](../cua/README.zh.md)：可执行文件身份和操作检查。
|
||||
- [插件管理](../../client/ui-plugin-manager/README.zh.md)：启用引导和页面插槽。
|
||||
- [界面基础组件](../../client/ui-primitives/README.zh.md)：共用弹窗、按钮和提示。
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## 模型体验
|
||||
|
||||
无。本包注册浏览器控件，不注册模型工具或提示词段落。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
无。模型可见的指引和操作结果由原生 provider 负责。
|
||||
|
||||
## 已知限制与后续工作
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
这套 CUA 专用界面随 Windows 支持一起开发，暂放在 `experimental`；转为正式功能后迁回 `client` 组的 `ui-cua`。
|
||||
|
||||
- 系统权限变更后，macOS 可能要求重启启动程序。申请尚未完成时，不能视为已授权。
|
||||
- 应用归属依赖可选的本机系统查询和可读取的应用信息；不可用时，页面使用通用指引，让用户按系统弹窗完成授权。桌面安装包的分发和权限归属留待后续处理。
|
||||
- 不提供 invariant companion：本界面投影 provider 查询，不独立保存授权决定。
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### 开发备注
|
||||
|
||||
<details>
|
||||
<summary>维护者工作记录</summary>
|
||||
|
||||
无。
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-client-ui-cua",
|
||||
"description": "Computer Use permission setup and application access controls",
|
||||
"version": "0.2.1-rc.2",
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
||||
"directory": "packages/experimental/client-ui-cua"
|
||||
},
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./client": {
|
||||
"types": "./lib/types/client/index.d.ts",
|
||||
"default": "./lib/client.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"dsh": {
|
||||
"client": {
|
||||
"inject": [
|
||||
"@deepseek-ai/dsh-api-remotes",
|
||||
"@deepseek-ai/dsh-client-locale",
|
||||
"@deepseek-ai/dsh-client-ui-plugin-manager",
|
||||
"@deepseek-ai/dsh-client-ui-settings",
|
||||
"@deepseek-ai/dsh-client-ui-layout"
|
||||
],
|
||||
"platform": "web"
|
||||
}
|
||||
},
|
||||
"scripts": {
|
||||
"bundle": "tsdown",
|
||||
"watch": "tsdown --watch"
|
||||
},
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:~"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:~",
|
||||
"@deepseek-ai/dsh-api-remotes": "workspace:*",
|
||||
"@deepseek-ai/dsh-api-gateway": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-locale": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-store": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-ui-primitives": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-ui-renderer": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-ui-slots": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-ui-plugin-manager": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-ui-settings": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-ui-layout": "workspace:*",
|
||||
"@deepseek-ai/dsh-cua": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-test-runtime": "workspace:*",
|
||||
"react": "^18.2.0",
|
||||
"react-dom": "^18.2.0",
|
||||
"@testing-library/react": "^16.1.0",
|
||||
"@deepseek-ai/dsh-typert-protocol": "workspace:*"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/client.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
.panels { display: grid; gap: 24px; margin: 18px 0; font-size: 13px; }
|
||||
.panel { display: grid; gap: 12px; min-width: 0; }
|
||||
.card { min-width: 0; border: 0.5px solid var(--dsw-alias-border-l1); border-radius: var(--dsw-radius-md); }
|
||||
.card > * + * { border-top: 0.5px solid var(--dsw-alias-border-l1); }
|
||||
.panel p { margin: 0; color: var(--dsw-alias-label-secondary); line-height: 1.6; }
|
||||
.panel h3 { margin: 0; font-size: 14px; font-weight: 500; }
|
||||
.panel h4 { margin: 0; font-size: 13px; font-weight: 500; color: var(--dsw-alias-label-secondary); }
|
||||
.permission, .apps li { display: flex; flex-wrap: wrap; align-items: center; gap: 12px; }
|
||||
.permission, .group, .emptyState, .notice { padding: 16px; }
|
||||
.permission { min-height: 80px; box-sizing: border-box; }
|
||||
.defaultSetting > p { margin: 0 16px 16px; }
|
||||
.permission > div:first-child, .apps li > div { flex: 1; min-width: 140px; }
|
||||
.actions, .notice { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; }
|
||||
.group { display: grid; gap: 4px; }
|
||||
.apps { display: grid; padding: 0; margin: 0; list-style: none; }
|
||||
.apps li { padding: 12px 0; }
|
||||
.apps li:last-child { padding-bottom: 0; }
|
||||
.panel strong { font-weight: 500; }
|
||||
.panel .error { color: var(--dsw-alias-state-error-primary); overflow-wrap: anywhere; }
|
||||
.application { display: flex; align-items: center; gap: 10px; }
|
||||
.application strong { min-width: 0; overflow-wrap: anywhere; }
|
||||
.application img { flex-shrink: 0; object-fit: contain; }
|
||||
.applicationBadge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
max-width: 100%;
|
||||
padding: 2px 6px;
|
||||
border: 0.5px solid var(--dsw-alias-border-l1);
|
||||
border-radius: var(--dsw-radius-sm);
|
||||
color: var(--dsw-alias-label-primary);
|
||||
vertical-align: middle;
|
||||
}
|
||||
.applicationBadge img { flex-shrink: 0; object-fit: contain; }
|
||||
.applicationBadge > span { min-width: 0; overflow-wrap: anywhere; }
|
||||
.permissionStatus { min-width: 72px; }
|
||||
.buttonPlaceholder, .statusPlaceholder, .appPlaceholder, .appsPlaceholder {
|
||||
display: inline-block;
|
||||
max-width: 100%;
|
||||
border-radius: var(--dsw-radius-xs);
|
||||
background: var(--dsw-alias-bg-skeleton);
|
||||
}
|
||||
.buttonPlaceholder { width: 120px; height: 28px; }
|
||||
.statusPlaceholder { width: 64px; height: 12px; }
|
||||
.appPlaceholder { width: 160px; height: 20px; }
|
||||
.appsPlaceholder { width: 240px; height: 40px; }
|
||||
|
||||
.iconPlaceholder { display: inline-block; width: 32px; height: 32px; flex-shrink: 0; }
|
||||
@@ -0,0 +1,183 @@
|
||||
/** Native permission status and revocable application consent on the existing plugin page. */
|
||||
import { useEffect, useState } from 'react'
|
||||
import { Button, Modal, Toast } from '@deepseek-ai/dsh-client-ui-primitives'
|
||||
import type { InjectFace, PropsLocale, PropsRuntime } from '@deepseek-ai/dsh-client-ui-slots'
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-plugin-manager/client'
|
||||
import type { ConfigForm } from '@deepseek-ai/dsh-client-ui-settings/client'
|
||||
import type { CuaApplicationGrant, CuaSystemPermissionState } from '@deepseek-ai/dsh-cua/types'
|
||||
import type { PermissionFace } from './controller.ts'
|
||||
import type { NS } from './locales.ts'
|
||||
import css from './PermissionPanel.module.css'
|
||||
|
||||
/** Renderer-bound permission observation and localized controls. */
|
||||
export type PermissionProps = InjectFace<PermissionFace> & PropsLocale<typeof NS>
|
||||
|
||||
/** Permission guidance only navigates to the existing bundle details. */
|
||||
export type PermissionSetupProps = PermissionProps
|
||||
& Pick<PropsRuntime<'plugins.bundle.activation'>, 'onDismiss' | 'onOpenDetails'>
|
||||
|
||||
type DefaultPresetProps = PropsLocale<typeof NS> & InjectFace<{
|
||||
hooks: { cuaDefaultPreset: ConfigForm<{ default: string; selectedDefault?: string }> }
|
||||
setPtcDefault: () => Promise<boolean>
|
||||
}>
|
||||
|
||||
/** The detail page owns the optional new-session default action. */
|
||||
export type PermissionPanelProps = PermissionProps & DefaultPresetProps
|
||||
|
||||
function DefaultPresetSetting(props: DefaultPresetProps) {
|
||||
const { t } = props
|
||||
const preset = props.useCuaDefaultPreset(value => value)
|
||||
const isPtc = (preset.value?.selectedDefault ?? preset.value?.default) === 'ptc'
|
||||
const loading = preset.value === undefined && preset.status === 'loading'
|
||||
const [saving, setSaving] = useState(false)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const setDefault = async (): Promise<void> => {
|
||||
setSaving(true)
|
||||
setError(null)
|
||||
try {
|
||||
if (!await props.setPtcDefault()) setError(t('defaultNotSaved'))
|
||||
} catch (error) {
|
||||
setError(t('defaultFailed', { message: error instanceof Error ? error.message : String(error) }))
|
||||
} finally { setSaving(false) }
|
||||
}
|
||||
return <div className={css.defaultSetting} role="group" aria-label={t('defaultTitle')}>
|
||||
<div className={css.permission}>
|
||||
<div><strong>{t('defaultTitle')}</strong><p>{t('defaultBody')}</p></div>
|
||||
{loading ? <span className={css.buttonPlaceholder} aria-hidden />
|
||||
: <Button size="sm" variant={isPtc ? 'outline' : 'primary'} disabled={saving || isPtc || !preset.writable || preset.status !== 'ready'}
|
||||
onClick={() => { void setDefault() }}>{t(isPtc ? 'defaultSet' : 'defaultConfirm')}</Button>}
|
||||
</div>
|
||||
{!isPtc && (preset.status === 'unavailable' ? <p className={css.error} role="alert">{t('defaultUnavailable')}</p>
|
||||
: preset.status === 'ready' && !preset.writable && <p className={css.error} role="alert">{t('defaultReadOnly')}</p>)}
|
||||
{error !== null && <p className={css.error} role="alert">{error}</p>}
|
||||
</div>
|
||||
}
|
||||
|
||||
function Applications({ title, apps, icons, onAction, label, disabled }: {
|
||||
title: string
|
||||
apps: readonly CuaApplicationGrant[]
|
||||
icons: Record<string, string> | null
|
||||
disabled: boolean
|
||||
label: string
|
||||
onAction: PermissionProps['revoke']
|
||||
}) {
|
||||
return <section className={css.group}>
|
||||
<h4>{title}</h4>
|
||||
<ul className={css.apps}>{apps.map(app => <li key={app.id}>
|
||||
<div className={css.application} title={app.executablePath}>
|
||||
{icons?.[app.executablePath] !== undefined ? <img src={icons[app.executablePath]} alt="" width={32} height={32} draggable={false} />
|
||||
: <span className={css.iconPlaceholder} aria-hidden />}
|
||||
<strong>{app.name}</strong>
|
||||
</div>
|
||||
<Button size="sm" variant="outline" disabled={disabled} onClick={() => { onAction(app.id) }}>{label}</Button>
|
||||
</li>)}</ul>
|
||||
</section>
|
||||
}
|
||||
|
||||
function PermissionDescription({ application, permission, loading, t }: Pick<PermissionProps, 't'> & {
|
||||
application: CuaSystemPermissionState['launchApplication']
|
||||
permission: 'accessibility' | 'screenRecording'
|
||||
loading: boolean
|
||||
}) {
|
||||
if (!loading && application === undefined) return <p>{t(permission === 'accessibility' ? 'accessibilityUse' : 'screenRecordingUse')}</p>
|
||||
return <p>
|
||||
{t('permissionPrefix')}{' '}
|
||||
<span className={css.applicationBadge}>
|
||||
{application === undefined ? <span className={css.appPlaceholder} aria-hidden /> : <>
|
||||
{application.icon !== undefined && <img src={application.icon} alt="" width={20} height={20} draggable={false} />}
|
||||
<span>{application.name}</span>
|
||||
</>}
|
||||
</span>{' '}
|
||||
{t(permission === 'accessibility' ? 'accessibilityAction' : 'screenRecordingAction')}
|
||||
</p>
|
||||
}
|
||||
|
||||
/** @param props Host observation and explicit user actions. @returns Permission controls without automatic system prompts. */
|
||||
export function PermissionPanel(props: PermissionPanelProps) {
|
||||
const { t, refresh, setActive } = props
|
||||
const state = props.useCuaPermissions(value => value)
|
||||
const observed = state.system.value
|
||||
const apps = state.applications.value
|
||||
const busy = state.system.busy
|
||||
const appsBusy = state.applications.busy
|
||||
const loading = observed === null && (busy || state.system.error === null)
|
||||
const application = !busy && state.system.error !== null ? undefined : observed?.launchApplication
|
||||
useEffect(() => {
|
||||
setActive(true)
|
||||
window.addEventListener('focus', refresh)
|
||||
return () => { window.removeEventListener('focus', refresh) }
|
||||
}, [refresh, setActive])
|
||||
return <div className={css.panels} data-cua-settings aria-busy={state.busy || loading}>
|
||||
<section className={css.panel} aria-label={t('title')}>
|
||||
<h3>{t('title')}</h3>
|
||||
<div className={css.card}>
|
||||
<DefaultPresetSetting {...props} />
|
||||
{!busy && state.system.error !== null && <div className={css.notice} role="alert">
|
||||
<span className={css.error}>{t('queryFailed', { message: state.system.error })}</span>
|
||||
<Button size="sm" variant="outline" disabled={state.system.busy} onClick={refresh}>{t('retry')}</Button>
|
||||
</div>}
|
||||
{(['accessibility', 'screenRecording'] as const).map(permission => <div className={css.permission} key={permission}
|
||||
role="group" aria-label={t(permission)}>
|
||||
<div>
|
||||
<strong>{t(permission)}</strong>
|
||||
<PermissionDescription application={application} permission={permission} loading={loading} t={t} />
|
||||
</div>
|
||||
<span className={css.permissionStatus}>{loading ? <span className={css.statusPlaceholder} aria-hidden />
|
||||
: t(observed === null || state.system.error !== null ? 'unknown' : observed[permission] ? 'granted' : 'missing')}</span>
|
||||
<div className={css.actions}>
|
||||
{loading ? <span className={css.buttonPlaceholder} aria-hidden />
|
||||
: observed?.[permission] === false && <Button size="sm" variant="primary" disabled={busy} onClick={() => { props.request(permission) }}>{t('request')}</Button>}
|
||||
<Button size="sm" variant="outline" disabled={busy || loading} onClick={() => { props.openSettings(permission) }}>{t('settings')}</Button>
|
||||
</div>
|
||||
</div>)}
|
||||
</div>
|
||||
{observed !== null && (!observed.accessibility || !observed.screenRecording) && <p>
|
||||
{t(application === undefined ? 'permissionHintUnknown' : 'permissionHint')}
|
||||
</p>}
|
||||
</section>
|
||||
<section className={css.panel} aria-label={t('applicationsTitle')}>
|
||||
<h3>{t('applicationsTitle')}</h3>
|
||||
<div className={css.card}>
|
||||
{(['applications', 'icons'] as const).map(key => !state[key].busy && state[key].error !== null && <div className={css.notice} role="alert" key={key}>
|
||||
<span className={css.error}>{t(key === 'applications' ? 'applicationsFailed' : 'iconsFailed', { message: state[key].error })}</span>
|
||||
<Button size="sm" variant="outline" onClick={refresh}>{t('retry')}</Button>
|
||||
</div>)}
|
||||
{apps === null ? <div className={css.emptyState}><span className={css.appsPlaceholder} aria-hidden /></div> : <>
|
||||
{apps.allowedApps.length === 0 && apps.blockedApps.length === 0 && <p className={css.emptyState}>{t('noApps')}</p>}
|
||||
{apps.allowedApps.length > 0 && <Applications title={t('allowedApps')} apps={apps.allowedApps} icons={state.icons.value}
|
||||
onAction={props.revoke} label={t('revoke')} disabled={appsBusy || !apps.persistentAvailable} />}
|
||||
{apps.blockedApps.length > 0 && <Applications title={t('blockedApps')} apps={apps.blockedApps} icons={state.icons.value}
|
||||
onAction={props.retryApproval} label={t('retryApproval')} disabled={appsBusy} />}
|
||||
</>}
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
}
|
||||
|
||||
/**
|
||||
* @param props Permission observation and activation navigation callbacks.
|
||||
* @returns Setup guidance when system permissions need attention.
|
||||
*/
|
||||
export function PermissionSetup({ t, setActive, useCuaPermissions, onDismiss, onOpenDetails }: PermissionSetupProps) {
|
||||
const state = useCuaPermissions(value => value.system)
|
||||
useEffect(() => { setActive(true) }, [setActive])
|
||||
const ready = state.value !== null && state.error === null
|
||||
&& state.value.accessibility && state.value.screenRecording
|
||||
useEffect(() => { if (ready && !state.busy) onDismiss() }, [ready, state.busy, onDismiss])
|
||||
return <Modal open={!ready && (state.value !== null || (!state.busy && state.error !== null))}
|
||||
title={t('setupTitle')} closeLabel={t('close')} onClose={onDismiss}
|
||||
footer={<>
|
||||
<Button variant="ghost" onClick={onDismiss}>{t('later')}</Button>
|
||||
<Button variant="primary" data-modal-autofocus onClick={onOpenDetails}>{t('openSetup')}</Button>
|
||||
</>}>
|
||||
<p>{t(state.error === null ? 'setupBody' : 'setupQueryFailed')}</p>
|
||||
</Modal>
|
||||
}
|
||||
|
||||
/** @param props Shared operation outcome, independent of page lifetime. @returns The existing global toast presentation. */
|
||||
export function PermissionToast(props: PermissionProps) {
|
||||
const toast = props.useCuaPermissions(value => value.toast)
|
||||
if (!toast) return null
|
||||
return <Toast key={toast.id} text={toast.kind === 'failed' ? props.t('failed', { message: toast.message }) : props.t(toast.kind)}
|
||||
{...toast.kind === 'revoked' ? { tone: 'success' as const } : {}} onDone={props.dismissToast} />
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
/** One query projection for the permission page and activation guidance. */
|
||||
import { createSnapshotStore, type SnapshotStore } from '@deepseek-ai/dsh-client-store'
|
||||
import type { CuaApplicationId, CuaApplicationState, CuaSystemPermission, CuaSystemPermissionState } from '@deepseek-ai/dsh-cua/types'
|
||||
|
||||
/** Host operations available to the permission UI. */
|
||||
export interface PermissionActions {
|
||||
readSystem: () => Promise<CuaSystemPermissionState>
|
||||
readApplications: () => Promise<CuaApplicationState>
|
||||
readIcons: () => Promise<Record<string, string>>
|
||||
request: (permission: CuaSystemPermission) => Promise<void>
|
||||
openSettings: (permission: CuaSystemPermission) => Promise<void>
|
||||
revoke: (id: CuaApplicationId) => Promise<void>
|
||||
retryApproval: (id: CuaApplicationId) => Promise<void>
|
||||
}
|
||||
interface PermissionData {
|
||||
system: CuaSystemPermissionState
|
||||
applications: CuaApplicationState
|
||||
icons: Record<string, string>
|
||||
}
|
||||
/** Query failures retain the last observation without presenting it as current. */
|
||||
interface Query<T> {
|
||||
value: T | null
|
||||
busy: boolean
|
||||
error: string | null
|
||||
}
|
||||
/** Independently settled observations; the overall busy flag follows the latest refresh. */
|
||||
export type PermissionView = { [K in keyof PermissionData]: Query<PermissionData[K]> } & {
|
||||
busy: boolean
|
||||
toast: { id: number; kind: 'requested' | 'opened' | 'revoked' | 'retryAllowed' } | { id: number; kind: 'failed'; message: string } | null
|
||||
}
|
||||
/** Slot actions and the renderer-bound permission projection. */
|
||||
export interface PermissionFace {
|
||||
hooks: { cuaPermissions: SnapshotStore<PermissionView> }
|
||||
/** Pause observation during bundle changes, or resume with a fresh read. */
|
||||
setActive: (active: boolean) => void
|
||||
refresh: () => void
|
||||
request: (permission: CuaSystemPermission) => void
|
||||
openSettings: (permission: CuaSystemPermission) => void
|
||||
revoke: (id: CuaApplicationId) => void
|
||||
retryApproval: (id: CuaApplicationId) => void
|
||||
dismissToast: () => void
|
||||
}
|
||||
|
||||
/** Serializes user actions and drops superseded refresh results or results arriving after disposal. */
|
||||
export class PermissionController {
|
||||
/** Last observed permissions and the lifetime of the current refresh/action. */
|
||||
readonly state = createSnapshotStore<PermissionView>({
|
||||
system: { value: null, busy: false, error: null },
|
||||
applications: { value: null, busy: false, error: null },
|
||||
icons: { value: null, busy: false, error: null },
|
||||
busy: false, toast: null,
|
||||
})
|
||||
private tail: Promise<void> = Promise.resolve()
|
||||
private reads: Promise<void> = Promise.resolve()
|
||||
private closed = false
|
||||
private active = true
|
||||
private epoch = 0
|
||||
private sequence = 0
|
||||
/** @param actions Browser-owned Remote calls that reject on failed responses. */
|
||||
constructor(private readonly actions: PermissionActions) {}
|
||||
/**
|
||||
* Bind the permission projection to its user actions.
|
||||
* @returns Reactive state and controls shared by the page, prompt, and toast host.
|
||||
*/
|
||||
face(): PermissionFace {
|
||||
return {
|
||||
hooks: { cuaPermissions: this.state },
|
||||
setActive: (active) => {
|
||||
if (this.closed) return
|
||||
if (active !== this.active) { this.active = active; this.epoch++ }
|
||||
if (active) this.enqueue()
|
||||
else this.state.update((state) => {
|
||||
state.busy = true
|
||||
for (const key of ['system', 'applications', 'icons'] as const) {
|
||||
state[key].busy = true; state[key].error = null
|
||||
}
|
||||
})
|
||||
},
|
||||
refresh: () => { this.enqueue() },
|
||||
request: (permission) => { this.enqueue(() => this.actions.request(permission), 'requested') },
|
||||
openSettings: (permission) => { this.enqueue(() => this.actions.openSettings(permission), 'opened') },
|
||||
revoke: (id) => { this.enqueue(() => this.actions.revoke(id), 'revoked') },
|
||||
retryApproval: (id) => { this.enqueue(() => this.actions.retryApproval(id), 'retryAllowed') },
|
||||
dismissToast: () => { if (!this.closed) this.state.update((state) => { state.toast = null }) },
|
||||
}
|
||||
}
|
||||
/** Prevent publication after UI withdrawal and join the accepted Remote work. */
|
||||
async dispose(): Promise<void> { this.closed = true; await this.tail; await this.reads }
|
||||
private enqueue(action?: () => Promise<void>, kind?: 'requested' | 'opened' | 'revoked' | 'retryAllowed'): void {
|
||||
if (this.closed || !this.active) return
|
||||
const epoch = this.epoch
|
||||
const current = () => !this.closed && this.active && epoch === this.epoch
|
||||
const sequence = ++this.sequence
|
||||
this.state.update((state) => {
|
||||
state.busy = true
|
||||
for (const key of ['system', 'applications', 'icons'] as const) state[key].busy = true
|
||||
})
|
||||
this.tail = this.tail.then(async () => {
|
||||
if (!current()) return
|
||||
try {
|
||||
await action?.()
|
||||
if (!current()) return
|
||||
if (kind) this.state.update((state) => { state.toast = { id: sequence, kind } })
|
||||
} catch (error) {
|
||||
if (!current()) return
|
||||
const message = error instanceof Error ? error.message : String(error)
|
||||
this.state.update((state) => { state.toast = { id: sequence, kind: 'failed', message } })
|
||||
}
|
||||
const latest = () => current() && this.sequence === sequence
|
||||
const refresh = Promise.all([
|
||||
this.read('system', this.actions.readSystem, latest),
|
||||
this.read('applications', this.actions.readApplications, latest),
|
||||
this.read('icons', this.actions.readIcons, latest),
|
||||
]).then(() => {
|
||||
if (latest()) this.state.update((state) => { state.busy = false })
|
||||
})
|
||||
this.reads = Promise.all([this.reads, refresh]).then(() => {})
|
||||
})
|
||||
}
|
||||
|
||||
private async read<K extends keyof PermissionData>(
|
||||
key: K, read: () => Promise<PermissionData[K]>, current: () => boolean,
|
||||
): Promise<void> {
|
||||
if (!current()) return
|
||||
try {
|
||||
const value = await read()
|
||||
if (!current()) return
|
||||
this.state.update((state) => { state[key].value = value; state[key].error = null; state[key].busy = false })
|
||||
} catch (error) {
|
||||
if (!current()) return
|
||||
this.state.update((state) => {
|
||||
state[key].error = error instanceof Error ? error.message : String(error); state[key].busy = false
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
/** Bind the source-owned permission UI to its generated Remote contribution. */
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import cuaRemote from '@deepseek-ai/dsh-cua/remote'
|
||||
import { mountCuaPermissions } from './mount.ts'
|
||||
|
||||
export { inject } from './mount.ts'
|
||||
|
||||
/**
|
||||
* Activate the optional Computer Use permission controls.
|
||||
* @param ctx Browser plugin context.
|
||||
* @returns Cleanup of the UI and Remote contribution.
|
||||
*/
|
||||
export async function apply(ctx: Context): Promise<() => Promise<void>> {
|
||||
return mountCuaPermissions(ctx, cuaRemote)
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
/** Product copy for native permission onboarding and application revocation. */
|
||||
export const NS = 'cuaPermissions'
|
||||
/** English permission-page dictionary. */
|
||||
export const en = {
|
||||
title: 'Authorize Computer Use',
|
||||
applicationsTitle: 'Manage applications Computer Use can access',
|
||||
setupTitle: 'Set up Computer Use before use',
|
||||
setupBody: 'Computer Use is enabled. Before using it, switch to PTC mode and grant Accessibility and Screen Recording permissions. Open the plugin details to complete setup.',
|
||||
setupQueryFailed: 'Computer Use is enabled, but system permissions could not be checked. Open the plugin details to see the error and retry.',
|
||||
defaultTitle: 'Default mode for new sessions',
|
||||
defaultBody: 'Computer Use is available only in PTC mode.',
|
||||
defaultConfirm: 'Set to PTC', defaultSet: 'PTC is the default',
|
||||
defaultUnavailable: 'Could not read the default mode. Check it in Settings.',
|
||||
defaultReadOnly: 'Default-mode settings are read-only.',
|
||||
defaultNotSaved: 'The default mode was not saved. Try again.',
|
||||
defaultFailed: 'Could not change the default mode: {message}',
|
||||
openSetup: 'Open settings', later: 'Later', close: 'Close',
|
||||
accessibility: 'Accessibility', screenRecording: 'Screen Recording',
|
||||
accessibilityUse: 'Read interface controls, click, and type.', screenRecordingUse: 'Capture the selected application window.',
|
||||
permissionPrefix: 'In System Settings, allow',
|
||||
accessibilityAction: 'to read interface controls, click, and type.',
|
||||
screenRecordingAction: 'to capture application windows.',
|
||||
permissionHint: 'Enable the application in System Settings. Follow the macOS prompt if it names another application.',
|
||||
permissionHintUnknown: 'Enable these permissions for the application named in the macOS prompt.',
|
||||
granted: 'Allowed', missing: 'Not allowed', unknown: 'Unknown', request: 'Request permission', settings: 'Open System Settings', retry: 'Retry',
|
||||
allowedApps: 'Always-allowed applications',
|
||||
blockedApps: 'Applications not allowed', retryApproval: 'Allow asking again',
|
||||
retryAllowed: 'The next application selection will ask again; access has not been granted',
|
||||
noApps: 'No always-allowed or blocked applications.',
|
||||
revoke: 'Revoke', revoked: 'Application access revoked', requested: 'Review the macOS prompt, or open System Settings if no prompt appears', opened: 'System Settings opened',
|
||||
applicationsFailed: 'Could not load application access: {message}', iconsFailed: 'Could not load application icons: {message}',
|
||||
failed: 'Could not update permissions: {message}', queryFailed: 'Could not check permissions: {message}',
|
||||
}
|
||||
/** Keys shared by all supported locale dictionaries. */
|
||||
export type CuaPermissionLocaleKey = keyof typeof en
|
||||
/** Chinese permission-page dictionary. */
|
||||
export const zh: Record<CuaPermissionLocaleKey, string> = {
|
||||
title: '给 Computer Use 授权', applicationsTitle: '管理 Computer Use 可访问的应用', setupTitle: '使用 Computer Use 前需要设置',
|
||||
setupBody: 'Computer Use 插件已开启。使用前请切换到 PTC 模式，并授予辅助功能和屏幕录制权限。请前往插件详情页完成设置。',
|
||||
setupQueryFailed: 'Computer Use 插件已开启，但暂时无法检测系统权限。请前往插件详情页查看原因并重试。',
|
||||
defaultTitle: '新会话默认模式',
|
||||
defaultBody: 'Computer Use 仅在 PTC 模式中可用。',
|
||||
defaultConfirm: '设为 PTC', defaultSet: '已设为 PTC',
|
||||
defaultUnavailable: '无法读取默认模式，请在设置中检查',
|
||||
defaultReadOnly: '默认模式设置为只读',
|
||||
defaultNotSaved: '默认模式未保存，请重试',
|
||||
defaultFailed: '无法切换默认模式：{message}',
|
||||
openSetup: '前往设置', later: '稍后', close: '关闭',
|
||||
accessibility: '辅助功能', screenRecording: '屏幕录制',
|
||||
accessibilityUse: '读取界面控件、点击和输入文字', screenRecordingUse: '截取所选应用的窗口',
|
||||
permissionPrefix: '在系统设置中允许',
|
||||
accessibilityAction: '读取界面、点击和输入',
|
||||
screenRecordingAction: '截取应用窗口',
|
||||
permissionHint: '在系统设置中开启对应应用；若 macOS 提示其他应用，以系统提示为准',
|
||||
permissionHintUnknown: '在系统设置中，为 macOS 提示的应用开启对应权限',
|
||||
granted: '已允许', missing: '未允许', unknown: '未确认', request: '申请权限', settings: '打开系统设置', retry: '重试',
|
||||
allowedApps: '始终允许的应用',
|
||||
blockedApps: '不允许访问的应用', retryApproval: '允许再次询问',
|
||||
retryAllowed: '下次选择此应用时将重新询问，当前尚未授权',
|
||||
noApps: '暂无始终允许或不允许访问的应用。',
|
||||
revoke: '撤销', revoked: '已撤销应用访问权限', requested: '请查看 macOS 提示；若未出现，请打开系统设置', opened: '已打开系统设置',
|
||||
applicationsFailed: '无法读取应用授权：{message}', iconsFailed: '无法读取应用图标：{message}',
|
||||
failed: '无法更新权限：{message}', queryFailed: '无法检测权限：{message}',
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
/** Optional Computer Use controls mounted through existing plugin-page and overlay slots. */
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import type {} from '@deepseek-ai/dsh-cua/remote'
|
||||
import type { TypertRemoteContribution } from '@deepseek-ai/dsh-typert-protocol'
|
||||
import type {} from '@deepseek-ai/dsh-api-remotes/client'
|
||||
import type {} from '@deepseek-ai/dsh-client-locale/client'
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-renderer/client'
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-layout/client'
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-plugin-manager/client'
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-settings/client'
|
||||
import { PermissionController } from './controller.ts'
|
||||
import { PermissionPanel, PermissionSetup, PermissionToast } from './PermissionPanel.tsx'
|
||||
import { en, NS, zh, type CuaPermissionLocaleKey } from './locales.ts'
|
||||
|
||||
declare module '@deepseek-ai/dsh-client-ui-slots' {
|
||||
interface LocaleNamespaceMap {
|
||||
/** Native permission controls and application consent. */
|
||||
cuaPermissions: CuaPermissionLocaleKey
|
||||
}
|
||||
}
|
||||
|
||||
/** Existing browser registries needed by the optional permission UI. */
|
||||
export const inject = ['remote', 'slots', 'locale', 'configForms']
|
||||
|
||||
/**
|
||||
* Mount the optional permission Remote and its browser controls.
|
||||
* @param ctx Browser plugin context.
|
||||
* @param contribution Generated permission Remote definitions.
|
||||
* @returns Cleanup for the optional Remote and UI registrations.
|
||||
*/
|
||||
export async function mountCuaPermissions(ctx: Context, contribution: TypertRemoteContribution): Promise<() => Promise<void>> {
|
||||
const transport = ctx.plugin(async (owner) => { await owner.remote.$mount(contribution) })
|
||||
const ui = ctx.inject(['remote.cuaAccess', 'slots', 'locale'], (ctx) => {
|
||||
ctx.effect(() => ctx.locale.register(NS, { en, zh }))
|
||||
const controller = new PermissionController({
|
||||
readSystem: async () => {
|
||||
const result = await ctx.remote.cuaAccess.systemStatus(); if (!result.ok) throw result.error; return result.value
|
||||
},
|
||||
readApplications: async () => {
|
||||
const result = await ctx.remote.cuaAccess.applications(); if (!result.ok) throw result.error; return result.value
|
||||
},
|
||||
readIcons: async () => {
|
||||
const result = await ctx.remote.cuaAccess.applicationIcons(); if (!result.ok) throw result.error; return result.value
|
||||
},
|
||||
request: async (permission) => {
|
||||
const result = await ctx.remote.cuaAccess.requestPermission(permission); if (!result.ok) throw result.error
|
||||
},
|
||||
openSettings: async (permission) => {
|
||||
const result = await ctx.remote.cuaAccess.openSettings(permission); if (!result.ok) throw result.error
|
||||
},
|
||||
revoke: async (id) => { const result = await ctx.remote.cuaAccess.revoke(id); if (!result.ok) throw result.error },
|
||||
retryApproval: async (id) => { const result = await ctx.remote.cuaAccess.retryApproval(id); if (!result.ok) throw result.error },
|
||||
})
|
||||
const face = controller.face()
|
||||
const preset = ctx.configForms.get<{ default: string; selectedDefault?: string }>('agent-preset-registry')
|
||||
const settings = {
|
||||
...face,
|
||||
hooks: { ...face.hooks, cuaDefaultPreset: preset },
|
||||
setPtcDefault: () => preset.set('selectedDefault', 'ptc'),
|
||||
}
|
||||
ctx.effect(() => () => controller.dispose())
|
||||
ctx.on('connection/reset', face.refresh)
|
||||
ctx.effect(() => ctx.remote.$on('settings/document-updated', (namespace) => {
|
||||
if (namespace === controller.state.getSnapshot().applications.value?.settingsNamespace) face.refresh()
|
||||
}))
|
||||
ctx.slots.inject('plugins.bundle.config', () => ctx.slots.register({
|
||||
name: 'plugins.bundle.config', key: '@deepseek-ai/dsh-cua-bundle', locale: NS, inject: () => settings,
|
||||
}, PermissionPanel))
|
||||
ctx.slots.inject('plugins.bundle.activation', () => ctx.slots.register({
|
||||
name: 'plugins.bundle.activation', key: '@deepseek-ai/dsh-cua-bundle', locale: NS, inject: () => face,
|
||||
}, PermissionSetup))
|
||||
ctx.slots.inject('shell.overlay', () => ctx.slots.register({
|
||||
name: 'shell.overlay', id: 'cua-permission-outcome', locale: NS, inject: () => face,
|
||||
}, PermissionToast))
|
||||
})
|
||||
const dispose = ctx.effect(function* () {
|
||||
yield transport.dispose
|
||||
yield ui.dispose
|
||||
}, 'cua-permissions.mount')
|
||||
try { await Promise.all([transport, ui]) } catch (error) { await dispose(); throw error }
|
||||
return dispose
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
declare module '*.module.css' {
|
||||
const classes: Record<string, string>
|
||||
export default classes
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
/** Host marker; the client export owns the optional permission UI, while dsh-cua owns native policy. */
|
||||
export function apply(): void {}
|
||||
@@ -0,0 +1,253 @@
|
||||
/** Permission refreshes preserve observations and join user actions before withdrawal. */
|
||||
import { setImmediate } from 'node:timers/promises'
|
||||
import { expect, it, onTestFinished, vi } from 'vitest'
|
||||
import type { CuaApplicationId, CuaApplicationState, CuaSystemPermissionState } from '@deepseek-ai/dsh-cua/types'
|
||||
import { PermissionController } from '../src/client/controller.ts'
|
||||
|
||||
const value: CuaSystemPermissionState = { accessibility: true, screenRecording: false, helper: { pid: 9 }, parent: { pid: 8 } }
|
||||
const applications: CuaApplicationState = {
|
||||
allowedApps: [], blockedApps: [], settingsNamespace: 'cua', persistentAvailable: true }
|
||||
function fixture() {
|
||||
const actions = { readSystem: vi.fn(async () => value), readApplications: vi.fn(async () => applications),
|
||||
readIcons: vi.fn(async (): Promise<Record<string, string>> => ({})), request: vi.fn(async () => {}),
|
||||
openSettings: vi.fn(async () => {}), revoke: vi.fn(async () => {}), retryApproval: vi.fn(async () => {}) }
|
||||
const controller = new PermissionController(actions)
|
||||
return { actions, controller, face: controller.face() }
|
||||
}
|
||||
|
||||
it('refreshes after a user request without treating the request itself as a grant', async () => {
|
||||
const f = fixture()
|
||||
f.face.request('screenRecording')
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) })
|
||||
expect(f.actions.request).toHaveBeenCalledWith('screenRecording')
|
||||
expect(f.controller.state.getSnapshot().system.value?.screenRecording).toBe(false)
|
||||
expect(f.controller.state.getSnapshot().toast?.kind).toBe('requested')
|
||||
f.face.dismissToast()
|
||||
expect(f.controller.state.getSnapshot().toast).toBeNull()
|
||||
await f.controller.dispose()
|
||||
})
|
||||
|
||||
it('retains the last observation while marking a failed refresh', async () => {
|
||||
const f = fixture()
|
||||
f.face.refresh()
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().system.value).toEqual(value) })
|
||||
f.actions.readSystem.mockRejectedValueOnce(new Error('connection closed'))
|
||||
f.face.refresh()
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().system.error).toBe('connection closed') })
|
||||
expect(f.controller.state.getSnapshot().system.value).toEqual(value)
|
||||
await f.controller.dispose()
|
||||
})
|
||||
|
||||
it.each([new Error('write refused'), 'write refused'])('reports settings and revocation outcomes through the shared toast state', async (error) => {
|
||||
const f = fixture()
|
||||
f.face.openSettings('accessibility')
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast?.kind).toBe('opened') })
|
||||
f.face.revoke('a'.repeat(64) as CuaApplicationId)
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast?.kind).toBe('revoked') })
|
||||
f.face.retryApproval('a'.repeat(64) as CuaApplicationId)
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast?.kind).toBe('retryAllowed') })
|
||||
f.actions.revoke.mockRejectedValueOnce(error)
|
||||
f.face.revoke('a'.repeat(64) as CuaApplicationId)
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast).toMatchObject({ kind: 'failed', message: 'write refused' }) })
|
||||
await f.controller.dispose()
|
||||
})
|
||||
|
||||
it('joins a pending read without publishing after disposal or starting queued actions', async () => {
|
||||
const f = fixture()
|
||||
const pending = Promise.withResolvers<CuaSystemPermissionState>()
|
||||
const entered = Promise.withResolvers<undefined>()
|
||||
f.actions.readSystem.mockImplementationOnce(() => { entered.resolve(undefined); return pending.promise })
|
||||
f.face.refresh()
|
||||
await entered.promise
|
||||
f.face.request('accessibility')
|
||||
const before = f.controller.state.getSnapshot()
|
||||
const closed = f.controller.dispose()
|
||||
pending.resolve(value)
|
||||
await closed
|
||||
expect(f.controller.state.getSnapshot()).toBe(before)
|
||||
expect(f.actions.request).not.toHaveBeenCalled()
|
||||
f.face.refresh()
|
||||
expect(f.actions.readSystem).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('revokes access while icons are pending, serializes writes, and joins reads on disposal', async () => {
|
||||
const f = fixture(), icons = Promise.withResolvers<Record<string, string>>()
|
||||
const action = Promise.withResolvers<undefined>(), entered = Promise.withResolvers<undefined>()
|
||||
onTestFinished(async () => { icons.resolve({}); action.resolve(undefined); await f.controller.dispose() })
|
||||
f.actions.readIcons.mockReturnValueOnce(icons.promise)
|
||||
f.actions.revoke.mockImplementationOnce(() => { entered.resolve(undefined); return action.promise })
|
||||
f.face.refresh()
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().applications.busy).toBe(false) })
|
||||
expect(f.controller.state.getSnapshot().icons.busy).toBe(true)
|
||||
f.face.revoke('a'.repeat(64) as CuaApplicationId)
|
||||
await entered.promise
|
||||
f.face.retryApproval('a'.repeat(64) as CuaApplicationId)
|
||||
await setImmediate()
|
||||
expect(f.actions.retryApproval).not.toHaveBeenCalled()
|
||||
action.resolve(undefined)
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) })
|
||||
expect(f.actions.retryApproval).toHaveBeenCalledOnce()
|
||||
expect(f.controller.state.getSnapshot().toast?.kind).toBe('retryAllowed')
|
||||
const before = f.controller.state.getSnapshot()
|
||||
let disposed = false
|
||||
const closed = f.controller.dispose().then(() => { disposed = true })
|
||||
await setImmediate()
|
||||
expect(disposed).toBe(false)
|
||||
icons.resolve({ '/old/app': 'old-icon' })
|
||||
await closed
|
||||
expect(f.controller.state.getSnapshot()).toBe(before)
|
||||
})
|
||||
|
||||
it('ignores stale toast completion after the permission UI is disposed', async () => {
|
||||
const f = fixture()
|
||||
f.face.request('accessibility')
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast?.kind).toBe('requested') })
|
||||
await f.controller.dispose()
|
||||
const before = f.controller.state.getSnapshot()
|
||||
f.face.dismissToast()
|
||||
f.face.setActive(true)
|
||||
expect(f.controller.state.getSnapshot()).toBe(before)
|
||||
})
|
||||
|
||||
it('contains a late read rejection and supports errors without an Error instance', async () => {
|
||||
const f = fixture()
|
||||
f.actions.readSystem.mockRejectedValueOnce('offline')
|
||||
f.face.refresh()
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().system.error).toBe('offline') })
|
||||
const pending = Promise.withResolvers<CuaSystemPermissionState>(), entered = Promise.withResolvers<undefined>()
|
||||
f.actions.readSystem.mockImplementationOnce(() => { entered.resolve(undefined); return pending.promise })
|
||||
f.face.refresh()
|
||||
await entered.promise
|
||||
const before = f.controller.state.getSnapshot(), closed = f.controller.dispose()
|
||||
pending.reject(new Error('late disconnect'))
|
||||
await closed
|
||||
expect(f.controller.state.getSnapshot()).toBe(before)
|
||||
})
|
||||
|
||||
|
||||
it.each(['success', 'failure'])('does not publish an older refresh %s while the activation refresh is queued', async (outcome) => {
|
||||
const f = fixture()
|
||||
const first = Promise.withResolvers<CuaSystemPermissionState>(), firstEntered = Promise.withResolvers<undefined>()
|
||||
const latest = Promise.withResolvers<CuaSystemPermissionState>(), latestEntered = Promise.withResolvers<undefined>()
|
||||
onTestFinished(async () => { first.resolve(value); latest.resolve(value); await f.controller.dispose() })
|
||||
f.actions.readSystem.mockImplementationOnce(() => { firstEntered.resolve(undefined); return first.promise })
|
||||
f.actions.readSystem.mockImplementationOnce(() => { latestEntered.resolve(undefined); return latest.promise })
|
||||
const observations: Array<ReturnType<typeof f.controller.state.getSnapshot>> = []
|
||||
const unsubscribe = f.controller.state.subscribe(() => { observations.push(f.controller.state.getSnapshot()) })
|
||||
onTestFinished(unsubscribe)
|
||||
f.face.refresh()
|
||||
await firstEntered.promise
|
||||
f.face.refresh()
|
||||
if (outcome === 'failure') first.reject(new Error('provider is reloading'))
|
||||
else first.resolve({ ...value, screenRecording: true })
|
||||
await latestEntered.promise
|
||||
expect(observations.every(state => state.system.error === null && state.system.value === null && state.busy)).toBe(true)
|
||||
latest.resolve(value)
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot()).toMatchObject({ busy: false, system: { error: null, value } }) })
|
||||
expect(observations.every(state => state.system.error === null)).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps a user-action failure visible when a refresh is queued behind it', async () => {
|
||||
const f = fixture(), action = Promise.withResolvers<undefined>(), entered = Promise.withResolvers<undefined>()
|
||||
onTestFinished(async () => { action.resolve(undefined); await f.controller.dispose() })
|
||||
f.actions.request.mockImplementationOnce(() => { entered.resolve(undefined); return action.promise })
|
||||
f.face.request('accessibility')
|
||||
await entered.promise
|
||||
f.face.refresh()
|
||||
action.reject(new Error('permission request failed'))
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) })
|
||||
expect(f.controller.state.getSnapshot()).toMatchObject({ system: { value, error: null },
|
||||
toast: { kind: 'failed', message: 'permission request failed' } })
|
||||
})
|
||||
|
||||
|
||||
it.each(['success', 'failure'])('drops a permission read %s from the previous bundle activation', async (outcome) => {
|
||||
const f = fixture()
|
||||
f.face.setActive(true)
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) })
|
||||
const pending = Promise.withResolvers<CuaSystemPermissionState>(), entered = Promise.withResolvers<undefined>()
|
||||
onTestFinished(async () => { pending.resolve(value); await f.controller.dispose() })
|
||||
f.actions.readSystem.mockImplementationOnce(() => { entered.resolve(undefined); return pending.promise })
|
||||
f.face.refresh()
|
||||
await entered.promise
|
||||
f.face.setActive(false)
|
||||
f.face.setActive(false)
|
||||
expect(f.controller.state.getSnapshot()).toMatchObject({ busy: true, system: { error: null, value } })
|
||||
f.face.request('accessibility')
|
||||
expect(f.actions.request).not.toHaveBeenCalled()
|
||||
if (outcome === 'failure') pending.reject(new Error('old helper stopped'))
|
||||
else pending.resolve({ ...value, screenRecording: true })
|
||||
await Promise.allSettled([pending.promise])
|
||||
expect(f.controller.state.getSnapshot()).toMatchObject({ busy: true, system: { error: null, value } })
|
||||
f.face.setActive(true)
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) })
|
||||
expect(f.controller.state.getSnapshot()).toMatchObject({ system: { error: null, value } })
|
||||
expect(f.actions.readSystem).toHaveBeenCalledTimes(3)
|
||||
})
|
||||
|
||||
it.each(['success', 'failure'])('does not publish a user-action %s after its bundle is replaced', async (outcome) => {
|
||||
const f = fixture(), pending = Promise.withResolvers<undefined>(), entered = Promise.withResolvers<undefined>()
|
||||
onTestFinished(async () => { pending.resolve(undefined); await f.controller.dispose() })
|
||||
f.actions.request.mockImplementationOnce(() => { entered.resolve(undefined); return pending.promise })
|
||||
f.face.request('accessibility')
|
||||
await entered.promise
|
||||
f.face.setActive(false)
|
||||
if (outcome === 'failure') pending.reject(new Error('old request cancelled'))
|
||||
else pending.resolve(undefined)
|
||||
f.face.setActive(true)
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) })
|
||||
expect(f.controller.state.getSnapshot()).toMatchObject({ toast: null, system: { error: null, value } })
|
||||
expect(f.actions.readSystem).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('publishes each observation before the other queries finish', async () => {
|
||||
const f = fixture()
|
||||
const system = Promise.withResolvers<CuaSystemPermissionState>()
|
||||
const apps = Promise.withResolvers<CuaApplicationState>()
|
||||
const icons = Promise.withResolvers<Record<string, string>>()
|
||||
onTestFinished(async () => { system.resolve(value); apps.resolve(applications); icons.resolve({}); await f.controller.dispose() })
|
||||
f.actions.readSystem.mockReturnValueOnce(system.promise)
|
||||
f.actions.readApplications.mockReturnValueOnce(apps.promise)
|
||||
f.actions.readIcons.mockReturnValueOnce(icons.promise)
|
||||
f.face.refresh()
|
||||
apps.resolve(applications)
|
||||
await vi.waitFor(() => {
|
||||
expect(f.controller.state.getSnapshot().applications).toEqual({ value: applications, error: null, busy: false })
|
||||
})
|
||||
expect(f.controller.state.getSnapshot().system).toMatchObject({ value: null, busy: true })
|
||||
system.resolve(value)
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().system).toEqual({ value, error: null, busy: false }) })
|
||||
expect(f.controller.state.getSnapshot().icons).toMatchObject({ value: null, busy: true })
|
||||
icons.reject(new Error('icon unavailable'))
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) })
|
||||
expect(f.controller.state.getSnapshot()).toMatchObject({
|
||||
system: { value, error: null }, applications: { value: applications, error: null },
|
||||
icons: { value: null, error: 'icon unavailable', busy: false },
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps application decisions available when system inspection fails', async () => {
|
||||
const f = fixture()
|
||||
f.actions.readSystem.mockRejectedValueOnce(new Error('helper offline'))
|
||||
f.face.refresh()
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) })
|
||||
expect(f.controller.state.getSnapshot()).toMatchObject({
|
||||
system: { value: null, error: 'helper offline' },
|
||||
applications: { value: applications, error: null, busy: false },
|
||||
})
|
||||
await f.controller.dispose()
|
||||
})
|
||||
|
||||
it.each(['success', 'failure'])('drops old icon %s after disabling and reenabling the bundle', async (outcome) => {
|
||||
const f = fixture(), pending = Promise.withResolvers<Record<string, string>>()
|
||||
onTestFinished(async () => { pending.resolve({}); await f.controller.dispose() })
|
||||
f.actions.readIcons.mockReturnValueOnce(pending.promise)
|
||||
f.face.refresh()
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().system.busy).toBe(false) })
|
||||
f.face.setActive(false)
|
||||
f.face.setActive(true)
|
||||
if (outcome === 'success') pending.resolve({ '/old/app': 'old-icon' })
|
||||
else pending.reject(new Error('old helper stopped'))
|
||||
await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) })
|
||||
expect(f.controller.state.getSnapshot().icons).toEqual({ value: {}, error: null, busy: false })
|
||||
})
|
||||
@@ -0,0 +1,181 @@
|
||||
/** The optional permission namespace and existing page slots share one disposal lifetime. */
|
||||
import assert from 'node:assert/strict'
|
||||
import { setImmediate } from 'node:timers/promises'
|
||||
import { Context, Service } from '@deepseek-ai/cordis'
|
||||
import { LocaleRuntime } from '@deepseek-ai/dsh-client-locale/client'
|
||||
import { SlotRegistry } from '@deepseek-ai/dsh-client-ui-renderer/client'
|
||||
import { createSnapshotStore } from '@deepseek-ai/dsh-client-store'
|
||||
import { RemoteError, type RemoteResult, type TypertRemoteContribution } from '@deepseek-ai/dsh-typert-protocol'
|
||||
import type { CuaApplicationState, CuaSystemPermissionState } from '@deepseek-ai/dsh-cua/types'
|
||||
import { expect, it, vi } from 'vitest'
|
||||
import type { PermissionFace } from '../src/client/controller.ts'
|
||||
import { mountCuaPermissions, inject } from '../src/client/mount.ts'
|
||||
import { apply as hostApply } from '../src/index.ts'
|
||||
|
||||
const contribution: TypertRemoteContribution = { package: '@deepseek-ai/dsh-cua', descriptors: [] }
|
||||
const value: CuaApplicationState & { system: CuaSystemPermissionState } = {
|
||||
system: { accessibility: false, screenRecording: false, helper: { pid: 2 }, parent: { pid: 1 } },
|
||||
allowedApps: [], blockedApps: [], settingsNamespace: 'cua', persistentAvailable: true }
|
||||
|
||||
function actionsOf(value: Record<string, unknown>): asserts value is Record<string, unknown> & PermissionFace {
|
||||
assert(typeof value.setActive === 'function')
|
||||
assert(typeof value.refresh === 'function' && typeof value.request === 'function' && typeof value.openSettings === 'function')
|
||||
assert(typeof value.revoke === 'function' && typeof value.dismissToast === 'function')
|
||||
assert(typeof value.retryApproval === 'function')
|
||||
assert(typeof value.hooks === 'object' && value.hooks !== null)
|
||||
}
|
||||
|
||||
async function fixture(fail = false) {
|
||||
const ctx = new Context(), unmount = vi.fn(async () => {}), unsubscribe = vi.fn()
|
||||
let settingsListener: ((namespace: string, revision: number) => void) | undefined
|
||||
class Remote extends Service {
|
||||
constructor() { super(ctx, 'remote') }
|
||||
async $mount(received: TypertRemoteContribution) {
|
||||
expect(received).toBe(contribution)
|
||||
const owned = this.ctx.effect(() => unmount)
|
||||
return () => owned()
|
||||
}
|
||||
$on(_event: string, listener: (namespace: string, revision: number) => void) { settingsListener = listener; return unsubscribe }
|
||||
}
|
||||
new Remote()
|
||||
const setDefault = vi.fn(async () => true)
|
||||
const preset = createSnapshotStore({ status: 'ready' as const, value: { default: 'standard' },
|
||||
base: {}, user: {}, revision: 1, writable: true })
|
||||
ctx.provide('configForms', { get: (namespace: string) => {
|
||||
expect(namespace).toBe('agent-preset-registry')
|
||||
return { ...preset, set: setDefault }
|
||||
} })
|
||||
const status = vi.fn<() => Promise<RemoteResult<CuaApplicationState>>>().mockResolvedValue({ ok: true, value })
|
||||
const systemStatus = vi.fn<() => Promise<RemoteResult<CuaSystemPermissionState>>>().mockResolvedValue({ ok: true, value: value.system })
|
||||
const applicationIcons = vi.fn<() => Promise<RemoteResult<Record<string, string>>>>().mockResolvedValue({ ok: true, value: {} })
|
||||
const requestPermission = vi.fn<() => Promise<RemoteResult<CuaSystemPermissionState>>>()
|
||||
.mockResolvedValue({ ok: true, value: value.system })
|
||||
const openSettings = vi.fn<() => Promise<RemoteResult<void>>>().mockResolvedValue({ ok: true, value: undefined })
|
||||
const revoke = vi.fn<() => Promise<RemoteResult<void>>>().mockResolvedValue({ ok: true, value: undefined })
|
||||
const retryApproval = vi.fn<() => Promise<RemoteResult<void>>>().mockResolvedValue({ ok: true, value: undefined })
|
||||
ctx.provide('remote.cuaAccess', { applications: status, systemStatus, applicationIcons, requestPermission, openSettings, revoke, retryApproval })
|
||||
ctx.provide('locale', new LocaleRuntime(ctx))
|
||||
await ctx.plugin(SlotRegistry)
|
||||
ctx.slots.register({ name: 'root', children: {
|
||||
'plugins.bundle.activation': { kind: 'keyed', scope: 'root' },
|
||||
'plugins.bundle.config': { kind: 'keyed', scope: 'root' }, 'shell.overlay': { kind: 'list', scope: 'root' },
|
||||
} } as never, () => null)
|
||||
if (fail) vi.spyOn(ctx.slots, 'inject').mockImplementationOnce(() => { throw new Error('slot unavailable') })
|
||||
return { ctx, unmount, unsubscribe, status, systemStatus, applicationIcons,
|
||||
requestPermission, openSettings, revoke, retryApproval, setDefault,
|
||||
updateSettings(namespace: string) { settingsListener!(namespace, 1) },
|
||||
}
|
||||
}
|
||||
|
||||
it('withdraws the permission Remote and all controls with the optional plugin', async () => {
|
||||
hostApply()
|
||||
const f = await fixture()
|
||||
try {
|
||||
const fiber = f.ctx.plugin({ inject: [...inject], apply: ctx => mountCuaPermissions(ctx, contribution) })
|
||||
await fiber
|
||||
const entry = f.ctx.slots.entries('plugins.bundle.config')[0]!
|
||||
const actions = entry.inject!()
|
||||
actionsOf(actions)
|
||||
actions.refresh()
|
||||
await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().applications.value).toEqual(value) })
|
||||
actions.request('accessibility')
|
||||
await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().toast?.kind).toBe('requested') })
|
||||
actions.openSettings('screenRecording')
|
||||
await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().toast?.kind).toBe('opened') })
|
||||
actions.revoke('a'.repeat(64) as import('@deepseek-ai/dsh-cua/types').CuaApplicationId)
|
||||
await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().toast?.kind).toBe('revoked') })
|
||||
actions.retryApproval('a'.repeat(64) as import('@deepseek-ai/dsh-cua/types').CuaApplicationId)
|
||||
await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().toast?.kind).toBe('retryAllowed') })
|
||||
const failed = { ok: false as const, error: new RemoteError('gateway/internal', 'offline', {}) }
|
||||
for (const [target, invoke] of [
|
||||
[f.status, actions.refresh], [f.systemStatus, actions.refresh], [f.applicationIcons, actions.refresh], [f.requestPermission, () => { actions.request('accessibility') }],
|
||||
[f.openSettings, () => { actions.openSettings('accessibility') }],
|
||||
[f.revoke, () => { actions.revoke('a'.repeat(64) as import('@deepseek-ai/dsh-cua/types').CuaApplicationId) }],
|
||||
[f.retryApproval, () => { actions.retryApproval('a'.repeat(64) as import('@deepseek-ai/dsh-cua/types').CuaApplicationId) }],
|
||||
] as const) {
|
||||
actions.refresh()
|
||||
await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().busy).toBe(false) })
|
||||
target.mockResolvedValueOnce(failed)
|
||||
invoke()
|
||||
await vi.waitFor(() => {
|
||||
const state = actions.hooks.cuaPermissions.getSnapshot()
|
||||
const error = target === f.status ? state.applications.error : target === f.systemStatus ? state.system.error
|
||||
: target === f.applicationIcons ? state.icons.error : state.toast?.kind === 'failed' ? state.toast.message : null
|
||||
expect(error).toContain('offline')
|
||||
})
|
||||
}
|
||||
for (const name of ['plugins.bundle.activation', 'shell.overlay'] as const) {
|
||||
const other = f.ctx.slots.entries(name)[0]!
|
||||
const otherActions = other.inject!()
|
||||
actionsOf(otherActions)
|
||||
expect(otherActions.hooks.cuaPermissions).toBe(actions.hooks.cuaPermissions)
|
||||
}
|
||||
const settings = f.ctx.slots.entries('plugins.bundle.config')[0]!.inject!()
|
||||
assert(typeof settings.setPtcDefault === 'function')
|
||||
expect(f.ctx.slots.entries('plugins.bundle.activation')[0]!.inject!()).not.toHaveProperty('setPtcDefault')
|
||||
expect(f.setDefault).not.toHaveBeenCalled()
|
||||
await (settings.setPtcDefault as () => Promise<boolean>)()
|
||||
expect(f.setDefault).toHaveBeenCalledExactlyOnceWith('selectedDefault', 'ptc')
|
||||
await fiber.dispose()
|
||||
expect(f.ctx.slots.entries('conversation.composer')).toHaveLength(0)
|
||||
expect(f.ctx.slots.entries('plugins.bundle.config')).toHaveLength(0)
|
||||
expect(f.ctx.slots.entries('shell.overlay')).toHaveLength(0)
|
||||
expect(f.unsubscribe).toHaveBeenCalledOnce()
|
||||
expect(f.unmount).toHaveBeenCalledOnce()
|
||||
} finally { await f.ctx.fiber.dispose() }
|
||||
})
|
||||
|
||||
it('withdraws the Remote contribution when UI registration fails', async () => {
|
||||
const f = await fixture(true)
|
||||
try {
|
||||
await expect(mountCuaPermissions(f.ctx, contribution)).rejects.toThrow('slot unavailable')
|
||||
expect(f.unmount).toHaveBeenCalledOnce()
|
||||
} finally { await f.ctx.fiber.dispose() }
|
||||
})
|
||||
|
||||
it.each(['plugin', 'root'])('keeps the Remote mounted until an accepted revoke completes during %s disposal', async (owner) => {
|
||||
const f = await fixture()
|
||||
const response = Promise.withResolvers<RemoteResult<void>>(), entered = Promise.withResolvers<undefined>()
|
||||
f.revoke.mockImplementationOnce(() => { entered.resolve(undefined); return response.promise })
|
||||
f.unmount.mockImplementation(async () => { response.resolve({ ok: false, error: new RemoteError('gateway/internal', 'unmounted', {}) }) })
|
||||
let closing: Promise<void> | undefined
|
||||
try {
|
||||
const fiber = await f.ctx.plugin({ inject: [...inject], apply: ctx => mountCuaPermissions(ctx, contribution) })
|
||||
const actions = f.ctx.slots.entries('plugins.bundle.config')[0]!.inject!()
|
||||
actionsOf(actions)
|
||||
actions.revoke('a'.repeat(64) as import('@deepseek-ai/dsh-cua/types').CuaApplicationId)
|
||||
await entered.promise
|
||||
let closed = false
|
||||
closing = Promise.resolve(owner === 'plugin' ? fiber.dispose() : f.ctx.fiber.dispose()).then(() => { closed = true })
|
||||
await setImmediate()
|
||||
expect(f.unmount).not.toHaveBeenCalled()
|
||||
expect(closed).toBe(false)
|
||||
response.resolve({ ok: true, value: undefined })
|
||||
await closing
|
||||
expect(f.unmount).toHaveBeenCalledOnce()
|
||||
} finally {
|
||||
response.resolve({ ok: true, value: undefined })
|
||||
await closing
|
||||
await f.ctx.fiber.dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('refreshes only the settings namespace reported by the permission owner', async () => {
|
||||
const f = await fixture()
|
||||
f.status.mockResolvedValue({ ok: true, value: { ...value, settingsNamespace: 'custom-cua-entry' } })
|
||||
try {
|
||||
await f.ctx.plugin({ inject: [...inject], apply: ctx => mountCuaPermissions(ctx, contribution) })
|
||||
const actions = f.ctx.slots.entries('plugins.bundle.config')[0]!.inject!()
|
||||
actionsOf(actions)
|
||||
f.updateSettings('locale')
|
||||
expect(f.status).not.toHaveBeenCalled()
|
||||
actions.refresh()
|
||||
await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().busy).toBe(false) })
|
||||
expect(f.status).toHaveBeenCalledOnce()
|
||||
f.updateSettings('cua')
|
||||
f.updateSettings('locale')
|
||||
expect(f.status).toHaveBeenCalledOnce()
|
||||
f.updateSettings('custom-cua-entry')
|
||||
await vi.waitFor(() => { expect(f.status).toHaveBeenCalledTimes(2) })
|
||||
} finally { await f.ctx.fiber.dispose() }
|
||||
})
|
||||
@@ -0,0 +1,332 @@
|
||||
// @vitest-environment jsdom
|
||||
/** Permission UI distinguishes missing privileges, uncertainty, and application consent. */
|
||||
import { act, cleanup, fireEvent, render, screen, waitFor, within } from '@testing-library/react'
|
||||
import { bindSnapshotSelector, makeTranslate } from '@deepseek-ai/dsh-client-test-runtime'
|
||||
import { createSnapshotStore } from '@deepseek-ai/dsh-client-store'
|
||||
import type { CuaApplicationId, CuaApplicationState, CuaSystemPermissionState } from '@deepseek-ai/dsh-cua/types'
|
||||
import type { ConfigFormSnapshot } from '@deepseek-ai/dsh-client-ui-settings/client'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import { PermissionPanel, PermissionSetup, PermissionToast, type PermissionPanelProps } from '../src/client/PermissionPanel.tsx'
|
||||
import type { PermissionView } from '../src/client/controller.ts'
|
||||
import { en, zh } from '../src/client/locales.ts'
|
||||
|
||||
afterEach(() => { cleanup(); vi.useRealTimers() })
|
||||
const value: CuaApplicationState & { system: CuaSystemPermissionState } = { system: { accessibility: false, screenRecording: true,
|
||||
helper: { pid: 9, path: '/native/helper' }, parent: { pid: 8, path: '/host/node' } },
|
||||
allowedApps: [], blockedApps: [], settingsNamespace: 'cua', persistentAvailable: true }
|
||||
|
||||
function observe(value: CuaApplicationState & { system: CuaSystemPermissionState }) {
|
||||
return { system: { value: value.system, busy: false, error: null },
|
||||
applications: { value, busy: false, error: null }, icons: { value: {}, busy: false, error: null }, busy: false }
|
||||
}
|
||||
|
||||
function fixture() {
|
||||
const state = createSnapshotStore<PermissionView>({
|
||||
system: { value: null, busy: true, error: null }, applications: { value: null, busy: true, error: null },
|
||||
icons: { value: null, busy: true, error: null }, busy: true, toast: null,
|
||||
})
|
||||
const preset = createSnapshotStore<ConfigFormSnapshot<{ default: string; selectedDefault?: string }>>({
|
||||
status: 'ready', value: { default: 'standard', selectedDefault: 'ptc' },
|
||||
base: {}, user: {}, revision: 1, writable: true,
|
||||
})
|
||||
const refresh = vi.fn()
|
||||
const props = { useCuaPermissions: bindSnapshotSelector(state), t: makeTranslate(en),
|
||||
useCuaDefaultPreset: bindSnapshotSelector(preset), setPtcDefault: vi.fn(async () => true),
|
||||
refresh, setActive: vi.fn((active: boolean) => { if (active) refresh() }), request: vi.fn(), openSettings: vi.fn(), revoke: vi.fn(),
|
||||
retryApproval: vi.fn(), dismissToast: vi.fn(),
|
||||
} satisfies PermissionPanelProps
|
||||
return { state, preset, props,
|
||||
publish(patch: Partial<PermissionView>) { act(() => { state.set({ ...state.getSnapshot(), ...patch }) }) },
|
||||
}
|
||||
}
|
||||
|
||||
it('requests only a missing privilege and leaves OS attribution to the system prompt', () => {
|
||||
const f = fixture()
|
||||
const shown = render(<PermissionPanel {...f.props} />)
|
||||
expect(f.props.refresh).toHaveBeenCalledOnce()
|
||||
f.publish({ ...observe(value) })
|
||||
fireEvent.click(screen.getByRole('button', { name: en.request }))
|
||||
expect(f.props.request).toHaveBeenCalledWith('accessibility')
|
||||
const controls = screen.getAllByRole('button', { name: en.settings })
|
||||
fireEvent.click(controls[0]!)
|
||||
fireEvent.click(controls[1]!)
|
||||
expect(f.props.openSettings.mock.calls).toEqual([['accessibility'], ['screenRecording']])
|
||||
fireEvent.focus(window)
|
||||
expect(f.props.refresh).toHaveBeenCalledTimes(2)
|
||||
shown.unmount()
|
||||
fireEvent.focus(window)
|
||||
expect(f.props.refresh).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it.each([en, zh])('changes the default from the detail page and retains a refused write for retry', async (dictionary) => {
|
||||
const f = fixture()
|
||||
f.preset.set({ ...f.preset.getSnapshot(), value: { default: 'standard' } })
|
||||
f.publish({ ...observe(value) })
|
||||
render(<PermissionPanel {...f.props} t={makeTranslate(dictionary)} />)
|
||||
const group = screen.getByRole('group', { name: dictionary.defaultTitle })
|
||||
expect(within(group).getByText(dictionary.defaultBody)).toBeTruthy()
|
||||
expect(f.props.setPtcDefault).not.toHaveBeenCalled()
|
||||
const write = Promise.withResolvers<boolean>()
|
||||
f.props.setPtcDefault.mockReturnValueOnce(write.promise)
|
||||
fireEvent.click(within(group).getByRole('button', { name: dictionary.defaultConfirm }))
|
||||
expect(within(group).getByRole('button', { name: dictionary.defaultConfirm }).hasAttribute('disabled')).toBe(true)
|
||||
await act(async () => { write.resolve(false); await write.promise })
|
||||
expect(screen.getByRole('alert').textContent).toBe(dictionary.defaultNotSaved)
|
||||
fireEvent.click(within(group).getByRole('button', { name: dictionary.defaultConfirm }))
|
||||
await waitFor(() => { expect(screen.queryByRole('alert')).toBeNull() })
|
||||
expect(f.props.setPtcDefault).toHaveBeenCalledTimes(2)
|
||||
act(() => { f.preset.set({ ...f.preset.getSnapshot(), value: { default: 'standard', selectedDefault: 'ptc' } }) })
|
||||
expect(within(group).getByRole('button', { name: dictionary.defaultSet }).hasAttribute('disabled')).toBe(true)
|
||||
})
|
||||
|
||||
it('dismisses activation when permissions are ready without changing the default', () => {
|
||||
const f = fixture(), onDismiss = vi.fn()
|
||||
f.preset.set({ ...f.preset.getSnapshot(), value: { default: 'standard', selectedDefault: 'minimal' } })
|
||||
f.publish({ ...observe({ ...value, system: { ...value.system, accessibility: true } }) })
|
||||
render(<PermissionSetup {...f.props} onDismiss={onDismiss} onOpenDetails={vi.fn()} />)
|
||||
expect(onDismiss).toHaveBeenCalledOnce()
|
||||
expect(screen.queryByRole('dialog')).toBeNull()
|
||||
expect(f.props.setPtcDefault).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it.each([new Error('offline'), 'offline'])('reports a failed default-mode request on the detail page', async (error) => {
|
||||
const f = fixture()
|
||||
f.preset.set({ ...f.preset.getSnapshot(), value: { default: 'standard' } })
|
||||
f.props.setPtcDefault.mockRejectedValueOnce(error)
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
fireEvent.click(screen.getByRole('button', { name: en.defaultConfirm }))
|
||||
await waitFor(() => { expect(screen.getByRole('alert').textContent).toContain('offline') })
|
||||
})
|
||||
|
||||
it('waits for the settings document and recognizes PTC as the deployment default', () => {
|
||||
const f = fixture()
|
||||
f.preset.set({ ...f.preset.getSnapshot(), status: 'loading', value: undefined })
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
expect(screen.queryByRole('button', { name: en.defaultConfirm })).toBeNull()
|
||||
act(() => { f.preset.set({ ...f.preset.getSnapshot(), status: 'ready', value: { default: 'ptc' } }) })
|
||||
expect(screen.getByRole('button', { name: en.defaultSet }).hasAttribute('disabled')).toBe(true)
|
||||
expect(f.props.setPtcDefault).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it.each(['unavailable', 'ready'] as const)('explains why the %s settings document cannot change the default', (status) => {
|
||||
const f = fixture()
|
||||
f.preset.set({ ...f.preset.getSnapshot(), status, writable: false,
|
||||
value: status === 'ready' ? { default: 'standard' } : undefined })
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
expect(screen.getByRole('alert').textContent).toBe(status === 'ready' ? en.defaultReadOnly : en.defaultUnavailable)
|
||||
expect(screen.getByRole('button', { name: en.defaultConfirm }).hasAttribute('disabled')).toBe(true)
|
||||
expect(f.props.setPtcDefault).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('shows query failures as unknown while retaining previously displayed apps', () => {
|
||||
const f = fixture(), app = { id: 'a'.repeat(64) as CuaApplicationId, name: 'Fixture', executablePath: '/fixture/app' }
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
f.publish({ ...observe({ ...value, allowedApps: [app] }), system: { value: value.system, busy: false, error: 'offline' }, busy: false })
|
||||
expect(screen.getAllByRole('button', { name: en.revoke })).toHaveLength(1)
|
||||
expect(screen.getAllByText(en.unknown)).toHaveLength(2)
|
||||
expect(screen.queryByText(en.granted)).toBeNull()
|
||||
expect(screen.getByRole('alert').textContent).toContain('offline')
|
||||
fireEvent.click(screen.getAllByRole('button', { name: en.revoke })[0]!)
|
||||
expect(f.props.revoke).toHaveBeenCalledWith(app.id)
|
||||
fireEvent.click(screen.getByRole('button', { name: en.retry }))
|
||||
expect(f.props.refresh).toHaveBeenCalledTimes(2)
|
||||
f.publish({ system: { ...f.state.getSnapshot().system, busy: true }, busy: true })
|
||||
expect(screen.queryByRole('button', { name: en.retry })).toBeNull()
|
||||
f.publish({ system: { ...f.state.getSnapshot().system, busy: false, error: null }, busy: false })
|
||||
expect(screen.queryByRole('button', { name: en.retry })).toBeNull()
|
||||
f.publish(observe(value))
|
||||
expect(screen.getByText(en.noApps)).toBeTruthy()
|
||||
expect(screen.queryByRole('button', { name: en.revoke })).toBeNull()
|
||||
})
|
||||
|
||||
it('keeps unknown initial status distinct from a confirmed missing privilege in Chinese', () => {
|
||||
const f = fixture()
|
||||
render(<PermissionPanel {...f.props} t={makeTranslate(zh)} />)
|
||||
f.publish({ system: { ...f.state.getSnapshot().system, busy: false, error: 'connection lost' }, busy: false })
|
||||
expect(screen.queryByText(zh.granted)).toBeNull()
|
||||
expect(screen.queryByRole('button', { name: zh.request })).toBeNull()
|
||||
f.publish({ ...observe({ ...value, system: { ...value.system, accessibility: true, screenRecording: false } }) })
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.request }))
|
||||
expect(f.props.request).toHaveBeenCalledWith('screenRecording')
|
||||
})
|
||||
|
||||
it('offers setup after inspection, permits dismissal, and dismisses once both privileges are observed', () => {
|
||||
const f = fixture(), onDismiss = vi.fn(), onOpenDetails = vi.fn()
|
||||
render(<PermissionSetup {...f.props} onDismiss={onDismiss} onOpenDetails={onOpenDetails} />)
|
||||
expect(screen.queryByRole('dialog')).toBeNull()
|
||||
f.publish({ ...observe(value) })
|
||||
const dialog = screen.getByRole('dialog', { name: en.setupTitle })
|
||||
fireEvent.click(within(dialog).getByRole('button', { name: en.openSetup }))
|
||||
expect(onOpenDetails).toHaveBeenCalledOnce()
|
||||
fireEvent.click(within(dialog).getByRole('button', { name: en.later }))
|
||||
expect(onDismiss).toHaveBeenCalledOnce()
|
||||
f.publish({ ...observe({ ...value, system: { ...value.system, accessibility: true } }) })
|
||||
expect(screen.queryByRole('dialog')).toBeNull()
|
||||
expect(onDismiss).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('offers the details page when permission inspection fails', () => {
|
||||
const f = fixture()
|
||||
render(<PermissionSetup {...f.props} onDismiss={vi.fn()} onOpenDetails={vi.fn()} />)
|
||||
f.publish({ system: { ...f.state.getSnapshot().system, busy: false, error: 'offline' }, busy: false })
|
||||
expect(screen.getByRole('dialog')).toBeTruthy()
|
||||
expect(screen.queryByRole('alert')).toBeNull()
|
||||
expect(screen.getByText(en.setupQueryFailed)).toBeTruthy()
|
||||
expect(screen.queryByText(en.setupBody)).toBeNull()
|
||||
})
|
||||
|
||||
it('exposes a new-request action for a blocked application without calling grant or revoke', () => {
|
||||
const f = fixture(), app = { id: 'a'.repeat(64) as CuaApplicationId, name: 'Fixture', executablePath: '/fixture/app' }
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
f.publish({ ...observe({ ...value, blockedApps: [app] }) })
|
||||
expect(screen.queryByText(en.noApps)).toBeNull()
|
||||
fireEvent.click(screen.getByRole('button', { name: en.retryApproval }))
|
||||
expect(f.props.retryApproval).toHaveBeenCalledWith(app.id)
|
||||
expect(f.props.revoke).not.toHaveBeenCalled()
|
||||
expect(f.props.request).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('renders action outcomes outside the permission page and clears finished toasts', async () => {
|
||||
vi.useFakeTimers()
|
||||
const f = fixture()
|
||||
render(<PermissionToast {...f.props} />)
|
||||
expect(screen.queryByRole('alert')).toBeNull()
|
||||
f.publish({ toast: { id: 1, kind: 'revoked' } })
|
||||
expect(screen.getByRole('alert').textContent).toBe(en.revoked)
|
||||
await act(async () => { await vi.advanceTimersByTimeAsync(4000) })
|
||||
expect(f.props.dismissToast).toHaveBeenCalledOnce()
|
||||
f.publish({ toast: { id: 2, kind: 'failed', message: 'write refused' } })
|
||||
expect(screen.getByRole('alert').textContent).toContain('write refused')
|
||||
})
|
||||
|
||||
it('keeps permission guidance open while a fresh observation is pending', () => {
|
||||
const f = fixture()
|
||||
render(<PermissionSetup {...f.props} onDismiss={vi.fn()} onOpenDetails={vi.fn()} />)
|
||||
f.publish({ ...observe(value) })
|
||||
const dialog = screen.getByRole('dialog')
|
||||
f.publish({ system: { ...f.state.getSnapshot().system, busy: true }, busy: true })
|
||||
expect(screen.getByRole('dialog')).toBe(dialog)
|
||||
})
|
||||
|
||||
it('shows application icons and names while keeping the executable path in the tooltip', () => {
|
||||
const f = fixture()
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
f.publish({ ...observe({ ...value, allowedApps: [{ id: 'a'.repeat(64) as CuaApplicationId,
|
||||
name: 'Calculator', executablePath: '/Applications/Calculator.app/Contents/MacOS/Calculator' }] }),
|
||||
icons: { value: { '/Applications/Calculator.app/Contents/MacOS/Calculator': 'data:image/png;base64,aWNvbg==' }, busy: false, error: null }, busy: false })
|
||||
expect(screen.getByText('Calculator')).toBeTruthy()
|
||||
expect(screen.queryByText('/Applications/Calculator.app/Contents/MacOS/Calculator')).toBeNull()
|
||||
const identity = screen.getByTitle('/Applications/Calculator.app/Contents/MacOS/Calculator')
|
||||
expect(identity.querySelector('img')?.getAttribute('src')).toBe('data:image/png;base64,aWNvbg==')
|
||||
})
|
||||
|
||||
it.each([en, zh])('shows three requirements with the target application beside each system permission', (dictionary) => {
|
||||
const f = fixture()
|
||||
render(<PermissionPanel {...f.props} t={makeTranslate(dictionary)} />)
|
||||
const application = { name: 'Visual Studio Code.app', path: '/Applications/Visual Studio Code.app', icon: 'data:image/png;base64,aWNvbg==' }
|
||||
f.publish({ ...observe({ ...value, system: { ...value.system, parent: { pid: 1, path: '/sbin/launchd' }, launchApplication: application } }) })
|
||||
const requirements = screen.getByRole('region', { name: dictionary.title })
|
||||
expect(within(requirements).getAllByRole('group')).toHaveLength(3)
|
||||
expect(screen.getByRole('region', { name: dictionary.applicationsTitle })).toBeTruthy()
|
||||
const accessibility = within(requirements).getByRole('group', { name: dictionary.accessibility })
|
||||
expect(accessibility.textContent).toContain(dictionary.permissionPrefix)
|
||||
expect(within(accessibility).getByText(application.name)).toBeTruthy()
|
||||
expect(accessibility.textContent).toContain(dictionary.accessibilityAction)
|
||||
expect(accessibility.querySelector('img')?.getAttribute('src')).toBe(application.icon)
|
||||
const recording = within(requirements).getByRole('group', { name: dictionary.screenRecording })
|
||||
expect(within(recording).getByText(application.name)).toBeTruthy()
|
||||
expect(recording.textContent).toContain(dictionary.screenRecordingAction)
|
||||
expect(within(requirements).getByText(dictionary.permissionHint)).toBeTruthy()
|
||||
f.publish({ system: { ...f.state.getSnapshot().system, error: 'offline' } })
|
||||
expect(screen.queryByText(application.name)).toBeNull()
|
||||
expect(within(requirements).getByText(dictionary.permissionHintUnknown)).toBeTruthy()
|
||||
})
|
||||
|
||||
it('names the authorization target even when its icon is unavailable', () => {
|
||||
const f = fixture()
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
f.publish({ ...observe({ ...value, system: { ...value.system,
|
||||
launchApplication: { name: 'iTerm.app', path: '/Applications/iTerm.app' } } }) })
|
||||
const accessibility = screen.getByRole('group', { name: en.accessibility })
|
||||
expect(within(accessibility).getByText('iTerm.app')).toBeTruthy()
|
||||
expect(accessibility.querySelector('img')).toBeNull()
|
||||
})
|
||||
|
||||
it('waits for the first current observation instead of opening setup with a stale error', () => {
|
||||
const f = fixture()
|
||||
render(<PermissionSetup {...f.props} onDismiss={vi.fn()} onOpenDetails={vi.fn()} />)
|
||||
f.publish({ system: { ...f.state.getSnapshot().system, busy: true, error: 'provider is reloading' }, busy: true })
|
||||
expect(screen.queryByRole('dialog')).toBeNull()
|
||||
f.publish({ ...observe(value) })
|
||||
expect(screen.getByRole('dialog', { name: en.setupTitle })).toBeTruthy()
|
||||
expect(screen.queryByRole('alert')).toBeNull()
|
||||
})
|
||||
|
||||
|
||||
it('omits setup instructions once both system permissions are granted', () => {
|
||||
const f = fixture()
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
f.publish({ ...observe({ ...value, system: { ...value.system, accessibility: true, screenRecording: true } }) })
|
||||
expect(screen.getAllByText(en.granted)).toHaveLength(2)
|
||||
expect(screen.queryByText(en.permissionHintUnknown)).toBeNull()
|
||||
})
|
||||
|
||||
|
||||
it('keeps all three requirements visible before the first permission response', () => {
|
||||
const f = fixture()
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
const requirements = screen.getByRole('region', { name: en.title })
|
||||
expect(within(requirements).getAllByRole('group')).toHaveLength(3)
|
||||
expect(screen.queryByRole('alert')).toBeNull()
|
||||
expect(screen.queryByRole('button', { name: en.request })).toBeNull()
|
||||
for (const button of screen.getAllByRole('button', { name: en.settings })) expect(button.hasAttribute('disabled')).toBe(true)
|
||||
})
|
||||
|
||||
it('fills application names, system permissions, and icons as their observations arrive', () => {
|
||||
const f = fixture()
|
||||
render(<PermissionPanel {...f.props} />)
|
||||
const requirements = screen.getByRole('region', { name: en.title })
|
||||
const access = screen.getByRole('region', { name: en.applicationsTitle })
|
||||
const app = { id: 'a'.repeat(64) as CuaApplicationId, name: 'Calculator', executablePath: '/Applications/Calculator.app' }
|
||||
f.publish({ applications: { value: { ...value, allowedApps: [app] }, busy: false, error: null } })
|
||||
expect(within(access).getByText(app.name)).toBeTruthy()
|
||||
expect(within(access).getByRole('button', { name: en.revoke }).hasAttribute('disabled')).toBe(false)
|
||||
expect(screen.queryByText(en.granted)).toBeNull()
|
||||
const identity = within(access).getByTitle(app.executablePath)
|
||||
expect(identity.querySelector('img')).toBeNull()
|
||||
const placeholder = identity.querySelector('[aria-hidden]')!
|
||||
expect(placeholder).toBeTruthy()
|
||||
f.publish({ system: { value: value.system, busy: false, error: null } })
|
||||
expect(within(requirements).getByText(en.granted)).toBeTruthy()
|
||||
expect(within(requirements).getByRole('button', { name: en.request }).hasAttribute('disabled')).toBe(false)
|
||||
expect(within(access).getByTitle(app.executablePath)).toBe(identity)
|
||||
expect(identity.querySelector('[aria-hidden]')).toBe(placeholder)
|
||||
f.publish({ icons: { value: { [app.executablePath]: 'data:image/png;base64,aWNvbg==' }, busy: false, error: null } })
|
||||
expect(identity.querySelector('img')?.getAttribute('src')).toBe('data:image/png;base64,aWNvbg==')
|
||||
expect(identity.querySelector('[aria-hidden]')).toBeNull()
|
||||
expect(screen.getByRole('region', { name: en.title })).toBe(requirements)
|
||||
})
|
||||
|
||||
it.each(['applications', 'icons'] as const)('contains a failed %s query in the application section', (key) => {
|
||||
const f = fixture()
|
||||
f.publish({ ...observe(value), [key]: { value: null, busy: false, error: 'offline' } })
|
||||
const view = render(<PermissionPanel {...f.props} />)
|
||||
const requirements = screen.getByRole('region', { name: en.title })
|
||||
const access = screen.getByRole('region', { name: en.applicationsTitle })
|
||||
expect(within(requirements).getByText(en.granted)).toBeTruthy()
|
||||
expect(within(requirements).queryByRole('alert')).toBeNull()
|
||||
expect(within(access).getByRole('alert').textContent).toContain('offline')
|
||||
fireEvent.click(within(access).getByRole('button', { name: en.retry }))
|
||||
expect(f.props.refresh).toHaveBeenCalledTimes(2)
|
||||
view.rerender(<PermissionPanel {...f.props} busy />)
|
||||
expect(screen.queryByRole('alert')).toBeNull()
|
||||
})
|
||||
|
||||
it('finishes activation guidance without waiting for application icons', () => {
|
||||
const f = fixture(), onDismiss = vi.fn()
|
||||
render(<PermissionSetup {...f.props} onDismiss={onDismiss} onOpenDetails={vi.fn()} />)
|
||||
f.publish({ system: { value: { ...value.system, accessibility: true }, busy: false, error: null } })
|
||||
expect(onDismiss).toHaveBeenCalledOnce()
|
||||
expect(screen.queryByRole('dialog')).toBeNull()
|
||||
expect(f.state.getSnapshot().icons.busy).toBe(true)
|
||||
})
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.client.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../client/locale/tsconfig.client.json"
|
||||
},
|
||||
{
|
||||
"path": "../../client/store"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-primitives"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-renderer"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-slots"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-plugin-manager/tsconfig.client.json"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-settings/tsconfig.client.json"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-layout"
|
||||
},
|
||||
{
|
||||
"path": "../../api/remotes/tsconfig.client.json"
|
||||
},
|
||||
{
|
||||
"path": "../../api/gateway/tsconfig.client.json"
|
||||
},
|
||||
{
|
||||
"path": "../cua"
|
||||
},
|
||||
{
|
||||
"path": "../../typert/protocol"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
import { clientBundle } from '../../client/tsdown.client.ts'
|
||||
|
||||
export default clientBundle(import.meta.dirname, ['lib/types/index.js'])
|
||||
@@ -0,0 +1,46 @@
|
||||
# Bilingual-pair consistency record for README.md (docs/i18n/README.md): per heading
|
||||
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
|
||||
# After editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/experimental/cua-bundle/README.md
|
||||
/:
|
||||
en: cc518dcd97629a96
|
||||
zh: 465e999011282dc3
|
||||
/deepseek-ai-dsh-cua-bundle:
|
||||
en: 9084fb48d0f6489c
|
||||
zh: ce692972d91cf663
|
||||
/deepseek-ai-dsh-cua-bundle/summary:
|
||||
en: 06d8bf4ee1ef6f3b
|
||||
zh: 8568f649fb846615
|
||||
/deepseek-ai-dsh-cua-bundle/table-of-contents:
|
||||
en: b563e4ec4727d0de
|
||||
zh: d0228da978238e05
|
||||
/deepseek-ai-dsh-cua-bundle/use-this-package:
|
||||
en: 7d378196959b1629
|
||||
zh: e735e217d1e35216
|
||||
/deepseek-ai-dsh-cua-bundle/understand-the-implementation:
|
||||
en: 057771fe13aaa337
|
||||
zh: 4fdb2b98ff3e7a78
|
||||
/deepseek-ai-dsh-cua-bundle/further-exploration:
|
||||
en: 9f15d56b5b16e1ca
|
||||
zh: 75eb08e8828c8877
|
||||
/deepseek-ai-dsh-cua-bundle/model-experience:
|
||||
en: 215e7ba838619b7b
|
||||
zh: a311da3843709f90
|
||||
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions:
|
||||
en: e8f11f45b375771e
|
||||
zh: e8f11f45b375771e
|
||||
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions/what-the-model-sees:
|
||||
en: 591af735d1eb92f6
|
||||
zh: 3632bffa33c77b28
|
||||
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions/token-effect:
|
||||
en: e8de232e96220fe7
|
||||
zh: 5c791307e2123667
|
||||
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions/kv-cache-effect:
|
||||
en: 417c40acd93df532
|
||||
zh: 69cc5cf7140c75a9
|
||||
/deepseek-ai-dsh-cua-bundle/known-limitations-and-deferred-work:
|
||||
en: 41f8c54d76d92c95
|
||||
zh: 44c9662e437f4cbe
|
||||
/deepseek-ai-dsh-cua-bundle/known-limitations-and-deferred-work/dev-note:
|
||||
en: 0a1253c6f7b5e8b9
|
||||
zh: ce96645b3e75874e
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
description: "Enable optional macOS Computer Use and its permission page."
|
||||
kind: "package-bundle"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-cua-bundle
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
Enable macOS Computer Use from Plugins for existing and new PTC sessions. Windows support is in development. This bundle remains in `experimental` until that support is complete, then moves to the `computer-use` group. It is off by default; disabled profiles prepare no native components and register neither CUA tools nor their model guidance.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Use this package](#use-this-package)
|
||||
- [Understand the implementation](#understand-the-implementation)
|
||||
- [Further Exploration](#further-exploration)
|
||||
- [Model Experience](#model-experience)
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## Use this package
|
||||
|
||||
This workflow targets developers running macOS Web or unpackaged Desktop from source. Open Plugins → Computer Use and enable the switch. The permission page identifies the responsible application reported by macOS when the optional system query is available. Request access and enable that application in System Settings; follow the application named by macOS if it differs from the detected source. For Web, run these commands from the repository root:
|
||||
|
||||
```sh
|
||||
pnpm run build
|
||||
pnpm dsh web
|
||||
```
|
||||
|
||||
For the source Desktop launcher and its separate development profile, follow [Desktop development](../../../apps/desktop/README.md). Enable Computer Use in each profile that needs it.
|
||||
|
||||
The switch saves bundle selection in the profile’s `package.json`. Computer Use is installed only in PTC sessions. Enabling from the plugin list offers setup guidance when system permissions need attention; enabling from its detail page displays the permission controls directly. The detail page offers **Set to PTC** for the new-session default; existing sessions keep their preset. The switch leaves PTC execution mode and other settings unchanged. Native preparation runs when the provider loads and reuses the source-hashed executable when available.
|
||||
|
||||
Disabling unloads the provider and withdraws CUA tools from its Agent scopes. Runtime configuration, the new-session default and saved grants remain. Session-only application decisions end with provider disposal.
|
||||
|
||||
Computer Use requires a macOS DSH Host. Other operating systems can display the entry, but native provider startup fails. A browser on another operating system can connect to a supported Mac Host.
|
||||
|
||||
If preparation fails, the saved selection remains and CUA operations stay unavailable. Read the Host startup diagnostic, correct its cause, then restart DSH.
|
||||
|
||||
Explicit activation runs a read-only permission check, which may start a helper without opening a system dialog. Open the [permission page](../client-ui-cua/README.md) for system requests, settings navigation and automatic refresh. Tasks ask separately before accessing an application. Saved application decisions use the existing profile configuration; session decisions remain in memory across turns and connections until DSH restarts or Computer Use is disabled. Revoke access on the same page.
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## Understand the implementation
|
||||
|
||||
<details>
|
||||
<summary>Maintainer details</summary>
|
||||
|
||||
The bundle patch mounts the native provider and permission UI with `agentPreset: ptc`. Existing Agent scopes register the tools without changing preset documents. The detail-page action uses the existing settings form to save `agent-preset-registry.selectedDefault: ptc` only when clicked. [CUA](../cua/README.md) owns registration and disposal.
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## Further Exploration
|
||||
|
||||
[Computer use](../../../docs/subsystems/computer-use.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## Model Experience
|
||||
|
||||
### Computer Use in PTC sessions
|
||||
|
||||
#### What the model sees
|
||||
|
||||
Existing and new PTC sessions receive `cua` through the generated SDK and `run_code`, plus CUA guidance. Disabling withdraws these contributions and keeps PTC presentation. Other presets receive no CUA contributions. [CUA](../cua/README.md) and [PTC](../../ptc-runtime/ptc-runtime-node/README.md#persistent-repl) own the interfaces and output semantics.
|
||||
|
||||
#### Token effect
|
||||
|
||||
CUA activation adds its guidance. Disabling removes that guidance; the existing PTC execution instructions remain.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
CUA activation and removal change the PTC SDK and system guidance; subsequent requests use the updated prefix. The top-level `run_code` transport remains present.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- Source Web and Desktop share permission controls on macOS. Packaged Desktop helper distribution, signing, and permission attribution are deferred.
|
||||
- PTC execution is shared by the profile and is configured independently. A Host without HMR applies saved changes at restart. Saving or revoking application grants while another HMR transaction unloads the provider can block; queued-write cancellation is a separate pending lifecycle fix.
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### Dev Note
|
||||
|
||||
None.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
description: "开启可选 macOS Computer Use 及权限页面。"
|
||||
kind: "package-bundle"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-cua-bundle
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
## 概述
|
||||
|
||||
从插件页为已有和新建 PTC 会话开启 macOS Computer Use。Windows 支持正在开发中。本组合包暂放在 `experimental`，待 Windows 支持完成后迁回 `computer-use` 组。默认关闭，关闭时不准备本机组件、不注册 CUA 工具或添加其模型指引。
|
||||
|
||||
## 目录
|
||||
|
||||
- [使用本包](#use-this-package)
|
||||
- [理解实现](#understand-the-implementation)
|
||||
- [进一步探索](#further-exploration)
|
||||
- [模型体验](#model-experience)
|
||||
- [已知限制与延后工作](#known-limitations-and-deferred-work)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## 使用本包
|
||||
|
||||
本流程面向从源码运行 macOS Web 或未打包 Desktop 的开发者。打开“插件 → Computer Use”并启用。权限页通过可选系统查询显示 macOS 报告的责任应用，指引用户申请权限并在系统设置中开启对应开关。若 macOS 显示其他应用名称，以系统提示为准。Web 版在仓库根目录运行以下命令：
|
||||
|
||||
```sh
|
||||
pnpm run build
|
||||
pnpm dsh web
|
||||
```
|
||||
|
||||
源码 Desktop 的启动方式和独立开发配置档见[桌面开发说明](../../../apps/desktop/README.zh.md)。需要使用 Computer Use 的配置档分别开启该功能。
|
||||
|
||||
开关将组合包选择保存到 profile 的 `package.json`。Computer Use 仅安装在 PTC 会话中。从插件列表启用时，系统权限未就绪会显示设置引导；从详情页启用后直接显示权限控件。详情页提供“设为 PTC”按钮来修改新会话默认模式，已有会话保留原 preset。开关保持 PTC 执行方式和其他设置不变。提供方加载时准备本机组件，已有与源码摘要匹配的可执行文件时直接复用。
|
||||
|
||||
关闭会卸载提供方，并从其 Agent 作用域中撤回 CUA 工具。执行器配置、新会话默认值和已保存的授权保留。会话级应用授权随提供方卸载而失效。
|
||||
|
||||
Computer Use 要求 DSH Host 运行在 macOS 上。其他系统可以显示入口，但原生提供方启动会失败。其他操作系统上的浏览器可以连接受支持的 Mac Host。
|
||||
|
||||
准备失败时，已保存的选择保留，CUA 操作不可用。查看 Host 的启动诊断，修复原因后重启 DSH。
|
||||
|
||||
显式启用会进行只读权限检查，可能启动 helper，但不会打开系统授权弹窗。[权限页面](../client-ui-cua/README.zh.md) 提供系统申请入口、设置跳转和自动检测。任务访问应用前会单独询问用户。长期应用授权保存在现有 profile 配置中，会话授权保存在内存中，跨 turn 和连接复用，重启 DSH 或关闭 Computer Use 后失效。可以在同一页面撤销授权。
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## 理解实现
|
||||
|
||||
<details>
|
||||
<summary>维护者说明</summary>
|
||||
|
||||
组合包 patch 通过 `agentPreset: ptc` 挂载原生提供方和权限界面。工具通过现有 Agent 作用域注册，preset 文档保持不变。详情页按钮复用现有设置表单，仅在点击后保存 `agent-preset-registry.selectedDefault: ptc`。[CUA](../cua/README.zh.md)负责注册和销毁。
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## 进一步探索
|
||||
|
||||
[Computer use](../../../docs/subsystems/computer-use.zh.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## 模型体验
|
||||
|
||||
### Computer Use in PTC sessions
|
||||
|
||||
#### What the model sees
|
||||
|
||||
已有和新建 PTC 会话通过生成的 SDK 与 `run_code` 使用 `cua`，并获得 CUA 指引。关闭撤回这些贡献，保留 PTC 展示方式。其他 preset 不获得 CUA 贡献。完整接口与输出约定由 [CUA](../cua/README.zh.md) 和 [PTC](../../ptc-runtime/ptc-runtime-node/README.zh.md#persistent-repl) 说明。
|
||||
|
||||
#### Token effect
|
||||
|
||||
CUA 生效时添加其指引。关闭撤回这些指引，现有 PTC 执行说明保留。
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
CUA 生效和卸载会改变 PTC SDK 与系统指引，后续请求使用更新后的前缀。顶层 `run_code` 工具保持存在。
|
||||
|
||||
## 已知限制与延后工作
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- 源码 Web 与 Desktop 共用 macOS 权限控件。桌面安装包的 helper 分发、签名和权限归属留待后续处理。
|
||||
- PTC 执行方式由 profile 共享，需单独配置。未启用 HMR 的 Host 在重启时应用已保存的变更。保存或撤销应用授权与另一次卸载提供方的 HMR 同时发生时可能阻塞；排队写入取消另行修复。
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### 开发备注
|
||||
|
||||
None.
|
||||
@@ -0,0 +1,9 @@
|
||||
- insert:
|
||||
- id: cua-provider-registry
|
||||
name: '@deepseek-ai/dsh-computer-use'
|
||||
- id: cua
|
||||
name: '@deepseek-ai/dsh-experimental-cua-macos'
|
||||
config:
|
||||
agentPreset: ptc
|
||||
- id: ui-cua
|
||||
name: '@deepseek-ai/dsh-client-ui-cua'
|
||||
@@ -0,0 +1,12 @@
|
||||
<svg width="36" height="36" viewBox="0 0 36 36" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<circle cx="15.5" cy="15.5" r="8.5" fill="url(#a)" />
|
||||
<circle cx="15.5" cy="15.5" r="5" fill="#FFFFFF" />
|
||||
<path d="M13.2 15.5L14.9 17.2L18.2 13.9" stroke="#145AF3" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" />
|
||||
<path d="M22 22L29 29" stroke="#145AF3" stroke-width="3.2" stroke-linecap="round" />
|
||||
<defs>
|
||||
<linearGradient id="a" x1="7" y1="7" x2="24" y2="24" gradientUnits="userSpaceOnUse">
|
||||
<stop stop-color="#7CB7FF" />
|
||||
<stop offset="1" stop-color="#45D9E7" />
|
||||
</linearGradient>
|
||||
</defs>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 642 B |
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"meta": {
|
||||
"title": "Computer Use",
|
||||
"description": "Currently supports macOS; Windows support is in development. Loads Computer Use for existing and new PTC sessions. Switch to PTC mode and grant Accessibility and Screen Recording permissions to use it."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"meta": {
|
||||
"title": "Computer Use",
|
||||
"description": "目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-cua-bundle",
|
||||
"description": "Optional macOS Computer Use and permission setup",
|
||||
"version": "0.2.1-rc.2",
|
||||
"icon": "./icon.svg",
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
||||
"directory": "packages/experimental/cua-bundle"
|
||||
},
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./cordis.patch.yml": "./cordis.patch.yml",
|
||||
"./locale/*.json": "./locale/*.json",
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"icon.svg",
|
||||
"locale/*.json",
|
||||
"lib/index.js",
|
||||
"cordis.patch.yml",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "MIT",
|
||||
"dsh": {
|
||||
"bundle": {
|
||||
"patch": "./cordis.patch.yml"
|
||||
}
|
||||
},
|
||||
"dependencies": {
|
||||
"@deepseek-ai/dsh-experimental-cua-macos": "workspace:*",
|
||||
"@deepseek-ai/dsh-computer-use": "workspace:*",
|
||||
"@deepseek-ai/dsh-ptc-runtime-node": "workspace:*",
|
||||
"@deepseek-ai/dsh-client-ui-cua": "workspace:*"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:~"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:~",
|
||||
"@deepseek-ai/cordis-plugin-include": "workspace:~",
|
||||
"js-yaml": "^4.2.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
/** Optional Computer Use composition; configuration rows live in cordis.patch.yml. */
|
||||
export {}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -12,11 +12,11 @@
|
||||
en: 4549f02fc53854a5
|
||||
zh: 6f74f1130b6623a8
|
||||
/deepseek-ai-dsh-experimental-cua-macos/table-of-contents:
|
||||
en: d152484eb41ac6b4
|
||||
zh: 7b30b973a849247d
|
||||
en: 867d9ec544719acf
|
||||
zh: 92348599582dc33a
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package:
|
||||
en: e2e1797ad6b73f25
|
||||
zh: 35b7a06e471e433c
|
||||
en: f2ceaaa0c834a931
|
||||
zh: 0edd75ea5323d07f
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
|
||||
en: e7909c9b43e44fbc
|
||||
zh: 98949877d4ea3a86
|
||||
@@ -45,8 +45,8 @@
|
||||
en: a2c9acb5a4320dda
|
||||
zh: e8d1586af60215c4
|
||||
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work:
|
||||
en: 9d8af4c39fe407a3
|
||||
zh: afec49eedfd702cb
|
||||
en: 1c42b446c186b01b
|
||||
zh: 0f1ecea4fd0a1a13
|
||||
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work/dev-note:
|
||||
en: b836f01bd0bceb4a
|
||||
zh: 8122fa08b2aa95fa
|
||||
@@ -20,12 +20,16 @@ Inspect macOS permissions, select running applications and windows, read accessi
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
- [Dev Note](#dev-note)
|
||||
|
||||
The helper supplies bounded 64-pixel PNG application icons for permission prompts and saved-application lists. Icon reads do not launch an application or inspect its windows; executable and signing checks still determine authorization.
|
||||
|
||||
The shared helper queries the process macOS considers responsible for it and captures that application’s name and icon before announcing readiness. A Desktop app launched from Terminal can therefore identify Terminal even when another GUI application sits between it and the helper. Permission queries retain this metadata after the responsible application exits and across connections sharing the helper. If the system query or application metadata is unavailable, the field is omitted and the UI uses the system prompt for guidance.
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## Use this package
|
||||
|
||||
Enable this experimental provider through explicit source composition on macOS 14 or later. Windows support is in development. Shipped profiles do not mount it. Prepare a native helper and load the registry/provider overlay:
|
||||
For repository Web use, enable [Computer Use](../cua-bundle/README.md) in Plugins. This prepares the local executable on macOS 14 or later; Windows support is in development. Subsequent builds check it while the existing Web profile keeps the bundle selected. Disabled profiles do not prepare native components. Explicit source composition is also available:
|
||||
|
||||
```sh
|
||||
swift build --package-path packages/experimental/cua-macos/native --configuration release
|
||||
@@ -34,7 +38,7 @@ export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
|
||||
pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config
|
||||
```
|
||||
|
||||
The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped executable is identified before approval and checked again before and after launch; unsigned code only receives task access. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s `raise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
|
||||
The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped executable is identified before approval and checked again before and after launch; unsigned code only receives session access for the selected instance. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s `raise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
|
||||
|
||||
| Field | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
@@ -59,7 +63,7 @@ Keypad names (`KP_0` through `KP_9`, `KP_Enter`, and keypad operators) retain th
|
||||
|
||||
Agent disposal and plugin unload cancel and await the complete tool call, including image admission and attachment storage. An image write already in progress must settle before disposal returns; a canceled call cannot return a successful screenshot.
|
||||
|
||||
Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. User Questions offers task access or denial, plus saved access when the application has a verifiable signature and the composition provides settings persistence. Task access is bound to the selected process handle; saved access is bound to its canonical executable path and designated code-signing requirement. Denial prevents repeated prompts in that task. Removing a saved `allowedApps` entry cancels affected operations and invalidates their helper handles; reauthorization requires fresh selection. Other handles on the same Agent connection also expire, while saved grants remain reusable. Agent/provider disposal joins cleanup of the corresponding native connection. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools retain their own execution authority.
|
||||
Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. The first selection asks the human for session access, denial, or saved access when the application has a verifiable signature. Session access is kept in memory by session and application identity, survives agent and helper replacement, and expires when DSH restarts or Computer Use is disabled. Signed application identities combine the canonical executable path and designated code-signing requirement; unverifiable applications remain bound to their process instance. Denial prevents repeated prompts in that session. Revocation reads and removes saved access in the same queue as grant creation, invalidates each affected Agent connection, and awaits its native cleanup without replaying input. Every app handle on those Agent connections becomes unusable; other applications need fresh selection, but their session and saved grants remain reusable. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools are not an operating-system sandbox.
|
||||
|
||||
Raw `getState` exposes validated state fields and the complete text for the requested disclosure options. The native helper also returns structured element diagnostics for native checks; the tool projection omits that duplicate array from model text and structured tool values.
|
||||
|
||||
@@ -147,7 +151,7 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
|
||||
##### Operation guidance (verbatim)
|
||||
|
||||
```markdown
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user allows another request on the permission page. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
|
||||
@@ -184,7 +188,7 @@ App and Window methods:
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to complete the Computer Use permission setup for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
@@ -210,7 +214,7 @@ The queue orders individual native requests. Holding an app object does not rese
|
||||
- macOS Stage Manager can reduce background window bounds and screenshots to thumbnails. Full-window observation and coordinate input in this state are not guaranteed; model guidance requires explaining foreground needs and avoiding repeated focus reclamation.
|
||||
- Background input depends on private SkyLight symbols; macOS updates can make them unavailable. Validate each target OS version.
|
||||
- Observed failures include dispatched Unicode text not appearing in Blender and background Command+A not selecting all text in AppKit. Inspect application state after input; a successful tool result confirms dispatch only.
|
||||
- Native binaries require a local build. Signing, distribution, and system-permission onboarding remain deferred.
|
||||
- Native binaries require a local build. Signing and Desktop distribution remain future work. The [permission page](../client-ui-cua/README.md) offers macOS prompts and app revocation; packaged Desktop permission attribution remains deferred.
|
||||
- Persistent PTC is an external dependency assumption, not an implementation in this package. Real-model task success requires separate evaluation beyond native and protocol fixtures.
|
||||
|
||||
-----
|
||||
|
||||
@@ -20,12 +20,16 @@ kind: "package-reference"
|
||||
- [已知限制与后续工作](#known-limitations-and-deferred-work)
|
||||
- [开发备注](#dev-note)
|
||||
|
||||
helper 为授权提示和已授权应用列表提供 64 像素 PNG 图标。读取图标不会启动应用或检查窗口，授权仍由可执行文件和签名校验决定。
|
||||
|
||||
共享 helper 查询 macOS 为它记录的责任进程，在发布就绪状态前保存对应应用的名称和图标。因此，从 Terminal 启动 Desktop 时，即使中间还有另一个 GUI 应用，也能识别到 Terminal。责任应用退出后，权限查询仍保留这些信息，后续复用该 helper 的连接读取同一份记录。系统查询或应用信息不可用时省略该字段，由界面引导用户按系统弹窗选择授权对象。
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## 使用本包
|
||||
|
||||
该实验性提供方需要在 macOS 14 及以上通过源码配置显式启用。Windows 支持正在开发中。默认配置不加载它。先准备原生组件，再加载注册表和提供方覆盖层：
|
||||
从仓库运行 Web 时，在插件页开启 [Computer Use](../cua-bundle/README.zh.md)。首次启用会在 macOS 14 及以上准备本机组件，Windows 支持正在开发中。现有 Web profile 保持选中时，后续构建继续检查更新。未开启的 profile 不准备原生组件。也可以显式加载源码覆盖层：
|
||||
|
||||
```sh
|
||||
swift build --package-path packages/experimental/cua-macos/native --configuration release
|
||||
@@ -34,7 +38,7 @@ export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
|
||||
pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config
|
||||
```
|
||||
|
||||
配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图。`getCapabilities` 查询权限和锁屏状态，不弹出系统授权窗口。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。启动前先确认可执行文件身份，授权后在启动前后再次核对；未签名代码只取得任务许可。返回的对象读取初始 AX，不自动激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口的 `raise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。
|
||||
配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图。`getCapabilities` 查询权限和锁屏状态，不弹出系统授权窗口。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。启动前先确认可执行文件身份，授权后在启动前后再次核对；未签名代码只能取得所选实例的会话许可。返回的对象读取初始 AX，不自动激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口的 `raise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。
|
||||
|
||||
| 字段 | 默认值 | 含义 |
|
||||
| --- | --- | --- |
|
||||
@@ -59,7 +63,7 @@ pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.
|
||||
|
||||
Agent 销毁和插件卸载会取消并等待完整工具调用，包括图像接入检查和附件存储。已经开始的图像写入必须结束后，销毁才会返回；已取消的调用不会返回成功截图。
|
||||
|
||||
CUA 工具在读取 AX、窗口标题、截图、激活和输入之前检查应用授权。User Questions 提供任务级允许和拒绝选项；应用签名可验证且组合支持配置持久化时，也可保存授权。任务授权绑定所选进程句柄，长期授权绑定真实可执行路径和指定代码签名要求。拒绝后，同一任务不再反复询问。删除保存的 `allowedApps` 条目会取消相关操作并使原生组件中的句柄失效；重新授权需要重新选择应用。同一 Agent 连接中的其他句柄也会失效，已保存的授权仍可复用。Agent 或提供方销毁时等待对应连接的原生状态清理。设计理由见[应用授权决策](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md)。这些检查约束 CUA 调用；持久 PTC 和另行授权的 shell 工具保留各自的执行权限。
|
||||
CUA 工具在读取 AX、窗口标题、截图、激活和输入前检查应用授权。首次选择应用时询问用户，可选择本会话允许、拒绝；签名可验证的应用还可保存长期授权。会话许可按会话和应用身份保存在内存中，Agent 或 helper 重建后仍可复用，重启 DSH 或关闭 Computer Use 后失效。已签名应用的身份由规范化可执行文件路径和代码签名指定要求组成；签名不可验证的应用仍绑定进程实例。拒绝后，同一会话不会反复询问。撤销与长期许可的保存共用队列，在队列内读取并移除许可，然后使相关 Agent 连接失效并等待原生清理完成，不自动重放输入。同一 Agent 连接中的其他应用也需要重新选择，其他应用的会话许可和长期许可仍可复用。设计理由见[应用授权决策](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md)。这些检查的范围是 CUA 调用。persistent PTC 与已授权的 shell 仍按各自执行权限运行。
|
||||
|
||||
原始 `getState` 操作返回经过校验的状态字段，以及按本次披露选项生成的完整文本。原生 helper 还返回供原生检查使用的结构化元素诊断数组；工具投影会从模型文本和结构化工具值中移除这份重复数组。
|
||||
|
||||
@@ -147,7 +151,7 @@ AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档
|
||||
##### 操作指导文本（原文）
|
||||
|
||||
```markdown
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user allows another request on the permission page. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
|
||||
@@ -184,7 +188,7 @@ App and Window methods:
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to complete the Computer Use permission setup for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
@@ -210,7 +214,7 @@ Use selectText only on controls advertising selectionWritable. It selects a uniq
|
||||
- macOS 台前调度可能将后台窗口的尺寸与截图变为缩略图，当前无法保证这种状态下的完整画面与坐标输入；模型指引要求说明前台需求并避免反复抢回焦点。
|
||||
- 后台输入依赖私有 SkyLight 符号，macOS 升级可能使其不可用，需在目标系统版本验证。
|
||||
- 实测 Blender Unicode 输入曾出现分派成功但界面未显示文本，后台 AppKit 的 Command+A 曾未选中全文。输入后必须检查应用状态；工具返回成功只表示已分派。
|
||||
- 原生二进制需要本地构建。签名、分发和系统权限引导留待后续。
|
||||
- 原生二进制需要本地构建，签名和桌面分发留待后续。[权限页面](../client-ui-cua/README.zh.md)提供 macOS 申请入口和应用授权撤销；桌面安装包的权限归属留待后续处理。
|
||||
- 持久 PTC 是对外部运行环境的假设，本包不实现它。真实模型任务成功率需要在原生测试和协议测试之外单独评估。
|
||||
|
||||
-----
|
||||
|
||||
@@ -4,4 +4,5 @@
|
||||
- id: cua
|
||||
name: '../src/index.ts'
|
||||
config:
|
||||
agentPreset: false
|
||||
helperPath: !!js process.env.DSH_CUA_HELPER
|
||||
@@ -15,17 +15,17 @@
|
||||
en: e8cbc43d38786c40
|
||||
zh: 52b3ba85d98011b9
|
||||
/dsh-cua-native-helper/build-and-service:
|
||||
en: 824e4a2b9036d9bf
|
||||
zh: 37c34a454abc6965
|
||||
en: 02b25ac1126b6094
|
||||
zh: 3995a1f46354531b
|
||||
/dsh-cua-native-helper/request-protocol:
|
||||
en: 3202c567a6e1a654
|
||||
zh: 143eebd0096a0487
|
||||
en: c8ad13843989b314
|
||||
zh: 2a42acb2faee8803
|
||||
/dsh-cua-native-helper/observation-and-input:
|
||||
en: f2db99ae46a21c61
|
||||
zh: 48c6752bd71dbf27
|
||||
en: ce99636fae06e66c
|
||||
zh: d3c983e8a47916b2
|
||||
/dsh-cua-native-helper/verification:
|
||||
en: 2afe5ceb0a3c4965
|
||||
zh: 86bf7187ddae489f
|
||||
en: ba3277817f4da729
|
||||
zh: 4642abf830234a3b
|
||||
/dsh-cua-native-helper/further-exploration:
|
||||
en: 265e1927952ce6d7
|
||||
zh: 7c35d9fd98b6e6c8
|
||||
|
||||
@@ -27,7 +27,7 @@ Build with `swift build --package-path packages/experimental/cua-macos/native --
|
||||
|
||||
The shared service uses an owner-only Unix socket under `/tmp/dsh-cua-<uid>`, keyed by the executable SHA-256. A version lock elects one process; an additional real-executable-path lock excludes a different build at the same path. The first frame is protocol-1 `hello` with the runtime key and this connection’s byte, AX, operation, screenshot, event-spacing and cleanup limits. `ready` reports the verified runtime key, PID and instance ID. Every connection owns an independent AXWorker. The stdin/stdout entry is a diagnostic proxy to this service.
|
||||
|
||||
AppKit runs on the main event loop. Blocking AX and input use each connection’s serial actor; all connections share a FIFO for complete desktop operations. Socket writes do not hold that operation slot. A request cancellation waits for native work to stop; it does not cancel the connection’s lifetime latch. Screenshot timeout ends only the request. An unfinished system capture retains the single capture slot until its real callback arrives, rejects new captures with CAPTURE_BUSY and leaves AX/input available. Only an operation or cleanup that cannot stop can invalidate the shared process. The proxy accepts `--max-request-bytes`, `--max-response-bytes`, `--ax-timeout-ms`, `--timeout-ms`, `--screenshot-timeout-ms`, `--shutdown-grace-ms` and `--text-event-interval-ms`; the Host sets the same limits in hello. An AX deadline bounds an individual system call. Cancellation can follow partial input, and uncertain input is never replayed automatically.
|
||||
AppKit runs on the main event loop. Blocking socket accepts and handshakes use background queues; server callbacks explicitly carry `@MainActor @Sendable` isolation and weak ownership. Shutdown releases accepted connections even when their handshake is incomplete. Blocking AX and input use each connection’s serial actor; all connections share a FIFO for complete desktop operations. Socket writes do not hold that operation slot. A request cancellation waits for native work to stop; it does not cancel the connection’s lifetime latch. Screenshot timeout ends only the request. An unfinished system capture retains the single capture slot until its real callback arrives, rejects new captures with CAPTURE_BUSY and leaves AX/input available. Only an operation or cleanup that cannot stop can invalidate the shared process. The proxy accepts `--max-request-bytes`, `--max-response-bytes`, `--ax-timeout-ms`, `--timeout-ms`, `--screenshot-timeout-ms`, `--shutdown-grace-ms` and `--text-event-interval-ms`; the Host sets the same limits in hello. An AX deadline bounds an individual system call. Cancellation can follow partial input, and uncertain input is never replayed automatically.
|
||||
|
||||
|
||||
-----
|
||||
@@ -57,7 +57,7 @@ Every app-targeted operation requires `appHandle` from `getApp` in the same help
|
||||
|
||||
Stable error codes are `INVALID_REQUEST` (malformed envelope), `INVALID_ARGUMENT` (invalid parameters), `UNSUPPORTED` (unknown operation), `APP_UNAVAILABLE`, `AMBIGUOUS_APP`, `STALE_APP`, `STALE_SNAPSHOT`, `WINDOW_UNAVAILABLE`, `WINDOW_CHANGED`, `PERMISSION_DENIED`, `FOREGROUND_REQUIRED` (hover target is not the unobscured foreground window), `PANEL_NOT_READY` (system panel recipient unavailable before input), and `NATIVE_ERROR` (remaining operating-system failures). An error never promises that a partially dispatched input was rolled back. The host must not retry input automatically.
|
||||
|
||||
Application selection verifies the executable path and code-signing requirement before returning an authorization identity to the Host. Unsigned running applications receive a process-lifetime identity; stopped unsigned applications use an executable digest and cannot receive persistent consent. The Host obtains human consent before launch, observation or input.
|
||||
Application selection returns an authorization record to the Host: a digest of canonical executable path and verified designated signing requirement, display name, executable path, and persistable flag. Unsigned or ad-hoc-signed applications cannot receive saved consent; running instances use process-lifetime identity, while stopped applications use an executable digest. The Host strips this record from model results and obtains human consent before launch, observation or input. This pipe protocol is trusted Host transport, not a sandbox against a separately authorized shell process.
|
||||
|
||||
Protocol declarations are generated from [the shared protocol](../../cua/src/protocol.ts). Run `pnpm exec tsx packages/experimental/cua-macos/native/generate-protocol.ts` from the repository root; package tests verify generated JSON and Swift field/enum tables. Model-submittable parameters exclude Host resource bounds and the private launch continuation.
|
||||
|
||||
@@ -77,6 +77,8 @@ Capture and pointer delivery read CG bounds before and after an exact-window AX
|
||||
|
||||
The responsible host/helper needs macOS Accessibility permission for AX state and input, and Screen Recording permission for screenshots. Capability preflight and permission errors do not grant permissions. Capture refuses a window that moves or resizes while the image is being read. Unassociated system panels and accessory-only applications are not selectable.
|
||||
|
||||
Host-only permission controls are `permissionStatus {}`, `requestPermission {permission}`, and `openPermissionSettings {permission}`, where permission is `accessibility` or `screenRecording`. They are excluded from model tools and from `getCapabilities.methods`. Permission status includes observed helper/parent process paths; these paths do not identify the TCC responsible application. The operating-system prompt names the application and asks for the user’s decision. The helper identifies the responsible process with the optional libSystem SPI `responsibility_get_pid_responsible_for_pid`, then captures its application name and icon before publishing readiness. The function is resolved dynamically, following the same signature used by [Chromium](https://chromium.googlesource.com/chromium/src/+/refs/tags/134.0.6998.28/base/process/process_info_mac.mm). If the symbol, process, or application metadata is unavailable, attribution stays unknown; it is not inferred from the nearest GUI ancestor. Permission checks remain independent of this metadata.
|
||||
|
||||
Observation reads each non-layout node’s descriptive properties and children through `AXUIElementCopyMultipleAttributeValues`. Per-attribute errors retain the individual-read rules; unsupported batch APIs use individual reads. Values are retained only during that observation, and input reads live attributes. Application state can change between nodes; a batch does not make the whole tree atomic.
|
||||
|
||||
System file-panel keyboard routing resolves the remote recipient through the selected app’s focused AX sheet. Only Apple’s Open and Save Panel Service is accepted, and its focused ancestry must still contain the identified remote window. Element pointer fallbacks resolve an app-owned window or sheet through AX parents when AXWindow is unavailable. AX actions retain priority over pointer input; visual click markers need no system-cursor movement.
|
||||
@@ -101,6 +103,8 @@ Use `python3 packages/experimental/cua-macos/native/tests/test_swift.py` for off
|
||||
|
||||
Run `python3 packages/experimental/cua-macos/native/tests/shared_helper_e2e.py --helper /absolute/helper` to verify concurrent startup, a shared service PID, independent handles, same-application handoff, request-local screenshot deadlines, connection cleanup and signal shutdown with an owned AppKit fixture. Existing Accessibility and Screen Recording grants are required. `--without-desktop` limits the check to protocol and process lifetime.
|
||||
|
||||
Run `python3 packages/experimental/cua-macos/native/tests/launcher_identity_e2e.py --helper /absolute/helper` in a GUI login session to verify a Launch Services application that starts a nested GUI launcher and then the helper. The helper identifies the outer responsible application and retains its name and icon after the launchers exit. This check owns both applications and the helper, cleans up its Launch Services registrations, and does not request Accessibility or Screen Recording access.
|
||||
|
||||
Run `python3 packages/experimental/cua-macos/native/tests/axless_e2e.py` after building and unlocking the Mac to verify an owned window whose application removes and later restores its AX roots. The check requires existing Accessibility/Screen Recording grants, rejects stale element indices, captures the AXless window, verifies a coordinate click in the fixture state file, and checks restoration. It never changes system permission settings.
|
||||
|
||||
Run `python3 packages/experimental/cua-macos/native/tests/observation_benchmark.py --reference-helper /absolute/reference --candidate-helper /absolute/candidate --artifacts /private/evidence` to compare complete observations of the same owned AppKit window with 35 and 400 synthetic rows. Each helper receives one warmup and three alternating measured requests. All result fields must match except helper-generated app and snapshot IDs. The report keeps raw timings, byte counts, hashes and replies; it sets no CI speed threshold and makes no memory claim. Fixture startup can activate the application. Run without other desktop automation or compilation jobs.
|
||||
|
||||
@@ -27,7 +27,7 @@ kind: "package-reference"
|
||||
|
||||
共享服务在 `/tmp/dsh-cua-<uid>` 下使用仅所有者可访问的 Unix socket，以可执行文件的 SHA-256 区分实例。版本锁选出一个进程，另一个真实可执行路径锁排除同一路径的不同构建。首帧为协议版本 1 的 `hello`，包含运行时标识以及该连接的字节、AX、操作、截图、事件间隔和清理限制。`ready` 返回已核实的运行时标识、PID 和实例 ID。每条连接拥有独立的 AXWorker。stdin/stdout 入口是该服务的诊断代理。
|
||||
|
||||
AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的串行 actor，所有连接共享用于完整桌面操作的 FIFO。socket 写出不占用操作槽位。请求取消会等待原生工作停止，不取消连接的生命期控制器。截图超时只结束该请求。未完成的系统捕获保留唯一捕获槽位，直至真实回调到达；期间新截图返回 CAPTURE_BUSY，AX 和输入仍可用。只有无法停止的操作或清理才可能使共享进程失效。代理接受 `--max-request-bytes`、`--max-response-bytes`、`--ax-timeout-ms`、`--timeout-ms`、`--screenshot-timeout-ms`、`--shutdown-grace-ms` 和 `--text-event-interval-ms`；Host 在 hello 中设置相同限制。AX 超时约束单次系统调用。取消可能发生在部分输入之后，结果不确定的输入不自动重放。
|
||||
AppKit 在主事件循环上运行。阻塞式 socket 接收与握手使用后台队列，服务回调明确声明 `@MainActor @Sendable` 隔离，并弱引用服务对象。关闭时会释放已接收的连接，包括尚未完成握手的连接。阻塞式 AX 和输入使用每条连接的串行 actor，所有连接共享用于完整桌面操作的 FIFO。socket 写出不占用操作槽位。请求取消会等待原生工作停止，不取消连接的生命期控制器。截图超时只结束该请求。未完成的系统捕获保留唯一捕获槽位，直至真实回调到达；期间新截图返回 CAPTURE_BUSY，AX 和输入仍可用。只有无法停止的操作或清理才可能使共享进程失效。代理接受 `--max-request-bytes`、`--max-response-bytes`、`--ax-timeout-ms`、`--timeout-ms`、`--screenshot-timeout-ms`、`--shutdown-grace-ms` 和 `--text-event-interval-ms`；Host 在 hello 中设置相同限制。AX 超时约束单次系统调用。取消可能发生在部分输入之后，结果不确定的输入不自动重放。
|
||||
|
||||
|
||||
-----
|
||||
@@ -57,7 +57,7 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
|
||||
|
||||
稳定错误码包括 `INVALID_REQUEST`（信封无效）、`INVALID_ARGUMENT`（参数无效）、`UNSUPPORTED`（操作未知）、`APP_UNAVAILABLE`、`AMBIGUOUS_APP`、`STALE_APP`、`STALE_SNAPSHOT`、`WINDOW_UNAVAILABLE`、`WINDOW_CHANGED`、`PERMISSION_DENIED`、`FOREGROUND_REQUIRED`（悬停目标不是未被遮挡的前台窗口）、`PANEL_NOT_READY`（输入前系统面板接收方尚不可用）及 `NATIVE_ERROR`（其余操作系统失败）。错误不保证已分派的部分输入被回滚，Host 不得自动重试输入。
|
||||
|
||||
应用选择先核实可执行路径和代码签名要求，再向 Host 返回授权身份。未签名的运行中应用使用进程生命期身份，未启动的未签名应用使用可执行文件摘要，不能获得持久授权。Host 在启动、观察和输入前取得用户许可。
|
||||
应用选择向 Host 返回授权记录：规范化可执行路径与经验证的代码签名指定要求的摘要、显示名称、可执行路径和可持久化标记。未签名或临时签名的应用不能获得长期授权；运行中实例使用进程生命期身份，未启动应用使用可执行文件摘要。Host 从模型结果中移除此记录，并在启动、观察和输入前取得用户许可。此管道协议用于可信 Host 传输，无法隔离另一个已获授权的 shell 进程。
|
||||
|
||||
协议声明从[共享协议](../../cua/src/protocol.ts)生成。在仓库根目录运行 `pnpm exec tsx packages/experimental/cua-macos/native/generate-protocol.ts`，包测试检查生成的 JSON 和 Swift 字段/枚举表。模型参数不包含 Host 资源上限和私有启动续接操作。
|
||||
|
||||
@@ -77,6 +77,8 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
|
||||
|
||||
负责执行的 host/helper 需要 macOS 辅助功能权限以读取 AX 和输入，需要屏幕录制权限以截图。能力预检和权限错误不授予权限。截图会拒绝在读取图片期间移动或缩放的窗口。无法确认归属的系统面板及仅后台辅助类型的应用不可选择。
|
||||
|
||||
仅供 Host 使用的权限操作为 `permissionStatus {}`、`requestPermission {permission}` 和 `openPermissionSettings {permission}`，其中 permission 为 `accessibility` 或 `screenRecording`。这些操作不出现在模型工具及 `getCapabilities.methods` 中。权限状态包含观察到的 helper 和父进程路径，这些路径不能确定 TCC 归属的应用。系统授权提示负责指出应用并询问用户决定。helper 使用可选的 libSystem 内部接口 `responsibility_get_pid_responsible_for_pid` 查询责任进程，在报告就绪前记录对应应用的名称和图标。接口通过动态加载解析，函数签名与 [Chromium](https://chromium.googlesource.com/chromium/src/+/refs/tags/134.0.6998.28/base/process/process_info_mac.mm) 使用的一致。符号、进程或应用信息不可用时，归属保持未知，不根据最近的 GUI 祖先推断。权限检查独立于这些信息。
|
||||
|
||||
观察通过 `AXUIElementCopyMultipleAttributeValues` 读取每个非布局节点的描述属性和子节点。逐属性错误保留单独读取的规则；不支持批量 API 时使用单独读取。值只在本次观察期间保留，输入读取即时属性。应用状态可能在节点之间变化，批量读取不使整棵树具备原子性。
|
||||
|
||||
系统文件面板的键盘路由通过所选应用的 AX 焦点 sheet 解析远端接收方。只接受 Apple 的 Open and Save Panel Service，且其焦点祖先链仍须包含已识别的远端窗口。AXWindow 不可用时，元素鼠标后备路径沿 AX 父节点解析应用所属窗口或 sheet。AX 动作保持优先，可视点击标记不需要系统鼠标移动。
|
||||
@@ -101,6 +103,8 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
|
||||
|
||||
运行 `python3 packages/experimental/cua-macos/native/tests/shared_helper_e2e.py --helper /absolute/helper`，通过测试自建的 AppKit fixture 核实并发启动、共享服务 PID、独立句柄、同一应用交接、请求级截图期限、连接清理和信号退出。需要已有辅助功能和屏幕录制权限。`--without-desktop` 将检查限定为协议与进程生命期。
|
||||
|
||||
在 GUI 登录会话中运行 `python3 packages/experimental/cua-macos/native/tests/launcher_identity_e2e.py --helper /absolute/helper`，验证由 Launch Services 启动的应用先启动另一个 GUI 启动器，再启动 helper。helper 识别外层责任应用，并在启动器退出后保留其名称和图标。检查只使用测试自建的两个应用和 helper，结束后清理对应的 Launch Services 注册，不申请辅助功能或屏幕录制权限。
|
||||
|
||||
构建并解锁 Mac 后运行 `python3 packages/experimental/cua-macos/native/tests/axless_e2e.py`，验证应用移除并恢复 AX 根节点的窗口。检查需要已有辅助功能/屏幕录制权限，会拒绝过期元素索引，捕获无 AX 窗口，通过 fixture 状态文件核实坐标点击，并检查恢复。不会修改系统权限设置。
|
||||
|
||||
运行 `python3 packages/experimental/cua-macos/native/tests/observation_benchmark.py --reference-helper /absolute/reference --candidate-helper /absolute/candidate --artifacts /private/evidence`，对比同一自建 AppKit 窗口中 35 行和 400 行合成内容的完整观察。每个 helper 先预热一次，再交替测量三次。除 helper 生成的应用和快照 ID 外，结果字段必须一致。报告保留原始耗时、字节数、哈希和回复，不设置 CI 性能阈值，也不声称内存表现。fixture 启动可能激活应用。运行时避免其他桌面自动化或编译任务。
|
||||
|
||||
@@ -36,6 +36,22 @@ struct ApplicationCatalogSnapshot: Sendable {
|
||||
}
|
||||
return ApplicationCatalogSnapshot(applications: apps, frontPID: NSWorkspace.shared.frontmostApplication?.processIdentifier)
|
||||
}
|
||||
/// A bounded PNG for human recognition; authorization continues to use executable/signing identity.
|
||||
static func icon(at path: String) -> String? {
|
||||
guard let bitmap = NSBitmapImageRep(bitmapDataPlanes: nil, pixelsWide: 64, pixelsHigh: 64,
|
||||
bitsPerSample: 8, samplesPerPixel: 4, hasAlpha: true, isPlanar: false,
|
||||
colorSpaceName: .deviceRGB, bytesPerRow: 0, bitsPerPixel: 0),
|
||||
let context = NSGraphicsContext(bitmapImageRep: bitmap) else { return nil }
|
||||
NSGraphicsContext.saveGraphicsState()
|
||||
defer { NSGraphicsContext.restoreGraphicsState() }
|
||||
NSGraphicsContext.current = context
|
||||
var url = URL(fileURLWithPath: path)
|
||||
while url.path != "/" && url.pathExtension != "app" { url.deleteLastPathComponent() }
|
||||
let iconPath = url.pathExtension == "app" ? url.path : path
|
||||
NSWorkspace.shared.icon(forFile: iconPath).draw(in: NSRect(x: 0, y: 0, width: 64, height: 64),
|
||||
from: .zero, operation: .copy, fraction: 1)
|
||||
return bitmap.representation(using: .png, properties: [:]).map { "data:image/png;base64," + $0.base64EncodedString() }
|
||||
}
|
||||
static func activate(_ identity: ProcessIdentity) throws {
|
||||
try NativePlatform.verify(identity)
|
||||
guard let app = NSRunningApplication(processIdentifier: identity.pid), !app.isTerminated else {
|
||||
@@ -47,9 +63,18 @@ struct ApplicationCatalogSnapshot: Sendable {
|
||||
}
|
||||
|
||||
enum NativePlatform {
|
||||
/// Optional libSystem SPI: unavailable attribution stays unknown rather than guessing from parentage.
|
||||
static func responsiblePID(_ pid: Int32) -> Int32? {
|
||||
typealias Resolve = @convention(c) (pid_t) -> pid_t
|
||||
guard let symbol = dlsym(UnsafeMutableRawPointer(bitPattern: -2), "responsibility_get_pid_responsible_for_pid") else { return nil }
|
||||
let resolve = unsafeBitCast(symbol, to: Resolve.self)
|
||||
let responsible = resolve(pid)
|
||||
return responsible > 0 ? responsible : nil
|
||||
}
|
||||
static func executablePath(_ pid: Int32) -> String? {
|
||||
var buffer = [CChar](repeating: 0, count: 4 * Int(MAXPATHLEN))
|
||||
return proc_pidpath(pid, &buffer, UInt32(buffer.count)) > 0 ? String(cString: buffer) : nil
|
||||
guard proc_pidpath(pid, &buffer, UInt32(buffer.count)) > 0 else { return nil }
|
||||
return String(decoding: buffer.prefix { $0 != 0 }.map { UInt8(bitPattern: $0) }, as: UTF8.self)
|
||||
}
|
||||
static func identity(_ pid: Int32) throws -> ProcessIdentity {
|
||||
var info = proc_bsdinfo()
|
||||
|
||||
+6
@@ -3,6 +3,7 @@ enum NativeProtocol {
|
||||
static let modelMethods: [String] = ["activate", "click", "drag", "findApps", "getApp", "getCapabilities", "getState", "hover", "listApps", "listWindows", "paste", "performSecondaryAction", "pressKey", "screenshot", "scroll", "selectText", "setValue", "show", "typeText"]
|
||||
static let fields: [String: Set<String>] = [
|
||||
"activate": ["appHandle"],
|
||||
"applicationIcons": ["paths"],
|
||||
"click": ["appHandle", "clickCount", "elementIndex", "modifiers", "mouseButton", "snapshotId", "windowId", "x", "y"],
|
||||
"drag": ["appHandle", "durationMs", "fromX", "fromY", "modifiers", "mouseButton", "snapshotId", "steps", "toX", "toY", "windowId"],
|
||||
"findApps": ["maxAppMatches", "query"],
|
||||
@@ -13,9 +14,12 @@ enum NativeProtocol {
|
||||
"launchApp": ["appHandle"],
|
||||
"listApps": [],
|
||||
"listWindows": ["appHandle"],
|
||||
"openPermissionSettings": ["permission"],
|
||||
"paste": ["appHandle", "format", "maxClipboardBytes", "maxDepth", "maxElements", "pasteReadTimeoutMs", "text"],
|
||||
"performSecondaryAction": ["action", "appHandle", "elementIndex", "snapshotId"],
|
||||
"permissionStatus": [],
|
||||
"pressKey": ["appHandle", "key", "modifiers"],
|
||||
"requestPermission": ["permission"],
|
||||
"screenshot": ["appHandle", "windowId"],
|
||||
"scroll": ["appHandle", "deltaLinesX", "deltaLinesY", "snapshotId", "windowId", "x", "y"],
|
||||
"selectText": ["appHandle", "elementIndex", "prefix", "selectionType", "snapshotId", "suffix", "text"],
|
||||
@@ -26,8 +30,10 @@ enum NativeProtocol {
|
||||
static let stringEnums: [String: [String: Set<String>]] = [
|
||||
"click": ["modifiers": ["command", "control", "fn", "option", "shift"], "mouseButton": ["left", "middle", "right"]],
|
||||
"drag": ["modifiers": ["command", "control", "fn", "option", "shift"], "mouseButton": ["left", "middle", "right"]],
|
||||
"openPermissionSettings": ["permission": ["accessibility", "screenRecording"]],
|
||||
"paste": ["format": ["html", "text"]],
|
||||
"pressKey": ["modifiers": ["command", "control", "fn", "option", "shift"]],
|
||||
"requestPermission": ["permission": ["accessibility", "screenRecording"]],
|
||||
"selectText": ["selectionType": ["cursor_after", "cursor_before", "text"]],
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
// System consent is requested only by the Host's user-facing permission controls.
|
||||
import AppKit
|
||||
@preconcurrency import ApplicationServices
|
||||
import Darwin
|
||||
import Foundation
|
||||
|
||||
@MainActor struct SystemPermissions {
|
||||
private let launchApplication: JSONValue?
|
||||
|
||||
// Capture the system-reported responsible app before startup completes and the launcher can exit.
|
||||
init() { launchApplication = Self.responsibleApplication(for: getpid()) }
|
||||
|
||||
func status() -> JSONValue {
|
||||
var result: [String: JSONValue] = [
|
||||
"accessibility": .bool(AXIsProcessTrusted()),
|
||||
"screenRecording": .bool(CGPreflightScreenCaptureAccess()),
|
||||
"helper": Self.process(getpid()),
|
||||
"parent": Self.process(getppid()),
|
||||
]
|
||||
if let launchApplication { result["launchApplication"] = launchApplication }
|
||||
return .object(result)
|
||||
}
|
||||
|
||||
func request(_ permission: String) throws -> JSONValue {
|
||||
switch permission {
|
||||
case "accessibility":
|
||||
if !AXIsProcessTrusted() {
|
||||
let options = [kAXTrustedCheckOptionPrompt.takeUnretainedValue() as String: true] as CFDictionary
|
||||
_ = AXIsProcessTrustedWithOptions(options)
|
||||
}
|
||||
case "screenRecording":
|
||||
if !CGPreflightScreenCaptureAccess() { _ = CGRequestScreenCaptureAccess() }
|
||||
default: throw RequestParameters.invalid("permission must be accessibility or screenRecording")
|
||||
}
|
||||
return status()
|
||||
}
|
||||
|
||||
func openSettings(_ permission: String) throws -> JSONValue {
|
||||
let pane: String
|
||||
switch permission {
|
||||
case "accessibility": pane = "Privacy_Accessibility"
|
||||
case "screenRecording": pane = "Privacy_ScreenCapture"
|
||||
default: throw RequestParameters.invalid("permission must be accessibility or screenRecording")
|
||||
}
|
||||
guard let url = URL(string: "x-apple.systempreferences:com.apple.preference.security?\(pane)"),
|
||||
NSWorkspace.shared.open(url) else {
|
||||
throw NativeFailure(code: "SETTINGS_UNAVAILABLE", message: "Open System Settings > Privacy & Security manually")
|
||||
}
|
||||
return .object(["opened": .bool(true)])
|
||||
}
|
||||
|
||||
/// Resolve macOS process responsibility, including launchers outside the nearest GUI application.
|
||||
static func responsibleApplication(for pid: Int32) -> JSONValue? {
|
||||
guard let responsible = NativePlatform.responsiblePID(pid),
|
||||
let app = NSRunningApplication(processIdentifier: responsible), !app.isTerminated,
|
||||
let bundle = app.bundleURL else { return nil }
|
||||
var result: [String: JSONValue] = ["name": .string(bundle.lastPathComponent), "path": .string(bundle.path)]
|
||||
if let icon = ApplicationCatalog.icon(at: bundle.path) { result["icon"] = .string(icon) }
|
||||
return .object(result)
|
||||
}
|
||||
|
||||
private static func process(_ pid: Int32) -> JSONValue {
|
||||
let path = NativePlatform.executablePath(pid)
|
||||
var result: [String: JSONValue] = ["pid": .number(Double(pid))]
|
||||
if let path { result["path"] = .string(path) }
|
||||
if let name = NSRunningApplication(processIdentifier: pid)?.localizedName { result["name"] = .string(name) }
|
||||
return .object(result)
|
||||
}
|
||||
}
|
||||
@@ -26,6 +26,13 @@ struct RequestParameters: Sendable {
|
||||
guard let value = values[key]?.string else { throw Self.invalid("\(key) must be a string") }
|
||||
return value
|
||||
}
|
||||
func strings(_ key: String) throws -> [String] {
|
||||
guard let raw = values[key], case .array(let items) = raw else { throw Self.invalid("\(key) must be a string array") }
|
||||
return try items.map { value in
|
||||
guard let path = value.string, !path.isEmpty else { throw Self.invalid("\(key) must contain nonempty strings") }
|
||||
return path
|
||||
}
|
||||
}
|
||||
func optionalString(_ key: String) throws -> String? { contains(key) ? try string(key) : nil }
|
||||
func optionalBoolean(_ key: String) throws -> Bool? {
|
||||
guard let raw = values[key] else { return nil }
|
||||
|
||||
@@ -9,6 +9,7 @@ import Darwin
|
||||
private let cancellation: CancellationLatch
|
||||
private let worker: AXWorker
|
||||
private let capture: WindowCapture
|
||||
private let permissions: SystemPermissions
|
||||
private let clipboard = ClipboardPaste()
|
||||
private let input: JSONLInput
|
||||
private let writer: JSONLWriter
|
||||
@@ -27,11 +28,12 @@ import Darwin
|
||||
private var nextCancelled = false
|
||||
|
||||
init(configuration: HelperConfiguration, capture: WindowCapture, channel: SharedSocket,
|
||||
cancellation: CancellationLatch, scheduler: RequestScheduler, writer: JSONLWriter,
|
||||
cancellation: CancellationLatch, scheduler: RequestScheduler, writer: JSONLWriter, permissions: SystemPermissions,
|
||||
onClose: @escaping @MainActor @Sendable () -> Void,
|
||||
onFatal: @escaping @Sendable (NativeFailure, Double) -> Void) {
|
||||
self.configuration = configuration
|
||||
self.capture = capture
|
||||
self.permissions = permissions
|
||||
self.cancellation = cancellation
|
||||
self.channel = channel
|
||||
self.scheduler = scheduler
|
||||
@@ -159,11 +161,25 @@ import Darwin
|
||||
}
|
||||
private func execute(_ request: NativeRequest, trace: ScreenshotTrace?) async throws -> JSONValue {
|
||||
let params = try RequestParameters(request)
|
||||
try cancellation.check()
|
||||
if request.method == "screenshot" { try capture.requireAvailable() }
|
||||
let requiresCatalog = ["getApp", "listApps", "findApps", "typeText", "pressKey"].contains(request.method)
|
||||
let catalog = requiresCatalog ? ApplicationCatalog.snapshot() : ApplicationCatalogSnapshot(applications: [], frontPID: nil)
|
||||
let result: JSONValue
|
||||
if request.method == "findApps" {
|
||||
if request.method == "permissionStatus" {
|
||||
result = permissions.status()
|
||||
} else if request.method == "applicationIcons" {
|
||||
var icons: [String: JSONValue] = [:]
|
||||
for path in try params.strings("paths") {
|
||||
try cancellation.check()
|
||||
if let icon = ApplicationCatalog.icon(at: path) { icons[path] = .string(icon) }
|
||||
}
|
||||
result = .object(icons)
|
||||
} else if request.method == "requestPermission" {
|
||||
result = try permissions.request(params.string("permission"))
|
||||
} else if request.method == "openPermissionSettings" {
|
||||
result = try permissions.openSettings(params.string("permission"))
|
||||
} else if request.method == "findApps" {
|
||||
result = .array(try ApplicationDiscovery.applications(matching: params.string("query"), exact: false,
|
||||
limit: params.integer("maxAppMatches", in: 1...1000), cancellation: cancellation).map { $0.json(catalog: catalog) })
|
||||
} else {
|
||||
@@ -179,7 +195,12 @@ import Darwin
|
||||
let outcome = try await worker.execute(request.method, params: params, catalog: catalog, installed: installed, trace: trace)
|
||||
trace?.mark("worker.end")
|
||||
switch outcome {
|
||||
case .result(let value): result = value
|
||||
case .result(let value):
|
||||
if request.method == "getApp", var fields = value.object {
|
||||
let path = installed?.path ?? catalog.applications.first { Double($0.pid) == value["pid"].number }?.path
|
||||
if let path, let icon = ApplicationCatalog.icon(at: path) { fields["applicationIcon"] = .string(icon) }
|
||||
result = .object(fields)
|
||||
} else { result = value }
|
||||
case .capture(let target):
|
||||
let image = try await capture.capture(target, cancellation: cancellation, trace: trace)
|
||||
result = try await worker.finishCapture(target, image: image, trace: trace)
|
||||
|
||||
@@ -67,6 +67,29 @@ final class NativeSignals {
|
||||
deinit { sources.forEach { $0.cancel() } }
|
||||
}
|
||||
|
||||
// Blocking accepts own only Sendable state; server access belongs to the MainActor callback.
|
||||
private func acceptConnections(listener: SharedSocket, lifetime: SharedHelperLifetime,
|
||||
onAccept: @escaping @MainActor @Sendable (UUID, SharedSocket, CancellationLatch) -> Void) {
|
||||
while !lifetime.isStopping {
|
||||
let descriptor = accept(listener.descriptor, nil, nil)
|
||||
if descriptor < 0 && errno == EINTR { continue }
|
||||
guard descriptor >= 0 else {
|
||||
if !lifetime.isStopping { lifetime.abort(grace: lifetime.shutdownGrace) }
|
||||
return
|
||||
}
|
||||
let channel = SharedSocket(descriptor: descriptor)
|
||||
var uid: uid_t = 0
|
||||
var gid: gid_t = 0
|
||||
guard getpeereid(descriptor, &uid, &gid) == 0, uid == getuid() else { channel.shutdown(); continue }
|
||||
let id = UUID()
|
||||
let cancellation = CancellationLatch()
|
||||
guard lifetime.add(id, channel: channel, cancellation: cancellation) else { channel.shutdown(); return }
|
||||
Task { @MainActor [onAccept, id, channel, cancellation] in
|
||||
onAccept(id, channel, cancellation)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@MainActor final class SharedHelperServer {
|
||||
private let options: SharedHelperOptions
|
||||
private let versionLock: SharedHelperLock
|
||||
@@ -75,6 +98,7 @@ final class NativeSignals {
|
||||
private let lifetime: SharedHelperLifetime
|
||||
private let scheduler = RequestScheduler()
|
||||
private let capture = WindowCapture()
|
||||
private let permissions = SystemPermissions()
|
||||
private let instanceID = UUID().uuidString
|
||||
private var connections: [UUID: RequestCoordinator] = [:]
|
||||
private var startupDeadline: NativeDeadline?
|
||||
@@ -99,12 +123,18 @@ final class NativeSignals {
|
||||
|
||||
func start() throws {
|
||||
let lifetime = self.lifetime
|
||||
signals = NativeSignals { [weak self] number in
|
||||
lifetime.abort(grace: lifetime.shutdownGrace, status: 128 + number)
|
||||
Task { @MainActor in self?.closeConnections(); self?.exitIfEmpty() }
|
||||
let onStop: @MainActor @Sendable () -> Void = { [weak self] in
|
||||
guard let self else { return }
|
||||
self.closeConnections()
|
||||
self.exitIfEmpty()
|
||||
}
|
||||
startupDeadline = NativeDeadline(seconds: options.startupTimeout) { [weak self] in
|
||||
Task { @MainActor in self?.exitIfEmpty() }
|
||||
signals = NativeSignals { [lifetime, onStop] number in
|
||||
lifetime.abort(grace: lifetime.shutdownGrace, status: 128 + number)
|
||||
Task { @MainActor [onStop] in onStop() }
|
||||
}
|
||||
let onIdle: @MainActor @Sendable () -> Void = { [weak self] in self?.exitIfEmpty() }
|
||||
startupDeadline = NativeDeadline(seconds: options.startupTimeout) { [onIdle] in
|
||||
Task { @MainActor [onIdle] in onIdle() }
|
||||
}
|
||||
let ready = options.ready(pid: getpid(), instanceID: instanceID)
|
||||
let data = try JSONEncoder().encode(ready)
|
||||
@@ -113,38 +143,45 @@ final class NativeSignals {
|
||||
try FileHandle.standardOutput.write(contentsOf: data + Data([10]))
|
||||
try FileHandle.standardOutput.close()
|
||||
let listener = self.listener
|
||||
DispatchQueue.global(qos: .utility).async { [weak self] in
|
||||
while !lifetime.isStopping {
|
||||
let descriptor = accept(listener.descriptor, nil, nil)
|
||||
if descriptor < 0 && errno == EINTR { continue }
|
||||
guard descriptor >= 0 else {
|
||||
if !lifetime.isStopping { lifetime.abort(grace: lifetime.shutdownGrace) }
|
||||
return
|
||||
}
|
||||
let channel = SharedSocket(descriptor: descriptor)
|
||||
var uid: uid_t = 0
|
||||
var gid: gid_t = 0
|
||||
guard getpeereid(descriptor, &uid, &gid) == 0, uid == getuid() else { channel.shutdown(); continue }
|
||||
let id = UUID()
|
||||
let cancellation = CancellationLatch()
|
||||
guard lifetime.add(id, channel: channel, cancellation: cancellation) else { channel.shutdown(); return }
|
||||
Task { @MainActor in self?.admit(id, channel: channel, cancellation: cancellation) }
|
||||
let onAccept: @MainActor @Sendable (UUID, SharedSocket, CancellationLatch) -> Void = { [weak self] id, channel, cancellation in
|
||||
guard let self else {
|
||||
channel.shutdown()
|
||||
lifetime.remove(id)
|
||||
return
|
||||
}
|
||||
self.admit(id, channel: channel, cancellation: cancellation)
|
||||
}
|
||||
DispatchQueue.global(qos: .utility).async { [listener, lifetime, onAccept] in
|
||||
acceptConnections(listener: listener, lifetime: lifetime, onAccept: onAccept)
|
||||
}
|
||||
}
|
||||
|
||||
private func admit(_ id: UUID, channel: SharedSocket, cancellation: CancellationLatch) {
|
||||
guard !lifetime.isStopping else { channel.shutdown(); disconnected(id); return }
|
||||
let options = self.options
|
||||
DispatchQueue.global(qos: .utility).async { [weak self] in
|
||||
let lifetime = self.lifetime
|
||||
let onHello: @MainActor @Sendable (HelperConfiguration) async -> Void = { [weak self] configuration in
|
||||
guard let self else {
|
||||
channel.shutdown()
|
||||
lifetime.remove(id)
|
||||
return
|
||||
}
|
||||
await self.connected(id, channel: channel, cancellation: cancellation, configuration: configuration)
|
||||
}
|
||||
let onFailure: @MainActor @Sendable () -> Void = { [weak self] in
|
||||
lifetime.remove(id)
|
||||
self?.exitIfEmpty()
|
||||
}
|
||||
DispatchQueue.global(qos: .utility).async { [options, channel, onHello, onFailure] in
|
||||
do {
|
||||
let configuration = try HelperConfiguration(hello: channel.readHello(timeout: options.startupTimeout), runtimeKey: options.runtimeKey)
|
||||
Task { @MainActor in await self?.connected(id, channel: channel, cancellation: cancellation, configuration: configuration) }
|
||||
Task { @MainActor [onHello, configuration] in await onHello(configuration) }
|
||||
} catch {
|
||||
let failure = error as? NativeFailure ?? NativeFailure(code: "NATIVE_PROTOCOL", message: "Invalid shared helper hello")
|
||||
let response = JSONValue.object(["type": .string("error"), "code": .string(failure.code), "message": .string(failure.message)])
|
||||
if let data = try? JSONEncoder().encode(response) { try? channel.write(data + Data([10]), timeout: options.startupTimeout) }
|
||||
channel.shutdown()
|
||||
Task { @MainActor in self?.disconnected(id) }
|
||||
Task { @MainActor [onFailure] in onFailure() }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -155,14 +192,18 @@ final class NativeSignals {
|
||||
let writer = JSONLWriter(limit: configuration.maxResponseBytes, channel: channel, writeTimeout: configuration.shutdownGraceSeconds)
|
||||
do { try await writer.write(options.ready(pid: getpid(), instanceID: instanceID)) }
|
||||
catch { channel.shutdown(); disconnected(id); return }
|
||||
// Signal shutdown can run while the ready frame is being written.
|
||||
guard !lifetime.isStopping else { channel.shutdown(); disconnected(id); return }
|
||||
let lifetime = self.lifetime
|
||||
let onFatal: @MainActor @Sendable () -> Void = { [weak self] in self?.closeConnections() }
|
||||
let coordinator = RequestCoordinator(configuration: configuration, capture: capture, channel: channel, cancellation: cancellation,
|
||||
scheduler: scheduler, writer: writer, onClose: { [weak self] in
|
||||
scheduler: scheduler, writer: writer, permissions: permissions, onClose: { [weak self] in
|
||||
channel.shutdown()
|
||||
self?.disconnected(id)
|
||||
}, onFatal: { [weak self] _, grace in
|
||||
guard let self else { lifetime.remove(id); return }
|
||||
self.disconnected(id)
|
||||
}, onFatal: { [lifetime, onFatal] _, grace in
|
||||
lifetime.abort(grace: grace)
|
||||
Task { @MainActor in self?.closeConnections() }
|
||||
Task { @MainActor [onFatal] in onFatal() }
|
||||
})
|
||||
connections[id] = coordinator
|
||||
startupDeadline?.cancel()
|
||||
|
||||
@@ -1025,6 +1025,131 @@
|
||||
"params"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"method": {
|
||||
"type": "string",
|
||||
"const": "permissionStatus"
|
||||
},
|
||||
"params": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"id",
|
||||
"method",
|
||||
"params"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"method": {
|
||||
"type": "string",
|
||||
"const": "applicationIcons"
|
||||
},
|
||||
"params": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"paths": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"paths"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"id",
|
||||
"method",
|
||||
"params"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"method": {
|
||||
"type": "string",
|
||||
"const": "requestPermission"
|
||||
},
|
||||
"params": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"permission": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"accessibility",
|
||||
"screenRecording"
|
||||
]
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"permission"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"id",
|
||||
"method",
|
||||
"params"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"method": {
|
||||
"type": "string",
|
||||
"const": "openPermissionSettings"
|
||||
},
|
||||
"params": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"permission": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"accessibility",
|
||||
"screenRecording"
|
||||
]
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"permission"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"id",
|
||||
"method",
|
||||
"params"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
// An owned GUI launcher with no windows; its child outlives the launch application.
|
||||
import AppKit
|
||||
import Foundation
|
||||
import Darwin
|
||||
|
||||
@MainActor final class Launcher: NSObject, NSApplicationDelegate {
|
||||
private let child = Process()
|
||||
func applicationDidFinishLaunching(_ notification: Notification) {
|
||||
let arguments = Array(CommandLine.arguments.dropFirst())
|
||||
guard let executable = arguments.first else { exit(64) }
|
||||
child.executableURL = URL(fileURLWithPath: executable)
|
||||
child.arguments = Array(arguments.dropFirst())
|
||||
child.standardOutput = FileHandle.standardOutput
|
||||
child.standardError = FileHandle.standardError
|
||||
child.terminationHandler = { @Sendable process in exit(process.terminationStatus) }
|
||||
do { try child.run() }
|
||||
catch { fputs("Launching fixture child failed: \(error)\n", stderr); exit(1) }
|
||||
}
|
||||
}
|
||||
|
||||
@main enum LauncherMain {
|
||||
@MainActor static func main() {
|
||||
let app = NSApplication.shared
|
||||
app.setActivationPolicy(.accessory)
|
||||
let launcher = Launcher()
|
||||
app.delegate = launcher
|
||||
withExtendedLifetime(launcher) { app.run() }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Verify system responsibility across nested GUI launchers and helper reuse; no TCC grants are required."""
|
||||
import argparse
|
||||
import base64
|
||||
from contextlib import ExitStack
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import plistlib
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import tempfile
|
||||
import uuid
|
||||
|
||||
from native_test_support import BINARY, ROOT
|
||||
from shared_helper_e2e import Connection, wait_released
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--helper", type=Path, default=BINARY)
|
||||
args = parser.parse_args()
|
||||
with tempfile.TemporaryDirectory(prefix="cua-", dir="/tmp") as temporary, ExitStack() as cleanup:
|
||||
root = Path(temporary)
|
||||
helper = root / "helper"
|
||||
shutil.copy2(args.helper.resolve(strict=True), helper)
|
||||
key = hashlib.sha256(helper.read_bytes()).hexdigest()
|
||||
runtime = root / "r"
|
||||
bundle = root / "Visual Studio Code.app"
|
||||
binaries = bundle / "Contents" / "MacOS"
|
||||
binaries.mkdir(parents=True)
|
||||
launcher = binaries / "launcher"
|
||||
subprocess.run(["xcrun", "swiftc", "-parse-as-library", str(ROOT / "tests/LauncherFixture.swift"), "-o", str(launcher)],
|
||||
check=True, timeout=60)
|
||||
nested = root / "Harness Dev.app"
|
||||
nested_launcher = nested / "Contents" / "MacOS" / "launcher"
|
||||
nested_launcher.parent.mkdir(parents=True)
|
||||
shutil.copy2(launcher, nested_launcher)
|
||||
identity = uuid.uuid4().hex
|
||||
register = "/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister"
|
||||
for role, application, name in [("owner", bundle, "Code"), ("nested", nested, "Harness Dev")]:
|
||||
with (application / "Contents" / "Info.plist").open("wb") as output:
|
||||
plistlib.dump({"CFBundleExecutable": "launcher",
|
||||
"CFBundleIdentifier": f"dev.dsh.cua.launcher-fixture.{role}-{identity}",
|
||||
"CFBundleName": name, "CFBundleDisplayName": name, "CFBundlePackageType": "APPL",
|
||||
"LSUIElement": True}, output)
|
||||
cleanup.callback(subprocess.run, [register, "-u", str(application)], check=True,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=10)
|
||||
ready_file, stderr_file = root / "ready.json", root / "stderr.log"
|
||||
# Launch Services establishes the outer app's responsibility; its GUI child inherits it.
|
||||
subprocess.run(["/usr/bin/open", "-n", "-g", "-W", "--stdout", str(ready_file), "--stderr", str(stderr_file),
|
||||
str(bundle), "--args", str(nested_launcher), str(helper), "--ensure-shared", "--runtime-key", key,
|
||||
"--runtime-directory", str(runtime), "--startup-timeout-ms", "10000"],
|
||||
check=True, capture_output=True, text=True, timeout=25)
|
||||
ready = json.loads(ready_file.read_text())
|
||||
owner = ready["pid"]
|
||||
cleanup.callback(wait_released, runtime / (key + ".lock"), owner)
|
||||
first = Connection(runtime / (key + ".sock"), key)
|
||||
cleanup.callback(first.close)
|
||||
state = first.call("permissionStatus", {})
|
||||
assert state["parent"]["pid"] == 1, state["parent"]
|
||||
source = state["launchApplication"]
|
||||
assert source["name"] == bundle.name, source["name"]
|
||||
assert Path(source["path"]).resolve() == bundle.resolve(), source["path"]
|
||||
assert source["icon"].startswith("data:image/png;base64,")
|
||||
image = base64.b64decode(source["icon"].split(",", 1)[1], validate=True)
|
||||
assert image[:8] == b"\x89PNG\r\n\x1a\n"
|
||||
assert struct.unpack(">II", image[16:24]) == (64, 64)
|
||||
second = Connection(runtime / (key + ".sock"), key)
|
||||
cleanup.callback(second.close)
|
||||
assert second.call("permissionStatus", {})["launchApplication"] == source
|
||||
print("PASS: system-responsible app wins over the nearest GUI ancestor and survives launcher exit and helper reuse")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -5,6 +5,7 @@ Accessibility/Screen Recording grants are required unless --without-desktop is s
|
||||
"""
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
from contextlib import ExitStack
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
@@ -128,6 +129,15 @@ def main():
|
||||
assert a.ready == b.ready == ready[0]
|
||||
for client in clients:
|
||||
client.call("protocol-unsupported", {}, error="UNSUPPORTED")
|
||||
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as invalid:
|
||||
invalid.settimeout(15)
|
||||
invalid.connect(str(socket_path))
|
||||
invalid.sendall(b'{"type":"hello","protocol":2}\n')
|
||||
with invalid.makefile("rb") as reader:
|
||||
failure = json.loads(reader.readline())
|
||||
assert failure["type"] == "error" and failure["code"] == "NATIVE_PROTOCOL", failure
|
||||
assert reader.readline() == b"", "Rejected handshake retained its connection"
|
||||
b.call("protocol-after-invalid-hello", {}, error="UNSUPPORTED")
|
||||
# Coalesced frames retain socket order, including cancellation before execution starts.
|
||||
a.socket.sendall(b'{"id":"coalesced","method":"listApps","params":{}}\n'
|
||||
b'{"type":"cancel","id":"coalesced"}\n')
|
||||
@@ -218,20 +228,39 @@ def main():
|
||||
wait_released(lock_path, owner_pid)
|
||||
owner_pid = None
|
||||
|
||||
daemon = subprocess.Popen([base[0], "--serve", *base[1:]], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
try:
|
||||
assert select.select([daemon.stdout], [], [], 15)[0], "Daemon did not report ready"
|
||||
signalled_ready = json.loads(daemon.stdout.readline())
|
||||
assert signalled_ready["type"] == "ready", signalled_ready
|
||||
daemon.send_signal(signal.SIGTERM)
|
||||
status = daemon.wait(timeout=15)
|
||||
assert status == 128 + signal.SIGTERM, (status, daemon.stderr.read())
|
||||
wait_released(lock_path, daemon.pid)
|
||||
finally:
|
||||
stop(daemon)
|
||||
daemon.stdout.close()
|
||||
daemon.stderr.close()
|
||||
print("PASS: concurrent launch, shared PID, client isolation, cleanup/lock release and SIGTERM")
|
||||
for shutdown in ["signal", "incomplete-hello", "idle"]:
|
||||
serve_options = base[1:] if shutdown != "idle" else [*base[1:-1], "100"]
|
||||
daemon = subprocess.Popen([base[0], "--serve", *serve_options], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
try:
|
||||
with ExitStack() as connections:
|
||||
assert select.select([daemon.stdout], [], [], 15)[0], "Daemon did not report ready"
|
||||
signalled_ready = json.loads(daemon.stdout.readline())
|
||||
assert signalled_ready["type"] == "ready", signalled_ready
|
||||
if shutdown == "incomplete-hello":
|
||||
pending = connections.enter_context(socket.socket(socket.AF_UNIX, socket.SOCK_STREAM))
|
||||
pending.settimeout(15)
|
||||
pending.connect(str(socket_path))
|
||||
pending.sendall(b'{"type":"hello"')
|
||||
# A later completed handshake observes the accept loop progressing past this socket.
|
||||
active = Connection(socket_path, key)
|
||||
connections.callback(active.socket.close)
|
||||
connections.callback(active.reader.close)
|
||||
if shutdown != "idle":
|
||||
daemon.send_signal(signal.SIGTERM)
|
||||
status = daemon.wait(timeout=15)
|
||||
expected_status = 0 if shutdown == "idle" else 128 + signal.SIGTERM
|
||||
assert status == expected_status, (status, daemon.stderr.read())
|
||||
wait_released(lock_path, daemon.pid)
|
||||
# The forced-exit deadline does not unlink these files; require normal cleanup.
|
||||
assert not socket_path.exists(), "Shutdown left its listening socket"
|
||||
assert not (runtime / (key + ".ready")).exists(), "Shutdown left its ready record"
|
||||
if shutdown == "incomplete-hello":
|
||||
assert pending.recv(1) == b"", "Signal shutdown retained an incomplete handshake"
|
||||
finally:
|
||||
stop(daemon)
|
||||
daemon.stdout.close()
|
||||
daemon.stderr.close()
|
||||
print("PASS: concurrent launch, shared PID, client isolation, rejected/incomplete handshakes, cleanup/lock release, SIGTERM and idle exit")
|
||||
finally:
|
||||
errors = []
|
||||
for client in reversed(clients):
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
// The selected catalog entry must still identify the executable inspected for consent.
|
||||
import AppKit
|
||||
import Darwin
|
||||
import Foundation
|
||||
import Testing
|
||||
@testable import CuaNative
|
||||
|
||||
@@ -22,3 +24,37 @@ import Testing
|
||||
#expect(error.code == "APP_UNAVAILABLE")
|
||||
}
|
||||
}
|
||||
|
||||
@MainActor @Test func applicationIconsUseBoundedNativePNGs() throws {
|
||||
let encoded = try #require(ApplicationCatalog.icon(at: "/System/Library/CoreServices/Finder.app/Contents/MacOS/Finder"))
|
||||
#expect(encoded.hasPrefix("data:image/png;base64,"))
|
||||
let data = try #require(Data(base64Encoded: String(encoded.dropFirst("data:image/png;base64,".count))))
|
||||
let bitmap = try #require(NSBitmapImageRep(data: data))
|
||||
#expect(bitmap.pixelsWide == 64)
|
||||
#expect(bitmap.pixelsHigh == 64)
|
||||
#expect(encoded.utf8.count <= 65536)
|
||||
}
|
||||
|
||||
@Test func resolvesTheCurrentProcessResponsibility() throws {
|
||||
let responsible = try #require(NativePlatform.responsiblePID(getpid()))
|
||||
#expect(responsible > 0)
|
||||
#expect(NativePlatform.responsiblePID(responsible) == responsible)
|
||||
}
|
||||
|
||||
@MainActor @Test func unknownResponsibilityDoesNotInventAnApplication() {
|
||||
#expect(SystemPermissions.responsibleApplication(for: 1) == nil)
|
||||
#expect(SystemPermissions.responsibleApplication(for: Int32.max) == nil)
|
||||
#expect(NativePlatform.responsiblePID(Int32.max) == nil)
|
||||
}
|
||||
|
||||
@MainActor @Test func permissionStatusKeepsProcessDiagnosticsSeparateFromLaunchMetadata() {
|
||||
let permissions = SystemPermissions()
|
||||
let state = permissions.status()
|
||||
#expect(state["helper"]["pid"] == .number(Double(getpid())))
|
||||
#expect(state["parent"]["pid"] == .number(Double(getppid())))
|
||||
if let source = state["launchApplication"].object {
|
||||
#expect(source["name"]?.string?.isEmpty == false)
|
||||
#expect(source["path"]?.string?.hasSuffix(".app") == true)
|
||||
}
|
||||
#expect(permissions.status()["launchApplication"] == state["launchApplication"])
|
||||
}
|
||||
@@ -116,3 +116,9 @@ func rejectsInvalidWindowIDs(_ value: String) throws {
|
||||
#expect(throws: NativeFailure.self) { try params.optionalBoolean("includeMenus") }
|
||||
}
|
||||
}
|
||||
|
||||
@Test(arguments: ["null", "true", "[1]", "[\"\"]"])
|
||||
func rejectsMalformedApplicationIconPaths(_ value: String) throws {
|
||||
let request = try NativeRequest(data: Data("{\"id\":\"icons\",\"method\":\"applicationIcons\",\"params\":{\"paths\":\(value)}}".utf8))
|
||||
#expect(throws: NativeFailure.self) { try RequestParameters(request).strings("paths") }
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
/** macOS operation guidance supplied with the shared CUA object API. */
|
||||
/** Instructions for raw operation calls and optional object-API bindings. */
|
||||
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user allows another request on the permission page. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
|
||||
@@ -37,7 +37,7 @@ App and Window methods:
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to complete the Computer Use permission setup for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
|
||||
@@ -73,7 +73,7 @@ it.skipIf(process.platform === 'win32')('delivers a valid PNG larger than 4 MiB
|
||||
const configPath = join(root, 'cordis.yml')
|
||||
const configs: Record<string, object> = {
|
||||
tools: { mode: 'ptc' }, attachments: { dshHome: join(root, 'home') }, workingDirectory: { defaultDirectory: root },
|
||||
fs: { cwd: root }, sandboxPolicy: { mode: 'danger-full-access', workspaceRoot: root }, cua: { helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] },
|
||||
fs: { cwd: root }, sandboxPolicy: { mode: 'danger-full-access', workspaceRoot: root }, cua: { agentPreset: false, helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] },
|
||||
}
|
||||
await writeFile(configPath, Object.keys(modules).flatMap(name => [
|
||||
`- name: 'cordis:${name}'`, ...configs[name] === undefined ? [] : [` config: ${JSON.stringify(configs[name])}`],
|
||||
|
||||
@@ -85,7 +85,7 @@ it.skipIf(process.platform === 'win32')('shares one application between retained
|
||||
await writeFile(configPath, Object.keys(modules).flatMap(name => [
|
||||
`- name: 'cordis:${name}'`,
|
||||
...name === 'attachments' ? [' config:', ` dshHome: ${JSON.stringify(root)}`] : [],
|
||||
...name === 'cua' ? [' config:', ` helperPath: ${JSON.stringify(helperPath)}`, ' allowedApps: [{"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app"}]'] : [],
|
||||
...name === 'cua' ? [' config:', ' agentPreset: false', ` helperPath: ${JSON.stringify(helperPath)}`, ' allowedApps: [{"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app"}]'] : [],
|
||||
]).join('\n') + '\n')
|
||||
context.baseUrl = pathToFileURL(root).href + '/'
|
||||
await context.plugin(Loader)
|
||||
|
||||
@@ -50,7 +50,7 @@ it.skipIf(process.platform === 'win32' || !existsSync(bridge) || !existsSync(boo
|
||||
await ctx.plugin(NodeRuntime, { executionMode: 'persistent-in-memory', launch: { kind: 'node-script', executable: process.execPath, bootstrapPath: bootstrap } })
|
||||
await ctx.plugin(Loader)
|
||||
ctx.loader.builtins.cua = Cua
|
||||
const entry = await ctx.loader.create({ name: 'cordis:cua', config: { helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] } })
|
||||
const entry = await ctx.loader.create({ name: 'cordis:cua', config: { agentPreset: false, helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] } })
|
||||
await ctx.loader.resolve(entry).fiber!.await()
|
||||
await mountAgentLoopTestHarness(ctx)
|
||||
ctx.llm.registerAdapter(['bridge-fixture'], new VisualAdapter())
|
||||
|
||||
@@ -18,6 +18,15 @@ describe('native protocol generation', () => {
|
||||
{ method: 'paste', params: { appHandle: 'app', text: 'text', pasteReadTimeoutMs: 1000 } },
|
||||
{ method: 'findApps', params: { query: 'Editor', maxAppMatches: 1 } },
|
||||
{ method: 'launchApp', params: { appHandle: 'pending' } },
|
||||
{ method: 'permissionStatus', params: {} },
|
||||
{ method: 'requestPermission', params: { permission: 'accessibility' } },
|
||||
{ method: 'openPermissionSettings', params: { permission: 'screenRecording' } },
|
||||
{ method: 'permissionStatus', params: {} },
|
||||
{ method: 'requestPermission', params: { permission: 'accessibility' } },
|
||||
{ method: 'openPermissionSettings', params: { permission: 'screenRecording' } },
|
||||
{ method: 'permissionStatus', params: {} },
|
||||
{ method: 'requestPermission', params: { permission: 'accessibility' } },
|
||||
{ method: 'openPermissionSettings', params: { permission: 'screenRecording' } },
|
||||
])('rejects Host-only request data at model admission: $method', (request) => {
|
||||
expect(Request.safeParse(request).success).toBe(false)
|
||||
})
|
||||
|
||||
@@ -1,16 +1,9 @@
|
||||
/** Native helper replacement is synchronized with Agent and provider disposal. */
|
||||
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { mkdtemp, rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join, resolve } from 'node:path'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { setImmediate } from 'node:timers/promises'
|
||||
import { Context, FiberState } from '@deepseek-ai/cordis'
|
||||
import Loader from '@deepseek-ai/cordis-plugin-loader'
|
||||
import Timer from '@deepseek-ai/cordis-plugin-timer'
|
||||
import { initProfile, mountRootInclude, readProfilePatches, type ProfileContext } from '@deepseek-ai/dsh-app-boot'
|
||||
import ConfigEditor from '@deepseek-ai/dsh-config-editor'
|
||||
import Settings from '@deepseek-ai/dsh-settings'
|
||||
import Hmr from '@deepseek-ai/dsh-hmr'
|
||||
import { mountAgentLoopTestDependencies, mountAgentLoopTestHarness } from '@deepseek-ai/dsh-agent-loop-testkit'
|
||||
import ComputerUseRegistry from '@deepseek-ai/dsh-computer-use'
|
||||
import LocalAttachmentStore from '@deepseek-ai/dsh-attachment-local'
|
||||
@@ -55,6 +48,7 @@ vi.mock('@deepseek-ai/dsh-cua/native-client', async importOriginal => ({
|
||||
}))
|
||||
|
||||
const config = {
|
||||
agentPreset: false,
|
||||
helperPath: resolve('fixture-helper'), buildTimeoutMs: 300_000, timeoutMs: 10_000, connectTimeoutMs: 5000, screenshotTimeoutMs: 5000, axTimeoutMs: 3000, maxRequestBytes: 1024, maxResponseBytes: 4096, maxClipboardBytes: 8192, pasteReadTimeoutMs: 1000, maxAppMatches: 20,
|
||||
graceMs: 10, textEventIntervalMs: 8, maxElements: 64, maxDepth: 8, dragDurationMs: 200, dragSteps: 10,
|
||||
} satisfies Omit<NativeProvider.Config, 'allowedApps'>
|
||||
@@ -93,9 +87,9 @@ async function setup(model?: { provider: string; model: string }) {
|
||||
|
||||
async function selectApplication(f: Awaited<ReturnType<typeof setup>>) {
|
||||
await f.ctx.plugin(UserQuestions)
|
||||
f.ctx.on('user-questions/request', async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this task'] }] }))
|
||||
f.ctx.on('user-questions/request', async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] }))
|
||||
native.request.mockResolvedValueOnce({ appHandle: 'app-one', pid: 17, name: 'Fixture',
|
||||
authorization: { id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: true } })
|
||||
authorization: { id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: false } })
|
||||
expect((await f.execute('getApp', { selector: 'Fixture' })).isError).toBe(false)
|
||||
}
|
||||
|
||||
@@ -105,7 +99,7 @@ it('rejects a relative helper path before publishing computer-use operations', a
|
||||
await expect(NativeProvider.apply(ctx, { ...config, allowedApps: { get: () => [] }, helperPath: 'relative-helper' })).rejects.toThrow('helperPath must be absolute')
|
||||
})
|
||||
|
||||
it.each(['Allow for this task', 'Deny'])('obtains %s before a stopped application can launch', async (choice) => {
|
||||
it.each(['Allow for this session', 'Deny'])('obtains %s before a stopped application can launch', async (choice) => {
|
||||
const f = await setup()
|
||||
await f.ctx.plugin(UserQuestions)
|
||||
const asked: PromiseWithResolvers<void> = Promise.withResolvers()
|
||||
@@ -130,7 +124,7 @@ it.each(['Allow for this task', 'Deny'])('obtains %s before a stopped applicatio
|
||||
it('rejects a launch result that changes the approved handle', async () => {
|
||||
const f = await setup()
|
||||
await f.ctx.plugin(UserQuestions)
|
||||
f.ctx.on('user-questions/request', async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this task'] }] }))
|
||||
f.ctx.on('user-questions/request', async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] }))
|
||||
native.request.mockResolvedValueOnce({ appHandle: 'pending-app', pid: null, name: 'Fixture', authorization: {
|
||||
id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: false,
|
||||
} }).mockResolvedValueOnce({ appHandle: 'another-app', pid: 17, name: 'Fixture' })
|
||||
@@ -257,6 +251,24 @@ it('keeps Agent disposal pending until its helper cleanup completes', async () =
|
||||
} finally { release.resolve(); await disposing }
|
||||
})
|
||||
|
||||
it('retains the computer-use registration until permission helper cleanup settles', async () => {
|
||||
const f = await setup()
|
||||
native.request.mockResolvedValue({ accessibility: false, screenRecording: false, helper: { pid: 9 }, parent: { pid: 8 } })
|
||||
await f.ctx.cuaAccess.systemStatus()
|
||||
const entered = Promise.withResolvers<undefined>(), release = Promise.withResolvers<undefined>()
|
||||
native.dispose.mockImplementation(async () => { entered.resolve(undefined); await release.promise })
|
||||
onTestFinished(() => { release.resolve(undefined) })
|
||||
const registry = f.ctx.computerUse
|
||||
const closing = f.provider.dispose()
|
||||
try {
|
||||
await entered.promise
|
||||
expect(registry.providerName).toBe('dsh-cua')
|
||||
release.resolve(undefined)
|
||||
await closing
|
||||
expect(registry.providerName).toBeUndefined()
|
||||
} finally { release.resolve(undefined); await closing }
|
||||
})
|
||||
|
||||
it('joins helper cleanup already started by Agent disposal when the provider unloads', async () => {
|
||||
const f = await setup()
|
||||
await f.execute()
|
||||
@@ -497,6 +509,8 @@ it.each([
|
||||
{ owner: 'provider', stage: 'storage' },
|
||||
{ owner: 'agent', stage: 'admission' },
|
||||
{ owner: 'agent', stage: 'storage' },
|
||||
{ owner: 'revocation', stage: 'admission' },
|
||||
{ owner: 'revocation', stage: 'storage' },
|
||||
])('waits for screenshot $stage and cancels its result during $owner disposal', async ({ owner, stage }) => {
|
||||
const f = await setup({ provider: 'image-fixture', model: 'vision' })
|
||||
await selectApplication(f)
|
||||
@@ -526,7 +540,9 @@ it.each([
|
||||
const reached = await Promise.race([entered.promise.then(() => true), pending.then(() => false)])
|
||||
expect(reached).toBe(true)
|
||||
let closed = false
|
||||
closing = (owner === 'provider' ? f.provider.dispose() : f.handle.dispose()).then(() => { closed = true })
|
||||
closing = (owner === 'provider' ? f.provider.dispose()
|
||||
: owner === 'agent' ? f.handle.dispose()
|
||||
: f.ctx.cuaAccess.revoke('a'.repeat(64) as import('@deepseek-ai/dsh-cua/types').CuaApplicationId)).then(() => { closed = true })
|
||||
await setImmediate()
|
||||
expect(admissionSignal?.aborted).toBe(true)
|
||||
expect(closed).toBe(false)
|
||||
@@ -622,79 +638,3 @@ it('rejects an unsupported native host before starting preparation', async () =>
|
||||
await expect(NativeProvider.apply(f.ctx, { ...automaticConfig, allowedApps: { get: () => [] } })).rejects.toThrow('unsupported host')
|
||||
expect(native.prepare).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
|
||||
it('stores signed application consent in the current profile and refuses unavailable settings', async () => {
|
||||
const f = await setup()
|
||||
await f.provider.dispose()
|
||||
const home = await mkdtemp(join(tmpdir(), 'cua-grant-profile-')), dir = join(home, 'profile')
|
||||
onTestFinished(async () => { await f.ctx.fiber.dispose(); await rm(home, { recursive: true, force: true }) })
|
||||
initProfile(dir, ['permission-fixture-bundle'])
|
||||
const bundle = join(dir, 'node_modules', 'permission-fixture-bundle')
|
||||
await mkdir(bundle, { recursive: true })
|
||||
await writeFile(join(home, 'package.json'), '{"name":"test-installation"}\n')
|
||||
await writeFile(join(bundle, 'package.json'), JSON.stringify({ name: 'permission-fixture-bundle', version: '1.0.0',
|
||||
dsh: { bundle: { patch: 'cordis.patch.yml' } } }))
|
||||
await writeFile(join(bundle, 'cordis.patch.yml'), JSON.stringify([{ insert: [{ id: 'cua', name: 'cordis:cua', config }] }]))
|
||||
await writeFile(join(dir, 'cordis.yml'), '[]\n')
|
||||
const profile: ProfileContext = { name: 'test', startedBundles: ['permission-fixture-bundle'], dir,
|
||||
patchPath: join(dir, 'cordis.patch.yml'), installAnchor: join(home, 'package.json'), cwd: home, home,
|
||||
overlays: [], telemetryDisabledEnv: undefined }
|
||||
f.ctx.baseUrl = pathToFileURL(home).href + '/'
|
||||
f.ctx.provide('profileContext', profile)
|
||||
f.ctx.provide('appReady', { onReady: (listener: () => void) => { listener(); return () => {} } })
|
||||
await f.ctx.plugin(Loader)
|
||||
f.ctx.loader.builtins.cua = NativeProvider
|
||||
await f.ctx.plugin(ConfigEditor)
|
||||
await mountRootInclude(f.ctx, join(dir, 'cordis.yml'), readProfilePatches('test', profile))
|
||||
await f.ctx.loader.await()
|
||||
await f.ctx.plugin(UserQuestions)
|
||||
const answer = vi.fn(async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this task'] }] }))
|
||||
f.ctx.on('user-questions/request', answer)
|
||||
native.request.mockResolvedValue({ appHandle: 'task-app', pid: 17, name: 'Fixture',
|
||||
authorization: { id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: true } })
|
||||
expect((await f.execute('getApp', { selector: 'Fixture' })).isError).toBe(false)
|
||||
await f.ctx.plugin(Settings)
|
||||
await f.ctx.plugin(Timer)
|
||||
await f.ctx.plugin(Hmr, { root: [], ignored: [], debounce: 0 })
|
||||
await f.ctx.hmr.runExclusive(async () => {})
|
||||
expect(f.ctx.settings.describe().find(row => row.ns === 'cua')?.autoGenerate).toBe(false)
|
||||
answer.mockResolvedValue({ answers: [{ id: 'cua-app-access', selected: ['Always allow'] }] })
|
||||
const authorization = { id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: true }
|
||||
native.request.mockResolvedValue({ appHandle: 'signed-app', pid: 17, name: 'Fixture', authorization })
|
||||
expect((await f.execute('getApp', { selector: 'Fixture' })).isError).toBe(false)
|
||||
expect(await readFile(profile.patchPath, 'utf8')).toContain(authorization.id)
|
||||
const settings = f.ctx.settings.describe().find(row => row.ns === 'cua')!
|
||||
await f.ctx.settings.mutate('cua', [{ op: 'set', path: ['allowedApps'], value: [] }], settings.revision)
|
||||
expect(await readFile(profile.patchPath, 'utf8')).toContain('allowedApps: []')
|
||||
const revoked = await f.execute('getState', { appHandle: 'signed-app' })
|
||||
expect(revoked.isError).toBe(true)
|
||||
expect(JSON.stringify(revoked.content)).toContain('APP_ACCESS_REVOKED')
|
||||
vi.spyOn(f.ctx.settings, 'describe').mockReturnValue([])
|
||||
native.request.mockResolvedValue({ appHandle: 'different-app', pid: 18, name: 'Other fixture', authorization: { ...authorization, id: 'b'.repeat(64) } })
|
||||
const unavailable = await f.execute('getApp', { selector: 'Fixture' })
|
||||
expect(unavailable.isError).toBe(true)
|
||||
expect(JSON.stringify(unavailable.content)).toContain('APP_ACCESS_DENIED')
|
||||
})
|
||||
|
||||
it('rejects an application handle before any helper or consent has been acquired', async () => {
|
||||
const f = await setup()
|
||||
const result = await f.execute('getState', { appHandle: 'unselected-app' })
|
||||
expect(result.isError).toBe(true)
|
||||
expect(JSON.stringify(result.content)).toContain('APP_ACCESS_REQUIRED')
|
||||
expect(native.instances).toHaveLength(0)
|
||||
expect(native.request).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('uses configured signed application grants without requesting a human answer', async () => {
|
||||
const f = await setup()
|
||||
await f.provider.dispose()
|
||||
const grant = { id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app' }
|
||||
await f.ctx.plugin(NativeProvider, { ...config, allowedApps: [grant] })
|
||||
native.request.mockResolvedValueOnce({ appHandle: 'saved-app', pid: 17, name: 'Fixture', authorization: { ...grant, persistable: true } })
|
||||
expect((await f.execute('getApp', { selector: 'Fixture' })).isError).toBe(false)
|
||||
native.request.mockResolvedValue({ dispatched: true, verified: false })
|
||||
expect((await f.execute('activate', { appHandle: 'saved-app' })).isError).toBe(false)
|
||||
expect(native.instances).toHaveLength(1)
|
||||
expect(native.request).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
@@ -12,14 +12,14 @@
|
||||
en: f4d9cc48587d62a2
|
||||
zh: d1bac6c1fe7eb29e
|
||||
/deepseek-ai-dsh-cua/table-of-contents:
|
||||
en: d152484eb41ac6b4
|
||||
zh: 1db86285abf6c499
|
||||
en: 4081cf444a77bf20
|
||||
zh: def91d58969ec281
|
||||
/deepseek-ai-dsh-cua/use-this-package:
|
||||
en: 52cacbe2d2aeb2bc
|
||||
zh: b3c65814d9a17052
|
||||
en: e230429b9e15efd8
|
||||
zh: 10dcb9ba5194dc76
|
||||
/deepseek-ai-dsh-cua/understand-the-implementation:
|
||||
en: effcee57d8854c32
|
||||
zh: d02c6b76e7249051
|
||||
en: 751fe580f79dfaae
|
||||
zh: 3811db65f634ef5a
|
||||
/deepseek-ai-dsh-cua/further-exploration:
|
||||
en: b8b0aece886fadaa
|
||||
zh: ecd8c88d601f2179
|
||||
|
||||
@@ -20,6 +20,8 @@ Inspect applications, retain window objects across PTC calls, and send actions t
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
- [Dev Note](#dev-note)
|
||||
|
||||
Application consent uses the existing User Questions options and remains independent of the session tool-approval policy. The provider accepts only one exact offered label without custom text. Native icons accompany permission-page observations and remain outside saved identities and model results. System checks, saved grants and refusals, and icons use separate queries.
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
@@ -33,7 +35,7 @@ Observation methods emit and return their result. Use `void await app.getAXState
|
||||
|
||||
`tools.cua` returns operation values directly. Screenshot values retain the encoded image Base64 and add a durable `attachment` reference for the tool renderer; model content contains coordinate metadata and one image, without the encoded bytes. Before capture, the tool requires an attachment store and a resolved model route declaring image input. Admission and storage failures reject the call. The object bridge preserves the screenshot bytes and validated metadata without uploading or emitting the same image again.
|
||||
|
||||
Application consent precedes launch, observation, capture, and input. Task access belongs to the selected instance; saved grants require platform-verified executable identity. Revocation closes affected Agent connections, so their old objects require fresh selection. No input is replayed after cancellation, timeout, or reconnect.
|
||||
Application consent precedes launch, observation, capture, and input. Session access is kept in memory by session and application identity, survives agent and helper replacement, and expires when DSH restarts or Computer Use is disabled. Saved grants require platform-verified executable identity. Refusal and revocation close affected Agent connections, so their old objects require fresh selection. No input is replayed after cancellation, timeout, or reconnect.
|
||||
|
||||
Actions return `dispatched:true, verified:false`; read fresh state to verify the result. Coordinate actions default to pixels in the latest returned screenshot; `relative: true` accepts integers from 0 to 1000 after a successful window observation. The Host resolves attachment normalization and the selected model route’s image projection before returning the screenshot, then converts input to native window points. Resizing invalidates the screenshot mapping; `SCREENSHOT_REQUIRED` requires a new screenshot for absolute coordinates. Resizing rejects old coordinates with `WINDOW_CHANGED`; movement alone preserves window-local coordinates.
|
||||
|
||||
@@ -47,6 +49,8 @@ Actions return `dispatched:true, verified:false`; read fresh state to verify the
|
||||
|
||||
The provider calls [installCua](src/index.ts) with its prepared helper, connection factory, and guidance. [NativeClient](src/native-client.ts) owns request correlation, byte limits, cancellation acknowledgement, and failure settlement. The platform connector owns startup and connection cleanup. The shared executor retains one connection per Agent and joins in-flight image writes during disposal.
|
||||
|
||||
`agentPreset` defaults to `ptc`, restricting tools and guidance to Agents bound to that preset. The provider uses the existing `agent/created` and `agent-preset/selected` events and Agent-owned scopes, including Agents already present when it loads. Provider unload and Agent disposal release those registrations. Set `agentPreset: false` for global registration in standalone compositions without presets. The Host continues to own permissions, native connections and cancellation; preset definitions remain unchanged.
|
||||
|
||||
[The operation schema](src/protocol.ts) is the authority for model parameters and separately declared Host-only fields. Native implementations enforce target identity, window geometry, resource limits, and operating-system access at execution. No invariant companion is published because this library has no separately projected registry or durable state.
|
||||
|
||||
</details>
|
||||
|
||||
@@ -20,6 +20,8 @@ kind: "package-library"
|
||||
- [已知限制与后续工作](#known-limitations-and-deferred-work)
|
||||
- [开发备注](#dev-note)
|
||||
|
||||
应用授权使用现有 User Questions 选项，仍独立于会话的工具审批策略。提供方只接受一个准确匹配所给选项的标签，附带自由文本的回答不会授权。本机图标用于权限页展示，不进入保存的身份信息或模型结果。系统权限、持久授权与拒绝记录、图标分别查询。
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
@@ -33,7 +35,7 @@ kind: "package-library"
|
||||
|
||||
`tools.cua` 直接返回操作结果。截图结果保留图片 Base64，并增加供工具渲染使用的持久 `attachment` 引用；模型内容只包含坐标元数据和一张图片，不包含编码字节。截图前要求附件服务可用，并确认当前模型支持图片输入。图片接入或保存失败会拒绝调用。对象桥接保留截图字节和已校验的元数据，不重复上传或输出同一张图片。
|
||||
|
||||
应用授权发生在启动、观察、截图和输入之前。任务许可绑定已选择的应用实例；长期许可要求平台验证可执行文件身份。撤销会关闭受影响的 Agent 连接，因此旧对象必须重新选择目标。取消、超时或重新连接后均不重放输入。
|
||||
应用授权发生在启动、观察、截图和输入之前。会话许可按会话和应用身份保存在内存中，Agent 或 helper 重建后仍可复用，重启 DSH 或关闭 Computer Use 后失效。长期许可要求平台验证可执行文件身份。拒绝或撤销会关闭受影响的 Agent 连接，因此旧对象必须重新选择目标。取消、超时或重新连接后均不重放输入。
|
||||
|
||||
动作返回 `dispatched:true, verified:false`；请读取新状态以验证结果。坐标操作默认使用最近返回截图中的像素；成功观察窗口后，传入 `relative: true` 可使用 0～1000 的整数比例。Host 在返回截图前解析附件归一化和当前模型的图片缩放规则，再将输入换算为原生窗口 point。窗口尺寸变化会使截图映射失效，绝对坐标遇到 `SCREENSHOT_REQUIRED` 时需要重新截图。窗口缩放后以 `WINDOW_CHANGED` 拒绝旧坐标；单纯移动窗口不改变窗口内坐标。
|
||||
|
||||
@@ -47,6 +49,8 @@ kind: "package-library"
|
||||
|
||||
提供方将准备好的 helper、连接工厂和指引传给 [installCua](src/index.ts)。[NativeClient](src/native-client.ts) 负责请求关联、字节上限、取消确认和失败处理。平台连接器负责启动和连接清理。共享执行器为每个 Agent 保留一个连接，并在销毁时等待进行中的图像写入。
|
||||
|
||||
`agentPreset` 默认为 `ptc`，将工具和指引限制到绑定该 preset 的 Agent。提供方使用现有 `agent/created`、`agent-preset/selected` 事件及 Agent 拥有的作用域，也处理启用前已存在的 Agent。提供方卸载和 Agent 销毁都会释放这些注册。不使用 preset 的独立组合可设置 `agentPreset: false` 来全局注册。Host 仍拥有权限、本机连接和取消操作，preset 定义保持不变。
|
||||
|
||||
[操作 schema](src/protocol.ts) 定义模型参数，并单独声明 Host 专用字段。原生实现在执行时检查目标身份、窗口几何、资源上限和操作系统访问条件。本库没有独立投影的注册表或持久状态，因此不发布 invariant 模块。
|
||||
|
||||
</details>
|
||||
|
||||
@@ -28,6 +28,18 @@
|
||||
"types": "./lib/types/bridge.d.ts",
|
||||
"default": "./lib/bridge.js"
|
||||
},
|
||||
"./typert": {
|
||||
"types": "./lib/typert.host.d.ts",
|
||||
"default": "./lib/typert.host.js"
|
||||
},
|
||||
"./remote": {
|
||||
"types": "./lib/typert.remote-client.d.ts",
|
||||
"default": "./lib/typert.remote-client.js"
|
||||
},
|
||||
"./types": {
|
||||
"types": "./lib/types/types.d.ts",
|
||||
"default": "./lib/types.js"
|
||||
},
|
||||
"./native-client": {
|
||||
"types": "./lib/types/native-client.d.ts",
|
||||
"default": "./lib/native-client.js"
|
||||
@@ -39,11 +51,16 @@
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/types.js",
|
||||
"lib/api.js",
|
||||
"lib/bridge.js",
|
||||
"lib/native-client.js",
|
||||
"lib/protocol.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/typert.host.js",
|
||||
"lib/typert.host.d.ts",
|
||||
"lib/typert.remote-client.js",
|
||||
"lib/typert.remote-client.d.ts"
|
||||
],
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
@@ -55,14 +72,17 @@
|
||||
"@deepseek-ai/dsh-subprocess": "workspace:*",
|
||||
"@deepseek-ai/dsh-user-questions": "workspace:*",
|
||||
"@deepseek-ai/dsh-attachment": "workspace:*",
|
||||
"@deepseek-ai/dsh-llm": "workspace:*"
|
||||
"@deepseek-ai/dsh-llm": "workspace:*",
|
||||
"@deepseek-ai/dsh-scope": "workspace:*",
|
||||
"@deepseek-ai/dsh-agent-preset-registry": "workspace:*"
|
||||
},
|
||||
"dependencies": {
|
||||
"@deepseek-ai/dsh-util-values": "workspace:*",
|
||||
"@deepseek-ai/schemastery": "workspace:~",
|
||||
"zod": "^4.4.3",
|
||||
"@deepseek-ai/dsh-brand": "workspace:*",
|
||||
"@deepseek-ai/dsh-atomic-write": "workspace:*"
|
||||
"@deepseek-ai/dsh-atomic-write": "workspace:*",
|
||||
"@deepseek-ai/dsh-typert-protocol": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-working-directory": "workspace:*",
|
||||
@@ -89,11 +109,13 @@
|
||||
"@deepseek-ai/dsh-sandbox-policy": "workspace:*",
|
||||
"sharp": "^0.35.3",
|
||||
"@deepseek-ai/dsh-ptc-runtime": "workspace:*",
|
||||
"@deepseek-ai/dsh-user-questions": "workspace:*",
|
||||
"@deepseek-ai/dsh-settings": "workspace:*",
|
||||
"@deepseek-ai/dsh-user-questions": "workspace:*",
|
||||
"@deepseek-ai/dsh-app-boot": "workspace:*",
|
||||
"@deepseek-ai/dsh-config-editor": "workspace:*",
|
||||
"@deepseek-ai/dsh-hmr": "workspace:*",
|
||||
"@deepseek-ai/cordis-plugin-timer": "workspace:~"
|
||||
"@deepseek-ai/cordis-plugin-timer": "workspace:~",
|
||||
"@deepseek-ai/dsh-scope": "workspace:*",
|
||||
"@deepseek-ai/dsh-agent-preset-registry": "workspace:*"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
/** Client-safe Computer Use permission observations and saved application decisions. */
|
||||
import type { CuaApplicationGrant } from './application-types.ts'
|
||||
export type { CuaApplicationGrant, CuaApplicationId } from './application-types.ts'
|
||||
|
||||
/** macOS privileges used by the native provider. */
|
||||
export type CuaSystemPermission = 'accessibility' | 'screenRecording'
|
||||
/** Observed process; neither the helper nor its parent is assumed to be the TCC owner. */
|
||||
export interface CuaPermissionProcess {
|
||||
pid: number
|
||||
path?: string | undefined
|
||||
name?: string | undefined
|
||||
}
|
||||
/** Fresh permission observations and startup metadata from the shared native helper. */
|
||||
export interface CuaSystemPermissionState {
|
||||
accessibility: boolean
|
||||
screenRecording: boolean
|
||||
helper: CuaPermissionProcess
|
||||
parent: CuaPermissionProcess
|
||||
/** Application macOS reports as responsible for the helper, captured at startup; absent when it cannot be identified. */
|
||||
launchApplication?: { name: string; path: string; icon?: string | undefined } | undefined
|
||||
}
|
||||
/** Saved grants and refused applications, independent of native permission checks and icons. */
|
||||
export interface CuaApplicationState {
|
||||
/** Owning configuration entry for permission updates; null when this composition has no Loader entry. */
|
||||
settingsNamespace: string | null
|
||||
allowedApps: readonly CuaApplicationGrant[]
|
||||
/** Refusals from any session, excluding saved grants; other sessions may still hold session access. */
|
||||
blockedApps: readonly CuaApplicationGrant[]
|
||||
persistentAvailable: boolean
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
/** Browser controls for the provider's application decisions and native permission work. */
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import { Remote, TypertRemoteService } from '@deepseek-ai/dsh-typert-protocol'
|
||||
import { ApplicationAccess, type AccessOptions } from './application-access.ts'
|
||||
import type { JsonValue } from '@deepseek-ai/dsh-util-values'
|
||||
import { z } from 'zod'
|
||||
import { ApplicationIcon } from './protocol.ts'
|
||||
import { NativeClient, type NativeClientConfig, type NativeChannelFactory } from './native-client.ts'
|
||||
import type { CuaApplicationId, CuaApplicationState, CuaSystemPermission, CuaSystemPermissionState } from './access-types.ts'
|
||||
export { ApplicationAuthorizationResult } from './application-access.ts'
|
||||
export type * from './access-types.ts'
|
||||
|
||||
interface ControlOptions extends AccessOptions {
|
||||
native: NativeClientConfig
|
||||
createChannel: NativeChannelFactory
|
||||
settingsNamespace: string | null
|
||||
}
|
||||
type Control = (method: string, params: Record<string, JsonValue>) => Promise<JsonValue>
|
||||
const permissionProcess = z.object({ pid: z.number().int().positive(), path: z.string().optional(), name: z.string().optional() })
|
||||
const systemPermissions = z.object({
|
||||
accessibility: z.boolean(), screenRecording: z.boolean(), helper: permissionProcess, parent: permissionProcess,
|
||||
launchApplication: z.object({ name: z.string().min(1), path: z.string().min(1), icon: ApplicationIcon.optional() }).optional(),
|
||||
})
|
||||
|
||||
declare module '@deepseek-ai/cordis' {
|
||||
interface Context {
|
||||
/** Native permission controls and executor-owned application consent. */
|
||||
cuaAccess: CuaAccess
|
||||
}
|
||||
}
|
||||
|
||||
/** Browser controls for provider-owned native permission and application decisions. */
|
||||
export class CuaAccess extends TypertRemoteService {
|
||||
/**
|
||||
* @param ctx Provider context.
|
||||
* @param policy Application decision owner.
|
||||
* @param settingsNamespace Owning configuration entry.
|
||||
* @param control Serialized native permission dispatch.
|
||||
*/
|
||||
constructor(ctx: Context, private readonly policy: ApplicationAccess,
|
||||
private readonly settingsNamespace: string | null, private readonly control: Control) { super(ctx, 'cuaAccess') }
|
||||
|
||||
/**
|
||||
* Read application decisions without waiting for the native helper.
|
||||
* @returns Saved grants and refused applications with their persistence owner.
|
||||
*/
|
||||
@Remote
|
||||
applications(): CuaApplicationState {
|
||||
return { settingsNamespace: this.settingsNamespace, ...this.policy.status() }
|
||||
}
|
||||
|
||||
/**
|
||||
* Inspect system privileges independently of application decisions and icons.
|
||||
* @returns Fresh native preflight and launch metadata.
|
||||
*/
|
||||
@Remote
|
||||
async systemStatus(): Promise<CuaSystemPermissionState> {
|
||||
return systemPermissions.parse(await this.control('permissionStatus', {}))
|
||||
}
|
||||
|
||||
/**
|
||||
* Load presentation icons separately from application decisions.
|
||||
* @returns Optional native PNG icons keyed by executable path for saved and blocked applications.
|
||||
*/
|
||||
@Remote
|
||||
async applicationIcons(): Promise<Record<string, string>> {
|
||||
const state = this.policy.status()
|
||||
const paths = [...new Set([...state.allowedApps, ...state.blockedApps].map(app => app.executablePath))]
|
||||
return paths.length === 0 ? {} : z.record(z.string(), ApplicationIcon).parse(await this.control('applicationIcons', { paths }))
|
||||
}
|
||||
|
||||
/**
|
||||
* Request a system privilege explicitly selected by the user.
|
||||
* @param permission macOS privilege to request.
|
||||
* @returns Preflight state after requesting; a false value never implies consent.
|
||||
*/
|
||||
@Remote
|
||||
async requestPermission(permission: CuaSystemPermission): Promise<CuaSystemPermissionState> {
|
||||
return systemPermissions.parse(await this.control('requestPermission', { permission }))
|
||||
}
|
||||
|
||||
/**
|
||||
* Open the macOS privacy pane selected by the user.
|
||||
* @param permission Corresponding system privilege.
|
||||
* @returns Completion after macOS accepted the open request.
|
||||
*/
|
||||
@Remote
|
||||
async openSettings(permission: CuaSystemPermission): Promise<void> {
|
||||
z.object({ opened: z.literal(true) }).parse(await this.control('openPermissionSettings', { permission }))
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove saved and session access and join affected operations.
|
||||
* @param id Application identity displayed on the permission page.
|
||||
* @returns Completion after persistence and cancellation; dispatched input cannot be undone.
|
||||
*/
|
||||
@Remote
|
||||
revoke(id: CuaApplicationId): Promise<void> { return this.policy.revoke(id) }
|
||||
|
||||
/**
|
||||
* Clear this application's refusals across sessions so each session may ask for access again.
|
||||
* @param id Refused or revoked application displayed on the permission page.
|
||||
* @returns Completion after clearing refusals; no permission is granted and no action is replayed.
|
||||
*/
|
||||
/* oxlint-disable typescript/require-await -- Remote callers receive policy failures as rejected promises. */
|
||||
@Remote
|
||||
async retryApproval(id: CuaApplicationId): Promise<void> { this.policy.retryApproval(id) }
|
||||
/* oxlint-enable typescript/require-await */
|
||||
}
|
||||
|
||||
/**
|
||||
* Install human-facing controls while keeping executor capabilities private to the provider.
|
||||
* @param ctx Provider context that owns decisions and operation lifetimes.
|
||||
* @param options Native transport and profile persistence.
|
||||
* @returns Consent selection, guarded dispatch, and task cleanup capabilities.
|
||||
*/
|
||||
export function installCuaAccess(ctx: Context, options: ControlOptions): Pick<ApplicationAccess, 'select' | 'run' | 'forget' | 'dispose'> {
|
||||
const policy = new ApplicationAccess(ctx, options)
|
||||
const lifetime = new AbortController()
|
||||
let client: NativeClient | undefined
|
||||
let tail: Promise<unknown> = Promise.resolve()
|
||||
const control: Control = (method, params) => {
|
||||
lifetime.signal.throwIfAborted()
|
||||
const perform = async () => {
|
||||
lifetime.signal.throwIfAborted()
|
||||
if (client?.disposed) await client.dispose()
|
||||
lifetime.signal.throwIfAborted()
|
||||
if (!client || client.disposed) client = new NativeClient(ctx, options.native, options.createChannel)
|
||||
return client.request(method, params, lifetime.signal)
|
||||
}
|
||||
const operation = tail.then(perform, perform)
|
||||
tail = operation
|
||||
return operation
|
||||
}
|
||||
const dispose = ctx.effect(() => async () => {
|
||||
lifetime.abort()
|
||||
await Promise.allSettled([policy.dispose(), tail])
|
||||
await client?.dispose()
|
||||
}, 'cua-access.dispose')
|
||||
new CuaAccess(ctx, policy, options.settingsNamespace, control)
|
||||
return { select: policy.select.bind(policy), run: policy.run.bind(policy), forget: policy.forget.bind(policy), dispose }
|
||||
}
|
||||
Loaded 100 of 138 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user