test(bench): describe cases, measure storage share, and model core counts

Every recorded case states what it measures and which user scenario it
affects, shown on hover in the report. Endpoints whose workers report
process CPU time default to the measured non-CPU share and busy
parallelism; the Session corpus worker now reports CPU time per
endpoint. Machine profiles carry logical CPU counts, and the page
shows each default share with its source and can reset overrides.
This commit is contained in:
Turtle committed 2026-10-08 10:52:34 +08:00
1 parent 9baa8bc8fe
commit 1c29d96067
15 files changed
+331 -129

No files matched your search

@@ -9,20 +9,20 @@
en: d8e9dee3e22d691a
zh: 53d9fae666f032b7
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/decision:
en: aca7e6a17dc9c72d
zh: 7dd535ac26cfb843
en: 5587ef207abde949
zh: a8c92e13b980c7c1
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/decision/machine-scaling:
en: d6ecda154d834793
zh: 68b16ea6580d5526
en: f532650e0720e66a
zh: dfd6ffa6366c1f09
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/decision/memory-pressure:
en: bec0d7592ed58486
zh: 2be2350c142f133d
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/decision/exclusions:
en: 57d602db18c6e825
zh: ac731f65c0c253fa
en: dc685a11156d6cd3
zh: 519bbdbd0b4c69d7
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/alternatives-considered:
en: 0f6786650d04e6bf
zh: 50a0534c9d39ce7b
en: 9011ce1b4702608d
zh: 214f38794afe13fd
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/consequences:
en: b8724a183365bd49
zh: 2819469dc566f343
en: 077dafabe9929058
zh: 114ab9cafcf33779
@@ -10,11 +10,13 @@ Benchmark budgets state whether one endpoint regressed on the hosted runner. The
## Decision
Every `test:bench` run writes `benchmarks/.dsh-report/report.html`, a self-contained page without network resources. Bench files pass each budgeted wall-clock median to `recordTimings` and whole-process peak RSS to `recordPeakMemory` from [scaling-report.ts](../../../../benchmarks/support/scaling-report.ts) before asserting budgets. A Vitest global setup clears the results before the run and renders the page after it, including after failed cases. Budgets and assertions do not change.
Every `test:bench` run writes `benchmarks/.dsh-report/report.html`, a self-contained page without network resources. Bench files pass each budgeted wall-clock median to `recordTimings` and whole-process peak RSS to `recordPeakMemory` from [scaling-report.ts](../../../../benchmarks/support/scaling-report.ts) before asserting budgets. Each call carries a case description stating what the case measures and which user scenario it affects; the page shows it when the reader hovers the case. A Vitest global setup clears the results before the run and renders the page after it, including after failed cases. Budgets and assertions do not change.
### Machine scaling
[machine-profiles.ts](../../../../benchmarks/support/machine-profiles.ts) lists laptops from 2015, 2019, 2023, and 2026 with SSD storage, mainstream cloud instances with their default system disks, the hosted runners, and the M4 Pro calibration reference. Each profile has a PassMark single-thread rating, retrieved on 2026-10-07, and a typical 4 KiB queue-depth-1 latency of its storage. A scaled endpoint is `ms × ((1 − io) × cpuScore(measured) / cpuScore(target) + io × ioLatency(target) / ioLatency(measured))`. `io` is a commented storage-wait estimate declared per endpoint beside the benchmark; the page lets readers change it per endpoint. The measuring machine is matched from the CPU model and can be changed in the page.
[machine-profiles.ts](../../../../benchmarks/support/machine-profiles.ts) lists laptops from 2015, 2019, 2023, and 2026 with SSD storage, mainstream cloud instances with their default system disks, the hosted runners, and the M4 Pro calibration reference. Each profile has a PassMark single-thread rating, retrieved on 2026-10-07, its logical CPU count, and a typical 4 KiB queue-depth-1 latency of its storage. A scaled endpoint is `ms × ((1 − io) × cpuScore(measured) / cpuScore(target) × coreFactor + io × ioLatency(target) / ioLatency(measured))`.
Where the worker reports process CPU time over the endpoint interval, `io` is the measured share of wall time not spent on CPU, `max(0, 1 − cpuMs / ms)`, and the busy parallelism is `max(1, cpuMs / ms)`. The Session corpus, projection list, Session first history and Agent resume, and Agent continuation cases report it. Other endpoints declare a commented `io` beside the benchmark: in-memory Client and terminal work declares 0, and the browser, Session-open phase, and SDK profile endpoints declare estimates. `coreFactor` is `max(1, parallelism / cores(target)) / max(1, parallelism / cores(measured))`, so an endpoint that kept more CPUs busy than a target has slows down there. The page shows each default and its source, and readers can override and reset `io` per endpoint. The measuring machine is matched from the CPU model, uses its actual available CPU count, and can be changed in the page.
On Apple M5 Pro the first content-index search measured 58.7 s; the model predicts 168 s on the EPYC 7763 runner, against a recorded hosted median of 144 s.
@@ -24,18 +26,18 @@ The Session corpus list, search, and fork cases report whole-process peak RSS fo
### Exclusions
Long-Session `first`, `input`, and `streamWall` include paced replay waits and are not recorded. The browser page heap is read after a forced garbage collection, so it is not a peak and is not recorded. Core count, memory size, and thermal limits are not modelled.
Long-Session `first`, `input`, and `streamWall` include paced replay waits and are not recorded. The browser page heap is read after a forced garbage collection, so it is not a peak and is not recorded. Memory size and thermal limits are not modelled. Measured non-CPU time also contains timers, event-loop yields, and child scheduling, and the measuring run reads a just-written corpus that the page cache mostly holds, so cold-cache storage waits on a user machine can be larger. Garbage-collection helper threads add CPU time, so measured parallelism is typically 1–2 and storage wait overlapping that work is not visible. Declared endpoints, including the browser case where Chromium and the Host share the machine, assume one busy CPU.
## Alternatives considered
**Run cases under an operating-system memory limit.** A cgroup limit needs root on Linux CI and is unavailable on macOS. The measured peaks stay below 1 GB, so a 4 GB limit would not change the timed path while adding privileged setup to a required gate.
**Measure storage wait from CPU time.** Wall time minus CPU time also contains event-loop yields, timers, and child scheduling, so it does not isolate storage. Most workers do not report CPU time either.
**Only declared storage shares.** Readers cannot judge a share they have to guess. Process CPU time is cheap to measure in the workers and gives a reproducible default, although the non-CPU time it leaves includes waits other than storage.
**Benchmark-wide storage share.** Session open restores and projects after closing its file, so one share per benchmark overstates high-latency disks for memory-only endpoints.
**Multi-core or Geekbench scores.** Timed endpoints run on one JavaScript thread. Geekbench pages were not retrievable for citation; PassMark single-thread ratings were.
**Multi-core or Geekbench scores.** A multi-core score would overstate machines with many cores for endpoints dominated by one JavaScript thread; the measured busy parallelism applies core limits only where they matter. Geekbench pages were not retrievable for citation; PassMark single-thread ratings were.
## Consequences
Readers can see each endpoint on reviewed machines and adjust the storage estimate, without changing any gate. The page depends on reviewed constants: cross-ISA ratings, shared vCPUs, and disk latency classes are coarse, so results are estimates rather than measurements. The memory view does not measure swap or reclaim costs. Publishing the page from CI requires an upload step in the `node 24 / benchmarks` job.
Readers can see each endpoint on reviewed machines with measured defaults, and adjust the storage share, without changing any gate. The page depends on reviewed constants: cross-ISA ratings, shared vCPUs, and disk latency classes are coarse, so results are estimates rather than measurements. The memory view does not measure swap or reclaim costs. Publishing the page from CI requires an upload step in the `node 24 / benchmarks` job.
@@ -10,11 +10,13 @@ Status: implemented
## 决定
每次 `test:bench` 运行都会写出 `benchmarks/.dsh-report/report.html`,这是一个不依赖网络资源的独立页面。基准文件在断言预算之前,把每个有预算的墙钟中位数传给 [scaling-report.ts](../../../../benchmarks/support/scaling-report.ts) 的 `recordTimings`,把整进程峰值 RSS 传给 `recordPeakMemory`。Vitest global setup 在运行前清空结果,在运行后渲染页面,用例失败时也会渲染。预算和断言不变。
每次 `test:bench` 运行都会写出 `benchmarks/.dsh-report/report.html`,这是一个不依赖网络资源的独立页面。基准文件在断言预算之前,把每个有预算的墙钟中位数传给 [scaling-report.ts](../../../../benchmarks/support/scaling-report.ts) 的 `recordTimings`,把整进程峰值 RSS 传给 `recordPeakMemory`。每次调用都带有用例描述,说明该用例测量什么、影响哪个用户场景;读者悬停该用例时页面会显示它。Vitest global setup 在运行前清空结果,在运行后渲染页面,用例失败时也会渲染。预算和断言不变。
### 机器缩放
[machine-profiles.ts](../../../../benchmarks/support/machine-profiles.ts) 列出 2015、2019、2023、2026 年使用 SSD 的笔记本,带默认系统盘的主流云服务器,托管 runner,以及 M4 Pro 校准参考机。每个配置包含 2026-10-07 获取的 PassMark 单线程评分,以及其存储的典型 4 KiB 队列深度 1 延迟。缩放后的端点为 `ms × ((1 − io) × cpuScore(measured) / cpuScore(target) + io × ioLatency(target) / ioLatency(measured))`。`io` 是在基准旁按端点声明并注释的存储等待估计值;页面允许读者按端点修改。测量机器按 CPU 型号匹配,也可以在页面中修改。
[machine-profiles.ts](../../../../benchmarks/support/machine-profiles.ts) 列出 2015、2019、2023、2026 年使用 SSD 的笔记本,带默认系统盘的主流云服务器,托管 runner,以及 M4 Pro 校准参考机。每个配置包含 2026-10-07 获取的 PassMark 单线程评分、逻辑 CPU 数,以及其存储的典型 4 KiB 队列深度 1 延迟。缩放后的端点为 `ms × ((1 − io) × cpuScore(measured) / cpuScore(target) × coreFactor + io × ioLatency(target) / ioLatency(measured))`。
当 worker 报告端点区间内的进程 CPU 时间时,`io` 是实测的非 CPU 墙钟时间占比 `max(0, 1 − cpuMs / ms)`,繁忙并行度为 `max(1, cpuMs / ms)`。Session 语料、投影列表、Session 首次历史与 Agent 恢复,以及 Agent 续聊用例会报告 CPU 时间。其他端点在基准旁声明并注释 `io`:内存中的 Client 与终端工作声明为 0,浏览器、Session-open 分阶段和 SDK profile 端点声明估计值。`coreFactor` 为 `max(1, parallelism / cores(target)) / max(1, parallelism / cores(measured))`,因此繁忙 CPU 数超过目标机器 CPU 数的端点会在该机器上变慢。页面显示每个默认值及其来源,读者可以按端点覆盖和重置 `io`。测量机器按 CPU 型号匹配,使用实际可用 CPU 数,也可以在页面中修改。
在 Apple M5 Pro 上,首次内容索引搜索实测 58.7 s;模型预测 EPYC 7763 runner 上为 168 s,托管 CI 记录的中位数为 144 s。
@@ -24,18 +26,18 @@ Session 语料的列表、搜索和 fork 用例报告最大合成工作集的整
### 排除项
长 Session 的 `first`、`input` 和 `streamWall` 包含按节奏回放的等待,不记录。浏览器页面堆在强制垃圾回收后读取,不是峰值,不记录。不建模核心数、内存容量和散热限制。
长 Session 的 `first`、`input` 和 `streamWall` 包含按节奏回放的等待,不记录。浏览器页面堆在强制垃圾回收后读取,不是峰值,不记录。不建模内存容量和散热限制。实测的非 CPU 时间还包含定时器、事件循环让出和子进程调度;测量运行读取的是刚写入、主要位于页面缓存中的语料,因此用户机器上冷缓存的存储等待可能更长。垃圾回收辅助线程会增加 CPU 时间,因此实测并行度通常为 1–2,与这些工作重叠的存储等待不可见。声明的端点,包括 Chromium 与 Host 共用机器的浏览器用例,都假定只有一个繁忙 CPU。
## 备选方案
**在操作系统内存限制下运行用例。** Linux CI 上的 cgroup 限制需要 root,macOS 上不可用。实测峰值低于 1 GB,4 GB 限制不会改变计时路径,却会给必需门禁增加特权配置。
**根据 CPU 时间测量存储等待。** 墙钟时间减去 CPU 时间还包含事件循环让出、定时器和子进程调度,无法单独分离存储。大多数 worker 也不报告 CPU 时间。
**只使用声明的存储占比。** 读者无法判断需要猜测的占比。在 worker 中测量进程 CPU 时间成本很低,并能给出可复现的默认值,尽管剩余的非 CPU 时间包含存储以外的等待。
**按基准统一存储占比。** Session open 在关闭文件后才执行恢复和投影,按基准使用一个占比会高估高延迟磁盘对纯内存端点的影响。
**多核或 Geekbench 分数。** 被计时端点运行在单个 JavaScript 线程上。Geekbench 页面无法获取以供引用,PassMark 单线程评分可以获取。
**多核或 Geekbench 分数。** 对于以单个 JavaScript 线程为主的端点,多核分数会高估多核机器;实测繁忙并行度只在相关时施加核心数限制。Geekbench 页面无法获取以供引用,PassMark 单线程评分可以获取。
## 影响
读者可以在经审阅的机器上查看每个端点并调整存储估计值,而不改变任何门禁。页面依赖经审阅的常量:跨指令集评分、共享 vCPU 和磁盘延迟等级都较粗略,因此结果是估计值而不是测量值。内存视图不测量交换和回收成本。要从 CI 发布该页面,需要在 `node 24 / benchmarks` job 中添加上传步骤。
读者可以在经审阅的机器上查看每个端点及其实测默认值,并调整存储占比,而不改变任何门禁。页面依赖经审阅的常量:跨指令集评分、共享 vCPU 和磁盘延迟等级都较粗略,因此结果是估计值而不是测量值。内存视图不测量交换和回收成本。要从 CI 发布该页面,需要在 `node 24 / benchmarks` job 中添加上传步骤。
+1 -1
View File
@@ -8,7 +8,7 @@ This tree owns required, repository-level performance gates whose measured user
- Browser workflow cases drive built Client bundles through the shared shipped-composition Web scaffold. Report its source-resolved test Host separately from published-Host evidence; two animation frames prove a rendering opportunity, not hardware presentation. Use fresh browsers and private scaffold worlds per sample.
- Synthesize fixed inputs from reviewed constants. Never use recorded Sessions, user material, ambient repositories, or network services.
- Run process-level wall-clock and retained-memory samples in fresh children with private `mkdtemp` roots. Pure synchronous folds create a fresh object graph per sample and must not mutate process-global state. Bound every child, await exit, and remove owned roots after failure as well as success.
- Pass every budgeted wall-clock endpoint to `recordTimings` from `support/scaling-report.ts` before asserting it, with a commented storage-wait share; omit endpoints that include paced or fixed waits. Pass whole-process peak RSS to `recordPeakMemory`. Profiles live in `support/machine-profiles.ts`; each run writes `.dsh-report/report.html`.
- Pass every budgeted wall-clock endpoint to `recordTimings` from `support/scaling-report.ts` before asserting it, with a `BenchmarkCase` stating what it measures and which user scenario it affects. Give each endpoint its process CPU time over the same interval when the worker can measure it, otherwise a commented storage-wait share; omit endpoints that include paced or fixed waits. Pass whole-process peak RSS to `recordPeakMemory`. Profiles live in `support/machine-profiles.ts`; each run writes `.dsh-report/report.html`.
- Record reference-machine expectations separately from the shared CI time scale and variance headroom. Do not apply the time scale to memory or dimensionless ratios.
- Report enough raw and aggregate measurements to explain each verdict, including whether a budget uses a median, minimum, absolute value, or ratio. Enforce reviewed source constants; environment variables must not override performance budgets.
- Keep scenario-specific support beside its benchmark. Move a helper into `benchmarks/support/` only after at least two benchmark directories require the same behavior.
@@ -3,7 +3,7 @@ import { join } from 'node:path'
import { expect, it } from 'vitest'
import { runBuiltBenchmarkWorker } from '../support/built-worker.ts'
import { ciTimeBudget, PERFORMANCE_BUDGET_HEADROOM } from '../support/calibration.ts'
import { recordTimings } from '../support/scaling-report.ts'
import { recordTimings, type BenchmarkCase } from '../support/scaling-report.ts'
import type { ReconnectReport } from './reconnect.worker.client.ts'
const REFERENCE_REPLACE_MS = 50
@@ -12,6 +12,11 @@ const REFERENCE_RETAINED_MB = 24
const SAMPLES = 3
/** Replacement folds in-memory Client state; no storage access is timed. */
const IO_SHARE = 0
const CASE: BenchmarkCase = {
id: 'active-stream-reconnect',
measures: 'Client rebuild of an unfinished Assistant reply from a 100,000-delta reasoning prefix delivered on reconnect.',
affects: 'Reloading or reconnecting the Web GUI while a long reply is still streaming.',
}
function expectReplacementWithinBudget(value: number, budget: number): void {
expect(value).toBeLessThanOrEqual(budget)
@@ -52,7 +57,7 @@ it('reconstructs a 100000-delta live prefix within baseline time and retained-me
const budgetMs = REPLACE_BUDGET_MS
const budgetMb = REFERENCE_RETAINED_MB * PERFORMANCE_BUDGET_HEADROOM
console.log(JSON.stringify({ benchmark: 'active-stream-reconnect', samples, median: { replaceMs, retainedMb }, referenceMs: REFERENCE_REPLACE_MS, referenceMb: REFERENCE_RETAINED_MB, budgetMs, budgetMb }))
recordTimings('active-stream-reconnect', IO_SHARE, { replaceMs }, { replaceMs: budgetMs })
recordTimings(CASE, { replaceMs: { ms: replaceMs, ioShare: IO_SHARE } }, { replaceMs: budgetMs })
expectReplacementWithinBudget(replaceMs, budgetMs)
expect.soft(retainedMb).toBeLessThanOrEqual(budgetMb)
})
@@ -6,7 +6,7 @@ import { join } from 'node:path'
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import { runBuiltBenchmarkWorker } from '../support/built-worker.ts'
import { ciTimeBudget, PERFORMANCE_BUDGET_HEADROOM } from '../support/calibration.ts'
import { recordTimings } from '../support/scaling-report.ts'
import { recordTimings, type BenchmarkCase } from '../support/scaling-report.ts'
import type { ContinuationReport } from './agent-continuation.worker.ts'
import type { CatalogReport } from './child-catalog.worker.ts'
import type { ProfileReport } from './profile-continuation.worker.ts'
@@ -29,8 +29,30 @@ const CATALOG_BUDGET_MS = Math.ceil(EXPECTED_CATALOG_CI_MS * PERFORMANCE_BUDGET_
const REQUEST_HISTORY_BUDGET_MS = 297
const EXPECTED_RETAINED_HEAP_MB = 23
const WORKERS = join(import.meta.dirname, '..', '.dsh-build', 'agent-continuation')
/** Estimated storage wait: continuation reads a copied Session tree and appends synchronized log records. */
const IO_SHARE = 0.15
/** Estimated storage wait of the SDK profile case, whose worker does not report CPU time: it reads files and appends synchronized log records. */
const PROFILE_IO_SHARE = 0.15
const CASES: Readonly<Record<Scenario, BenchmarkCase>> = {
'request-history': {
id: 'agent-continuation/request-history',
measures: 'Building 40 model requests over an 800-turn tool-heavy history, without provider or network work.',
affects: 'Every new turn in a long Session; the model request leaves later.',
},
'tool-continuation': {
id: 'agent-continuation/tool-continuation',
measures: 'Cold resume of the 800-turn Session, then 20 turns with 8 synthetic tool calls each through the real tool pipeline.',
affects: 'Continuing a long tool-heavy Session after restarting DSH.',
},
'catalog': {
id: 'agent-continuation/catalog',
measures: 'First and repeated discovery of 16 inactive fork children with 80-turn histories.',
affects: 'Listing forks and subagents of a Session that has many children.',
},
'profile-continuation': {
id: 'agent-continuation/profile-continuation',
measures: 'Boot plus 100 turns with 800 real file reads through the shipped sdk-minimal profile.',
affects: 'Headless SDK automation running a long agent task.',
},
}
type Scenario = 'request-history' | 'catalog' | 'tool-continuation' | keyof typeof EXPECTED_MS
type Report = ContinuationReport | CatalogReport | ProfileReport
@@ -178,7 +200,11 @@ describe('continuing tool-heavy Sessions with large histories', () => {
samples, totalMs: { min: Math.min(...totalMs), median: median(totalMs), max: Math.max(...totalMs) },
budgetMs, ...(scenario === 'tool-continuation' ? { retainedHeapBudgetMb } : {}),
}))
recordTimings('agent-continuation/' + scenario, IO_SHARE, { totalMs: median(totalMs) }, { totalMs: budgetMs })
recordTimings(CASES[scenario], {
totalMs: scenario === 'profile-continuation'
? { ms: median(totalMs), ioShare: PROFILE_IO_SHARE }
: { ms: median(totalMs), cpuMs: median((samples as (ContinuationReport | CatalogReport)[]).map(sample => sample.cpuUserMs + sample.cpuSystemMs)) },
}, { totalMs: budgetMs })
if (scenario === 'request-history') assertRequestHistoryBudget(median(totalMs))
else expectTotalWithinBudget(median(totalMs), budgetMs)
if (scenario === 'tool-continuation') {
@@ -10,7 +10,7 @@ import {
ciTimeBudget,
PERFORMANCE_BUDGET_HEADROOM,
} from '../support/calibration.ts'
import { recordTimings } from '../support/scaling-report.ts'
import { recordTimings, type BenchmarkCase } from '../support/scaling-report.ts'
import type { ConversationFoldWorkerReport, PreparingToolWorkerReport } from './conversation-fold.worker.client.ts'
/** Replies in the folded window; each carries one reasoning block and one text block. */
@@ -23,6 +23,12 @@ const SMALL_DELTAS = 100
const ATTEMPTS = 3
/** Folding and argument preparation run on in-memory Client state without storage access. */
const IO_SHARE = 0
const LARGE_WINDOW_CASE: BenchmarkCase = {
id: 'conversation-fold/large-window',
measures: 'Client fold of 200 replies whose 500,000 streamed deltas are compacted into 1,600 records.',
affects: 'Opening or scrolling a long conversation in the Web GUI.',
}
const PREPARING_AFFECTS = { write: 'Watching the model stream a large file write.', bash: 'Watching the model stream a long shell command.' } as const
/** A stuck fold worker is reaped before the outer benchmark deadline. */
const WORKER_TIMEOUT_MS = 60_000
@@ -80,7 +86,7 @@ describe('cold Chat fold of a large v2 history window', () => {
budgetMs: LARGE_FOLD_BUDGET_MS,
maxScaling: MAX_DELTA_SCALING,
}))
recordTimings('conversation-fold/large-window', IO_SHARE, { largeFoldMs: report.largeFoldMs }, { largeFoldMs: LARGE_FOLD_BUDGET_MS })
recordTimings(LARGE_WINDOW_CASE, { largeFoldMs: { ms: report.largeFoldMs, ioShare: IO_SHARE } }, { largeFoldMs: LARGE_FOLD_BUDGET_MS })
expect(report.chatNodes).toBeGreaterThan(0)
expect(report.largeFoldMs).toBeLessThanOrEqual(LARGE_FOLD_BUDGET_MS)
expect(report.scaling).toBeLessThanOrEqual(MAX_DELTA_SCALING)
@@ -112,7 +118,11 @@ describe('preparing tool arguments', () => {
const budgetMs = ciTimeBudget(workload.expectedMs)
const budgetMb = workload.expectedMb * PERFORMANCE_BUDGET_HEADROOM
console.log(JSON.stringify({ benchmark: `conversation-fold/preparing-${workload.tool}`, samples, medianMs, retainedMb, budgetMs, budgetMb }))
recordTimings(`conversation-fold/preparing-${workload.tool}`, IO_SHARE, { medianMs }, { medianMs: budgetMs })
recordTimings({
id: `conversation-fold/preparing-${workload.tool}`,
measures: `Client publishing of ${String(workload.characters / 1024)} KiB of streamed ${workload.tool} tool arguments.`,
affects: PREPARING_AFFECTS[workload.tool],
}, { medianMs: { ms: medianMs, ioShare: IO_SHARE } }, { medianMs: budgetMs })
expect(medianMs).toBeLessThanOrEqual(budgetMs)
expect(retainedMb).toBeLessThanOrEqual(budgetMb)
})
@@ -8,7 +8,7 @@ import { expect, it, vi } from 'vitest'
import { launchWebScaffold, seedSession, watchConsole, webSnapshotMode } from '../../apps/web/tests/scaffold.ts'
import { newEnglishPage } from '../../apps/web/tests/support.ts'
import { ciTimeBudget, PERFORMANCE_BUDGET_HEADROOM } from '../support/calibration.ts'
import { recordTimings } from '../support/scaling-report.ts'
import { recordTimings, type BenchmarkCase } from '../support/scaling-report.ts'
import { HISTORY_TURNS, SESSION_ID, FIRST, DONE, DELTAS, PACE_MS, syntheticHistory, syntheticReply } from './synthetic-history.ts'
const SAMPLES = 3
@@ -25,6 +25,11 @@ const REPLAY_DURATION_MS = (DELTAS + 4) * PACE_MS
const SCALED_ENDPOINTS = ['open', 'page', 'trajectory', 'streamTask'] as const
/** Estimated storage wait: the Host reads the seeded Session log, which the page cache usually holds. */
const IO_SHARE = 0.05
const CASE: BenchmarkCase = {
id: 'long-session-browser',
measures: 'Chromium on the built Web GUI: opening a 240-turn Session, the slowest older page, first Trajectory view, and main-thread task time while a reply streams.',
affects: 'Everyday Web GUI use with a long Session.',
}
async function painted(page: Page): Promise<void> {
// Two rAF callbacks include a rendering opportunity, not a GPU presentation timestamp.
@@ -444,6 +449,6 @@ it('opens, pages, navigates and streams into a 240-turn browser history', async
open: OPEN_BUDGET_MS, page: PAGE_BUDGET_MS, trajectory: TRAJECTORY_BUDGET_MS,
}
console.log(JSON.stringify({ benchmark: 'long-session-browser/median', turns: HISTORY_TURNS, deltas: DELTAS, paceMs: PACE_MS, samples, aggregate, referenceMs: REFERENCE, expectedOpenCiMs: EXPECTED_OPEN_CI_MS, expectedPageCiMs: EXPECTED_PAGE_CI_MS, expectedTrajectoryCiMs: EXPECTED_TRAJECTORY_CI_MS, budgets }))
recordTimings('long-session-browser', IO_SHARE, Object.fromEntries(SCALED_ENDPOINTS.map(key => [key, aggregate[key]!])), budgets)
recordTimings(CASE, Object.fromEntries(SCALED_ENDPOINTS.map(key => [key, { ms: aggregate[key]!, ioShare: IO_SHARE }])), budgets)
for (const [key, value] of Object.entries(aggregate)) expectEndpointWithinBudget(value, budgets[key]!)
})
@@ -18,8 +18,10 @@ const LIST_REFERENCE_MS = { modest: 50, tail: 500, cheap: 150 } as const
const CALLBACK_REFERENCE_MS = 30
/** Reference retained-heap allowance; memory receives variance headroom but no CPU scaling. */
const RETAINED_HEAP_REFERENCE_BYTES = 240 * 1024 * 1024
/** Estimated storage wait in listing: projection-cache reads mostly hit the page cache of the just-written corpus. */
const IO_SHARE = 0.1
/** Queue delay is time other callbacks wait behind list work on the Host thread, which is CPU work. */
const CALLBACK_IO_SHARE = 0
/** Sessions per workload. */
const SESSIONS = { modest: 50, tail: 300, cheap: 3_000 } as const
function median(values: readonly number[]): number {
return [...values].sort((left, right) => left - right)[Math.floor(values.length / 2)] as number
@@ -71,12 +73,18 @@ it.each(['modest', 'tail', 'cheap'] as const)('serves the %s projection list wit
console.log(JSON.stringify({ benchmark: `session-corpus/projection-list-${workload}`,
reports, medians, budgets, environment: { node: process.version, platform: process.platform, arch: process.arch, cpu: cpus()[0]?.model } }))
recordTimings(`session-corpus/projection-list-${workload}`, IO_SHARE, {
firstListAndJsonMs: medians.firstListAndJsonMs, repeatListAndJsonMs: medians.repeatListAndJsonMs, worstCallbackDelayMs: medians.worstCallbackDelayMs,
recordTimings({
id: `session-corpus/projection-list-${workload}`,
measures: `Host Session-list projection and JSON for ${SESSIONS[workload].toLocaleString('en-US')} Sessions, and the longest Host queue delay meanwhile.`,
affects: 'Showing the Session sidebar; a long queue delay stalls other Web GUI requests.',
}, {
firstListAndJsonMs: { ms: medians.firstListAndJsonMs, cpuMs: medians.firstListAndJsonCpuMs },
repeatListAndJsonMs: { ms: medians.repeatListAndJsonMs, cpuMs: medians.repeatListAndJsonCpuMs },
worstCallbackDelayMs: { ms: medians.worstCallbackDelayMs, ioShare: CALLBACK_IO_SHARE },
}, { firstListAndJsonMs: budgets.listAndJsonMs, repeatListAndJsonMs: budgets.listAndJsonMs, worstCallbackDelayMs: budgets.callbackDelayMs })
for (const report of reports) {
expect(report.workload).toBe(workload)
expect(report.fixture.sessions).toBe({ modest: 50, tail: 300, cheap: 3_000 }[workload])
expect(report.fixture.sessions).toBe(SESSIONS[workload])
expect(report.fixture.baseRows).toBe({ modest: 100, tail: 1_000, cheap: 0 }[workload])
expect(report.samples).toHaveLength(4)
for (const sample of [...report.samples, report.responsiveness]) {
@@ -7,13 +7,14 @@ import { performance } from 'node:perf_hooks'
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import { runBuiltBenchmarkWorker } from '../support/built-worker.ts'
import { PERFORMANCE_BUDGET_HEADROOM } from '../support/calibration.ts'
import { recordPeakMemory, recordTimings } from '../support/scaling-report.ts'
import { recordPeakMemory, recordTimings, type BenchmarkCase, type Timing } from '../support/scaling-report.ts'
import { anchorShape, ANCHOR_COUNT, sessionShape } from './corpus-shape.ts'
import { subagentRank } from './synthetic-corpus.ts'
import type {
AnchorsReport,
ForkReport,
ListReport,
PhaseCpu,
SearchReport,
SeedReport,
SessionCorpusReport,
@@ -70,14 +71,40 @@ const EXPECTED_CI_PEAK_RSS_MB = {
fork: 750,
} as const satisfies Record<keyof typeof RECORDED_CI_PEAK_RSS_MB, number>
/**
* Estimated storage-wait shares: listing and search read a just-written corpus mostly from the page cache;
* forking appends and synchronizes a new Session log.
*/
const IO_SHARE = { list: 0.1, search: 0.1, fork: 0.3 } as const
const CASES: Readonly<Record<CorpusName, BenchmarkCase>> = {
list: {
id: `session-corpus/list-${String(CORPUS.list)}`,
measures: `Cold Web Host boot, then first and repeated session.list over ${CORPUS.list.toLocaleString('en-US')} Sessions of realistic length.`,
affects: 'Starting DSH and showing the Session list for a heavy user.',
},
search: {
id: 'session-corpus/search',
measures: `First session.search, which builds the content index over ${CORPUS.search.toLocaleString('en-US')} Sessions, then a second query.`,
affects: 'Searching Session content in deployments that enable search; shipped profiles disable it.',
},
fork: {
id: 'session-corpus/fork',
measures: `session.fork for ten length strata, the p99 Session, and the longest of ${CORPUS.fork.toLocaleString('en-US')} Sessions.`,
affects: 'Forking a conversation to try another direction.',
},
}
type CorpusName = keyof typeof CORPUS
/**
* Pair the median wall time of each list phase with the median process CPU time of the same phase.
* @param samples - Wall-clock samples per phase.
* @param cpu - CPU time per sample.
* @returns Endpoints for the scaled report.
*/
function phaseTimings(
samples: Readonly<Record<keyof PhaseCpu, readonly number[]>>,
cpu: readonly PhaseCpu[],
): Record<keyof PhaseCpu, Timing> {
const phase = (key: keyof PhaseCpu): Timing => ({ ms: median(samples[key]), cpuMs: median(cpu.map(entry => entry[key])) })
return { bootMs: phase('bootMs'), firstMs: phase('firstMs'), repeatMs: phase('repeatMs') }
}
function topLevelRank(rank: number): number {
return subagentRank(rank) ? rank - 1 : rank
}
@@ -228,10 +255,9 @@ describe('Session corpus operations', () => {
heapUsedMb: reports.map(report => report.memory.heapUsedMb),
}
console.log(JSON.stringify({ benchmark: `session-corpus/list-${String(CORPUS.list)}`, samples, budgets, environment: environment() }))
recordTimings(`session-corpus/list-${String(CORPUS.list)}`, IO_SHARE.list,
{ bootMs: median(samples.bootMs), firstMs: median(samples.firstMs), repeatMs: median(samples.repeatMs) },
recordTimings(CASES.list, phaseTimings(samples, reports.map(report => report.cpu)),
{ bootMs: budgets.bootMs, firstMs: budgets.firstMs, repeatMs: budgets.repeatMs })
recordPeakMemory(`session-corpus/list-${String(CORPUS.list)}`, { peakRssMb: Math.max(...samples.peakRssMb) })
recordPeakMemory(CASES.list, { peakRssMb: Math.max(...samples.peakRssMb) })
for (const report of reports) expect(report.itemsWithProjections).toBe(CORPUS.list)
expectWithinBudget(median(samples.bootMs), budgets.bootMs)
expectWithinBudget(median(samples.firstMs), budgets.firstMs)
@@ -248,9 +274,11 @@ describe('Session corpus operations', () => {
// One sample: the cold index build dominates this file's time.
const report = await run<SearchReport>([root('search'), 'search'])
console.log(JSON.stringify({ benchmark: 'session-corpus/search', report, budgets, environment: environment() }))
recordTimings('session-corpus/search', IO_SHARE.search, { firstMs: report.firstMs, repeatMs: report.repeatMs },
{ firstMs: budgets.firstMs, repeatMs: budgets.repeatMs })
recordPeakMemory('session-corpus/search', { peakRssMb: report.memory.peakRssMb })
recordTimings(CASES.search, {
firstMs: { ms: report.firstMs, cpuMs: report.cpu.firstMs },
repeatMs: { ms: report.repeatMs, cpuMs: report.cpu.repeatMs },
}, { firstMs: budgets.firstMs, repeatMs: budgets.repeatMs })
recordPeakMemory(CASES.search, { peakRssMb: report.memory.peakRssMb })
expectWithinBudget(report.firstMs, budgets.firstMs)
expectWithinBudget(report.repeatMs, budgets.repeatMs)
expectWithinBudget(report.memory.peakRssMb, budgets.peakRssMb)
@@ -268,11 +296,13 @@ describe('Session corpus operations', () => {
reports.push(await run<ForkReport>([root('fork'), 'fork', ...[...FORK_STRATA, FORK_P99_RANK].map(String)]))
}
const longest = await run<ForkReport>([root('fork'), 'fork', String(FORK_LONGEST_RANK)])
const forkMs = (report: ForkReport, rank: number): number => {
const forkEntry = (report: ForkReport, rank: number): ForkReport['forks'][number] => {
const fork = report.forks.find(entry => entry.rank === rank)
if (fork === undefined) throw new Error(`fork report omits rank ${String(rank)}`)
return fork.forkMs
return fork
}
const forkMs = (report: ForkReport, rank: number): number => forkEntry(report, rank).forkMs
const forkCpuMs = (report: ForkReport, rank: number): number => forkEntry(report, rank).forkCpuMs
const samples = {
stratumMedianMs: rounded(reports.map(report => median(FORK_STRATA.map(rank => forkMs(report, rank))))),
p99Ms: rounded(reports.map(report => forkMs(report, FORK_P99_RANK))),
@@ -281,10 +311,16 @@ describe('Session corpus operations', () => {
peakRssMb: [...reports, longest].map(report => report.memory.peakRssMb),
}
console.log(JSON.stringify({ benchmark: 'session-corpus/fork', samples, budgets, environment: environment() }))
recordTimings('session-corpus/fork', IO_SHARE.fork,
{ stratumMedianMs: median(samples.stratumMedianMs), p99Ms: median(samples.p99Ms), longestMs: samples.longestMs },
recordTimings(CASES.fork, {
stratumMedianMs: {
ms: median(samples.stratumMedianMs),
cpuMs: median(reports.map(report => median(FORK_STRATA.map(rank => forkCpuMs(report, rank))))),
},
p99Ms: { ms: median(samples.p99Ms), cpuMs: median(reports.map(report => forkCpuMs(report, FORK_P99_RANK))) },
longestMs: { ms: samples.longestMs, cpuMs: forkCpuMs(longest, FORK_LONGEST_RANK) },
},
{ stratumMedianMs: budgets.stratumMedianMs, p99Ms: budgets.p99Ms, longestMs: budgets.longestMs })
recordPeakMemory('session-corpus/fork', { peakRssMb: Math.max(...samples.peakRssMb) })
recordPeakMemory(CASES.fork, { peakRssMb: Math.max(...samples.peakRssMb) })
expectWithinBudget(median(samples.stratumMedianMs), budgets.stratumMedianMs)
expectWithinBudget(median(samples.p99Ms), budgets.p99Ms)
expectWithinBudget(samples.longestMs, budgets.longestMs)
@@ -60,12 +60,20 @@ export interface SeedReport {
readonly seedMs: number
}
/** Process CPU milliseconds, user plus system, over the same intervals as the wall-clock endpoints. */
export interface PhaseCpu {
readonly bootMs: number
readonly firstMs: number
readonly repeatMs: number
}
/** List report: Host boot, then a cold and a repeated Session list. */
export interface ListReport {
readonly mode: 'list'
readonly bootMs: number
readonly firstMs: number
readonly repeatMs: number
readonly cpu: PhaseCpu
readonly items: number
readonly itemsWithProjections: number
readonly memory: CorpusMemory
@@ -77,6 +85,7 @@ export interface SearchReport {
readonly bootMs: number
readonly firstMs: number
readonly repeatMs: number
readonly cpu: PhaseCpu
readonly firstItems: number
readonly repeatItems: number
readonly memory: CorpusMemory
@@ -90,6 +99,7 @@ export interface ForkReport {
readonly rank: number
readonly sourceEvents: number
readonly forkMs: number
readonly forkCpuMs: number
}[]
readonly memory: CorpusMemory
}
@@ -251,16 +261,25 @@ async function seed(root: string, name: string, count: number): Promise<SeedRepo
return { mode: 'seed', name, corpus: written.facts, seedMs: performance.now() - started }
}
function cpuMs(): number {
const usage = process.cpuUsage()
return (usage.user + usage.system) / 1_000
}
async function measureList(root: string, expected: number): Promise<ListReport> {
const started = performance.now()
const cpuStarted = cpuMs()
const { ctx, controller } = await mountHost(root)
try {
const booted = performance.now()
const cpuBooted = cpuMs()
const signal = new AbortController().signal
const first = await controller.list({}, signal)
const firstDone = performance.now()
const cpuFirstDone = cpuMs()
const repeated = await controller.list({}, signal)
const done = performance.now()
const cpuDone = cpuMs()
if (first.items.length !== expected || repeated.items.length !== expected) {
throw new Error(`Session list returned ${String(first.items.length)} of ${String(expected)} Sessions`)
}
@@ -269,6 +288,7 @@ async function measureList(root: string, expected: number): Promise<ListReport>
bootMs: booted - started,
firstMs: firstDone - booted,
repeatMs: done - firstDone,
cpu: { bootMs: cpuBooted - cpuStarted, firstMs: cpuFirstDone - cpuBooted, repeatMs: cpuDone - cpuFirstDone },
items: first.items.length,
itemsWithProjections: first.items.filter(item => item.projections !== undefined).length,
memory: await memory(),
@@ -280,14 +300,18 @@ async function measureList(root: string, expected: number): Promise<ListReport>
async function measureSearch(root: string): Promise<SearchReport> {
const started = performance.now()
const cpuStarted = cpuMs()
const { ctx, controller } = await mountHost(root)
try {
const booted = performance.now()
const cpuBooted = cpuMs()
const signal = new AbortController().signal
const first = await controller.search({ query: SEARCH_QUERIES[0] }, signal)
const firstDone = performance.now()
const cpuFirstDone = cpuMs()
const repeated = await controller.search({ query: SEARCH_QUERIES[1] }, signal)
const done = performance.now()
const cpuDone = cpuMs()
if (first.items.length === 0 || repeated.items.length === 0) {
throw new Error('Session search matched no synthetic Session')
}
@@ -296,6 +320,7 @@ async function measureSearch(root: string): Promise<SearchReport> {
bootMs: booted - started,
firstMs: firstDone - booted,
repeatMs: done - firstDone,
cpu: { bootMs: cpuBooted - cpuStarted, firstMs: cpuFirstDone - cpuBooted, repeatMs: cpuDone - cpuFirstDone },
firstItems: first.items.length,
repeatItems: repeated.items.length,
memory: await memory(),
@@ -313,11 +338,13 @@ async function measureFork(root: string, ranks: readonly number[]): Promise<Fork
const forks: ForkReport['forks'][number][] = []
for (const rank of ranks) {
const forkStarted = performance.now()
const forkCpuStarted = cpuMs()
const { sessionId } = await controller.fork({ sessionId: corpusSessionId(rank) })
const forkMs = performance.now() - forkStarted
const forkCpuMs = cpuMs() - forkCpuStarted
const child = ctx.sessions.get(sessionId)
if (child === undefined) throw new Error(`fork of rank ${String(rank)} did not publish its child`)
forks.push({ rank, sourceEvents: child.inheritedEventCount, forkMs })
forks.push({ rank, sourceEvents: child.inheritedEventCount, forkMs, forkCpuMs })
}
return { mode: 'fork', bootMs: booted - started, forks, memory: await memory() }
} finally {
+36 -11
View File
@@ -15,7 +15,7 @@ import {
ciTimeBudget,
PERFORMANCE_BUDGET_HEADROOM,
} from '../support/calibration.ts'
import { recordTimings } from '../support/scaling-report.ts'
import { recordTimings, type BenchmarkCase } from '../support/scaling-report.ts'
import type {
SessionOpenBenchmarkScenario,
SessionOpenWorkerReport,
@@ -78,10 +78,30 @@ const AGENT_RETAINED_HEAP_BUDGET_MB = Math.ceil(
const WORKER = join(import.meta.dirname, '..', '.dsh-build', 'session-open', 'session-open.worker.js')
/**
* Estimated storage wait: endpoints that include open or read access one Session log that the page cache
* usually holds; restore and projection start after the file handle closes and process memory only.
* Estimated storage wait of the phase endpoints, whose worker reports CPU time only for the whole run: open
* and read access one Session log that the page cache usually holds; restore and projection start after the
* file handle closes and process memory only.
*/
const IO_SHARE = { file: 0.1, memory: 0 } as const
const PHASE_IO_SHARE = { file: 0.1, memory: 0 } as const
/** User scenario of each access kind. */
const ACCESS_AFFECTS: Readonly<Record<SessionAccessKind, string>> = {
'first-open': 'Opening a large Session for the first time after upgrading DSH.',
'post-upgrade-reopen': 'Opening a large Session again in a new DSH process.',
}
/**
* Describe one Session-open case.
* @param access - Access kind and its label.
* @param endpoint - Case suffix and what it measures.
* @returns Case metadata for the scaled report.
*/
function openCase(access: AccessBenchmarkSpec, endpoint: { readonly suffix: string; readonly measures: string }): BenchmarkCase {
return {
id: `session-open/${access.accessKind}/${endpoint.suffix}`,
measures: `${endpoint.measures} One 127,400-event Session, ${access.label}.`,
affects: ACCESS_AFFECTS[access.accessKind],
}
}
type WorkerRun = BuiltBenchmarkWorkerRun<SessionOpenWorkerReport>
@@ -399,11 +419,12 @@ describe('opening a large Session for first open and post-upgrade reopen', () =>
projection: PROJECTION_BUDGET_MS,
},
}))
recordTimings(`session-open/${access.accessKind}/phases`, IO_SHARE.file,
{ openMs: result.openMs.median, readMs: result.readMs.median }, { openMs: access.openBudgetMs, readMs: READ_BUDGET_MS })
recordTimings(`session-open/${access.accessKind}/phases`, IO_SHARE.memory,
{ sessionRestoreMs: result.sessionRestoreMs.median, projectionMs: result.projectionMs.median },
{ sessionRestoreMs: SESSION_RESTORE_BUDGET_MS, projectionMs: PROJECTION_BUDGET_MS })
recordTimings(openCase(access, { suffix: 'phases', measures: 'File open, read, Session restore, and projection phases.' }), {
openMs: { ms: result.openMs.median, ioShare: PHASE_IO_SHARE.file },
readMs: { ms: result.readMs.median, ioShare: PHASE_IO_SHARE.file },
sessionRestoreMs: { ms: result.sessionRestoreMs.median, ioShare: PHASE_IO_SHARE.memory },
projectionMs: { ms: result.projectionMs.median, ioShare: PHASE_IO_SHARE.memory },
}, { openMs: access.openBudgetMs, readMs: READ_BUDGET_MS, sessionRestoreMs: SESSION_RESTORE_BUDGET_MS, projectionMs: PROJECTION_BUDGET_MS })
expectOpenWithinBudget(result.openMs.median, access.openBudgetMs)
expect(result.readMs.median).toBeLessThanOrEqual(READ_BUDGET_MS)
expect(result.sessionRestoreMs.median).toBeLessThanOrEqual(SESSION_RESTORE_BUDGET_MS)
@@ -428,7 +449,9 @@ describe('opening a large Session for first open and post-upgrade reopen', () =>
result,
budgetMs: access.firstHistoryBudgetMs,
}))
recordTimings(`session-open/${access.accessKind}/first-history`, IO_SHARE.file, { totalMs: result.totalMs.median }, { totalMs: access.firstHistoryBudgetMs })
recordTimings(openCase(access, { suffix: 'first-history', measures: 'Host delivery of the first history snapshot.' }),
{ totalMs: { ms: result.totalMs.median, cpuMs: result.cpuUserMs.median + result.cpuSystemMs.median } },
{ totalMs: access.firstHistoryBudgetMs })
expect(result.totalMs.median).toBeLessThanOrEqual(access.firstHistoryBudgetMs)
})
@@ -451,7 +474,9 @@ describe('opening a large Session for first open and post-upgrade reopen', () =>
budgetMs: access.agentResumeBudgetMs,
retainedHeapBudgetMb: AGENT_RETAINED_HEAP_BUDGET_MB,
}))
recordTimings(`session-open/${access.accessKind}/agent-resume`, IO_SHARE.file, { totalMs: result.totalMs.median }, { totalMs: access.agentResumeBudgetMs })
recordTimings(openCase(access, { suffix: 'agent-resume', measures: 'Cold Agent resume until the resumed Agent handle returns.' }),
{ totalMs: { ms: result.totalMs.median, cpuMs: result.cpuUserMs.median + result.cpuSystemMs.median } },
{ totalMs: access.agentResumeBudgetMs })
expect(result.totalMs.median).toBeLessThanOrEqual(access.agentResumeBudgetMs)
expect(result.retainedHeapMb.median).toBeLessThanOrEqual(AGENT_RETAINED_HEAP_BUDGET_MB)
})
+22 -19
View File
@@ -3,8 +3,9 @@
*
* `cpuScore` is the PassMark single-thread rating retrieved on 2026-10-07 from cpubenchmark.net. Measured
* endpoints run on one JavaScript thread, so the report scales their CPU-bound share by the single-thread
* ratio; core count and memory size are not modelled. PassMark ratings across ISAs, and for burstable or
* shared vCPUs, are coarse.
* ratio. `cores` limits endpoints whose measured CPU time shows more than one busy CPU; memory size is not
* modelled. PassMark ratings across ISAs, for burstable or shared vCPUs, and for hybrid efficiency cores, are
* coarse.
*
* `ioLatencyUs` is a typical synchronous 4 KiB random-access latency at queue depth 1 for the profile's
* default storage. Laptop values come from published SSD reviews for the interface generation; cloud values
@@ -17,6 +18,8 @@ export interface MachineProfile {
readonly label: string
readonly cpu: string
readonly cpuScore: number
/** Logical CPUs: hardware threads on laptops, vCPUs on cloud instances. */
readonly cores: number
readonly storage: string
readonly ioLatencyUs: number
/** Lower-case substrings of `os.cpus()[0].model` that identify the measuring machine as this profile. */
@@ -25,23 +28,23 @@ export interface MachineProfile {
/** Profiles in report order. */
export const MACHINE_PROFILES: readonly MachineProfile[] = [
{ id: 'laptop-2015-win', label: '2015 Windows laptop', cpu: 'Intel Core i5-5200U', cpuScore: 1486, storage: 'SATA SSD', ioLatencyUs: 90 },
{ id: 'laptop-2015-mac', label: '2015 MacBook Pro 13"', cpu: 'Intel Core i5-5257U', cpuScore: 1737, storage: 'PCIe AHCI SSD', ioLatencyUs: 90 },
{ id: 'laptop-2019-win', label: '2019 Windows laptop', cpu: 'Intel Core i5-8265U', cpuScore: 2003, storage: 'NVMe PCIe 3.0 SSD', ioLatencyUs: 70 },
{ id: 'laptop-2023-win', label: '2023 Windows laptop', cpu: 'Intel Core i5-1335U', cpuScore: 3241, storage: 'NVMe PCIe 4.0 SSD', ioLatencyUs: 55 },
{ id: 'laptop-2023-mac', label: '2023 MacBook Air', cpu: 'Apple M2', cpuScore: 4208, storage: 'Apple SSD', ioLatencyUs: 55, cpuModelMatches: ['apple m2'] },
{ id: 'laptop-2026-win', label: '2026 Windows laptop', cpu: 'Intel Core Ultra 5 225H', cpuScore: 4247, storage: 'NVMe PCIe 4.0 SSD', ioLatencyUs: 50 },
{ id: 'laptop-2026-mac', label: '2026 MacBook', cpu: 'Apple M5', cpuScore: 6341, storage: 'Apple SSD', ioLatencyUs: 50, cpuModelMatches: ['apple m5'] },
{ id: 'aws-t3', label: 'AWS t3.medium', cpu: 'Intel Xeon Platinum 8259CL', cpuScore: 1948, storage: 'EBS gp3', ioLatencyUs: 500 },
{ id: 'aws-m6i', label: 'AWS m6i.large', cpu: 'Intel Xeon Platinum 8375C', cpuScore: 2474, storage: 'EBS gp3', ioLatencyUs: 500 },
{ id: 'aws-m7i', label: 'AWS m7i.large', cpu: 'Intel Xeon Platinum 8488C', cpuScore: 3113, storage: 'EBS gp3', ioLatencyUs: 500 },
{ id: 'aws-m7g', label: 'AWS m7g.large', cpu: 'AWS Graviton3 (Neoverse V1)', cpuScore: 1554, storage: 'EBS gp3', ioLatencyUs: 500 },
{ id: 'gcp-n2', label: 'GCP n2-standard-4', cpu: 'Intel Xeon Platinum 8273CL', cpuScore: 2200, storage: 'pd-balanced', ioLatencyUs: 800 },
{ id: 'azure-d4s-v5', label: 'Azure D4s v5', cpu: 'Intel Xeon Ice Lake (8375C rating)', cpuScore: 2474, storage: 'Premium SSD P10', ioLatencyUs: 1000 },
{ id: 'aliyun-g7', label: 'Alibaba Cloud ecs.g7.xlarge', cpu: 'Intel Xeon Ice Lake (8375C rating)', cpuScore: 2474, storage: 'ESSD PL0', ioLatencyUs: 400 },
{ id: 'github-ubuntu-24.04', label: 'GitHub ubuntu-24.04 runner', cpu: 'AMD EPYC 7763', cpuScore: 2517, storage: 'Azure managed disk', ioLatencyUs: 300, cpuModelMatches: ['epyc 7763'] },
{ id: 'github-ubuntu-24.04-genoa', label: 'GitHub ubuntu-24.04 runner (Genoa)', cpu: 'AMD EPYC 9V74', cpuScore: 2888, storage: 'Azure managed disk', ioLatencyUs: 300, cpuModelMatches: ['epyc 9v74'] },
{ id: 'reference-m4-pro', label: 'Calibration reference', cpu: 'Apple M4 Pro', cpuScore: 4547, storage: 'Apple SSD', ioLatencyUs: 55, cpuModelMatches: ['apple m4'] },
{ id: 'laptop-2015-win', label: '2015 Windows laptop', cpu: 'Intel Core i5-5200U', cpuScore: 1486, cores: 4, storage: 'SATA SSD', ioLatencyUs: 90 },
{ id: 'laptop-2015-mac', label: '2015 MacBook Pro 13"', cpu: 'Intel Core i5-5257U', cpuScore: 1737, cores: 4, storage: 'PCIe AHCI SSD', ioLatencyUs: 90 },
{ id: 'laptop-2019-win', label: '2019 Windows laptop', cpu: 'Intel Core i5-8265U', cpuScore: 2003, cores: 8, storage: 'NVMe PCIe 3.0 SSD', ioLatencyUs: 70 },
{ id: 'laptop-2023-win', label: '2023 Windows laptop', cpu: 'Intel Core i5-1335U', cpuScore: 3241, cores: 12, storage: 'NVMe PCIe 4.0 SSD', ioLatencyUs: 55 },
{ id: 'laptop-2023-mac', label: '2023 MacBook Air', cpu: 'Apple M2', cpuScore: 4208, cores: 8, storage: 'Apple SSD', ioLatencyUs: 55, cpuModelMatches: ['apple m2'] },
{ id: 'laptop-2026-win', label: '2026 Windows laptop', cpu: 'Intel Core Ultra 5 225H', cpuScore: 4247, cores: 14, storage: 'NVMe PCIe 4.0 SSD', ioLatencyUs: 50 },
{ id: 'laptop-2026-mac', label: '2026 MacBook', cpu: 'Apple M5', cpuScore: 6341, cores: 10, storage: 'Apple SSD', ioLatencyUs: 50, cpuModelMatches: ['apple m5'] },
{ id: 'aws-t3', label: 'AWS t3.medium', cpu: 'Intel Xeon Platinum 8259CL', cpuScore: 1948, cores: 2, storage: 'EBS gp3', ioLatencyUs: 500 },
{ id: 'aws-m6i', label: 'AWS m6i.large', cpu: 'Intel Xeon Platinum 8375C', cpuScore: 2474, cores: 2, storage: 'EBS gp3', ioLatencyUs: 500 },
{ id: 'aws-m7i', label: 'AWS m7i.large', cpu: 'Intel Xeon Platinum 8488C', cpuScore: 3113, cores: 2, storage: 'EBS gp3', ioLatencyUs: 500 },
{ id: 'aws-m7g', label: 'AWS m7g.large', cpu: 'AWS Graviton3 (Neoverse V1)', cpuScore: 1554, cores: 2, storage: 'EBS gp3', ioLatencyUs: 500 },
{ id: 'gcp-n2', label: 'GCP n2-standard-4', cpu: 'Intel Xeon Platinum 8273CL', cpuScore: 2200, cores: 4, storage: 'pd-balanced', ioLatencyUs: 800 },
{ id: 'azure-d4s-v5', label: 'Azure D4s v5', cpu: 'Intel Xeon Ice Lake (8375C rating)', cpuScore: 2474, cores: 4, storage: 'Premium SSD P10', ioLatencyUs: 1000 },
{ id: 'aliyun-g7', label: 'Alibaba Cloud ecs.g7.xlarge', cpu: 'Intel Xeon Ice Lake (8375C rating)', cpuScore: 2474, cores: 4, storage: 'ESSD PL0', ioLatencyUs: 400 },
{ id: 'github-ubuntu-24.04', label: 'GitHub ubuntu-24.04 runner', cpu: 'AMD EPYC 7763', cpuScore: 2517, cores: 2, storage: 'Azure managed disk', ioLatencyUs: 300, cpuModelMatches: ['epyc 7763'] },
{ id: 'github-ubuntu-24.04-genoa', label: 'GitHub ubuntu-24.04 runner (Genoa)', cpu: 'AMD EPYC 9V74', cpuScore: 2888, cores: 2, storage: 'Azure managed disk', ioLatencyUs: 300, cpuModelMatches: ['epyc 9v74'] },
{ id: 'reference-m4-pro', label: 'Calibration reference', cpu: 'Apple M4 Pro', cpuScore: 4547, cores: 14, storage: 'Apple SSD', ioLatencyUs: 55, cpuModelMatches: ['apple m4'] },
]
/**
+79 -37
View File
@@ -6,17 +6,19 @@
* `vitest.bench.config.ts` clears {@link REPORT_DIRECTORY} before the run and renders `report.html` after
* it, including after failed cases.
*
* A scaled endpoint is `ms × ((1 − io) × cpuScore(measured) / cpuScore(target) + io × ioLatency(target) /
* ioLatency(measured))`, where `io` is the declared share of the endpoint's wall time spent waiting on
* storage. The report lets readers change that share per endpoint. Memory and dimensionless ratios are not
* scaled.
* A scaled endpoint is `ms × ((1 − io) × cpuScore(measured) / cpuScore(target) × coreFactor + io ×
* ioLatency(target) / ioLatency(measured))`. When the endpoint reports process CPU time, `io` is the measured
* share of wall time not spent on CPU, `max(0, 1 − cpuMs / ms)`, and its busy parallelism is
* `max(1, cpuMs / ms)`; otherwise `io` is a declared estimate and parallelism is 1. `coreFactor` is
* `max(1, parallelism / cores(target)) / max(1, parallelism / cores(measured))`. Readers can override `io`
* per endpoint. Memory and dimensionless ratios are not scaled.
*
* Bench files also pass peak memory to {@link recordPeakMemory}. The report compares those unscaled peaks
* with 4 GB and 8 GB of available memory; it does not run cases under an operating-system memory limit.
*/
import { appendFileSync, existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { cpus } from 'node:os'
import { availableParallelism, cpus } from 'node:os'
import { join } from 'node:path'
import { MACHINE_PROFILES, matchMachineProfile } from './machine-profiles.ts'
@@ -25,50 +27,74 @@ export const REPORT_DIRECTORY = join(import.meta.dirname, '..', '.dsh-report')
const RESULTS_PATH = join(REPORT_DIRECTORY, 'results.jsonl')
const REPORT_PATH = join(REPORT_DIRECTORY, 'report.html')
/** One recorded wall-clock endpoint. */
export interface TimingRecord {
readonly kind: 'time'
/** Identity and reader-facing description of one benchmark case, shown when hovering its rows in the report. */
export interface BenchmarkCase {
/** Case id, such as `session-open/first-open/phases`. */
readonly id: string
/** What the timed or sampled work is, including workload size. */
readonly measures: string
/** Which user-visible scenario the measurement represents. */
readonly affects: string
}
/** Fields that every record copies from its {@link BenchmarkCase}. */
interface CaseFields {
readonly benchmark: string
readonly measures: string
readonly affects: string
}
function caseFields(benchmarkCase: BenchmarkCase): CaseFields {
return { benchmark: benchmarkCase.id, measures: benchmarkCase.measures, affects: benchmarkCase.affects }
}
/**
* One endpoint: its wall-clock milliseconds and either the process CPU milliseconds measured over the same
* interval or, where the worker cannot measure CPU time, a declared share from 0 to 1 of wall time spent
* waiting on storage.
*/
export type Timing =
| { readonly ms: number; readonly cpuMs: number }
| { readonly ms: number; readonly ioShare: number }
/** One recorded wall-clock endpoint. */
export type TimingRecord = CaseFields & Timing & {
readonly kind: 'time'
readonly metric: string
readonly ms: number
readonly budgetMs?: number
readonly ioShare: number
}
/**
* Append measured endpoints of one benchmark case to the report results.
* @param benchmark - Benchmark case id, such as `session-open/current/phases`.
* @param ioShare - Declared share, from 0 to 1, of the endpoints' wall time spent waiting on storage.
* @param timings - Measured wall-clock milliseconds keyed by endpoint name; record before asserting budgets.
* @param benchmarkCase - Case id and description.
* @param timings - Endpoints keyed by name; record before asserting budgets.
* @param budgetsMs - CI budgets keyed by the same endpoint names, where the case enforces one.
*/
export function recordTimings(
benchmark: string,
ioShare: number,
timings: Readonly<Record<string, number>>,
benchmarkCase: BenchmarkCase,
timings: Readonly<Record<string, Timing>>,
budgetsMs: Readonly<Record<string, number>> = {},
): void {
appendRecords(Object.entries(timings).map(([metric, ms]): TimingRecord => {
appendRecords(Object.entries(timings).map(([metric, timing]): TimingRecord => {
const budgetMs = budgetsMs[metric]
return { kind: 'time', benchmark, metric, ms, ioShare, ...budgetMs === undefined ? {} : { budgetMs } }
return { kind: 'time', ...caseFields(benchmarkCase), metric, ...timing, ...budgetMs === undefined ? {} : { budgetMs } }
}))
}
/** One recorded peak memory measurement. */
export interface MemoryRecord {
export interface MemoryRecord extends CaseFields {
readonly kind: 'memory'
readonly benchmark: string
readonly metric: string
readonly mb: number
}
/**
* Append peak memory of one benchmark case to the report results.
* @param benchmark - Benchmark case id, such as `session-corpus/search`.
* @param benchmarkCase - Case id and description.
* @param peaksMb - Peak megabytes keyed by measurement name, such as `peakRssMb`.
*/
export function recordPeakMemory(benchmark: string, peaksMb: Readonly<Record<string, number>>): void {
appendRecords(Object.entries(peaksMb).map(([metric, mb]): MemoryRecord => ({ kind: 'memory', benchmark, metric, mb })))
export function recordPeakMemory(benchmarkCase: BenchmarkCase, peaksMb: Readonly<Record<string, number>>): void {
appendRecords(Object.entries(peaksMb).map(([metric, mb]): MemoryRecord => ({ kind: 'memory', ...caseFields(benchmarkCase), metric, mb })))
}
function appendRecords(records: readonly (TimingRecord | MemoryRecord)[]): void {
@@ -93,6 +119,7 @@ export function teardown(): void {
node: process.version,
revision: process.env['GITHUB_SHA'],
generatedAt: new Date().toISOString(),
cores: availableParallelism(),
measuredProfile: matchMachineProfile(cpuModel),
}))
console.log(`benchmark report: ${REPORT_PATH}`)
@@ -106,6 +133,8 @@ export interface ReportData {
readonly node: string
readonly revision: string | undefined
readonly generatedAt: string
/** Logical CPUs available to the measuring process; replaces the matched profile's core count. */
readonly cores: number
/** Profile id describing the measuring machine; the reader chooses one when absent. */
readonly measuredProfile: string | undefined
}
@@ -146,14 +175,16 @@ tbody tr { cursor: pointer; }
<h1>DeepSeek Harness benchmark scenarios</h1>
<p class="meta" id="meta"></p>
<h2>Machines</h2>
<p class="note">Each measured endpoint is rescaled to other machines: its CPU share by the PassMark single-thread rating ratio and its storage share by the typical 4 KiB queue-depth-1 latency ratio of the default disk. Profiles are estimates; core count and memory size are not modelled.</p>
<p class="note">Each measured endpoint is rescaled to other machines. Its CPU share scales by the PassMark single-thread rating ratio and slows further when the endpoint kept more CPUs busy than the target has. Its storage share scales by the typical 4 KiB queue-depth-1 latency ratio of the default disk. Endpoints that report process CPU time use their measured share of wall time not spent on CPU as the storage share; the others use a declared estimate. Profiles are estimates; memory size is not modelled.</p>
<div class="controls">
<label>Measured on <select id="measured"></select></label>
<label>Endpoint <select id="endpoint"></select></label>
<label>Storage-bound share <input id="io" type="range" min="0" max="1" step="0.05"> <output id="io-value"></output></label>
<label>Storage-bound share <input id="io" type="range" min="0" max="1" step="0.01"> <output id="io-value"></output></label>
<button id="io-reset" type="button">Reset to default</button>
</div>
<p class="note" id="case"></p>
<svg id="chart" width="960" role="img" aria-label="Scaled endpoint per machine"></svg>
<p class="legend"><span class="b0">&lt; 100 ms</span><span class="b1">&lt; 1 s</span><span class="b2">&lt; 10 s</span><span class="b3">&ge; 10 s</span> Click a row to chart it.</p>
<p class="legend"><span class="b0">&lt; 100 ms</span><span class="b1">&lt; 1 s</span><span class="b2">&lt; 10 s</span><span class="b3">&ge; 10 s</span> Hover an endpoint for what it measures; click a row to chart it.</p>
<div class="wrap"><table id="table"></table></div>
<h2>Memory pressure</h2>
<p class="note">Measured peak memory of each case against 4 GB and 8 GB of memory available to DeepSeek Harness. Peaks are not scaled. Cases were not run under an operating-system memory limit, so swapping and reclaim costs are not measured.</p>
@@ -168,7 +199,12 @@ const times = data.records.filter(r => r.kind === 'time');
const memories = data.records.filter(r => r.kind === 'memory');
const AVAILABLE_MB = [4096, 8192];
const keyOf = r => r.benchmark + ' · ' + r.metric;
const ioShares = times.map(r => r.ioShare);
const describe = r => 'Measures: ' + r.measures + '\\nAffects: ' + r.affects;
const defaultShare = r => r.cpuMs === undefined ? r.ioShare : Math.min(1, Math.max(0, 1 - r.cpuMs / r.ms));
const shareSource = r => r.cpuMs === undefined ? 'declared' : 'measured';
const parallelism = r => r.cpuMs === undefined ? 1 : Math.max(1, r.cpuMs / r.ms);
const pct = share => (share < 0.1 && share > 0 ? (share * 100).toFixed(1) : Math.round(share * 100)) + '%';
const ioShares = times.map(defaultShare);
let selected = 0;
const fmt = ms => ms >= 10000 ? (ms / 1000).toFixed(0) + ' s' : ms >= 1000 ? (ms / 1000).toFixed(1) + ' s' : ms >= 10 ? ms.toFixed(0) + ' ms' : ms.toFixed(1) + ' ms';
const fmtMb = mb => mb >= 1024 ? (mb / 1024).toFixed(1) + ' GB' : mb.toFixed(0) + ' MB';
@@ -179,9 +215,12 @@ function scale(index, target) {
const record = times[index];
const base = profiles.find(p => p.id === $('measured').value);
const io = ioShares[index];
return record.ms * ((1 - io) * base.cpuScore / target.cpuScore + io * target.ioLatencyUs / base.ioLatencyUs);
const busy = parallelism(record);
const baseCores = base.id === data.measuredProfile ? data.cores : base.cores;
const coreFactor = Math.max(1, busy / target.cores) / Math.max(1, busy / baseCores);
return record.ms * ((1 - io) * base.cpuScore / target.cpuScore * coreFactor + io * target.ioLatencyUs / base.ioLatencyUs);
}
function esc(text) { const node = document.createElement('span'); node.textContent = text; return node.innerHTML; }
function esc(text) { const node = document.createElement('span'); node.textContent = text; return node.innerHTML.replaceAll('"', '&quot;'); }
/** Draws one horizontal bar per row on a logarithmic axis with labelled vertical ticks. */
function bars(svg, rows, ticks, format) {
const left = 380, width = 500, rowH = 22, top = 24;
@@ -202,15 +241,16 @@ function bars(svg, rows, ticks, format) {
svg.setAttribute('height', String(top + rows.length * rowH + 8));
svg.innerHTML = html;
}
$('meta').textContent = 'Measured on ' + data.cpuModel + ' (' + data.platform + ', Node ' + data.node + ')' + (data.revision ? ', revision ' + data.revision.slice(0, 12) : '') + ', ' + data.generatedAt + '.';
for (const p of profiles) $('measured').add(new Option(p.label + ' — ' + p.cpu, p.id));
$('meta').textContent = 'Measured on ' + data.cpuModel + ' with ' + data.cores + ' available CPUs (' + data.platform + ', Node ' + data.node + ')' + (data.revision ? ', revision ' + data.revision.slice(0, 12) : '') + ', ' + data.generatedAt + '.';
const machine = p => p.cpu + ', ' + p.cores + ' CPUs, ' + p.storage;
for (const p of profiles) $('measured').add(new Option(p.label + ' — ' + machine(p), p.id));
$('measured').value = data.measuredProfile ?? 'github-ubuntu-24.04';
if (!data.measuredProfile) $('meta').textContent += ' No profile matches this CPU; choose the closest one.';
times.forEach((r, i) => $('endpoint').add(new Option(keyOf(r), String(i))));
function table() {
let html = '<thead><tr><th>Endpoint</th><th>Measured</th><th>CI budget</th>' + profiles.map(p => '<th title="' + esc(p.cpu + ' · ' + p.storage) + '">' + esc(p.label) + '</th>').join('') + '</tr></thead><tbody>';
let html = '<thead><tr><th>Endpoint</th><th>Measured</th><th>CI budget</th><th title="Default storage-bound share and its source">Storage share</th><th title="Average CPUs kept busy, from process CPU time">CPUs busy</th>' + profiles.map(p => '<th title="' + esc(machine(p)) + '">' + esc(p.label) + '</th>').join('') + '</tr></thead><tbody>';
times.forEach((r, i) => {
html += '<tr data-index="' + i + '"' + (i === selected ? ' class="selected"' : '') + '><td>' + esc(keyOf(r)) + '</td><td>' + fmt(r.ms) + '</td><td>' + (r.budgetMs === undefined ? '' : fmt(r.budgetMs)) + '</td>';
html += '<tr data-index="' + i + '"' + (i === selected ? ' class="selected"' : '') + '><td title="' + esc(describe(r)) + '">' + esc(keyOf(r)) + '</td><td>' + fmt(r.ms) + '</td><td>' + (r.budgetMs === undefined ? '' : fmt(r.budgetMs)) + '</td><td>' + pct(ioShares[i]) + ' ' + (ioShares[i] === defaultShare(r) ? shareSource(r) : 'override') + '</td><td>' + (r.cpuMs === undefined ? '' : parallelism(r).toFixed(2)) + '</td>';
for (const p of profiles) { const ms = scale(i, p); html += '<td class="b' + band(ms) + '">' + fmt(ms) + '</td>'; }
html += '</tr>';
});
@@ -220,20 +260,21 @@ function render() {
if (times.length === 0) return;
$('endpoint').value = String(selected);
$('io').value = String(ioShares[selected]);
$('io-value').textContent = Math.round(ioShares[selected] * 100) + '%';
$('io-value').textContent = pct(ioShares[selected]) + ' (default ' + pct(defaultShare(times[selected])) + ' ' + shareSource(times[selected]) + ')';
$('case').textContent = describe(times[selected]).replace('\\n', ' ');
bars($('chart'), profiles.map(p => {
const ms = scale(selected, p);
return { label: p.label + ' · ' + p.storage, title: p.cpu, value: ms, band: band(ms), current: p.id === $('measured').value };
return { label: p.label + ' · ' + p.storage, title: machine(p), value: ms, band: band(ms), current: p.id === $('measured').value };
}), [1, 10, 100, 1000, 10000, 100000, 1000000].map(value => ({ value, label: fmt(value), marked: value === 100 || value === 1000 })), fmt);
table();
}
function renderMemory() {
if (memories.length === 0) return;
bars($('memory-chart'), memories.map(r => ({ label: keyOf(r), title: keyOf(r), value: r.mb, band: memoryBand(r.mb / AVAILABLE_MB[0]), current: false })),
bars($('memory-chart'), memories.map(r => ({ label: keyOf(r), title: describe(r), value: r.mb, band: memoryBand(r.mb / AVAILABLE_MB[0]), current: false })),
[16, 64, 256, 1024].map(value => ({ value, label: fmtMb(value), marked: false })).concat(AVAILABLE_MB.map(value => ({ value, label: fmtMb(value) + ' available', marked: true }))), fmtMb);
let html = '<thead><tr><th>Case</th><th>Peak</th>' + AVAILABLE_MB.map(mb => '<th>' + fmtMb(mb) + ' available</th>').join('') + '</tr></thead><tbody>';
for (const r of memories) {
html += '<tr><td>' + esc(keyOf(r)) + '</td><td>' + fmtMb(r.mb) + '</td>';
html += '<tr><td title="' + esc(describe(r)) + '">' + esc(keyOf(r)) + '</td><td>' + fmtMb(r.mb) + '</td>';
for (const mb of AVAILABLE_MB) html += '<td class="b' + memoryBand(r.mb / mb) + '">' + Math.round(100 * r.mb / mb) + '%</td>';
html += '</tr>';
}
@@ -242,6 +283,7 @@ function renderMemory() {
$('measured').onchange = render;
$('endpoint').onchange = () => { selected = Number($('endpoint').value); render(); };
$('io').oninput = () => { ioShares[selected] = Number($('io').value); render(); };
$('io-reset').onclick = () => { ioShares[selected] = defaultShare(times[selected]); render(); };
$('table').onclick = event => { const row = event.target.closest('tr[data-index]'); if (row) { selected = Number(row.dataset.index); render(); } };
render();
renderMemory();
+16 -5
View File
@@ -3,7 +3,7 @@ import { join } from 'node:path'
import { expect, it } from 'vitest'
import { runBuiltBenchmarkWorker } from '../support/built-worker.ts'
import { ciTimeBudget } from '../support/calibration.ts'
import { recordTimings } from '../support/scaling-report.ts'
import { recordTimings, type BenchmarkCase, type Timing } from '../support/scaling-report.ts'
import type { TerminalIoReport } from './terminal-io.worker.ts'
const MIB = 1024 * 1024
@@ -14,6 +14,16 @@ const EXPECTED_MS = { steadyIngest: 20, steadyComplete: 50, fullComplete: 120 }
const MAX_CAPACITY_RATIO = 4
/** Terminal output is buffered in memory; no storage access is timed. */
const IO_SHARE = 0
const STEADY_CASE: BenchmarkCase = {
id: 'terminal-io/steady',
measures: 'Terminal ingest and completion of 1 MiB of output into 128 KiB and 4 MiB retained windows.',
affects: 'Commands that keep printing output in the terminal tool.',
}
const FIVE_MIB_CASE: BenchmarkCase = {
id: 'terminal-io/five-mib',
measures: 'Complete terminal send of 5 MiB of output with a 4 MiB retained window.',
affects: 'Commands that print large logs.',
}
const MAX_RETAINED_HEAP_BYTES = 16 * MIB
function median(values: readonly number[]): number {
@@ -41,9 +51,10 @@ it('bounds steady overflow cost as retained terminal capacity grows 32 times', a
const ingestBudgetMs = ciTimeBudget(EXPECTED_MS.steadyIngest)
const completeBudgetMs = ciTimeBudget(EXPECTED_MS.steadyComplete)
console.log(JSON.stringify({ scenario: 'terminal-steady', small, large, capacityRatio, ingestBudgetMs, completeBudgetMs }))
recordTimings('terminal-io/steady', IO_SHARE, {
smallIngestMs: median(small.map(row => row.ingestMs)), largeIngestMs: median(large.map(row => row.ingestMs)),
smallCompleteMs: median(small.map(row => row.completeMs)), largeCompleteMs: median(large.map(row => row.completeMs)),
const declared = (ms: number): Timing => ({ ms, ioShare: IO_SHARE })
recordTimings(STEADY_CASE, {
smallIngestMs: declared(median(small.map(row => row.ingestMs))), largeIngestMs: declared(median(large.map(row => row.ingestMs))),
smallCompleteMs: declared(median(small.map(row => row.completeMs))), largeCompleteMs: declared(median(large.map(row => row.completeMs))),
}, { smallIngestMs: ingestBudgetMs, largeIngestMs: ingestBudgetMs, smallCompleteMs: completeBudgetMs, largeCompleteMs: completeBudgetMs })
expect(capacityRatio).toBeLessThanOrEqual(MAX_CAPACITY_RATIO)
for (const rows of [small, large]) {
@@ -71,7 +82,7 @@ it('completes a five MiB terminal send with bounded retained output', async () =
const completeMedianMs = median(samples.map(row => row.completeMs))
const completeBudgetMs = ciTimeBudget(EXPECTED_MS.fullComplete)
console.log(JSON.stringify({ scenario: 'terminal-five-mib', samples, completeMedianMs, completeBudgetMs }))
recordTimings('terminal-io/five-mib', IO_SHARE, { completeMs: completeMedianMs }, { completeMs: completeBudgetMs })
recordTimings(FIVE_MIB_CASE, { completeMs: { ms: completeMedianMs, ioShare: IO_SHARE } }, { completeMs: completeBudgetMs })
expect(completeMedianMs).toBeLessThanOrEqual(completeBudgetMs)
expect(Math.max(...samples.map(row => row.retainedHeapBytes))).toBeLessThanOrEqual(MAX_RETAINED_HEAP_BYTES)
})