mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-10-26 16:55:08 +00:00
test(bench): scale storage share per endpoint and record the report decision
Session restore and projection run after the file closes, so they use no storage share; the page keeps one share per endpoint. Record the repeat projection-list budget, drop the post-GC page heap from the memory view, remove the unused profile group, and add the owning Agent Note.
This commit is contained in:
1 parent
12c33f6d0c
commit
9baa8bc8fe
9 files changed
+153
-43
No files matched your search
@@ -0,0 +1,28 @@
|
||||
# Bilingual-pair consistency record for 2026-10-07-scaled-benchmark-report.md (docs/i18n/README.md): per heading
|
||||
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
|
||||
# After editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-10-07-scaled-benchmark-report.md
|
||||
/agent-note-scaled-machine-and-memory-pressure-benchmark-report:
|
||||
en: 486b21866106fb62
|
||||
zh: eb0f2e90eeb426a3
|
||||
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/problem:
|
||||
en: d8e9dee3e22d691a
|
||||
zh: 53d9fae666f032b7
|
||||
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/decision:
|
||||
en: aca7e6a17dc9c72d
|
||||
zh: 7dd535ac26cfb843
|
||||
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/decision/machine-scaling:
|
||||
en: d6ecda154d834793
|
||||
zh: 68b16ea6580d5526
|
||||
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/decision/memory-pressure:
|
||||
en: bec0d7592ed58486
|
||||
zh: 2be2350c142f133d
|
||||
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/decision/exclusions:
|
||||
en: 57d602db18c6e825
|
||||
zh: ac731f65c0c253fa
|
||||
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/alternatives-considered:
|
||||
en: 0f6786650d04e6bf
|
||||
zh: 50a0534c9d39ce7b
|
||||
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/consequences:
|
||||
en: b8724a183365bd49
|
||||
zh: 2819469dc566f343
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: Scaled machine and memory-pressure benchmark report
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-10-07-scaled-benchmark-report.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
Benchmark budgets state whether one endpoint regressed on the hosted runner. They do not tell readers how slow the same endpoint is on an older laptop or a typical cloud instance, or whether its working set fits a machine with 4 GB or 8 GB of available memory, so a passing or failing number does not convey user impact.
|
||||
|
||||
## Decision
|
||||
|
||||
Every `test:bench` run writes `benchmarks/.dsh-report/report.html`, a self-contained page without network resources. Bench files pass each budgeted wall-clock median to `recordTimings` and whole-process peak RSS to `recordPeakMemory` from [scaling-report.ts](../../../../benchmarks/support/scaling-report.ts) before asserting budgets. A Vitest global setup clears the results before the run and renders the page after it, including after failed cases. Budgets and assertions do not change.
|
||||
|
||||
### Machine scaling
|
||||
|
||||
[machine-profiles.ts](../../../../benchmarks/support/machine-profiles.ts) lists laptops from 2015, 2019, 2023, and 2026 with SSD storage, mainstream cloud instances with their default system disks, the hosted runners, and the M4 Pro calibration reference. Each profile has a PassMark single-thread rating, retrieved on 2026-10-07, and a typical 4 KiB queue-depth-1 latency of its storage. A scaled endpoint is `ms × ((1 − io) × cpuScore(measured) / cpuScore(target) + io × ioLatency(target) / ioLatency(measured))`. `io` is a commented storage-wait estimate declared per endpoint beside the benchmark; the page lets readers change it per endpoint. The measuring machine is matched from the CPU model and can be changed in the page.
|
||||
|
||||
On Apple M5 Pro the first content-index search measured 58.7 s; the model predicts 168 s on the EPYC 7763 runner, against a recorded hosted median of 144 s.
|
||||
|
||||
### Memory pressure
|
||||
|
||||
The Session corpus list, search, and fork cases report whole-process peak RSS for the largest synthetic working sets. The page shows each peak as a share of 4 GB and 8 GB of available memory. Peaks are not scaled.
|
||||
|
||||
### Exclusions
|
||||
|
||||
Long-Session `first`, `input`, and `streamWall` include paced replay waits and are not recorded. The browser page heap is read after a forced garbage collection, so it is not a peak and is not recorded. Core count, memory size, and thermal limits are not modelled.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Run cases under an operating-system memory limit.** A cgroup limit needs root on Linux CI and is unavailable on macOS. The measured peaks stay below 1 GB, so a 4 GB limit would not change the timed path while adding privileged setup to a required gate.
|
||||
|
||||
**Measure storage wait from CPU time.** Wall time minus CPU time also contains event-loop yields, timers, and child scheduling, so it does not isolate storage. Most workers do not report CPU time either.
|
||||
|
||||
**Benchmark-wide storage share.** Session open restores and projects after closing its file, so one share per benchmark overstates high-latency disks for memory-only endpoints.
|
||||
|
||||
**Multi-core or Geekbench scores.** Timed endpoints run on one JavaScript thread. Geekbench pages were not retrievable for citation; PassMark single-thread ratings were.
|
||||
|
||||
## Consequences
|
||||
|
||||
Readers can see each endpoint on reviewed machines and adjust the storage estimate, without changing any gate. The page depends on reviewed constants: cross-ISA ratings, shared vCPUs, and disk latency classes are coarse, so results are estimates rather than measurements. The memory view does not measure swap or reclaim costs. Publishing the page from CI requires an upload step in the `node 24 / benchmarks` job.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: 按机器缩放与内存压力的基准报告
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-10-07-scaled-benchmark-report.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
基准预算只说明某个端点是否在托管 runner 上回归。读者无法据此判断同一端点在旧笔记本或典型云服务器上有多慢,也无法判断其工作集能否放进 4 GB 或 8 GB 可用内存,因此通过或失败的数字无法体现对用户的影响。
|
||||
|
||||
## 决定
|
||||
|
||||
每次 `test:bench` 运行都会写出 `benchmarks/.dsh-report/report.html`,这是一个不依赖网络资源的独立页面。基准文件在断言预算之前,把每个有预算的墙钟中位数传给 [scaling-report.ts](../../../../benchmarks/support/scaling-report.ts) 的 `recordTimings`,把整进程峰值 RSS 传给 `recordPeakMemory`。Vitest global setup 在运行前清空结果,在运行后渲染页面,用例失败时也会渲染。预算和断言不变。
|
||||
|
||||
### 机器缩放
|
||||
|
||||
[machine-profiles.ts](../../../../benchmarks/support/machine-profiles.ts) 列出 2015、2019、2023、2026 年使用 SSD 的笔记本,带默认系统盘的主流云服务器,托管 runner,以及 M4 Pro 校准参考机。每个配置包含 2026-10-07 获取的 PassMark 单线程评分,以及其存储的典型 4 KiB 队列深度 1 延迟。缩放后的端点为 `ms × ((1 − io) × cpuScore(measured) / cpuScore(target) + io × ioLatency(target) / ioLatency(measured))`。`io` 是在基准旁按端点声明并注释的存储等待估计值;页面允许读者按端点修改。测量机器按 CPU 型号匹配,也可以在页面中修改。
|
||||
|
||||
在 Apple M5 Pro 上,首次内容索引搜索实测 58.7 s;模型预测 EPYC 7763 runner 上为 168 s,托管 CI 记录的中位数为 144 s。
|
||||
|
||||
### 内存压力
|
||||
|
||||
Session 语料的列表、搜索和 fork 用例报告最大合成工作集的整进程峰值 RSS。页面把每个峰值显示为 4 GB 和 8 GB 可用内存的占比。峰值不做缩放。
|
||||
|
||||
### 排除项
|
||||
|
||||
长 Session 的 `first`、`input` 和 `streamWall` 包含按节奏回放的等待,不记录。浏览器页面堆在强制垃圾回收后读取,不是峰值,不记录。不建模核心数、内存容量和散热限制。
|
||||
|
||||
## 备选方案
|
||||
|
||||
**在操作系统内存限制下运行用例。** Linux CI 上的 cgroup 限制需要 root,macOS 上不可用。实测峰值低于 1 GB,4 GB 限制不会改变计时路径,却会给必需门禁增加特权配置。
|
||||
|
||||
**根据 CPU 时间测量存储等待。** 墙钟时间减去 CPU 时间还包含事件循环让出、定时器和子进程调度,无法单独分离存储。大多数 worker 也不报告 CPU 时间。
|
||||
|
||||
**按基准统一存储占比。** Session open 在关闭文件后才执行恢复和投影,按基准使用一个占比会高估高延迟磁盘对纯内存端点的影响。
|
||||
|
||||
**多核或 Geekbench 分数。** 被计时端点运行在单个 JavaScript 线程上。Geekbench 页面无法获取以供引用,PassMark 单线程评分可以获取。
|
||||
|
||||
## 影响
|
||||
|
||||
读者可以在经审阅的机器上查看每个端点并调整存储估计值,而不改变任何门禁。页面依赖经审阅的常量:跨指令集评分、共享 vCPU 和磁盘延迟等级都较粗略,因此结果是估计值而不是测量值。内存视图不测量交换和回收成本。要从 CI 发布该页面,需要在 `node 24 / benchmarks` job 中添加上传步骤。
|
||||
@@ -8,7 +8,7 @@ This tree owns required, repository-level performance gates whose measured user
|
||||
- Browser workflow cases drive built Client bundles through the shared shipped-composition Web scaffold. Report its source-resolved test Host separately from published-Host evidence; two animation frames prove a rendering opportunity, not hardware presentation. Use fresh browsers and private scaffold worlds per sample.
|
||||
- Synthesize fixed inputs from reviewed constants. Never use recorded Sessions, user material, ambient repositories, or network services.
|
||||
- Run process-level wall-clock and retained-memory samples in fresh children with private `mkdtemp` roots. Pure synchronous folds create a fresh object graph per sample and must not mutate process-global state. Bound every child, await exit, and remove owned roots after failure as well as success.
|
||||
- Pass every budgeted wall-clock endpoint to `recordTimings` from `support/scaling-report.ts` before asserting it, with a commented storage-wait share; omit endpoints that include paced or fixed waits. Pass whole-process or page memory peaks to `recordPeakMemory`. Profiles live in `support/machine-profiles.ts`; each run writes `.dsh-report/report.html`.
|
||||
- Pass every budgeted wall-clock endpoint to `recordTimings` from `support/scaling-report.ts` before asserting it, with a commented storage-wait share; omit endpoints that include paced or fixed waits. Pass whole-process peak RSS to `recordPeakMemory`. Profiles live in `support/machine-profiles.ts`; each run writes `.dsh-report/report.html`.
|
||||
- Record reference-machine expectations separately from the shared CI time scale and variance headroom. Do not apply the time scale to memory or dimensionless ratios.
|
||||
- Report enough raw and aggregate measurements to explain each verdict, including whether a budget uses a median, minimum, absolute value, or ratio. Enforce reviewed source constants; environment variables must not override performance budgets.
|
||||
- Keep scenario-specific support beside its benchmark. Move a helper into `benchmarks/support/` only after at least two benchmark directories require the same behavior.
|
||||
|
||||
@@ -8,7 +8,7 @@ import { expect, it, vi } from 'vitest'
|
||||
import { launchWebScaffold, seedSession, watchConsole, webSnapshotMode } from '../../apps/web/tests/scaffold.ts'
|
||||
import { newEnglishPage } from '../../apps/web/tests/support.ts'
|
||||
import { ciTimeBudget, PERFORMANCE_BUDGET_HEADROOM } from '../support/calibration.ts'
|
||||
import { recordPeakMemory, recordTimings } from '../support/scaling-report.ts'
|
||||
import { recordTimings } from '../support/scaling-report.ts'
|
||||
import { HISTORY_TURNS, SESSION_ID, FIRST, DONE, DELTAS, PACE_MS, syntheticHistory, syntheticReply } from './synthetic-history.ts'
|
||||
|
||||
const SAMPLES = 3
|
||||
@@ -445,6 +445,5 @@ it('opens, pages, navigates and streams into a 240-turn browser history', async
|
||||
}
|
||||
console.log(JSON.stringify({ benchmark: 'long-session-browser/median', turns: HISTORY_TURNS, deltas: DELTAS, paceMs: PACE_MS, samples, aggregate, referenceMs: REFERENCE, expectedOpenCiMs: EXPECTED_OPEN_CI_MS, expectedPageCiMs: EXPECTED_PAGE_CI_MS, expectedTrajectoryCiMs: EXPECTED_TRAJECTORY_CI_MS, budgets }))
|
||||
recordTimings('long-session-browser', IO_SHARE, Object.fromEntries(SCALED_ENDPOINTS.map(key => [key, aggregate[key]!])), budgets)
|
||||
recordPeakMemory('long-session-browser', { pageJsHeapMb: Math.max(...samples.map(sample => sample.heapMb)) })
|
||||
for (const [key, value] of Object.entries(aggregate)) expectEndpointWithinBudget(value, budgets[key]!)
|
||||
})
|
||||
@@ -73,7 +73,7 @@ it.each(['modest', 'tail', 'cheap'] as const)('serves the %s projection list wit
|
||||
|
||||
recordTimings(`session-corpus/projection-list-${workload}`, IO_SHARE, {
|
||||
firstListAndJsonMs: medians.firstListAndJsonMs, repeatListAndJsonMs: medians.repeatListAndJsonMs, worstCallbackDelayMs: medians.worstCallbackDelayMs,
|
||||
}, { firstListAndJsonMs: budgets.listAndJsonMs, worstCallbackDelayMs: budgets.callbackDelayMs })
|
||||
}, { firstListAndJsonMs: budgets.listAndJsonMs, repeatListAndJsonMs: budgets.listAndJsonMs, worstCallbackDelayMs: budgets.callbackDelayMs })
|
||||
for (const report of reports) {
|
||||
expect(report.workload).toBe(workload)
|
||||
expect(report.fixture.sessions).toBe({ modest: 50, tail: 300, cheap: 3_000 }[workload])
|
||||
|
||||
@@ -77,8 +77,11 @@ const AGENT_RETAINED_HEAP_BUDGET_MB = Math.ceil(
|
||||
)
|
||||
|
||||
const WORKER = join(import.meta.dirname, '..', '.dsh-build', 'session-open', 'session-open.worker.js')
|
||||
/** Estimated storage wait: each fresh process reads one Session log that the page cache usually holds. */
|
||||
const IO_SHARE = 0.1
|
||||
/**
|
||||
* Estimated storage wait: endpoints that include open or read access one Session log that the page cache
|
||||
* usually holds; restore and projection start after the file handle closes and process memory only.
|
||||
*/
|
||||
const IO_SHARE = { file: 0.1, memory: 0 } as const
|
||||
|
||||
type WorkerRun = BuiltBenchmarkWorkerRun<SessionOpenWorkerReport>
|
||||
|
||||
@@ -396,10 +399,11 @@ describe('opening a large Session for first open and post-upgrade reopen', () =>
|
||||
projection: PROJECTION_BUDGET_MS,
|
||||
},
|
||||
}))
|
||||
recordTimings(`session-open/${access.accessKind}/phases`, IO_SHARE, {
|
||||
openMs: result.openMs.median, readMs: result.readMs.median,
|
||||
sessionRestoreMs: result.sessionRestoreMs.median, projectionMs: result.projectionMs.median,
|
||||
}, { openMs: access.openBudgetMs, readMs: READ_BUDGET_MS, sessionRestoreMs: SESSION_RESTORE_BUDGET_MS, projectionMs: PROJECTION_BUDGET_MS })
|
||||
recordTimings(`session-open/${access.accessKind}/phases`, IO_SHARE.file,
|
||||
{ openMs: result.openMs.median, readMs: result.readMs.median }, { openMs: access.openBudgetMs, readMs: READ_BUDGET_MS })
|
||||
recordTimings(`session-open/${access.accessKind}/phases`, IO_SHARE.memory,
|
||||
{ sessionRestoreMs: result.sessionRestoreMs.median, projectionMs: result.projectionMs.median },
|
||||
{ sessionRestoreMs: SESSION_RESTORE_BUDGET_MS, projectionMs: PROJECTION_BUDGET_MS })
|
||||
expectOpenWithinBudget(result.openMs.median, access.openBudgetMs)
|
||||
expect(result.readMs.median).toBeLessThanOrEqual(READ_BUDGET_MS)
|
||||
expect(result.sessionRestoreMs.median).toBeLessThanOrEqual(SESSION_RESTORE_BUDGET_MS)
|
||||
@@ -424,7 +428,7 @@ describe('opening a large Session for first open and post-upgrade reopen', () =>
|
||||
result,
|
||||
budgetMs: access.firstHistoryBudgetMs,
|
||||
}))
|
||||
recordTimings(`session-open/${access.accessKind}/first-history`, IO_SHARE, { totalMs: result.totalMs.median }, { totalMs: access.firstHistoryBudgetMs })
|
||||
recordTimings(`session-open/${access.accessKind}/first-history`, IO_SHARE.file, { totalMs: result.totalMs.median }, { totalMs: access.firstHistoryBudgetMs })
|
||||
expect(result.totalMs.median).toBeLessThanOrEqual(access.firstHistoryBudgetMs)
|
||||
})
|
||||
|
||||
@@ -447,7 +451,7 @@ describe('opening a large Session for first open and post-upgrade reopen', () =>
|
||||
budgetMs: access.agentResumeBudgetMs,
|
||||
retainedHeapBudgetMb: AGENT_RETAINED_HEAP_BUDGET_MB,
|
||||
}))
|
||||
recordTimings(`session-open/${access.accessKind}/agent-resume`, IO_SHARE, { totalMs: result.totalMs.median }, { totalMs: access.agentResumeBudgetMs })
|
||||
recordTimings(`session-open/${access.accessKind}/agent-resume`, IO_SHARE.file, { totalMs: result.totalMs.median }, { totalMs: access.agentResumeBudgetMs })
|
||||
expect(result.totalMs.median).toBeLessThanOrEqual(access.agentResumeBudgetMs)
|
||||
expect(result.retainedHeapMb.median).toBeLessThanOrEqual(AGENT_RETAINED_HEAP_BUDGET_MB)
|
||||
})
|
||||
|
||||
@@ -11,13 +11,9 @@
|
||||
* come from the provider's documented latency class for its default system disk. They are estimates.
|
||||
*/
|
||||
|
||||
/** Group shown as one row of scenario choices in the report. */
|
||||
export type MachineProfileGroup = 'laptop' | 'cloud' | 'reference'
|
||||
|
||||
/** One modelled machine with its CPU and default-storage performance. */
|
||||
export interface MachineProfile {
|
||||
readonly id: string
|
||||
readonly group: MachineProfileGroup
|
||||
readonly label: string
|
||||
readonly cpu: string
|
||||
readonly cpuScore: number
|
||||
@@ -29,23 +25,23 @@ export interface MachineProfile {
|
||||
|
||||
/** Profiles in report order. */
|
||||
export const MACHINE_PROFILES: readonly MachineProfile[] = [
|
||||
{ id: 'laptop-2015-win', group: 'laptop', label: '2015 Windows laptop', cpu: 'Intel Core i5-5200U', cpuScore: 1486, storage: 'SATA SSD', ioLatencyUs: 90 },
|
||||
{ id: 'laptop-2015-mac', group: 'laptop', label: '2015 MacBook Pro 13"', cpu: 'Intel Core i5-5257U', cpuScore: 1737, storage: 'PCIe AHCI SSD', ioLatencyUs: 90 },
|
||||
{ id: 'laptop-2019-win', group: 'laptop', label: '2019 Windows laptop', cpu: 'Intel Core i5-8265U', cpuScore: 2003, storage: 'NVMe PCIe 3.0 SSD', ioLatencyUs: 70 },
|
||||
{ id: 'laptop-2023-win', group: 'laptop', label: '2023 Windows laptop', cpu: 'Intel Core i5-1335U', cpuScore: 3241, storage: 'NVMe PCIe 4.0 SSD', ioLatencyUs: 55 },
|
||||
{ id: 'laptop-2023-mac', group: 'laptop', label: '2023 MacBook Air', cpu: 'Apple M2', cpuScore: 4208, storage: 'Apple SSD', ioLatencyUs: 55, cpuModelMatches: ['apple m2'] },
|
||||
{ id: 'laptop-2026-win', group: 'laptop', label: '2026 Windows laptop', cpu: 'Intel Core Ultra 5 225H', cpuScore: 4247, storage: 'NVMe PCIe 4.0 SSD', ioLatencyUs: 50 },
|
||||
{ id: 'laptop-2026-mac', group: 'laptop', label: '2026 MacBook', cpu: 'Apple M5', cpuScore: 6341, storage: 'Apple SSD', ioLatencyUs: 50, cpuModelMatches: ['apple m5'] },
|
||||
{ id: 'aws-t3', group: 'cloud', label: 'AWS t3.medium', cpu: 'Intel Xeon Platinum 8259CL', cpuScore: 1948, storage: 'EBS gp3', ioLatencyUs: 500 },
|
||||
{ id: 'aws-m6i', group: 'cloud', label: 'AWS m6i.large', cpu: 'Intel Xeon Platinum 8375C', cpuScore: 2474, storage: 'EBS gp3', ioLatencyUs: 500 },
|
||||
{ id: 'aws-m7i', group: 'cloud', label: 'AWS m7i.large', cpu: 'Intel Xeon Platinum 8488C', cpuScore: 3113, storage: 'EBS gp3', ioLatencyUs: 500 },
|
||||
{ id: 'aws-m7g', group: 'cloud', label: 'AWS m7g.large', cpu: 'AWS Graviton3 (Neoverse V1)', cpuScore: 1554, storage: 'EBS gp3', ioLatencyUs: 500 },
|
||||
{ id: 'gcp-n2', group: 'cloud', label: 'GCP n2-standard-4', cpu: 'Intel Xeon Platinum 8273CL', cpuScore: 2200, storage: 'pd-balanced', ioLatencyUs: 800 },
|
||||
{ id: 'azure-d4s-v5', group: 'cloud', label: 'Azure D4s v5', cpu: 'Intel Xeon Ice Lake (8375C rating)', cpuScore: 2474, storage: 'Premium SSD P10', ioLatencyUs: 1000 },
|
||||
{ id: 'aliyun-g7', group: 'cloud', label: 'Alibaba Cloud ecs.g7.xlarge', cpu: 'Intel Xeon Ice Lake (8375C rating)', cpuScore: 2474, storage: 'ESSD PL0', ioLatencyUs: 400 },
|
||||
{ id: 'github-ubuntu-24.04', group: 'reference', label: 'GitHub ubuntu-24.04 runner', cpu: 'AMD EPYC 7763', cpuScore: 2517, storage: 'Azure managed disk', ioLatencyUs: 300, cpuModelMatches: ['epyc 7763'] },
|
||||
{ id: 'github-ubuntu-24.04-genoa', group: 'reference', label: 'GitHub ubuntu-24.04 runner (Genoa)', cpu: 'AMD EPYC 9V74', cpuScore: 2888, storage: 'Azure managed disk', ioLatencyUs: 300, cpuModelMatches: ['epyc 9v74'] },
|
||||
{ id: 'reference-m4-pro', group: 'reference', label: 'Calibration reference', cpu: 'Apple M4 Pro', cpuScore: 4547, storage: 'Apple SSD', ioLatencyUs: 55, cpuModelMatches: ['apple m4'] },
|
||||
{ id: 'laptop-2015-win', label: '2015 Windows laptop', cpu: 'Intel Core i5-5200U', cpuScore: 1486, storage: 'SATA SSD', ioLatencyUs: 90 },
|
||||
{ id: 'laptop-2015-mac', label: '2015 MacBook Pro 13"', cpu: 'Intel Core i5-5257U', cpuScore: 1737, storage: 'PCIe AHCI SSD', ioLatencyUs: 90 },
|
||||
{ id: 'laptop-2019-win', label: '2019 Windows laptop', cpu: 'Intel Core i5-8265U', cpuScore: 2003, storage: 'NVMe PCIe 3.0 SSD', ioLatencyUs: 70 },
|
||||
{ id: 'laptop-2023-win', label: '2023 Windows laptop', cpu: 'Intel Core i5-1335U', cpuScore: 3241, storage: 'NVMe PCIe 4.0 SSD', ioLatencyUs: 55 },
|
||||
{ id: 'laptop-2023-mac', label: '2023 MacBook Air', cpu: 'Apple M2', cpuScore: 4208, storage: 'Apple SSD', ioLatencyUs: 55, cpuModelMatches: ['apple m2'] },
|
||||
{ id: 'laptop-2026-win', label: '2026 Windows laptop', cpu: 'Intel Core Ultra 5 225H', cpuScore: 4247, storage: 'NVMe PCIe 4.0 SSD', ioLatencyUs: 50 },
|
||||
{ id: 'laptop-2026-mac', label: '2026 MacBook', cpu: 'Apple M5', cpuScore: 6341, storage: 'Apple SSD', ioLatencyUs: 50, cpuModelMatches: ['apple m5'] },
|
||||
{ id: 'aws-t3', label: 'AWS t3.medium', cpu: 'Intel Xeon Platinum 8259CL', cpuScore: 1948, storage: 'EBS gp3', ioLatencyUs: 500 },
|
||||
{ id: 'aws-m6i', label: 'AWS m6i.large', cpu: 'Intel Xeon Platinum 8375C', cpuScore: 2474, storage: 'EBS gp3', ioLatencyUs: 500 },
|
||||
{ id: 'aws-m7i', label: 'AWS m7i.large', cpu: 'Intel Xeon Platinum 8488C', cpuScore: 3113, storage: 'EBS gp3', ioLatencyUs: 500 },
|
||||
{ id: 'aws-m7g', label: 'AWS m7g.large', cpu: 'AWS Graviton3 (Neoverse V1)', cpuScore: 1554, storage: 'EBS gp3', ioLatencyUs: 500 },
|
||||
{ id: 'gcp-n2', label: 'GCP n2-standard-4', cpu: 'Intel Xeon Platinum 8273CL', cpuScore: 2200, storage: 'pd-balanced', ioLatencyUs: 800 },
|
||||
{ id: 'azure-d4s-v5', label: 'Azure D4s v5', cpu: 'Intel Xeon Ice Lake (8375C rating)', cpuScore: 2474, storage: 'Premium SSD P10', ioLatencyUs: 1000 },
|
||||
{ id: 'aliyun-g7', label: 'Alibaba Cloud ecs.g7.xlarge', cpu: 'Intel Xeon Ice Lake (8375C rating)', cpuScore: 2474, storage: 'ESSD PL0', ioLatencyUs: 400 },
|
||||
{ id: 'github-ubuntu-24.04', label: 'GitHub ubuntu-24.04 runner', cpu: 'AMD EPYC 7763', cpuScore: 2517, storage: 'Azure managed disk', ioLatencyUs: 300, cpuModelMatches: ['epyc 7763'] },
|
||||
{ id: 'github-ubuntu-24.04-genoa', label: 'GitHub ubuntu-24.04 runner (Genoa)', cpu: 'AMD EPYC 9V74', cpuScore: 2888, storage: 'Azure managed disk', ioLatencyUs: 300, cpuModelMatches: ['epyc 9v74'] },
|
||||
{ id: 'reference-m4-pro', label: 'Calibration reference', cpu: 'Apple M4 Pro', cpuScore: 4547, storage: 'Apple SSD', ioLatencyUs: 55, cpuModelMatches: ['apple m4'] },
|
||||
]
|
||||
|
||||
/**
|
||||
|
||||
@@ -8,7 +8,8 @@
|
||||
*
|
||||
* A scaled endpoint is `ms × ((1 − io) × cpuScore(measured) / cpuScore(target) + io × ioLatency(target) /
|
||||
* ioLatency(measured))`, where `io` is the declared share of the endpoint's wall time spent waiting on
|
||||
* storage. The report lets readers change that share. Memory and dimensionless ratios are not scaled.
|
||||
* storage. The report lets readers change that share per endpoint. Memory and dimensionless ratios are not
|
||||
* scaled.
|
||||
*
|
||||
* Bench files also pass peak memory to {@link recordPeakMemory}. The report compares those unscaled peaks
|
||||
* with 4 GB and 8 GB of available memory; it does not run cases under an operating-system memory limit.
|
||||
@@ -167,16 +168,17 @@ const times = data.records.filter(r => r.kind === 'time');
|
||||
const memories = data.records.filter(r => r.kind === 'memory');
|
||||
const AVAILABLE_MB = [4096, 8192];
|
||||
const keyOf = r => r.benchmark + ' · ' + r.metric;
|
||||
const ioShares = Object.fromEntries(times.map(r => [r.benchmark, r.ioShare]));
|
||||
const ioShares = times.map(r => r.ioShare);
|
||||
let selected = 0;
|
||||
const fmt = ms => ms >= 10000 ? (ms / 1000).toFixed(0) + ' s' : ms >= 1000 ? (ms / 1000).toFixed(1) + ' s' : ms >= 10 ? ms.toFixed(0) + ' ms' : ms.toFixed(1) + ' ms';
|
||||
const fmtMb = mb => mb >= 1024 ? (mb / 1024).toFixed(1) + ' GB' : mb.toFixed(0) + ' MB';
|
||||
const band = ms => ms < 100 ? 0 : ms < 1000 ? 1 : ms < 10000 ? 2 : 3;
|
||||
const memoryBand = share => share < 0.5 ? 0 : share < 0.75 ? 1 : share < 1 ? 2 : 3;
|
||||
const colors = ['#1a7f37', '#9a6700', '#cf222e', '#82071e'];
|
||||
function scale(record, target) {
|
||||
function scale(index, target) {
|
||||
const record = times[index];
|
||||
const base = profiles.find(p => p.id === $('measured').value);
|
||||
const io = ioShares[record.benchmark];
|
||||
const io = ioShares[index];
|
||||
return record.ms * ((1 - io) * base.cpuScore / target.cpuScore + io * target.ioLatencyUs / base.ioLatencyUs);
|
||||
}
|
||||
function esc(text) { const node = document.createElement('span'); node.textContent = text; return node.innerHTML; }
|
||||
@@ -209,19 +211,18 @@ function table() {
|
||||
let html = '<thead><tr><th>Endpoint</th><th>Measured</th><th>CI budget</th>' + profiles.map(p => '<th title="' + esc(p.cpu + ' · ' + p.storage) + '">' + esc(p.label) + '</th>').join('') + '</tr></thead><tbody>';
|
||||
times.forEach((r, i) => {
|
||||
html += '<tr data-index="' + i + '"' + (i === selected ? ' class="selected"' : '') + '><td>' + esc(keyOf(r)) + '</td><td>' + fmt(r.ms) + '</td><td>' + (r.budgetMs === undefined ? '' : fmt(r.budgetMs)) + '</td>';
|
||||
for (const p of profiles) { const ms = scale(r, p); html += '<td class="b' + band(ms) + '">' + fmt(ms) + '</td>'; }
|
||||
for (const p of profiles) { const ms = scale(i, p); html += '<td class="b' + band(ms) + '">' + fmt(ms) + '</td>'; }
|
||||
html += '</tr>';
|
||||
});
|
||||
$('table').innerHTML = html + '</tbody>';
|
||||
}
|
||||
function render() {
|
||||
if (times.length === 0) return;
|
||||
const record = times[selected];
|
||||
$('endpoint').value = String(selected);
|
||||
$('io').value = String(ioShares[record.benchmark]);
|
||||
$('io-value').textContent = Math.round(ioShares[record.benchmark] * 100) + '%';
|
||||
$('io').value = String(ioShares[selected]);
|
||||
$('io-value').textContent = Math.round(ioShares[selected] * 100) + '%';
|
||||
bars($('chart'), profiles.map(p => {
|
||||
const ms = scale(record, p);
|
||||
const ms = scale(selected, p);
|
||||
return { label: p.label + ' · ' + p.storage, title: p.cpu, value: ms, band: band(ms), current: p.id === $('measured').value };
|
||||
}), [1, 10, 100, 1000, 10000, 100000, 1000000].map(value => ({ value, label: fmt(value), marked: value === 100 || value === 1000 })), fmt);
|
||||
table();
|
||||
@@ -240,7 +241,7 @@ function renderMemory() {
|
||||
}
|
||||
$('measured').onchange = render;
|
||||
$('endpoint').onchange = () => { selected = Number($('endpoint').value); render(); };
|
||||
$('io').oninput = () => { ioShares[times[selected].benchmark] = Number($('io').value); render(); };
|
||||
$('io').oninput = () => { ioShares[selected] = Number($('io').value); render(); };
|
||||
$('table').onclick = event => { const row = event.target.closest('tr[data-index]'); if (row) { selected = Number(row.dataset.index); render(); } };
|
||||
render();
|
||||
renderMemory();
|
||||
|
||||
Reference in new issue
Block a user