ci: adapt concurrency to available runner CPUs (#4518)

* ci: adapt worker pools to available runner CPUs

* ci: preserve local snapshot worker defaults

* test: isolate CI worker overrides in gate fixtures

* docs: clarify shared Windows worker limits
This commit is contained in:
Turtle authored and GitHub committed 2026-09-18 13:17:34 +08:00
1 parent de742f7f45
commit 9d5fc30632
9 files changed
+139 -39

No files matched your search

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
2026-07-26-ci-failover-runbook.md: 10123fe1999c0ad03f977e1cc7c69d788a679c2b
2026-07-26-ci-failover-runbook.zh.md: cff99e6f644bf945f8366c2a723b32222ab902b4
2026-07-26-ci-failover-runbook.md: c2fd2fa6cf673ec4d202702c16a9a85a3a609b8d
2026-07-26-ci-failover-runbook.zh.md: 7fa5887bb552afd98cebd7502202fb5e9f37b1ee
@@ -32,7 +32,7 @@ The two switches are independent: flip only the one whose platform is degraded.
1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER_LINUX` (Linux pool outage) or `DSH_CI_FAILOVER_WINDOWS` (Windows pool outage), value `selfhosted`.
2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued.
3. That is the entire switch. Under the `selfhosted` Linux failover value the workflow also drops `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 for the shared VM and skips the hosted-path pnpm cache restores because the VM's persistent store serves warm installs. Coverage uses the same four single-worker instrumented partitions and two exempt workers on both Linux pools. The Windows switch has no concurrency or cache branches; it only retargets the native Windows jobs' pool.
3. That is the entire switch. Under the `selfhosted` Linux failover value the workflow also drops `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 for the shared VM and skips the hosted-path pnpm cache restores because the VM's persistent store serves warm installs. Self-hosted coverage uses four single-worker instrumented partitions and two exempt workers; isolated hosted runners derive worker defaults from available CPUs. The Windows switch selects the native Windows pool and retains explicit worker limits on shared self-hosted runners.
**Dependabot exception.** Both switches' `selfhosted` legs deliberately exclude `dependabot[bot]`: under self-hosted failover, Dependabot PRs stay queued for the hosted pool rather than executing dependency-supplied code on the persistent VMs. A Dependabot PR that remains queued during an outage is expected behavior, not a failed switch; it completes when the hosted pool recovers. The `blacksmith` value's branches carry no such exclusion, because Blacksmith runners are ephemeral (see the [blacksmith failover leg note](2026-09-09-blacksmith-failover-leg.md)).
@@ -32,7 +32,7 @@ Status: implemented
1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**:名称 `DSH_CI_FAILOVER_LINUX`(Linux 池故障)或 `DSH_CI_FAILOVER_WINDOWS`(Windows 池故障),值 `selfhosted`。
2. 重新触发必需作业,使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向,也无法原地 re-run,因此对于本手册所述的无限排队故障,应取消卡住的运行并 re-run all jobs,或推送一个新提交;“Re-run failed jobs”只有在作业真正失败(而非仍在排队)时才有用。
3. 切换到此完成。在 `selfhosted` 的 Linux 故障切换取值下,工作流还会把 `DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12,以限制共享虚拟机上的争抢,并跳过托管路径的 pnpm 缓存恢复,因为虚拟机的持久 store 会直接提供热安装。覆盖率在两个 Linux 池上都使用 4 个单 worker 插桩分区与 2 个豁免 worker。Windows 开关没有并发或缓存分支;它只重定向原生 Windows 作业的运行器池。
3. 切换到此完成。在 `selfhosted` 的 Linux 故障切换取值下,工作流还会把 `DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12,以限制共享虚拟机上的争抢,并跳过托管路径的 pnpm 缓存恢复,因为虚拟机的持久 store 会直接提供热安装。自托管覆盖率使用 4 个单 worker 插桩分区与 2 个豁免 worker;独占的托管运行器根据可用 CPU 计算默认 worker 数。Windows 开关选择原生 Windows 运行器池,并在共享的自托管运行器上保留显式 worker 限制。
**Dependabot 例外。**两个开关的 `selfhosted` 腿都刻意排除 `dependabot[bot]`:自托管故障切换期间,Dependabot 拉取请求继续在托管池排队,而不是把依赖项提供的代码放到持久化虚拟机上执行。故障期间 Dependabot PR 持续排队是预期行为而非切换失败;托管池恢复后它会自行完成。`blacksmith` 取值下的分支不带此类排除,因为 Blacksmith 运行器是临时的(见 [blacksmith 故障切换支路笔记](2026-09-09-blacksmith-failover-leg.zh.md))。
+18 -16
View File
@@ -49,7 +49,8 @@ jobs:
|| 'dsh-ubuntu-24-04-16core' }}
name: node 24 / static
env:
DSH_GATE_CONCURRENCY: '8'
# Shared hosts expose host CPUs, not an isolated allocation.
DSH_GATE_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '8' || '' }}
# Stop the aggregate at the first blocking gate failure so a red run
# does not keep burning enterprise runner time on the remaining gates.
DSH_GATE_FAIL_FAST: '1'
@@ -116,11 +117,10 @@ jobs:
|| 'dsh-ubuntu-24-04-16core' }}
name: node 24 / coverage
env:
# Partitioning replaces the instrumented share; this budget gives the
# exempt-heavy gate two workers on both hosted and failover runners.
DSH_COVERAGE_MAX_WORKERS: '6'
DSH_COVERAGE_PARTITIONS: '4'
DSH_GATE_CONCURRENCY: '3'
# Shared hosts expose host CPUs, not an isolated allocation.
DSH_COVERAGE_MAX_WORKERS: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '6' || '' }}
DSH_COVERAGE_PARTITIONS: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '4' || '' }}
DSH_GATE_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '3' || '' }}
# Managed-scope teardown cases run past the default per-test budget when
# this lane's partitions, workers, and sibling gates share one host: the
# disposal cases in subprocess-local and bash-sandbox exceeded 5000ms on
@@ -246,17 +246,17 @@ jobs:
|| 'dsh-ubuntu-24-04-16core' }}
name: node 24 / snapshots and artifacts
env:
DSH_GATE_CONCURRENCY: '10'
# Shared hosts expose host CPUs, not an isolated allocation.
DSH_GATE_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '10' || '' }}
DSH_OXLINT_THREADS: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '8' || '' }}
DSH_PUBLINT_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '8' || '' }}
DSH_WEB_SNAPSHOT_WORKERS: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '6' || '' }}
DSH_SNAPSHOT_MAX_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '12' || '' }}
DSH_NODE_COMPAT_SKIP_TYPECHECK: '1'
DSH_OXLINT_THREADS: '8'
DSH_PUBLINT_CONCURRENCY: '8'
DSH_WEB_SNAPSHOT_WORKERS: '6'
# A failing gate aborts its running siblings: a failing build stops the
# independent Node compatibility smoke, and a failing reader (e.g.
# publint) stops the remaining artifact consumers.
DSH_GATE_FAIL_FAST: '1'
# Failover halves snapshot concurrency for the shared 64-core VM.
DSH_SNAPSHOT_MAX_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '12' || '32' }}
steps:
# Runner cleanup owns tool caches and fixtures, including cancelled runs.
- name: Use runner-owned temporary storage
@@ -555,10 +555,11 @@ jobs:
name: windows node 24 / coverage
timeout-minutes: 120
env:
DSH_COVERAGE_MAX_WORKERS: '6'
DSH_COVERAGE_PARTITIONS: '4'
# Shared hosts expose host CPUs, not an isolated allocation.
DSH_COVERAGE_MAX_WORKERS: ${{ vars.DSH_CI_FAILOVER_WINDOWS == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '6' || '' }}
DSH_COVERAGE_PARTITIONS: ${{ vars.DSH_CI_FAILOVER_WINDOWS == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '4' || '' }}
DSH_GATE_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER_WINDOWS == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '3' || '' }}
DSH_COVERAGE_TEST_TIMEOUT_MS: '90000'
DSH_GATE_CONCURRENCY: '3'
# A gate failure aborts the sibling gate instead of waiting out its
# multi-minute instrumented run.
DSH_GATE_FAIL_FAST: '1'
@@ -682,7 +683,8 @@ jobs:
name: windows node 24 / observational
timeout-minutes: 60
env:
DSH_PUBLINT_CONCURRENCY: '8'
# Shared hosts expose host CPUs, not an isolated allocation.
DSH_PUBLINT_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER_WINDOWS == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '8' || '' }}
steps:
- uses: actions/checkout@v6
with:
+2 -4
View File
@@ -232,9 +232,8 @@ describe('CI workflow', () => {
expect(install!.run).not.toContain('$cloneFlag')
}
// windows-coverage uses the lower 4-partition profile.
expect(windowsCoverage.name).toBe('windows node 24 / coverage')
expect(windowsCoverage.env).toMatchObject({ DSH_COVERAGE_PARTITIONS: '4' })
expect(windowsCoverage.env).toMatchObject({ DSH_COVERAGE_PARTITIONS: "${{ vars.DSH_CI_FAILOVER_WINDOWS == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '4' || '' }}" })
const coverageSteps = windowsCoverage.steps as unknown[]
const coverageCommands = coverageSteps.filter((step): step is Record<string, unknown> & { run: string } => (
isRecord(step) && typeof step.run === 'string'
@@ -386,8 +385,7 @@ describe('CI workflow', () => {
// The observational lane stays complete: it is continue-on-error by design
// and exists to collect as much Windows-native evidence per run as
// possible, so the first failure must not truncate the rest.
expect(windowsObservational.env).toBeDefined()
expect(windowsObservational.env).not.toMatchObject({ DSH_GATE_FAIL_FAST: '1' })
expect(windowsObservational.env ?? {}).not.toMatchObject({ DSH_GATE_FAIL_FAST: '1' })
})
it('gates standalone keyless blacksmith jobs and benchmark tiers on the failover variables', () => {
+61 -3
View File
@@ -1,7 +1,8 @@
import { readFileSync } from 'node:fs'
import { describe, expect, it, vi, type MockInstance } from 'vitest'
import { afterEach, beforeEach, describe, expect, it, vi, type MockInstance } from 'vitest'
import {
cliGateOptions,
ciWorkerEnvironment,
collectDescendants,
defaultConcurrency,
formatGateResultReason,
@@ -14,6 +15,10 @@ import {
type GateResult,
} from './run-gates.ts'
// Graph fixtures select their own browser pool instead of inheriting the CI host's pool.
beforeEach(() => vi.stubEnv('DSH_WEB_SNAPSHOT_WORKERS', undefined))
afterEach(() => vi.unstubAllEnvs())
/**
* Capture output a gate streams through runGate's streamOutput path.
* @returns the accumulated chunks and the stdout spy to restore in finally.
@@ -135,6 +140,59 @@ function withEnv<T>(name: string, value: string | undefined, action: () => T): T
}
}
describe('CI worker allocation', () => {
it.each([1, 2, 4, 8, 16, 64])('shares a %i CPU coverage budget without multiplying pools', (cpus) => {
const env = ciWorkerEnvironment('ci-coverage', {}, cpus)
const exempt = Math.max(1, Math.floor(cpus / 3))
const instrumented = Number(env.DSH_COVERAGE_PARTITIONS ?? 1)
expect(Number(env.DSH_COVERAGE_MAX_WORKERS)).toBe(cpus)
expect(instrumented + exempt).toBe(Math.max(2, cpus))
expect(defaultConcurrency('ci-coverage', 3, cpus).workers).toBe(Math.min(3, cpus))
if (cpus <= 2) expect(env.DSH_COVERAGE_PARTITIONS).toBeUndefined()
})
it('bounds overlapping readers and lets the isolated browser pool use the runner', () => {
const env = ciWorkerEnvironment('ci-consumers', {}, 16)
expect(env).toMatchObject({
DSH_OXLINT_THREADS: '8',
DSH_PUBLINT_CONCURRENCY: '8',
DSH_SNAPSHOT_MAX_WORKERS: '1',
DSH_SNAPSHOT_MAX_CONCURRENCY: '8',
DSH_WEB_SNAPSHOT_WORKERS: '16',
})
})
it('preserves serial reference overrides on large hosts', () => {
const inherited = {
DSH_GATE_CONCURRENCY: '1',
DSH_COVERAGE_MAX_WORKERS: '1',
DSH_OXLINT_THREADS: '1',
DSH_PUBLINT_CONCURRENCY: '1',
DSH_SNAPSHOT_MAX_CONCURRENCY: '1',
DSH_WEB_SNAPSHOT_WORKERS: '1',
}
const additions = ciWorkerEnvironment('ci-primary', inherited, 64)
expect(additions).toEqual({ DSH_SNAPSHOT_MAX_WORKERS: '1' })
expect(inherited.DSH_COVERAGE_MAX_WORKERS).toBe('1')
})
it('honors an explicit coverage budget and partition override', () => {
expect(ciWorkerEnvironment('ci-coverage', { DSH_COVERAGE_MAX_WORKERS: '6' }, 16))
.toHaveProperty('DSH_COVERAGE_PARTITIONS', '4')
expect(ciWorkerEnvironment('ci-coverage', { DSH_COVERAGE_PARTITIONS: '3' }, 16))
.not.toHaveProperty('DSH_COVERAGE_PARTITIONS')
})
it.each(['0', '-1', 'NaN', '2.5'])('rejects invalid worker budget %s', (raw) => {
expect(() => ciWorkerEnvironment('ci-coverage', { DSH_COVERAGE_MAX_WORKERS: raw }, 16))
.toThrow('DSH_COVERAGE_MAX_WORKERS must be a positive integer')
})
it('keeps local documentation defaults unchanged', () => {
expect(ciWorkerEnvironment('doc-sync', {}, 64)).toEqual({})
})
})
describe('gate graph validation', () => {
it.each([
'ci-primary',
@@ -601,8 +659,8 @@ describe('Node 24 lane ownership', () => {
const subject = withPnpmEntrypoint(() => gatesForMode('ci-consumers'))
expect(defaultConcurrency('ci-consumers', subject.length, 4)).toEqual({
workers: 11,
source: 'ci-consumers gate count',
workers: 4,
source: '4 available CPU(s)',
})
expect(subject.map(item => item.id)).toEqual([
'build',
+43 -7
View File
@@ -102,6 +102,9 @@ if (import.meta.main) {
async function main(args: string[]): Promise<number> {
const mode = parseMode(args[0])
const workerEnv = ciWorkerEnvironment(mode, process.env)
Object.assign(process.env, workerEnv)
if (Object.keys(workerEnv).length > 0) console.log(`run-gates: worker settings ${JSON.stringify(workerEnv)}`)
const gates = gatesForMode(mode)
const concurrencyDefault = defaultConcurrency(mode, gates.length)
const concurrencyOverride = process.env.DSH_GATE_CONCURRENCY
@@ -170,7 +173,6 @@ export function defaultConcurrency(
total: number,
available = availableParallelism(),
): ConcurrencyDefault {
if (selectedMode === 'ci-consumers') return { workers: total, source: 'ci-consumers gate count' }
// Local modes cap workers: several doc gates each build a full ts.Program,
// so an uncapped default on a large host trades wall clock for memory blowups.
const localCap = selectedMode === 'check-all'
@@ -186,6 +188,41 @@ export function defaultConcurrency(
}
}
/**
* Fill CI worker defaults from the CPU allocation without replacing explicit overrides.
* Coverage shares one budget; artifact readers share CPUs with sibling commands.
* @param mode - aggregate selected by the caller.
* @param env - inherited worker overrides, including serial reference settings.
* @param available - CPUs available to this runner process.
* @returns environment additions for this aggregate and its children.
*/
export function ciWorkerEnvironment(
mode: Mode,
env: NodeJS.ProcessEnv,
available = availableParallelism(),
): Record<string, string> {
if (!mode.startsWith('ci-')) return {}
const additions: Record<string, string> = {}
const setDefault = (name: string, value: number): void => {
if (env[name] === undefined || env[name] === '') additions[name] = String(value)
}
const shared = Math.max(1, Math.floor(available / 2))
setDefault('DSH_OXLINT_THREADS', shared)
setDefault('DSH_PUBLINT_CONCURRENCY', shared)
setDefault('DSH_SNAPSHOT_MAX_WORKERS', 1)
setDefault('DSH_SNAPSHOT_MAX_CONCURRENCY', shared)
setDefault('DSH_WEB_SNAPSHOT_WORKERS', env.DSH_GATE_CONCURRENCY === '1' ? 1 : available)
setDefault('DSH_COVERAGE_MAX_WORKERS', available)
const coverageBudget = env.DSH_COVERAGE_MAX_WORKERS || String(available)
const total = Number(coverageBudget)
if (!Number.isSafeInteger(total) || total < 1) {
throw new Error(`run-gates: DSH_COVERAGE_MAX_WORKERS must be a positive integer, got ${JSON.stringify(coverageBudget)}.`)
}
const instrumented = Math.max(1, total - Math.max(1, Math.floor(total / 3)))
if (instrumented > 1) setDefault(COVERAGE_PARTITIONS_ENV, instrumented)
return additions
}
function concurrencyFromEnv(name: string, fallback: number): number {
const raw = process.env[name]
if (raw === undefined || raw === '') return fallback
@@ -498,8 +535,8 @@ function webSnapshotGate(needs: string[], after?: string[]): Gate {
const workerRaw = process.env.DSH_WEB_SNAPSHOT_WORKERS
if (workerRaw !== undefined && workerRaw !== '') {
const workers = Number.parseInt(workerRaw, 10)
if (!Number.isSafeInteger(workers) || workers < 2 || String(workers) !== workerRaw) {
throw new Error(`run-gates: DSH_WEB_SNAPSHOT_WORKERS must be an integer greater than 1, got ${JSON.stringify(workerRaw)}.`)
if (!Number.isSafeInteger(workers) || workers < 1 || String(workers) !== workerRaw) {
throw new Error(`run-gates: DSH_WEB_SNAPSHOT_WORKERS must be a positive integer, got ${JSON.stringify(workerRaw)}.`)
}
return pnpmScript('web-snapshot', 'test:web:ci', {
label: 'web browser snapshot',
@@ -593,10 +630,9 @@ function lintGate(options: { needs?: string[] } = {}): Gate {
// under v8 instrumentation while contributing nothing the thresholds need
// (membership rules in scripts/coverage-exempt.ts).
//
// DSH_COVERAGE_MAX_WORKERS is the ordinary lane's worker budget, so the two
// parallel gates split it instead of each claiming it whole. When
// DSH_COVERAGE_PARTITIONS is set, its single-worker processes replace the
// instrumented share while this budget still sizes the exempt gate. The exempt
// DSH_COVERAGE_MAX_WORKERS is shared between the two parallel gates. CI
// defaults its partition count to the instrumented share; an explicit
// DSH_COVERAGE_PARTITIONS overrides that share independently. The exempt
// gate's wall clock is dominated by its longest single file, so it takes the
// small share. A budget of 1 gives each gate 1 worker; lanes that need a strict
// total of one (the serial reference jobs) also set DSH_GATE_CONCURRENCY=1,
+2 -2
View File
@@ -7,8 +7,8 @@ const serialFiles = [
]
const workerRaw = process.env.DSH_WEB_SNAPSHOT_WORKERS
const workers = Number.parseInt(workerRaw ?? '', 10)
if (!Number.isSafeInteger(workers) || workers < 2 || String(workers) !== workerRaw) {
throw new Error(`DSH_WEB_SNAPSHOT_WORKERS must be an integer greater than 1, got ${JSON.stringify(workerRaw)}.`)
if (!Number.isSafeInteger(workers) || workers < 1 || String(workers) !== workerRaw) {
throw new Error(`DSH_WEB_SNAPSHOT_WORKERS must be a positive integer, got ${JSON.stringify(workerRaw)}.`)
}
const invocation = pnpmInvocation(['exec', 'vitest', 'run', '--config', 'vitest.web.config.ts'])
let serialStatus = 0
+9 -3
View File
@@ -21,6 +21,10 @@ const snapshotMaxConcurrency = positiveIntFromEnv(
Math.min(DEFAULT_SNAPSHOT_MAX_CONCURRENCY, availableParallelism()),
)
const snapshotMaxWorkers = process.env.DSH_SNAPSHOT_MAX_WORKERS
? positiveIntFromEnv('DSH_SNAPSHOT_MAX_WORKERS', availableParallelism())
: undefined
// Replay is the keyless default: boot real subprocess paths from recorded model responses and diff
// assembled requests, normalized protocol or transcript output, and persisted-log expected outputs.
// `record` calls the real API and updates fixtures and expected outputs; `refresh` replays committed scripts
@@ -54,14 +58,16 @@ export default defineConfig({
// Replay never writes committed outputs and every scenario owns its
// mutable runtime state (the subprocess suites use a unique temp dir and
// fixture set per scenario), so replay runs the snapshot files in
// parallel and bounds in-file concurrency with the environment knob
// (value 1 restores fully serial replay on constrained machines). Record
// parallel with separate file-worker and in-file concurrency budgets.
// Without a file-worker override, in-file concurrency 1 is fully serial. Record
// and refresh stay serial: record spends real API quota per scenario, and
// refresh write-back harvests volatile values from fixtures already on
// disk, so concurrent writers would corrupt expected outputs.
testTimeout: 120_000,
hookTimeout: 30_000,
fileParallelism: (process.env.DSH_SNAPSHOT || 'replay') === 'replay' && snapshotMaxConcurrency > 1,
fileParallelism: (process.env.DSH_SNAPSHOT || 'replay') === 'replay'
&& (snapshotMaxWorkers ?? snapshotMaxConcurrency) > 1,
maxConcurrency: snapshotMaxConcurrency,
...snapshotMaxWorkers === undefined ? {} : { maxWorkers: snapshotMaxWorkers },
},
})