diff --git a/docs/kuna-provider.md b/docs/kuna-provider.md index 479d1dfbd2..9c4b885cb1 100644 --- a/docs/kuna-provider.md +++ b/docs/kuna-provider.md @@ -68,7 +68,7 @@ The background export belongs to the session. `close_binary`, or the end of a CL The assembly label index is scanned once per export reader. Cancelling a request stops its wait without interrupting other readers; closing the session aborts and drains the shared scan before releasing its files. -Regex procedure and string search uses ECMAScript Unicode matching in a cancellable worker with a matching deadline and a worker old-generation heap. Literal matching remains the default. Cancellation or a matching deadline leaves the session available. The deadline is `REA_REGEX_MATCHING_DEADLINE_MS`, a positive decimal integer from 1 to 2147483647 ms. Unset uses 5000 ms. A larger value is refused because Node schedules a delay past that signed 32-bit bound as 1 ms. Invalid values are configuration errors and do not fall back to the default. The setting is read from the environment that starts REA, so a change needs a restart. A deadline failure names the effective milliseconds, whether they came from the default or the setting, the default, and the maximum. The worker old-generation heap does not raise it. The worker requests 128 MiB. A positive Node.js `--max-old-space-size` on the process that starts REA, including `NODE_OPTIONS`, overrides that request and sets the effective heap. `--max-old-space-size-percentage` takes precedence over that flag in either order. REA does not turn the percentage into a MiB, so the effective heap stays unestablished instead of the 128 MiB request or the absolute flag. The reported `NODE_OPTIONS` value is the process initial environment when that block can be read. A later edit to the environment is not the value Node applied, and an unreadable initial value leaves the effective heap unestablished. The launch Node options are the initial command line on Linux when that buffer still has a program entry, not a later edit to `process.execArgv`. A preload can clear that array after V8 has applied the flags. Setting `process.title` or `--title` overwrites the command line, so a buffer without a program entry leaves the effective heap unestablished. `--env-file` or `--env-file-if-exists` can supply `NODE_OPTIONS` when the initial environment has none. That file is not part of the initial environment block, so the effective heap stays unestablished instead of the 128 MiB request. An initial `NODE_OPTIONS` is still used, because Node does not let the file replace it. A positive `--max-heap-size`, including the underscore form and launch `NODE_OPTIONS` on Node versions that allow it, sets the total heap rather than the old generation and can be lower than the 128 MiB request. The effective old generation stays unestablished while that value is in effect. A zero or empty value clears it. `--experimental-config-file` or `--experimental-default-config-file` can set `nodeOptions`, including a heap percentage, from a file that is not read back. The effective heap stays unestablished when either flag is present, including when an absolute `--max-old-space-size` is also on the command line or in the recovered `NODE_OPTIONS`. Node accepts the config path as `=` or the next argument. +Regex procedure and string search uses ECMAScript Unicode matching in a cancellable worker with a matching deadline and a worker old-generation heap. Literal matching remains the default. Cancellation or a matching deadline leaves the session available. The deadline is `REA_REGEX_MATCHING_DEADLINE_MS`, a positive decimal integer from 1 to 2147483647 ms. Unset uses 5000 ms. A larger value is refused because Node schedules a delay past that signed 32-bit bound as 1 ms. Invalid values are configuration errors and do not fall back to the default. The setting is read from the environment that starts REA, so a change needs a restart. A deadline failure names the effective milliseconds, whether they came from the default or the setting, the default, and the maximum. The worker old-generation heap does not raise it. The worker heap request is `REA_REGEX_WORKER_HEAP_REQUEST_MIB`, a decimal integer from 7 to 16384 MiB. Unset uses 128 MiB. A request below 7 MiB aborts the worker during startup, and a larger value is refused. Invalid values are configuration errors and do not fall back to 128. The setting is read from the environment that starts REA, so a change needs a restart. A heap failure names the effective request, whether it came from the default or the setting, the default, the floor, and the maximum. A positive Node.js `--max-old-space-size` on the process that starts REA, including `NODE_OPTIONS`, overrides that request and sets the effective heap. `--max-old-space-size-percentage` takes precedence over that flag in either order. REA does not turn the percentage into a MiB, so the effective heap stays unestablished instead of the worker heap request or the absolute flag. The reported `NODE_OPTIONS` value is the process initial environment when that block can be read. A later edit to the environment is not the value Node applied, and an unreadable initial value leaves the effective heap unestablished. The launch Node options are the initial command line on Linux when that buffer still has a program entry, not a later edit to `process.execArgv`. A preload can clear that array after V8 has applied the flags. Setting `process.title` or `--title` overwrites the command line, so a buffer without a program entry leaves the effective heap unestablished. `--env-file` or `--env-file-if-exists` can supply `NODE_OPTIONS` when the initial environment has none. That file is not part of the initial environment block, so the effective heap stays unestablished instead of the worker heap request. An initial `NODE_OPTIONS` is still used, because Node does not let the file replace it. A positive `--max-heap-size`, including the underscore form and launch `NODE_OPTIONS` on Node versions that allow it, sets the total heap rather than the old generation and can be lower than the worker heap request. The effective old generation stays unestablished while that value is in effect. A zero or empty value clears it. `--experimental-config-file` or `--experimental-default-config-file` can set `nodeOptions`, including a heap percentage, from a file that is not read back. The effective heap stays unestablished when either flag is present, including when an absolute `--max-old-space-size` is also on the command line or in the recovered `NODE_OPTIONS`. Node accepts the config path as `=` or the next argument. Single-function subprocess output and each cached source facet share a 16 MiB buffered-text capacity. Cached spans are checked before allocation; larger text remains in the local export and the error identifies the file and range. `analyze_function` initially returns the complete dossier, which can still be large; later retained views do not remove that first transfer. Prefer the individual text tools for a focused question. diff --git a/docs/native-investigation.md b/docs/native-investigation.md index 7178c37368..76f2ea7701 100644 --- a/docs/native-investigation.md +++ b/docs/native-investigation.md @@ -212,12 +212,18 @@ from the environment that starts REA, so a change needs a restart. A deadline failure names the effective milliseconds, whether they came from the default or the setting, the default, and the maximum. The worker old-generation heap does not raise it. Kuna regex search uses the same deadline. The worker -requests 128 -MiB. A positive Node.js `--max-old-space-size` on the process that starts +heap request is `REA_REGEX_WORKER_HEAP_REQUEST_MIB`, a decimal integer from +7 to 16384 MiB. Unset uses 128 MiB. A request below 7 MiB aborts the worker +during startup, and a larger value is refused. Invalid values are +configuration errors and do not fall back to 128. The setting is read from +the environment that starts REA, so a change needs a restart. A heap failure +names the effective request, whether it came from the default or the setting, +the default, the floor, and the maximum. A positive Node.js +`--max-old-space-size` on the process that starts REA, including `NODE_OPTIONS`, overrides that request and sets the effective heap. `--max-old-space-size-percentage` takes precedence over that flag in either order. REA does not turn the percentage into a MiB, so the effective -heap stays unestablished instead of the 128 MiB request or the absolute flag. +heap stays unestablished instead of the worker heap request or the absolute flag. The reported `NODE_OPTIONS` value is the process initial environment when that block can be read. A later edit to the environment is not the value Node applied, and an unreadable initial value leaves the effective heap @@ -229,11 +235,11 @@ buffer without a program entry leaves the effective heap unestablished. `--env-file` or `--env-file-if-exists` can supply `NODE_OPTIONS` when the initial environment has none. That file is not part of the initial environment block, so the effective heap stays unestablished instead of the -128 MiB request. An initial `NODE_OPTIONS` is still used, because Node does +worker heap request. An initial `NODE_OPTIONS` is still used, because Node does not let the file replace it. A positive `--max-heap-size`, including the underscore form and launch `NODE_OPTIONS` on Node versions that allow it, sets the total heap rather than the old generation and can be lower than the -128 MiB request. The effective old generation stays unestablished while that +worker heap request. The effective old generation stays unestablished while that value is in effect. A zero or empty value clears it. `--experimental-config-file` or `--experimental-default-config-file` can set `nodeOptions`, including a heap percentage, from a file that is not read diff --git a/src/application/Doctor.test.ts b/src/application/Doctor.test.ts index 557b5d945a..03957e6365 100644 --- a/src/application/Doctor.test.ts +++ b/src/application/Doctor.test.ts @@ -165,6 +165,7 @@ describe("doctor native analysis configuration", () => { it.each([ ["REA_ANALYSIS_PROVIDER", "Hopper"], ["REA_REGEX_MATCHING_DEADLINE_MS", "0"], + ["REA_REGEX_WORKER_HEAP_REQUEST_MIB", "6"], ["REA_GHIDRA_STARTUP_TIMEOUT_MS", "abc"], ["REA_SWIFTC_COMPILE_TIMEOUT_MS", "0"], ["REA_LOG_LEVEL", "verbose"], diff --git a/src/application/SetupRegistrationEnvironment.ts b/src/application/SetupRegistrationEnvironment.ts index 53b5f95fa5..027acb77e8 100644 --- a/src/application/SetupRegistrationEnvironment.ts +++ b/src/application/SetupRegistrationEnvironment.ts @@ -8,6 +8,7 @@ const SELECTED_REGISTRATION_SETTINGS = [ "REA_JEB_MCP_TIMEOUT_MS", "REA_MCP_INPUT_SCHEMA_PROFILE", "REA_REGEX_MATCHING_DEADLINE_MS", + "REA_REGEX_WORKER_HEAP_REQUEST_MIB", ] as const; /** Preserve selected setup settings without copying unrelated process values. */ diff --git a/src/composition/binary.ts b/src/composition/binary.ts index bab6997522..01142e690e 100644 --- a/src/composition/binary.ts +++ b/src/composition/binary.ts @@ -55,6 +55,7 @@ const composeDeepBinarySession = ( const ida = new IdaProvider(config, environment); const kuna = new KunaProvider(environment, { regexMatchingDeadline: config.regexMatchingDeadline, + regexWorkerHeapRequest: config.regexWorkerHeapRequest, }); const auxiliaryDeclarations = auxiliaryAnalysisProviderDeclarations(environment); diff --git a/src/config/environment.ts b/src/config/environment.ts index acda5c4f2e..d90715c8d9 100644 --- a/src/config/environment.ts +++ b/src/config/environment.ts @@ -20,6 +20,7 @@ import { mcpOutputSchemaProfileSchema } from "./mcpOutputSchemaProfile.js"; import { mcpResponseBudgetSchema } from "./mcpResponseBudget.js"; import { asarMaxHeaderBytesSchema } from "./asarHeaderAdmission.js"; import { regexMatchingDeadlineMsSchema } from "./regexMatchingDeadline.js"; +import { regexWorkerHeapRequestMiBSchema } from "./regexWorkerHeapRequest.js"; import { parseLoaderArgs, parseStringArray } from "./parsers.js"; const environmentSchema = z.object({ @@ -27,6 +28,7 @@ const environmentSchema = z.object({ REA_MCP_MAX_RESPONSE_BYTES: mcpResponseBudgetSchema.optional(), REA_ASAR_MAX_HEADER_BYTES: asarMaxHeaderBytesSchema.optional(), REA_REGEX_MATCHING_DEADLINE_MS: regexMatchingDeadlineMsSchema.optional(), + REA_REGEX_WORKER_HEAP_REQUEST_MIB: regexWorkerHeapRequestMiBSchema.optional(), REA_MCP_INPUT_SCHEMA_PROFILE: mcpInputSchemaProfileSchema.optional(), REA_MCP_OUTPUT_SCHEMA_PROFILE: mcpOutputSchemaProfileSchema.optional(), REA_IDA_MCP_CONFIG: z diff --git a/src/config/parseConfig.ts b/src/config/parseConfig.ts index 4d01a9d6f0..28ff6a705c 100644 --- a/src/config/parseConfig.ts +++ b/src/config/parseConfig.ts @@ -5,6 +5,7 @@ import { ok, type Result } from "../domain/result.js"; import { parseEnvironment } from "./environment.js"; import { homeDirectoryFromEnvironment } from "./homeDirectory.js"; import { DEFAULT_REGEX_MATCHING_DEADLINE } from "./regexMatchingDeadline.js"; +import { DEFAULT_REGEX_WORKER_HEAP_REQUEST } from "./regexWorkerHeapRequest.js"; import type { AppConfig } from "./types.js"; import { linuxHopperLauncherCandidates, @@ -129,5 +130,12 @@ export const parseConfig = ( deadlineMs: env.REA_REGEX_MATCHING_DEADLINE_MS, source: "setting", }, + regexWorkerHeapRequest: + env.REA_REGEX_WORKER_HEAP_REQUEST_MIB === undefined + ? DEFAULT_REGEX_WORKER_HEAP_REQUEST + : { + heapMiB: env.REA_REGEX_WORKER_HEAP_REQUEST_MIB, + source: "setting", + }, }); }; diff --git a/src/config/regexWorkerHeapRequest.test.ts b/src/config/regexWorkerHeapRequest.test.ts new file mode 100644 index 0000000000..bef250b2e5 --- /dev/null +++ b/src/config/regexWorkerHeapRequest.test.ts @@ -0,0 +1,159 @@ +import { expect, it, vi } from "vitest"; + +import { HopperProvider } from "../hopper/HopperProvider.js"; +import { KunaProvider } from "../kuna/KunaProvider.js"; +import { silentLogger } from "../logger.js"; +import { ConfigurationError } from "../domain/configurationErrors.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import { regexWorkerHeapError } from "../process/RegexMatcher.js"; +import { parseEnvironment } from "./environment.js"; +import { parseConfig } from "./parseConfig.js"; +import { + DEFAULT_REGEX_WORKER_HEAP_REQUEST, + MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB, + MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB, + parseRegexWorkerHeapRequest, + REGEX_WORKER_HEAP_REQUEST_SETTING, + regexWorkerHeapRequestConstraint, + regexWorkerHeapRequestRecovery, +} from "./regexWorkerHeapRequest.js"; + +const configured = (heapMiB: number) => + ({ heapMiB, source: "setting" }) as const; + +it("keeps the 128 MiB request when the setting is unset", () => { + expect(parseRegexWorkerHeapRequest(undefined)).toEqual({ + ok: true, + value: DEFAULT_REGEX_WORKER_HEAP_REQUEST, + }); + const parsed = parseConfig({}); + if (!parsed.ok) throw new Error("unset heap request was rejected"); + expect(parsed.value.regexWorkerHeapRequest).toEqual( + DEFAULT_REGEX_WORKER_HEAP_REQUEST, + ); + const recovery = regexWorkerHeapRequestRecovery( + DEFAULT_REGEX_WORKER_HEAP_REQUEST, + ); + expect(recovery).toContain( + "The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 128 MiB (source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB).", + ); + expect(recovery).toContain( + "Set REA_REGEX_WORKER_HEAP_REQUEST_MIB in the environment that starts REA", + ); + const projected = projectAnalysisError( + regexWorkerHeapError("search_procedures", new Error("heap")), + ); + expect(projected.code).toBe("resource_constraint"); + expect(projected.remediation.action).toContain(recovery); + expect(projected.remediation.action).not.toContain("no runtime override"); +}); + +it("uses a valid REA_REGEX_WORKER_HEAP_REQUEST_MIB override", () => { + expect(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB).toBe(7); + expect( + parseRegexWorkerHeapRequest(String(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)), + ).toEqual({ + ok: true, + value: configured(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB), + }); + expect(parseRegexWorkerHeapRequest("256")).toEqual({ + ok: true, + value: configured(256), + }); + expect( + parseRegexWorkerHeapRequest(String(MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)), + ).toEqual({ + ok: true, + value: configured(MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB), + }); + const parsed = parseConfig({ REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256" }); + if (!parsed.ok) throw new Error("valid heap request was rejected"); + expect(parsed.value.regexWorkerHeapRequest).toEqual(configured(256)); + const hopper = new HopperProvider(parsed.value, silentLogger, { + REA_REGEX_WORKER_HEAP_REQUEST_MIB: "8", + }); + const hopperText = hopper + .capabilities() + .find((capability) => capability.operation === "search_procedures") + ?.limitations.join(" "); + expect(hopperText).toContain( + "The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 256 MiB (source setting; default 128 MiB, floor 7 MiB, maximum 16384 MiB).", + ); + expect(hopperText).toContain( + "Set REA_REGEX_WORKER_HEAP_REQUEST_MIB in the environment that starts REA", + ); + expect(hopperText).not.toContain("no runtime override"); + + vi.stubEnv(REGEX_WORKER_HEAP_REQUEST_SETTING, "8"); + try { + const kuna = new KunaProvider({ + REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256", + }); + const kunaText = kuna + .capabilities() + .find((capability) => capability.operation === "search_strings") + ?.limitations.join(" "); + expect(kunaText).toContain( + "The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 256 MiB (source setting; default 128 MiB, floor 7 MiB, maximum 16384 MiB).", + ); + expect(kunaText).not.toContain( + "The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 8 MiB", + ); + } finally { + vi.unstubAllEnvs(); + } + + const ambient = new KunaProvider({}); + expect( + ambient + .capabilities() + .find((capability) => capability.operation === "search_procedures") + ?.limitations.join(" "), + ).toContain( + "The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 128 MiB (source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB).", + ); +}); + +it.each([ + "", + "0", + "6", + "-1", + "0128", + "10_000", + "1e4", + " 128", + "128 ", + "1.5", + String(MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB + 1), + "9".repeat(40), +])("rejects invalid %j as a configuration error", (value) => { + const parsed = parseRegexWorkerHeapRequest(value); + if (parsed.ok) throw new Error(`Accepted ${JSON.stringify(value)}`); + expect(parsed.error).toBeInstanceOf(ConfigurationError); + expect(parsed.error.settings).toEqual([ + { + setting: "REA_REGEX_WORKER_HEAP_REQUEST_MIB", + constraint: regexWorkerHeapRequestConstraint, + }, + ]); + expect(parsed.error.message).toBe(regexWorkerHeapRequestConstraint); + const startup = parseEnvironment({ + REA_REGEX_WORKER_HEAP_REQUEST_MIB: value, + }); + if (startup.ok) throw new Error(`Startup accepted ${JSON.stringify(value)}`); + expect(startup.error.settings).toContainEqual({ + setting: "REA_REGEX_WORKER_HEAP_REQUEST_MIB", + constraint: expect.stringContaining("from 7 to 16384 MiB"), + }); + const config = parseConfig({ REA_REGEX_WORKER_HEAP_REQUEST_MIB: value }); + if (config.ok) + throw new Error(`parseConfig accepted ${JSON.stringify(value)}`); + expect(projectAnalysisError(config.error)).toMatchObject({ + code: "configuration_invalid", + message: expect.stringContaining("REA_REGEX_WORKER_HEAP_REQUEST_MIB"), + }); + expect( + () => new KunaProvider({ REA_REGEX_WORKER_HEAP_REQUEST_MIB: value }), + ).toThrow(ConfigurationError); +}); diff --git a/src/config/regexWorkerHeapRequest.ts b/src/config/regexWorkerHeapRequest.ts new file mode 100644 index 0000000000..5b92bbe1fd --- /dev/null +++ b/src/config/regexWorkerHeapRequest.ts @@ -0,0 +1,106 @@ +import { z } from "zod"; + +import { ConfigurationError } from "../domain/configurationErrors.js"; +import { err, ok, type Result } from "../domain/result.js"; + +export const REGEX_WORKER_HEAP_REQUEST_SETTING = + "REA_REGEX_WORKER_HEAP_REQUEST_MIB"; + +/** Worker old-generation request used when the setting is unset. Today's constant. */ +export const DEFAULT_REGEX_WORKER_HEAP_REQUEST_MIB = 128; + +/** + * Smallest request that completes a trivial regex match on supported Node. + * Linux runs of the production worker (`^a` against `["a"]`, no + * `NODE_OPTIONS`) return `ERR_WORKER_OUT_OF_MEMORY` at 6 MiB on Node 24.21.0 + * and 24.22.1, and on most runs of 24.11.0, 24.18.0, and 26.0.0. 7 MiB + * returned that match on Node 22.19.0, 22.23.4, 24.11.0, 24.18.0, 24.21.0, + * 24.22.1, 26.0.0, and 26.12.1. A lower accepted value passes configuration + * and then fails the search. + */ +export const MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB = 7; + +/** + * Largest request one regex worker may make. 16 GiB is the same ceiling REA + * already uses for an isolated JavaScript analysis worker. A positive Node + * `--max-old-space-size` can still set a different effective heap; this + * maximum bounds only the REA request. + */ +export const MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB = 16_384; + +/** Effective regex worker heap request and where its value came from. */ +export interface RegexWorkerHeapRequest { + readonly heapMiB: number; + readonly source: "default" | "setting"; +} + +export const DEFAULT_REGEX_WORKER_HEAP_REQUEST: RegexWorkerHeapRequest = { + heapMiB: DEFAULT_REGEX_WORKER_HEAP_REQUEST_MIB, + source: "default", +}; + +export const regexWorkerHeapRequestConstraint = `${REGEX_WORKER_HEAP_REQUEST_SETTING} must be a decimal integer from ${String(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)} to ${String(MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)} MiB; the default is ${String(DEFAULT_REGEX_WORKER_HEAP_REQUEST_MIB)} MiB. A request below ${String(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)} MiB aborts the worker during startup, and a larger value is refused.`; + +const decimalInteger = /^[1-9]\d*$/u; + +const withinHeapRange = (value: string): boolean => { + // The pattern check reports a non-decimal value. Refine only bounds integers. + if (!decimalInteger.test(value)) return true; + const parsed = BigInt(value); + return ( + parsed >= BigInt(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB) && + parsed <= BigInt(MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB) + ); +}; + +/** Optional decimal mebibyte count for the regex worker old-generation request. */ +export const regexWorkerHeapRequestMiBSchema = z + .string() + .regex(decimalInteger, { error: regexWorkerHeapRequestConstraint }) + .refine(withinHeapRange, { error: regexWorkerHeapRequestConstraint }) + .transform(Number); + +/** Resolve the effective request from one environment value. */ +export const parseRegexWorkerHeapRequest = ( + value: string | undefined, +): Result => { + if (value === undefined) return ok(DEFAULT_REGEX_WORKER_HEAP_REQUEST); + const parsed = regexWorkerHeapRequestMiBSchema.safeParse(value); + return parsed.success + ? ok({ heapMiB: parsed.data, source: "setting" }) + : err( + new ConfigurationError(regexWorkerHeapRequestConstraint, { + settings: [ + { + setting: REGEX_WORKER_HEAP_REQUEST_SETTING, + constraint: regexWorkerHeapRequestConstraint, + }, + ], + }), + ); +}; + +/** + * Resolve the request from a composition root's selected environment. An + * invalid setting is refused, not ignored. Callers that already resolved it + * at startup pass that value through instead of reading the environment again. + */ +export const resolveRegexWorkerHeapRequest = ( + environment: Readonly>, +): RegexWorkerHeapRequest => { + const parsed = parseRegexWorkerHeapRequest( + environment[REGEX_WORKER_HEAP_REQUEST_SETTING], + ); + if (!parsed.ok) throw parsed.error; + return parsed.value; +}; + +/** How to read and change the regex worker old-generation request. */ +export const regexWorkerHeapRequestRecovery = ( + request: RegexWorkerHeapRequest, +): string => + [ + `The effective ${REGEX_WORKER_HEAP_REQUEST_SETTING} value is ${String(request.heapMiB)} MiB (source ${request.source}; default ${String(DEFAULT_REGEX_WORKER_HEAP_REQUEST_MIB)} MiB, floor ${String(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)} MiB, maximum ${String(MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)} MiB).`, + `Set ${REGEX_WORKER_HEAP_REQUEST_SETTING} in the environment that starts REA to a decimal integer from ${String(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)} to ${String(MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)}, then restart REA.`, + "A positive Node --max-old-space-size, including NODE_OPTIONS, overrides this request and sets the effective heap. That flag is not limited by this setting's floor or maximum. Raising host memory alone does not raise the request.", + ].join(" "); diff --git a/src/config/types.ts b/src/config/types.ts index 6f621a11a8..97858d8b26 100644 --- a/src/config/types.ts +++ b/src/config/types.ts @@ -6,6 +6,7 @@ import type { GhidraLanguageOverride } from "./ghidraLanguageOverride.js"; import type { McpInputSchemaProfile } from "./mcpInputSchemaProfile.js"; import type { McpOutputSchemaProfile } from "./mcpOutputSchemaProfile.js"; import type { RegexMatchingDeadline } from "./regexMatchingDeadline.js"; +import type { RegexWorkerHeapRequest } from "./regexWorkerHeapRequest.js"; export interface AppConfig { readonly mcpMaxResponseBytes?: number; @@ -30,4 +31,6 @@ export interface AppConfig { readonly referenceSourcePolicy: ReferenceSourcePolicy; /** Supervised ECMAScript matching deadline resolved at startup. */ readonly regexMatchingDeadline: RegexMatchingDeadline; + /** Regex worker old-generation request resolved at startup. */ + readonly regexWorkerHeapRequest: RegexWorkerHeapRequest; } diff --git a/src/contracts/officialToolContracts.ts b/src/contracts/officialToolContracts.ts index 2fd201ad39..44e0ac2a04 100644 --- a/src/contracts/officialToolContracts.ts +++ b/src/contracts/officialToolContracts.ts @@ -20,7 +20,7 @@ import { HOPPER_MEMORY_TOOL_DEFINITIONS } from "./hopperMemoryContracts.js"; import { analysisSearchInput } from "./analysisSearchContract.js"; const hopperRegexDeadline = - "Hopper regex mode uses ECMAScript Unicode syntax in a cancellable worker. The matching deadline is REA_REGEX_MATCHING_DEADLINE_MS (a positive decimal integer of milliseconds from 1 to 2147483647, default 5000). Unset keeps the default. A deadline failure names the effective milliseconds, source (default or setting), the default, and the maximum, and leaves the API usable."; + "Hopper regex mode uses ECMAScript Unicode syntax in a cancellable worker. The matching deadline is REA_REGEX_MATCHING_DEADLINE_MS (a positive decimal integer of milliseconds from 1 to 2147483647, default 5000). Unset keeps the default. A deadline failure names the effective milliseconds, source (default or setting), the default, and the maximum, and leaves the API usable. The worker old-generation request is REA_REGEX_WORKER_HEAP_REQUEST_MIB (a decimal integer of mebibytes from 7 to 16384, default 128). Unset keeps the default. A positive Node --max-old-space-size, including NODE_OPTIONS, overrides that request and is not limited by the setting's floor or maximum. A heap failure names the effective request, its source (default or setting), the default, the floor, and the maximum."; const official = ( name: Name, diff --git a/src/hopper/HopperProvider.ts b/src/hopper/HopperProvider.ts index ddfefb47bd..ce8c4e27c8 100644 --- a/src/hopper/HopperProvider.ts +++ b/src/hopper/HopperProvider.ts @@ -77,7 +77,10 @@ export class HopperProvider implements AnalysisProviderCandidate { private readonly platform: NodeJS.Platform = process.platform, ) { this.environment = snapshotEnvironment(environment, platform); - this.#capabilities = hopperCapabilities(config.regexMatchingDeadline); + this.#capabilities = hopperCapabilities( + config.regexMatchingDeadline, + config.regexWorkerHeapRequest, + ); } identity(): ProviderIdentity { @@ -314,6 +317,7 @@ export class HopperProvider implements AnalysisProviderCandidate { const regexSearch = new HopperRegexSearch( client, this.config.regexMatchingDeadline, + this.config.regexWorkerHeapRequest, ); const projectExecution = ( operation: AnalysisOperation, diff --git a/src/hopper/HopperProviderCapabilities.ts b/src/hopper/HopperProviderCapabilities.ts index e3dd050d65..877f56421d 100644 --- a/src/hopper/HopperProviderCapabilities.ts +++ b/src/hopper/HopperProviderCapabilities.ts @@ -2,6 +2,10 @@ import { DEFAULT_REGEX_MATCHING_DEADLINE, type RegexMatchingDeadline, } from "../config/regexMatchingDeadline.js"; +import { + DEFAULT_REGEX_WORKER_HEAP_REQUEST, + type RegexWorkerHeapRequest, +} from "../config/regexWorkerHeapRequest.js"; import { launchRegexMatchingLimitation } from "../process/RegexMatcher.js"; import type { CapabilityDescriptor, @@ -68,6 +72,7 @@ const MUTATING_OPERATIONS = new Set([ /** Source-owned capabilities without acquiring or probing a Hopper session. */ export const hopperCapabilities = ( deadline: RegexMatchingDeadline = DEFAULT_REGEX_MATCHING_DEADLINE, + heapRequest: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, ): readonly CapabilityDescriptor[] => Object.freeze( [...HOPPER_OPERATIONS, "search_bytes" as const].map((operation) => @@ -99,7 +104,7 @@ export const hopperCapabilities = ( ...(operation === "search_strings" || operation === "search_procedures" ? [ - `${launchRegexMatchingLimitation(deadline)} Literal mode retains native casefold semantics.`, + `${launchRegexMatchingLimitation(deadline, heapRequest)} Literal mode retains native casefold semantics.`, ] : []), ...(operation === "list_strings" || diff --git a/src/hopper/HopperRegexSearch.ts b/src/hopper/HopperRegexSearch.ts index 1d12f7b02b..397c61f789 100644 --- a/src/hopper/HopperRegexSearch.ts +++ b/src/hopper/HopperRegexSearch.ts @@ -16,6 +16,10 @@ import { DEFAULT_REGEX_MATCHING_DEADLINE, type RegexMatchingDeadline, } from "../config/regexMatchingDeadline.js"; +import { + DEFAULT_REGEX_WORKER_HEAP_REQUEST, + type RegexWorkerHeapRequest, +} from "../config/regexWorkerHeapRequest.js"; import { matchRegexValues } from "../process/RegexMatcher.js"; import { analysisClientFailureForOperation } from "../providers/AnalysisClientFailure.js"; @@ -33,6 +37,7 @@ export class HopperRegexSearch { constructor( private readonly client: Pick, private readonly deadline: RegexMatchingDeadline = DEFAULT_REGEX_MATCHING_DEADLINE, + private readonly heapRequest: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, ) {} execute( @@ -137,6 +142,7 @@ export class HopperRegexSearch { values: parsed.data.map((item) => item.value), signal: options.signal, deadline: this.deadline, + heapRequest: this.heapRequest, invalidPattern: (reason) => new HopperRemoteError(-32000, `Invalid regex pattern: ${reason}`, { diagnosticType: "invalid_request", diff --git a/src/kuna/KunaProvider.ts b/src/kuna/KunaProvider.ts index b1a872b6d3..9889c021d6 100644 --- a/src/kuna/KunaProvider.ts +++ b/src/kuna/KunaProvider.ts @@ -20,6 +20,10 @@ import { resolveRegexMatchingDeadline, type RegexMatchingDeadline, } from "../config/regexMatchingDeadline.js"; +import { + resolveRegexWorkerHeapRequest, + type RegexWorkerHeapRequest, +} from "../config/regexWorkerHeapRequest.js"; import { snapshotEnvironment } from "../process/snapshotEnvironment.js"; import { kunaFileCleanupError, @@ -58,6 +62,7 @@ const VERIFIED_ARCHITECTURES: ReadonlySet = new Set([ export class KunaProvider implements AnalysisProviderCandidate { readonly #environment: Readonly; readonly #deadline: RegexMatchingDeadline; + readonly #heapRequest: RegexWorkerHeapRequest; readonly #platform: NodeJS.Platform; readonly #launcher: KunaProcessLauncher | undefined; readonly #probeTimeoutMs: number | undefined; @@ -79,6 +84,11 @@ export class KunaProvider implements AnalysisProviderCandidate { * selected environment; an invalid setting is refused. */ readonly regexMatchingDeadline?: RegexMatchingDeadline; + /** + * Heap request already resolved at startup. Omitted means resolve the + * selected environment; an invalid setting is refused. + */ + readonly regexWorkerHeapRequest?: RegexWorkerHeapRequest; } = {}, ) { this.#platform = options.platform ?? process.platform; @@ -86,6 +96,9 @@ export class KunaProvider implements AnalysisProviderCandidate { this.#deadline = options.regexMatchingDeadline ?? resolveRegexMatchingDeadline(this.#environment); + this.#heapRequest = + options.regexWorkerHeapRequest ?? + resolveRegexWorkerHeapRequest(this.#environment); this.#launcher = options.launcher; this.#probeTimeoutMs = options.probeTimeoutMs; this.#singleFunctionRuns = options.singleFunctionRuns; @@ -106,6 +119,7 @@ export class KunaProvider implements AnalysisProviderCandidate { code: availability.code, }, this.#deadline, + this.#heapRequest, ); } @@ -305,6 +319,7 @@ export class KunaProvider implements AnalysisProviderCandidate { runSlots: this.#foregroundSlots(configuration.value), environment: this.#environment, deadline: this.#deadline, + heapRequest: this.#heapRequest, ...(this.#launcher === undefined ? {} : { launcher: this.#launcher }), }); } diff --git a/src/kuna/KunaProviderCapabilities.ts b/src/kuna/KunaProviderCapabilities.ts index b130a264d3..3ef7882934 100644 --- a/src/kuna/KunaProviderCapabilities.ts +++ b/src/kuna/KunaProviderCapabilities.ts @@ -8,6 +8,10 @@ import { DEFAULT_REGEX_MATCHING_DEADLINE, type RegexMatchingDeadline, } from "../config/regexMatchingDeadline.js"; +import { + DEFAULT_REGEX_WORKER_HEAP_REQUEST, + type RegexWorkerHeapRequest, +} from "../config/regexWorkerHeapRequest.js"; import { launchRegexMatchingLimitation } from "../process/RegexMatcher.js"; import { KUNA_FUNCTION_TEXT_BYTES } from "./KunaLimits.js"; @@ -67,6 +71,7 @@ const KUNA_TEXT_CAPACITY_LIMITATION = `Each buffered source facet admits at most export const kunaLimitations = ( operation: KunaOperation, deadline: RegexMatchingDeadline = DEFAULT_REGEX_MATCHING_DEADLINE, + heapRequest: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, ): readonly string[] => { switch (operation) { case "list_documents": @@ -77,7 +82,7 @@ export const kunaLimitations = ( return [ KUNA_PROVENANCE, KUNA_STRING_LIMITATION, - launchRegexMatchingLimitation(deadline), + launchRegexMatchingLimitation(deadline, heapRequest), ]; case "list_procedures": case "procedure_address": @@ -87,7 +92,7 @@ export const kunaLimitations = ( return [ KUNA_PROVENANCE, KUNA_ENTRY_LIMITATION, - launchRegexMatchingLimitation(deadline), + launchRegexMatchingLimitation(deadline, heapRequest), ]; case "procedure_callers": case "procedure_callees": @@ -119,6 +124,7 @@ export const kunaCapabilities = ( readonly code: ProviderRejectionCode; }, deadline: RegexMatchingDeadline = DEFAULT_REGEX_MATCHING_DEADLINE, + heapRequest: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, ): readonly CapabilityDescriptor[] => KUNA_OPERATIONS.map((operation) => ({ provider: KUNA_PROVIDER_IDENTITY, @@ -142,5 +148,5 @@ export const kunaCapabilities = ( changesPermissions: false, requiresRoot: false, }, - limitations: kunaLimitations(operation, deadline), + limitations: kunaLimitations(operation, deadline, heapRequest), })); diff --git a/src/kuna/KunaQueryWork.ts b/src/kuna/KunaQueryWork.ts index 6df0009c91..864d4fc944 100644 --- a/src/kuna/KunaQueryWork.ts +++ b/src/kuna/KunaQueryWork.ts @@ -9,6 +9,10 @@ import { DEFAULT_REGEX_MATCHING_DEADLINE, type RegexMatchingDeadline, } from "../config/regexMatchingDeadline.js"; +import { + DEFAULT_REGEX_WORKER_HEAP_REQUEST, + type RegexWorkerHeapRequest, +} from "../config/regexWorkerHeapRequest.js"; import { matchRegexValues } from "../process/RegexMatcher.js"; import type { KunaJsonDocumentReader } from "./KunaJsonStream.js"; import { KUNA_DIAGNOSTIC_BYTES } from "./KunaLimits.js"; @@ -41,6 +45,7 @@ export class KunaQueryWork { readonly runSlots: KunaRunSlots; }, private readonly deadline: RegexMatchingDeadline = DEFAULT_REGEX_MATCHING_DEADLINE, + private readonly heapRequest: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, ) {} /** Unproven process cleanup retains both leases in the native command owner. */ @@ -138,6 +143,7 @@ export class KunaQueryWork { values: input.values(), signal, deadline: this.deadline, + heapRequest: this.heapRequest, }); } finally { release(); diff --git a/src/kuna/KunaSessionClient.ts b/src/kuna/KunaSessionClient.ts index 0626434efd..85c8fa6817 100644 --- a/src/kuna/KunaSessionClient.ts +++ b/src/kuna/KunaSessionClient.ts @@ -71,6 +71,7 @@ import { KunaQueryWork } from "./KunaQueryWork.js"; import { KunaStringQueries } from "./KunaStringQueries.js"; import { normalizeKunaAddress } from "./KunaAddress.js"; import type { RegexMatchingDeadline } from "../config/regexMatchingDeadline.js"; +import type { RegexWorkerHeapRequest } from "../config/regexWorkerHeapRequest.js"; import { KUNA_FUNCTION_TEXT_BYTES } from "./KunaLimits.js"; import type { KunaRunRelease, KunaRunSlots } from "./KunaRunSlots.js"; import { @@ -253,6 +254,7 @@ export class KunaSessionClient implements AnalysisClient { readonly #references: KunaReferenceQueries; readonly #strings: KunaStringQueries; readonly #deadline: RegexMatchingDeadline; + readonly #heapRequest: RegexWorkerHeapRequest; #installation: | SharedStartupAttempt> | undefined; @@ -285,6 +287,7 @@ export class KunaSessionClient implements AnalysisClient { readonly runSlots: KunaRunSlots; readonly environment: Readonly; readonly deadline: RegexMatchingDeadline; + readonly heapRequest: RegexWorkerHeapRequest; readonly launcher?: KunaProcessLauncher; }, ) { @@ -301,6 +304,7 @@ export class KunaSessionClient implements AnalysisClient { : { launcher: dependencies.launcher }), }); this.#deadline = dependencies.deadline; + this.#heapRequest = dependencies.heapRequest; this.#queryWork = new KunaQueryWork( this.#commands, { @@ -309,6 +313,7 @@ export class KunaSessionClient implements AnalysisClient { runSlots: dependencies.runSlots, }, dependencies.deadline, + dependencies.heapRequest, ); this.#references = new KunaReferenceQueries(this.#queryWork); this.#strings = new KunaStringQueries(this.#queryWork); @@ -419,7 +424,7 @@ export class KunaSessionClient implements AnalysisClient { ); return ok( this.#execution( - kunaLimitations(operation, this.#deadline), + kunaLimitations(operation, this.#deadline, this.#heapRequest), installation.value, answer.value, ), @@ -1271,7 +1276,11 @@ export class KunaSessionClient implements AnalysisClient { cfg: unavailable("Kuna exports no basic blocks."), }, limitations: [ - ...kunaLimitations("analyze_function", this.#deadline), + ...kunaLimitations( + "analyze_function", + this.#deadline, + this.#heapRequest, + ), ...pseudocode.value.limitations, KUNA_RELATIONSHIP_LIMITATION, ...(relationships.ok diff --git a/src/process/RegexMatcher.test.ts b/src/process/RegexMatcher.test.ts index 330161faa2..bdff4647ad 100644 --- a/src/process/RegexMatcher.test.ts +++ b/src/process/RegexMatcher.test.ts @@ -1,5 +1,6 @@ import { expect, it } from "vitest"; +import { DEFAULT_REGEX_MATCHING_DEADLINE } from "../config/regexMatchingDeadline.js"; import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import { execArgvFromCommandLine, @@ -454,6 +455,13 @@ it("describes the effective heap and the configurable matching deadline", () => expect(limitation).toContain("REA_REGEX_MATCHING_DEADLINE_MS"); expect(limitation).toContain("floor 1 ms"); expect(limitation).toContain("maximum 2147483647 ms"); + expect(limitation).toContain("REA_REGEX_WORKER_HEAP_REQUEST_MIB"); + expect(limitation).toContain( + "source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB", + ); + expect(limitation).toContain( + "Set REA_REGEX_WORKER_HEAP_REQUEST_MIB in the environment that starts REA", + ); expect(limitation).not.toContain("no runtime override"); expect(limitation).not.toContain("either cap"); expect(limitation).not.toContain("no runtime override for the heap"); @@ -476,10 +484,14 @@ it("describes the effective heap and the configurable matching deadline", () => ); expect(projected.message).toContain("was not established"); expect(projected.details).toMatchObject({ - reported_limits: { requested_worker_heap_mib: 128 }, + reported_limits: { + requested_worker_heap_mib: 128, + requested_worker_heap_source: "default", + }, }); expect(projected.details?.reported_limits).toEqual({ requested_worker_heap_mib: 128, + requested_worker_heap_source: "default", }); } else { expect(limitation).toContain(`effective ${String(heap.mib)} MiB`); @@ -488,8 +500,57 @@ it("describes the effective heap and the configurable matching deadline", () => reported_limits: { worker_heap_mib: heap.mib, requested_worker_heap_mib: 128, + requested_worker_heap_source: "default", }, }); } } }); + +it("keeps a selected heap request unless a positive Node flag overrides it", () => { + const request = { heapMiB: 256, source: "setting" as const }; + expect(resolveRegexWorkerHeap([], undefined, request)).toEqual( + effective(256, false), + ); + expect( + resolveRegexWorkerHeap(["--max-old-space-size=64"], undefined, request), + ).toEqual(effective(64, true)); + const selected = regexMatchingLimitation( + effective(256, false), + DEFAULT_REGEX_MATCHING_DEADLINE, + request, + ); + expect(selected).toContain("effective 256 MiB"); + expect(selected).toContain( + "The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 256 MiB (source setting; default 128 MiB, floor 7 MiB, maximum 16384 MiB).", + ); + const overridden = regexMatchingLimitation( + effective(64, true), + DEFAULT_REGEX_MATCHING_DEADLINE, + request, + ); + expect(overridden).toContain("effective 64 MiB"); + expect(overridden).toContain("source setting"); + expect(overridden).toContain("floor 7 MiB"); + expect(overridden).toContain("maximum 16384 MiB"); + const projected = projectAnalysisError( + regexWorkerHeapError( + "search_procedures", + new Error("heap"), + effective(64, true), + request, + ), + ); + expect(projected.details).toMatchObject({ + reported_limits: { + worker_heap_mib: 64, + requested_worker_heap_mib: 256, + requested_worker_heap_source: "setting", + }, + }); + expect(projected.remediation.action).toContain("source setting"); + expect(projected.remediation.action).toContain( + "Set REA_REGEX_WORKER_HEAP_REQUEST_MIB in the environment that starts REA", + ); + expect(projected.remediation.action).not.toContain("no runtime override"); +}); diff --git a/src/process/RegexMatcher.ts b/src/process/RegexMatcher.ts index 9e98e4baeb..4a822bd3ed 100644 --- a/src/process/RegexMatcher.ts +++ b/src/process/RegexMatcher.ts @@ -16,10 +16,15 @@ import { regexMatchingDeadlineRecovery, type RegexMatchingDeadline, } from "../config/regexMatchingDeadline.js"; - -const WORKER_HEAP_MIB = 128; +import { + DEFAULT_REGEX_WORKER_HEAP_REQUEST, + REGEX_WORKER_HEAP_REQUEST_SETTING, + regexWorkerHeapRequestRecovery, + type RegexWorkerHeapRequest, +} from "../config/regexWorkerHeapRequest.js"; /** - * Node's Worker `resourceLimits.maxOldGenerationSizeMb` request. A positive + * Node's Worker `resourceLimits.maxOldGenerationSizeMb` request. + * `REA_REGEX_WORKER_HEAP_REQUEST_MIB` is that request (default 128 MiB). A positive * `--max-old-space-size` on the process that starts REA, including * `NODE_OPTIONS`, overrides it. A zero or empty value leaves this request * in effect. `NODE_OPTIONS` is applied before `process.execArgv`, and the @@ -31,7 +36,7 @@ const WORKER_HEAP_MIB = 128; * before user code reads it, and some Node versions also cap total memory at * 4 GiB when pointer compression is on. A later read is not the value Node * applied, so a percentage flag leaves the effective heap unestablished - * instead of the 128 MiB request or an absolute flag. + * instead of the worker heap request or an absolute flag. * * `process.env.NODE_OPTIONS` is not that launch value. A preload can clear or * rewrite it after V8 has already applied the original flags, and those flags @@ -50,7 +55,7 @@ const WORKER_HEAP_MIB = 128; * is not a fallback. * * `--max-heap-size` is the total heap, not the old generation. A positive - * value can be lower than the 128 MiB request and still changes the result + * value can be lower than the worker heap request and still changes the result * when an old-space flag is also present. The old generation stays * unestablished while that value is in effect. Zero or empty clears it. * Node 24 and later also accept the flag in `NODE_OPTIONS`. @@ -80,17 +85,11 @@ export type RegexWorkerHeapLimit = | { readonly kind: "effective"; readonly mib: number; - /** True when a positive `--max-old-space-size` overrides the 128 MiB request. */ + /** True when a positive `--max-old-space-size` overrides the worker heap request. */ readonly overridden: boolean; } | { readonly kind: "unknown" }; -const requestedHeap = { - kind: "effective", - mib: WORKER_HEAP_MIB, - overridden: false, -} as const satisfies RegexWorkerHeapLimit; - /** Node's `ParseNodeOptionsEnvVar`: quotes group tokens and are not part of them. */ const nodeOptionsArgv = ( nodeOptions: string, @@ -144,7 +143,13 @@ const parsedFlagMiB = ( export const resolveRegexWorkerHeap = ( execArgv: readonly string[], nodeOptions: string | undefined, + request: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, ): RegexWorkerHeapLimit => { + const requestedHeap = { + kind: "effective" as const, + mib: request.heapMiB, + overridden: false as const, + }; const fromEnvironment = nodeOptions === undefined ? [] : nodeOptionsArgv(nodeOptions); const tokens = [...(fromEnvironment ?? []), ...execArgv]; @@ -375,44 +380,62 @@ export const readLaunchExecArgv = (): LaunchExecArgv => { export const resolveLaunchRegexWorkerHeap = ( launchExecArgv: LaunchExecArgv, launchNodeOptions: LaunchNodeOptions, + request: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, ): RegexWorkerHeapLimit => launchExecArgv.kind === "unrecoverable" || launchNodeOptions.kind === "unrecoverable" ? { kind: "unknown" } - : resolveRegexWorkerHeap(launchExecArgv.value, launchNodeOptions.value); + : resolveRegexWorkerHeap( + launchExecArgv.value, + launchNodeOptions.value, + request, + ); + +const launchExecArgv = readLaunchExecArgv(); +const launchNodeOptions = readLaunchNodeOptions(); const launchRegexWorkerHeap = resolveLaunchRegexWorkerHeap( - readLaunchExecArgv(), - readLaunchNodeOptions(), + launchExecArgv, + launchNodeOptions, ); -const heapBound = (heap: RegexWorkerHeapLimit): string => { +const heapBound = ( + heap: RegexWorkerHeapLimit, + request: RegexWorkerHeapRequest, +): string => { + const recovery = regexWorkerHeapRequestRecovery(request); if (heap.kind === "unknown") - return "The worker requests 128 MiB. The effective old-generation heap was not established from the launch flags. --max-old-space-size, --max-old-space-size-percentage, --max-heap-size, --env-file, --env-file-if-exists, --experimental-config-file, and --experimental-default-config-file can change that request."; - const override = - "A positive Node --max-old-space-size, including NODE_OPTIONS, overrides the 128 MiB worker request"; + return `The worker requests ${String(request.heapMiB)} MiB. The effective old-generation heap was not established from the launch flags. --max-old-space-size, --max-old-space-size-percentage, --max-heap-size, --env-file, --env-file-if-exists, --experimental-config-file, and --experimental-default-config-file can change that request. ${recovery}`; + const override = `A positive Node --max-old-space-size, including NODE_OPTIONS, overrides the ${String(request.heapMiB)} MiB worker request`; return heap.overridden - ? `${override} and sets this process's effective heap to ${String(heap.mib)} MiB.` - : `${override}. This process has no positive override, so the effective heap is ${String(heap.mib)} MiB.`; + ? `${override} and sets this process's effective heap to ${String(heap.mib)} MiB. ${recovery}` + : `${override}. This process has no positive override, so the effective heap is ${String(heap.mib)} MiB. ${recovery}`; }; /** Capability text for the supervised ECMAScript matching boundary. */ export const regexMatchingLimitation = ( heap: RegexWorkerHeapLimit, deadline: RegexMatchingDeadline = DEFAULT_REGEX_MATCHING_DEADLINE, + request: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, ): string => { const deadlineText = `${String(deadline.deadlineMs)} ms matching deadline (source ${deadline.source})`; const preface = heap.kind === "effective" ? `Regex mode uses ECMAScript Unicode semantics in a cancellable worker with a ${deadlineText} and an effective ${String(heap.mib)} MiB old-generation heap.` : `Regex mode uses ECMAScript Unicode semantics in a cancellable worker with a ${deadlineText}.`; - return `${preface} ${regexMatchingDeadlineRecovery(deadline)} ${heapBound(heap)} Matching failure leaves the service available.`; + return `${preface} ${regexMatchingDeadlineRecovery(deadline)} ${heapBound(heap, request)} Matching failure leaves the service available.`; }; -/** Capability text for this process's launch heap and one resolved deadline. */ +/** Capability text for this process's launch heap, deadline, and heap request. */ export const launchRegexMatchingLimitation = ( deadline: RegexMatchingDeadline = DEFAULT_REGEX_MATCHING_DEADLINE, -): string => regexMatchingLimitation(launchRegexWorkerHeap, deadline); + request: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, +): string => + regexMatchingLimitation( + resolveLaunchRegexWorkerHeap(launchExecArgv, launchNodeOptions, request), + deadline, + request, + ); /** * Capability text for the default deadline on this process's launch heap. @@ -444,6 +467,7 @@ export const regexWorkerHeapError = ( operation: string, cause: Error, heap: RegexWorkerHeapLimit = launchRegexWorkerHeap, + request: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST, ): AnalysisResourceConstraintError => new AnalysisResourceConstraintError( operation, @@ -454,12 +478,16 @@ export const regexWorkerHeapError = ( heap.kind === "effective" ? { worker_heap_mib: heap.mib, - requested_worker_heap_mib: WORKER_HEAP_MIB, + requested_worker_heap_mib: request.heapMiB, + requested_worker_heap_source: request.source, } - : { requested_worker_heap_mib: WORKER_HEAP_MIB }, + : { + requested_worker_heap_mib: request.heapMiB, + requested_worker_heap_source: request.source, + }, { cause, - remediationAction: `${heapBound(heap)} Restart REA after changing --max-old-space-size. A launch --max-old-space-size-percentage takes precedence over that flag in either order. Simplify the pattern or use literal mode.`, + remediationAction: `${heapBound(heap, request)} Restart REA after changing --max-old-space-size or ${REGEX_WORKER_HEAP_REQUEST_SETTING}. A launch --max-old-space-size-percentage takes precedence over that flag in either order. Simplify the pattern or use literal mode.`, }, ); const replySchema = z.discriminatedUnion("status", [ @@ -499,8 +527,11 @@ export const matchRegexValues = async (input: { readonly invalidPattern?: (reason: string) => AnalysisError; /** Resolved deadline. Omitted means the 5000 ms default, not ambient `process.env`. */ readonly deadline?: RegexMatchingDeadline | undefined; + /** Resolved heap request. Omitted means the 128 MiB default, not ambient `process.env`. */ + readonly heapRequest?: RegexWorkerHeapRequest | undefined; }): Promise> => { const deadline = input.deadline ?? DEFAULT_REGEX_MATCHING_DEADLINE; + const heapRequest = input.heapRequest ?? DEFAULT_REGEX_WORKER_HEAP_REQUEST; if (input.signal?.aborted) return err(new AnalysisCancelledError(input.operation)); let worker: Worker; @@ -512,7 +543,7 @@ export const matchRegexValues = async (input: { caseSensitive: input.caseSensitive, values: input.values, }, - resourceLimits: { maxOldGenerationSizeMb: WORKER_HEAP_MIB }, + resourceLimits: { maxOldGenerationSizeMb: heapRequest.heapMiB }, }); } catch (cause: unknown) { return err( @@ -591,7 +622,16 @@ export const matchRegexValues = async (input: { finish( err( cause.code === "ERR_WORKER_OUT_OF_MEMORY" - ? regexWorkerHeapError(input.operation, cause) + ? regexWorkerHeapError( + input.operation, + cause, + resolveLaunchRegexWorkerHeap( + launchExecArgv, + launchNodeOptions, + heapRequest, + ), + heapRequest, + ) : new AnalysisOutputError( input.operation, `Regex worker failed: ${cause.message}`, diff --git a/src/setupEnvironment.test.ts b/src/setupEnvironment.test.ts index 2a713be701..d24b4e574e 100644 --- a/src/setupEnvironment.test.ts +++ b/src/setupEnvironment.test.ts @@ -186,6 +186,7 @@ it.each([ REA_HOPPER_STARTUP_TIMEOUT_MS: "120000", REA_MCP_INPUT_SCHEMA_PROFILE: "full", REA_REGEX_MATCHING_DEADLINE_MS: "10000", + REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256", }, providerConfigured: false, installHopper: true, @@ -219,6 +220,7 @@ it.each([ REA_HOPPER_STARTUP_TIMEOUT_MS: "120000", REA_MCP_INPUT_SCHEMA_PROFILE: "full", REA_REGEX_MATCHING_DEADLINE_MS: "10000", + REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256", }); } else { expect(initial.providerEnvironment).not.toHaveProperty( diff --git a/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts b/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts index c926007748..cfe9589091 100644 --- a/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts +++ b/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts @@ -487,6 +487,7 @@ cliTest( REA_JEB_MCP_TIMEOUT_MS: "2900003", REA_MCP_INPUT_SCHEMA_PROFILE: "full", REA_REGEX_MATCHING_DEADLINE_MS: "10000", + REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256", REA_UNRELATED_MARKER: "do-not-persist", }; const run = (arguments_: readonly string[]) => @@ -516,6 +517,7 @@ cliTest( expect(codexRegistration).toMatchObject({ tool_timeout_sec: 3021 }); expect(codexConfig).toContain('REA_JEB_MCP_TIMEOUT_MS = "2900003"'); expect(codexConfig).toContain('REA_REGEX_MATCHING_DEADLINE_MS = "10000"'); + expect(codexConfig).toContain('REA_REGEX_WORKER_HEAP_REQUEST_MIB = "256"'); expect(codexConfig).not.toContain("REA_UNRELATED_MARKER"); const kimiPath = join(kimiHome, "mcp.json"); @@ -532,6 +534,7 @@ cliTest( REA_JEB_MCP_TIMEOUT_MS: "2900003", REA_MCP_INPUT_SCHEMA_PROFILE: "full", REA_REGEX_MATCHING_DEADLINE_MS: "10000", + REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256", }, }); expect(await readFile(kimiPath, "utf8")).not.toContain( diff --git a/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts b/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts index b100de8697..53aca18e93 100644 --- a/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts +++ b/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts @@ -496,6 +496,11 @@ it.each([ { REA_REGEX_MATCHING_DEADLINE_MS: "0" }, "REA_REGEX_MATCHING_DEADLINE_MS", ], + [ + "regex worker heap request below its floor", + { REA_REGEX_WORKER_HEAP_REQUEST_MIB: "6" }, + "REA_REGEX_WORKER_HEAP_REQUEST_MIB", + ], [ "malformed loader JSON", { HOPPER_LOADER_ARGS_JSON: "{" }, diff --git a/tests/boundary/process/regexMatcher.test.ts b/tests/boundary/process/regexMatcher.test.ts index 7ffe8dcdf7..b877e8266a 100644 --- a/tests/boundary/process/regexMatcher.test.ts +++ b/tests/boundary/process/regexMatcher.test.ts @@ -2,6 +2,7 @@ import { spawnSync } from "node:child_process"; import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; import { expect, it } from "vitest"; @@ -10,6 +11,7 @@ import { MAXIMUM_REGEX_MATCHING_DEADLINE_MS, REGEX_MATCHING_DEADLINE_SETTING, } from "../../../src/config/regexMatchingDeadline.js"; +import { MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB } from "../../../src/config/regexWorkerHeapRequest.js"; import { ok } from "../../../src/domain/result.js"; import { HopperRegexSearch } from "../../../src/hopper/HopperRegexSearch.js"; import { OFFICIAL_TOOL_CONTRACTS } from "../../../src/contracts/officialToolContracts.js"; @@ -78,6 +80,12 @@ it("names the configurable regex deadline and the effective worker heap", () => "5000 ms matching deadline (source default)", ); expect(REGEX_MATCHING_LIMITATION).toContain("REA_REGEX_MATCHING_DEADLINE_MS"); + expect(REGEX_MATCHING_LIMITATION).toContain( + "REA_REGEX_WORKER_HEAP_REQUEST_MIB", + ); + expect(REGEX_MATCHING_LIMITATION).toContain( + "source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB", + ); expect(REGEX_MATCHING_LIMITATION).toContain("floor 1 ms"); expect(REGEX_MATCHING_LIMITATION).toContain("maximum 2147483647 ms"); expect(REGEX_MATCHING_LIMITATION).not.toContain("no runtime override"); @@ -107,18 +115,28 @@ it("names the configurable regex deadline and the effective worker heap", () => ); expect(failure.remediation.action).not.toContain("no runtime override"); expect(failure.remediation.action).toContain("--max-old-space-size"); + expect(failure.remediation.action).toContain( + "REA_REGEX_WORKER_HEAP_REQUEST_MIB", + ); + expect(failure.remediation.action).toContain( + "source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB", + ); if (heap.kind === "effective") { expect(failure.message).toContain(`${String(heap.mib)} MiB`); expect(failure.details).toMatchObject({ reported_limits: { worker_heap_mib: heap.mib, requested_worker_heap_mib: 128, + requested_worker_heap_source: "default", }, }); } else { expect(failure.message).toContain("was not established"); expect(failure.details).toMatchObject({ - reported_limits: { requested_worker_heap_mib: 128 }, + reported_limits: { + requested_worker_heap_mib: 128, + requested_worker_heap_source: "default", + }, }); } for (const name of ["search_procedures", "search_strings"] as const) { @@ -131,6 +149,11 @@ it("names the configurable regex deadline and the effective worker heap", () => ); expect(description).toContain(String(MAXIMUM_REGEX_MATCHING_DEADLINE_MS)); expect(description).toContain("source (default or setting)"); + expect(description).toContain("REA_REGEX_WORKER_HEAP_REQUEST_MIB"); + expect(description).toContain("default 128"); + expect(description).toContain("from 7 to 16384"); + expect(description).toContain("floor"); + expect(description).toContain("maximum"); expect(description).not.toContain("no runtime override"); } }); @@ -482,6 +505,53 @@ it("stops a slow match at the configured deadline and names the setting", async }); }); +it("matches a trivial pattern in a real worker at the heap floor", () => { + const directory = mkdtempSync(join(tmpdir(), "rea-regex-heap-floor-")); + const probe = join(directory, "probe.mjs"); + const repository = fileURLToPath(new URL("../../../", import.meta.url)); + const tsxApi = pathToFileURL( + join(repository, "node_modules/tsx/dist/esm/api/index.mjs"), + ).href; + const matcher = pathToFileURL( + join(repository, "src/process/RegexMatcher.ts"), + ).href; + // Plain Node, with NODE_OPTIONS removed. Do not put `--import tsx` on this + // process: workers inherit execArgv, and that flag exhausts a floor-sized heap. + writeFileSync( + probe, + `import { tsImport } from ${JSON.stringify(tsxApi)}; +const { matchRegexValues } = await tsImport(${JSON.stringify(matcher)}, import.meta.url); +const result = await matchRegexValues({ + operation: "search_procedures", + pattern: "^a", + caseSensitive: true, + values: ["a"], + heapRequest: { + heapMiB: ${String(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)}, + source: "setting", + }, +}); +process.stdout.write(JSON.stringify(result.ok + ? { ok: true, indices: result.value } + : { ok: false, message: String(result.error?.message ?? "") })); +if (!result.ok) process.exitCode = 1; +`, + ); + const env = { ...process.env }; + delete env.NODE_OPTIONS; + try { + const result = spawnSync(process.execPath, [probe], { + env, + encoding: "utf8", + timeout: 30_000, + }); + expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(0); + expect(JSON.parse(result.stdout)).toEqual({ ok: true, indices: [0] }); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); + it("preserves Unicode matching order and refuses a pre-cancelled match", async () => { expect( await matchRegexValues({