mirror of
https://github.com/morluto/rea.git
synced 2026-11-19 05:02:40 +00:00
* fix(pcap): report a refused tool launch as an unavailable tool When a capture read's tool launch was refused with EPERM, pcap reported it as unreadable tool output (AnalysisProtocolError) instead of an unavailable tool. Pcap also never told "not executable" apart from "missing". Add classifyLaunchFailure in src/process as the one owner of which spawn errnos mean missing (ENOENT, ENOTDIR) and which mean present but not executable (EACCES, EPERM, ENOEXEC, EISDIR). Pcap, apktool and gdb used three different errno lists; all three now use it. * refactor(process): classify owned command outcomes through one owner Add ProviderOutcome, read once from the supervisor's facts, and classifyProviderOutcome, the one mapping from an outcome to a typed error. Migrate pcap and the shared artifact producer command failure (TShark, sigrok). Add the rea/provider-failure-classifier lint rule with a shrink-only baseline of per-site (cause: unknown) => AnalysisError functions. * refactor(process): classify Java and adb launch refusals with the shared owner (#10598) The Ghidra and JADX Java probes and adb's version probe now take missing-versus-not-executable from classifyLaunchFailure instead of their own errno checks. ELOOP and ENAMETOOLONG join the owner's missing set, so adb keeps its existing classification and both Java probes now treat a non-directory path component as a missing runtime. * fix(pcap): preserve command launch attribution * test(pcap): expect an unknown failing stage when a limited capture tool exits unsuccessfully * refactor(process): classify owned command outcomes through one owner Add ProviderOutcome, read once from the supervisor's facts, and classifyProviderOutcome, the one mapping from an outcome to a typed error. Migrate pcap and the shared artifact producer command failure (TShark, sigrok). Add the rea/provider-failure-classifier lint rule with a shrink-only baseline of per-site (cause: unknown) => AnalysisError functions. * test(lint): cover baselined files in the provider-failure classifier rule * fix(apktool): report the decoder JVM's heap exhaustion as memory, not a target refusal The supervisor now watches for a managed runtime's out-of-memory report while output streams (JVM java.lang.OutOfMemoryError, V8 'JavaScript heap out of memory'), so a retention budget cannot hide it. ProviderOutcome gains a runtime-exhausted kind and classifyProviderOutcome maps it once. Apktool decode opts in and moves to the shared classifier; Ghidra's single-purpose output token becomes the JVM runtime option and its area matching reuses the shared table. * fix(process): keep the runtime's out-of-memory report on the memory error it supports * fix(pcap): attribute a limited capture's exit through the prlimit wrapper * fix(process): keep a shortened out-of-memory line marked as shortened * fix(process): identify the refused executable for every launcher shape Name the executable on errnos Node throws synchronously from spawn (ENOEXEC, ENOTDIR, ...) and read the Windows native launcher's executable-launch constraint, so pcap attributes those refusals to the selected tool or limiter instead of an unknown stage. * test(pcap): prove synchronous launch naming with ENOTDIR on every POSIX host * fix(apktool): advise the JVM limit that governs the reported memory area * fix(ghidra): read the supervisor's parsed OOM area instead of reparsing lines * fix(sqlite,capture): read V8 heap exhaustion from streamed output, not retained stderr (#10724) The SQLite worker and the Node capture decoders now declare runtime: "v8", so the supervisor observes V8's heap report as output streams. Both classifiers read the runtime-exhausted outcome instead of matching retained stderr, so a report beyond the retention budget, or one whose process then overflowed its output budget, is still reported as memory exhaustion. * refactor(evm): classify EVMole worker outcomes through the shared provider classifier (#10716) * refactor(evm): classify EVMole worker outcomes through the shared provider classifier The EVMole worker's cancellation, deadline, output budget and CPU/file-size signals now go through classifyProviderOutcome; only the prlimit launch refusal and the reserved EFBIG exit remain in its dialect. The worker is opted into V8 exhaustion scanning so a heap report names its fixed 256 MiB heap. providerOutcomeOf now reads a launch refusal only from a spawn errno, so a private-storage or carrier read errno is no longer reported as a limiter launch failure. * fix(evm): keep a stream failure's exit status and update Linux worker expectations * refactor(process): read launch refusals through the launcher identity owner * refactor(wasm): classify WABT command outcomes through the shared provider classifier (#10727) * refactor(wasm): classify WABT command outcomes through the shared provider classifier WABT's cancellation, command deadline, output budget, and inherited-limit signals now go through classifyProviderOutcome; its dialect keeps only the configuration-probe and launch-refusal unavailability and the unclassified-exit output error. A tool ended by SIGXCPU/SIGXFSZ or the memory limit is now a resource constraint instead of an output error, and a refused launch in the producer phase is reported as unavailable. The shared timeout context gains the analysis setting that selected the deadline, and the output-limit context may name its exceeded budget. * fix(wasm): keep setup guidance when a WABT version probe's output cannot be collected * refactor(process): build classified cancellations with cancelledBy and the carried caller reason * chore(lint): count runOwned's close-failure mapper from main in the classifier baseline * fix(process): carry the runtime's exhaustion report on every memory error that claims it SQLite and historical-capture memory errors recognised a streamed V8 heap report but dropped it, so a report outside retained output reached callers without its source line or omission flags. One projection, runtimeExhaustionReport, now owns that shape for the shared classifier and both consumers. Also lowers the EVMole and WABT cancellation-constructor baselines. * fix(process): keep the provider's recovery for a native-thread exhaustion report The shared classifier replaced the provider-supplied recovery with generic capacity advice whenever the JVM reported it could not create a native thread, dropping Apktool's launcher guidance and flipping retryable to true. The provider's recovery now wins; the capacity advice remains the fallback for providers that supply none.
303 lines
10 KiB
TypeScript
303 lines
10 KiB
TypeScript
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
|
|
import { expect, it, vi } from "vitest";
|
|
|
|
import { NodeCaptureAdapter } from "../browser/history/NodeCaptureAdapter.js";
|
|
import { HistoricalCaptureDecoder } from "../browser/history/HistoricalCaptureDecoder.js";
|
|
import { decodeHarCapture } from "../browser/history/HarCapture.js";
|
|
import { historicalCaptureFailure } from "../browser/history/CaptureFailures.js";
|
|
import { ConfigurationError } from "../domain/configurationErrors.js";
|
|
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
|
|
import { inspectWebNetworkCaptureInputSchema } from "../domain/webNetworkCapture.js";
|
|
import { OwnedCommandFailure } from "../process/OwnedCommand.js";
|
|
import { createWebNetworkCaptureService } from "../composition/webNetworkCaptures.js";
|
|
import { createCli } from "../cli.js";
|
|
import { historicalHar } from "../../tests/fixtures/historicalHar.js";
|
|
import { parseEnvironment } from "./environment.js";
|
|
import { parseConfig } from "./parseConfig.js";
|
|
import {
|
|
CAPTURE_DECODER_OLD_GENERATION_SETTING,
|
|
DEFAULT_CAPTURE_DECODER_HEAP,
|
|
parseCaptureDecoderHeap,
|
|
resolveCaptureDecoderHeap,
|
|
} from "./captureDecoderHeap.js";
|
|
import { MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB } from "../domain/analysisResourceLimits.js";
|
|
|
|
const configured = (oldGenerationMiB: number) =>
|
|
({ oldGenerationMiB, source: "setting" }) as const;
|
|
|
|
const heapExhaustion = () => {
|
|
const stderr =
|
|
"FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory\n";
|
|
return new OwnedCommandFailure("process", "Decoder failed", {
|
|
stdout: { text: "", bytes: 0, observedBytes: 0 },
|
|
stderr: {
|
|
text: stderr,
|
|
bytes: Buffer.byteLength(stderr),
|
|
observedBytes: Buffer.byteLength(stderr),
|
|
},
|
|
exitCode: null,
|
|
signal: "SIGABRT",
|
|
runtimeExhaustion: {
|
|
runtime: "v8",
|
|
area: "heap",
|
|
stream: "stderr",
|
|
line: stderr.trim(),
|
|
lineComplete: true,
|
|
truncated: false,
|
|
invalidUtf8: false,
|
|
},
|
|
});
|
|
};
|
|
|
|
it("keeps the 192 MiB decoder heap when the setting is unset", async () => {
|
|
expect(parseCaptureDecoderHeap(undefined)).toEqual({
|
|
ok: true,
|
|
value: DEFAULT_CAPTURE_DECODER_HEAP,
|
|
});
|
|
const parsed = parseConfig({});
|
|
if (!parsed.ok) throw new Error("unset heap was rejected");
|
|
expect(new NodeCaptureAdapter("har", {}).heap).toEqual(
|
|
DEFAULT_CAPTURE_DECODER_HEAP,
|
|
);
|
|
const command = await new NodeCaptureAdapter("har").command(
|
|
"/owned/request.json",
|
|
);
|
|
expect(command.arguments).toContain("--max-old-space-size=192");
|
|
expect(command.arguments).toContain("--max-semi-space-size=8");
|
|
});
|
|
|
|
it("uses a valid REA_CAPTURE_DECODER_OLD_GENERATION_MIB override", async () => {
|
|
expect(parseCaptureDecoderHeap("256")).toEqual({
|
|
ok: true,
|
|
value: configured(256),
|
|
});
|
|
expect(
|
|
parseCaptureDecoderHeap(String(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB)),
|
|
).toEqual({
|
|
ok: true,
|
|
value: configured(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB),
|
|
});
|
|
const environment = { REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "256" };
|
|
const parsed = parseConfig(environment);
|
|
if (!parsed.ok) throw new Error("valid heap was rejected");
|
|
const adapter = new NodeCaptureAdapter("har", environment);
|
|
expect(adapter.heap).toEqual(configured(256));
|
|
const command = await adapter.command("/owned/request.json");
|
|
expect(command.arguments).toContain("--max-old-space-size=256");
|
|
expect(command.arguments).toContain("--max-semi-space-size=8");
|
|
const input = inspectWebNetworkCaptureInputSchema.parse({
|
|
capture_path: "/selected/input.har",
|
|
format: "har",
|
|
});
|
|
const projected = projectAnalysisError(
|
|
historicalCaptureFailure(
|
|
input,
|
|
heapExhaustion(),
|
|
"decoder",
|
|
undefined,
|
|
configured(256),
|
|
),
|
|
);
|
|
expect(projected).toMatchObject({
|
|
code: "resource_constraint",
|
|
details: { resource: "memory" },
|
|
});
|
|
expect(projected.remediation.action).toMatch(
|
|
/REA_CAPTURE_DECODER_OLD_GENERATION_MIB.*256 MiB.*source setting/iu,
|
|
);
|
|
expect(projected.remediation.action).toContain("--max-old-space-size=256");
|
|
expect(projected.remediation.action).toMatch(
|
|
/floor 8 MiB.*maximum 16384 MiB/iu,
|
|
);
|
|
expect(projected.remediation.action).toContain("update_analysis_settings");
|
|
expect(projected.remediation.action).toContain("launch seed");
|
|
expect(projected.remediation.action).toContain(
|
|
"capture.decoder_old_generation_mib",
|
|
);
|
|
expect(projected.remediation.action).toMatch(
|
|
/removes inherited NODE_OPTIONS/iu,
|
|
);
|
|
expect(projected.remediation.action).toMatch(/Bun.*V8 flags are ignored/iu);
|
|
expect(projected.remediation.action).toMatch(/Windows.*512 MiB reserve/iu);
|
|
expect(projected.remediation.action).toMatch(
|
|
/Other Bun hosts enforce no decoder heap limit/iu,
|
|
);
|
|
if (process.versions.bun === undefined)
|
|
expect(projected.details).toMatchObject({
|
|
limit: {
|
|
setting: "capture.decoder_old_generation_mib",
|
|
effective_value: 256,
|
|
source: "setting",
|
|
unit: "MiB",
|
|
applies_at: "next_worker",
|
|
agent_editable: true,
|
|
update_mode: "runtime",
|
|
pending_restart: false,
|
|
},
|
|
reported_limits: {
|
|
old_generation_heap_mib: 256,
|
|
old_generation_heap_source: "setting",
|
|
},
|
|
});
|
|
});
|
|
|
|
it("keeps a parsed heap instead of reading the adapter environment again", async () => {
|
|
const heap = configured(256);
|
|
const adapter = new NodeCaptureAdapter(
|
|
"har",
|
|
{ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "512" },
|
|
heap,
|
|
);
|
|
expect(adapter.heap).toBe(heap);
|
|
const command = await adapter.command("/owned/request.json");
|
|
expect(command.arguments).toContain("--max-old-space-size=256");
|
|
expect(command.arguments).not.toContain("--max-old-space-size=512");
|
|
});
|
|
|
|
it("reports the heap the selected adapter launches when decoder environment differs", async () => {
|
|
const launched = new NodeCaptureAdapter("har", {
|
|
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "256",
|
|
});
|
|
const other = new NodeCaptureAdapter("saz", {
|
|
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "512",
|
|
});
|
|
const root = await mkdtemp(join(tmpdir(), "rea-capture-heap-report-"));
|
|
const capturePath = join(root, "input.har");
|
|
const text = JSON.stringify(historicalHar());
|
|
await writeFile(capturePath, text);
|
|
const input = inspectWebNetworkCaptureInputSchema.parse({
|
|
capture_path: capturePath,
|
|
format: "har",
|
|
});
|
|
const decoder = new HistoricalCaptureDecoder([launched, other], {});
|
|
try {
|
|
vi.spyOn(launched, "command").mockImplementation(async (requestPath) => {
|
|
const runtimePath = join(requestPath, "..");
|
|
await writeFile(
|
|
join(runtimePath, "reply.json"),
|
|
JSON.stringify({ ok: true, value: decodeHarCapture(text, []) }),
|
|
);
|
|
return {
|
|
command: process.execPath,
|
|
arguments: ["-e", "process.exit(0)"],
|
|
};
|
|
});
|
|
const reported = await decoder.inspect(input);
|
|
if (!reported.ok)
|
|
throw new Error(projectAnalysisError(reported.error).message);
|
|
const limitations = reported.value.limitations.join("\n");
|
|
expect(limitations).toMatch(
|
|
/REA_CAPTURE_DECODER_OLD_GENERATION_MIB.*256 MiB.*source setting/iu,
|
|
);
|
|
|
|
vi.spyOn(launched, "command").mockRejectedValue(heapExhaustion());
|
|
const exhausted = await decoder.inspect(input);
|
|
if (exhausted.ok) throw new Error("Heap exhaustion must fail inspection");
|
|
const projected = projectAnalysisError(exhausted.error);
|
|
expect(projected.code).toBe("resource_constraint");
|
|
expect(projected.remediation.action).toMatch(
|
|
/REA_CAPTURE_DECODER_OLD_GENERATION_MIB.*256 MiB.*source setting/iu,
|
|
);
|
|
expect(projected.remediation.action).toContain("--max-old-space-size=256");
|
|
if (process.versions.bun === undefined)
|
|
expect(projected.details).toMatchObject({
|
|
limit: {
|
|
setting: "capture.decoder_old_generation_mib",
|
|
effective_value: 256,
|
|
source: "setting",
|
|
},
|
|
reported_limits: {
|
|
old_generation_heap_mib: 256,
|
|
old_generation_heap_source: "setting",
|
|
},
|
|
});
|
|
} finally {
|
|
await decoder.close();
|
|
await rm(root, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it("registers CLI commands when the capture decoder heap setting is invalid", async () => {
|
|
const cli = createCli({
|
|
REA_LOG_LEVEL: "silent",
|
|
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "7",
|
|
});
|
|
let text = "";
|
|
await cli.serve(["inspect-web-network-capture", "--help"], {
|
|
env: {},
|
|
exit: () => undefined,
|
|
stdout: (chunk: string) => {
|
|
text += chunk;
|
|
},
|
|
});
|
|
expect(text).toContain("inspect-web-network-capture");
|
|
});
|
|
|
|
it.each([
|
|
"",
|
|
"0",
|
|
"7",
|
|
"-1",
|
|
"0192",
|
|
"192.5",
|
|
" 192",
|
|
"192 ",
|
|
"1e3",
|
|
String(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB + 1),
|
|
"9".repeat(40),
|
|
])("rejects invalid %j as a configuration error", (value) => {
|
|
const parsed = parseCaptureDecoderHeap(value);
|
|
if (parsed.ok) throw new Error(`Accepted ${JSON.stringify(value)}`);
|
|
expect(parsed.error).toBeInstanceOf(ConfigurationError);
|
|
expect(parsed.error.message).toContain(
|
|
CAPTURE_DECODER_OLD_GENERATION_SETTING,
|
|
);
|
|
expect(parsed.error.message).toMatch(/decimal integer/i);
|
|
expect(parsed.error.message).toMatch(/8 to 16384 MiB/i);
|
|
expect(parsed.error.message).toMatch(/default.*192 MiB/i);
|
|
expect(parsed.error.message).toMatch(/semi-space.*abort.*Node.*startup/i);
|
|
expect(parsed.error.settings).toEqual([
|
|
{
|
|
setting: CAPTURE_DECODER_OLD_GENERATION_SETTING,
|
|
constraint: parsed.error.message,
|
|
},
|
|
]);
|
|
const startup = parseEnvironment({
|
|
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
|
|
});
|
|
if (!startup.ok)
|
|
throw new Error(`Startup rejected a runtime seed ${JSON.stringify(value)}`);
|
|
const config = parseConfig({
|
|
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
|
|
});
|
|
if (!config.ok)
|
|
throw new Error(
|
|
`parseConfig rejected a runtime seed ${JSON.stringify(value)}`,
|
|
);
|
|
expect(() =>
|
|
resolveCaptureDecoderHeap({
|
|
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
|
|
}),
|
|
).toThrow(ConfigurationError);
|
|
expect(
|
|
() =>
|
|
new NodeCaptureAdapter("har", {
|
|
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
|
|
}),
|
|
).toThrow(ConfigurationError);
|
|
expect(
|
|
() =>
|
|
new HistoricalCaptureDecoder([], {
|
|
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
|
|
}),
|
|
).toThrow(ConfigurationError);
|
|
expect(() =>
|
|
createWebNetworkCaptureService({
|
|
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
|
|
}),
|
|
).toThrow(ConfigurationError);
|
|
});
|