refactor(domain): give strict UTF-8 decoding one owner with an explicit BOM policy (#10728)

* refactor(domain): give strict UTF-8 decoding one owner with an explicit BOM policy

* fix(domain): keep the leading-BOM behaviour every migrated UTF-8 decoder had

* refactor(domain): read the domain layer's strict UTF-8 through the BOM-policy owner (#10803)

Twelve domain decoders built their own fatal TextDecoder with ignoreBOM: true. They now use strictUtf8Decoder("preserve"), which keeps that behaviour: each reads bytes where a leading U+FEFF is content (binary-format names and fields, JSON and XML read byte-for-byte, recorded bodies, registry seeds whose kept bytes are re-emitted). The counted baseline drops from 74 to 62.

* refactor(artifacts): read artifact readers' strict UTF-8 through the BOM-policy owner (#10871)

Fourteen artifact reader modules (17 sites) built their own fatal TextDecoder. They now take it from strictUtf8Decoder with the policy each had: every one kept a leading BOM (ignoreBOM: true), so they use "preserve"; JavaScriptArtifactFiles keeps it for JavaScript and HTML entries and strips it otherwise, as before. The strict-utf8-decoder baseline drops those 17 sites.

* refactor(domain): drop TextDecoder imports left by the strict UTF-8 migration

* refactor(application): read MCP registration metadata through the strict UTF-8 owner

* refactor(formats): read binary-format readers' strict UTF-8 through the BOM-policy owner (#10903)

* refactor(formats): read binary-format readers' strict UTF-8 through the BOM-policy owner

* refactor(formats): drop the model readers' unused TextDecoder imports

* refactor(browser): read browser capture and fetch strict UTF-8 through the BOM-policy owner (#10935)

* refactor(browser): read browser capture and fetch strict UTF-8 through the BOM-policy owner

* refactor(native): read LLDB and Mach-O name strict UTF-8 through the BOM-policy owner (#10936)

* refactor(native): read LLDB and Mach-O name strict UTF-8 through the BOM-policy owner

* refactor(process): read MCP and NDJSON transport frames' strict UTF-8 through the BOM-policy owner (#10937)

* refactor(process): read MCP and NDJSON transport frames' strict UTF-8 through the BOM-policy owner

* refactor(application): read JSON and file inputs' strict UTF-8 through the BOM-policy owner (#10938)

* refactor(application): read JSON and file inputs' strict UTF-8 through the BOM-policy owner

* refactor(providers): read IDA, Electron, Ghidra and reference strict UTF-8 through the BOM-policy owner (#10939)

* refactor(providers): read IDA, Electron, Ghidra and reference strict UTF-8 through the BOM-policy owner

* refactor(evm): read the hex bytecode carrier's strict UTF-8 through the owner and refuse new local decoders (#10940)

* refactor(evm): read the hex bytecode carrier's strict UTF-8 through the BOM-policy owner

* chore(architecture): match local strict UTF-8 decoders in any option order
This commit is contained in:
morluto authored and GitHub committed 2026-11-15 19:54:39 +08:00
1 parent abd36dd250
commit cabd56ea71
103 files changed
+322 -251

No files matched your search

+6 -5
View File
@@ -1,3 +1,4 @@
import { decodeStrictUtf8 } from "../domain/strictUtf8.js";
import { OwnerCancellation } from "../domain/analysisErrorCore.js";
import {
deserializeMessage,
@@ -386,11 +387,11 @@ export class OwnedMcpStdioTransport implements Transport {
);
this.#appendFrame(fragment, frameBytes);
if (newline === -1) return;
const line = new TextDecoder("utf-8", {
fatal: true,
// Preserve a leading BOM so the JSON-RPC parser can reject it.
ignoreBOM: true,
}).decode(this.#frameBuffer.subarray(0, this.#frameLength));
// Preserve a leading BOM so the JSON-RPC parser can reject it.
const line = decodeStrictUtf8(
this.#frameBuffer.subarray(0, this.#frameLength),
"preserve",
);
this.#frameBuffer = Buffer.alloc(0);
this.#frameLength = 0;
this.onmessage?.(deserializeMessage(line));
+2 -6
View File
@@ -37,6 +37,7 @@ import {
type ManagedRuntime,
type RuntimeExhaustion,
} from "./RuntimeExhaustion.js";
import { isStrictUtf8 } from "../domain/strictUtf8.js";
const DEFAULT_TERMINATION_GRACE_MS = 250;
const DEFAULT_KILL_GRACE_MS = 1_000;
@@ -1143,12 +1144,7 @@ class ProcessOutputCapture {
observedBytes: number,
): ProviderProcessSnapshot["stdout"] {
const text = buffer.toString("utf8");
let invalidUtf8 = false;
try {
new TextDecoder("utf-8", { fatal: true }).decode(buffer);
} catch {
invalidUtf8 = true;
}
const invalidUtf8 = !isStrictUtf8(buffer);
return {
text,
bytes: buffer.byteLength,
+4 -2
View File
@@ -32,6 +32,7 @@ import {
regexWorkerHeapRequestRecovery,
type RegexWorkerHeapRequest,
} from "../config/regexWorkerHeapRequest.js";
import { decodeStrictUtf8, strictUtf8Decoder } from "../domain/strictUtf8.js";
/**
* Node's Worker `resourceLimits.maxOldGenerationSizeMb` request.
* `REA_REGEX_WORKER_HEAP_REQUEST_MIB` is that request (default 128 MiB). A positive
@@ -232,8 +233,9 @@ const nodeOptionsFromEnviron = (
entry.subarray(0, nodeOptionsKey.length).equals(nodeOptionsKey)
) {
try {
return new TextDecoder("utf-8", { fatal: true }).decode(
return decodeStrictUtf8(
entry.subarray(nodeOptionsKey.length),
"strip-signature",
);
} catch {
return "invalid";
@@ -288,7 +290,7 @@ const commandLineTokens = (raw: Buffer): readonly string[] | "invalid" => {
if (raw.length === 0) return "invalid";
const end = raw[raw.length - 1] === 0 ? raw.length - 1 : raw.length;
const tokens: string[] = [];
const decoder = new TextDecoder("utf-8", { fatal: true });
const decoder = strictUtf8Decoder("strip-signature");
let offset = 0;
while (offset < end) {
const next = raw.indexOf(0, offset);
+2 -4
View File
@@ -1,3 +1,4 @@
import { strictUtf8Decoder } from "../domain/strictUtf8.js";
import {
malformedInput,
type ParseFailureError,
@@ -12,10 +13,7 @@ export class Utf8NdjsonLineBuffer {
#buffer: Buffer = Buffer.alloc(0);
#fragments: Buffer[] = [];
#nextNewline = -1;
readonly #decoder = new TextDecoder("utf-8", {
fatal: true,
ignoreBOM: true,
});
readonly #decoder = strictUtf8Decoder("preserve");
/**
* Yield each complete trimmed nonblank line; preserve unconsumed lines on