refactor(domain): give strict UTF-8 decoding one owner with an explicit BOM policy (#10728)

* refactor(domain): give strict UTF-8 decoding one owner with an explicit BOM policy

* fix(domain): keep the leading-BOM behaviour every migrated UTF-8 decoder had

* refactor(domain): read the domain layer's strict UTF-8 through the BOM-policy owner (#10803)

Twelve domain decoders built their own fatal TextDecoder with ignoreBOM: true. They now use strictUtf8Decoder("preserve"), which keeps that behaviour: each reads bytes where a leading U+FEFF is content (binary-format names and fields, JSON and XML read byte-for-byte, recorded bodies, registry seeds whose kept bytes are re-emitted). The counted baseline drops from 74 to 62.

* refactor(artifacts): read artifact readers' strict UTF-8 through the BOM-policy owner (#10871)

Fourteen artifact reader modules (17 sites) built their own fatal TextDecoder. They now take it from strictUtf8Decoder with the policy each had: every one kept a leading BOM (ignoreBOM: true), so they use "preserve"; JavaScriptArtifactFiles keeps it for JavaScript and HTML entries and strips it otherwise, as before. The strict-utf8-decoder baseline drops those 17 sites.

* refactor(domain): drop TextDecoder imports left by the strict UTF-8 migration

* refactor(application): read MCP registration metadata through the strict UTF-8 owner

* refactor(formats): read binary-format readers' strict UTF-8 through the BOM-policy owner (#10903)

* refactor(formats): read binary-format readers' strict UTF-8 through the BOM-policy owner

* refactor(formats): drop the model readers' unused TextDecoder imports

* refactor(browser): read browser capture and fetch strict UTF-8 through the BOM-policy owner (#10935)

* refactor(browser): read browser capture and fetch strict UTF-8 through the BOM-policy owner

* refactor(native): read LLDB and Mach-O name strict UTF-8 through the BOM-policy owner (#10936)

* refactor(native): read LLDB and Mach-O name strict UTF-8 through the BOM-policy owner

* refactor(process): read MCP and NDJSON transport frames' strict UTF-8 through the BOM-policy owner (#10937)

* refactor(process): read MCP and NDJSON transport frames' strict UTF-8 through the BOM-policy owner

* refactor(application): read JSON and file inputs' strict UTF-8 through the BOM-policy owner (#10938)

* refactor(application): read JSON and file inputs' strict UTF-8 through the BOM-policy owner

* refactor(providers): read IDA, Electron, Ghidra and reference strict UTF-8 through the BOM-policy owner (#10939)

* refactor(providers): read IDA, Electron, Ghidra and reference strict UTF-8 through the BOM-policy owner

* refactor(evm): read the hex bytecode carrier's strict UTF-8 through the owner and refuse new local decoders (#10940)

* refactor(evm): read the hex bytecode carrier's strict UTF-8 through the BOM-policy owner

* chore(architecture): match local strict UTF-8 decoders in any option order
This commit is contained in:
morluto authored and GitHub committed 2026-11-15 19:54:39 +08:00
1 parent abd36dd250
commit cabd56ea71
103 files changed
+322 -251

No files matched your search

+2 -1
View File
@@ -1,3 +1,4 @@
import { strictUtf8Decoder } from "../domain/strictUtf8.js";
import { decode } from "@msgpack/msgpack";
import {
BinaryResourceFailure,
@@ -12,7 +13,7 @@ const AMD_METADATA_CONTAINER_ENTRIES = 65536;
const AMD_METADATA_VALUE_BYTES = 65536;
/** Validate allocation and nesting costs before the standard decoder allocates containers. */
const utf8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
const utf8 = strictUtf8Decoder("preserve");
const preflight = (bytes: Buffer): Set<string> => {
const strings = new Set<string>();
let at = 0,
+3 -1
View File
@@ -1,3 +1,4 @@
import { decodeStrictUtf8 } from "../domain/strictUtf8.js";
import { scheduler } from "node:timers/promises";
import type { GpuObject } from "../domain/gpu/gpuCodeObjects.js";
import {
@@ -69,8 +70,9 @@ export const readClangBundleDirectory = async (
);
let name: string;
try {
name = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(
name = decodeStrictUtf8(
await source.read(position + 24, nameSize),
"preserve",
);
} catch (cause) {
if (cause instanceof TypeError)
+2 -4
View File
@@ -1,3 +1,4 @@
import { decodeStrictUtf8 } from "../domain/strictUtf8.js";
import {
CCOB_MAGIC,
CLANG_BUNDLE_MAGIC,
@@ -366,10 +367,7 @@ export class GpuInventory {
const content = bytes.subarray(0, length);
let text: string;
try {
text = new TextDecoder("utf-8", {
fatal: true,
ignoreBOM: true,
}).decode(content);
text = decodeStrictUtf8(content, "preserve");
} catch {
return rejectGpu("PTX is not valid UTF-8 text");
}