Files
reverse-engineering-anything/src/sqlite/SqliteDatabaseProvider.ts
T
morluto b98fbdc442 refactor(ida): let IDA close failures read the session's later close (#10811)
* refactor(errors): read cleanup completion from dependencies instead of pushing it to dependents

AnalysisError.cleanupIncomplete becomes a getter over the error's own unresolved state and its bound dependencies. The WeakRef dependents graph and resolveCleanup's push propagation are removed, and subclasses mark their initial state through markCleanupIncomplete instead of overriding a public mutable field.

* fix(errors): settle bound dependencies before binding more

A composite whose dependencies had released but whose status was never read
stayed incomplete when a later dependency was bound.

* refactor(errors): let errors read a lease group's release instead of resolving them (#10759)

AnalysisCleanupLeaseGroup no longer calls resolveCleanup() on its error.
The error holds the group as a release claim and reads cleanupReleased when
asked, so completion follows the lease owner without a push.

* chore(verify): count resolveCleanup and bindCleanupDependencies calls so they only shrink (#10763)

* refactor(errors): let errors read a lease group's release instead of resolving them

AnalysisCleanupLeaseGroup no longer calls resolveCleanup() on its error.
The error holds the group as a release claim and reads cleanupReleased when
asked, so completion follows the lease owner without a push.

* chore(verify): count resolveCleanup and bindCleanupDependencies calls so they only shrink

* refactor(process): let worker cleanup errors read their root epoch's release

OwnedWorkerRuntime's root epoch is now an AnalysisCleanupReleaseClaim. Errors bound to it read its release when asked instead of the epoch pushing resolveCleanup() through weak references, and a runtime without a retained root binds an already-released owner. Removes the OwnedWorkerCleanupFailure structural type and four resolveCleanup() pushes; the counted baseline shrinks accordingly.

* refactor(pcap): let pcap cleanup errors read command and snapshot release (#10792)

Retained pcap commands, retained snapshot failures and snapshot completion descriptors now own their release and errors read it: commands and snapshot failures implement AnalysisCleanupReleaseClaim, and a failed completion-descriptor close is retried through an AnalysisCleanupLease bound to its error. Removes four resolveCleanup() pushes; the counted baseline shrinks accordingly.

* refactor(process): let Darwin helper cleanup errors read their owner's release

Darwin helper cache and run-token reader root cleanup now record an AnalysisCleanupRelease that every failure reports; a successful retry confirms it. Replaces the first-failure anchor that later failures depended on and the resolveCleanup() push. Removes two resolveCleanup() and two bindCleanupDependencies() calls; both counted baselines shrink.

* refactor(ghidra): let Ghidra cleanup errors read the client's and checkpoint's release

GhidraClient and GhidraCheckpoint bind each cleanup failure to an AnalysisCleanupRelease and confirm it when a later close or retry releases everything, instead of keeping a set of failures and pushing resolveCleanup() to each. Removes two resolveCleanup() calls; the counted baseline shrinks.

* refactor(process): let debugger session cleanup errors read a confirmed stop

OwnedDebuggerSessionRegistry binds each retained gdb or rizin cleanup failure to the session's AnalysisCleanupRelease and confirms it when a stop is confirmed, keeping the latest failure for retry reporting, instead of pushing resolveCleanup() to a set of failures. Removes one resolveCleanup() call; the counted baseline shrinks.

* refactor(process): drop the debugger registry's unused closeOperation option

Returning the latest retained failure removed the last reader of closeOperation; remove the option and the values gdb and rizin supplied.

* refactor(server): let environment shutdown errors read a successful close

serveEnvironment binds each shutdown failure to an AnalysisCleanupRelease that a later successful close confirms, instead of pushing resolveCleanup() to a set of failures. The state-file retry test now also requires the earlier report to read the release. Removes one resolveCleanup() call; the counted baseline shrinks.

* refactor(desktop): release retained screen-judge cleanup through its lease

Each retained judge or screenshot cleanup owner is wrapped in an AnalysisCleanupLease bound to its failure; close() releases the lease, which records the release the failure reads, instead of closing the owner and pushing resolveCleanup(). Removes one resolveCleanup() call; the counted baseline shrinks.

* refactor(ida): let IDA close failures read the session's later close

IdaSessionClient's close failure binds an owner that is both its resource owner and its release claim, read from the client's closed state, instead of keeping weak references and pushing resolveCleanup() after a successful retry. Removes one resolveCleanup() call; the counted baseline shrinks.

* refactor(javascript): let source-map codec cleanup errors read a later close (#10814)

* refactor(javascript): let source-map codec cleanup errors read a later close

OwnedCodec binds each close failure to an AnalysisCleanupRelease that a later successful close confirms, replacing the first-failure anchor that later failures depended on and the resolveCleanup() push. Removes one resolveCleanup() and one bindCleanupDependencies() call; both counted baselines shrink.

* refactor(cutter): let pending Cutter command errors read the completion receipt (#10815)

* refactor(cutter): let pending Cutter command errors read the completion receipt

Each pending Cutter command keeps an AnalysisCleanupRelease bound to its timeout failure; the producer's terminal completion receipt confirms it instead of pushing resolveCleanup(). Removes one resolveCleanup() call; the counted baseline shrinks.

* refactor(application): let provider candidate cleanup errors read the candidate's close (#10817)

* refactor(application): let provider candidate cleanup errors read the candidate's close

AnalysisProviderRegistry keeps one AnalysisCleanupRelease per candidate whose resource close is unconfirmed; failures bind to it and a later confirmed close confirms it, instead of pushing resolveCleanup() to a set of failures. Removes one resolveCleanup() call; the counted baseline shrinks.

* fix(errors): bind a cleanup release claim to an error at most once

AnalysisError kept release claims in an array, so a provider whose retried close returned the same error bound the same claim again on every retry and each cleanupIncomplete read scanned the duplicates. Keep the claims in a set.

* refactor(application): let reverse-tool cleanup errors read a verified stop (#10819)

* refactor(application): let reverse-tool cleanup errors read a verified stop

Each retained provider process group in ReverseEngineeringService keeps an AnalysisCleanupRelease; its command and retry failures bind to it and a later verified stop confirms it, instead of collecting failures and pushing resolveCleanup(). Removes one resolveCleanup() call; the counted baseline shrinks.

* refactor(sqlite): let SQLite worker and root cleanup errors read their release (#10821)

Each retained SQLite snapshot cleanup records an AnalysisCleanupRelease for its worker and for its runtime root; the resource failures bind to them and a verified stop or root removal confirms them, instead of pushing resolveCleanup(). Composite failures keep reading those resource failures. Removes two resolveCleanup() calls; the counted baseline shrinks.

* refactor(application): let JADX and session close failures read their owner's release (#10854)

* refactor(android): let JADX cleanup errors read their retained cleanup's release

Each retained JADX cleanup records an AnalysisCleanupRelease for the admission failure's descriptor owners and one for the whole retained cleanup. The admission failure and every error the cleanup reports bind to them, and a confirmed close confirms them, instead of pushing resolveCleanup() through WeakRefs to reported errors. Removes two resolveCleanup() calls; the counted baseline shrinks.

* refactor(application): let session close failures read the client's later close

A retained session cleanup records an AnalysisCleanupRelease; unowned cleanup failures the client's close reported bind to it and the successful close confirms it, instead of the session collecting them and pushing resolveCleanup(). Failures that already follow their own owner keep reading it. Adds a regression for an unowned close failure. Removes one resolveCleanup() call; the counted baseline shrinks.
2026-11-15 19:53:53 +08:00

561 lines
19 KiB
TypeScript

import {
createAnalysisExecution,
type AnalysisExecution,
type ExecutionOptions,
} from "../application/AnalysisProvider.js";
import type { SqliteDatabasePort } from "../application/sqlite/SqliteDatabasePort.js";
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
import { bindArtifactFileCleanup } from "../artifacts/ArtifactFileOperation.js";
import type {
AnalysisCapturedOutput,
AnalysisError,
} from "../domain/analysisErrorBase.js";
import { AnalysisCleanupRelease } from "../domain/analysisCleanupRelease.js";
import {
AnalysisOutputError,
AnalysisResourceConstraintError,
cancelledBy,
OwnerCancellation,
} from "../domain/analysisErrorCore.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { createAnalysisExecutionEvidence } from "../domain/analysisObservation.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import { err, ok, type Result } from "../domain/result.js";
import {
inspectSqliteDatabaseInputSchema,
sqliteDatabaseSchema,
type InspectSqliteDatabaseInput,
DEFAULT_SQLITE_WORKER_TIMEOUT_MS,
sqliteInspectionParameters,
} from "../domain/sqlite/sqliteDatabase.js";
import { SQLITE_PROVIDER_IDENTITY } from "../domain/sqlite/SqliteProviderIdentity.js";
import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js";
import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js";
import {
OwnedCommandFailure,
ownedCommandResources,
} from "../process/OwnedCommand.js";
import type {
ProviderProcessStopResult,
ProviderProcessSupervisor,
} from "../process/ProviderProcess.js";
import {
createSqliteCleanupError,
sqliteDatabaseFailure,
} from "./SqliteDatabaseFailures.js";
import { captureSqliteDatabaseSnapshot } from "./SqliteDatabaseSnapshot.js";
import { SQLITE_DATABASE_LIMITS } from "./SqliteDatabaseLimits.js";
import {
executeSqliteDatabaseWorker,
sqliteReplyFinalizationFailure,
type SqliteWorkerLauncher,
} from "./SqliteDatabaseWorkerExecution.js";
const operation = "inspect_sqlite_database";
type SqliteOutcome = Result<AnalysisExecution, AnalysisError>;
type SqliteRuntimeRoot = Pick<PrivateRuntimeRoot, "path" | "close">;
interface PendingSnapshotCleanup {
readonly input: InspectSqliteDatabaseInput;
readonly result: SqliteOutcome;
readonly output: AnalysisCapturedOutput | undefined;
/** Reports the runtime root until its removal confirms `rootRelease`. */
readonly rootFailure: AnalysisError;
readonly rootRelease: AnalysisCleanupRelease;
/** Reports the worker until a verified stop confirms `processRelease`. */
readonly processFailure: AnalysisError | undefined;
readonly processRelease: AnalysisCleanupRelease;
process: ProviderProcessSupervisor | undefined;
}
/** The runtime root a retained cleanup still owns. */
const sqliteRootFailure = (root: SqliteRuntimeRoot): ProviderCleanupError =>
new ProviderCleanupError(
SQLITE_PROVIDER_IDENTITY.id,
[root.path],
{ reason: "SQLite runtime root cleanup remains unresolved." },
{ operation },
);
/** Inspect a provider-owned copy through a cancellable SQLite child, without source writes. */
export class SqliteDatabaseProvider implements SqliteDatabasePort {
readonly #active = new Set<Promise<SqliteOutcome>>();
readonly #artifactResources = new ArtifactResourceScope();
readonly #pendingCleanup = new Map<
SqliteRuntimeRoot,
PendingSnapshotCleanup
>();
#closed = false;
#closePromise: Promise<void> | undefined;
#retryPromise: Promise<Result<null, AnalysisError>> | undefined;
constructor(
readonly environment: Readonly<NodeJS.ProcessEnv> = process.env,
readonly launcher?: SqliteWorkerLauncher,
readonly createRuntime: () => Promise<
Pick<PrivateRuntimeRoot, "path" | "close">
> = () => PrivateRuntimeRoot.create({ prefix: "rea-sqlite-database-" }),
) {}
/** Return exact original DB/WAL identities and current schema/selected committed rows. */
async inspect(
input: InspectSqliteDatabaseInput,
options?: ExecutionOptions,
): Promise<SqliteOutcome> {
if (options?.signal?.aborted)
return err(cancelledBy(operation, options.signal));
if (this.#closed)
return err(cancelledBy(operation, OwnerCancellation.ownerClosed));
if (this.#active.size >= SQLITE_DATABASE_LIMITS.concurrentInspections)
return err(
new AnalysisResourceConstraintError(
operation,
"capacity",
"SQLite inspection capacity is occupied. Retry after an active inspection completes.",
{
active_inspections: this.#active.size,
max_concurrent_inspections:
SQLITE_DATABASE_LIMITS.concurrentInspections,
input_bytes_per_inspection: SQLITE_DATABASE_LIMITS.inputBytes,
},
{
remediationAction: `The concurrent SQLite inspection allowance is ${String(SQLITE_DATABASE_LIMITS.concurrentInspections)}. This release has no runtime override. Waiting for an active inspection to finish can free a slot. worker_timeout_ms does not raise it. Retry after an active SQLite inspection completes.`,
},
),
);
const inspection = this.#artifactResources
.run(() => this.#inspect(input, options))
.catch((cause: unknown) =>
err(
sqliteDatabaseFailure(
cause,
input.path,
"artifact-read",
options?.signal,
),
),
);
this.#active.add(inspection);
try {
return await inspection;
} finally {
this.#active.delete(inspection);
}
}
/** Await inspections and retry retained descriptors, workers, and snapshot roots. */
close(): Promise<void> {
this.#closed = true;
this.#closePromise ??= Promise.allSettled(this.#active)
.then(async () => {
let descriptorFailure: AnalysisError | undefined;
try {
await this.#artifactResources.close();
} catch (cause: unknown) {
const cleanup = ArtifactReaderFailure.cleanupObservation(
cause,
"SQLite artifact descriptors",
);
descriptorFailure = new ProviderCleanupError(
SQLITE_PROVIDER_IDENTITY.id,
cleanup.resources,
{ reason: cleanup.reason },
{ operation, cause },
);
bindArtifactFileCleanup(cause, descriptorFailure);
}
const cleaned = await this.#retryCleanup();
if (descriptorFailure !== undefined && !cleaned.ok) {
const failure = new ProviderCleanupError(
SQLITE_PROVIDER_IDENTITY.id,
[
...descriptorFailure.cleanupResources,
...cleaned.error.cleanupResources,
],
{
failures: [
projectAnalysisError(descriptorFailure),
projectAnalysisError(cleaned.error),
],
},
{ operation },
);
failure.bindCleanupDependencies([descriptorFailure, cleaned.error]);
throw failure;
}
if (descriptorFailure !== undefined) throw descriptorFailure;
if (!cleaned.ok) throw cleaned.error;
})
.catch((cause: unknown) => {
this.#closePromise = undefined;
throw cause;
});
return this.#closePromise;
}
#retryCleanup(): Promise<Result<null, AnalysisError>> {
this.#retryPromise ??= this.#cleanupPending().finally(() => {
this.#retryPromise = undefined;
});
return this.#retryPromise;
}
async #cleanupPending(): Promise<Result<null, AnalysisError>> {
const failures: AnalysisError[] = [];
for (const [root, pending] of [...this.#pendingCleanup]) {
if (pending.process !== undefined) {
const stopped = await pending.process.stop();
if (stopped.status === "incomplete" || stopped.status === "not-owned") {
const reason =
stopped.status === "incomplete"
? stopped.reason
: "SQLite worker ownership remains unresolved.";
const failure = sqliteCleanupFailure(
[
pending.process.launch.ownership?.runId ?? "owned-sqlite-worker",
root.path,
],
reason,
pending.result,
pending.input,
pending.output,
);
failure.bindCleanupDependencies(
[pending.processFailure, pending.rootFailure].filter(
(dependency): dependency is AnalysisError =>
dependency !== undefined,
),
);
failures.push(failure);
continue;
}
pending.process = undefined;
pending.processRelease.confirm();
}
const cleaned = await cleanupSqliteRoot(
root,
pending.result,
pending.input,
pending.output,
);
if (cleaned.ok) {
pending.rootRelease.confirm();
this.#pendingCleanup.delete(root);
} else {
cleaned.error.bindCleanupDependencies([pending.rootFailure]);
failures.push(cleaned.error);
}
}
if (failures.length === 1 && failures[0] !== undefined)
return err(failures[0]);
return failures.length === 0
? ok(null)
: err(
new ProviderCleanupError(
SQLITE_PROVIDER_IDENTITY.id,
failures.flatMap((failure) => failure.cleanupResources),
{
failures: failures.map((failure) =>
projectAnalysisError(failure),
),
},
{ operation },
).bindCleanupDependencies(failures),
);
}
async #inspect(
input: InspectSqliteDatabaseInput,
options?: ExecutionOptions,
): Promise<SqliteOutcome> {
let root: Pick<PrivateRuntimeRoot, "path" | "close"> | undefined;
let processOwner: ProviderProcessSupervisor | undefined;
let processCleanup:
| Extract<ProviderProcessStopResult, { status: "incomplete" }>
| undefined;
let retainedOutput: AnalysisCapturedOutput | undefined;
let phase = "configuration";
let result: Result<AnalysisExecution, AnalysisError>;
const workerTimeoutMs =
input.worker_timeout_ms ?? DEFAULT_SQLITE_WORKER_TIMEOUT_MS;
try {
if (options?.signal?.aborted)
throw cancelledBy(operation, options.signal);
validateSqliteInput(input);
const retired = await this.#retryCleanup();
if (!retired.ok) return retired;
if (options?.signal?.aborted)
throw cancelledBy(operation, options.signal);
if (this.#closed)
throw cancelledBy(operation, OwnerCancellation.ownerClosed);
root = await this.createRuntime();
phase = "artifact-read";
const snapshot = await captureSqliteDatabaseSnapshot(
input.path,
root.path,
this.#artifactResources,
options?.signal,
);
phase = "worker";
const completed = await executeSqliteDatabaseWorker({
input,
root: root.path,
snapshotPath: snapshot.snapshotPath,
timeoutMs: workerTimeoutMs,
environment: this.environment,
resourceScope: this.#artifactResources,
...(this.launcher === undefined ? {} : { launcher: this.launcher }),
...(options?.signal === undefined ? {} : { signal: options.signal }),
captured: (output) => {
retainedOutput = output;
},
retainCleanup: (owner, result) => {
processOwner = owner;
processCleanup = result;
},
});
const validated = sqliteDatabaseSchema.safeParse({
artifact: snapshot.artifact,
wal: snapshot.wal,
...completed.inspection,
worker_timeout_ms: workerTimeoutMs,
});
if (!validated.success)
throw sqliteReplyFinalizationFailure(
new AnalysisOutputError(
operation,
"SQLite worker report dimensions or identity fields are invalid",
retainedOutput === undefined
? undefined
: { capturedOutput: retainedOutput },
),
completed.finalization,
);
const execution = createSqliteObservation(validated.data, input.path);
result =
completed.finalization === undefined
? ok(execution)
: err(
completed.finalization.retainPartialObservation(
completedSqliteEvidence(input, execution),
),
);
} catch (cause: unknown) {
// A failure rethrown after retainCleanup carries no owner of its own;
// keep the retained worker instead of discarding it.
if (
cause instanceof OwnedCommandFailure &&
cause.cleanupOwner !== undefined
)
processOwner = cause.cleanupOwner;
const failure = sqliteDatabaseFailure(
cause,
input.path,
phase,
options?.signal,
workerTimeoutMs,
);
retainedOutput ??= failure.capturedOutput;
result = err(failure);
}
if (root !== undefined) {
if (processOwner !== undefined) {
const ownership = processOwner.launch.ownership;
const rootRelease = new AnalysisCleanupRelease();
const rootFailure = rootRelease.bind(sqliteRootFailure(root));
const processReason =
processCleanup?.reason ?? "SQLite worker cleanup remains unresolved.";
const processRelease = new AnalysisCleanupRelease();
const processFailure = processRelease.bind(
new ProviderCleanupError(
SQLITE_PROVIDER_IDENTITY.id,
ownership === undefined
? ["owned-sqlite-worker"]
: ownedCommandResources(ownership),
{ reason: processReason },
{ operation },
),
);
this.#pendingCleanup.set(root, {
input,
result,
output: retainedOutput,
rootFailure,
rootRelease,
processFailure,
processRelease,
process: processOwner,
});
if (processCleanup !== undefined) {
const failure = sqliteCleanupFailure(
ownership === undefined
? ["owned-sqlite-worker", root.path]
: ownedCommandResources(ownership, root.path),
processCleanup.reason,
result,
input,
retainedOutput,
processCleanup.cleanupResult === undefined
? undefined
: {
...(processCleanup.cleanupResult.failures === undefined
? {}
: {
failures: processCleanup.cleanupResult.failures.map(
(failure) => ({ ...failure }),
),
}),
...(processCleanup.cleanupResult.unverified === undefined
? {}
: {
unverified: processCleanup.cleanupResult.unverified.map(
(process) => ({ ...process }),
),
}),
},
);
failure.bindCleanupDependencies([processFailure, rootFailure]);
return err(failure);
}
if (!result.ok) {
result.error.bindCleanupDependencies([processFailure, rootFailure]);
return result;
}
const failure = sqliteCleanupFailure(
[
...(ownership === undefined
? ["owned-sqlite-worker"]
: ownedCommandResources(ownership)),
root.path,
],
processReason,
result,
input,
retainedOutput,
);
failure.bindCleanupDependencies([processFailure, rootFailure]);
return err(failure);
}
const cleaned = await cleanupSqliteRoot(
root,
result,
input,
retainedOutput,
);
if (!cleaned.ok) {
const rootRelease = new AnalysisCleanupRelease();
const rootFailure = rootRelease.bind(sqliteRootFailure(root));
cleaned.error.bindCleanupDependencies([rootFailure]);
this.#pendingCleanup.set(root, {
input,
result,
output: retainedOutput,
rootFailure,
rootRelease,
processFailure: undefined,
processRelease: new AnalysisCleanupRelease(),
process: undefined,
});
return cleaned;
}
}
if (result.ok && options?.signal?.aborted)
return err(
cancelledBy(operation, options.signal, {
partialObservation: completedSqliteEvidence(input, result.value),
}),
);
return result;
}
}
const createSqliteObservation = (
report: ReturnType<typeof sqliteDatabaseSchema.parse>,
path: string,
): AnalysisExecution => {
const locations: AnalysisExecution["locations"][number][] = [
{ kind: "artifact-path", path },
];
if (report.wal !== null)
locations.push({ kind: "artifact-path", path: report.wal.path });
return createAnalysisExecution(
report,
{ ...SQLITE_PROVIDER_IDENTITY, version: `SQLite ${report.engine.version}` },
{
rawResult: null,
subject: { path, format: "file", sha256: report.artifact.sha256 },
locations,
limitations: report.limitations,
},
);
};
const cleanupSqliteRoot = async (
root: Pick<PrivateRuntimeRoot, "path" | "close">,
result: Result<AnalysisExecution, AnalysisError>,
input: InspectSqliteDatabaseInput,
output?: AnalysisCapturedOutput,
): Promise<Result<null, AnalysisError>> => {
try {
await root.close();
return ok(null);
} catch (cause: unknown) {
return err(
sqliteCleanupFailure(
[root.path],
cause instanceof Error ? cause.message : String(cause),
result,
input,
output,
undefined,
cause,
),
);
}
};
const sqliteCleanupFailure = (
resources: readonly string[],
reason: string,
result: SqliteOutcome,
input: InspectSqliteDatabaseInput,
output?: AnalysisCapturedOutput,
details?: Readonly<Record<string, JsonValue>>,
cause?: unknown,
): ProviderCleanupError => {
const partialObservation = result.ok
? completedSqliteEvidence(input, result.value)
: result.error.partialObservation;
return createSqliteCleanupError(
resources,
{
...details,
reason,
},
{
...(output === undefined ? {} : { capturedOutput: output }),
...(result.ok ? {} : { cleanupPrimary: result.error }),
...(cause === undefined ? {} : { cause }),
...(partialObservation === undefined ? {} : { partialObservation }),
},
);
};
const completedSqliteEvidence = (
input: InspectSqliteDatabaseInput,
execution: AnalysisExecution,
) =>
createAnalysisExecutionEvidence(
operation,
sqliteInspectionParameters(input),
execution,
"observed",
);
const validateSqliteInput = (input: InspectSqliteDatabaseInput): void => {
const parsed = inspectSqliteDatabaseInputSchema.safeParse(input);
if (!parsed.success)
throw analysisInputErrorFromIssues(operation, parsed.error.issues, input, {
cause: parsed.error,
});
};