mirror of
https://github.com/morluto/rea.git
synced 2026-11-21 14:05:49 +00:00
experiment(kuna): preserve typed reference facts at their owner
Expose source-owned reference kind, endpoints, function identities and instruction text for experiments. Keep the existing address-only public result and retained-cache shape; materialize richer facts only when explicitly requested. Record prior art, composition rules and the distinction between fact, transport and compute units. Verified 18 producer boundary cases and an independently checked native interior-reference example. Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
1 parent
27f981352d
commit
2e19ece025
3 files changed
+238
-22
No files matched your search
@@ -0,0 +1,112 @@
|
||||
# What might count as an RE atom?
|
||||
|
||||
This is an experiment record, not a claim that a new universal instruction set
|
||||
has been discovered. APIs already expose many relevant primitives. The open
|
||||
question here is how to make their evidence, cost and composition useful to an
|
||||
agent investigating a real program.
|
||||
|
||||
## Existing mechanisms and what they establish
|
||||
|
||||
- Ghidra has a program database, `ReferenceManager.getReferencesTo`, function-body
|
||||
ownership through `FunctionManager.getFunctionContaining`, listing objects and
|
||||
decompiler IR. Fine-grained **engine APIs already exist**. A poor MCP projection
|
||||
can still discard their metadata or require excessive round trips.
|
||||
Sources: [ReferenceManager](https://ghidra.re/ghidra_docs/api/ghidra/program/model/symbol/ReferenceManager.html),
|
||||
[FunctionManager](https://ghidra.re/ghidra_docs/api/ghidra/program/model/listing/FunctionManager.html).
|
||||
- Binary Ninja distinguishes code/data, type and variable references, including
|
||||
IL-level references. This shows why one undifferentiated `xref` is insufficient
|
||||
for some questions. It does not establish that an MCP interface using these
|
||||
APIs will be token-efficient or that another provider can reproduce the facts.
|
||||
Sources: [type/variable references](https://binary.ninja/2021/04/12/type-and-variable-xrefs.html),
|
||||
[BinaryView API](https://dev-api.binary.ninja/binaryninja.binaryview-module.html).
|
||||
- Kuna's Rust engine and SLEIGH compiler derive from Ghidra, while its standalone
|
||||
loader/analyzer tier and CLI are separate components. Port lineage does not
|
||||
imply identical observations or a shared live program database.
|
||||
Source: [NOTICE](https://github.com/Noelo-Lab/kuna/blob/fe01d6294eb0d1053ce343c8943646df88e143fc/NOTICE).
|
||||
- In **tested Kuna 1.833**, an `xrefs` command loads a program and builds an
|
||||
`XrefIndex` for its query. A different CLI invocation does that again.
|
||||
`strings` with attribution instead loads/indexes once and joins references over
|
||||
each string's byte extent. That is already internal composition, even though
|
||||
the CLI does not expose an arbitrary persistent query interface.
|
||||
Sources: [xref producer](https://github.com/Noelo-Lab/kuna/blob/fe01d6294eb0d1053ce343c8943646df88e143fc/decompiler/crates/kuna-cli/src/xrefs.rs),
|
||||
[string attribution](https://github.com/Noelo-Lab/kuna/blob/fe01d6294eb0d1053ce343c8943646df88e143fc/decompiler/crates/kuna-cli/src/strings.rs).
|
||||
|
||||
These are source-derived observations. They are not measurements of current
|
||||
upstream main, a guarantee of decompilation correctness, or a novelty claim.
|
||||
|
||||
## Granularity has at least three independent dimensions
|
||||
|
||||
| Dimension | Example | Consequence |
|
||||
| --- | --- | --- |
|
||||
| Fact unit | One typed reference with both endpoints and provenance | Can be retained and joined without reading unrelated C text |
|
||||
| Transport unit | A batch of references for several selectors | Amortizes headers and agent turns; need not change native work |
|
||||
| Compute unit | One image load and reference walk | Can serve many fact queries; may cost much more than one returned row |
|
||||
|
||||
Calling all three "atomic tools" hides the main optimization opportunity. A tiny
|
||||
getter can repeat a whole-program analysis. Conversely, one expensive shared
|
||||
analysis can provide precise, small, independently addressable facts.
|
||||
|
||||
## Candidate semantic core
|
||||
|
||||
1. **Identified objects and located observations**: artifact, image/address-space,
|
||||
byte span, decoded instruction, symbol, string candidate, recovered function,
|
||||
variable and type. Keep width, encoding, storage and confidence where needed
|
||||
to interpret a value. A recovered object is an analysis claim, not necessarily
|
||||
a source-language object that existed before compilation.
|
||||
2. **Relations and constraints**: maps-to, owns/contains, references, calls,
|
||||
flows-to, defines/uses, argument binding, type/storage association and branch
|
||||
guard. Keep source and destination, reference site, kind, supporting evidence
|
||||
and unknown/partial coverage. Do not collapse a call site into just a callee.
|
||||
3. **Evidence views**: retain the original observations and expose the requested
|
||||
fields, range or neighborhood. A readable decompilation is an important view;
|
||||
it does not replace a machine-grounded relation or prove original source.
|
||||
|
||||
Useful query operations resemble selection, projection, traversal, join and
|
||||
aggregation. This is a design vocabulary, **not a proposal to add a SQL/graph
|
||||
language or Code Mode**. Start with named, typed analyst questions; add more
|
||||
general execution only if repeated real workflows demonstrate the need.
|
||||
|
||||
## Composition rules that prevent false conclusions
|
||||
|
||||
- Entity identity and an observation's identity differ. Artifact/profile/location
|
||||
can identify a function candidate; `evidence_id` identifies one exact claim
|
||||
about it. Keep disagreeing claims instead of deduplicating them by address.
|
||||
- A session lease prevents using a handle after close/reopen. It is not a frozen
|
||||
analysis revision. In particular, Kuna's whole-project and early single-function
|
||||
reconstructions can differ under the same selected profile. Target admission
|
||||
prevents a target switch; it does not establish a consistent analysis snapshot.
|
||||
- A string spans bytes. A point-address xref and a reference anywhere inside that
|
||||
object answer different questions. A data-pointer reference is not automatically
|
||||
a runtime call, string consumption, or a reachable execution path.
|
||||
- Omission, an unsupported operation, cancellation, an unowned reference and an
|
||||
examined empty set are different outcomes. No empty set proves global absence
|
||||
unless the producing analysis actually establishes that coverage.
|
||||
- Field selection must not detach a value from the metadata that makes it
|
||||
meaningful. For example, the first prototype keeps REA's complete `string`
|
||||
metadata record as one selectable field, including its encoding status.
|
||||
- Expensive producer work should be shared with explicit identity, lifetime,
|
||||
invalidation and cancellation ownership. A client-side batch is insufficient
|
||||
evidence that native work was shared.
|
||||
|
||||
## What complex-program effectiveness would require
|
||||
|
||||
Navigation is only the first tier. Finding a string and its users is weaker than
|
||||
explaining why a branch rejects input or how data reaches a privileged operation.
|
||||
The next tier needs definition/use, call-argument and memory-flow relationships,
|
||||
with explicit analysis assumptions. The existing prototypes do **not** supply
|
||||
those relationships, and slicing C text cannot manufacture them.
|
||||
|
||||
For every harder trial, specify the analyst question, independent answer/evidence
|
||||
oracle, input artifact/profile, initial knowledge and allowed budget. Compare:
|
||||
|
||||
- whether the actual question was answered correctly;
|
||||
- decisive evidence recovered and relevant evidence omitted;
|
||||
- remaining unknowns and unsafe conclusions;
|
||||
- all calls, follow-up reads, failures, latency, CPU, family memory and estimated
|
||||
token traffic, including tool discovery and retained Evidence reads.
|
||||
|
||||
Source inspection can establish architectural opportunities. Scripted equivalent
|
||||
workflows can establish transport and engine costs. Neither alone establishes
|
||||
that a live agent reasons better on complex software; a controlled agent trial
|
||||
is a separate validation step. Likewise, byte/token reductions alone do not
|
||||
establish improved correctness or analyst productivity.
|
||||
@@ -159,6 +159,28 @@ const xrefsDocument = z.object({
|
||||
xrefs: z.array(z.unknown()),
|
||||
});
|
||||
|
||||
/** One owning interpretation of native endpoints, identities and alias scope. */
|
||||
const parseReferenceRow = (raw: unknown, aliases: ReadonlySet<string>) => {
|
||||
const row = xrefRow.parse(raw);
|
||||
if (
|
||||
row.address !== row.address_hex ||
|
||||
row.address !== row.from_address ||
|
||||
row.from_address !== row.from_address_hex ||
|
||||
row.to_address !== row.to_address_hex
|
||||
)
|
||||
throw new Error(
|
||||
"xref endpoints disagree with numeric or directional fields",
|
||||
);
|
||||
for (const fn of [row.from_function, row.to_function])
|
||||
if (fn !== null && fn.address !== fn.address_hex)
|
||||
throw new Error("xref function address disagrees with its hex field");
|
||||
if (!aliases.has(row.to_address))
|
||||
throw new Error(
|
||||
"xref row targets neither the requested address nor an advertised alias",
|
||||
);
|
||||
return row;
|
||||
};
|
||||
|
||||
export interface KunaIncomingReferences {
|
||||
readonly addresses: readonly string[];
|
||||
readonly sha256: string;
|
||||
@@ -167,12 +189,32 @@ export interface KunaIncomingReferences {
|
||||
readonly excludedAliasRows: number;
|
||||
}
|
||||
|
||||
/** Native --to unifies forwarding aliases; REA xrefs asks for the exact address. */
|
||||
export const parseKunaIncomingReferences = (
|
||||
/** Experimental fact projection, owned by the native protocol parser. */
|
||||
export type KunaReferenceFact = {
|
||||
readonly from: string;
|
||||
readonly to: string;
|
||||
readonly kind: z.infer<typeof kind>;
|
||||
readonly fromFunction: {
|
||||
readonly address: string;
|
||||
readonly name: string;
|
||||
} | null;
|
||||
readonly toFunction: {
|
||||
readonly address: string;
|
||||
readonly name: string;
|
||||
} | null;
|
||||
readonly instruction: string;
|
||||
};
|
||||
|
||||
/** Native --to unifies aliases; these facts cover only the requested address. */
|
||||
const parseIncomingReferenceDocument = (
|
||||
text: string,
|
||||
input: string,
|
||||
target: string,
|
||||
): Result<KunaIncomingReferences, string> => {
|
||||
retainFacts: boolean,
|
||||
): Result<
|
||||
KunaIncomingReferences & { readonly facts: readonly KunaReferenceFact[] },
|
||||
string
|
||||
> => {
|
||||
try {
|
||||
const document = xrefsDocument.parse(
|
||||
parseKunaJson(text, KUNA_XREF_OUTPUT_BYTES, addressKeys),
|
||||
@@ -192,27 +234,34 @@ export const parseKunaIncomingReferences = (
|
||||
aliases.add(alias.address);
|
||||
}
|
||||
const addresses = new Set<string>();
|
||||
const facts: KunaReferenceFact[] = [];
|
||||
let excludedAliasRows = 0;
|
||||
for (const raw of document.xrefs) {
|
||||
const row = xrefRow.parse(raw);
|
||||
if (
|
||||
row.address !== row.address_hex ||
|
||||
row.address !== row.from_address ||
|
||||
row.from_address !== row.from_address_hex ||
|
||||
row.to_address !== row.to_address_hex
|
||||
)
|
||||
throw new Error(
|
||||
"xref endpoints disagree with numeric or directional fields",
|
||||
);
|
||||
for (const fn of [row.from_function, row.to_function])
|
||||
if (fn !== null && fn.address !== fn.address_hex)
|
||||
throw new Error("xref function address disagrees with its hex field");
|
||||
if (!aliases.has(row.to_address))
|
||||
throw new Error(
|
||||
"xref row targets neither the requested address nor an advertised alias",
|
||||
);
|
||||
if (row.to_address === target) addresses.add(row.from_address);
|
||||
else excludedAliasRows += 1;
|
||||
const row = parseReferenceRow(raw, aliases);
|
||||
if (row.to_address === target) {
|
||||
addresses.add(row.from_address);
|
||||
if (retainFacts)
|
||||
facts.push({
|
||||
from: row.from_address,
|
||||
to: row.to_address,
|
||||
kind: row.kind,
|
||||
fromFunction:
|
||||
row.from_function === null
|
||||
? null
|
||||
: {
|
||||
address: row.from_function.address,
|
||||
name: row.from_function.name,
|
||||
},
|
||||
toFunction:
|
||||
row.to_function === null
|
||||
? null
|
||||
: {
|
||||
address: row.to_function.address,
|
||||
name: row.to_function.name,
|
||||
},
|
||||
instruction: row.instruction,
|
||||
});
|
||||
} else excludedAliasRows += 1;
|
||||
}
|
||||
return ok({
|
||||
addresses: [...addresses].sort(compareKunaAddresses),
|
||||
@@ -220,8 +269,30 @@ export const parseKunaIncomingReferences = (
|
||||
excludedAliasRows,
|
||||
bytes: Buffer.byteLength(text),
|
||||
sha256: createHash("sha256").update(text).digest("hex"),
|
||||
facts,
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
return err(cause instanceof Error ? cause.message : String(cause));
|
||||
}
|
||||
};
|
||||
|
||||
export const parseKunaIncomingReferenceFacts = (
|
||||
text: string,
|
||||
input: string,
|
||||
target: string,
|
||||
): Result<
|
||||
KunaIncomingReferences & { readonly facts: readonly KunaReferenceFact[] },
|
||||
string
|
||||
> => parseIncomingReferenceDocument(text, input, target, true);
|
||||
|
||||
/** Existing address-only contract; richer facts do not enter its retained cache. */
|
||||
export const parseKunaIncomingReferences = (
|
||||
text: string,
|
||||
input: string,
|
||||
target: string,
|
||||
): Result<KunaIncomingReferences, string> => {
|
||||
const parsed = parseIncomingReferenceDocument(text, input, target, false);
|
||||
if (!parsed.ok) return parsed;
|
||||
const { addresses, sha256, bytes, rows, excludedAliasRows } = parsed.value;
|
||||
return ok({ addresses, sha256, bytes, rows, excludedAliasRows });
|
||||
};
|
||||
@@ -4,6 +4,7 @@ import { normalizeKunaAddress } from "../../../../src/kuna/KunaAddress.js";
|
||||
import {
|
||||
parseKunaCallGraph,
|
||||
parseKunaIncomingReferences,
|
||||
parseKunaIncomingReferenceFacts,
|
||||
} from "../../../../src/kuna/KunaReferenceFiles.js";
|
||||
|
||||
const input = "/snapshot/program";
|
||||
@@ -189,6 +190,38 @@ it("keeps exact endpoints, deduplicates sites and counts excluded forwarding ali
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves typed relationships and explicit unknown ownership at the producing boundary", () => {
|
||||
const value = xrefs();
|
||||
value.xrefs[1]!.kind = "data";
|
||||
const parsed = parseKunaIncomingReferenceFacts(wire(value), input, "0x2000");
|
||||
expect(parsed.ok).toBe(true);
|
||||
if (!parsed.ok) return;
|
||||
expect(parsed.value.facts).toEqual([
|
||||
{
|
||||
from: "0x1004",
|
||||
to: "0x2000",
|
||||
kind: "call",
|
||||
fromFunction: { address: "0x1000", name: "caller" },
|
||||
toFunction: null,
|
||||
instruction: "CALL",
|
||||
},
|
||||
{
|
||||
from: "0x1004",
|
||||
to: "0x2000",
|
||||
kind: "data",
|
||||
fromFunction: { address: "0x1000", name: "caller" },
|
||||
toFunction: null,
|
||||
instruction: "CALL",
|
||||
},
|
||||
]);
|
||||
const addresses = parseKunaIncomingReferences(wire(value), input, "0x2000");
|
||||
expect(addresses.ok && Object.hasOwn(addresses.value, "facts")).toBe(false);
|
||||
value.xrefs[2]!.from_address_hex = "0xdead";
|
||||
expect(parseKunaIncomingReferenceFacts(wire(value), input, "0x2000").ok).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
"count",
|
||||
|
||||
Reference in new issue
Block a user