experiment(kuna): preserve typed reference facts at their owner

Expose source-owned reference kind, endpoints, function identities and instruction text for experiments. Keep the existing address-only public result and retained-cache shape; materialize richer facts only when explicitly requested. Record prior art, composition rules and the distinction between fact, transport and compute units. Verified 18 producer boundary cases and an independently checked native interior-reference example.

Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
committed 2026-10-16 02:59:46 +00:00
1 parent 27f981352d
commit 2e19ece025
3 files changed
+238 -22

No files matched your search

+112
View File
@@ -0,0 +1,112 @@
# What might count as an RE atom?
This is an experiment record, not a claim that a new universal instruction set
has been discovered. APIs already expose many relevant primitives. The open
question here is how to make their evidence, cost and composition useful to an
agent investigating a real program.
## Existing mechanisms and what they establish
- Ghidra has a program database, `ReferenceManager.getReferencesTo`, function-body
ownership through `FunctionManager.getFunctionContaining`, listing objects and
decompiler IR. Fine-grained **engine APIs already exist**. A poor MCP projection
can still discard their metadata or require excessive round trips.
Sources: [ReferenceManager](https://ghidra.re/ghidra_docs/api/ghidra/program/model/symbol/ReferenceManager.html),
[FunctionManager](https://ghidra.re/ghidra_docs/api/ghidra/program/model/listing/FunctionManager.html).
- Binary Ninja distinguishes code/data, type and variable references, including
IL-level references. This shows why one undifferentiated `xref` is insufficient
for some questions. It does not establish that an MCP interface using these
APIs will be token-efficient or that another provider can reproduce the facts.
Sources: [type/variable references](https://binary.ninja/2021/04/12/type-and-variable-xrefs.html),
[BinaryView API](https://dev-api.binary.ninja/binaryninja.binaryview-module.html).
- Kuna's Rust engine and SLEIGH compiler derive from Ghidra, while its standalone
loader/analyzer tier and CLI are separate components. Port lineage does not
imply identical observations or a shared live program database.
Source: [NOTICE](https://github.com/Noelo-Lab/kuna/blob/fe01d6294eb0d1053ce343c8943646df88e143fc/NOTICE).
- In **tested Kuna 1.833**, an `xrefs` command loads a program and builds an
`XrefIndex` for its query. A different CLI invocation does that again.
`strings` with attribution instead loads/indexes once and joins references over
each string's byte extent. That is already internal composition, even though
the CLI does not expose an arbitrary persistent query interface.
Sources: [xref producer](https://github.com/Noelo-Lab/kuna/blob/fe01d6294eb0d1053ce343c8943646df88e143fc/decompiler/crates/kuna-cli/src/xrefs.rs),
[string attribution](https://github.com/Noelo-Lab/kuna/blob/fe01d6294eb0d1053ce343c8943646df88e143fc/decompiler/crates/kuna-cli/src/strings.rs).
These are source-derived observations. They are not measurements of current
upstream main, a guarantee of decompilation correctness, or a novelty claim.
## Granularity has at least three independent dimensions
| Dimension | Example | Consequence |
| --- | --- | --- |
| Fact unit | One typed reference with both endpoints and provenance | Can be retained and joined without reading unrelated C text |
| Transport unit | A batch of references for several selectors | Amortizes headers and agent turns; need not change native work |
| Compute unit | One image load and reference walk | Can serve many fact queries; may cost much more than one returned row |
Calling all three "atomic tools" hides the main optimization opportunity. A tiny
getter can repeat a whole-program analysis. Conversely, one expensive shared
analysis can provide precise, small, independently addressable facts.
## Candidate semantic core
1. **Identified objects and located observations**: artifact, image/address-space,
byte span, decoded instruction, symbol, string candidate, recovered function,
variable and type. Keep width, encoding, storage and confidence where needed
to interpret a value. A recovered object is an analysis claim, not necessarily
a source-language object that existed before compilation.
2. **Relations and constraints**: maps-to, owns/contains, references, calls,
flows-to, defines/uses, argument binding, type/storage association and branch
guard. Keep source and destination, reference site, kind, supporting evidence
and unknown/partial coverage. Do not collapse a call site into just a callee.
3. **Evidence views**: retain the original observations and expose the requested
fields, range or neighborhood. A readable decompilation is an important view;
it does not replace a machine-grounded relation or prove original source.
Useful query operations resemble selection, projection, traversal, join and
aggregation. This is a design vocabulary, **not a proposal to add a SQL/graph
language or Code Mode**. Start with named, typed analyst questions; add more
general execution only if repeated real workflows demonstrate the need.
## Composition rules that prevent false conclusions
- Entity identity and an observation's identity differ. Artifact/profile/location
can identify a function candidate; `evidence_id` identifies one exact claim
about it. Keep disagreeing claims instead of deduplicating them by address.
- A session lease prevents using a handle after close/reopen. It is not a frozen
analysis revision. In particular, Kuna's whole-project and early single-function
reconstructions can differ under the same selected profile. Target admission
prevents a target switch; it does not establish a consistent analysis snapshot.
- A string spans bytes. A point-address xref and a reference anywhere inside that
object answer different questions. A data-pointer reference is not automatically
a runtime call, string consumption, or a reachable execution path.
- Omission, an unsupported operation, cancellation, an unowned reference and an
examined empty set are different outcomes. No empty set proves global absence
unless the producing analysis actually establishes that coverage.
- Field selection must not detach a value from the metadata that makes it
meaningful. For example, the first prototype keeps REA's complete `string`
metadata record as one selectable field, including its encoding status.
- Expensive producer work should be shared with explicit identity, lifetime,
invalidation and cancellation ownership. A client-side batch is insufficient
evidence that native work was shared.
## What complex-program effectiveness would require
Navigation is only the first tier. Finding a string and its users is weaker than
explaining why a branch rejects input or how data reaches a privileged operation.
The next tier needs definition/use, call-argument and memory-flow relationships,
with explicit analysis assumptions. The existing prototypes do **not** supply
those relationships, and slicing C text cannot manufacture them.
For every harder trial, specify the analyst question, independent answer/evidence
oracle, input artifact/profile, initial knowledge and allowed budget. Compare:
- whether the actual question was answered correctly;
- decisive evidence recovered and relevant evidence omitted;
- remaining unknowns and unsafe conclusions;
- all calls, follow-up reads, failures, latency, CPU, family memory and estimated
token traffic, including tool discovery and retained Evidence reads.
Source inspection can establish architectural opportunities. Scripted equivalent
workflows can establish transport and engine costs. Neither alone establishes
that a live agent reasons better on complex software; a controlled agent trial
is a separate validation step. Likewise, byte/token reductions alone do not
establish improved correctness or analyst productivity.
+93 -22
View File
@@ -159,6 +159,28 @@ const xrefsDocument = z.object({
xrefs: z.array(z.unknown()),
});
/** One owning interpretation of native endpoints, identities and alias scope. */
const parseReferenceRow = (raw: unknown, aliases: ReadonlySet<string>) => {
const row = xrefRow.parse(raw);
if (
row.address !== row.address_hex ||
row.address !== row.from_address ||
row.from_address !== row.from_address_hex ||
row.to_address !== row.to_address_hex
)
throw new Error(
"xref endpoints disagree with numeric or directional fields",
);
for (const fn of [row.from_function, row.to_function])
if (fn !== null && fn.address !== fn.address_hex)
throw new Error("xref function address disagrees with its hex field");
if (!aliases.has(row.to_address))
throw new Error(
"xref row targets neither the requested address nor an advertised alias",
);
return row;
};
export interface KunaIncomingReferences {
readonly addresses: readonly string[];
readonly sha256: string;
@@ -167,12 +189,32 @@ export interface KunaIncomingReferences {
readonly excludedAliasRows: number;
}
/** Native --to unifies forwarding aliases; REA xrefs asks for the exact address. */
export const parseKunaIncomingReferences = (
/** Experimental fact projection, owned by the native protocol parser. */
export type KunaReferenceFact = {
readonly from: string;
readonly to: string;
readonly kind: z.infer<typeof kind>;
readonly fromFunction: {
readonly address: string;
readonly name: string;
} | null;
readonly toFunction: {
readonly address: string;
readonly name: string;
} | null;
readonly instruction: string;
};
/** Native --to unifies aliases; these facts cover only the requested address. */
const parseIncomingReferenceDocument = (
text: string,
input: string,
target: string,
): Result<KunaIncomingReferences, string> => {
retainFacts: boolean,
): Result<
KunaIncomingReferences & { readonly facts: readonly KunaReferenceFact[] },
string
> => {
try {
const document = xrefsDocument.parse(
parseKunaJson(text, KUNA_XREF_OUTPUT_BYTES, addressKeys),
@@ -192,27 +234,34 @@ export const parseKunaIncomingReferences = (
aliases.add(alias.address);
}
const addresses = new Set<string>();
const facts: KunaReferenceFact[] = [];
let excludedAliasRows = 0;
for (const raw of document.xrefs) {
const row = xrefRow.parse(raw);
if (
row.address !== row.address_hex ||
row.address !== row.from_address ||
row.from_address !== row.from_address_hex ||
row.to_address !== row.to_address_hex
)
throw new Error(
"xref endpoints disagree with numeric or directional fields",
);
for (const fn of [row.from_function, row.to_function])
if (fn !== null && fn.address !== fn.address_hex)
throw new Error("xref function address disagrees with its hex field");
if (!aliases.has(row.to_address))
throw new Error(
"xref row targets neither the requested address nor an advertised alias",
);
if (row.to_address === target) addresses.add(row.from_address);
else excludedAliasRows += 1;
const row = parseReferenceRow(raw, aliases);
if (row.to_address === target) {
addresses.add(row.from_address);
if (retainFacts)
facts.push({
from: row.from_address,
to: row.to_address,
kind: row.kind,
fromFunction:
row.from_function === null
? null
: {
address: row.from_function.address,
name: row.from_function.name,
},
toFunction:
row.to_function === null
? null
: {
address: row.to_function.address,
name: row.to_function.name,
},
instruction: row.instruction,
});
} else excludedAliasRows += 1;
}
return ok({
addresses: [...addresses].sort(compareKunaAddresses),
@@ -220,8 +269,30 @@ export const parseKunaIncomingReferences = (
excludedAliasRows,
bytes: Buffer.byteLength(text),
sha256: createHash("sha256").update(text).digest("hex"),
facts,
});
} catch (cause: unknown) {
return err(cause instanceof Error ? cause.message : String(cause));
}
};
export const parseKunaIncomingReferenceFacts = (
text: string,
input: string,
target: string,
): Result<
KunaIncomingReferences & { readonly facts: readonly KunaReferenceFact[] },
string
> => parseIncomingReferenceDocument(text, input, target, true);
/** Existing address-only contract; richer facts do not enter its retained cache. */
export const parseKunaIncomingReferences = (
text: string,
input: string,
target: string,
): Result<KunaIncomingReferences, string> => {
const parsed = parseIncomingReferenceDocument(text, input, target, false);
if (!parsed.ok) return parsed;
const { addresses, sha256, bytes, rows, excludedAliasRows } = parsed.value;
return ok({ addresses, sha256, bytes, rows, excludedAliasRows });
};
@@ -4,6 +4,7 @@ import { normalizeKunaAddress } from "../../../../src/kuna/KunaAddress.js";
import {
parseKunaCallGraph,
parseKunaIncomingReferences,
parseKunaIncomingReferenceFacts,
} from "../../../../src/kuna/KunaReferenceFiles.js";
const input = "/snapshot/program";
@@ -189,6 +190,38 @@ it("keeps exact endpoints, deduplicates sites and counts excluded forwarding ali
});
});
it("preserves typed relationships and explicit unknown ownership at the producing boundary", () => {
const value = xrefs();
value.xrefs[1]!.kind = "data";
const parsed = parseKunaIncomingReferenceFacts(wire(value), input, "0x2000");
expect(parsed.ok).toBe(true);
if (!parsed.ok) return;
expect(parsed.value.facts).toEqual([
{
from: "0x1004",
to: "0x2000",
kind: "call",
fromFunction: { address: "0x1000", name: "caller" },
toFunction: null,
instruction: "CALL",
},
{
from: "0x1004",
to: "0x2000",
kind: "data",
fromFunction: { address: "0x1000", name: "caller" },
toFunction: null,
instruction: "CALL",
},
]);
const addresses = parseKunaIncomingReferences(wire(value), input, "0x2000");
expect(addresses.ok && Object.hasOwn(addresses.value, "facts")).toBe(false);
value.xrefs[2]!.from_address_hex = "0xdead";
expect(parseKunaIncomingReferenceFacts(wire(value), input, "0x2000").ok).toBe(
false,
);
});
it.each([
[
"count",