* feat(sqlite): page selected rows with a stated order and row offset
readRows read `NOT INDEXED LIMIT ?` with no ORDER BY and reported
order: "unspecified", so an agent could not reach rows past row_limit
deterministically: raising row_limit hits the fixed 16 MiB reply budget,
and the sampled order depends on whichever index SQLite happened to scan.
Read the window in a unique, stated order and expose the position:
- rowid tables order by rowid unless all of rowid, _rowid_ and oid are
declared columns, in which case the order stays unspecified.
- WITHOUT ROWID tables order by the primary-key columns, reusing each
key's DESC flag and declared collation from pragma_index_xinfo so the
plan keeps index order instead of materialising a sorter.
- rows.order, rows.row_offset and rows.next_row_offset report the window;
next_row_offset is null once the window reaches the end.
- row_offset (CLI --row-offset) requires table and defaults to 0. A
non-zero offset is refused when the table declares no usable order,
because the skipped prefix would be arbitrary.
The schema now checks next_row_offset against truncated and the offset so
a worker reply cannot claim a page it did not read.
One attribution fix came with it: adding ORDER BY text can push an
over-wide projection past SQLite's SQL-text limit, which fails as
SQLITE_TOOBIG before the result-column limit is reported. Attribute that
case to column capacity too, so a 250k-column request still reads as
capacity rather than a byte budget.
Verified: npm run check:fast, npm run format:check on the touched files,
npm run verify:module-boundaries, npm run docs:check, and 196 tests
across the sqlite boundary, application and CLI/MCP suites. A 1002-row
inspection and in-memory EXPLAIN QUERY PLAN checks confirmed the
WITHOUT ROWID path adds no sorter. Full check:pr and the complete test
suite were not run.
* fix(sqlite): resolve the rowid alias and keep paging claims reachable
Three defects in the row paging added alongside row_offset:
- readRowOrder emitted ORDER BY rowid for every rowid table, but a table
that declares a rowid column shadows that spelling, so the sort used the
stored column and could repeat or drop rows across pages. Order now names
the first alias the table leaves unbound, read from raw name bytes so an
unrelated malformed identifier is not decoded just to choose an order.
- The inventory projection could hide the primary key a WITHOUT ROWID table
orders by, silently degrading a paginated read to unspecified order. The
order key is now read from the table itself rather than from the columns
the caller asked to see.
- A truncated table with no stated order reported a next_row_offset that the
row_offset rule then refused. It is null there, and the reply limitation
no longer claims an unspecified order for windows that have one.
rowEvaluationFailure also probed the first rows of the table instead of the
requested window, so a failure confined to a later page was attributed to
columns that evaluate fine there.
* test(sqlite): assert the concatenated pages
The paging regression checked only counts, offsets and flags, so a second
page that repeated the first page's rows would still pass. Assert the values
read across all three pages instead.
* fix(sqlite): decode row keys in the database encoding
readRowKeyFacts decoded primary-key names as UTF-8 and probed a fixed
hidden set, so two supported schemas could not be paged at all:
- A UTF-16 database stores key names as UTF-16 bytes, so "k" decoded to
NUL-padded text and ORDER BY named an identifier that does not exist.
Pass the detected encoding, as readColumns and the index-metadata path
already do.
- A virtual generated column named rowid (hidden=2) was absent from the
shadow check, so ORDER BY rowid evaluated the generated expression
instead of the rowid and could sort non-uniquely or fail outright.
Include generated columns in the alias probe.
Also stop decoding primary-key names for rowid tables: their ORDER BY
names an alias, never a declared key, so an unselected malformed key
name could abort an otherwise readable selection.
The page-concatenation assertion now narrows SqliteValue before reading
.value, which the null/blob arms do not carry; this was failing the
Static checks typecheck job.
* fix(sqlite): order WITHOUT ROWID rows by the index's own key sequence
`readRowOrder` filed `pragma_index_xinfo` key terms into a map keyed by
column name, so a valid composite key that repeats a column —
`PRIMARY KEY(a COLLATE BINARY, a COLLATE NOCASE DESC)` — collapsed to one
term. The generated clause then overstated uniqueness: it omitted the
BINARY term, so `A`/`a` ties survived, and the `row_offset` paging
advertised a `primary_key` order that the actual key does not establish.
Build the terms from the index rows in `seqno` order instead, keeping
repetitions, and bail to `unspecified` if any declared key column never
appears in the index. The declared collation stays explicit so a caller
column's own collation cannot re-sort the window.
Verified against a real SQLite database that the repeated key is accepted
and that the old clause orders A, a, B while the index orders A, B, a.
* fix(sqlite): accept a null next offset on unordered truncated rows
readRows already withholds next_row_offset when a table shadows rowid, _rowid_ and oid and declares no primary key, because every non-zero offset is refused there. The reply schema still required the offset whenever truncated was set, so the provider rejected its own worker result and the first sample never reached CLI or MCP. Require the offset only when the order is stated.
* fix(sqlite): probe row keys only for ordinary tables and name row_offset
readRowKeyFacts ran for a selected virtual shadow table before readRows
could reject it, so an unavailable module surfaced as SQLite's
"no such module" instead of the selection refusal. Probe the key only
when the selected kind is an ordinary table and leave the order
unspecified otherwise, since no rows are read for that kind.
The unordered-offset refusal also reported the generic selection reason,
which callers see against table. Give it its own reason so the failure is
attributed to row_offset.
* feat(sqlite)!: page selected rows with a stated order and row offset
Selected-row inspection currently reports only rows.order: "unspecified" and
offers no way to reach rows past the first sample deterministically. Main also
refuses non-zero row_offset today, so nothing relies on that omission; this
change makes the stated order and the offset the contract, reports the next
offset the caller feeds back, and keeps the existing byte, heap, statement and
deadline limits.
BREAKING CHANGE: rows.order becomes rowid | primary_key | unspecified, and
rows gains required row_offset and next_row_offset members. A client that
validates selected-table output against the previous schema (where order was
the only permitted value) rejects a now-valid reply. Migration: accept the two
new members and treat order: "rowid" | "primary_key" as the stated order the
offset applies to; a null next_row_offset means the window reached the end.
* docs(sqlite): say a null next offset also means no stated order
The selected-rows limitation described next_row_offset as null only when the window reached the end. An unordered table whose row_limit truncates the sample also publishes null, because an offset there would name a request the row-offset rule refuses, so the Evidence read as if the sample were complete. State the second cause and pin it in the existing regression.
* test(sqlite): drop the limitation wording assertion and state the unordered exception
The boundary test pinned the sentence a caller reads in `limitations`,
so a harmless rewrite of that text failed a paging test whose real
subject is `order`, `truncated` and `next_row_offset`.
docs/development/testing.md asks for facts over frozen explanatory prose,
and the three structured expectations already fail if the paging fact
changes.
The tool contract promised that retained rows always have a unique order
and can be paged. A rowid table that shadows rowid, _rowid_ and oid and
declares no primary key has no such order: it returns
order=unspecified with a null next offset and refuses a non-zero
row_offset. Say so in the description so callers do not plan a
continuation the tool always refuses.
* docs(sqlite): say unordered tables cannot be paged
The description told callers that a truncated table without a stated order
needs "the table read without offsets", but omitting row_offset starts at
zero again: with an unchanged row_limit the next call repeats the first
sample instead of continuing past it. Say that these tables cannot be
paged and point at row_limit for a larger first sample.
* style(cli): wrap the native data type path description
The `inspect-native-data-type` argument description line is 82 columns, over the repository's 80-column `.oxfmtrc.json` limit, so `npm run format:check` fails on main and on every branch merged from it. It came in with #9465 and the following commits to this file did not touch it. Reformat the one call the formatter objects to; no behaviour change.
* fix(sqlite): bound rowid alias comparison by name byte length
readRowKeyFacts decoded every column name before testing it against three short aliases, so a table declaring a multi-megabyte name could exhaust the worker heap even when the caller projected the column away. Reject names that cannot be an alias by their byte length, accounting for the database encoding, before building any string.
---------
Co-authored-by: 2SinceJuly <202121119+2SinceJuly@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
* test(ci): require provider coverage and reuse test sessions
* fix(ci): prepare the runtime before Python producer tests
The fast Python job ran the suite on a fresh checkout, so analysis-efficiency
checks failed without compiled process modules. Install locked dependencies
and prepare the runtime before the suite.
* fix(ci): keep provider test reports inside the checkout
The provider-report reader accepted an arbitrary path from argv. Resolve the
path and reject anything outside the checkout before reading it. The packet
and signal workflows write those reports under .cache.
* fix(ci): require a test module in each named Python bridge family
Discovery still runs every bridge test. The mitmproxy, native, and pwndbg
families must each keep a test module, so deleting one cannot pass through
the others.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Reuse the pinned SDK's combined array-element and object-member admission
before canonical validation, deferred runtime loading and tool execution.
Share a configurable finite allowance across eager/deferred HTTP and stdio,
and project refusals to bounded, actionable transport resource errors.
Preserve canonical CLI/application contracts, caller-selected input values,
ordinary validation semantics and complete Evidence access. Exercise actual
transport boundaries, real artifact/TShark workflows, packaged installation,
continued service and downstream identity/capacity regressions.
Fixes#7882
BREAKING CHANGE: MCP tool arguments now default to 1,024 combined array
elements and object members per call. Select a larger positive finite
REA_MCP_MAX_INPUT_ELEMENTS allowance and restart for larger inline inputs,
or use retained/file selectors where supported. Inputs are never truncated.
Assisted-by: Reimu (gensokyo-skills:reimu-incident-triage)
Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
Assisted-by: Yukari (gensokyo-skills:yukari-boundary-analysis)
* fix(mcp)!: return a compact binary_session status by default
binary_session {} with no target open returned about 66 KB, most of it
provider rejection diagnostics and candidate capability sets that the
skill's recovery flow re-reads on every retry. 6.8.0 grew it by 21%
because the shared regex capability limitation began repeating setting
bounds and step-by-step recovery for every search operation of every
provider (#6099, #6267, #6301, #7049) and IDA gained that limitation for
its two search operations (#4862).
The regex limitation now states the effective deadline and heap, their
sources and where to change them; the deadline and heap errors keep the
full recovery. binary_session gains detail: summary (default) or
complete. The summary keeps open state, target identity, the binding,
route capabilities, run, activity and health status, each candidate's
availability and target-support status, code and reason, and every
tool's availability, reason, remediation and modes, and lists the
fields it leaves out in omitted_fields. Complete detail returns the
previous shape. The CLI capabilities and providers commands keep
complete provider facts through the same application projection.
BREAKING CHANGE: binary_session {} no longer returns
analysis_provider_candidates[].availability.diagnostics,
target_support.diagnostics or capabilities, or tool_availability group
surface, effects and annotations. Pass {"detail": "complete"} to get
them.
Fixes#7109
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mcp): key binary_session output fields by detail
The output schema accepted detail: "complete" with summary candidates or
tool groups, and a summary without omitted_fields, so detail did not
tell callers which fields existed. Each session kind is now crossed
with a detail-specific shape: a summary requires omitted_fields and
summary-only candidates and tool groups; complete requires diagnostics,
candidate capabilities and static tool flags and forbids omitted_fields.
Both the canonical schema and the SDK-validated advertised schema reject
mismatched shapes.
The verifier helper accepts summary candidates only for a validated MCP
summary; CLI provider output still requires candidate capabilities.
Regex limitation tests check values, sources and setting names instead
of whole sentences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Charge the whole-catalog admission guard by escaped decoded names, preserving raw-byte lookup identity and final worker reply limits. Keep UTF-16 inventory and detailed-schema results whose returned facts fit the documented allowance.
Fixes#6888
Use one bound JSON selection set for inventory metadata, keeping raw-name byte
comparison and filtering inside SQLite while preserving caller projection order
and repetition. Large requests reach native result-column capacity attribution.
Separate unavailable row sources from generated-expression failures. Preserve
SQLite's reason and selected table/path, offer schema-only recovery, and avoid
claiming corruption or recommending unreadable columns.
Validated UTF-8/UTF-16 identity, native inspection/worker tests and CLI/MCP failure
and recovery workflows. The large CLI selection uses the existing framed CLI
fixture to avoid host argv limits. Source bytes and identity remain unchanged.
Fixes#6855Fixes#6604
Match table metadata by stored name bytes and fold only ASCII code units for relationships, reusing parsed facts without per-trigger queries. Preserve caller paths on structured worker limits and V8 heap failures.
Validated invalid UTF-8 collisions, distinct UTF-16 names and real worker transport refusal through CLI and stdio MCP. Controlled stderr covers OOM classification; actual memory exhaustion was not induced.
Fixes#6795Fixes#6607
Retain bounded FAT and offset-tagged PE observations through archive hashing,
classification, and content nodes. Preserve PyInstaller member paths according
to the observed carrier dialect and share APK v1 signing-name semantics.
Read SQLite's implicit sqlite_schema table without inventing a stored schema
declaration. Distinguish unsupported ELF numbering and decoded-name admission
from malformed data and fixed scan allowances. Preserve Sigrok basenames and
classify matching explicit input-format refusals at their owning boundary.
Validation: 554 distinct tests passed, 3 skipped across serial focused runs;
build, typecheck/lint, docs, formatting, and discovery (2141 files) passed.
Sigrok command-boundary simulators do not establish real-tool decoding.
Small repeated public PyInstaller CLI runs had equivalent inventory facts;
warm median elapsed 0.5431s before / 0.5426s after, with no speed claim.
Fixes#6733Fixes#6741Fixes#6722Fixes#6732Fixes#6736Fixes#6734Fixes#6764Fixes#6766
Refs #6739
* fix(sqlite): attribute generated-column evaluation failures to the selection
A selected virtual generated column whose expression needs a function the
read-only connection lacks (or raises an SQL error for a sampled row) was
classified by the worker's catch-all as a malformed database and projected as
an invalid_format issue on `path`.
Row reads now attribute SQLITE_ERROR from selected cells by probing each
selected virtual generated column over the sampled rows, and report a
typed capability_unavailable refusal naming the table, each failing column
with SQLite's message, and the remaining columns to select. Other causes,
including corruption, TOOBIG and NOMEM, keep their classification.
Fixes#5719
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(sqlite): recommend only probed-readable columns after evaluation failures
Record every per-column probe failure with its own reason, discover stored
columns outside an inventory-scope selection without decoding unrelated
malformed names, and present recovery as the structured columns selection
instead of shell-quoted command text built from database-supplied names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Expose a per-call SQLite worker deadline with exact CLI/MCP recovery controls,
parent-owned execution metadata and historical unknown preservation. Keep fixed
resource limits and cleanup admission intact. Share typed cleanup composition
with one captured-output owner and preserve the timeout after cleanup resolves.
Fixes#5812. Follow-up tracking: #5808. Existing CLI test-oracle failure: #5845.
Verified 155 SQLite tests (3 skipped), static/docs/discovery checks, six real
CLI/MCP pairs and 13 actual Codex sessions including four corrected deadline
recoveries. Three pre-dispatch discovery failures remain explicit, not passes.
Review findings include real retained-cleanup and fixture ownership regressions.
Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
The owned worker inspects only a private snapshot, so its format reason
cannot name the caller's selection. Project path-scoped worker rejections
with the selected database path so distinct malformed inputs report their
own source, matching the documented failure contract.
Fixes#5520
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Refuse oversized database and WAL inputs before acquisition with exact observed bytes and export guidance. Keep initial path failures separate from file-set changes and retain cleanup and partial observations on late growth.
Refs #4208, #4188, #4141, #4127
Co-authored-by: hima <1ep23ic023@eastpoint.ac.in>
Co-authored-by: Royce <roycelam@umich.edu>
Add offline SQLite database/WAL inspection through the shared CLI/MCP.
Use compact inventory, selected-table schema, exact ordered columns and
bounded row samples while preserving source identity and lossless values.
Retain completed Evidence through late cancellation and private-root cleanup
failures, bound concurrent inspections, and recheck journal validity after
acquisition. Reuse prepared CI runtime artifacts across supported Node lines.
Co-authored-by: Y0UW1SH <29498139+Y0UW1SH@users.noreply.github.com>
Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
