OpenSource/reverse-engineering-anything
1
0 Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-11-09 16:26:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
1f9bf8af0ec5cc39f896cf81f751609304cdb10c
reverse-engineering-anything/docs/guides/data/sqlite-databases.md
T
2SinceJuly2SinceJulymorluto
 3230f50b16 feat(sqlite)!: page selected rows with a stated order and row offset (#9331) 
* feat(sqlite): page selected rows with a stated order and row offset

readRows read `NOT INDEXED LIMIT ?` with no ORDER BY and reported
order: "unspecified", so an agent could not reach rows past row_limit
deterministically: raising row_limit hits the fixed 16 MiB reply budget,
and the sampled order depends on whichever index SQLite happened to scan.

Read the window in a unique, stated order and expose the position:

- rowid tables order by rowid unless all of rowid, _rowid_ and oid are
  declared columns, in which case the order stays unspecified.
- WITHOUT ROWID tables order by the primary-key columns, reusing each
  key's DESC flag and declared collation from pragma_index_xinfo so the
  plan keeps index order instead of materialising a sorter.
- rows.order, rows.row_offset and rows.next_row_offset report the window;
  next_row_offset is null once the window reaches the end.
- row_offset (CLI --row-offset) requires table and defaults to 0. A
  non-zero offset is refused when the table declares no usable order,
  because the skipped prefix would be arbitrary.

The schema now checks next_row_offset against truncated and the offset so
a worker reply cannot claim a page it did not read.

One attribution fix came with it: adding ORDER BY text can push an
over-wide projection past SQLite's SQL-text limit, which fails as
SQLITE_TOOBIG before the result-column limit is reported. Attribute that
case to column capacity too, so a 250k-column request still reads as
capacity rather than a byte budget.

Verified: npm run check:fast, npm run format:check on the touched files,
npm run verify:module-boundaries, npm run docs:check, and 196 tests
across the sqlite boundary, application and CLI/MCP suites. A 1002-row
inspection and in-memory EXPLAIN QUERY PLAN checks confirmed the
WITHOUT ROWID path adds no sorter. Full check:pr and the complete test
suite were not run.

* fix(sqlite): resolve the rowid alias and keep paging claims reachable

Three defects in the row paging added alongside row_offset:

- readRowOrder emitted ORDER BY rowid for every rowid table, but a table
  that declares a rowid column shadows that spelling, so the sort used the
  stored column and could repeat or drop rows across pages. Order now names
  the first alias the table leaves unbound, read from raw name bytes so an
  unrelated malformed identifier is not decoded just to choose an order.

- The inventory projection could hide the primary key a WITHOUT ROWID table
  orders by, silently degrading a paginated read to unspecified order. The
  order key is now read from the table itself rather than from the columns
  the caller asked to see.

- A truncated table with no stated order reported a next_row_offset that the
  row_offset rule then refused. It is null there, and the reply limitation
  no longer claims an unspecified order for windows that have one.

rowEvaluationFailure also probed the first rows of the table instead of the
requested window, so a failure confined to a later page was attributed to
columns that evaluate fine there.

* test(sqlite): assert the concatenated pages

The paging regression checked only counts, offsets and flags, so a second
page that repeated the first page's rows would still pass. Assert the values
read across all three pages instead.

* fix(sqlite): decode row keys in the database encoding

readRowKeyFacts decoded primary-key names as UTF-8 and probed a fixed
hidden set, so two supported schemas could not be paged at all:

- A UTF-16 database stores key names as UTF-16 bytes, so "k" decoded to
  NUL-padded text and ORDER BY named an identifier that does not exist.
  Pass the detected encoding, as readColumns and the index-metadata path
  already do.
- A virtual generated column named rowid (hidden=2) was absent from the
  shadow check, so ORDER BY rowid evaluated the generated expression
  instead of the rowid and could sort non-uniquely or fail outright.
  Include generated columns in the alias probe.

Also stop decoding primary-key names for rowid tables: their ORDER BY
names an alias, never a declared key, so an unselected malformed key
name could abort an otherwise readable selection.

The page-concatenation assertion now narrows SqliteValue before reading
.value, which the null/blob arms do not carry; this was failing the
Static checks typecheck job.

* fix(sqlite): order WITHOUT ROWID rows by the index's own key sequence

`readRowOrder` filed `pragma_index_xinfo` key terms into a map keyed by
column name, so a valid composite key that repeats a column —
`PRIMARY KEY(a COLLATE BINARY, a COLLATE NOCASE DESC)` — collapsed to one
term. The generated clause then overstated uniqueness: it omitted the
BINARY term, so `A`/`a` ties survived, and the `row_offset` paging
advertised a `primary_key` order that the actual key does not establish.

Build the terms from the index rows in `seqno` order instead, keeping
repetitions, and bail to `unspecified` if any declared key column never
appears in the index. The declared collation stays explicit so a caller
column's own collation cannot re-sort the window.

Verified against a real SQLite database that the repeated key is accepted
and that the old clause orders A, a, B while the index orders A, B, a.

* fix(sqlite): accept a null next offset on unordered truncated rows

readRows already withholds next_row_offset when a table shadows rowid, _rowid_ and oid and declares no primary key, because every non-zero offset is refused there. The reply schema still required the offset whenever truncated was set, so the provider rejected its own worker result and the first sample never reached CLI or MCP. Require the offset only when the order is stated.

* fix(sqlite): probe row keys only for ordinary tables and name row_offset

readRowKeyFacts ran for a selected virtual shadow table before readRows
could reject it, so an unavailable module surfaced as SQLite's
"no such module" instead of the selection refusal. Probe the key only
when the selected kind is an ordinary table and leave the order
unspecified otherwise, since no rows are read for that kind.

The unordered-offset refusal also reported the generic selection reason,
which callers see against table. Give it its own reason so the failure is
attributed to row_offset.

* feat(sqlite)!: page selected rows with a stated order and row offset

Selected-row inspection currently reports only rows.order: "unspecified" and
offers no way to reach rows past the first sample deterministically. Main also
refuses non-zero row_offset today, so nothing relies on that omission; this
change makes the stated order and the offset the contract, reports the next
offset the caller feeds back, and keeps the existing byte, heap, statement and
deadline limits.

BREAKING CHANGE: rows.order becomes rowid | primary_key | unspecified, and
rows gains required row_offset and next_row_offset members. A client that
validates selected-table output against the previous schema (where order was
the only permitted value) rejects a now-valid reply. Migration: accept the two
new members and treat order: "rowid" | "primary_key" as the stated order the
offset applies to; a null next_row_offset means the window reached the end.

* docs(sqlite): say a null next offset also means no stated order

The selected-rows limitation described next_row_offset as null only when the window reached the end. An unordered table whose row_limit truncates the sample also publishes null, because an offset there would name a request the row-offset rule refuses, so the Evidence read as if the sample were complete. State the second cause and pin it in the existing regression.

* test(sqlite): drop the limitation wording assertion and state the unordered exception

The boundary test pinned the sentence a caller reads in `limitations`,
so a harmless rewrite of that text failed a paging test whose real
subject is `order`, `truncated` and `next_row_offset`.
docs/development/testing.md asks for facts over frozen explanatory prose,
and the three structured expectations already fail if the paging fact
changes.

The tool contract promised that retained rows always have a unique order
and can be paged. A rowid table that shadows rowid, _rowid_ and oid and
declares no primary key has no such order: it returns
order=unspecified with a null next offset and refuses a non-zero
row_offset. Say so in the description so callers do not plan a
continuation the tool always refuses.

* docs(sqlite): say unordered tables cannot be paged

The description told callers that a truncated table without a stated order
needs "the table read without offsets", but omitting row_offset starts at
zero again: with an unchanged row_limit the next call repeats the first
sample instead of continuing past it. Say that these tables cannot be
paged and point at row_limit for a larger first sample.

* style(cli): wrap the native data type path description

The `inspect-native-data-type` argument description line is 82 columns, over the repository's 80-column `.oxfmtrc.json` limit, so `npm run format:check` fails on main and on every branch merged from it. It came in with #9465 and the following commits to this file did not touch it. Reformat the one call the formatter objects to; no behaviour change.

* fix(sqlite): bound rowid alias comparison by name byte length

readRowKeyFacts decoded every column name before testing it against three short aliases, so a table declaring a multi-megabyte name could exhaust the worker heap even when the caller projected the column away. Reject names that cannot be an alias by their byte length, accounting for the database encoding, before building any string.

---------

Co-authored-by: 2SinceJuly <202121119+2SinceJuly@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-11-09 21:01:58 +08:00

230 lines
13 KiB
Markdown
Raw Blame History

# SQLite database snapshots
`inspect_sqlite_database` / `inspect-sqlite-database` inspects one explicitly
selected local SQLite database snapshot without an active native analysis
target. Use it to understand application storage, configuration or retained
database evidence. No separate SQLite installation is required.
This operation requires Node.js 24.x >=24.15 or 26+ with the native
`DatabaseSync.limits` API. The worker verifies that capability before querying
the copied database. Other REA features retain their published Node.js 22.x,
24.x or 26+ requirements.
`rea capabilities` reports `inspect_sqlite_database` availability and the
runtime requirement without opening a database. Unsupported runtimes show
`available: false`, `availability_code: "unsupported_version"`, and an upgrade
note while the inventory command still succeeds.
```sh
rea capabilities --json
rea inspect-sqlite-database ./application.sqlite --json
rea inspect-sqlite-database ./application.sqlite --table settings --column key --column value --row-limit 20 --json
```
```json
{
"name": "inspect_sqlite_database",
"arguments": {
"path": "/artifacts/application.sqlite",
"table": "settings",
"columns": ["key", "value"],
"row_limit": 20
}
}
```
MCP paths must be absolute filesystem paths on the current host. CLI paths
resolve against the operator's working directory. Select a quiescent snapshot:
stop the writer or obtain a consistent database/WAL pair using the application's
backup workflow before inspection. A file copy of a live pair does not establish
an atomic SQLite snapshot, even if no change is detected during inspection.
## Returned evidence
Without a table selection, the default `schema_scope` is `inventory`: compact
table metadata and index, view and trigger names, without SQL declarations or
column details. Selecting a table defaults to `selected_table`, which expands
that table and its indexes and triggers. Explicit `schema_scope: "database"`
(or CLI `--schema-scope database`) expands all stored declarations and supported
columns. `schema.completeness` refers to the reported scope; omitted column
details are marked `not_requested`. SQL declarations retain the original text. Virtual and
shadow tables are identified as such; their supported metadata coverage is
reported. If a virtual table cannot be resolved, possible backing tables have
`kind: "unknown"` and an explicit limitation; their rows are refused. Unrelated
ordinary tables remain selectable. Schema inventory does not execute views,
triggers or arbitrary SQL.
The exact table name `sqlite_schema` is also a supported read-only row source.
SQLite's [`table_list`](https://www.sqlite.org/pragma.html#pragma_table_list)
and `table_xinfo` pragmas establish its row-source type and columns. The
[schema catalog](https://www.sqlite.org/schematab.html) has no self-entry for
`sqlite_schema`, so REA does not invent a declaration, SQL statement or root
page for it in `schema.tables`.
Rows are optional. `table` selects one exact ordinary-table name. `columns`
(CLI repeated `--column`) selects exact visible column names in the requested
order; omission selects all visible columns, including generated columns.
Unselected cells are not evaluated or projected. A selected virtual generated
column whose expression needs a function REA's connection lacks (extensions and
application functions are not loaded), or raises an SQL error for a sampled
row, is refused as `capability_unavailable`. The error names the table and
each failing selected column with SQLite's message, and recommends only columns
whose sampled values were read; stored and other generated cells stay readable. Malformed column-name bytes
cause a format refusal when all columns are requested. In inventory scope, an explicit valid subset can
still be sampled without decoding unrelated malformed column names.
An unavailable row source, including a `WITHOUT ROWID` primary key requiring an
application collation, is also refused as `capability_unavailable`. The error
retains the selected table and SQLite's reason without identifying unproven
dependencies or diagnosing file corruption. Such a source does not provide
readable-column recommendations. Omit `table` and select `schema_scope: "database"`
to read stored declarations without requesting rows; the application's supported
consistent export may be needed to obtain those values.
Column selection requires a
table and at least one name; empty-string identifiers are valid. SQLite counts
every projected name, including a repeated name, against its native
result-column capacity; a wider projection is refused as `resource_constraint`
with `resource: "capacity"`, reporting the selected count and this connection's
column limit. The selected database is readable and is not reported as
malformed; request fewer columns per call instead. `row_limit`
requires `table`, defaults to 20 and accepts positive safe integers. Samples
remain subject to the byte and worker resource limits below, regardless of row count. The result reports
the selected limit, returned count and whether more rows were observed.
Rows are read in a stated unique order reported as `rows.order`: `rowid` for
rowid tables, and `primary_key` for `WITHOUT ROWID` tables, reusing the
primary-key index's own direction and declared collation. `rows.row_offset`
reports the rows skipped before the window and `rows.next_row_offset` the offset
to request for the following page; it is `null` once the window reaches the end.
Pass `row_offset` (CLI `--row-offset`, requires `table`, default 0) to skip into
a table rather than repeating earlier rows. A table that declares all of
`rowid`, `_rowid_` and `oid` as columns and has no primary key reports
`order: "unspecified"` and refuses a non-zero `row_offset`, because the skipped
prefix would be arbitrary. Pages compose only while the reported database and
WAL SHA-256 match; re-inspecting a changed snapshot restarts the sequence.
View, virtual-table and shadow-table row reads are unsupported.
SQLite integers are decimal strings so every signed 64-bit value survives JSON
serialization. Real values retain their numbers; `"Infinity"`, `"-Infinity"`
and `"-0"` explicitly preserve infinities and negative zero through JSON.
Text returns the decoded string together with its bytes as hex and the database
encoding, preserving malformed text that the SQLite binding may replace for
display. Blobs are hex, and SQL NULL is a distinct tagged value.
Database and existing sibling `-wal` files have separate original SHA-256 and
byte counts. The Evidence subject identifies the original database file;
the WAL identity is also necessary to interpret pending committed data. The
SQLite engine version is reported with the observed result. Neither digest is
a claim that the database is healthy, complete or trusted.
## Source files and failures
REA copies the selected database and existing sibling WAL to an owned private
temporary directory before SQLite opens them. Source files are not opened by
SQLite, so its locks or shared-memory sidecars cannot modify the original
snapshot. Temporary copies and the owned worker are cleaned up on success,
failure and cancellation. Extension loading and schema trust are disabled.
Empty rollback journals and fully zeroed journal headers left by successful
`TRUNCATE` and `PERSIST` commits are accepted after stable identity checks.
Journal headers are checked both before and after acquisition; unchanged file
metadata alone does not establish that a journal remains inactive.
Other retained rollback journals are refused because recovery is outside this
profile. Journal files are never modified or copied into the worker snapshot.
A provider admits at most two inspections through acquisition, worker execution
and cleanup. Additional requests receive a typed capacity error before reading
files or creating workers; retry after an active inspection finishes. This
bounds simultaneous input buffering and private snapshot/worker counts.
The database and WAL together are limited to 256 MiB. SQLite values/records,
SQL statements and the complete worker JSON reply each have a 16 MiB limit.
Schema names count by their returned JSON text, including escaping, for UTF-8,
UTF-16le and UTF-16be databases. Internal raw-byte lookup keys do not consume
the reply allowance; exact name identity still comes from those stored bytes.
Actual row byte/hex expansion still counts toward the reply.
The owned worker has a 256 MiB JavaScript heap limit and defaults to a 30,000 ms
deadline. Select optional MCP `worker_timeout_ms`, or CLI
`--worker-timeout-ms <milliseconds>`, to change the budget for that call. Values
must be integer milliseconds from 1 to 2,147,483,647 (the runtime timer range).
It covers worker startup, queries and reply writing, **not** DB/WAL snapshot
acquisition, final reply reads, cleanup or delivery. Allow corresponding headroom
in the MCP client's whole-call deadline; increasing only that deadline does not
change the worker budget. No server restart is needed.
The result's `worker_timeout_ms` reports the actual parent-supervised budget,
including the default. Historical Evidence without this field has unknown
execution settings, not an inferred default. Explicit input selection is also
retained in Evidence parameters; omission stays omitted. Completed observations
retained under cancellation or cleanup failure keep the same budget facts.
Timeout errors name the selected milliseconds, exact CLI/MCP controls, and
cleanup-before-retry requirements. Incomplete cleanup remains a blocker even
when a larger budget is available.
For example, `rea inspect-sqlite-database ./application.sqlite
--worker-timeout-ms 60000 --json` selects 60 seconds; equivalent MCP input is
`{"path":"/artifacts/application.sqlite","worker_timeout_ms":60000}`.
SQLite native allocations are separate from the JavaScript heap;
overall native heap and total resident memory are not capped. Row limits do not
override these byte and resource limits. Resource-limit failures do not return
an incomplete schema as successful complete evidence.
If the combined database and WAL exceed the snapshot budget, the refusal is a
`resource_constraint` and reports both the configured byte limit and the exact
observed combined size as a decimal string. The remediation directs callers to
create a smaller consistent offline snapshot containing only relevant data
through the application's supported export workflow. A generic backup or copy
does not guarantee a smaller result; REA does not truncate or modify the
selected source.
On hosts that cannot address the derived WAL or rollback-journal path for a
long selected database name, REA reports that the SQLite capability is
unavailable. It does not assume an inaccessible sidecar is absent. Use the
application's supported consistent offline export or copy workflow to place
the complete known file set at a shorter path.
MCP also applies its configured response budget to the complete Evidence
envelope. If delivery exceeds that budget, it reports a transport constraint
with guidance for exporting retained Evidence, requesting a matching larger
client/server response budget or using complete CLI JSON output.
A cancellation received after the worker has completed does not erase the
completed observation. If subsequent private-root cleanup fails, the error
retains the completed Evidence and the resource owner for cleanup retry. MCP
records that Evidence before delivering the error, which carries its retained
evidence ID for later retrieval or export.
If a worker fails and cleanup remains incomplete, the error retains the original
cancellation, timeout, diagnostic overflow or memory failure. Successful cleanup
retry restores that failure's classification; it does not make the inspection
successful.
Changed input identities, special files, malformed databases, unsupported
formats and resource failures are reported as errors with the selected path and
meaningful reason. Worker resource refusals expose the caller-selected file in
`details.diagnostics.selected_path`. Encrypted databases, database repair, arbitrary SQL,
cross-database queries and live capture are outside this inspection's coverage.
## Compatibility verification
Run `npm run test:sqlite` after changing the SQLite adapter or its shared
process, filesystem, CLI or MCP boundaries. It prepares the compiled runtime
and runs SQLite service, snapshot, inspection, cleanup and public CLI/MCP
tests. Directory-based selection includes new SQLite service and boundary
tests automatically. `npm run test:sqlite:run` runs the same cases when the
compiled runtime and generated test artifacts are already current.
The reusable SQLite CI workflow consumes the shared build artifact and tests
the latest Node 22.x unavailable-runtime diagnostics, the minimum supported
24.15.0 release, and the latest 24.x and 26.x releases. The repository-pinned
runtime also runs the full suite. CI's required gate waits for every selected
SQLite matrix job; a failed, cancelled, skipped or missing selected lane blocks
the gate. Workflow changes themselves select the build and compatibility lane.
PR checks run against GitHub's merge with the current base branch. A later
change to `main` can introduce failures outside SQLite; inspect the failing
job and recheck the updated merge before treating an earlier green run as
current verification. Compatibility checks detect regressions, but cannot
guarantee that an arbitrary future upstream change remains compatible.
Reference in new issue View Git Blame Copy Permalink