From df2d018b30946a00360592faf9e94982c7c39ff6 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Mon, 9 Nov 2026 04:48:08 -0800 Subject: [PATCH] feat(desktop): read gzip documents as the document inside (#9111) * fix(desktop): release a capture when its start is cancelled after launch Cancelling start_build_capture during the opened app's screenshot left the capture running, so the next start was refused. The capture is now kept for the owner's cleanup, and the next start stops it first. * fix(desktop): keep a published witness error and tag an app data refusal A second failure to close a published witness returns the original error, so its record and cleanup lease stay bound. A reset the app itself refuses is an AppDataRefusal; only a reply that is not that refusal stays unsettled. A missing child of a cut view is undecided with the same description in both evaluators, and the shared cases cover it. * fix(desktop): align a cut view's missing text with the Python check A missing child of a truncated view is undecided for the same sentence in both evaluators, including the pointer the view does not contain. * fix(desktop): keep one published witness cleanup across retries A published record and its bound cleanup error stay together. A later close failure returns that error, and the unused path that built a fresh adapter error is gone. * fix(desktop): keep a bundle task read inside the witness result A cancelled, refused, or failed read of the bundle's task files returns the witness's Result, including a cleanup lease when a handle would not close. * test(desktop): pass the settle timing when reading a witness bundle The witness takes the settle timing before the bundle path, so the bundle-read tests keep the path in its own argument. * fix(desktop): bound a guest file answer and split a long controller argument A data/get whose base64 answer would exceed a quarter of the V8 heap is refused before it is read. Controller JSON that exceeds Linux's single-argument limit travels as several arguments the controller joins. * fix(desktop): keep an underreported guest file from raising its read A data/get answer is admitted only when its base64 body fits in the heap quarter after the copies held during parse. Decoded bytes larger than the computer reported are dropped. * fix(desktop): grade shipped seeds and keep selected settings and sqlite logs Seed checks follow the files the bundle ships, so an omitted lock is absent. An incomplete checkpoint is not reported as no change. A selected link keeps its spelling, and a database's wal and journal stay supporting state. Closing the fake computer stops a detached capture. * fix(desktop): keep added-record checks off ids, dates, and joined timestamps A position-named record's fields keep their paths, so an identifier, a stamped date, and text past the limit are not suggested checks. A whole-text check does not climb into an ancestor whose text joins a timestamp. * fix(desktop): bound home fetches and keep unsettled edits readable Split home-folder reads by the guest answer budget, keep a capacity error from the capture output, and drop the superseded task-argument refusal. Grade kept text per line, leave unretained dconf unread, and reserve by-position notes for XML children. Map a linked database's sidecars through the selected spelling and stop a detached capture only when its start identity still owns the pid. * fix(desktop): describe unretained dconf as unread The key comparison treats a missing copy as unknown. The comment still said that copy counted as an empty key list. * test(desktop): keep a capture capacity error through endedCapture An output past the guest answer ceiling stays the resource constraint, with its heap recovery and non-retryable status, instead of a generic capture-exit message. * fix(desktop): keep nested save times and formatting runs out of checks A field check drops save-time leaves before it joins the field, so a nested timestamp is not part of the suggested text. A text check beside a save time uses the paragraph when the diff names a formatting run. * test(desktop): read paragraph pointers from checks that have them * test(desktop): expect a large task to export intact The 96 KiB refusal is gone. A prompt past that size reaches the computer and comes back unchanged, instead of failing before export. * fix(desktop): keep real edits that save-side heuristics were dropping Repeated unnamed fields stay content checks. A formula that still has text is not renumbered with its ExprID. One gained style attribute is the edit; a repeated fill-in stays upkeep. Phone numbers and plain clock times stay text, and a decimal is the element's whole text. An older verifier matches on the task it states, not its evaluator source. * fix(desktop): cover repeated records and keep timestamps out of checks An added record whose words are already in the seed still gets the list's text when that text fits. Readiness is a start reason, so a final check names seed locks without matching the message. Field text split by a timestamp stays separate, and a timestamp line is not offered as a guard. * fix(desktop): compare a cell's joined decimal within tolerance The whole-element check keeps numeric tolerance, so 437.80 and a long float still match 437.8, including when a run splits the digits. A different number or other text does not. Both evaluators read an element's text for that comparison. * fix(desktop): grade a number from a complete XML element only A tolerance check joins an element's text only in an XML file or an XML archive entry, and leaves that check undecided when the view is cut, so a dropped descendant cannot turn the kept runs into a passing number. A JSON object with the same shape stays the value itself. Equivalent decimals and a split run on a whole cell still match. * fix(desktop): accept an older grader only when its index helper agrees The witness still matches a bundle whose evaluator is this REA's, or the older index expression alone when no path segment is a leading-zero number. A truncated verifier, a changed grader, or that older helper beside a segment such as 01 no longer counts as the same task. * fix(desktop): require the added record and leave save times out of list checks A record a selector names is checked itself, so its words have to be in that record. When an unnamed record only repeats words the seed already has, a list that holds a save time is checked by the text around that time. A later save of the same records still passes, and a missing or rewritten record fails. * fix(desktop): judge the older index helper by pointer segments only A leading zero in a file name or a URL does not change how the older index helper reads a pointer. The witness still rejects that helper when a pointer segment is a number such as 01. * fix(desktop): do not suggest list text the seed already passes A repeated record is shown by text the seed does not hold. The seed's own text is not a check. A pointer inside a check value is not a pointer segment for the older index helper. * fix(desktop): keep a linked settings file in the folder the link points at A selected ~/.config alias no longer moves MuseScore settings to that alias. The seed keeps settings/MuseScore or Documents, which is the path the export opens the note from. * fix(desktop): split home fetches by answer size and keep a size refusal A batch of tiny files stops before each path's envelope exceeds the host answer budget. A capture output past that ceiling stays a resource error instead of becoming a generic exit. * fix(desktop): seed a selected link under the path the app uses The selected spelling wins, so ~/.config/MuseScore is not rewritten to the folder it points at and the note stays the one launched document. A database wal and journal follow that database. A launch argument is respelled the same way. Teardown signals a detached PID only when its start identity still matches. * fix(desktop): keep a witness's actions and an incomplete checkpoint A failed witness save retries the action log it first recorded. A deletion left unconfirmed between the seed and the end is not reported as no change. Windows packs npm through a shell, which can start npm.cmd. * fix(desktop): read verifier pointers from the parsed checks A brace inside a check value is not an object boundary. A leading-zero pointer after that value still rejects the older index helper. * fix(desktop): prefer the most specific home spelling and retarget launch files A selected parent directory no longer hides a settings folder walked through a link. A launch filename under that folder is replaced with the seeded file's absolute path. * fix(desktop): pack the build runtime through npm's JavaScript entry npm pack and npm ci no longer spawn npm.cmd through a shell, so a temporary directory whose path contains spaces stays one argument. * fix(desktop): spell launch settings and keep a database log exact A database log matches only that file, so a neighboring folder is not moved. A launch option and the working directory use the same home spelling as the seed. * fix(desktop): move a state database's logs with the database The write-ahead log and journal follow the spelling the database received, including a walked state link. A neighboring folder stays put. A separate --config argument uses that spelling too. * fix(desktop): keep a selected file ahead of a database-log guess A file selected as itself keeps that spelling even when its name ends in -journal. Relative launch arguments are resolved in the captured working directory before that directory moves. * fix(desktop): keep outside-home database logs and option values A selected database outside the home keeps its write-ahead log. Launch options are rewritten only when they name a captured file or settings path. * fix(desktop): follow a database mapping and own limitation text A selected database keeps its log when a parent folder moves or the file sits outside the home. Trash and seed-lock tests compare the owning notice with the paths, and launch options that are not captured paths stay as written. * fix(desktop): retarget a settings folder that holds captured files A nested profile directory is a captured path when a seeded file is inside it. An option value that only resolves under the home stays as written. * fix(desktop): keep a program word and a directory's neighbor The program after env stays the program even when a captured file sits in a folder of that name. A selected directory is not a database, so a neighboring file keeps the parent folder's spelling. * fix(desktop): record whether a selected path is a directory Capture stores the walked path's kind. A neighboring file of a selected directory keeps the parent folder's spelling even when that directory never changes and its children are not content roots. * fix(desktop): mark containing paths without a pairwise scan A recorded path marks its known ancestors as directories in one walk up the path, instead of comparing every recorded path with every other. * fix(desktop): keep a selected folder that is named like a database log A folder the caller selected stays a selected root when its name ends in -journal or -wal. A generated log beside a database file still does not. * fix(desktop): refuse a build command that exceeds the argument vector Splitting one argument still leaves the whole command under Linux's E2BIG limit. A request past that budget is refused before the process starts. * fix(desktop): remap a captured path in an env assignment The program word stays the program. A NAME=value assignment keeps its name and follows a captured path value when that path's spelling moves. * fix(desktop): keep oversized text out of draft guards A value or line longer than the draft's text bound is not copied into a guard, so one large frame cannot fill the task by itself. * fix(desktop): keep an empty env value empty An empty assignment is not a path. Resolving it named the working directory and rewrote the value when that directory's spelling moved. * fix(desktop): check build runtime export capability * fix(desktop): provide complete runtime repack command * fix(desktop): retain runtime probe failures * test(desktop): identify capture independently of schema preflight * fix(desktop): preserve document URLs and value formats * fix(desktop): bound added checks and omit noisy text guards * fix(desktop): preserve launch URLs and JSON draft values * fix(desktop): recognize captured file URL documents * fix(desktop): render captured file URLs for runtime home * fix(desktop): distinguish phone numbers from save metadata * fix(desktop): preserve launch URLs and user text * fix(desktop): preserve local file URL authority * fix(desktop): preserve local file URL authority * fix(desktop): render captured file URLs at launch * fix(desktop): classify numeric timestamps using field context * fix(desktop): classify numeric timestamps using field context * fix(desktop): retain context for generated draft values * fix(desktop): retain context for generated draft values * test(desktop): cover generated IDs in text guards * test(desktop): cover generated IDs in text guards * fix(desktop): isolate launcher URL encoding * fix(desktop): isolate launcher URL encoding * fix(desktop): isolate launcher URL encoding * fix(desktop): retain clean text in noisy record fields * style(desktop): format the noisy-record draft files with the pinned oxfmt Co-Authored-By: Claude Opus 5.5 * test(desktop): move launch execution to boundary suite * fix(desktop): redact build schema failure output * refactor(desktop): normalize schema failure once * fix(desktop): omit unsafe schema parser snippets * fix(desktop): preserve draft change and selected path evidence * test(desktop): group retained dconf note regressions * fix(desktop): respect archive entry format in draft checks * test(desktop): separate unowned-file draft note cases * docs(desktop): clarify task checks before scratch export * fix(desktop): respell selected paths in file URLs * test(desktop): group captured document spelling cases * fix(desktop): bound draft preview serialization * fix(desktop): restore selected state aliases * fix(desktop): respect entry format in draft noise * test(desktop): match exported alias evidence at its JSON boundary * style: fix desktop launch test formatting * style: format native command path schema * fix(desktop): map selected external launch paths * style(desktop): format seed document regressions * test(desktop): group launch URL regressions * fix(desktop): omit redundant nested state aliases * fix(desktop): compare added free text as a whole * fix(desktop): check recompressed seeds as shipped * fix(desktop): map captured launch paths once * refactor: remove unused whole-text guard state * refactor: stop passing unused text children * test: align free-text checks with whole-note policy * fix: validate recompressed desktop archive seeds * style(desktop): format archive seed regression * fix(desktop): redact runtime schema errors * fix(desktop): normalize captured file URLs * fix(desktop): report seeded files by capture path * fix(desktop): retain startup readiness failure stage * fix(desktop): retain JSON date checks * fix(desktop): preserve archive formatting checks * fix(desktop): preserve explicitly selected journal files * fix(desktop): report build command transport limits * style(desktop): format capture review fixes --------- Co-authored-by: Cursor Agent Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- .../references/desktop-build.md | 196 +++- .../references/environments.md | 15 +- bridge/agentenv/capture.py | 53 +- docs/desktop-application-capture.md | 36 +- docs/desktop-application-export.md | 119 ++- docs/desktop-application-state.md | 45 +- docs/desktop-application-tasks.md | 191 +++- docs/desktop-build-computer.md | 12 +- docs/desktop-build-witness.md | 124 ++- .../build-witness/spec.md | 38 +- .../build-witness/verification.md | 117 +++ .../capture-for-building/spec.md | 4 +- .../desktop-app-environments/capture/spec.md | 7 +- .../state-view/spec.md | 9 +- .../desktop-app-environments/tasks/spec.md | 9 +- scripts/pack-build-computer-runtime.mjs | 12 +- .../desktop/DesktopApplicationCapture.test.ts | 4 +- .../desktop/DesktopApplicationCapture.ts | 16 +- .../desktop/DesktopApplicationExport.test.ts | 665 ++++++++++++- .../desktop/DesktopApplicationExport.ts | 95 +- .../desktop/DesktopApplicationState.test.ts | 38 + .../desktop/DesktopApplicationState.ts | 26 +- .../desktop/DesktopArchiveTask.test.ts | 101 ++ src/application/desktop/DesktopBuildBundle.ts | 40 +- .../desktop/DesktopBuildCapture.ts | 152 ++- .../desktop/DesktopBuildCaptureFailure.ts | 176 ++++ .../desktop/DesktopBuildCommands.ts | 104 +- .../desktop/DesktopBuildComputers.ts | 51 +- src/application/desktop/DesktopBuildExport.ts | 19 +- .../desktop/DesktopBuildOutputRead.test.ts | 123 +++ src/application/desktop/DesktopBuildSettle.ts | 97 ++ src/application/desktop/DesktopBuildSteps.ts | 15 +- .../desktop/DesktopBuildSummary.ts | 99 +- .../desktop/DesktopBuildWitness.test.ts | 8 +- .../desktop/DesktopBuildWitness.ts | 87 +- .../desktop/DesktopBuildWitnessRecord.ts | 24 + .../desktop/DesktopCaptureScope.test.ts | 36 +- .../desktop/DesktopCaptureScope.ts | 84 +- src/application/desktop/DesktopExportFiles.ts | 59 +- .../desktop/DesktopExportLaunch.test.ts | 213 +++++ .../desktop/DesktopExportLaunch.ts | 79 +- .../desktop/DesktopGzipDocument.test.ts | 291 ++++++ src/application/desktop/DesktopTaskDraft.ts | 70 +- .../desktop/DesktopTaskDraftNotes.test.ts | 393 ++++++++ .../desktop/DesktopTaskExport.test.ts | 12 + src/application/desktop/DesktopTaskExport.ts | 40 +- src/composition/capabilityServices.ts | 9 +- src/contracts/desktopBuildToolContracts.ts | 88 +- src/desktop/AgentEnvComputer.ts | 172 +++- src/desktop/ApplicationStateDirectories.ts | 8 +- src/desktop/BuildComputerController.test.ts | 172 +++- src/desktop/BuildComputerController.ts | 137 ++- src/desktop/ComputerStartFailure.test.ts | 25 + src/desktop/DesktopApplicationSession.test.ts | 24 + src/desktop/DesktopApplicationSession.ts | 6 +- src/desktop/GuestFileAnswer.test.ts | 77 ++ src/desktop/LinuxApplicationInventory.test.ts | 19 + src/desktop/NpmPackInvocation.test.ts | 23 + src/domain/desktop/appExport.test.ts | 276 ++++++ src/domain/desktop/appExport.ts | 245 ++++- src/domain/desktop/appExportPython.ts | 47 +- src/domain/desktop/appExportPythonZip.ts | 98 +- src/domain/desktop/appExportUnowned.test.ts | 244 +++++ src/domain/desktop/appExportUnowned.ts | 259 +++++ src/domain/desktop/appSettle.ts | 63 ++ src/domain/desktop/appTaskChecks.fixture.ts | 186 ++++ src/domain/desktop/appTaskChecks.test.ts | 78 ++ src/domain/desktop/appTaskChecks.ts | 40 +- src/domain/desktop/appTaskDraft.fixture.ts | 80 ++ src/domain/desktop/appTaskDraft.test.ts | 35 +- src/domain/desktop/appTaskDraft.ts | 372 ++++---- src/domain/desktop/appTaskDraftAdded.test.ts | 898 ++++++++++++++++++ src/domain/desktop/appTaskDraftAdded.ts | 385 ++++++++ .../desktop/appTaskDraftFileNotes.test.ts | 58 ++ src/domain/desktop/appTaskDraftFileNotes.ts | 104 ++ src/domain/desktop/appTaskDraftGuards.test.ts | 290 ++++++ src/domain/desktop/appTaskDraftGuards.ts | 282 ++++++ src/domain/desktop/appTaskDraftLineGuards.ts | 58 ++ src/domain/desktop/appTaskDraftNamed.ts | 610 ++++++++++-- src/domain/desktop/appTaskDraftNoise.test.ts | 357 +++++++ src/domain/desktop/appTaskDraftNoise.ts | 380 ++++++++ src/domain/desktop/appTaskDraftNotes.ts | 259 +++++ .../desktop/appTaskDraftPositional.test.ts | 226 +++++ src/domain/desktop/appTaskDraftRows.ts | 164 ++++ src/domain/desktop/appTaskDraftScalars.ts | 43 + src/domain/desktop/appTaskDraftText.ts | 401 +++++++- .../desktop/appTaskDraftTextAdded.test.ts | 69 ++ src/domain/desktop/appTaskDraftValues.test.ts | 572 +++++++++++ src/domain/desktop/appTaskDraftValues.ts | 559 +++++++++++ src/domain/desktop/appTaskPython.ts | 260 ++++- .../desktop/appTaskPythonDescription.test.ts | 137 +++ src/domain/desktop/attributeRank.ts | 54 ++ src/domain/desktop/buildCapture.ts | 38 + src/domain/desktop/buildWitness.test.ts | 2 + src/domain/desktop/buildWitness.ts | 4 + src/domain/desktop/desktopCapture.test.ts | 147 +++ src/domain/desktop/desktopCapture.ts | 99 +- src/domain/desktop/desktopEntry.ts | 20 + src/domain/desktop/desktopState.ts | 4 +- src/domain/desktop/gzipDocument.test.ts | 117 +++ src/domain/desktop/gzipDocument.ts | 152 +++ src/domain/desktop/positionalKeys.ts | 130 +++ src/domain/desktop/runtimeFiles.ts | 72 ++ src/domain/desktop/seedDocument.test.ts | 247 +++++ src/domain/desktop/seedDocument.ts | 172 +++- src/domain/desktop/seedScrub.test.ts | 187 ++++ src/domain/desktop/seedScrub.ts | 56 +- src/domain/desktop/selectorCandidates.ts | 127 +++ src/domain/desktop/stateChanges.ts | 24 +- src/domain/desktop/stateFormats.ts | 2 +- src/domain/desktop/stateRoles.test.ts | 98 ++ src/domain/desktop/stateRoles.ts | 95 +- src/domain/desktop/viewPointer.test.ts | 102 ++ src/domain/desktop/viewPointer.ts | 212 ++++- src/domain/desktop/zipView.ts | 4 +- src/server/registerDesktopBuildTools.ts | 15 +- .../boundary/mcp/buildCaptureDraftMcp.test.ts | 121 +++ .../mcp/buildCaptureFailureMcp.test.ts | 214 +++++ tests/boundary/mcp/buildCaptureMcp.test.ts | 177 +++- .../boundary/mcp/buildCaptureRetryMcp.test.ts | 39 + .../mcp/buildCaptureStatePathsMcp.test.ts | 245 +++++ tests/boundary/mcp/buildComputerMcp.test.ts | 57 ++ tests/boundary/mcp/buildWitnessMcp.test.ts | 180 +++- .../providers/desktopExportPython.test.ts | 80 +- .../providers/desktopGzipPython.test.ts | 162 ++++ .../providers/desktopTaskPython.test.ts | 192 +++- tests/fixtures/buildCaptureFakes.ts | 67 +- tests/fixtures/buildComputerMcp.ts | 13 +- tests/fixtures/fakeAgentEnvComputer.ts | 122 ++- tests/fixtures/fakeBuildComputerRea.ts | 87 +- tests/support/desktopExportPython.ts | 27 +- 131 files changed, 16095 insertions(+), 967 deletions(-) create mode 100644 src/application/desktop/DesktopBuildCaptureFailure.ts create mode 100644 src/application/desktop/DesktopBuildOutputRead.test.ts create mode 100644 src/application/desktop/DesktopBuildSettle.ts create mode 100644 src/application/desktop/DesktopExportLaunch.test.ts create mode 100644 src/application/desktop/DesktopGzipDocument.test.ts create mode 100644 src/application/desktop/DesktopTaskDraftNotes.test.ts create mode 100644 src/desktop/ComputerStartFailure.test.ts create mode 100644 src/desktop/GuestFileAnswer.test.ts create mode 100644 src/desktop/NpmPackInvocation.test.ts create mode 100644 src/domain/desktop/appExportUnowned.test.ts create mode 100644 src/domain/desktop/appExportUnowned.ts create mode 100644 src/domain/desktop/appSettle.ts create mode 100644 src/domain/desktop/appTaskDraft.fixture.ts create mode 100644 src/domain/desktop/appTaskDraftAdded.test.ts create mode 100644 src/domain/desktop/appTaskDraftAdded.ts create mode 100644 src/domain/desktop/appTaskDraftFileNotes.test.ts create mode 100644 src/domain/desktop/appTaskDraftFileNotes.ts create mode 100644 src/domain/desktop/appTaskDraftGuards.test.ts create mode 100644 src/domain/desktop/appTaskDraftGuards.ts create mode 100644 src/domain/desktop/appTaskDraftLineGuards.ts create mode 100644 src/domain/desktop/appTaskDraftNoise.test.ts create mode 100644 src/domain/desktop/appTaskDraftNoise.ts create mode 100644 src/domain/desktop/appTaskDraftNotes.ts create mode 100644 src/domain/desktop/appTaskDraftPositional.test.ts create mode 100644 src/domain/desktop/appTaskDraftRows.ts create mode 100644 src/domain/desktop/appTaskDraftScalars.ts create mode 100644 src/domain/desktop/appTaskDraftTextAdded.test.ts create mode 100644 src/domain/desktop/appTaskDraftValues.test.ts create mode 100644 src/domain/desktop/appTaskDraftValues.ts create mode 100644 src/domain/desktop/appTaskPythonDescription.test.ts create mode 100644 src/domain/desktop/attributeRank.ts create mode 100644 src/domain/desktop/gzipDocument.test.ts create mode 100644 src/domain/desktop/gzipDocument.ts create mode 100644 src/domain/desktop/positionalKeys.ts create mode 100644 src/domain/desktop/runtimeFiles.ts create mode 100644 src/domain/desktop/selectorCandidates.ts create mode 100644 tests/boundary/mcp/buildCaptureDraftMcp.test.ts create mode 100644 tests/boundary/mcp/buildCaptureFailureMcp.test.ts create mode 100644 tests/boundary/mcp/buildCaptureStatePathsMcp.test.ts create mode 100644 tests/boundary/providers/desktopGzipPython.test.ts diff --git a/.agents/skills/reverse-engineer-anything/references/desktop-build.md b/.agents/skills/reverse-engineer-anything/references/desktop-build.md index efad507a94..233b49af47 100644 --- a/.agents/skills/reverse-engineer-anything/references/desktop-build.md +++ b/.agents/skills/reverse-engineer-anything/references/desktop-build.md @@ -72,6 +72,13 @@ the first source that has the app: `/usr/share/applications` without `.desktop`, such as `org.inkscape.Inkscape`. Add `window_class` when you know it; Inkscape's is `Inkscape`. AbiWord's entry is `abiword` and its class `Abiword`. +- To find the entry, read the package's file list: its page on + packages.ubuntu.com, or `apt-get download PKG && dpkg -c PKG_*.deb` where + apt runs. The entry may ship in a companion `-data` or `-common` package, + as Planner's in `planner-data`: read that package's list too. The entry's `StartupWMClass` may not be the window's real + class, so do not take `window_class` from it: MuseScore 3's says + `mscore3`, its window's is `MuseScore3`. The export uses the class the + capture saw. ## Drive the app @@ -97,16 +104,25 @@ the first source that has the app: environment's name. - Prefer shortcuts and typing to long pointer paths: `key` takes `keys`, an array holding one chord's keys, such as - `{"action": "key", "keys": ["ctrl", "s"]}`, and `type` turns `\n` into + `{"action": "key", "keys": ["Control", "s"]}`, and `type` turns `\n` into Enter. - Before `type`, check in a screenshot that a text cursor or a focused field is there. In apps with single-key shortcuts, such as Inkscape, text typed without one runs as commands. -- Save As opens in the home: click Documents in its sidebar and type only - the file name. Answer the app's save questions, such as keeping the - format, then check the saved path in the title or status bar, or the - dialog's folder when the app shows only the name, and that it shows no - unsaved mark, before you mark a checkpoint or check. +- Save As differs by app. Check the dialog's current folder first: most + open in the home, so click Documents in the sidebar, but Scribus opens in + `~/Documents` and its sidebar has no Documents entry. Type only the file + name. A dialog taller than the 1280×800 screen hides its buttons: + maximize it from its title bar, or confirm it with `key` Return, as when + HomeBank's Save button is off-screen. +- Answer the app's save questions, such as keeping the format, then check + the saved path in the title or status bar, or the dialog's folder when + the app shows only the name, and that it shows no unsaved mark, before + you mark a checkpoint or check. Where the title never shows one, as in + HomeBank, a greyed Save or Revert, or the file reopened from recent + files, confirms the save. +- Qt spin boxes change their value on scroll: move the pointer off them + before you scroll. - What the app saved under the home before `seed`, such as a document in `~/Documents` or its settings, is part of the seed. A file you move to the Trash is left out, and the export lists it in `limitations`. Trash a stray @@ -126,12 +142,30 @@ the first source that has the app: 1. `start_build_computer` with `{"source": {"kind": "request", "desktop_entry": ..., "install": ...}}`. - The first start of an app builds its image and can take minutes. A - failed start returns the build's error: fix the request, or stop when the + The first start of an app builds its image, and starts at the same time + on this host wait for each other; either can take minutes. A failed + start returns the build's error: fix the request, or stop when the install cannot work, and close any `computer_id` the error names. 2. `start_build_capture`. It returns once the app's window is open. A `package_lists_error` may stop the capture finding the package's - sources; the export says when that matters. + sources; the export says when that matters. When the app exits before + its window opens, the error gives its exit code and the end of its + stderr, which may name what it lacks, such as a Java runtime: try + another install source, or stop as for an install this flow cannot do. + - REA counts a file as the app's document only when the app's window + title shows its file name or `~/` path. When the title shows something + else, such as MuseScore's score title, pass the path you will save the + document at in `state_paths`, relative to the home: + `{"computer_id": ..., "state_paths": ["Documents/MuseScore3/Scores/Morning_Song.mscz"]}`. + It need not exist yet. Add the app's settings folder when its name + differs from the app's, such as `.config/MuseScore`, so the seed keeps + the dismissed first-run dialogs. Do not pass the document as a launch + argument to make REA see it: `state_paths` is the way. + - An app whose document is a folder of pages, such as a Zim notebook + (`Notebooks/Notes`, a `.txt` file per page), never shows a file in its + title: pass the folder in `state_paths`. Every page in it is then a + document, so the bundle opens none at launch and the app starts where + it keeps its place; check the page the task changes. 3. Make the seed through the app and save it from the app under `~/Documents`: build the document in the app, or write its source where the app edits its format as text, then save it again from the app. Make @@ -141,14 +175,57 @@ the first source that has the app: otherwise starts on its own screen. For a starting file of the requester's, read it here and rebuild it in the app with synthetic data in place of personal data; REA cannot copy a file into the computer. + - An app that keeps its own data and saves after every change, such as + Osmo's contacts in `~/.local/share/osmo/contacts_records.xml`, has no + document: that state is the seed. Build it in the app and mark `seed` + once the app has written. To see whether it has, mark a throwaway + checkpoint, such as `probe-1`, and read its `changed_files`: up to ten + files that changed since the checkpoint before, those under + `state_paths` first, then those the app held open, caches last, with + `more_changed_files` counting the rest. + - The app's own compressed format works when it is gzip, such as + Gnumeric's `.gnumeric`, Dia's `.dia` or Inkscape's `.svgz`: REA reads + the document inside, and checks point into it as into the same + document uncompressed. A format REA reads as binary, such as one + compressed with xz, has no view: save in a format the app offers that + REA reads, then reopen the file once and check that a plain Ctrl+S + keeps that format. 4. Only once the seed is saved, `mark_build_checkpoint` with name `seed`. 5. Make the change the task asks for, as a user would, and save it. 6. Only once the change is saved, `mark_build_checkpoint` with name `end`. 7. `stop_build_capture`. Read `draft.files` (each file that changed, with - its net change), `draft.task.checks` (suggested checks) and - `limitations`. When `draft` is null, read `draft_refusal` first: a - missing `seed` or `end` checkpoint means capturing again on a new build - computer, while a failed draft command names its own cause to fix. + its net change and its `values`), `draft.task.checks` (suggested + checks), `draft.guards`, `draft.notes` and `limitations`. When `draft` + is null, read `draft_refusal` first: a missing `seed` or `end` + checkpoint means capturing again on a new build computer, while a failed + draft command names its own cause to fix. When `draft.files` is empty, + `draft.notes` says why: + - `no_change`: nothing was saved between `seed` and `end`. Capture again + on a new build computer, and save before marking `end`. + - `unowned_changes` with `no_document`: the app saved your document + where REA did not count it as the app's. Its `files` list what changed, + with their roles; the document is among them. Close the computer and + capture again on a new one with that path in `state_paths`; the export + refuses a check on a file outside the owned paths. + - `uncheckable_changes`: the app saved the change in dconf, as gedit does + its preferences, and no check reads dconf. Saving again does not help: + choose a change the app saves in a file, and capture again on a new + build computer. + - `unreadable_changes`: a dconf database changed but REA could not read + it; `error` says why. This is not evidence the app saved there: report + the error rather than changing the task. + + A note with a `file` says why that file's suggestions are thin: + - `by_position`: only position names what changed; its `elements` lack + a naming attribute. Check an ancestor's text, or give the element a + name in the app, such as an id, and capture again. + - `view_state_only`: every change is view state, a renumbered id or a + time the app stamps, and its `pointers` are the checks kept. The edit + was likely not saved: capture again and save it before `end`. + - `long_text`: the element's text is too long to check whole; check a + phrase in it with `text_contains`. + - `cut_view` or `unreadable_view`: REA's view holds only part of the + file, or none a check can read, such as a binary format (step 3). ## Write the task @@ -158,18 +235,37 @@ Write `{"name", "prompt", "checks"}` from the draft and what you did. - `prompt`: what the requester wants, in their words and the app's terms, naming the document as a user would and ending with saving it: "Change the poster's headline in Documents/poster.svg to Summer Bake Sale and save - it." Never say how the file stores it, and never give the checks. + it." Leave saving out for an app that saves on its own, after every change + or a few seconds after the last one, such as Osmo or Gnote: grading waits + until the app's files have held for 3 s, so the solving agent need not + wait for the save. A save more than 3 s after the agent stops is still + missed; name such an app in the report. Never say how the file stores it, + and never give the checks. - Checks are file-based: content checks that prove the edit, never only that a file exists or changed. Start from the suggestions; keep those that say what the user did and drop what follows from how the app saves. - Compare exact values, so a typo or a wrong number fails. +- Read each file's `values` before writing checks: every value the change + added, changed and removed, with `before` and `after` (a scalar, or an + element as compact XML, cut at 600 characters) at the pointer a check + names it by, so copy the pointer and the `after` value into a check. + Only XML files, archives, JSON and key files have values: a text file's + or SQLite database's `change` already lists its lines and rows. + `refers_to` says what a number stands for, such as `category="44"` + naming ``; use it for the prompt's words. + Skip entries whose `note` is `by_position` (no check can use that + pointer) or noise (`view_state`, `renumbered`, `volatile`, + `timestamp`, `upkeep`). `values_cut` means a limit left some out. - Select elements by what they hold, never by position: SVG text by id, `/root/children/[name=g][@inkscape:label=Layer 1]/children/[name=text][@id=headline]/children/[name=tspan]/children/0`, or an ODF paragraph by style, `/entries/[name=content.xml]/view/root/children/[name=office:body]/children/[name=office:text]/children/[name=text:p][@text:style-name=Title]/children/0`. A selector must match exactly one element, so give the target a unique id - or style in the seed. The draft may suggest positional pointers; rewrite - them. Only an element's text is read by position, as `children/0`. + or style in the seed. The draft's pointers already do this: a name-like + attribute such as Scribus's `ANNAME` before a coordinate, and two + attributes together where one is not enough, as a spreadsheet cell's + `[@Col=3][@Row=2]`. Only an element's text is read by position, as + `children/0`. - When an app may split an element's text into runs, check it whole: `text_equals` (or `text_contains` for a value inside a sentence) on the element compares all its text, however it is split. Add @@ -178,15 +274,43 @@ Write `{"name", "prompt", "checks"}` from the draft and what you did. `{"kind": "text_contains", "file": "Documents/memo.abw", "pointer": "/root/children/[name=section]/children/[name=p][@xid=6]", "value": "at 3:30 PM"}`. Point a text check at the paragraph or another element the edit keeps, never at a run such as AbiWord's `[name=c]`: a reopened document splits - the text into several runs. The draft lists only what changed; to name an - unchanged element, such as another paragraph's `xid`, use the app's usual - numbering and let the export confirm it, or read it in the exported - bundle's `envs//seed/`. + the text into several runs. The draft lists only what changed; an + unchanged element is in `draft.guards` when it is a like sibling of a + changed one, and otherwise use the app's usual numbering, such as another + paragraph's `xid`, and let the export confirm it. +- In an ODS spreadsheet no attribute names a row or a cell. Name the sheet + in the seed, and check its whole text: `text_equals` on + `/entries/[name=content.xml]/view/root/children/[name=office:body]/children/[name=office:spreadsheet]/children/[name=table:table][@table:name=Q1 Sales]`, + or a stretch of it with `text_contains`. Cell texts join with nothing + between, so a row reads `February23100`. +- For a line added to free text, such as an item on a note's list, the + draft compares the whole text with `text_equals`. This rejects longer or + negated items, but can reject a correct edit with different ordering or + spacing. Report that limitation when the task permits such variations. +- Some apps keep text in an attribute, such as Scribus's ``. + `text_equals` cannot join it, so check the attribute with `value_equals`: + `/root/children/[name=DOCUMENT]/children/[name=PAGEOBJECT][@ANNAME=Headline]/children/[name=StoryText]/children/[name=ITEXT]/attributes/CH`. + A partial edit the app restyles splits the text into several runs and + fails that check; list it in the report's limitations. +- When the draft shows no values or no checks, a scratch export can expose + `envs//seed/` only after the task has at least one valid check: it must + pass on the captured end state and fail on the seed. Before the first + successful export, use `act_on_build_computer` to open the saved file in an + available GUI text editor or viewer and inspect its actual representation + without changing it. Use the visible saved content and the task's edit to + write a check; an export refusal can also name the end state's value, such + as `is "44", not "1"`. If the file cannot be inspected or expressed by a + supported check, report the limitation instead of guessing a check. - Where easy, add one check that the rest is untouched, such as another - text's value the edit must leave alone. It is the one unchanged value - worth checking; it passes on the seed too, which is fine, since the export - needs only one check to fail there. -- Give `value_equals` a `tolerance` for a number the app writes as text. + text's value the edit must leave alone. `draft.guards` offers them at + their seed values, which the end holds too: a field on the nearest like + siblings, one the user sets, such as Planner's `work`, before a default + they all share; the text an addition kept; a text file's lines beside its + new one. Copy one or two into `checks`; they pass on the seed, which is + fine, since the export needs only one check to fail there. +- The draft suggests a decimal the app writes as text, such as `146.83`, + as a number with `tolerance: 0.001`; keep the tolerance for any number + you check that way. - When the draft shows an XML document's changes as `lines`, REA read it as text, for example because its DOCTYPE declares entities. Selectors are impossible there, and `line_count` with `contains` only approximates a @@ -215,6 +339,17 @@ is no longer empty. Keep `seed_files`, `scrubbed_files`, `unscrubbed_files` and `limitations` for the report. +The limitations name the folders the app wrote that the bundle neither owns +nor seeds, such as `~/.config/MuseScore`, so the app starts without them: +expect its first-run wizard, a tour or a "restore session?" question at every +boot. If the witness shows one, capture again on a new build computer with +those folders in `state_paths`, and dismiss the dialogs before `seed`. Lock +files the app holds while it runs, such as LibreOffice's `.lock` and +`.~lock.#`, are left out of the seed and listed after "Left out of the +seed", so you need not close the document or delete them before `seed`. The +limitations also name seeded WebKit or Chromium storage and caches; when the +witness's boot is clean, list them in the report. + Keep the build computer open until the witness holds: a task fix exports again from it. @@ -231,12 +366,19 @@ checks. Give every `check_build_task` the task exactly as exported. seed holds exactly one and the app takes a file. When the export's `limitations` say it opens no document, open it from the prompt's path, as a solving agent would. Take a `wait` screenshot first, since the - desktop may not have drawn yet. + desktop may not have drawn yet. When the start fails because the + computer never got ready, and its error names lock files in the bundle's + seed, delete them from the bundle and start again. 2. `check_build_task` with `stage: "seed"` and `directory`, the absolute path of a new or empty folder for the record. A check must fail. Recording starts. Later stages take no `directory`. 3. Do the task from the prompt with `act_on_build_computer` and save. `check_build_task` with `stage: "after_edit"`: every check must pass. + Each check waits until the app's files have held for 3 s, as the + bundle's verifier does, so an app that saves on its own needs no wait of + yours. A result with `settled: false` means the files were still + changing at the 15 s bound, and `read_cut: true` that a slow read was + still running there. 4. Make a near miss on top of it: a plausible wrong edit, such as a typo or a wrong number, and save. `check_build_task` with `stage: "near_miss"`: a check must fail. @@ -286,7 +428,9 @@ Report: `witness.json` and the recording; - the seed's contents: `seed_files` and what each holds; - `scrubbed_files` and `unscrubbed_files`; an unscrubbed file still names - the host or user, so it needs review before sharing; + the host or user, so it needs review before sharing. `recompressed_files` + are gzip seeds whose headers were cleared, their content unchanged: give + their count; - limitations: the capture's, the export's and anything you could not do; - solver runs, if any: the model, the runs and the passes; - the build time and the questions you asked. diff --git a/.agents/skills/reverse-engineer-anything/references/environments.md b/.agents/skills/reverse-engineer-anything/references/environments.md index 63dd4e78b6..521357983a 100644 --- a/.agents/skills/reverse-engineer-anything/references/environments.md +++ b/.agents/skills/reverse-engineer-anything/references/environments.md @@ -308,9 +308,20 @@ Run `rea describe-desktop-application-task CAPTURE --format json`. Read: and last version, with its `role`, the `baseline` and `final` versions and the net `change`. Text changes list each added and removed line once, and `line_counts` gives lines found at both ends whose count changed. +- `values` in each of those files: what the change added, changed and + removed, `before` and `after`, at the pointer a check names it by, with + `refers_to` naming the element a key such as `category="44"` stands for. + Skip entries noted `by_position` or as noise. Only XML files, archives, + JSON and key files have values: a text file's or database's change + already lists its lines and rows. - `task`: a draft whose `checks` are suggestions from the net change of the documents the user worked on, or of the app's own state when no document changed. +- `guards`: optional checks that the rest kept its seed values, such as a + like sibling's field or a text file's neighbouring lines; add one or two + to show the rest is untouched. +- `notes`: why `files` is empty, or why a file's suggestions are thin, such + as `by_position` when no attribute names the changed element. ### Write the prompt @@ -346,7 +357,9 @@ document saved as a zip archive, such as FreeCAD's `.FCStd`, reads as its entries: point into an entry's view, and name array members by selector rather than position, as `/entries/[name=Document.xml]/view/root/children/[name=Object][@name=Hole]`, -so the check survives an app version that writes them in another order. Give +so the check survives an app version that writes them in another order. A +gzip-compressed document, such as Gnumeric's `.gnumeric`, reads as the +document inside: point into it as into the same document uncompressed. Give `value_equals` a `tolerance` to compare a number the app writes as text, such as `"5.0000000000000000"`, with a number. Check what the user wrote in a document with `text_equals` on its paragraph element, not `value_equals` on a diff --git a/bridge/agentenv/capture.py b/bridge/agentenv/capture.py index 0162918f17..599bd2e7f0 100644 --- a/bridge/agentenv/capture.py +++ b/bridge/agentenv/capture.py @@ -1,7 +1,9 @@ """Run REA inside an agentenv build computer for REA on the host: capture the app, draft its task and export it. REA sends this file's source through the computer's ``computer-exec/v1`` route as -``python3 -I -c ``; each command prints one JSON line and exits. ``start`` +``python3 -I -c ``. JSON that would exceed the bytes Linux +passes as one string travels as several arguments, which this file joins before parsing. Each command prints +one JSON line and exits. ``start`` leaves the capture running on its own: its stdin is a FIFO it holds open for reading and writing, so it never sees the end of its input, and ``mark`` and ``stop`` write their requests there and wait for the capture's own acknowledgement on its progress log. A small shell around the capture writes its exit status to a file, the @@ -91,6 +93,13 @@ def _exit_code(capture: str): return None +def _ended(capture: str, code: int) -> dict: + """A capture that has exited: its status, the end of its progress log and its output's size, which REA reads + whole for the Evidence or failure it ended with, whichever step was waiting on it.""" + return {"exited": code, "progress": _tail(os.path.join(capture, "progress.log")), + "output_bytes": os.path.getsize(os.path.join(capture, "capture.out"))} + + def _event_line(path: str, offset: int, kinds: set, name): """The first progress line after ``offset`` whose event is of one of ``kinds``, for ``name`` when given.""" try: @@ -119,17 +128,49 @@ def _wait(capture: str, offset: int, kinds: set, name, timeout: float) -> dict: return {"line": line} code = _exit_code(capture) if code is not None: - return {"exited": code, "progress": _tail(log), "output": _tail(os.path.join(capture, "capture.out"))} + return _ended(capture, code) if time.monotonic() > deadline: return {"timed_out": True, "progress": _tail(log)} time.sleep(POLL_S) +def _within(path: str, folder: str) -> bool: + return path == folder or path.startswith(folder + os.sep) + + +def _state_paths(values: list, work: str): + """Each state path as an absolute path strictly under the home and outside REA's work folder, both as named + and with its links resolved, or the refusal of the first that is not. The capture resolves the links again + and retains what they lead to, so a link out of the home is refused here. Each passes on as named, so the + capture keeps what its spelling means, such as a hidden settings folder.""" + home = os.path.normpath(os.path.expanduser("~")) + real_home, real_work = os.path.realpath(home), os.path.realpath(work) + spellings = {home, real_home} + paths = [] + for value in values: + relative = value[2:] if value.startswith("~/") else value + path = os.path.normpath(os.path.join(home, relative)) + if ".." in relative.split("/") or not any(path.startswith(spelling + os.sep) for spelling in spellings): + return None, f"The state path {value} is not under the computer's home, {home}." + # Every link on the way is followed, a dangling one too; a path that does not exist yet resolves + # through its nearest existing folder. + resolved = os.path.realpath(path) + if not resolved.startswith(real_home + os.sep): + return None, f"The state path {value} leads through a link to {resolved}, which is not under the computer's home, {home}." + if _within(path, work) or _within(resolved, real_work): + return None, f"The state path {value} is in REA's work folder {work}, which every capture leaves out." + paths.append(path) + return paths, None + + def start(arguments: dict) -> int: work, capture = os.path.abspath(arguments["work"]), os.path.abspath(arguments["capture"]) rea = shutil.which("rea") if rea is None: return _say({"error": "REA is not installed in this computer; start the computer from a request, which installs it."}) + state_paths, refusal = _state_paths(arguments.get("state_paths", []), work) + if refusal is not None: + return _say({"refused": refusal}) if os.path.exists(capture): return _say({"error": f"the capture folder {capture} is already used"}) apt = _apt(work, arguments.get("apt_timeout_s", 600)) @@ -143,6 +184,8 @@ def start(arguments: dict) -> int: "--output", os.path.join(capture, "capture"), "--exclude", work, "--json"] for value in arguments.get("arguments", []): argv += ["--argument", value] + for path in state_paths: + argv += ["--state-path", path] if arguments.get("package_file"): argv += ["--package-file", arguments["package_file"]] # Reading and writing, so the capture never reads the end of its input between requests. @@ -175,11 +218,13 @@ def _request(capture: str, line: str) -> bool: def mark(arguments: dict) -> int: + """Request a checkpoint and answer with the capture's acknowledgement line as it wrote it: the checkpoint's + counts and up to ten of the files that changed, home-relative, with a count of the rest.""" capture, name = os.path.abspath(arguments["capture"]), arguments["name"] code = _exit_code(capture) log = os.path.join(capture, "progress.log") if code is not None: - return _say({"exited": code, "progress": _tail(log)}) + return _say(_ended(capture, code)) # A checkpoint already taken, whose answer was lost, is answered from the capture's own log. taken = _event_line(log, 0, {"checkpoint"}, name) if taken is not None: @@ -270,4 +315,4 @@ def listing(arguments: dict) -> int: COMMANDS = {"start": start, "mark": mark, "stop": stop, "run": run, "list": listing} if __name__ == "__main__": - sys.exit(COMMANDS[sys.argv[1]](json.loads(sys.argv[2]))) + sys.exit(COMMANDS[sys.argv[1]](json.loads("".join(sys.argv[2:])))) diff --git a/docs/desktop-application-capture.md b/docs/desktop-application-capture.md index 6a72342ea6..7b8f8ef14c 100644 --- a/docs/desktop-application-capture.md +++ b/docs/desktop-application-capture.md @@ -94,12 +94,20 @@ Progress lines on stderr are JSON, `{"rea_progress": {"phase": "desktop_capture", "message": ...}}`. Those a driver waits on also carry an `event`: `{"kind": "window_open"}` once the window opens; `{"kind": "checkpoint", "name", "trigger", "at_ms", "entries", "changes", -"retained_files", "retained_bytes"}` once a checkpoint has been taken, the -baseline, `stop` and `exit` included, with the paths walked, the paths -changed since the checkpoint before and the files retained; and `{"kind": -"checkpoint_refused", "name", "reason"}` for a requested name it refused. A -driver that sends `checkpoint ` waits for the event naming it rather -than for a fixed time. +"retained_files", "retained_bytes", "changed_files", "more_changed_files"}` +once a checkpoint has been taken, the baseline, `stop` and `exit` included, +with the paths walked, the paths changed since the checkpoint before, the +files retained, and up to ten of the files (not folders) that changed, +home-relative: those under a `--state-path` or absolute `--argument` first, +then those the application held open, then those in its state directories, +then the rest, and last caches (a folder whose name ends in `cache`, such as +`~/.cache`) and WebKit or Chromium storage, each by path, with a count of the +rest. An application can rewrite dozens of cache files on a save, and KDE's +holds its service cache open, so the saved document would otherwise fall past +the ten; +and `{"kind": "checkpoint_refused", "name", "reason"}` for a requested name it +refused. A driver that sends `checkpoint ` waits for the event naming it +rather than for a fixed time. When the session ends the capture sends SIGTERM to the launched process, its process group and every process that carries the run token, waits up to five @@ -310,7 +318,9 @@ the whole name is compared, so `~/.config/GIMPshop` does not. Dotfiles in the home named after the application (`.rc`, or `.` followed by `-`, `.` or `_`, for a name of three or more characters) that exist at the start join them, since an application may only read such a -file during the capture. A `flatpak run` launch is named after its application +file during the capture. So does KDE's settings file in the config folder, +`rc` such as Tellico's `~/.config/tellicorc`, for each such name of three +or more characters, whether or not it exists yet. A `flatpak run` launch is named after its application id rather than the `flatpak` launcher, and adds `~/.var/app/`. **Session.** When it started, how long it ran and why it ended; the launched @@ -318,8 +328,9 @@ process's exit status; every process seen in the application's tree; every client window with its class pair, titles, owner pid and type, marked when it belongs to the application; when the first application window appeared; the most common class pair of the application's windows, for matching its window -later; files the application's processes held open under the walked roots; the first -8 KiB of the launched process's stderr; and whether cleanup was verified. Each +later; files the application's processes held open under the walked roots; the last +8 KiB of the launched process's stderr, where a failing app's last words are, +with `stderr_truncated` when earlier output was dropped; and whether cleanup was verified. Each process records whether the capture started it: the launched process, a process carrying the capture's run token, or a descendant of one, even when its environment cannot be read, as for a zombie or a sandboxed renderer. @@ -345,7 +356,9 @@ directory, which are named in the limitations instead; a selected path that is not a directory brings the `-wal` and `-journal` paths beside it, so a database selected on its own is captured with its logs), the state directories as walked (their physical spelling, which differs from the inventory's when one -is a symlink), the paths every walk left out (the output folder and +is a symlink), the selected paths (each `--state-path` and content-root +`--argument` as named, absolute, beside the physical path walked, so a +symlinked settings folder keeps what its name means), the paths every walk left out (the output folder and `--exclude` values), and the checkpoints in order: `baseline` before launch, each requested checkpoint, `stop` or `time-limit`, and `exit` after the application has gone. Each checkpoint records how many paths its walk saw, @@ -374,7 +387,8 @@ unreadable SQLite sidecar from an absent one. Captures made before this field was added must be made again before state inspection, task drafting or export. Directories the application held files open in, but whose starting content was not retained, are listed so a second capture can pass them with -`--state-path`. +`--state-path`, or `state_paths` on a build computer's `start_build_capture`, +and a limitation names them, spelled from `~/` under the home. **Limitations.** What the capture could not establish, such as a truncated walk or a missing window. diff --git a/docs/desktop-application-export.md b/docs/desktop-application-export.md index c3b15bc6b8..c3487dfa43 100644 --- a/docs/desktop-application-export.md +++ b/docs/desktop-application-export.md @@ -33,8 +33,10 @@ The command returns Evidence with operation `export_desktop_application` and predicate type `rea.desktop-application-export`, listing the environment name, the seed and end `checkpoints`, owned paths, seed files, the seeds it rewrote to remove the capture host's name and user (`scrubbed_files`), the -bundle files that still hold them (`unscrubbed_files`), and written files, -the last three by their path in the bundle. A failed export, cancellation +gzip seeds it compressed again only to clear their headers +(`recompressed_files`), the bundle files that still hold them +(`unscrubbed_files`), and written files, the last four by their path in the +bundle. A failed export, cancellation included, removes the bundle it had not yet published, so no partial copy of the user's data remains. If the bundle was published but its output handles could not be closed, the command reports a cleanup error that carries the export @@ -154,9 +156,11 @@ The class declares: Files the capture recorded under a symlinked state directory's physical spelling are mapped back to the innermost such directory's home spelling, unless that spelling is outside the captured home, as when the home itself is - a symbolic link. When two files map to one home path, the export uses the one - present before the capture, else the first by path, and a limitation names - the other. + a symbolic link. When a selected link names an application state directory, + the bundle seeds the application's state path and restores the selected + alias at boot. Selections nested under that alias are seeded under the same + state path. An existing incompatible alias stops setup without replacing it. When two files map to one home path, the export uses the one present + before the capture, else the first by path, and a limitation names the other. - Data hooks, because declaring any hook switches off agentenv's default data plane: the class seeds the home once per boot and keeps it as the initial state with `keep_initial`, `data/add` and `data/reset` call agentenv's `add` @@ -169,10 +173,43 @@ retain is not seeded: one in the state view without retained content at that checkpoint, or one the capture omitted (for example too large or past the content budget). The result lists them in `unseeded_files` and the README names them. When the application held files open in folders whose starting -content the capture did not retain, a limitation names those folders. The -desktop Trash, `~/.local/share/Trash`, is never the application's, so a file -the user moved there is neither owned nor seeded, even when a window title -names it; a limitation lists the files the Trash held at the seed. +content the capture did not retain, a limitation names those folders, spelled +from `~/` under the home: a file there is seeded only when the capture +retained it, as it does one written during the capture, and the limitation +names the seed files there or says there are none. The desktop Trash, +`~/.local/share/Trash`, is never the application's, so a file the user moved +there is neither owned nor seeded, even when a window title names it; a +limitation lists the files the Trash held at the seed. + +Files that changed during the capture outside the owned paths are neither +owned nor seeded, so the app starts without what it saved there: MuseScore +keeps its settings in `~/.config/MuseScore`, which its name rules miss, and +showed its first-run wizard on every boot. A limitation and the README name +those folders, leaving out desktop services' files, each collapsed to its +folder in `~/.config`, `~/.local/share`, `~/.local/state` or `~/.local`, the +hidden folder of the home it lies in, or else its own folder, up to eight and +a count of the rest. Capture again naming them with `--state-path`, or +`state_paths` on a build computer, to own them. + +Lock files the app holds while it runs are left out of the seed: a copied one +makes the app find its profile or document in use, and a LibreOffice Calc +bundle whose seed held its profile's `.lock` showed no window at boot. They +are `lockfile`, `lock`, `.lock`, `.parentlock`, `parent.lock` and +`SingletonLock` anywhere, LibreOffice's document lock `.~lock.#`, and +`*.lock` or `*.lck` directly in one of the application's state directories. +Elsewhere such a file may be a project's own, such as `Cargo.lock` or +`yarn.lock`, which pin dependencies, and is seeded, even under a folder whose +name starts with a dot. A file the state view makes a document, or one named +itself by `--state-path` or an absolute `--argument`, is never left out, +whatever its name. A limitation and the README list them, up to +eight and a count, after "Left out of the seed". They stay under the owned +paths, so `data/get` returns the lock the running app holds, and `data/reset` +restores the seed without them. Other seed files the app keeps for its own +running are seeded as captured and named, collapsed to the folder that holds +them: WebKit or Chromium storage (such as `hsts-storage.sqlite`, `storage/`, +`mediakeys/`, `Local Storage/`, `IndexedDB/`, `Cookies`), which may hold the +app's own data, and caches (a folder whose name ends in `cache`); a stale one +can change how the app starts. The application's dconf keys at the seed, if any, are written once when the app first starts, also for an application that owns no files, and the class adds @@ -182,7 +219,11 @@ existing user database through a `file-db:` profile, then runs `dconf compile` on that dump and the captured keys, in that name order so the captured keys win, and replaces `~/.config/dconf/user`. If that fails, the server prints a one-line warning and the app starts without the captured keys. `data/reset` does -not restore them. +not restore them. No file of the bundle holds them, so `data/get` does not +return them and no task check reads them; a draft whose only change between +its checkpoints is to the application's dconf keys says so with +`uncheckable_changes`, and one whose dconf database changed but could not be +read says so with `unreadable_changes`. ## The seed checkpoint @@ -252,6 +293,26 @@ export removes them from what the bundle ships: boot, as it did the captured home before. The replacements read as the computer's own: its user is `user`, with its home in `/home/user`, and `localhost` is the name every host has for itself. +- A gzip-compressed seed, such as a Gnumeric workbook, is decompressed, up to + 256 MiB, scrubbed as the format the state view read its document as, and + compressed again whether or not its document changed, so no file name, + comment or time its header held ships: with no file name, comment or extra + field, modification time 0 and the operating system unknown, so the same + document gives the same bytes on every host. It is deflated at the fastest + level, or stored when deflating would make it expand more than the 100 times + a view reads, so a seed that read still reads. It is listed in + `scrubbed_files` when its document changed, and otherwise in + `recompressed_files`, which the README counts rather than lists, so the + example and template files an app copies on first run do not bury the + document scrubbed; one already in that form ships as captured. The class copies it at + boot as it is, so the home it holds stays `/home/user`. One that does not + decompress ships as captured, its bytes searched, and one the state view did + not read is only searched. One whose document needs no edit but a member of + which has an extra field, where BGZF (`.vcf.gz`, `.bam`) and dictzip + (`.dict.dz`) files index their blocks, also ships as captured, its bytes + searched, so its blocks stay where its index says; one whose document + changed is still compressed again as one member, losing them, as the + export's limitation says. - Binary files, SQLite databases among them, and text files over 256 MiB are copied as captured. Then every file the bundle ships, the class module with its dconf keys and the README included, is searched by a looser rule than @@ -318,6 +379,39 @@ declared vendor repository installs the package already; when the URL is not written as `http://` or `https://` or holds a user name or password; and when the file is built for an architecture other than `amd64`, `arm64` or `all`. +## Grading once the app has saved + +A task's verifier, `artifacts/-check/verify.py`, grades the app's files +only once they have settled, because an app can save on its own timer after the +last change. Gnote saves a note 4 s after the last key, so a read taken as +soon as the agent stops finds the text on screen but not yet in the file. The +verifier reads the app's `data/get` every 0.5 s. It compares each file's bytes +between reads by their SHA-256, as `data/get` returns them, and grades once +every file has held for 3 s. The 3 s count from when a read returned the +current bytes, not when it began, so a slow read that brings a change late +does not count its own run as quiet. 15 s after the first read began, the +bound, the verifier grades the last completed read: no read starts at or +after the bound, and a read still running then is cut. Each result records +whether the files settled in `settled`, and whether the bound cut a read in +`read_cut`. + +The window has a cost and a limit: + +- An app whose files do not change during the check, such as one that saved + at once, is graded 3 s after the first read returns. +- A save that lands more than 3 s after the first read returns is still + missed. An app that saves 4 s after the last key is caught only when the + agent stops at least a second after that key. +- Files the app rewrites more often than every 3 s, such as a log, never + settle, so every check waits until the bound. +- The bound counts the reads too, so a slow `data/get` ends the check at + 15 s. Only the first read, which every grade needs, runs to its own + timeout. + +The bound and window are `SETTLE_BOUND_S` and `SETTLE_WINDOW_S` in the +verifier. `check_build_task` waits the same way before it grades each +witness stage; see [witnessing](desktop-build-witness.md#witness). + ## Typed views The views follow the [state view](desktop-application-state.md)'s formats and @@ -332,7 +426,10 @@ as their entries, read with Python's `zipfile` from the file in place, with REA's entry, size, compression ratio and archive limits and its wording for an entry it does not read. The module reads the central directory itself, as REA does, and shows `zipfile` only the records it lists, reading each entry in -bounded chunks no further than a byte past its declared size. A JSON file +bounded chunks no further than a byte past its declared size. A gzip file is +the view of the document inside it, with `"compression": "gzip"` beside its +format, decompressed with Python's `zlib` within the same entry size and ratio +limits, its members joined as REA joins them. A JSON file holding a number JavaScript cannot read exactly, such as an integer above 2^53, or whose retained value nests more than 500 levels deep, gets an `error` instead of a view, as in REA. Duplicate keys keep their last value before the depth check. Text that is not one JSON diff --git a/docs/desktop-application-state.md b/docs/desktop-application-state.md index 5bc2cd6fec..c586838fc1 100644 --- a/docs/desktop-application-state.md +++ b/docs/desktop-application-state.md @@ -40,7 +40,16 @@ The role follows from the reasons, in this order: `shared_desktop` for a file in the desktop Trash, which the user deleted, whatever else names it; `application_state` for application keys in dconf, or a state directory outside shared desktop locations; `document` for a file named in a window title or selected by the -caller, even if the application holds it open; `application_state` for another +caller, even if the application holds it open, except that a selected file +under a hidden folder of the home, such as `~/.config/MuseScore`, a dotfile +there, or a lock an application holds, such as LibreOffice's `.~lock.#` +beside a document, is `application_state`, judged by the path as the caller +named it, so a file reached through `~/.config/MuseScore` stays the application's +when that folder is a link to `~/settings/MuseScore`; the innermost selected +path holding a file decides, so a score selected itself inside the link's +target keeps its own name and is a document (a capture written before it +recorded selected paths is judged by their physical spelling); +`application_state` for another open file outside shared locations; `shared_desktop` for other files in shared locations; `unrelated` otherwise. A window title names a file only when no other file name character adjoins the name, so `day.txt` does not match @@ -75,6 +84,9 @@ whether the file was present, the SHA-256 of its retained bytes, and a view: | `text` | Other UTF-8 text | Lines, up to 5,000 | | `binary` | NUL bytes or invalid UTF-8 | None | +A gzip file is read as the document inside it; see +[compressed documents](#compressed-documents). + A JSON value holding a number JavaScript would round, such as an integer above 2^53, is not read rather than shown rounded. @@ -193,6 +205,37 @@ so an archive of hundreds of thousands of entries costs no more to read than one of a thousand. An archive whose central directory cannot be read has a `view_error` of its own. +## Compressed documents + +Gnumeric's `.gnumeric`, Dia's `.dia`, Xournal++'s `.xopp`, GnuCash's default +book and Inkscape's `.svgz` are XML compressed with gzip. A file that starts +with gzip's magic bytes, `1f 8b 08`, whatever its name, is read as the +document inside it: its members joined, then given a format and view by the +rules above under the file's own name, so a pointer or check into it is +written exactly as for the same document uncompressed, and the task draft is +the same. The version records the compression beside the format: + +```json +{ + "format": "xml", + "compression": "gzip", + "view": { + "root": { "name": "gnm:Workbook", "attributes": {}, "children": [] }, + "truncated": false + }, + "view_error": null +} +``` + +The document is read only when it is at most 16 MiB and at most 100 times the +file's size, an archive entry's limits, so decompressing stops there; past +them, or when the stream is damaged or cut short, the version has `format` +null and a `view_error`, and a check into it is undecided. A SQLite or dconf +database inside keeps its format and has a `view_error`, an archive inside is +read as an archive, and another gzip stream inside is binary. A gzip file +inside an archive reads as binary. Other compression, such as xz, Zstandard +or bzip2, is not read: such a file is binary. + ## Kinds of change Between consecutive versions a change is `created`, `deleted`, or a diff --git a/docs/desktop-application-tasks.md b/docs/desktop-application-tasks.md index bc01fa5118..7e644ebf04 100644 --- a/docs/desktop-application-tasks.md +++ b/docs/desktop-application-tasks.md @@ -42,9 +42,31 @@ operation `describe_desktop_application_task` and predicate type database is compared by its rows, since its hash leaves out the `-wal` file. Text changes list each line once and add `line_counts`, the lines found at both ends whose number of - occurrences changed. Views appear only for the files the suggestions cover, - with arrays past 200 items and strings past 2,000 characters cut + occurrences changed. An XML file's change names its members by the + [selectors](#pointers-and-selectors) checks use, as an archive's already + does, and by position only where nothing names a member or where only the + members' order changed. Views appear only for the files the suggestions + cover, with arrays past 200 items and strings past 2,000 characters cut (`view_cut`). +- `values`, in each of those files: the values its change added, changed and + removed, each with the `pointer` a check names it by, its `kind` (`added`, + `changed` or `removed`), and `before` and `after`: the scalar, an element as + compact XML with the text the XML reader kept, spaces between runs included, + or a JSON value as compact JSON, cut at 600 characters, and null at the end + that lacks it. A file created or deleted lists what its one end holds, + `added` or `removed`: an archive's entries, an XML file's root element, a + JSON file's members or a key file's keys, each whole once. A value only + position reaches has `note: "by_position"`, and its pointer cannot go into a + check; noise an app writes as it saves has `view_state`, `renumbered`, + `volatile`, `timestamp` or `upkeep` (see `task`). `refers_to` names up to + three other elements a value equals the `key` or `id` of, with the + `attribute` holding it, as HomeBank's `category="44"` names + ``; where several elements share that key, + the one whose tag shares the attribute's stem, as `payee` and ``. What a check can + name comes first, then what only position reaches, then noise. A file lists + at most 20 values and 8 KB, the draft 24 KB, and `values_cut` says when a + limit left some out. Values cover XML, archives, JSON and key files; a text + or SQLite file's change and suggestions already carry its lines and rows. - `task`: a draft with the name `change`, a placeholder prompt the export refuses, and suggested checks derived from the net change of the files with role `document`, or of the app's own state files when no document changed. @@ -56,21 +78,151 @@ operation `describe_desktop_application_task` and predicate type and the JSON, key file and XML views of a zip archive's entries, gain checks by [selectors](#pointers-and-selectors) and never through an element by position: in XML, first [`text_equals`](#whole-text) for each element whose - text changed, then an element added whole that it exists, one removed whole - that it is absent, then values changed, added and removed, an entry or a - JSON value added whole through its members, and an empty array or object as - it is. The text check names the paragraph rather than a run inside it: it - goes up from an element sitting in its parent's text, and from one named by - its tag or position alone to the nearest ancestor named by an attribute, as - AbiWord's `p` by its `xid`; the text nodes and runs inside it get no other - check, but an element holding no text that is added or removed there, such - as an image or a line break, keeps its own, since the text check never - reads it. Whitespace an app lays out between elements is not checked, and - values that look like a time the app stamps, in ISO 8601 or as C's - `asctime` writes it (`Thu Nov 5 09:20:31 2026`), are left out. + text changed, then elements added or removed whole, then values changed, + added and removed, an entry or a JSON value added whole through its + members, and an empty array or object as it is. + - A selector names a member by its tag, then by its identifiers (`name`, + `id`, `xid`, `xml:id`), then by one other attribute, a name-like key such + as Scribus's `ANNAME` or `inkscape:label`, a worded value or an integer of + up to five digits first, then by two such attributes together, as + Gnumeric's cells by `[@Col=3][@Row=2]`, and last by a coordinate, a long + number or a token, such as `XPOS` or `ItemID`, or an attribute naming + view state, such as which layer is `selected`. A pair counts every + member holding both its values, whichever attributes that member ranks + first. An element with identifiers is named by those alone. The selector + must name the member alone at both ends, with the same values. An + element whose one identifier changed, at its place among its tag's + members, with every other attribute and its children alike, is that + element renamed, as a MyPaint layer renamed from `Untitled` to `Sun`: it + is named by its first other attribute naming it alone at both ends, as + `[@src=data/layer-00.png]`, so the rename reads as its `name` changed, + and the old name gets a check that it names nothing, unless its change + is noise, as an id the app renumbered. + An identifier the app numbers by position names nothing: where the + members of a tag hold it as 1..N (or 0..N-1) in document order at both + ends, members were inserted or removed, and every member another + attribute names alone at both ends kept its number or moved it the way + the count did, one at least, as Planner's task `id` after a task is + inserted, members are paired by their other attributes, so an insertion + reads as one task added. A value two members swap, or an id appended at + the end, as Tellico's next ``, leaves the ids identity. + - The text check names the paragraph rather than a run inside it: it goes up + from an element sitting in its parent's text, and from one named by its + tag or position alone to the nearest ancestor named by an attribute, as + AbiWord's `p` by its `xid`; where no attribute names one, to the nearest + ancestor within three levels its tag names with no step by position, as + MuseScore's `VBox` around untitled `` elements; and where only + position reaches the element, to the ancestor above the outermost step by + position, by an attribute of its own that names it alone where one does, + as an ODS sheet's `table:table[@table:name=Q1 Sales]` above its rows. The + text nodes and runs inside it get no other check, but an element holding + no text that is added or removed there, such as an image or a line break, + keeps its own, since the text check never reads it. A decimal that is an + element's only text, such as a spreadsheet cell's, is checked by its + number instead. Free text that gains a line, such as a shopping list, + uses `text_equals` on the whole text: the current XML checks cannot + compare an unstructured line as a whole item. This can reject a correct + edit that puts the item elsewhere or changes spacing. A record added + or removed whole, an element holding only elements, its fields, such as + Tellico's `` or a record of repeated `` elements, never + makes the list holding records of its kind a check, whose text would hold + the records the edit left alone, except a record removed whole that only + position names, whose absence only the list's text shows. Text made of + runs is no record: an element whose tag names a paragraph or a piece of + text (`p`, Writer's `text:p` and `text:h`, `h1` to `h6`, `li`, `para`, + SVG's `text` and `tspan`, a `span` or an `a`), or whose runs split a + sentence at a space, as `Meet ` and a linked `Monday`. Nor does an + element holding a date the app stamps in parts take a text check. + - An element added to a list, where another element of its tag is at either + end, gets a `value_equals` for each of its attributes, best ranked first, + at most six, leaving out its selector's attributes, identifiers, times, + volatile values and view state, as HomeBank's new `` gets its + `amount`, `category` and `payee`. An element whose other attributes, two + or more, a like sibling holds every one of, as a new Planner task copies + its neighbour's defaults, and any other element added whole, get a check + that it exists. A record added to a list that a selector names, as + Tellico's ``, gets a `text_equals` for each field, at most + five, each by its selector among the record's fields, as `[name=title]`, + leaving out a field repeating the record's identifier, a date the app + stamps in parts, and blank text or times. A record only position names, + such as Osmo's `` of fields, also gets a `text_contains` for each + new field of three or more characters, at most five, at the nearest + ancestor a check can name, each absent from that ancestor's text at the + seed. + - A decimal written as text, such as `146.83`, is suggested as a number + with `tolerance: 0.001`, unless the seed's value is that close too. + - Noise leaves the suggestions when the file's change holds anything + else, a change only position reaches included: view state, which a tag + or attribute on the way names (the cursor, a selection, scrolling, zoom, + the viewport, windows, splitters, Inkscape's `namedview`, view and + configuration settings, the printer, a save's history, edit time, + editing duration and cycles, last-saved time, the generator and + document statistics), or the value of a name-like attribute or `key` on + the way, when it is one token of several words, as LibreOffice's + `config:config-item[@config:name=CursorPositionX]` inside + `[@config:name=ooo:view-settings]`; an attribute ending in `ID` or `Id`, + or an `rsid`, that the app renumbered, with everything else in its + element, as Gnumeric empties a formula cell whose shared expression it + renumbers, and an identifier the app numbers by position; volatile + values, UUIDs and epoch times, and durations and counts under a + document's metadata; times the app stamps, also as a date written in + parts whose tag, or one it sits in, names when the app made, changed or + saved something, as Tellico's `` of ``, `` and + `` (`cdate`, `mdate`, `created`, `modified`, `saved`, `updated`), + which no check ever reads, while a date a user enters in parts, as a + birthday, is an edit like any other; and `upkeep`, the + `loext:tab-stop-distance` attributes LibreOffice fills in `styles.xml` + while another archive entry holds an edit. However much noise + comes first, it never takes a check's place. When the change holds + nothing else, the checks stay and a note flags them: up to three in a + document, and as many as fit in the app's own state, where a view + setting may be the task. + - Whitespace an app lays out between elements is not checked, and values + that look like a time the app stamps, in ISO 8601, as C's `asctime` + writes it (`Thu Nov 5 09:20:31 2026`) or as a time of day to the + second (`05:16:21.675302540`), are left out; noise never makes its + ancestor's whole text a check. +- `guards`: checks an author may add to show the edit left the rest alone, + kept out of `task.checks`, with ids `guard-1` and on: for each suggested + `value_equals` or `text_equals` on a member of a list, a field on up to + two like siblings, nearest first, at its seed value, which the end holds + too: the check's own field where like siblings hold different values in + it, and otherwise one that varies among them, as Planner's `work` rather + than the `type="normal"` every task shares; never through a sibling only + position or a renumbered id pairs. A text file gets a `lines_present` of the + unchanged list items beside its first new line with the same marker, as + Zim's other `[ ]` action items, or else its nearest lines with words in + them. At most four a file. They pass on the seed, which the export allows, + since one check failing there is enough. +- `notes`: why `files` is empty, or why a file's suggestions are thin. Each + note has a `code` and a `message`. For a draft with no files: `no_change` + when no file under the home changed between the two checkpoints and nor + did the application's dconf keys; `unowned_changes` when files changed but + none under the owned paths, with up to five of them in `files`, each with + its home path and role (desktop services' files last), and how many more + in `more`; `uncheckable_changes` when the application's keys in a dconf + database changed, with the database in `files`: no check reads dconf, and + the export only writes the keys as they were at the seed checkpoint, so + make the task's change one the app saves in a file; `unreadable_changes` + when a dconf database changed but REA could not read it, with the read + failure in `error` (install `dconf-cli` or set `REA_DCONF_COMMAND` when it + is missing), since whose keys changed is then unknown; and `no_document` + when no file has the document role, as when no window title shows the + file's name. A capture again with `--state-path` naming the file, or + `state_paths` on a build computer's `start_build_capture`, makes it the + app's. For a drafted file, named in `file`: `by_position` when + changes reach their element only by position, the tags no attribute names + alone in `elements`, or when only the elements' order changed; + `view_state_only` when every change is noise, the checks kept in + `pointers`; `long_text` when an element's text changed past the 2,000 + characters a text check compares, the elements in `pointers`; `cut_view` + when the views hold only part of the file; and `unreadable_view` when no + check can read it, as a document REA reads as binary. The suggestions are a starting point. Keep the ones that say what the user -did; drop the ones that only follow from how the app saves its file. A LibreCAD +did; drop the ones that only follow from how the app saves its file. Read +`values` for what changed and its pointers, and `refers_to` for what a number +stands for, before writing the prompt and any check of your own. A LibreCAD capture that added four holes to a drawing rewrote it from DXF R12 to DXF 2007, so most of its lines changed; the draft suggests counts of `SECTION`, `ENDSEC` and `CIRCLE`, and only `CIRCLE` going from 5 to 9 is the user's change. @@ -129,7 +281,9 @@ The rules both evaluators follow: `{"groups": {"GROUP": {"KEY": "VALUE"}}}`, so a setting is `/groups/GROUP/KEY`; XML as `{"root": {"name", "attributes", "children"}}`; SQLite as `{"tables": {"TABLE": {"columns", "rows"}}}`; a zip archive as - `{"entries": [{"name", "format", "view", …}]}`. + `{"entries": [{"name", "format", "view", …}]}`; a gzip file, such as a + Gnumeric `.gnumeric`, as the document inside it, so its pointers are those + of the same document uncompressed. - With `tolerance`, a non-negative number, `value` must be a number, and the value found passes when it is a number, or a string that is a decimal numeral such as `"5.0000000000000000"`, at most `tolerance` from `value`. @@ -322,7 +476,10 @@ The bundle then holds, besides the app and `tasks/smoke.json`: The verifier finds the app among the computer's child environments, calls its `data/get` under the run's grant of the computer's control token, and reads -the `state` data part. It raises when the app cannot be reached, so the run +the `state` data part. It reads again until the app's files settle, as +[grading once the app has saved](desktop-application-export.md#grading-once-the-app-has-saved) +describes, and each result's `settled` says whether they did, and `read_cut` +whether the bound cut a read still running. It raises when the app cannot be reached, so the run fails rather than scoring the agent. The README lists the task, its prompt, its checks and the recipe below. diff --git a/docs/desktop-build-computer.md b/docs/desktop-build-computer.md index 979b92651f..ebb94b3ec4 100644 --- a/docs/desktop-build-computer.md +++ b/docs/desktop-build-computer.md @@ -45,9 +45,11 @@ From a request, REA renders a bundle in a private temporary folder: a computer app on the `ubuntu-24.04` base with the `reference` driver that installs the app, opens it on demand with `gtk-launch ` from the computer user's home, so its open and save dialogs start there rather than in the -framework's folder for the app, and keeps no seed and no data hooks. The app is not open when the computer starts: agentenv lists it -in the desktop's Applications menu under Other by the environment's name, such -as `mousepad`, and that entry launches it through the app's server. +framework's folder for the app, and keeps no seed and no data hooks. The app is not open when the computer starts: +`start_build_capture` opens it under REA's capture. When no capture is under +way, open it from the desktop's Applications menu, where agentenv lists it +under Other by the environment's name, such as `mousepad`; that entry launches +it through the app's server. When the app is Electron, set `electron` to true. The install rewrites that desktop entry's command with the same policy an export uses: `--no-sandbox` @@ -96,6 +98,10 @@ in the folder and its files stay as they were shared. REA runs the deploy with `run_bundle(keep=True, emulate=False)` on the local sandbox provider, so only the host's own architecture is built. The first start of a bundle builds its image and can take minutes; later starts reuse it. +agentenv makes a start wait while another agentenv run on this host writes its +bundle's ids, so starts at the same time wait for each other, several minutes +when a few run together; the progress line then says it is waiting for another +agent-env run to finish writing this bundle's ids. While it waits, REA sends an MCP progress notification every five seconds with agentenv's latest line, so a client that resets its request timeout on progress keeps waiting. REA then waits until the computer's readiness checks diff --git a/docs/desktop-build-witness.md b/docs/desktop-build-witness.md index 0d064b7429..7f7f159714 100644 --- a/docs/desktop-build-witness.md +++ b/docs/desktop-build-witness.md @@ -10,7 +10,7 @@ grades the witness with the same checks the exported verifier runs. | Tool | What it does | | -------------------------- | ------------------------------------------------------------------------------------------- | | `start_build_capture` | Downloads the package lists as the computer's user and launches the app under REA's capture | -| `mark_build_checkpoint` | Takes a named checkpoint and returns the capture's acknowledgement with its counts | +| `mark_build_checkpoint` | Takes a named checkpoint and returns the capture's counts and the files that changed | | `stop_build_capture` | Stops the capture and returns its checkpoints and the task draft between `seed` and `end` | | `export_build_environment` | Exports the bundle with the agent's task inside the computer and copies it to a folder here | | `check_build_task` | Grades a task against the app's live state; with a `stage`, files the check in the witness | @@ -41,6 +41,34 @@ See [desktop builds](https://github.com/morluto/rea/blob/main/.agents/skills/rev from a `.deb` file, the build layer keeps that file at `/opt/rea/package.deb` and the capture records it with `--package-file`. Do not open the app from the menu first: the capture refuses an app whose window is already open. + When the capture ends before the window opens, REA reads its output. An app + that exits at launch, such as one missing a runtime, is reported with its + program, exit code, run time and the last lines of its stderr, at most 8 + lines and 800 characters of the last 8 KiB the capture keeps, and + `details.diagnostics` keeps the capture's record of the session. A capture + that failed, such as one refusing to start, gives its own message and + remediation, with the whole failure in `details.diagnostics.command_failure` + and, when its cleanup is incomplete, the run tokens and paths it left in + `details.resources`. A checkpoint or stop that finds the capture ended + reads its output the same way. + + `state_paths` names files or folders in the computer's home that hold the + app's state, relative to the home, after `~/` or absolute under it, at + most 16; each becomes a `--state-path` of the capture, so it is retained + from the start and its files count as the app's even before they exist. + REA counts a file as a document only when an app window's title shows its + name or `~/` path, so name the file the app will save when its title shows + something else, as MuseScore's shows a score's title: a file outside + hidden folders, other than a lock such as LibreOffice's `.~lock.#`, + becomes a document, which the bundle owns, seeds and opens + at launch, and one under a hidden folder such as `~/.config/MuseScore` the + app's own state, also when that folder is a link elsewhere in the home. A + path with `..`, or naming the home itself, is refused before the computer + is asked; the computer refuses an absolute path outside its home, one in + `~/.rea-build`, and one a link leads out of the home or into + `~/.rea-build`, resolving a path that does not exist yet through its + nearest existing folder, and starts nothing. + 3. Make the seed through the app with `act_on_build_computer`, saving it where the user keeps such files, such as `~/Documents`, then `mark_build_checkpoint` `seed`. @@ -48,22 +76,56 @@ See [desktop builds](https://github.com/morluto/rea/blob/main/.agents/skills/rev `mark_build_checkpoint` `end`. 5. `stop_build_capture`. The app is closed and the capture finishes. The result lists the checkpoints with their counts, the package and the - sources the capture found, its limitations and the draft: each file that - changed between `seed` and `end` with its net change, and suggested - checks with a placeholder prompt. + sources the capture found, its limitations and the draft: each file under + the paths the export owns that changed between `seed` and `end`, with its + net change and, for XML files, archives, JSON and key files, its `values`, + the values the change added, changed and removed with what each was and + became, at the pointer a check names it by; suggested checks with a + placeholder prompt; and `guards`, optional checks that the rest kept its + seed values. The draft's + [task section](desktop-application-tasks.md#describe-the-change) describes + each. `notes`, each with a `code` and a `message`, say why a draft has no + files: `no_change` when no file under the home changed between them and + nor did the app's dconf keys; `unowned_changes` when files changed but + none under the owned paths, listing up to five in `files` with their + roles, desktop files last, and counting the rest in `more`; + `uncheckable_changes` when the app's keys in dconf were read and changed, + listing the database in `files`, since no check reads dconf (a version + the capture kept no copy of, as it keeps none of the database at the + start, counts as holding none of the app's keys); + `unreadable_changes` when a dconf database changed but REA could not read + it, such as without `dconf-cli`, with the read `error`, so whose keys + changed is unknown; and `no_document` when no file has the document role. + `unowned_changes` and `no_document` point at `state_paths`. They also say, + for a drafted `file`, why its suggestions are thin: `by_position`, + `view_state_only`, `long_text`, `cut_view` or `unreadable_view`. 6. Write the task: its name, the prompt in the user's own words, and checks - that hold after the change and not on the seed. + that hold after the change and not on the seed, from the suggestions, + `values` and any `guards` worth keeping. 7. `export_build_environment` with the task and a new or empty `directory`. The export runs inside the computer with `--seed-checkpoint seed --end-checkpoint end`, so it seeds what was saved at `seed`, scrubs the computer's host name and user from it, and refuses a task whose checks - all pass on the seed. A task larger than 96 KiB as REA passes it to the - computer is refused before anything runs. It also passes `--synthetic-seed`, - since the agent made the seed for the task, so the README does not call it - the user's real data. When the seed holds exactly one document and the - app's desktop entry takes files, the bundle's app opens that document at - launch. A file moved to the desktop Trash is not seeded, and the export's - limitations name what the Trash held at `seed`. For a + all pass on the seed. It also passes `--synthetic-seed`, since the agent + made the seed for the task, so the README does not call it the user's real + data. When the seed holds exactly one document and the app's desktop entry + takes files, the bundle's app opens that document at launch. A file moved + to the desktop Trash is not seeded, and the export's limitations name what + the Trash held at `seed`. The limitations and the README also name the + folders where files changed during the capture that the bundle neither + owns nor seeds, such as settings that keep a first-run wizard away, up to + eight collapsed to their top folder, such as `~/.config/MuseScore`, and a + count of the rest; capture again with them in `state_paths` to own them. + Lock files the app holds while it runs, such as LibreOffice's profile + `.lock` and its document lock `.~lock.#`, are left out of the seed, + since a copied one can keep the app from starting, and the limitations + and the README list them after "Left out of the seed"; a project's + `Cargo.lock` or `yarn.lock`, a document or a path in `state_paths` is + seeded. They name seeded + WebKit or Chromium storage and caches too, which the app keeps for its + own running. When the application held files open in a folder whose + starting content the capture did not retain, a limitation names the + folder and the seed files in it. For a `.deb` app that no apt source the capture found offers, the export takes the request's URL with `--package-url`; one apt offers installs from apt. REA lists the bundle's files with their sizes and SHA-256 in the computer, into a file beside the bundle that it reads @@ -78,7 +140,15 @@ A checkpoint name is lowercase letters, digits, dot, dash and underscore; the capture's own `baseline`, `stop`, `time-limit` and `exit` are refused, and so is a name already taken. The acknowledgement comes from the capture after it has walked the home and retained what changed: `entries` walked, `changes` -since the checkpoint before, `retained_files` and `retained_bytes`. +since the checkpoint before, `retained_files` and `retained_bytes`, and up to +ten `changed_files`, home-relative, with `more_changed_files` counting the +rest. Folders are left out; files under `state_paths` come first, then +those the app held open, then those in its state directories, then the rest, +and caches and WebKit or Chromium storage last, each by path, so a save's +document leads the list. When the computer's REA is older than these lists, +as when `REA_BUILD_COMPUTER_REA_ARCHIVE` still names an earlier build, the +checkpoint is still taken and both are `null`: which files changed is +unknown. Pack the current REA into the archive to get them back. Each step can be asked again after its answer was lost: @@ -102,7 +172,9 @@ computer's control token. REA sends each command through agentenv's `computer-exec/v1` route with that token, which stays in REA, and reads larger outputs, such as the draft, the export's Evidence and the bundle's file list, from files under `~/.rea-build` through `data/get`, since a command's -output is capped at 1 MiB. +output is capped at 1 MiB. In a command's Evidence or failure read this way, +as in its own output, REA replaces the token in every string, after checking +the Evidence's digest; the bundle's files arrive as they are. Calls on one computer run one at a time, in the order REA receives them: an action, a capture step, a check or a close waits until the call before @@ -116,7 +188,11 @@ cancels the call in progress on it and refuses the calls still waiting. Start a fresh computer from the exported folder with `start_build_computer` and `{"kind": "bundle", "path": ...}`; agentenv boots the app with the seed, -on the seed's document when the export opens one. +on the seed's document when the export opens one. When the computer never +gets ready and the bundle's seed holds lock files, the start's error names +them: delete them from the bundle, or export again, which leaves them out. +A bundle records no state directories, so the error names only files by a +profile lock's name and LibreOffice's document locks. Then, with the task exactly as exported: | `stage` | Before it | Its checks must | @@ -129,7 +205,18 @@ Then, with the task exactly as exported: `check_build_task` reads the app's `data/get`, takes the views it returns beside the files and evaluates each check with REA's evaluator, the one the export validates tasks with and the exported verifier mirrors, one result per -check. Without `stage` it only grades. The `seed` stage begins a recording +check. Like the verifier, it grades only once the app's files settle, since an +app such as Gnote saves a few seconds after the last key. It reads `data/get` +every 0.5 s and grades once every file's bytes have held for 3 s, counted +from the read that returned them. 15 s after the first read began, it grades +the last completed read: no read starts then, and one still running is cut. +It sends progress notifications while it waits. The result's `settled` says +whether the files settled, and `read_cut` whether the bound cut a read; each +stage in the record keeps both. Every check therefore costs at least 3 s, +and a save that lands more than 3 s after the first read returns is still +missed; see +[grading once the app has saved](desktop-application-export.md#grading-once-the-app-has-saved). +Without `stage` it only grades. The `seed` stage begins a recording of the display under REA's own request id, one per computer; a seed check that fails after it, such as on a failed read of the app's state or a lost reply to the recording's start, can be asked again, and asks for the @@ -171,13 +258,14 @@ A stage whose checks do not show what it must ends the witness, and so does - `recording/NNNN.mp4`, the recording's segments; - `witness.json`, REA Evidence (`rea.desktop-build-witness`) holding the verdict, the stage that failed if one did, the task's name and SHA-256, - each stage's results with how many logged actions came before it, the + each stage's results with how many logged actions came before it and + whether the app's files had settled (`settled`), the computer's [action log](desktop-build-computer.md#act) as it keeps it, each action with its arguments, timing and outcome, with `actions_dropped` counting the earliest actions it no longer keeps, and the recording's segments with their sizes and SHA-256. The record's limitations say when the log dropped actions or cut arguments, which each entry names in - `truncated_arguments`. + `truncated_arguments`, and which stages graded files that had not settled. The stages' rules read the log's count of actions and the last applied action of each kind, which it keeps for the whole session, so they hold diff --git a/docs/specs/desktop-app-environments/build-witness/spec.md b/docs/specs/desktop-app-environments/build-witness/spec.md index 4cded391d1..8acb70e8a5 100644 --- a/docs/specs/desktop-app-environments/build-witness/spec.md +++ b/docs/specs/desktop-app-environments/build-witness/spec.md @@ -28,23 +28,49 @@ and the recording and writes the record as Evidence. out of every walk. A start returns once the capture reports the window open; a refused or early-ended start stops the capture and reports its output. +- `start_build_capture` takes `state_paths`, at most 16 files or folders in + the home, relative to it, after `~/` or absolute, without `..` and not the + home itself; the controller resolves each under the computer's home, both + as named and with its links followed (a path not there yet through its + nearest existing folder), refuses one outside it or in `~/.rea-build` + either way before launching anything, and passes each as named to the + capture as `--state-path`. A selected file is a document, or the app's own + state under a hidden folder of the home as named, before links, or when it + is a lock the app holds, such as LibreOffice's `.~lock.#`. - Acknowledgement: `capture-desktop-application` reports, beside each progress message, an `event`: `window_open`; `checkpoint` with its name, trigger, - time, entries walked, changes and retained files and bytes, once it is - taken, the baseline, `stop` and `exit` included; and `checkpoint_refused` - with the reason. `mark_build_checkpoint` writes `checkpoint ` and - waits for the event naming it, returning its counts. Names follow a narrow - pattern, without the capture's own names; a name already taken is refused. + time, entries walked, changes and retained files and bytes, and up to ten + changed files, home-relative, those under a selected path first, then + those the app held open, caches and browser-engine storage last, with a count + of the rest, once it is taken, the baseline, `stop` and `exit` included; + and `checkpoint_refused` with the reason. `mark_build_checkpoint` writes + `checkpoint ` and waits for the event naming it, returning its counts + and changed files, both `null` when the REA in the computer predates them + and its event leaves them out. Names follow a narrow pattern, without the capture's own + names; a name already taken is refused. - `stop_build_capture` sends `stop`, waits for the exit status, reads the capture's Evidence and, when `seed` and `end` were acknowledged, runs `describe-desktop-application-task --seed-checkpoint seed --end-checkpoint end` and returns the changed files, their net change and suggested checks. + A draft with no files carries `notes`, coded `no_change`, + `unowned_changes` (up to five changed home paths with their roles, and a + count of the rest), `uncheckable_changes` (dconf databases whose app keys + were read and changed, which no check reads, a version kept without its + bytes counting as holding none), `unreadable_changes` (dconf + databases that changed but could not be read, with the read `error`) and + `no_document`; a drafted file whose suggestions are thin carries one + naming its `file`, coded `by_position`, `view_state_only`, `long_text`, + `cut_view` or `unreadable_view`. - `export_build_environment` writes the agent's task under `~/.rea-build`, runs `export-desktop-application` with it and the same checkpoints into a new folder there, reports a refused task with the export's own reason, lists the bundle with sizes and SHA-256, fetches it through the driver's `data/get` in answers of at most 8 MiB, checks each file and writes the - caller's new or empty folder through the safe output tree. + caller's new or empty folder through the safe output tree. The export + leaves lock files the app holds while it runs out of the seed, but never a + document or selected path, listing them in its limitations and README; a + start from a bundle whose computer never gets ready names the files its + seed holds by a profile lock's name or as a document lock. - `check_build_task` reads the one app's `data/get` through the card's child URL, takes the state views beside its files and evaluates the task's checks with `evaluateTaskChecks`, one result per check. With `stage`, the diff --git a/docs/specs/desktop-app-environments/build-witness/verification.md b/docs/specs/desktop-app-environments/build-witness/verification.md index f0d53e8048..2db8bbd157 100644 --- a/docs/specs/desktop-app-environments/build-witness/verification.md +++ b/docs/specs/desktop-app-environments/build-witness/verification.md @@ -280,6 +280,123 @@ taken 18, 23 and 21 turns for $0.70, $1.55 and $0.51 ($0.92). Cache reads fell from 585-714K tokens to 181-225K; the cost is mostly cache writes, which varied from 17K to 189K tokens whatever the turns. +## Naming the app's state + +Run on macOS with source tests and gates; no build computer was started. +Each test below failed on the head before this change, with only the tests +and their fixtures added, and passes now. + +- `buildCaptureStatePathsMcp.test.ts`: `start_build_capture` with + `state_paths` passes the note's path to the capture as `--state-path`, the + draft covers it as a document with no notes, and an export with a check on + it is accepted; without it the draft is empty with `unowned_changes` + (the note, `unrelated`) and `no_document`, and the export's refusal names + `state_paths`. A path through `..`, the home itself and 17 paths are + refused by the schema, `/etc/hosts` by the computer, and none starts a + capture. Before, `state_paths` was an unknown argument. +- `DesktopTaskDraftNotes.test.ts`: a draft with files has no notes; one with + no change between seed and end has only `no_change`; one whose changes lie + outside the owned paths lists five with their roles, desktop files last, + and counts two more, beside `no_document`. +- `stateRoles.test.ts`: a selected file under a hidden folder of the home is + application state, elsewhere a document. +- `DesktopApplicationExport.test.ts`, `appExportUnowned.test.ts`: the export's + limitations and README name folders the capture saw change outside the + owned paths, collapsed to their top folder and without desktop caches, up + to eight and a count, and the seeded lock file, WebKit storage and cache + folders. + +## Lock files, changed files and held-open folders + +Run on macOS with source tests and gates; no build computer was started. A +hold-out LibreOffice Calc bundle seeded `~/.config/libreoffice/4/.lock` and +showed no window at boot; its export also said nothing in `~/Documents` was +seeded while `seed_files` held `Documents/q1-sales.ods`. Each test below +failed on the head before this change, with only the tests and their +fixtures added, and passes now. + +- `DesktopApplicationExport.test.ts`: a capture whose seed holds a profile + `.lock` and a document lock `.~lock.today.txt#` exports without either in + `seed_files` or the bundle, and lists both after "Left out of the seed" in + one limitation and the README; with `~/Documents` held open, its + limitation names `~/Documents/today.txt` as seeded from there. +- `appExportUnowned.test.ts`: profile, browser and document locks are lock + files, and `*.lock` or `*.lck` only in a hidden folder, so `Cargo.lock` + and `yarn.lock` in a project are not; the capture's limitation names the + held-open folders from `~/`, and the export's names the seed files there + or says there are none. +- `desktopCapture.test.ts`: a checkpoint's acknowledgement names its changed + files home-relative, the held-open document first, then state, then the + rest, without folders, and names ten of thirteen with three more. +- `buildCaptureMcp.test.ts`: `mark_build_checkpoint` returns the changed + files the capture acknowledged. +- `buildComputerMcp.test.ts`: a bundle's start that never got ready names + the lock files its seed holds; one without locks, or failing otherwise, + names none. + +The Linux-only capture test's acknowledgement now expects the changed state +file; it is skipped on macOS and was not run. + +## Linked state paths, dconf changes, project lock files and older checkpoint replies + +Run on macOS with source tests and gates; no build computer was started. +Each test below failed on the head before this change, with only the tests +and their fixtures added, and passes now. + +- `buildCaptureStatePathsMcp.test.ts`: the computer refuses state paths that + a link leads out of the home, existing or not yet (through the nearest + existing folder, or a dangling link), or into `~/.rea-build`, and starts + nothing; a link that stays in the home is followed and the path passed to + the capture as named. Before, the first such path started a capture. +- `DesktopCaptureScope.test.ts`: the capture scope records each selected + path as named beside the physical path it walks. +- `DesktopApplicationExport.test.ts`: with `~/.config/MuseScore` selected as + a link to `~/settings/MuseScore` beside a selected note, the bundle's + launch opens the note. With only the capture's record of selected paths + added, the settings file was still a document and the launch opened none. +- `DesktopTaskDraftNotes.test.ts`: a draft whose only change between `seed` + and `end` is to the app's keys in dconf carries `uncheckable_changes` + naming `~/.config/dconf/user`, not `no_change`; one where only another + group's keys changed still has `no_change`. +- `appExportUnowned.test.ts`: `*.lock` and `*.lck` are locks only directly + in a state directory, so `Projects/.repo/Cargo.lock` is not; and 20,000 + separate unowned folders collapse in tens of milliseconds, where comparing + every pair took 12.9 seconds. +- `DesktopApplicationExport.test.ts`: `Projects/.repo/Cargo.lock` and + `.dotfiles/web/yarn.lock`, which the capture recognized as documents, and + a lock-named file the caller selected are seeded, a `*.lock` directly in a + state directory is left out, and the profile and document locks still are. +- `buildCaptureMcp.test.ts`: checkpoints acknowledged without + `changed_files` and `more_changed_files`, as the previous REA in a build + computer does, are taken with both `null`, also when a lost reply is read + again from the capture's log. Before, the host refused the acknowledgement + although the checkpoint was taken. +- `desktopCapture.test.ts`: a checkpoint's changed files name a document + under a selected path first, though the app did not hold it open, and + caches and browser-engine storage last, even one the app held open or in a + state directory, so ten KDE service-cache files no longer push the saved + document past the cut, as they did in a held-out Tellico build. +- `DesktopApplicationExport.test.ts`: with `~/.config/MuseScore` selected as + a link to `~/Documents` and the note inside it selected itself, the + launch opens the note while the settings beside it stay the app's; before, + the link's hidden name made the note application state and the launch + opened none. +- `DesktopTaskDraftNotes.test.ts`: a dconf database that changed but could + not be read gives `unreadable_changes` with the reader's error, not + `uncheckable_changes`. +- `buildCaptureMcp.test.ts`: `stop_build_capture` on a capture whose only + change between `seed` and `end` is a dconf database the draft cannot read + (no `dconf` on its `PATH`) returns the `unreadable_changes` note with its + `error`; before, the build tool dropped the field. +- `LinuxApplicationInventory.test.ts`: the inventory lists KDE's + `~/.config/rc` before it exists, so a Tellico capture owns + `~/.config/tellicorc`, whose absence brought back Tip of the Day at boot; + `DesktopApplicationExport.test.ts` shows such a file, written during the + capture, owned and seeded (that test passed before, given the inventory). + +The capture's recording of selected paths runs only on Linux; its scope is +tested on macOS, and the capture itself was not run. + ## Limits of this verification - The witness acts were scripted by this agent from the task's prompt; no diff --git a/docs/specs/desktop-app-environments/capture-for-building/spec.md b/docs/specs/desktop-app-environments/capture-for-building/spec.md index acf5dcaa71..6ea8b9a8cc 100644 --- a/docs/specs/desktop-app-environments/capture-for-building/spec.md +++ b/docs/specs/desktop-app-environments/capture-for-building/spec.md @@ -55,7 +55,9 @@ and the bundle holds neither the capture host's name nor its user's. path and content, escapes decoded, for the host name and `user@host` anywhere and the captured home as a whole path, with the replacements set aside. `scrubbed_files` lists the rewritten seeds and `unscrubbed_files` - every file found, by bundle path; the README lists both. + every file found, by bundle path; the README lists both. gzip seeds + compressed again only to clear their headers, their documents unchanged, + are in `recompressed_files` instead, which the README counts. - Tasks: checks whose values name `user@host` or the host name by that rule are refused, like those that name the captured home, but not for their field names, kind or id; and the checks that must fail on the seed are diff --git a/docs/specs/desktop-app-environments/capture/spec.md b/docs/specs/desktop-app-environments/capture/spec.md index 637516e6c1..b8dc38fd42 100644 --- a/docs/specs/desktop-app-environments/capture/spec.md +++ b/docs/specs/desktop-app-environments/capture/spec.md @@ -68,7 +68,8 @@ folder to build the environment; this step builds nothing. lists candidate state directories under the XDG folders and dot-folders named after the package, executable, desktop entry's full id and last part, and name, and every folder present there whose name is one of those in - another letter case, such as GIMP's `~/.config/GIMP`; a `flatpak run` + another letter case, such as GIMP's `~/.config/GIMP`, and KDE's + `rc` in the config folder, such as `~/.config/tellicorc`; a `flatpak run` launch uses the application id (its first operand after options and their values) instead of the launcher's name and adds `~/.var/app/`. An `--executable` runs from the caller's @@ -78,7 +79,9 @@ folder to build the environment; this step builds nothing. `--state-path` values and absolute arguments other than `/` or a directory containing the home directory, which the limitations name instead, plus the `-wal` and `-journal` paths beside each of those selected paths that is not - a directory, so a database selected on its own is captured with its logs. The + a directory, so a database selected on its own is captured with its logs. + Each selected path is also recorded as named, beside its physical + spelling, so a role can follow from the name the caller gave it. The baseline walk runs before launch and retains every file under the content roots; each later checkpoint reports created, modified, deleted and retyped paths against the previous walk and retains every changed file. A creation diff --git a/docs/specs/desktop-app-environments/state-view/spec.md b/docs/specs/desktop-app-environments/state-view/spec.md index d63791fbe7..0e56b7f86e 100644 --- a/docs/specs/desktop-app-environments/state-view/spec.md +++ b/docs/specs/desktop-app-environments/state-view/spec.md @@ -41,7 +41,11 @@ generation reads the kinds of change. ([XML text checks](../xml-text-checks/spec.md)); key files up to 4 MiB whose every line parses; zip archives by their local file header, as their entries with each entry's own format and - view ([document views](../document-views/spec.md)); text; binary. + view ([document views](../document-views/spec.md)); text; binary. A gzip + file, told by its magic bytes, is read as the document inside, its members + joined, within a zip entry's 16 MiB and 100-times limits, and its version + records `compression: "gzip"` beside that document's format; past the + limits, or damaged, it has a `view_error`. - Changes between versions are created, deleted, or lines, values, keys or rows with added, removed and changed lists (100 each, with counts); content for binary files and unreadable when a view is missing. Values are JSON @@ -57,5 +61,6 @@ generation reads the kinds of change. - No export, seed selection or task generation. - No parsing of other binary formats (GVDB other than dconf, LevelDB, Realm, - protobuf); they are listed as binary. + protobuf), or of compression other than gzip (xz, Zstandard, bzip2); they + are listed as binary. - No redaction: views contain the user's real data, as the capture does. diff --git a/docs/specs/desktop-app-environments/tasks/spec.md b/docs/specs/desktop-app-environments/tasks/spec.md index b8715b73ea..7cbb35c63a 100644 --- a/docs/specs/desktop-app-environments/tasks/spec.md +++ b/docs/specs/desktop-app-environments/tasks/spec.md @@ -21,7 +21,14 @@ never runs code the agent wrote. both ends whose count changed); and a draft task with a placeholder prompt and checks suggested from the net change of documents, or of application state when no document changed. Views are shown only for those files, cut at - 200 items and 2,000 characters. + 200 items and 2,000 characters. A draft with no files says why in `notes`: + `no_change`, `unowned_changes` with up to five changed home paths and their + roles, `uncheckable_changes` with the dconf databases whose application + keys were read and changed, `unreadable_changes` with the dconf databases + that changed but could not be read and the read `error`, and + `no_document`; a drafted file whose suggestions are thin gets one naming + its `file`: `by_position`, `view_state_only`, `long_text`, `cut_view` or + `unreadable_view`. - The task file is JSON with a portable `name` other than `smoke`, a `prompt` and 1 to 32 `checks`, parsed by a zod schema in `src/domain/desktop/`. Check kinds: `file_present`, `file_absent`, `lines_present`, `lines_absent`, diff --git a/scripts/pack-build-computer-runtime.mjs b/scripts/pack-build-computer-runtime.mjs index 3acafbb0c8..f3b10f3876 100644 --- a/scripts/pack-build-computer-runtime.mjs +++ b/scripts/pack-build-computer-runtime.mjs @@ -9,8 +9,9 @@ import { tmpdir } from "node:os"; import { dirname, join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; +import { npmInvocation } from "./lib/npm-invocation.mjs"; + const root = join(dirname(fileURLToPath(import.meta.url)), ".."); -const npm = process.platform === "win32" ? "npm.cmd" : "npm"; const run = (command, args, cwd) => { const result = spawnSync(command, args, { cwd, stdio: "inherit" }); @@ -31,7 +32,8 @@ if (destination === undefined || !destination.endsWith(".tar.xz")) { const archive = resolve(destination); const work = await mkdtemp(join(tmpdir(), "rea-build-runtime-")); try { - run(npm, ["pack", "--pack-destination", work], root); + const pack = npmInvocation(["pack", "--pack-destination", work], process.env); + run(pack.command, pack.arguments, root); const [packed] = (await readdir(work)).filter((name) => name.endsWith(".tgz"), ); @@ -43,7 +45,11 @@ try { join(root, "package-lock.json"), join(runtime, "package-lock.json"), ); - run(npm, ["ci", "--omit=dev", "--omit=optional"], runtime); + const install = npmInvocation( + ["ci", "--omit=dev", "--omit=optional"], + process.env, + ); + run(install.command, install.arguments, runtime); run("tar", ["-cJf", archive, "-C", work, "rea-prod"], work); const { size } = await stat(archive); process.stdout.write( diff --git a/src/application/desktop/DesktopApplicationCapture.test.ts b/src/application/desktop/DesktopApplicationCapture.test.ts index 4dc0d54ee9..6ac6d439bf 100644 --- a/src/application/desktop/DesktopApplicationCapture.test.ts +++ b/src/application/desktop/DesktopApplicationCapture.test.ts @@ -475,7 +475,7 @@ describe.skipIf(process.platform !== "linux")( evidence.normalized_result, ); const seed = capture.state.checkpoints[1]; - // The acknowledgement comes after the checkpoint is taken and carries its counts. + // The acknowledgement comes after the checkpoint is taken and carries its counts and the file the app wrote. expect(parsed[2]).toEqual({ kind: "checkpoint", name: "seed", @@ -485,6 +485,8 @@ describe.skipIf(process.platform !== "linux")( changes: seed?.changes.length, retained_files: 1, retained_bytes: 8, + changed_files: [".config/fakeapp/state.ini"], + more_changed_files: 0, }); expect(parsed[3]).toEqual({ kind: "checkpoint_refused", diff --git a/src/application/desktop/DesktopApplicationCapture.ts b/src/application/desktop/DesktopApplicationCapture.ts index b825dd358d..c825121bf6 100644 --- a/src/application/desktop/DesktopApplicationCapture.ts +++ b/src/application/desktop/DesktopApplicationCapture.ts @@ -6,6 +6,7 @@ import { projectAnalysisError, type AnalysisErrorProjection, } from "../../domain/analysisErrorProjection.js"; +import { unretainedOpenedLimitation } from "../../domain/desktop/appExportUnowned.js"; import { checkpointTaken, type DesktopApplicationCapture, @@ -110,6 +111,7 @@ const limitationsFor = ( recorder: CheckpointRecorder, facts: { readonly unretained: readonly string[]; + readonly home: string; readonly broadArguments: readonly string[]; readonly unreadableWindowSamples: number; }, @@ -156,10 +158,8 @@ const limitationsFor = ( session.processes.some((process) => !process.started_by_capture), "An application window belonged to a process the capture did not start; that process was left running.", ); - add( - facts.unretained.length > 0, - "The application held files open in directories whose starting content was not retained; pass them with --state-path to retain it.", - ); + const unretained = unretainedOpenedLimitation(facts.unretained, facts.home); + if (unretained !== undefined) limitations.push(unretained); return limitations; }; @@ -264,7 +264,11 @@ const recordCapture = async ( const acknowledge = async () => { const taken = recorder.checkpoints.at(-1); if (taken === undefined) return; - const event = checkpointTaken(taken); + const event = checkpointTaken( + taken, + home, + context.scope.selectedPaths.map(({ physical }) => physical), + ); await host.report( `Recorded checkpoint ${taken.name}: ${String(event.changes)} change(s), ${String(event.retained_files)} file(s) retained.`, event, @@ -314,6 +318,7 @@ const recordCapture = async ( content_roots: [...scope.contentRoots], baseline_non_files: recorder.baselineNonFiles, state_directories: [...scope.stateDirectories], + selected_paths: [...context.scope.selectedPaths], excluded: [...scope.excluded], checkpoints: recorder.checkpoints, content_files: store.files, @@ -325,6 +330,7 @@ const recordCapture = async ( ...admitted.limitations, ...limitationsFor(session, recorder, { unretained, + home, broadArguments: context.scope.broadArguments, unreadableWindowSamples: outcome.unreadableWindowSamples, }), diff --git a/src/application/desktop/DesktopApplicationExport.test.ts b/src/application/desktop/DesktopApplicationExport.test.ts index ebac7538ba..b0f2e65438 100644 --- a/src/application/desktop/DesktopApplicationExport.test.ts +++ b/src/application/desktop/DesktopApplicationExport.test.ts @@ -3,6 +3,7 @@ import { access, readFile } from "node:fs/promises"; import { join } from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; +import { z } from "zod"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { SafeOutputTree } from "../../artifacts/SafeOutputTree.js"; @@ -22,10 +23,12 @@ import { import { exportDesktopApplication, seedLimitation, + trashedFilesLimitation, } from "./DesktopApplicationExport.js"; import { EDITED_NOTE, HOME, + NOTE, capture, change, checkpoint, @@ -248,6 +251,165 @@ describe("exportDesktopApplication", () => { }); }); +it("seeds the app's state at its own path and retains a selected alias", async () => { + const local = capture(); + const physical = `${HOME}/.config/notes`; + const { result, directory } = await exportCapture({ + ...local, + state: { + ...local.state, + content_roots: [physical], + state_directories: [physical, `${HOME}/.local/share/notes`], + selected_paths: [ + { path: `${HOME}/settings`, physical, kind: "directory" }, + { + path: `${HOME}/settings/profiles/work`, + physical: `${physical}/profiles/work`, + kind: "directory", + }, + ], + checkpoints: [ + checkpoint( + "baseline", + "baseline", + [], + [ + retained(`${physical}/settings.ini`, "[ui]\nwrap=false\n"), + retained(`${physical}/profiles/work/profile.ini`, "milk\n"), + ], + ), + ], + }, + }); + if (result instanceof Error) throw result; + expect(result.normalized_result).toMatchObject({ + paths: [".config/notes"], + seed_files: [ + ".config/notes/profiles/work/profile.ini", + ".config/notes/settings.ini", + ], + }); + expect( + await readFile( + join(directory, "envs/notes/seed/.config/notes/settings.ini"), + "utf8", + ), + ).toBe("[ui]\nwrap=false\n"); + expect( + await readFile( + join( + directory, + "envs/notes/seed/.config/notes/profiles/work/profile.ini", + ), + "utf8", + ), + ).toBe("milk\n"); + const module = await readFile( + join(directory, "envs/notes/notes_app.py"), + "utf8", + ); + expect(module).toContain('("settings", ".config/notes")'); +}); + +it.each(["settings/notes", "chosen/notes"])( + "restores the selected %s spelling alongside the app's walked state path", + async (selected) => { + const local = capture(); + const physical = `${HOME}/settings/notes`; + const { result, directory } = await exportCapture({ + ...local, + state: { + ...local.state, + roots: [HOME], + content_roots: [physical], + state_directories: [physical, `${HOME}/.local/share/notes`], + selected_paths: [ + { path: `${HOME}/${selected}`, physical, kind: "directory" }, + ], + checkpoints: [ + checkpoint( + "baseline", + "baseline", + [], + [retained(`${physical}/settings.ini`, "[ui]\nwrap=false\n")], + ), + ], + }, + }); + if (result instanceof Error) throw result; + expect(result.normalized_result).toMatchObject({ + seed_files: expect.arrayContaining([".config/notes/settings.ini"]), + }); + expect( + await readFile( + join(directory, "envs/notes/seed/.config/notes/settings.ini"), + "utf8", + ), + ).toBe("[ui]\nwrap=false\n"); + const module = await readFile( + join(directory, "envs/notes/notes_app.py"), + "utf8", + ); + expect(module).toContain(`("${selected}", ".config/notes")`); + }, +); + +it.each([true, false])( + "omits only a child alias supplied by its parent (%s)", + async (redundant) => { + const local = capture(); + const parent = `${HOME}/.config/notes`; + const child = `${parent}/state/notes`; + const { result, directory } = await exportCapture({ + ...local, + application: { + ...local.application, + state_directories: [ + { path: parent, exists_at_start: true }, + { + path: redundant ? child : `${HOME}/.local/share/separate`, + exists_at_start: true, + }, + ], + }, + state: { + ...local.state, + content_roots: [parent], + state_directories: [parent, child], + selected_paths: [ + { path: `${HOME}/settings`, physical: parent, kind: "directory" }, + { + path: `${HOME}/settings/state/notes`, + physical: child, + kind: "directory", + }, + ], + checkpoints: [ + checkpoint( + "baseline", + "baseline", + [], + [retained(`${child}/settings.ini`, "[ui]\nwrap=false\n")], + ), + ], + }, + }); + if (result instanceof Error) throw result; + const module = await readFile( + join(directory, "envs/notes/notes_app.py"), + "utf8", + ); + expect(module).toContain('("settings", ".config/notes")'); + expect(module).not.toContain( + '("settings/state/notes", ".config/notes/state/notes")', + ); + if (!redundant) + expect(module).toContain( + '("settings/state/notes", ".local/share/separate")', + ); + }, +); + describe("exportDesktopApplication and the desktop Trash", () => { it("leaves a file the user moved to the Trash out of the seed and the owned paths, and says so", async () => { const local = capture(); @@ -306,12 +468,297 @@ describe("exportDesktopApplication and the desktop Trash", () => { "Documents/today.txt", ], }); - const warning = result.limitations.find((line) => - line.includes("Trash/files/today.txt"), + const trashPaths = [ + ".local/share/Trash/files/today.txt", + ".local/share/Trash/info/today.txt.trashinfo", + ]; + const notice = trashedFilesLimitation(trashPaths); + expect(result.limitations).toContain(notice); + for (const path of trashPaths) expect(notice).toContain(`~/${path}`); + }); +}); + +describe("exportDesktopApplication of state it does not own or keeps for itself", () => { + it("names the folders the app wrote that the bundle neither owns nor seeds, and the web storage and caches it seeds", async () => { + const local = capture(); + const [baseline, edited] = local.state.checkpoints; + if (baseline === undefined || edited === undefined) + throw new Error("the fixture has two checkpoints"); + const text = "x\n"; + // In the state directories from the start, so seeded. + const kept = [ + `${HOME}/.local/share/notes/storage/salt`, + `${HOME}/.local/share/notes/hsts-storage.sqlite`, + `${HOME}/.config/notes/GPUCache/data_0`, + ]; + // Written during the capture outside them; the font cache is the desktop's. + const written = [ + `${HOME}/.config/MuseScore/MuseScore3.ini`, + `${HOME}/.local/share/MuseScore/MuseScore3/session`, + `${HOME}/.cache/fontconfig/cache-1`, + ]; + const folder = await writeCaptureFolder( + { + ...local, + state: { + ...local.state, + checkpoints: [ + { + ...baseline, + retained: [ + ...baseline.retained, + ...kept.map((path) => retained(path, text)), + ], + }, + { + ...edited, + changes: [ + ...edited.changes, + ...written.map((path) => change(path, { after: text })), + ], + retained: [ + ...edited.retained, + ...written.map((path) => retained(path, text)), + ], + }, + ], + }, + }, + undefined, + [text], ); - expect(warning).toContain("~/.local/share/Trash/info/today.txt.trashinfo"); - expect(warning).toMatch(/Trash.*2 file.*not seeded/iu); - expect(warning).toMatch(/restore.*capture again/iu); + const directory = join( + await createTestTempDirectory("desktop-export-"), + "bundle", + ); + + const result = await exportDesktopApplication( + { capture: folder, directory, name: undefined, tasks: [] }, + readers, + ); + + if (result instanceof Error) throw result; + expect(result.normalized_result).toMatchObject({ + seed_files: expect.arrayContaining( + kept.map((path) => path.slice(HOME.length + 1)), + ), + }); + const unowned = result.limitations.find((limitation) => + limitation.includes("~/.config/MuseScore"), + ); + expect(unowned).toContain("~/.local/share/MuseScore"); + expect(unowned).toContain("state_paths"); + const runtime = result.limitations.find((limitation) => + limitation.includes("~/.local/share/notes/storage"), + ); + for (const path of [ + "~/.local/share/notes/hsts-storage.sqlite", + "~/.config/notes/GPUCache", + ]) + expect(runtime).toContain(path); + // Desktop caches are left out, and paths collapse to the folder that holds them. + const listed = result.limitations.join("\n"); + for (const path of ["fontconfig", "MuseScore3/session", "storage/salt"]) + expect(listed).not.toContain(path); + const readme = await readFile(join(directory, "README.md"), "utf8"); + for (const path of [ + "~/.config/MuseScore", + "~/.local/share/MuseScore", + "~/.local/share/notes/storage", + "~/.config/notes/GPUCache", + ]) + expect(readme).toContain(`\`${path}\``); + expect(readme).toContain("`state_paths`"); + }); +}); + +/** The fixture with a profile lock and a document lock written by `edited`, and its notes folder held open. */ +const lockedCapture = (): DesktopApplicationCapture => { + const local = capture(); + const [baseline, edited] = local.state.checkpoints; + if (baseline === undefined || edited === undefined) + throw new Error("the fixture has two checkpoints"); + const locks = [ + [PROFILE_LOCK, PROFILE_LOCK_TEXT], + [DOCUMENT_LOCK, DOCUMENT_LOCK_TEXT], + ] as const; + return { + ...local, + state: { + ...local.state, + checkpoints: [ + baseline, + { + ...edited, + changes: [ + ...edited.changes, + ...locks.map(([path, text]) => change(path, { after: text })), + ], + retained: [ + ...edited.retained, + ...locks.map(([path, text]) => retained(path, text)), + ], + }, + ], + unretained_opened_directories: [`${HOME}/Documents`], + }, + }; +}; +const PROFILE_LOCK = `${HOME}/.config/notes/.lock`; +const PROFILE_LOCK_TEXT = "[Lockdata]\nUser=ada\nHost=localhost\n"; +const DOCUMENT_LOCK = `${HOME}/Documents/.~lock.today.txt#`; +const DOCUMENT_LOCK_TEXT = + ",ada,localhost,04.11.2026 00:00,file:///home/ada/.config/notes;\n"; + +const exportLocked = async ( + recorded = lockedCapture(), + extraContent: readonly string[] = [], +) => { + const folder = await writeCaptureFolder(recorded, undefined, [ + PROFILE_LOCK_TEXT, + DOCUMENT_LOCK_TEXT, + ...extraContent, + ]); + const directory = join( + await createTestTempDirectory("desktop-export-"), + "bundle", + ); + const result = await exportDesktopApplication( + { + capture: folder, + directory, + name: undefined, + tasks: [], + seedCheckpoint: "edited", + }, + readers, + ); + if (result instanceof Error) throw result; + return { result, directory }; +}; + +describe("exportDesktopApplication of lock files and folders the app held files open in", () => { + it("leaves the profile lock and the document lock out of the seed and lists them as left out", async () => { + const { result, directory } = await exportLocked(); + + const seedFiles = z + .object({ seed_files: z.array(z.string()) }) + .parse(result.normalized_result).seed_files; + expect(seedFiles).toContain("Documents/today.txt"); + expect(seedFiles).toContain(".config/notes/settings.ini"); + expect(seedFiles).not.toContain(".config/notes/.lock"); + expect(seedFiles).not.toContain("Documents/.~lock.today.txt#"); + expect( + await exists(join(directory, "envs/notes/seed/.config/notes/.lock")), + ).toBe(false); + expect( + await exists( + join(directory, "envs/notes/seed/Documents/.~lock.today.txt#"), + ), + ).toBe(false); + const leftOut = result.limitations.filter((limitation) => + limitation.startsWith("Left out of the seed:"), + ); + expect(leftOut).toHaveLength(1); + expect(leftOut[0]).toContain( + "~/.config/notes/.lock, ~/Documents/.~lock.today.txt#.", + ); + // Nothing else calls them seeded. + expect( + result.limitations.filter((limitation) => limitation.includes(".lock")), + ).toEqual(leftOut); + const readme = await readFile(join(directory, "README.md"), "utf8"); + expect(readme).toContain("- Left out of the seed:"); + expect(readme).toContain( + "`~/.config/notes/.lock`, `~/Documents/.~lock.today.txt#`", + ); + }); + + it("seeds project lock files the capture recognized as documents and a lock-named file the caller selected, and leaves out a *.lock file directly in a state directory", async () => { + const locked = lockedCapture(); + const [baseline, edited] = locked.state.checkpoints; + if (baseline === undefined || edited === undefined) + throw new Error("the fixture has two checkpoints"); + const text = "x\n"; + // Under folders whose names start with a dot, which made them look like an app's locks. + const projects = [ + `${HOME}/Projects/.repo/Cargo.lock`, + `${HOME}/.dotfiles/web/yarn.lock`, + ]; + const selected = `${HOME}/.config/notes/notes.lock`; + const stateLock = `${HOME}/.config/notes/db.lock`; + const written = [...projects, selected, stateLock]; + const recorded: DesktopApplicationCapture = { + ...locked, + session: { + ...locked.session, + windows: locked.session.windows.map((window) => ({ + ...window, + titles: [...window.titles, "Cargo.lock - Notes", "yarn.lock - Notes"], + })), + }, + state: { + ...locked.state, + content_roots: [...locked.state.content_roots, selected], + checkpoints: [ + baseline, + { + ...edited, + changes: [ + ...edited.changes, + ...written.map((path) => change(path, { after: text })), + ], + retained: [ + ...edited.retained, + ...written.map((path) => retained(path, text)), + ], + }, + ], + }, + }; + + const { result, directory } = await exportLocked(recorded, [text]); + + const seedFiles = z + .object({ seed_files: z.array(z.string()) }) + .parse(result.normalized_result).seed_files; + expect(seedFiles).toEqual( + expect.arrayContaining([ + "Projects/.repo/Cargo.lock", + ".dotfiles/web/yarn.lock", + ".config/notes/notes.lock", + ]), + ); + for (const project of [ + "Projects/.repo/Cargo.lock", + ".dotfiles/web/yarn.lock", + ]) + expect( + await exists(join(directory, "envs/notes/seed", project)), + project, + ).toBe(true); + expect(seedFiles).not.toContain(".config/notes/db.lock"); + expect(seedFiles).not.toContain(".config/notes/.lock"); + const leftOut = result.limitations.filter((limitation) => + limitation.startsWith("Left out of the seed:"), + ); + expect(leftOut).toHaveLength(1); + expect(leftOut[0]).toContain( + "~/.config/notes/.lock, ~/.config/notes/db.lock, ~/Documents/.~lock.today.txt#.", + ); + }); + + it("names the seed files under a folder the app held files open in, rather than saying nothing there is seeded", async () => { + const { result } = await exportLocked(); + + const heldOpen = result.limitations.find((limitation) => + limitation.startsWith("The application held files open in"), + ); + expect(heldOpen).toContain("held files open in ~/Documents,"); + expect(heldOpen).toContain( + "the seed holds ~/Documents/today.txt from there", + ); + expect(heldOpen).not.toContain("nothing there is seeded"); }); }); @@ -592,6 +1039,68 @@ describe("exportDesktopApplication home paths", () => { const seed = (directory: string, path: string) => readFile(join(directory, "envs/notes/seed", path), "utf8"); + it("owns and seeds KDE's ~/.config/rc, which the app first wrote during the capture", async () => { + const local = capture(); + const [baseline, edited] = local.state.checkpoints; + if (baseline === undefined || edited === undefined) + throw new Error("the fixture has two checkpoints"); + const rc = `${HOME}/.config/notesrc`; + const text = "[General]\nShowTipOfDay=false\n"; + // The inventory names it before it exists, as it names the app's folders. + const folder = await writeCaptureFolder( + { + ...local, + application: { + ...local.application, + state_directories: [ + ...local.application.state_directories, + { path: rc, exists_at_start: false }, + ], + }, + state: { + ...local.state, + content_roots: [...local.state.content_roots, rc], + state_directories: [...(local.state.state_directories ?? []), rc], + checkpoints: [ + baseline, + { + ...edited, + changes: [ + ...edited.changes, + change(rc, { after: text }, { state: true }), + ], + retained: [...edited.retained, retained(rc, text)], + }, + ], + }, + }, + undefined, + [text], + ); + const directory = join( + await createTestTempDirectory("desktop-export-"), + "bundle", + ); + + const result = await exportDesktopApplication( + { + capture: folder, + directory, + name: undefined, + tasks: [], + seedCheckpoint: "edited", + }, + readers, + ); + + if (result instanceof Error) throw result; + expect(result.normalized_result).toMatchObject({ + paths: expect.arrayContaining([".config/notesrc"]), + seed_files: expect.arrayContaining([".config/notesrc"]), + }); + expect(await seed(directory, ".config/notesrc")).toBe(text); + }); + it("keeps a file under a symlinked home at its path under the walked home", async () => { const home = "/data/ada"; const { result, directory } = await exportWalked({ @@ -670,12 +1179,20 @@ describe("exportDesktopApplication home paths", () => { }); }); -/** Export the seed-then-end capture, launched from a desktop entry with `exec`, seeded at `seed`. */ +/** Export the seed-then-end capture, or `recorded` with `extraContent`, launched from a desktop entry with `exec`, seeded at `seed`. */ const exportSeeded = async ( exec: string | null, - options: { readonly syntheticSeed?: boolean } = {}, + { + recorded, + extraContent = [], + ...options + }: { + readonly syntheticSeed?: boolean; + readonly recorded?: DesktopApplicationCapture; + readonly extraContent?: readonly string[]; + } = {}, ) => { - const local = seedEndCapture(); + const local = recorded ?? seedEndCapture(); const folder = await writeCaptureFolder( { ...local, @@ -698,7 +1215,7 @@ const exportSeeded = async ( }, }, undefined, - [EDITED_NOTE], + [EDITED_NOTE, ...extraContent], ); const directory = join( await createTestTempDirectory("desktop-export-"), @@ -729,6 +1246,45 @@ const exportSeeded = async ( const launchLimitations = (limitations: readonly string[]) => limitations.filter((line) => line.includes(NO_LAUNCH_DOCUMENT)); +/** + * The seed-then-end capture of an app whose titles name no file, given the + * note and `~/.config/MuseScore`, a link to `target`, as state paths, with + * MuseScore's settings there from the start. + */ +const linkedSettingsCapture = (target: string): DesktopApplicationCapture => { + const base = seedEndCapture(); + const [baseline, ...later] = base.state.checkpoints; + if (baseline === undefined) throw new Error("the fixture has a baseline"); + return { + ...base, + session: { + ...base.session, + windows: base.session.windows.map((window) => ({ + ...window, + titles: ["Untitled - Notes"], + })), + }, + state: { + ...base.state, + content_roots: [...base.state.content_roots, target, NOTE], + selected_paths: [ + { path: `${HOME}/.config/MuseScore`, physical: target }, + { path: NOTE, physical: NOTE }, + ], + checkpoints: [ + { + ...baseline, + retained: [ + ...baseline.retained, + retained(`${target}/MuseScore3.ini`, "[ui]\nwrap=false\n"), + ], + }, + ...later, + ], + }, + }; +}; + describe("exportDesktopApplication launch document", () => { it("opens the seed's one document when the desktop entry takes files", async () => { const { result, module, readme } = await exportSeeded("notes %U"); @@ -745,6 +1301,37 @@ describe("exportDesktopApplication launch document", () => { expect(launchLimitations(result.limitations)).toEqual([]); }); + it.each([ + ["elsewhere in the home", "settings/MuseScore"], + [ + "to the note's own folder, where the note's deeper selection decides", + "Documents", + ], + ])( + "opens the selected note when a selected settings link leads %s", + async (_, target) => { + const { result, module } = await exportSeeded("notes %U", { + recorded: linkedSettingsCapture(`${HOME}/${target}`), + }); + + // The selected link is ~/.config/MuseScore, so the settings keep that + // spelling and the note stays the one document. + expect(result.normalized_result).toMatchObject({ + seed_files: expect.arrayContaining([ + "Documents/today.txt", + ".config/MuseScore/MuseScore3.ini", + ]), + }); + expect(result.normalized_result).not.toMatchObject({ + seed_files: expect.arrayContaining([`${target}/MuseScore3.ini`]), + }); + expect(module).toContain( + `exec 'notes' \\"$HOME\\"/'Documents/today.txt'"]`, + ); + expect(launchLimitations(result.limitations)).toEqual([]); + }, + ); + it.each(["notes", null])( "launches as captured, and says why in the README and limitations, when Exec is %j", async (exec) => { @@ -817,6 +1404,66 @@ describe("exportDesktopApplication launch document", () => { }); }); +describe("exportDesktopApplication launch document beside its lock", () => { + it("leaves LibreOffice's lock out of a selected folder's seed, so the document beside it is the one the launch opens", async () => { + const base = seedEndCapture(); + const [baseline, seed, ...later] = base.state.checkpoints; + if (baseline === undefined || seed === undefined) + throw new Error("the fixture has a baseline and a seed"); + const documents = `${HOME}/Documents`; + const recorded: DesktopApplicationCapture = { + ...base, + session: { + ...base.session, + windows: base.session.windows.map((window) => ({ + ...window, + titles: ["Untitled - Notes"], + })), + }, + state: { + ...base.state, + content_roots: [...base.state.content_roots, documents], + selected_paths: [{ path: documents, physical: documents }], + checkpoints: [ + baseline, + { + ...seed, + changes: [ + ...seed.changes, + change(DOCUMENT_LOCK, { after: DOCUMENT_LOCK_TEXT }), + ], + retained: [ + ...seed.retained, + retained(DOCUMENT_LOCK, DOCUMENT_LOCK_TEXT), + ], + }, + ...later, + ], + }, + }; + + const { result, module } = await exportSeeded("notes %U", { + recorded, + extraContent: [DOCUMENT_LOCK_TEXT], + }); + + const seedFiles = z + .object({ seed_files: z.array(z.string()) }) + .parse(result.normalized_result).seed_files; + expect(seedFiles).toContain("Documents/today.txt"); + expect(seedFiles).not.toContain("Documents/.~lock.today.txt#"); + expect( + result.limitations.filter((line) => + line.startsWith("Left out of the seed:"), + ), + ).toEqual([expect.stringContaining("~/Documents/.~lock.today.txt#")]); + expect(module).toContain( + `exec 'notes' \\"$HOME\\"/'Documents/today.txt'"]`, + ); + expect(launchLimitations(result.limitations)).toEqual([]); + }); +}); + describe("exportDesktopApplication bundle writes", () => { it.each(["cancelled", "out of space"])( "removes the unpublished bundle when its write fails (%s)", diff --git a/src/application/desktop/DesktopApplicationExport.ts b/src/application/desktop/DesktopApplicationExport.ts index eba8d18622..4c6b017805 100644 --- a/src/application/desktop/DesktopApplicationExport.ts +++ b/src/application/desktop/DesktopApplicationExport.ts @@ -16,6 +16,14 @@ import { type SeedScrubReport, } from "../../domain/desktop/appExport.js"; import { INSTALL_SCRIPT } from "../../domain/desktop/appExportInstall.js"; +import { + heldOpenLimitation, + leftOutLocksLimitation, + runtimeSeeds, + runtimeSeedsLimitation, + unownedChangedFolders, + unownedFoldersLimitation, +} from "../../domain/desktop/appExportUnowned.js"; import { installLimitations } from "../../domain/desktop/appExportInstallLimitations.js"; import type { DesktopApplicationCapture } from "../../domain/desktop/desktopCapture.js"; import { @@ -88,6 +96,8 @@ export interface DesktopExportResult { readonly unseeded_files: readonly string[]; /** Seed files rewritten to remove the capture host's name and user. */ readonly scrubbed_files: readonly string[]; + /** gzip seed files compressed again only to clear their headers, their documents as captured. */ + readonly recompressed_files: readonly string[]; /** Seed files that still hold them, in bytes the export does not rewrite or in their name. */ readonly unscrubbed_files: readonly string[]; readonly dconf_groups: number; @@ -187,16 +197,23 @@ const planExport = ( request.packageUrl, ); const electron = application.executable.electron_markers.length > 0; - const { homePath, files, collisions, paths } = exportedStateFiles( - capture, - state, - ); + const { homePath, stateAliases, files, collisions, paths, isLeftOutLock } = + exportedStateFiles(capture, state); const { text: dconf, ...dconfScrub } = seedDconf( state, context.seed, context.identity, ); - const { seeds, textSeeds } = seedVersions(files, paths, context.seed); + const { seeds, textSeeds, locks } = seedVersions( + files, + paths, + context.seed, + isLeftOutLock, + ); + const unseeded = unseededFiles(capture, homePath, files, { + paths, + seed: context.seed, + }); const launch = planLaunch(capture, electron, seeds, files); const plan: DesktopAppExportPlan = { name: source, @@ -222,10 +239,13 @@ const planExport = ( syntheticSeed: request.syntheticSeed === true, seedFiles: [...seeds.keys()].sort(), textSeeds, - unseededFiles: unseededFiles(capture, homePath, files, { - paths, - seed: context.seed, - }), + stateAliases, + unseededFiles: unseeded.filter((file) => !isLeftOutLock(file)), + leftOutLocks: [ + ...new Set([...locks, ...unseeded.filter(isLeftOutLock)]), + ].sort(), + unownedFolders: unownedChangedFolders(files, paths), + runtimeSeeds: runtimeSeeds([...seeds.keys()]), dconf, captureEvidenceId: context.evidenceId, }; @@ -245,6 +265,14 @@ const planExport = ( }; }; +/** The limitation naming files the desktop Trash held at the seed; none when it held none. */ +export const trashedFilesLimitation = ( + files: readonly string[], +): string | undefined => + files.length === 0 + ? undefined + : `The desktop Trash held ${String(files.length)} file(s) at the seed checkpoint, which are not seeded: ${files.map((home) => `~/${home}`).join(", ")}. Restore one the task needs and capture again.`; + /** The limitation on where the seed came from: made for the task, or the user's real state, kept local. */ export const seedLimitation = (syntheticSeed: boolean): string => syntheticSeed @@ -252,7 +280,7 @@ export const seedLimitation = (syntheticSeed: boolean): string => : "The seed is the user's real state at the seed checkpoint; keep the bundle local."; const SCRUB_LIMITATION = - "Binary state files are seeded as captured; only text files, UTF-8 or UTF-16 with a byte-order mark, have the capture host's name, user@host and home path replaced."; + "Binary state files are seeded as captured; only text files, UTF-8 or UTF-16 with a byte-order mark, and gzip documents, scrubbed inside, have the capture host's name, user@host and home path replaced. A gzip document so rewritten is compressed again as one member with no extra field, so a BGZF or dictzip file loses the blocks its index names."; /** The bundle's generated files, the class module flagged when its dconf keys hold the identity, and its README, made once the seeds' scrub is known. */ const bundleText = ( @@ -299,10 +327,22 @@ const exportLimitations = ( readonly collisions: readonly HomePathCollision[]; readonly identity: HostIdentity; readonly trashed: readonly string[]; + readonly files: ReadonlyMap; }, ): string[] => { - const opened = capture.state.unretained_opened_directories; + const opened = heldOpenLimitation( + capture.state.unretained_opened_directories, + capture.host.home, + plan.seedFiles.flatMap((seed) => { + const source = facts.files.get(seed)?.path; + return source === undefined ? [] : [{ seed, source }]; + }), + ); const opens = launchDocumentLimitation(plan.opens); + const unowned = unownedFoldersLimitation(plan.unownedFolders); + const runtime = runtimeSeedsLimitation(plan.runtimeSeeds); + const locks = leftOutLocksLimitation(plan.leftOutLocks); + const trashed = trashedFilesLimitation(facts.trashed); return [ seedLimitation(plan.syntheticSeed), SCRUB_LIMITATION, @@ -313,21 +353,16 @@ const exportLimitations = ( : [ `${String(plan.unseededFiles.length)} file(s) under the owned paths existed at the seed but were not retained, so they are not seeded.`, ]), - ...(facts.trashed.length === 0 - ? [] - : [ - `The desktop Trash held ${String(facts.trashed.length)} file(s) at the seed checkpoint, which are not seeded: ${facts.trashed.map((home) => `~/${home}`).join(", ")}. Restore one the task needs and capture again.`, - ]), + ...(locks === undefined ? [] : [locks]), + ...(unowned === undefined ? [] : [unowned]), + ...(runtime === undefined ? [] : [runtime]), + ...(trashed === undefined ? [] : [trashed]), ...(facts.identity.hostname === null || facts.identity.user === null ? [ "The capture does not record the host's name and user, so only the captured home is removed from the seed; capture again to remove them too.", ] : []), - ...(opened.length === 0 - ? [] - : [ - `The application held files open in ${opened.join(", ")}, whose starting content the capture did not retain, so nothing there is seeded.`, - ]), + ...(opened === undefined ? [] : [opened]), ...facts.collisions.map( ({ home, kept, dropped }) => `${dropped} and ${kept} both map to ~/${home}; the export uses ${kept} and leaves out ${dropped}.`, @@ -379,11 +414,22 @@ const runExport = async ( ); const admitted = await admitDesktopTask( request.tasks, - { ...plan, files, identity, ends, seedViews }, + { + ...plan, + files, + identity, + ends, + seedViews, + seeded: new Set(seeds.keys()), + }, signal, ); const { generated, readme } = bundleText(plan, admitted, dconf.holdsIdentity); - let scrub: SeedScrubReport = { scrubbed: [], unscrubbed: [] }; + let scrub: SeedScrubReport = { + scrubbed: [], + recompressed: [], + unscrubbed: [], + }; const unclosed = await withRetainedContent( request.capture, (content) => @@ -403,6 +449,7 @@ const runExport = async ( collisions, identity, trashed, + files, }); const result: DesktopExportResult = { directory: request.directory, @@ -416,6 +463,7 @@ const runExport = async ( seed_files: plan.seedFiles, unseeded_files: plan.unseededFiles, scrubbed_files: [...scrub.scrubbed].sort(), + recompressed_files: [...scrub.recompressed].sort(), unscrubbed_files: [...scrub.unscrubbed].sort(), dconf_groups: dconf.groups, files: [ @@ -460,6 +508,7 @@ export const exportDesktopApplication = async ( seed_files: [...result.seed_files], unseeded_files: [...result.unseeded_files], scrubbed_files: [...result.scrubbed_files], + recompressed_files: [...result.recompressed_files], unscrubbed_files: [...result.unscrubbed_files], files: [...result.files], limitations: [...result.limitations], diff --git a/src/application/desktop/DesktopApplicationState.test.ts b/src/application/desktop/DesktopApplicationState.test.ts index b6a1086a8b..17e847f6fd 100644 --- a/src/application/desktop/DesktopApplicationState.test.ts +++ b/src/application/desktop/DesktopApplicationState.test.ts @@ -60,6 +60,44 @@ const statePathsOf = async (folder: string, using = readers) => { return new Map(state.paths.map((entry) => [entry.home_path, entry])); }; +it("treats both explicitly selected files as documents when one ends in -journal", async () => { + const report = `${HOME}/Documents/report`; + const journal = `${report}-journal`; + const base = extended({ + baseline: [ + retained(report, "old report"), + retained(journal, "old journal"), + ], + changes: [ + change(report, { before: "old report", after: "new report" }), + change(journal, { before: "old journal", after: "new journal" }), + ], + later: [retained(report, "new report"), retained(journal, "new journal")], + }); + const folder = await writeCaptureFolder( + { + ...base, + state: { + ...base.state, + content_roots: [...base.state.content_roots, report, journal], + selected_paths: [report, journal].map((path) => ({ + path, + physical: path, + kind: "file" as const, + })), + }, + }, + undefined, + ["old report", "old journal", "new report", "new journal"], + ); + const byPath = await statePathsOf(folder); + for (const file of ["Documents/report", "Documents/report-journal"]) + expect(byPath.get(file)).toMatchObject({ + role: "document", + reasons: ["selected_path"], + }); +}); + describe("inspectDesktopApplicationState", () => { it("assigns roles and reads each version into typed views and changes", async () => { const folder = await writeCaptureFolder(capture()); diff --git a/src/application/desktop/DesktopApplicationState.ts b/src/application/desktop/DesktopApplicationState.ts index ff2510a067..15a3326cc1 100644 --- a/src/application/desktop/DesktopApplicationState.ts +++ b/src/application/desktop/DesktopApplicationState.ts @@ -12,9 +12,9 @@ import type { DesktopApplicationState, DesktopStatePath, } from "../../domain/desktop/desktopState.js"; +import { openStateDocument } from "../../domain/desktop/gzipDocument.js"; import { diffStateViews } from "../../domain/desktop/stateChanges.js"; import { - detectStateFormat, keyfileView, plainRecord, SQLITE_SIDECARS, @@ -24,6 +24,7 @@ import { applicationNames, isApplicationDconfGroup, roleFacts, + selectedHomePath, stateReasons, stateRole, } from "../../domain/desktop/stateRoles.js"; @@ -146,7 +147,7 @@ const dconfGroups = (text: string, names: ReadonlySet) => { * removal keeps the directory's owner and is reported with the read's outcome. */ const withStaged = async ( - files: readonly { readonly name: string; readonly bytes: Buffer }[], + files: readonly { readonly name: string; readonly bytes: Uint8Array }[], read: (paths: readonly string[]) => Promise, ): Promise => { const root = await PrivateRuntimeRoot.create({ @@ -184,10 +185,10 @@ type ViewOutcome = const failed = (error: string): ViewOutcome => ({ view: null, error }); -/** One retained version's verified bytes, with its format already detected. */ +/** One retained version's verified document, a gzip file's decompressed, with its format already detected. */ interface ReadInput { readonly path: string; - readonly bytes: Buffer; + readonly bytes: Uint8Array; readonly version: Version; } @@ -340,8 +341,14 @@ const readVersion = async ( version.view_error = `retained content ${bytes.error}`; return; } - version.format = detectStateFormat(path, bytes.value); - const outcome = await viewOf({ path, bytes: bytes.value, version }, context); + const document = openStateDocument(path, bytes.value); + version.format = document.format; + if (document.compression !== undefined) + version.compression = document.compression; + const outcome = + "error" in document + ? failed(document.error) + : await viewOf({ path, bytes: document.bytes, version }, context); version.view = outcome.view; version.view_error = outcome.error; }; @@ -377,10 +384,14 @@ const describePath = async ( ), ]; const home = capture.host.home; + const role = stateRole(reasons, selectedHomePath(facts, path)); return { path, home_path: path.startsWith(`${home}/`) ? path.slice(home.length + 1) : null, - role: stateRole(reasons), + role: + facts.sqliteSidecars?.has(path) === true && role === "unrelated" + ? "application_state" + : role, reasons, format: versions.findLast((version) => version.format !== null)?.format ?? null, @@ -400,6 +411,7 @@ const STATE_LIMITATIONS = [ "JSON and key file views read files of at most 4 MiB, JSON nested at most 500 levels and only numbers JavaScript holds exactly; other files of those formats have a view_error.", "dconf views keep only groups whose path names the application.", "Zip views list at most 1,000 entries and read an entry only when it is unencrypted, stored or deflated, at most 16 MiB and at most 100 times its compressed size, up to 64 MiB per archive; databases and archives inside an archive are not read.", + "A gzip file's view is its document's, read only when it is at most 16 MiB and at most 100 times the file's size; a database inside one is not read, nor a gzip file inside an archive.", ]; /** Read a capture folder into the application's state view. */ diff --git a/src/application/desktop/DesktopArchiveTask.test.ts b/src/application/desktop/DesktopArchiveTask.test.ts index 28a0824664..7588c8c3eb 100644 --- a/src/application/desktop/DesktopArchiveTask.test.ts +++ b/src/application/desktop/DesktopArchiveTask.test.ts @@ -1,5 +1,6 @@ import { readFile, writeFile } from "node:fs/promises"; import { join } from "node:path"; +import { gzipSync } from "node:zlib"; import { describe, expect, it } from "vitest"; @@ -144,6 +145,55 @@ const exportTask = async (folder: string, checks: unknown[]) => { return { result, directory }; }; +it.each([ + { seedTitle: "done", accepted: false }, + { seedTitle: "before", accepted: true }, +])( + "checks the recompressed gzip archive's real seed view (accepted: $accepted)", + async ({ seedTitle, accepted }) => { + const archive = (title: string) => + zipArchive([ + { name: "state.json", data: JSON.stringify({ title }), method: 0 }, + { name: "padding.txt", data: "x".repeat(60_000), method: 0 }, + ]); + const seed = gzipSync(archive(seedTitle), { level: 9 }); + const end = gzipSync(archive("done"), { level: 0 }); + const folder = await archiveCapture([seed, end]); + const state = await inspectDesktopApplicationState(folder, readers); + if (!("normalized_result" in state)) throw state; + const versions = desktopApplicationStateSchema + .parse(state.normalized_result) + .paths.find((entry) => entry.home_path === LIST)?.versions; + expect(versions?.[0]).toMatchObject({ + format: null, + compression: "gzip", + }); + expect(versions?.at(-1)).toMatchObject({ + format: "zip", + compression: "gzip", + }); + const { result } = await exportTask(folder, [ + { + kind: "value_equals", + file: LIST, + pointer: "/entries/[name=state.json]/view/title", + value: "done", + }, + ]); + if (accepted) { + if (result instanceof Error) throw result; + expect(result.normalized_result).toMatchObject({ + task: { failing_on_seed: ["check-1"] }, + }); + } else { + expect(result).toBeInstanceOf(Error); + expect((result as Error & { userMessage: string }).userMessage).toContain( + "already passes on the seed", + ); + } + }, +); + describe("documents saved as zip archives", () => { it("reads each version into its entries and lists the change by entry and element name", async () => { const evidence = await inspectDesktopApplicationState( @@ -272,6 +322,57 @@ describe("documents saved as zip archives", () => { }); }); +describe("task draft values and guards", () => { + /** The draft's files, checks and guards, as describe-desktop-application-task returns them. */ + const draftOf = async (folder: string) => { + const drafted = await draftDesktopApplicationTask(folder, readers); + if (drafted instanceof Error) throw drafted; + return drafted.normalized_result as { + files: { + file: string; + values: { pointer: string; kind: string }[]; + values_cut: boolean; + }[]; + task: { checks: { id: string }[] }; + guards: { id: string }[]; + }; + }; + + it("lists each archive's values by the pointers checks use, and guards the export accepts beside the checks", async () => { + const folder = await archiveCapture(); + + const { files, task, guards } = await draftOf(folder); + + const plate = files.find(({ file }) => file === PLATE); + expect(plate?.values_cut).toBe(false); + expect( + plate?.values.map(({ kind, pointer }) => [kind, pointer]), + ).toContainEqual([ + "added", + "/entries/[name=Document.xml]/view/root/children/[name=Objects]/children/[name=Object][@name=Hole]", + ]); + expect(guards.map(({ id }) => id)).toEqual( + guards.map((_, index) => `guard-${String(index + 1)}`), + ); + expect(guards.length).toBeGreaterThan(0); + // Guards hold at the seed and the end, so the task still fails on the seed only through its checks. + const { result } = await exportTask(folder, [...task.checks, ...guards]); + if (result instanceof Error) throw result; + }); + + it("points a refused check at the draft's values", async () => { + const folder = await archiveCapture(); + + const { result } = await exportTask(folder, [{ ...HOLE, value: 7 }]); + + expect(result).toBeInstanceOf(Error); + const remediation = (result as Error & { remediationAction?: string }) + .remediationAction; + for (const name of ["values", "stop_build_capture"]) + expect(remediation).toContain(name); + }); +}); + describe("paragraphs split into runs", () => { it("drafts and exports a paragraph's whole text once an edit split it into runs, which its first run does not hold", async () => { const run = (text: string) => diff --git a/src/application/desktop/DesktopBuildBundle.ts b/src/application/desktop/DesktopBuildBundle.ts index 014ad26e06..08495ade24 100644 --- a/src/application/desktop/DesktopBuildBundle.ts +++ b/src/application/desktop/DesktopBuildBundle.ts @@ -1,5 +1,12 @@ -import { copyFile, mkdir, readFile, stat, writeFile } from "node:fs/promises"; -import { dirname, isAbsolute, join } from "node:path"; +import { + copyFile, + mkdir, + readdir, + readFile, + stat, + writeFile, +} from "node:fs/promises"; +import { dirname, isAbsolute, join, relative, sep } from "node:path"; import { renderBuildBundle, @@ -10,6 +17,7 @@ import { AnalysisCapabilityUnavailableError, AnalysisInputError, } from "../../domain/analysisErrorCore.js"; +import { isLockFile } from "../../domain/desktop/appExportUnowned.js"; import { AGENTENV_PROVIDER } from "../../desktop/AgentEnvComputerHelper.js"; import { PrivateRuntimeRoot } from "../../process/PrivateRuntimeRoot.js"; @@ -108,3 +116,31 @@ export const writeRequestBundle = async ( request: BuildRequest, environment: Readonly, ) => writeBuildBundle(request, await reaPayload(environment)); + +/** + * The lock files in a bundle folder's seeds, as `envs//seed/`; none when it has no seed. A bundle records no state directories, so + * only a profile lock's name or LibreOffice's document lock counts here. + */ +export const seededLockFiles = async ( + bundle: string, +): Promise => { + const environments = join(bundle, "envs"); + const names = await readdir(environments).catch(() => []); + const locks: string[] = []; + for (const name of names) { + const seed = join(environments, name, "seed"); + const entries = await readdir(seed, { + recursive: true, + withFileTypes: true, + }).catch(() => []); + for (const entry of entries) { + const home = relative(seed, join(entry.parentPath, entry.name)) + .split(sep) + .join("/"); + if (entry.isFile() && isLockFile(home, [])) + locks.push(`envs/${name}/seed/${home}`); + } + } + return locks.sort(); +}; diff --git a/src/application/desktop/DesktopBuildCapture.ts b/src/application/desktop/DesktopBuildCapture.ts index c5997d61c5..816d5650bd 100644 --- a/src/application/desktop/DesktopBuildCapture.ts +++ b/src/application/desktop/DesktopBuildCapture.ts @@ -16,6 +16,11 @@ import { ProviderAdapterError } from "../../domain/providerAdapterError.js"; import { err, ok, type Result } from "../../domain/result.js"; import { AGENTENV_PROVIDER } from "../../desktop/AgentEnvComputerHelper.js"; import type { ControlledComputer } from "../../desktop/BuildComputerController.js"; +import { + endedCapture, + notAnswered, + type CaptureStep, +} from "./DesktopBuildCaptureFailure.js"; import { adapterError, controller, @@ -39,11 +44,31 @@ const START = "start_build_capture"; const MARK = "mark_build_checkpoint"; const STOP = "stop_build_capture"; const EXPORT = "export_build_environment"; +/** The start's step, for a capture that ended before the app's window opened. */ +const STARTING: CaptureStep = { + operation: START, + when: "before the app's window opened", + afterAppExit: + "The app does not run as installed: check what it needs, such as a runtime its stderr names, or install it from another source on a new build computer, and close this one.", +}; +/** A checkpoint's step, for a capture that ended before acknowledging it. */ +const marking = (name: string): CaptureStep => ({ + operation: MARK, + when: `before checkpoint ${name}`, + afterAppExit: + "The capture ended with the app: stop_build_capture reads what it recorded, then start_build_capture captures again.", +}); const NOT_RUNNING = [ "No capture is running on this computer.", "Start one with start_build_capture.", ] as const; +/** What the agent asks of a capture's launch: arguments for the app, and files or folders in the home that are its state. */ +export interface CaptureInput { + readonly arguments?: readonly string[] | undefined; + readonly state_paths?: readonly string[] | undefined; +} + const exportOptionsSchema = z.object({ options: z.object({ properties: z.record(z.string(), z.unknown()) }), }); @@ -68,7 +93,9 @@ const checkExportRuntime = async ( { stdout: path, timeoutS: 30, signal }, ); if (command.exitCode !== 0 && command.stderr !== "") - return err(schemaCommandError(command.exitCode, command.stderr)); + return err( + schemaCommandError(command.exitCode, computer.redacted(command.stderr)), + ); if (command.stdoutBytes > 1024 * 1024) return err( adapterError( @@ -90,12 +117,23 @@ const checkExportRuntime = async ( return err( schemaCommandError( command.exitCode, - content.toString("utf8").slice(-2048), + computer.redacted(content.toString("utf8")).slice(-2048), ), ); - const schema = exportOptionsSchema.safeParse( - JSON.parse(content.toString("utf8")), - ); + let rawSchema: unknown; + try { + rawSchema = JSON.parse(content.toString("utf8")); + } catch { + // Parser errors can quote only a fragment of a credential, which cannot + // be removed reliably by exact-token redaction. + return err( + adapterError( + START, + "The build computer's REA export schema is invalid JSON.", + ), + ); + } + const schema = exportOptionsSchema.safeParse(rawSchema); if (!schema.success) return err( adapterError( @@ -113,37 +151,33 @@ const checkExportRuntime = async ( ); return ok(undefined); } catch (cause: unknown) { + if (signal.aborted) + return err(new AnalysisCancelledError(START, { cause: signal.reason })); + const message = errorText(cause); + const redacted = computer.redacted(message); return err( - signal.aborted - ? new AnalysisCancelledError(START, { cause: signal.reason }) - : adapterError( - START, - `Could not read the build computer's REA export schema: ${errorText(cause)}`, - cause, - ), + adapterError( + START, + `Could not read the build computer's REA export schema: ${redacted}`, + redacted === message ? cause : undefined, + ), ); } }; +/** A taken checkpoint as the tool reports it: changed files null when the REA in the computer predates them. */ type CheckpointTaken = Omit< Extract, - "kind" ->; + "kind" | "changed_files" | "more_changed_files" +> & { + readonly changed_files: string[] | null; + readonly more_changed_files: number | null; +}; /** The capture's progress event in a line the controller waited for. */ const eventOf = (line: string): DesktopCaptureEvent | undefined => captureProgressEvent(line); -/** Why the capture is not answering, from what the controller saw instead of the event. */ -const notAnswered = (operation: string, answer: Answer): AnalysisError => - adapterError( - operation, - answer.error ?? - (answer.exited === undefined - ? `The capture did not answer in time. Its last progress: ${answer.progress ?? ""}` - : `The capture ended with exit status ${String(answer.exited)}. ${answer.output ?? ""} ${answer.progress ?? ""}`.trim()), - ); - /** * One capture in the computer. `unreleased` is a capture that may be * running although its start failed, kept until a stop is confirmed; @@ -204,7 +238,7 @@ export class DesktopBuildCapture { } async start( - input: { readonly arguments?: readonly string[] }, + input: CaptureInput, signal: AbortSignal, ): Promise< Result< @@ -261,10 +295,20 @@ export class DesktopBuildCapture { return err(started.error.failure); } + /** + * The capture launched, then the caller cancelled before the start was + * reported. The next start stops it; until then it is not a second capture. + */ + abandon(): void { + const run = this.#run; + if (run?.status === "running" && run.stopped === undefined) + run.status = "unreleased"; + } + async #launch( run: CaptureRun, entry: string, - input: { readonly arguments?: readonly string[] }, + input: CaptureInput, signal: AbortSignal, ): Promise< Result< @@ -286,6 +330,7 @@ export class DesktopBuildCapture { capture: run.folder, desktop_entry: entry, arguments: [...(input.arguments ?? [])], + state_paths: [...(input.state_paths ?? [])], ...(this.#packageFile === undefined ? {} : { package_file: BUILD_PACKAGE_FILE }), @@ -295,11 +340,33 @@ export class DesktopBuildCapture { { timeoutS: 720, signal }, ); } catch (cause: unknown) { - return err({ failure: adapterError(START, errorText(cause), cause) }); + return err({ + failure: + cause instanceof AnalysisError + ? cause + : adapterError(START, errorText(cause), cause), + }); } + if (answer.refused !== undefined) + return err({ + failure: refused( + START, + answer.refused, + "Name files or folders under the computer's home in state_paths, relative to it or absolute, with no link on the way leading out of it.", + ), + answer, + }); const event = answer.line === undefined ? undefined : eventOf(answer.line); if (event?.kind !== "window_open") - return err({ failure: notAnswered(START, answer), answer }); + return err({ + failure: await endedCapture( + this.computer, + { ...STARTING, folder: run.folder }, + answer, + signal, + ), + answer, + }); return ok({ desktop_entry: entry, package_lists: answer.apt?.lists ?? 0, @@ -352,12 +419,33 @@ export class DesktopBuildCapture { answer.line === undefined ? undefined : eventOf(answer.line); if (event?.kind === "checkpoint_refused") return err(refused(MARK, event.reason, "Use another checkpoint name.")); - if (event?.kind !== "checkpoint") return err(notAnswered(MARK, answer)); + if (event?.kind !== "checkpoint") + return err( + await endedCapture( + this.computer, + { ...marking(name), folder: run.folder }, + answer, + signal, + ), + ); run.checkpoints.add(name); - const { kind: _kind, ...taken } = event; - return ok(taken); + const { + kind: _kind, + changed_files: changed, + more_changed_files: more, + ...taken + } = event; + return ok({ + ...taken, + changed_files: changed ?? null, + more_changed_files: more ?? null, + }); } catch (cause: unknown) { - return err(adapterError(MARK, errorText(cause), cause)); + return err( + cause instanceof AnalysisError + ? cause + : adapterError(MARK, errorText(cause), cause), + ); } } diff --git a/src/application/desktop/DesktopBuildCaptureFailure.ts b/src/application/desktop/DesktopBuildCaptureFailure.ts new file mode 100644 index 0000000000..c7a92c9d5c --- /dev/null +++ b/src/application/desktop/DesktopBuildCaptureFailure.ts @@ -0,0 +1,176 @@ +import { AnalysisError } from "../../domain/analysisErrorBase.js"; +import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js"; +import { + desktopApplicationCaptureSchema, + desktopCaptureProgressSchema, + type DesktopApplicationCapture, +} from "../../domain/desktop/desktopCapture.js"; +import { ProviderAdapterError } from "../../domain/providerAdapterError.js"; +import { AGENTENV_PROVIDER } from "../../desktop/AgentEnvComputerHelper.js"; +import type { ControlledComputer } from "../../desktop/BuildComputerController.js"; +import { + adapterError, + readOutput, + reportedFailure, + type Answer, +} from "./DesktopBuildCommands.js"; + +/** The most lines, and characters, a message quotes from the end of the app's stderr. */ +const STDERR_QUOTE = { lines: 8, characters: 800 } as const; +/** The most characters a message quotes of a program or a progress line. */ +const LINE_QUOTE = 300; + +/** The end of `text`, at most `limit` characters, marked when cut. */ +const ending = (text: string, limit: number): string => { + const characters = [...text.trim()]; + return characters.length <= limit + ? characters.join("") + : `…${characters.slice(-limit).join("")}`; +}; + +/** The last whole lines of the app's stderr within the quote's bounds. */ +const stderrEnding = (stderr: string): string => { + const lines = stderr.trim().split("\n"); + const kept: string[] = []; + let size = 0; + for (const line of lines.toReversed()) { + const grown = size + line.length + 1; + if ( + kept.length === STDERR_QUOTE.lines || + (kept.length > 0 && grown > STDERR_QUOTE.characters) + ) + break; + kept.unshift(line); + size = grown; + } + const quoted = ending(kept.join("\n"), STDERR_QUOTE.characters); + return kept.length < lines.length ? `…\n${quoted}` : quoted; +}; + +/** The capture's last progress message in a tail of its progress log, or its last line. */ +const lastProgress = (progress: string | undefined): string | undefined => { + const lines = (progress ?? "") + .split("\n") + .map((line) => line.trim()) + .filter((line) => line.length > 0); + for (let index = lines.length - 1; index >= 0; index -= 1) { + try { + const parsed = desktopCaptureProgressSchema.safeParse( + JSON.parse(lines[index] ?? ""), + ); + if (parsed.success) return parsed.data.rea_progress.message; + } catch { + // A line cut by the tail, or one the capture's runtime wrote. + } + } + return lines.at(-1); +}; + +/** Why the capture is not answering, from what the controller saw instead of the event; its output stays out of the message. */ +export const notAnswered = ( + operation: string, + answer: Answer, +): AnalysisError => { + if (answer.error !== undefined) return adapterError(operation, answer.error); + const last = lastProgress(answer.progress); + const progress = + last === undefined ? "" : ` Its last progress: ${ending(last, LINE_QUOTE)}`; + return adapterError( + operation, + answer.exited === undefined + ? `The capture did not answer in time.${progress}` + : `The capture ended with exit status ${String(answer.exited)} before it answered.${progress}`, + ); +}; + +/** The step a capture that ended did not answer, as its errors name it. */ +export interface CaptureStep { + readonly operation: string; + /** When the capture failed, after "The capture failed", such as "before checkpoint seed". */ + readonly when: string; + /** What to do when the app itself exited and so ended the capture. */ + readonly afterAppExit: string; +} + +/** The app exited and so ended the capture: how it exited and what it said last, with the capture's record kept as details. */ +const appExited = ( + step: CaptureStep, + recorded: DesktopApplicationCapture, + exitStatus: number, +): AnalysisError | undefined => { + const { application, session } = recorded; + if (session.end_reason !== "application_exited") return undefined; + const { exit } = session; + const ended = + exit === null + ? "ended" + : exit.signal === null + ? `exited with code ${String(exit.code)}` + : `was ended by ${exit.signal}`; + const window = + session.first_window_ms === null ? " without opening a window" : ""; + const stderr = + session.stderr.trim() === "" + ? "it wrote nothing to stderr" + : `its stderr ends: ${stderrEnding(session.stderr)}`; + return new ProviderAdapterError(AGENTENV_PROVIDER, step.operation, { + userMessage: `${ending(application.argv[0] ?? "The app", LINE_QUOTE)} ${ended} after ${String(session.duration_ms)} ms${window}; ${stderr}`, + remediationAction: step.afterAppExit, + diagnostics: { + capture_exit_status: exitStatus, + application: { argv: [...application.argv] }, + session: { + end_reason: session.end_reason, + duration_ms: session.duration_ms, + exit: session.exit, + first_window_ms: session.first_window_ms, + stderr: session.stderr, + stderr_truncated: session.stderr_truncated, + }, + limitations: [...recorded.limitations], + }, + }); +}; + +/** + * Why a capture step got no answer. When the capture has ended, its own + * output says how, read the same way for every step that waits on it: a + * failure it reported, with its details, or the app exiting; otherwise, or + * when that output cannot be read, what the controller saw. + */ +export const endedCapture = async ( + computer: ControlledComputer, + step: CaptureStep & { readonly folder: string }, + answer: Answer, + signal: AbortSignal, +): Promise => { + if (answer.exited === undefined || answer.output_bytes === undefined) + return notAnswered(step.operation, answer); + try { + const output = await readOutput( + computer, + `${step.folder}/capture.out`, + answer.output_bytes, + signal, + ); + if (output.kind === "failure") + return reportedFailure( + step.operation, + output, + `The capture failed ${step.when}: `, + ); + const recorded = desktopApplicationCaptureSchema.safeParse(output.result); + return ( + (recorded.success + ? appExited(step, recorded.data, answer.exited) + : undefined) ?? notAnswered(step.operation, answer) + ); + } catch (cause: unknown) { + if (cause instanceof AnalysisError) return cause; + if (signal.aborted) + return new AnalysisCancelledError(step.operation, { + cause: signal.reason, + }); + return notAnswered(step.operation, answer); + } +}; diff --git a/src/application/desktop/DesktopBuildCommands.ts b/src/application/desktop/DesktopBuildCommands.ts index 356817e6a0..ac4fb4f4e2 100644 --- a/src/application/desktop/DesktopBuildCommands.ts +++ b/src/application/desktop/DesktopBuildCommands.ts @@ -6,7 +6,16 @@ import { SEED_CHECKPOINT, } from "../../domain/desktop/buildCapture.js"; import { parseEvidence } from "../../domain/evidence.js"; +import { + jsonObjectSchema, + jsonValueSchema, + type JsonValue, +} from "../../domain/jsonValue.js"; import { ProviderAdapterError } from "../../domain/providerAdapterError.js"; +import { + guestFileAnswerCeiling, + guestFileAnswerRefusal, +} from "../../desktop/AgentEnvComputer.js"; import { AGENTENV_PROVIDER } from "../../desktop/AgentEnvComputerHelper.js"; import { runRea as runControllerRea, @@ -32,15 +41,19 @@ const cliFailureSchema = z.looseObject({ code: z.string(), message: z.string(), remediation: z.looseObject({ action: z.string() }).optional(), + details: z.record(z.string(), jsonValueSchema).optional(), }); +/** A failure a REA command in the computer reported, and the whole projection it reported it as. */ +export interface CommandFailure { + readonly failure: z.infer; + readonly reported: Readonly>; +} + /** What REA reads from a command it ran in the computer: its Evidence result, or the failure it reported. */ export type CommandOutput = | { readonly kind: "result"; readonly result: unknown } - | { - readonly kind: "failure"; - readonly failure: z.infer; - }; + | ({ readonly kind: "failure" } & CommandFailure); export const errorText = (cause: unknown): string => cause instanceof Error ? cause.message : String(cause); @@ -55,6 +68,44 @@ export const adapterError = ( ...(cause === undefined ? {} : { cause }), }); +const cleanupTargetsSchema = z.object({ + cleanup: z.literal("incomplete"), + resources: z.array(z.string()), + cleanup_reason: z.string().optional(), +}); + +/** + * A failure a REA command in the computer reported, as this step's error: + * its message after `lead`, its remediation, the whole failure under + * `details.diagnostics.command_failure`, and an incomplete cleanup's targets, + * such as run tokens and paths, kept as this error's own, since its + * remediation names them as listed. + */ +export const reportedFailure = ( + operation: string, + { failure, reported }: CommandFailure, + lead = "", +): ProviderAdapterError => { + const cleanup = cleanupTargetsSchema.safeParse(failure.details); + return new ProviderAdapterError(AGENTENV_PROVIDER, operation, { + userMessage: `${lead}${failure.message}`, + ...(failure.remediation === undefined + ? {} + : { remediationAction: failure.remediation.action }), + diagnostics: { command_failure: reported }, + ...(cleanup.success + ? { + cleanup: { + reason: + cleanup.data.cleanup_reason ?? + "the command's cleanup is incomplete", + resources: cleanup.data.resources, + }, + } + : {}), + }); +}; + /** A valid request the build's current state cannot take, with what to do instead. */ export const refused = ( operation: string, @@ -80,18 +131,53 @@ export const runRea = ( ): ReturnType => runControllerRea(computer, argv, { ...options, work: WORK }); -/** The Evidence result, or the reported failure, a REA command wrote to a file of `bytes` in the computer. */ +/** `value` with `redact` applied to each of its strings and keys. */ +const redactedJson = ( + value: JsonValue, + redact: (text: string) => string, +): JsonValue => { + if (typeof value === "string") return redact(value); + if (Array.isArray(value)) + return value.map((item) => redactedJson(item, redact)); + if (value === null || typeof value !== "object") return value; + return Object.fromEntries( + Object.entries(value).map(([key, item]) => [ + redact(key), + redactedJson(item, redact), + ]), + ); +}; + +/** + * The Evidence result, or the reported failure, a REA command wrote to a file + * of `bytes` in the computer, with the control token replaced in every + * string, as in a command's own output; the Evidence is checked first, since + * its digest covers what the command wrote. + */ export const readOutput = async ( computer: ControlledComputer, path: string, bytes: number, signal: AbortSignal, ): Promise => { + const refusal = guestFileAnswerRefusal(bytes, 1, guestFileAnswerCeiling()); + if (refusal !== undefined) throw refusal; const content = (await computer.homeFiles([path], bytes, signal)).get(path); if (content === undefined) throw new Error(`The computer has no ${path}.`); const value: unknown = JSON.parse(content.toString("utf8")); - const failure = cliFailureSchema.safeParse(value); - return failure.success - ? { kind: "failure", failure: failure.data } - : { kind: "result", result: parseEvidence(value).normalized_result }; + const redact = (text: string) => computer.redacted(text); + if (cliFailureSchema.safeParse(value).success) { + const reported = jsonObjectSchema.parse( + redactedJson(jsonObjectSchema.parse(value), redact), + ); + return { + kind: "failure", + failure: cliFailureSchema.parse(reported), + reported, + }; + } + return { + kind: "result", + result: redactedJson(parseEvidence(value).normalized_result, redact), + }; }; diff --git a/src/application/desktop/DesktopBuildComputers.ts b/src/application/desktop/DesktopBuildComputers.ts index 9277cbe9c2..aa2e93a7df 100644 --- a/src/application/desktop/DesktopBuildComputers.ts +++ b/src/application/desktop/DesktopBuildComputers.ts @@ -1,6 +1,8 @@ import { randomUUID } from "node:crypto"; import type { BuildSource } from "../../domain/desktop/appBuild.js"; +import { counted } from "../../domain/desktop/appExportUnowned.js"; +import { SETTLE_TIMING } from "../../domain/desktop/appSettle.js"; import { driverCall, type ComputerAction, @@ -23,6 +25,7 @@ import { AgentEnvComputer, AppDataRefusal, ComputerStartFailure, + computerFailedOnReadiness, type DriverResult, } from "../../desktop/AgentEnvComputer.js"; import { @@ -36,8 +39,9 @@ import { BuildComputerActionLog, type ReadonlyActionLog, } from "./DesktopBuildActionLog.js"; -import { writeRequestBundle } from "./DesktopBuildBundle.js"; +import { seededLockFiles, writeRequestBundle } from "./DesktopBuildBundle.js"; import { DesktopBuildCapture } from "./DesktopBuildCapture.js"; +import type { BuildSettle } from "./DesktopBuildSettle.js"; import { DesktopBuildTurns } from "./DesktopBuildTurns.js"; import { DesktopBuildWitness } from "./DesktopBuildWitness.js"; @@ -195,6 +199,7 @@ const liveComputer = ( computer: AgentEnvComputer, workspace: PrivateRuntimeRoot | undefined, source: BuildSource, + settle: BuildSettle, ): HeldComputer => { const actions = new BuildComputerActionLog(); const turns = new DesktopBuildTurns(); @@ -212,6 +217,7 @@ const liveComputer = ( witness: new DesktopBuildWitness( computer, actions, + settle, source.kind === "bundle" ? source.path : undefined, ), turns, @@ -221,6 +227,32 @@ const liveComputer = ( }; }; +/** + * A bundle's start that failed on readiness, naming the lock files its seed + * holds, which an app that finds its profile or document in use may never + * start past; any other failure as it is. Readiness is `reason`, not the + * message. + */ +/** The start failure's notice that the bundle seed still holds lock files. */ +export const seededLockStartNotice = (locks: readonly string[]): string => + `The bundle's seed holds lock files, which can keep the app from starting when it finds its profile or document in use: ${counted(locks)}. Delete them from the bundle, or export it again, which leaves them out, and start again.`; + +const withSeededLocks = async ( + source: BuildSource, + cause: unknown, +): Promise => { + if (source.kind !== "bundle" || !computerFailedOnReadiness(cause)) + return cause; + const locks = await seededLockFiles(source.path); + return locks.length === 0 + ? cause + : new ComputerStartFailure( + `${cause.message} ${seededLockStartNotice(locks)}`, + cause.remediation, + { cause: cause.cause, reason: cause.reason }, + ); +}; + /** * Why a start failed, with the helper's output, and, when its teardown is * unconfirmed, the id REA keeps that teardown under for close to retry. @@ -274,7 +306,10 @@ export class DesktopBuildComputers { readonly #starts = new Set(); #closed = false; - constructor(private readonly environment: Readonly) {} + constructor( + private readonly environment: Readonly, + private readonly settle: BuildSettle = SETTLE_TIMING, + ) {} async start( input: { @@ -367,7 +402,10 @@ export class DesktopBuildComputers { shot.value.arguments, signal, ); - this.#computers.set(id, liveComputer(computer, target.workspace, source)); + this.#computers.set( + id, + liveComputer(computer, target.workspace, source, this.settle), + ); return ok({ value: { computer_id: id, @@ -395,7 +433,12 @@ export class DesktopBuildComputers { closing: true, }); const release = await this.#retire(id); - return err(startFailure(cause, signal, owner?.output(), { id, release })); + const failed = signal.aborted + ? cause + : await withSeededLocks(source, cause); + return err( + startFailure(failed, signal, owner?.output(), { id, release }), + ); } } diff --git a/src/application/desktop/DesktopBuildExport.ts b/src/application/desktop/DesktopBuildExport.ts index becee0b379..98203190e2 100644 --- a/src/application/desktop/DesktopBuildExport.ts +++ b/src/application/desktop/DesktopBuildExport.ts @@ -17,6 +17,7 @@ import { errorText, readOutput, refused, + reportedFailure, runRea, } from "./DesktopBuildCommands.js"; import { @@ -25,13 +26,12 @@ import { } from "./DesktopOutputTree.js"; const EXPORT = "export_build_environment"; -/** The most bytes the task takes in the controller's one JSON argument, which Linux caps at 128 KiB. */ -const TASK_ARGUMENT_BYTES = 96 * 1024; - const exportResultSchema = z.looseObject({ environment: z.string(), seed_files: z.array(z.string()), scrubbed_files: z.array(z.string()), + // An export from before gzip seeds were listed apart has none. + recompressed_files: z.array(z.string()).default([]), unscrubbed_files: z.array(z.string()), limitations: z.array(z.string()), task: jsonValueSchema, @@ -53,6 +53,7 @@ export type ExportedBuild = { readonly task: JsonValue; readonly seed_files: string[]; readonly scrubbed_files: string[]; + readonly recompressed_files: string[]; readonly unscrubbed_files: string[]; readonly limitations: string[]; }; @@ -127,15 +128,6 @@ export const exportBuild = async ( const task = `${folder}/task-${attempt}.json`; const stdout = `${folder}/export-${attempt}.json`; const taskFile = `${JSON.stringify(request.task, null, 2)}\n`; - const taskBytes = Buffer.byteLength(JSON.stringify(taskFile)); - if (taskBytes > TASK_ARGUMENT_BYTES) - return err( - refused( - EXPORT, - `The task takes ${String(taskBytes)} bytes in the command REA runs in the computer, more than the ${String(TASK_ARGUMENT_BYTES)} it allows a task.`, - "Shorten the task's prompt or checks and export again.", - ), - ); try { const { stdoutBytes: bytes } = await runRea( computer, @@ -170,7 +162,7 @@ export const exportBuild = async ( output.failure.remediation?.action ?? "Rewrite the task and export again.", ) - : adapterError(EXPORT, output.failure.message), + : reportedFailure(EXPORT, output), ); const result = exportResultSchema.parse(output.result); const written = await writeBundle( @@ -188,6 +180,7 @@ export const exportBuild = async ( task: result.task, seed_files: result.seed_files, scrubbed_files: result.scrubbed_files, + recompressed_files: result.recompressed_files, unscrubbed_files: result.unscrubbed_files, limitations: result.limitations, }); diff --git a/src/application/desktop/DesktopBuildOutputRead.test.ts b/src/application/desktop/DesktopBuildOutputRead.test.ts new file mode 100644 index 0000000000..43d779fc04 --- /dev/null +++ b/src/application/desktop/DesktopBuildOutputRead.test.ts @@ -0,0 +1,123 @@ +import { describe, expect, it } from "vitest"; + +import { AnalysisResourceConstraintError } from "../../domain/analysisErrorCore.js"; +import { + GUEST_FILE_ANSWER_SOURCE, + guestFileAnswerCeiling, +} from "../../desktop/AgentEnvComputer.js"; +import { readOutput } from "./DesktopBuildCommands.js"; +import { endedCapture } from "./DesktopBuildCaptureFailure.js"; + +describe("endedCapture", () => { + it("returns the size refusal instead of a generic exit when the capture output is too large", async () => { + const error = await endedCapture( + { + homeFiles: () => Promise.resolve(new Map()), + redacted: (text) => text, + exec: () => Promise.reject(new Error("unused")), + }, + { + operation: "desktop.capture", + when: "before checkpoint seed", + afterAppExit: "unused", + folder: "/tmp/capture", + }, + { exited: 0, output_bytes: Number.MAX_SAFE_INTEGER }, + new AbortController().signal, + ); + + expect(error).toBeInstanceOf(AnalysisResourceConstraintError); + expect(error.message).toContain("max-old-space-size"); + }); +}); + +describe("readOutput", () => { + it("does not fetch a command output whose reported size exceeds the host answer ceiling", async () => { + let fetched = false; + const failure = await readOutput( + { + homeFiles: () => { + fetched = true; + return Promise.resolve(new Map()); + }, + redacted: (text) => text, + exec: () => Promise.reject(new Error("unused")), + }, + "out.json", + Number.MAX_SAFE_INTEGER, + new AbortController().signal, + ).catch((cause: unknown) => cause); + expect(fetched).toBe(false); + expect(failure).toBeInstanceOf(AnalysisResourceConstraintError); + if (!(failure instanceof AnalysisResourceConstraintError)) return; + expect(failure.reportedLimits).toMatchObject({ + guest_file_bytes: Number.MAX_SAFE_INTEGER, + file_count: 1, + answer_ceiling_bytes: guestFileAnswerCeiling(), + ceiling_source: GUEST_FILE_ANSWER_SOURCE, + }); + }); + + it("reads a reported failure that fits the ceiling", async () => { + const body = Buffer.from( + JSON.stringify({ code: "busy", message: "the app is busy" }), + ); + let reported = 0; + const output = await readOutput( + { + homeFiles: (_paths, bytes) => { + reported = bytes; + return Promise.resolve(new Map([["out.json", body]])); + }, + redacted: (text) => text, + exec: () => Promise.reject(new Error("unused")), + }, + "out.json", + body.length, + new AbortController().signal, + ); + expect(reported).toBe(body.length); + expect(output.kind).toBe("failure"); + if (output.kind !== "failure") return; + expect(output.failure).toMatchObject({ + code: "busy", + message: "the app is busy", + }); + }); +}); + +describe("endedCapture", () => { + it("keeps a capacity error from the capture output, including its recovery", async () => { + let fetched = false; + const failure = await endedCapture( + { + homeFiles: () => { + fetched = true; + return Promise.resolve(new Map()); + }, + redacted: (text) => text, + exec: () => Promise.reject(new Error("unused")), + }, + { + operation: "desktop_build_capture_start", + when: "before checkpoint seed", + afterAppExit: "Start the capture again.", + folder: "capture", + }, + { exited: 0, output_bytes: Number.MAX_SAFE_INTEGER }, + new AbortController().signal, + ); + expect(fetched).toBe(false); + expect(failure).toBeInstanceOf(AnalysisResourceConstraintError); + if (!(failure instanceof AnalysisResourceConstraintError)) return; + expect(failure.operation).toBe("data/get"); + expect(failure.retryable).toBe(false); + expect(failure.remediationAction).toContain("--max-old-space-size"); + expect(failure.reportedLimits).toMatchObject({ + guest_file_bytes: Number.MAX_SAFE_INTEGER, + file_count: 1, + answer_ceiling_bytes: guestFileAnswerCeiling(), + ceiling_source: GUEST_FILE_ANSWER_SOURCE, + }); + }); +}); diff --git a/src/application/desktop/DesktopBuildSettle.ts b/src/application/desktop/DesktopBuildSettle.ts new file mode 100644 index 0000000000..8d523635c6 --- /dev/null +++ b/src/application/desktop/DesktopBuildSettle.ts @@ -0,0 +1,97 @@ +import { setTimeout as delay } from "node:timers/promises"; + +import { + appFileDigests, + type SettleTiming, +} from "../../domain/desktop/appSettle.js"; + +/** The time a settle keeps; tests substitute a virtual clock. */ +export interface SettleClock { + now(): number; + delay(ms: number, signal: AbortSignal): Promise; + /** A signal that aborts once `ms` have passed. */ + timeout(ms: number): AbortSignal; +} + +const monotonicClock: SettleClock = { + now: () => performance.now(), + delay: (ms, signal) => delay(ms, undefined, { signal }), + // A timeout takes whole milliseconds. + timeout: (ms) => AbortSignal.timeout(Math.max(1, Math.floor(ms))), +}; + +/** How a build computer reads an app until its files settle, on real time unless a clock is given. */ +export interface BuildSettle extends SettleTiming { + readonly clock?: SettleClock; +} + +/** + * The last completed data/get answer of a settled read, whether the app's + * files had held for the window by then, and whether the bound cut a read + * still in flight. + */ +export interface SettledRead { + readonly parts: readonly unknown[]; + readonly settled: boolean; + readonly readCut: boolean; +} + +const seconds = (ms: number) => `${(ms / 1000).toFixed(1)} s`; + +/** + * Read an app's data/get every interval until each file's bytes have held + * for the window, counted from the read that returned them, or until the + * bound, as the exported verifier does; `waiting` hears why it reads again. + * No read starts at or past the bound, and one in flight there is cut. + */ +export const settledAppData = async ( + read: (signal: AbortSignal) => Promise, + timing: BuildSettle, + signal: AbortSignal, + waiting: (message: string) => Promise, +): Promise => { + const clock = timing.clock ?? monotonicClock; + const start = clock.now(); + const elapsed = () => clock.now() - start; + // The first read is the one a grade needs, so only its own timeout bounds it. + let parts = await read(signal); + let files = appFileDigests(parts); + // Offsets from the first read's start: when the last completed read began, + // and when a read first returned the bytes it holds. + let began = 0; + let since = elapsed(); + const unsettled = () => began - since < timing.windowMs; + while (unsettled()) { + // Reads come an interval apart, and the one that can settle the files as soon as the window allows. + const next = Math.max( + Math.min(began + timing.intervalMs, since + timing.windowMs), + elapsed(), + ); + if (next >= timing.boundMs) + return { parts, settled: false, readCut: false }; + await clock.delay(Math.max(0, next - elapsed()), signal); + const at = Math.max(next, elapsed()); + if (at >= timing.boundMs) return { parts, settled: false, readCut: false }; + const bound = clock.timeout(timing.boundMs - at); + let latest: readonly unknown[]; + try { + latest = await read(AbortSignal.any([signal, bound])); + } catch (cause: unknown) { + if (bound.aborted && !signal.aborted) + return { parts, settled: false, readCut: true }; + throw cause; + } + began = at; + parts = latest; + const digests = appFileDigests(latest); + if (digests !== files) { + since = elapsed(); + files = digests; + } + if (unsettled()) + await waiting( + `waiting for the app's files to hold for ${seconds(timing.windowMs)}: unchanged for ${seconds(Math.max(0, began - since))}, ${seconds(began)} of at most ${seconds(timing.boundMs)}`, + ); + } + return { parts, settled: true, readCut: false }; +}; diff --git a/src/application/desktop/DesktopBuildSteps.ts b/src/application/desktop/DesktopBuildSteps.ts index bfd82865d8..ceff338b5e 100644 --- a/src/application/desktop/DesktopBuildSteps.ts +++ b/src/application/desktop/DesktopBuildSteps.ts @@ -5,6 +5,7 @@ import { } from "../../domain/analysisErrorCore.js"; import type { JsonValue } from "../../domain/jsonValue.js"; import { err, ok, type Result } from "../../domain/result.js"; +import type { ProgressReporter } from "../ProgressReporter.js"; import type { StoppedCapture } from "./DesktopBuildSummary.js"; import type { BuildComputerSteps, @@ -41,6 +42,7 @@ export class DesktopBuildSteps { input: { readonly computer_id: string; readonly arguments?: readonly string[] | undefined; + readonly state_paths?: readonly string[] | undefined; }, signal: AbortSignal, ): Promise< @@ -60,21 +62,21 @@ export class DesktopBuildSteps { if (steps === undefined) return err(unknownComputer("start_build_capture")); // The screenshot of the opened app takes the same turn as the start. return steps.turns.run("start_build_capture", signal, async (turn) => { - const started = await steps.capture.start( - input.arguments === undefined ? {} : { arguments: input.arguments }, - turn, - ); + const { computer_id: _id, ...launch } = input; + const started = await steps.capture.start(launch, turn); if (!started.ok) return started; // A failed screenshot only leaves it out, but a close or abort meanwhile cancels the start: it saved nothing to report. const shot = await steps.computer .callTool("computer_screenshot", {}, turn) .catch(() => undefined); - if (turn.aborted) + if (turn.aborted) { + steps.capture.abandon(); return err( new AnalysisCancelledError("start_build_capture", { cause: turn.reason, }), ); + } return ok({ value: { computer_id: input.computer_id, @@ -131,10 +133,11 @@ export class DesktopBuildSteps { readonly computer_id: string; }, signal: AbortSignal, + progress?: ProgressReporter, ): Promise, AnalysisError>> { const { computer_id: id, ...request } = input; return this.#on(id, "check_build_task", signal, (steps, turn) => - steps.witness.check(request, turn), + steps.witness.check(request, turn, progress), ); } diff --git a/src/application/desktop/DesktopBuildSummary.ts b/src/application/desktop/DesktopBuildSummary.ts index a7c63a08c3..e0b83a202d 100644 --- a/src/application/desktop/DesktopBuildSummary.ts +++ b/src/application/desktop/DesktopBuildSummary.ts @@ -6,6 +6,8 @@ import { SEED_CHECKPOINT, } from "../../domain/desktop/buildCapture.js"; import { installsFromPackageFile } from "../../domain/desktop/appExportInstall.js"; +import { draftNoteSchema } from "../../domain/desktop/appTaskDraftNotes.js"; +import { draftValueSchema } from "../../domain/desktop/appTaskDraftValues.js"; import { desktopApplicationCaptureSchema } from "../../domain/desktop/desktopCapture.js"; import { jsonObjectSchema, @@ -15,9 +17,9 @@ import { import { err, ok, type Result } from "../../domain/result.js"; import type { ControlledComputer } from "../../desktop/BuildComputerController.js"; import { - adapterError, ENDS, readOutput, + reportedFailure, runRea, } from "./DesktopBuildCommands.js"; @@ -29,6 +31,8 @@ const draftResultSchema = z.looseObject({ file: z.string(), role: z.string(), change: jsonValueSchema, + values: z.array(draftValueSchema).default([]), + values_cut: z.boolean().default(false), }), ), task: z.looseObject({ @@ -36,6 +40,82 @@ const draftResultSchema = z.looseObject({ prompt: z.string(), checks: z.array(jsonObjectSchema), }), + guards: z.array(jsonObjectSchema).default([]), + notes: z.array(draftNoteSchema).default([]), +}); + +/** A drafted file's value as the stop returns it. */ +type DraftValue = { + readonly pointer: string; + readonly kind: string; + readonly before: JsonValue; + readonly after: JsonValue; + readonly note?: string; + readonly refers_to?: { + readonly attribute?: string; + readonly value: string; + readonly element: string; + }[]; +}; + +/** A draft's note as the stop returns it. */ +type DraftNote = { + readonly code: string; + readonly message: string; + readonly file?: string; + readonly files?: { readonly file: string; readonly role: string }[]; + readonly more?: number; + readonly elements?: string[]; + readonly pointers?: string[]; + readonly error?: string; +}; + +/** A value without the optional fields the draft left out. */ +const draftValue = ({ + pointer, + kind, + before, + after, + note, + refers_to, +}: z.infer): DraftValue => ({ + pointer, + kind, + before, + after, + ...(note === undefined ? {} : { note }), + ...(refers_to === undefined + ? {} + : { + refers_to: refers_to.map(({ attribute, value, element }) => ({ + ...(attribute === undefined ? {} : { attribute }), + value, + element, + })), + }), +}); + +/** A note without the optional fields the draft left out. */ +const draftNote = ({ + code, + message, + file, + files, + more, + elements, + pointers, + error, +}: z.infer): DraftNote => ({ + code, + message, + ...(file === undefined ? {} : { file }), + ...(files === undefined + ? {} + : { files: files.map(({ file: home, role }) => ({ file: home, role })) }), + ...(more === undefined ? {} : { more }), + ...(elements === undefined ? {} : { elements }), + ...(pointers === undefined ? {} : { pointers }), + ...(error === undefined ? {} : { error }), }); /** A stopped capture: its checkpoints, what it found of the app and the draft between seed and end. */ @@ -61,12 +141,19 @@ export type StoppedCapture = { readonly file: string; readonly role: string; readonly change: JsonValue; + /** The values the change added, changed and removed, at the pointers a check names them by. */ + readonly values: DraftValue[]; + readonly values_cut: boolean; }[]; readonly task: { readonly name: string; readonly prompt: string; readonly checks: Readonly>[]; }; + /** Checks that the rest stays as it was, outside the task until an author adds them. */ + readonly guards: Readonly>[]; + /** Why the draft has no files, or why a file's suggestions are thin. */ + readonly notes: DraftNote[]; } | null; readonly draft_refusal?: string; }; @@ -105,16 +192,20 @@ const draftTask = async ( draft: { seed_checkpoint: SEED_CHECKPOINT, end_checkpoint: END_CHECKPOINT, - files: draft.files.map(({ file, role, change }) => ({ + files: draft.files.map(({ file, role, change, values, values_cut }) => ({ file, role, change, + values: values.map(draftValue), + values_cut, })), task: { name: draft.task.name, prompt: draft.task.prompt, checks: draft.task.checks, }, + guards: draft.guards, + notes: draft.notes.map(draftNote), }, }; }; @@ -147,9 +238,7 @@ export const summarizeCapture = async ( signal, ); if (output.kind === "failure") - return err( - adapterError(STOP, `The capture failed: ${output.failure.message}`), - ); + return err(reportedFailure(STOP, output, "The capture failed: ")); const recorded = desktopApplicationCaptureSchema.parse(output.result); const checkpoints = new Set( recorded.state.checkpoints.map((checkpoint) => checkpoint.name), diff --git a/src/application/desktop/DesktopBuildWitness.test.ts b/src/application/desktop/DesktopBuildWitness.test.ts index 5e836e4ebf..911d7cfad1 100644 --- a/src/application/desktop/DesktopBuildWitness.test.ts +++ b/src/application/desktop/DesktopBuildWitness.test.ts @@ -7,6 +7,7 @@ import { AnalysisCancelledError, AnalysisInputError, } from "../../domain/analysisErrorCore.js"; +import { SETTLE_TIMING } from "../../domain/desktop/appSettle.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { BuildComputerActionLog } from "./DesktopBuildActionLog.js"; import { @@ -34,7 +35,12 @@ const computer = { } as WitnessedComputer; const witnessAt = async (bundle: string) => - new DesktopBuildWitness(computer, new BuildComputerActionLog(), bundle); + new DesktopBuildWitness( + computer, + new BuildComputerActionLog(), + SETTLE_TIMING, + bundle, + ); describe("bundle task reads", () => { it("returns a cancelled result when reading the bundle's task is aborted", async () => { diff --git a/src/application/desktop/DesktopBuildWitness.ts b/src/application/desktop/DesktopBuildWitness.ts index b77bfd0951..bd3aaabddd 100644 --- a/src/application/desktop/DesktopBuildWitness.ts +++ b/src/application/desktop/DesktopBuildWitness.ts @@ -12,6 +12,7 @@ import { type desktopTaskSchema, type DesktopTask, } from "../../domain/desktop/appTask.js"; +import { SETTLE_TIMING } from "../../domain/desktop/appSettle.js"; import { evaluateTaskChecks, type TaskCheckOutcome, @@ -19,6 +20,7 @@ import { import { renderTaskSteps, renderTaskVerifier, + sameTaskVerifier, taskVerifierArtifact, } from "../../domain/desktop/appTaskPython.js"; import { @@ -42,9 +44,14 @@ import type { import { AGENTENV_PROVIDER } from "../../desktop/AgentEnvComputerHelper.js"; import { readHostFile } from "../../desktop/HostFileRead.js"; import { isPortableBundleName } from "../environment/environmentBundle.js"; +import { + silentProgressReporter, + type ProgressReporter, +} from "../ProgressReporter.js"; import type { ReadonlyActionLog } from "./DesktopBuildActionLog.js"; import { errorText } from "./DesktopBuildCommands.js"; import { desktopOperationFailure } from "./DesktopCaptureRefusals.js"; +import { settledAppData, type BuildSettle } from "./DesktopBuildSettle.js"; import { writeWitnessRecord, type EndedWitness, @@ -70,12 +77,23 @@ type WitnessSummary = { readonly record: WitnessRecord | null; }; +/** A task's results on the app's views, whether its files had settled, and whether the bound cut a read. */ +type Graded = { + readonly results: TaskCheckOutcome[]; + readonly settled: boolean; + readonly read_cut: boolean; +}; + /** One check of the task against the app's live state, and the witness stage it filed, if any. */ export type CheckedTask = { readonly task: string; readonly passed: number; readonly failed: number; readonly results: TaskCheckOutcome[]; + /** Whether the app's files had held for the settle window when they were graded, rather than at its bound. */ + readonly settled: boolean; + /** Whether a read still in flight at the settle bound was cut, so the last completed read was graded. */ + readonly read_cut: boolean; readonly witness: WitnessSummary | null; }; @@ -128,7 +146,10 @@ const bundleTaskDifference = async ( !isDeepStrictEqual(listed, JSON.parse(renderTaskSteps(task, app))) ) return `the bundle's ${stepsPath} differs from this task's steps.`; - return verifier.toString("utf8") === renderTaskVerifier(task, app) + return sameTaskVerifier( + renderTaskVerifier(task, app), + verifier.toString("utf8"), + ) ? undefined : `the bundle's ${verifierPath} differs from this task's checks.`; }; @@ -144,6 +165,11 @@ interface OpenWitness extends EndedWitness { record?: WitnessRecord; /** A manifest whose segments still need to be fetched. */ pendingManifest?: RecordingManifest | undefined; + /** + * The action log as it stood when the record was first written. A retry + * uses this copy, so later activity cannot drop the witnessed actions. + */ + logged?: Pick; /** * A published record whose handles are still open. The error already carries * the record and the cleanup lease; every later close failure returns it. @@ -171,9 +197,16 @@ const summaryOf = ( }); /** The pass and fail counts of one set of results. */ -const counted = (task: DesktopTask, results: TaskCheckOutcome[]) => { +const counted = (task: DesktopTask, { results, settled, read_cut }: Graded) => { const passed = results.filter((result) => result.passed).length; - return { task: task.name, passed, failed: results.length - passed, results }; + return { + task: task.name, + passed, + failed: results.length - passed, + results, + settled, + read_cut, + }; }; /** @@ -210,6 +243,7 @@ export class DesktopBuildWitness { constructor( private readonly computer: WitnessedComputer, private readonly actions: ReadonlyActionLog, + private readonly settle: BuildSettle = SETTLE_TIMING, private readonly bundle?: string, ) {} @@ -220,6 +254,7 @@ export class DesktopBuildWitness { readonly directory?: string | undefined; }, signal: AbortSignal, + progress: ProgressReporter = silentProgressReporter, ): Promise> { const parsed = parseDesktopTask(input.task, isPortableBundleName); if (!parsed.ok) @@ -243,27 +278,38 @@ export class DesktopBuildWitness { ? undefined : await this.#admit(task, stage, input.directory, signal); if (admitted?.ok === false) return admitted; - const graded = await this.#grade(task, signal); + const graded = await this.#grade(task, signal, progress); if (!graded.ok) return graded; - const results = graded.value; - const checked = counted(task, results); + const checked = counted(task, graded.value); if (stage === undefined || admitted === undefined) return ok({ ...checked, witness: null }); - const filed = await this.#file(stage, admitted.value, results, signal); + const filed = await this.#file(stage, admitted.value, graded.value, signal); return filed.ok ? ok({ ...checked, witness: filed.value }) : filed; } - /** The task's checks against the app's live views, one result per check. */ + /** The task's checks against the app's views once its files settle, one result per check. */ async #grade( task: DesktopTask, signal: AbortSignal, - ): Promise> { + progress: ProgressReporter, + ): Promise> { try { - const views = appStateViews( - await this.computer.appData("data/get", signal), + const read = await settledAppData( + (turn) => this.computer.appData("data/get", turn), + this.settle, + signal, + (message) => + progress + .report({ phase: CHECK, completed: 0, total: null, message }) + .catch(() => undefined), ); + const views = appStateViews(read.parts); return views.ok - ? ok([...evaluateTaskChecks(task.checks, views.value)]) + ? ok({ + results: [...evaluateTaskChecks(task.checks, views.value)], + settled: read.settled, + read_cut: read.readCut, + }) : err( refused( views.error, @@ -290,7 +336,7 @@ export class DesktopBuildWitness { async #file( stage: WitnessStage, actionsBefore: number, - results: TaskCheckOutcome[], + { results, settled, read_cut }: Graded, signal: AbortSignal, ): Promise> { const witness = this.#witness; @@ -328,6 +374,8 @@ export class DesktopBuildWitness { checked_at: new Date().toISOString(), expectation: stageExpectation(stage), held, + settled, + read_cut, results, ...(unrestored.length === 0 ? {} @@ -361,10 +409,14 @@ export class DesktopBuildWitness { witness.state = "saved"; return ok(summaryOf(filed, null, witness.record)); } + witness.logged ??= { + entries: [...this.actions.entries], + dropped: this.actions.dropped, + }; const written = await writeWitnessRecord( this.computer, witness, - this.actions, + witness.logged, signal, witness.pendingManifest, ); @@ -410,7 +462,7 @@ export class DesktopBuildWitness { ), ); return ok({ - ...counted(task, filed.results), + ...counted(task, filed), witness: summaryOf(filed, null, witness.record), }); } @@ -436,7 +488,10 @@ export class DesktopBuildWitness { ); const saved = await this.#save(witness, filed, signal); return saved.ok - ? ok({ ...counted(task, filed.results), witness: saved.value }) + ? ok({ + ...counted(task, filed), + witness: saved.value, + }) : saved; } diff --git a/src/application/desktop/DesktopBuildWitnessRecord.ts b/src/application/desktop/DesktopBuildWitnessRecord.ts index 5f9c2e9ba3..7095c15756 100644 --- a/src/application/desktop/DesktopBuildWitnessRecord.ts +++ b/src/application/desktop/DesktopBuildWitnessRecord.ts @@ -213,6 +213,29 @@ const logLimitations = ( : []), ]; +/** Which stages graded files that had not settled by the bound, and which of them had a read cut there. */ +const unsettledLimitation = ( + stages: readonly WitnessStageRecord[], +): string[] => { + const named = (cut: boolean) => + stages.flatMap(({ stage, settled, read_cut }) => + settled || (cut && !read_cut) ? [] : [stage], + ); + const [unsettled, cut] = [named(false), named(true)]; + return [ + ...(unsettled.length === 0 + ? [] + : [ + `The app's files had not held for the settle window by its bound when these stages were checked, so each graded its last completed read, with settled false: ${unsettled.join(", ")}.`, + ]), + ...(cut.length === 0 + ? [] + : [ + `A read still running at the settle bound was cut, with read_cut true, in: ${cut.join(", ")}.`, + ]), + ]; +}; + /** How a witness write ended: saved, retryable, or published with cleanup still open. */ export type WitnessWrite = | { readonly kind: "saved"; readonly record: WitnessRecord } @@ -286,6 +309,7 @@ export const writeWitnessRecord = async ( ? [`The recording is missing: ${recording.error}`] : []), ...logLimitations(actions, dropped), + ...unsettledLimitation(witness.stages), ], }); const files = [ diff --git a/src/application/desktop/DesktopCaptureScope.test.ts b/src/application/desktop/DesktopCaptureScope.test.ts index 7218deb6da..9f71ab792a 100644 --- a/src/application/desktop/DesktopCaptureScope.test.ts +++ b/src/application/desktop/DesktopCaptureScope.test.ts @@ -1,4 +1,4 @@ -import { mkdir, realpath, writeFile } from "node:fs/promises"; +import { mkdir, realpath, symlink, writeFile } from "node:fs/promises"; import { dirname, join } from "node:path"; import { describe, expect, it } from "vitest"; @@ -107,6 +107,40 @@ describe("captureScope", () => { expect.objectContaining({ path: `${database}-journal`, kind: "created" }), ]); }); + + it("records each selected path as named beside the physical path it walks, so a link keeps what its name means", async () => { + const root = await realpath( + await createTestTempDirectory("desktop-scope-"), + ); + const home = join(root, "home/ada"); + const settings = join(home, "settings/MuseScore"); + const linked = join(home, ".config/MuseScore"); + const score = join(home, "Documents/song.mscz"); + await mkdir(settings, { recursive: true }); + await mkdir(dirname(linked)); + await symlink(settings, linked); + + const { scope, selectedPaths } = await captureScope( + { ...request([`${linked}/MuseScore3.ini`]), statePaths: [linked, score] }, + inventory(), + home, + join(root, "out"), + ); + + expect(selectedPaths).toEqual([ + { path: linked, physical: settings, kind: "directory" }, + { path: score, physical: score, kind: "file" }, + { + path: `${linked}/MuseScore3.ini`, + physical: `${settings}/MuseScore3.ini`, + kind: "file", + }, + ]); + expect(scope.contentRoots).toEqual( + expect.arrayContaining([settings, score]), + ); + expect(scope.contentRoots).not.toContain(linked); + }); }); describe("applicationWindowMatching", () => { diff --git a/src/application/desktop/DesktopCaptureScope.ts b/src/application/desktop/DesktopCaptureScope.ts index 9b441a44c6..e06b22f65a 100644 --- a/src/application/desktop/DesktopCaptureScope.ts +++ b/src/application/desktop/DesktopCaptureScope.ts @@ -1,5 +1,5 @@ import { lstat } from "node:fs/promises"; -import { basename, isAbsolute } from "node:path"; +import { basename, isAbsolute, resolve } from "node:path"; import type { DesktopApplicationInventory } from "../../domain/desktop/desktopCapture.js"; import { isWithinAny } from "../../domain/desktop/desktopStateDiff.js"; @@ -10,33 +10,35 @@ import type { CheckpointScope } from "./DesktopCheckpointRecorder.js"; const unique = (paths: readonly string[]) => [...new Set(paths)]; -/** - * The `-wal` and `-journal` paths beside each selected path that is not a - * directory: a database selected on its own is unreadable without them, and - * only a content root records whether they exist. - */ -const sidecarsOf = async (paths: readonly string[]) => { - const sidecars: string[] = []; - for (const path of paths) { - const directory = await lstat(path).then( - (stats) => stats.isDirectory(), - () => false, - ); - if (!directory) - sidecars.push(...SQLITE_SIDECARS.map(({ suffix }) => `${path}${suffix}`)); - } - return sidecars; -}; +/** Whether the walked path is a directory. A missing path is not one, matching the sidecar decision. */ +const pathKind = async (path: string): Promise<"file" | "directory"> => + (await lstat(path).then( + (stats) => stats.isDirectory(), + () => false, + )) + ? "directory" + : "file"; -/** The checkpoint scope, and absolute arguments too broad to be part of it. */ +/** A path the caller selected, absolute as named, and the physical path walked for it. */ +export interface SelectedPath { + readonly path: string; + readonly physical: string; + /** The walked path's kind, so a later export does not infer a directory from changes. */ + readonly kind: "file" | "directory"; +} + +/** The checkpoint scope, the selected paths in it, and absolute arguments too broad to be part of it. */ export interface CaptureScope { readonly scope: CheckpointScope; + /** `--state-path` values, then absolute arguments, in the scope. */ + readonly selectedPaths: readonly SelectedPath[]; /** `/` or a directory containing the home directory, named by `--argument`. */ readonly broadArguments: readonly string[]; } /** - * Paths in their physical spelling, so walks and exclusions agree. An + * Paths in their physical spelling, so walks and exclusions agree; a selected + * path keeps the spelling it was named by too, which a link would lose. An * absolute argument becomes a content root unless it is `/` or contains the * home directory, which would retain and walk whole trees. A selected path * that is not a directory brings its SQLite sidecars as content roots too. @@ -54,23 +56,50 @@ export const captureScope = async ( ): Promise => { const canonical = (paths: readonly string[]) => Promise.all(paths.map((path) => canonicalPath(path))); + const named = (paths: readonly string[]) => + Promise.all( + paths.map(async (path) => ({ + path: resolve(path), + physical: await canonicalPath(path), + })), + ); const stateDirectories = await canonical( inventory.state_directories.map((entry) => entry.path), ); - const argumentPaths = await canonical( + const argumentPaths = await named( request.arguments.filter((value) => isAbsolute(value)), ); const broadArguments = unique( - argumentPaths.filter((path) => isWithinAny(home, [path])), + argumentPaths + .map(({ physical }) => physical) + .filter((path) => isWithinAny(home, [path])), ); - const selected = [ - ...(await canonical(request.statePaths)), - ...argumentPaths.filter((path) => !broadArguments.includes(path)), - ]; + const selectedPaths = await Promise.all( + [ + ...new Map( + [ + ...(await named(request.statePaths)), + ...argumentPaths.filter( + ({ physical }) => !broadArguments.includes(physical), + ), + ].map((entry) => [entry.path, entry]), + ).values(), + ].map(async (entry) => ({ + ...entry, + kind: await pathKind(entry.physical), + })), + ); + const selected = unique(selectedPaths.map(({ physical }) => physical)); + // A database selected on its own is unreadable without its logs. A directory + // is not a database, so a neighbor named like a log stays a normal file. const contentRoots = unique([ ...stateDirectories, ...selected, - ...(await sidecarsOf(selected)), + ...selectedPaths.flatMap(({ physical, kind }) => + kind === "directory" + ? [] + : SQLITE_SIDECARS.map(({ suffix }) => `${physical}${suffix}`), + ), ]); return { scope: { @@ -83,6 +112,7 @@ export const captureScope = async ( excluded: unique([output, ...(await canonical(request.excludes))]), maxEntries: request.maxEntries, }, + selectedPaths, broadArguments, }; }; diff --git a/src/application/desktop/DesktopExportFiles.ts b/src/application/desktop/DesktopExportFiles.ts index 6479d34908..bd77a580de 100644 --- a/src/application/desktop/DesktopExportFiles.ts +++ b/src/application/desktop/DesktopExportFiles.ts @@ -7,18 +7,23 @@ import type { SeedScrubReport, } from "../../domain/desktop/appExport.js"; import type { TaskFileView } from "../../domain/desktop/appTaskChecks.js"; +import { + compressionOf, + openStateDocument, +} from "../../domain/desktop/gzipDocument.js"; import { identityLeftIn, type HostIdentity, } from "../../domain/desktop/hostIdentity.js"; import { scrubSeed } from "../../domain/desktop/seedScrub.js"; import { - detectStateFormat, isTextualFormat, textualView, } from "../../domain/desktop/stateFormats.js"; +import { err } from "../../domain/result.js"; import { refuse } from "./DesktopExportInstall.js"; import type { RetainedContent } from "./DesktopRetainedContent.js"; +import { readZipView } from "./DesktopZipView.js"; /** A file the export writes from bytes it made. */ export interface GeneratedFile { @@ -90,8 +95,8 @@ const retainedBytes = async ( }; /** - * The view of each text seed the scrub rewrites, as the bundle ships it, for - * a task's checks on the seed: the computer starts from these bytes. + * The view of each rewritten seed, including a formerly unreadable gzip + * seed, as the bundle ships it: the computer starts from these bytes. */ export const shippedSeedViews = async ( content: RetainedContent, @@ -102,20 +107,31 @@ export const shippedSeedViews = async ( const views = new Map(); for (const copy of copies) { const seed = copy.seed; - if (seed === undefined || !isTextualFormat(seed.format)) continue; + if ( + seed === undefined || + (seed.format !== null && !isTextualFormat(seed.format)) + ) + continue; const bytes = await retainedBytes(content, copy, signal); const shipped = scrubSeed(bytes, seed, identity).bytes; if (shipped === undefined) continue; - const format = detectStateFormat(seed.home, shipped); - const view = isTextualFormat(format) - ? textualView(format, shipped) - : undefined; - views.set( - seed.home, - view === undefined || view.ok - ? { format, view: view?.value ?? null } - : { format, view: null, error: view.error }, - ); + const document = openStateDocument(seed.home, shipped); + const { format } = document; + const view = + "error" in document + ? err(document.error) + : format === "zip" + ? await readZipView(document.bytes, signal) + : isTextualFormat(format) + ? textualView(format, document.bytes) + : undefined; + views.set(seed.home, { + format, + ...compressionOf(document), + ...(view === undefined || view.ok + ? { view: view?.value ?? null } + : { view: null, error: view.error }), + }); } return views; }; @@ -123,8 +139,10 @@ export const shippedSeedViews = async ( /** * Write the generated files, then each retained file as it is read, so one at * a time is in memory: a seed whose text names the capture host becomes a - * generated file with its own digest. The README follows, listing them. Every - * file is listed as unscrubbed whose path or content still holds the identity. + * generated file with its own digest, and a gzip seed whose document did not + * change is listed apart as recompressed. The README follows, listing them. + * Every file is listed as unscrubbed whose path or content still holds the + * identity. */ export const writeFiles = async ( tree: SafeOutputTree, @@ -133,6 +151,7 @@ export const writeFiles = async ( signal: AbortSignal | undefined, ): Promise => { const scrubbed: string[] = []; + const recompressed: string[] = []; const unscrubbed: string[] = []; const write = async ( path: string, @@ -173,10 +192,12 @@ export const writeFiles = async ( if (scrub?.bytes === undefined) await write(copy.path, bytes, { sha256: copy.digest, holdsIdentity }); else { - scrubbed.push(copy.path); + (scrub.recompressed === true ? recompressed : scrubbed).push(copy.path); await write(copy.path, scrub.bytes, { holdsIdentity }); } } - await generated(bundle.readme({ scrubbed, unscrubbed: [...unscrubbed] })); - return { scrubbed, unscrubbed }; + await generated( + bundle.readme({ scrubbed, recompressed, unscrubbed: [...unscrubbed] }), + ); + return { scrubbed, recompressed, unscrubbed }; }; diff --git a/src/application/desktop/DesktopExportLaunch.test.ts b/src/application/desktop/DesktopExportLaunch.test.ts new file mode 100644 index 0000000000..b332ae35d8 --- /dev/null +++ b/src/application/desktop/DesktopExportLaunch.test.ts @@ -0,0 +1,213 @@ +import { describe, expect, it } from "vitest"; + +import type { DesktopApplicationCapture } from "../../domain/desktop/desktopCapture.js"; +import type { DesktopApplicationState } from "../../domain/desktop/desktopState.js"; +import { planLaunch } from "./DesktopExportLaunch.js"; + +const HOME = "/home/ada"; +type StatePath = DesktopApplicationState["paths"][number]; + +const file = (physical: string, role: StatePath["role"]): StatePath => + ({ path: physical, role }) as StatePath; + +describe("planLaunch captured paths", () => { + it.each([ + ["/mnt/data/books.db", "/mnt/data/books.db"], + ["file:///mnt/data/books.db?mode=edit#page", "/mnt/data/books.db"], + ])("retargets selected outside-home argument %s", (argument, physical) => { + const capture = { + host: { home: HOME }, + application: { + argv: ["editor", argument], + working_directory: HOME, + desktop_entry: { exec: "editor %U" }, + state_directories: [], + }, + state: { + state_directories: [], + selected_paths: [ + { path: `${HOME}/Documents/books.db`, physical, kind: "file" }, + { + path: `${HOME}/.config/vendor`, + physical: `${HOME}/Documents`, + kind: "directory", + }, + ], + }, + } as unknown as DesktopApplicationCapture; + const files = new Map([["Documents/books.db", file(physical, "document")]]); + const seeds = new Map([ + ["Documents/books.db", { sha256: "a".repeat(64), format: "text" }], + ]); + const launch = planLaunch(capture, false, seeds, files); + expect(launch.opens).toEqual({ kind: "captured" }); + expect(launch.command.join(" ")).toContain("Documents/books.db"); + expect(launch.command.join(" ")).not.toContain("/mnt/data/books.db"); + expect(launch.command.join(" ")).not.toContain(".config/vendor/books.db"); + if (argument.startsWith("file:")) + expect(launch.command.join(" ")).toContain("?mode=edit#page"); + }); + + it("leaves an unrelated outside-home argument untouched", () => { + const capture = { + host: { home: HOME }, + application: { + argv: ["editor", "/mnt/other/unselected.db"], + working_directory: HOME, + desktop_entry: { exec: "editor %U" }, + state_directories: [], + }, + state: { + state_directories: [], + selected_paths: [ + { + path: `${HOME}/Documents/books.db`, + physical: "/mnt/data/books.db", + kind: "file", + }, + ], + }, + } as unknown as DesktopApplicationCapture; + const files = new Map([ + ["Documents/books.db", file("/mnt/data/books.db", "document")], + ]); + const command = planLaunch(capture, false, new Map(), files).command.join( + " ", + ); + expect(command).toContain("/mnt/other/unselected.db"); + }); +}); + +describe("planLaunch URL and argument spelling", () => { + it("keeps an encoded filename when a file URL has repeated separators", () => { + const physical = `${HOME}/data/A budget.svg`; + const capture = { + host: { home: HOME }, + application: { + argv: [ + "editor", + `file://localhost${HOME}//data//%41%20budget.svg?mode=edit#page`, + ], + working_directory: HOME, + desktop_entry: { exec: "editor %U" }, + state_directories: [], + }, + state: { + state_directories: [], + selected_paths: [{ path: `${HOME}/Documents/A budget.svg`, physical }], + }, + } as unknown as DesktopApplicationCapture; + const files = new Map([ + ["Documents/A budget.svg", file(physical, "document")], + ]); + const command = planLaunch(capture, false, new Map(), files).command.join( + " ", + ); + expect(command).toContain("/Documents/%41%20budget.svg?mode=edit#page"); + }); + + it("opens the selected document spelling when its captured file URL names the physical path", () => { + const physical = `${HOME}/data/A budget.svg`; + const capture = { + host: { home: HOME }, + application: { + argv: [ + "editor", + `file://localhost${HOME}/data/%41%20budget.svg?mode=edit#page`, + ], + working_directory: HOME, + desktop_entry: { exec: "editor %U" }, + state_directories: [], + }, + state: { + state_directories: [], + selected_paths: [{ path: `${HOME}/Documents/A budget.svg`, physical }], + }, + } as unknown as DesktopApplicationCapture; + const files = new Map([ + ["Documents/A budget.svg", file(physical, "document")], + ]); + const command = planLaunch(capture, false, new Map(), files).command.join( + " ", + ); + expect(command).toContain("file://localhost"); + expect(command).toContain("/Documents/%41%20budget.svg?mode=edit#page"); + expect(command).not.toContain("/data/%41%20budget.svg"); + }); + + it("rewrites a nested settings folder and leaves a numeric option", () => { + const profile = `${HOME}/settings/geany/profiles/work/geany.conf`; + const files = new Map([ + [ + ".config/geany/profiles/work/geany.conf", + file(profile, "application_state"), + ], + [ + "Documents/poster.svg", + file(`${HOME}/Documents/poster.svg`, "document"), + ], + ]); + const capture = { + host: { home: HOME }, + application: { + argv: ["geany", "--config=settings/geany/profiles/work", "--line=10"], + working_directory: HOME, + desktop_entry: { exec: "geany" }, + state_directories: [{ path: `${HOME}/.config/geany` }], + }, + state: { + state_directories: [`${HOME}/settings/geany`], + selected_paths: [ + { + path: `${HOME}/.config/geany`, + physical: `${HOME}/settings/geany`, + }, + ], + }, + } as unknown as DesktopApplicationCapture; + + const command = planLaunch(capture, false, new Map(), files).command.join( + " ", + ); + + expect(command).toContain(`"$HOME"/'.config/geany/profiles/work'`); + expect(command).toContain("'--line=10'"); + expect(command).not.toContain("settings/geany"); + }); + + it("keeps the program an env prefix runs when a captured file sits in a folder of that name", () => { + const config = `${HOME}/work/geany/config.ini`; + const files = new Map([ + [".config/vendor/geany/config.ini", file(config, "application_state")], + ]); + const capture = { + host: { home: HOME }, + application: { + argv: ["env", "GDK_BACKEND=x11", "geany", "note.txt"], + working_directory: `${HOME}/work`, + desktop_entry: { exec: "geany %f" }, + state_directories: [{ path: `${HOME}/.config/vendor` }], + }, + state: { + state_directories: [`${HOME}/work`], + selected_paths: [ + { path: `${HOME}/.config/vendor`, physical: `${HOME}/work` }, + ], + }, + } as unknown as DesktopApplicationCapture; + const note = file(`${HOME}/work/note.txt`, "document"); + files.set("work/note.txt", note); + + const command = planLaunch( + capture, + false, + new Map([["work/note.txt", { sha256: "a".repeat(64), format: "text" }]]), + files, + ).command.join(" "); + + expect(command).toContain("'GDK_BACKEND=x11'"); + expect(command).toContain("'geany'"); + expect(command).toContain(`"$HOME"/'.config/vendor/note.txt'`); + expect(command).not.toContain(".config/vendor/geany"); + }); +}); diff --git a/src/application/desktop/DesktopExportLaunch.ts b/src/application/desktop/DesktopExportLaunch.ts index 84b40ec2f9..92f069a139 100644 --- a/src/application/desktop/DesktopExportLaunch.ts +++ b/src/application/desktop/DesktopExportLaunch.ts @@ -1,4 +1,5 @@ import { + capturedHomePath, launchCommand, type DesktopAppExportPlan, } from "../../domain/desktop/appExport.js"; @@ -15,51 +16,105 @@ import type { SeedVersion } from "./DesktopExportFiles.js"; type StatePath = DesktopApplicationState["paths"][number]; -/** The owned files present at the seed with their digests, and those read as text. */ +/** + * The owned files present at the seed with their digests, and those read as + * uncompressed text, which boot respells as UTF-8, but for the lock files + * `isLeftOutLock` names, which the app holds while it runs: a copied one makes + * the app find its profile or document in use, so those are left out. + */ export const seedVersions = ( files: ReadonlyMap, paths: readonly string[], seed: number, + isLeftOutLock: (home: string) => boolean, ): { readonly seeds: ReadonlyMap; readonly textSeeds: readonly string[]; + readonly locks: readonly string[]; } => { const seeds = new Map(); const textSeeds: string[] = []; + const locks: string[] = []; for (const [home, path] of files) { const version = versionAt(path, seed); if (!isWithinAny(home, paths) || version?.present !== true) continue; - if (version.sha256 === null) continue; - seeds.set(home, { sha256: version.sha256, format: version.format }); - if (isTextualFormat(version.format)) textSeeds.push(home); + if (isLeftOutLock(home)) locks.push(home); + else if (version.sha256 !== null) { + seeds.set(home, { sha256: version.sha256, format: version.format }); + if (isTextualFormat(version.format) && version.compression === undefined) + textSeeds.push(home); + } } - return { seeds, textSeeds: textSeeds.sort() }; + return { seeds, textSeeds: textSeeds.sort(), locks: locks.sort() }; +}; + +/** Home paths of captured files and of settings directories, in either spelling. */ +const capturedLaunchPaths = ( + capture: DesktopApplicationCapture, + files: ReadonlyMap, + homeOf: (path: string) => string | null, +): ((path: string) => boolean) => { + const { home } = capture.host; + const captured = new Set(files.keys()); + const remember = (absolute: string) => { + if (!absolute.startsWith(`${home}/`)) return; + const relative = absolute.slice(home.length + 1); + captured.add(relative); + const spelled = homeOf(absolute); + if (spelled !== null) captured.add(spelled); + }; + for (const entry of capture.application.state_directories) + remember(entry.path); + for (const directory of capture.state.state_directories ?? []) + remember(directory); + for (const selected of capture.state.selected_paths ?? []) { + remember(selected.path); + remember(selected.physical); + } + for (const file of files.values()) remember(file.path); + const known = [...captured]; + // A settings folder between a selected root and a captured file is a path + // too. An option value that merely resolves under the home is not. + return (path) => + captured.has(path) || known.some((file) => isWithinAny(file, [path])); }; /** The launch command, opening the seed's one document when the app takes files. */ export const planLaunch = ( - { application, host }: DesktopApplicationCapture, + capture: DesktopApplicationCapture, electron: boolean, seeds: ReadonlyMap, files: ReadonlyMap, ): Pick => { + const { application, host } = capture; const isDocument = (home: string) => files.get(home)?.role === "document"; + const homeOf = capturedHomePath(capture); + const capturedDirectory = application.working_directory; + const workingDirectory = (() => { + if (!capturedDirectory.startsWith(`${host.home}/`)) + return capturedDirectory; + const relative = capturedDirectory.slice(host.home.length + 1); + const seed = homeOf(capturedDirectory) ?? relative; + return seed === relative ? capturedDirectory : `${host.home}/${seed}`; + })(); const launch = openSeedDocument( electronLaunchArguments(application.argv, electron), { home: host.home, - workingDirectory: application.working_directory, + // Resolve captured relative arguments against the directory the app used, + // then place the process in the seeded spelling of that directory. + workingDirectory: capturedDirectory, + launchDirectory: workingDirectory, exec: application.desktop_entry?.exec ?? null, documents: [...seeds.keys()].filter(isDocument).sort(), isDocument, + isCaptured: capturedLaunchPaths(capture, files, homeOf), + spell: (path) => homeOf(`${host.home}/${path}`) ?? path, + mapOutside: homeOf, }, ); return { - command: launchCommand( - launch.argv, - host.home, - application.working_directory, - ), + command: launchCommand(launch.argv, host.home, workingDirectory), opens: launch.opens, }; }; diff --git a/src/application/desktop/DesktopGzipDocument.test.ts b/src/application/desktop/DesktopGzipDocument.test.ts new file mode 100644 index 0000000000..d8f9a432fd --- /dev/null +++ b/src/application/desktop/DesktopGzipDocument.test.ts @@ -0,0 +1,291 @@ +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { gunzipSync, gzipSync } from "node:zlib"; + +import { describe, expect, it } from "vitest"; + +import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; +import type { DesktopApplicationCapture } from "../../domain/desktop/desktopCapture.js"; +import { desktopApplicationStateSchema } from "../../domain/desktop/desktopState.js"; +import { openStateDocument } from "../../domain/desktop/gzipDocument.js"; +import { exportDesktopApplication } from "./DesktopApplicationExport.js"; +import { inspectDesktopApplicationState } from "./DesktopApplicationState.js"; +import { draftDesktopApplicationTask } from "./DesktopTaskDraft.js"; +import { + HOME, + capture, + change, + readers, + retained, + writeCaptureFolder, +} from "./desktopCapture.fixture.js"; + +const BUDGET = "Documents/Budget.gnumeric"; +const HOSTNAME = "e81232be457d"; +const SEED = "envs/notes/seed/Documents/Budget.gnumeric"; +const CELLS = + "/root/children/[name=gnm:Sheets]/children/[name=gnm:Sheet]/children/[name=gnm:Cells]/children"; + +/** A Gnumeric workbook's XML with its rent, author and any comment as given. */ +const workbook = ( + rent: string, + author: string, + comment = "", +) => ` + + author${author} + ${comment} + Budget + Rent + ${rent} + + +`; + +/** + * The notes capture of a host with the workbook, named in a window title, + * saved before and after the user changed the rent and set the author to the + * value the scrub writes, each save made by `save`, with any other files the + * app kept unchanged by their home path. + */ +const budgetCapture = ( + save: (text: string) => Buffer, + comment = "", + kept: Readonly> = {}, +) => { + const before = save(workbook("1200", `ada@${HOSTNAME}`, comment)); + const after = save(workbook("1350", "user@localhost", comment)); + const base = capture(); + const [first, second] = base.state.checkpoints; + if (first === undefined || second === undefined) throw new Error("fixture"); + const path = `${HOME}/${BUDGET}`; + const captured: DesktopApplicationCapture = { + ...base, + host: { ...base.host, hostname: HOSTNAME, user: "ada" }, + session: { + ...base.session, + windows: base.session.windows.map((window) => ({ + ...window, + titles: [...window.titles, "Budget.gnumeric - Notes"], + })), + }, + state: { + ...base.state, + checkpoints: [ + { + ...first, + retained: [ + ...first.retained, + retained(path, before), + ...Object.entries(kept).map(([home, bytes]) => + retained(`${HOME}/${home}`, bytes), + ), + ], + }, + { + ...second, + changes: [...second.changes, change(path, { before, after })], + retained: [...second.retained, retained(path, after)], + }, + ], + }, + }; + return writeCaptureFolder(captured, undefined, [ + before, + after, + ...Object.values(kept), + ]); +}; + +const gzipped = (text: string) => gzipSync(text); +const plain = (text: string) => Buffer.from(text); + +/** The workbook's path in the capture's state view. */ +const budgetState = async (folder: string) => { + const evidence = await inspectDesktopApplicationState(folder, readers); + if (!("normalized_result" in evidence)) throw evidence; + return desktopApplicationStateSchema + .parse(evidence.normalized_result) + .paths.find((path) => path.home_path === BUDGET); +}; + +const draftOf = async (folder: string) => { + const drafted = await draftDesktopApplicationTask(folder, readers); + if (drafted instanceof Error) throw drafted; + return drafted.normalized_result; +}; + +/** Export the capture with a task that checks the rent and the author the scrub writes. */ +const exportBudget = async (folder: string) => { + const scratch = await createTestTempDirectory("desktop-gzip-task-"); + const task = join(scratch, "task.json"); + const directory = join(scratch, "bundle"); + await writeFile( + task, + JSON.stringify({ + name: "rent", + prompt: "Set the rent to 1350 in my budget and save it.", + checks: [ + { + id: "rent", + kind: "value_equals", + file: BUDGET, + pointer: `${CELLS}/[name=gnm:Cell][@Row=0][@Col=1]/children/0`, + value: "1350", + }, + { + id: "author", + kind: "text_equals", + file: BUDGET, + pointer: + "/root/children/[name=gnm:Summary]/children/[name=gnm:Item]/children/[name=gnm:val-string]", + value: "user@localhost", + }, + ], + }), + ); + + const result = await exportDesktopApplication( + { capture: folder, directory, name: undefined, tasks: [task] }, + readers, + ); + return { result, directory }; +}; + +describe("documents saved gzip-compressed", () => { + it("reads each version as the XML inside, the view the same workbook has uncompressed, and records the compression", async () => { + const compressed = await budgetState(await budgetCapture(gzipped)); + const uncompressed = await budgetState(await budgetCapture(plain)); + + expect(compressed).toMatchObject({ + role: "document", + format: "xml", + versions: [ + { format: "xml", compression: "gzip", view_error: null }, + { format: "xml", compression: "gzip", view_error: null }, + ], + changes: [{ kind: "values", changed_count: 2 }], + }); + expect(uncompressed?.versions[0]?.compression).toBeUndefined(); + expect(compressed?.versions.map(({ view }) => view)).toEqual( + uncompressed?.versions.map(({ view }) => view), + ); + expect(compressed?.changes).toEqual(uncompressed?.changes); + }); + + it("drafts the same files, values, checks, guards and notes as the same workbook uncompressed", async () => { + const compressed = await draftOf(await budgetCapture(gzipped)); + + expect(compressed).toEqual(await draftOf(await budgetCapture(plain))); + expect( + (compressed as { task: { checks: unknown[] } }).task.checks, + ).toContainEqual( + expect.objectContaining({ + file: BUDGET, + pointer: `${CELLS}/[name=gnm:Cell][@Col=1]`, + value: "1350", + }), + ); + }); + + it("ships the seed scrubbed inside and still gzip, copied as it is at boot, and checks the seed as shipped", async () => { + const { result, directory } = await exportBudget( + await budgetCapture(gzipped), + ); + + if (result instanceof Error) throw result; + expect(result.normalized_result).toMatchObject({ + scrubbed_files: [SEED], + unscrubbed_files: [], + task: { failing_on_seed: ["rent"] }, + }); + const shipped = await readFile(join(directory, SEED)); + expect([...shipped.subarray(0, 4)]).toEqual([0x1f, 0x8b, 8, 0]); + expect(gunzipSync(shipped).toString("utf8")).toBe( + workbook("1200", "user@localhost"), + ); + const module = await readFile( + join(directory, "envs/notes/notes_app.py"), + "utf8", + ); + // Boot respells text seeds as UTF-8, so the gzip workbook is not one of them. + expect(module).toContain( + 'TEXT_SEEDS = frozenset([".config/notes/settings.ini"])', + ); + }); + + it("lists gzip seeds recompressed only to clear their headers apart from the document whose content was scrubbed", async () => { + const templates = ["blank", "invoice", "loan"].map( + (name) => `.local/share/notes/templates/${name}.xml.gz`, + ); + const kept = Object.fromEntries( + templates.map((home) => [home, gzipSync(`