From 970969cb53f788976e64eae387076fcea104b603 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Wed, 4 Nov 2026 23:14:04 -0800 Subject: [PATCH] feat(desktop): capture and export apps installed from a downloaded .deb (#8954) * feat(desktop): capture a Linux desktop application in use Add `capture-desktop-application`, a CLI-only, Linux and X11 capture that launches an installed desktop application, watches it while it is used and writes capture Evidence plus SHA-256 addressed content: - inventory: desktop entry, launch command, executable, package and its apt sources (or snap, flatpak, AppImage), Electron markers and candidate state directories; - session: the launched process tree, client windows from xprop with the first application window time and a window match, and files held open; - state: a stat walk of the home directory before launch, at checkpoints read from stdin, at stop and after exit, with created, modified, deleted and retyped paths and the content of changed files and state directories. It refuses non-Linux hosts, missing DISPLAY or xprop, a non-empty output folder and an application whose window is already open, and signals only processes it started. Co-Authored-By: Claude Opus 5.5 * fix(desktop): own launched processes, confirm walk coverage and keep cancelled captures - Launch through the owned provider path with a run token; release with SIGTERM, then the token-verified group cleanup, after every session. Processes without the token are never signalled; each records whether it carries it, and the session records whether cleanup was verified. - Report creations and deletions only where a walk listed the nearest existing directory; count the rest as unconfirmed. - Retain a file only when it is still the walked version, through owned no-follow reads; charge the budget only for new content; create files exclusively in a private output folder owned by the caller. - Walk content roots outside the home directory, compare paths in their physical spelling, and run executables from the caller's directory. - Refuse when the window list cannot be read, re-evaluate window ownership each sample, handle one queued request per sample with a bounded stdin queue, and write the capture with end reason `cancelled` on interruption. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep owned cleanup failures and fresh digests - Allow a capture cancelled before any requested checkpoint to hold only the baseline. - Retry an incomplete stop once; if cleanup is still unverified, write the capture and fail with the run token and process group as cleanup resources. - Classify path changes after a walk (ENOTDIR and other selected-file errors) as changed_since_walk instead of failing the capture. - Treat a non-directory ancestor as proof that nothing existed beneath it. - Report a before digest only for the file version it was read from. - Derive stderr_truncated from stderr's own byte counts. Co-Authored-By: Claude Opus 5.5 * wip(desktop): state view over a capture folder Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep failed file-handle closes owned for retry Content reads pass retainFailedClose, so a handle whose close fails stays owned by the content store; the capture retries those closes after it is written and fails with the output folder as the cleanup resource if one still cannot be closed. Co-Authored-By: Claude Opus 5.5 * feat(desktop): read a captured application's state Add inspect-desktop-application-state, a CLI command that reads a capture folder into the application's state: every retained or changed file with its role and reasons, a typed JSON view of each version (SQLite through the shared worker, dconf through a private dconf dump, JSON, XML, key files, text) and the kinds of change between versions (lines, values, keys, rows). Co-Authored-By: Claude Opus 5.5 * fix(desktop): report process and handle cleanup failures together When both the launched processes and a retained file's handle remain unresolved after the capture is written, one cleanup error now lists both reasons and both resource sets. Co-Authored-By: Claude Opus 5.5 * feat(desktop): export a captured application as an agentenv computer app Add export-desktop-application, which writes an agentenv bundle with one computer_app environment from a capture folder and its state view: the apt package, a boot launch matched on the captured window class with a window ready check, owned paths from the state roles, a seed of the baseline files, data hooks that call agentenv's add/get/reset helpers with a typed JSON view on get, a skill, a README and a deploy smoke task. The desktop CLI commands now load their services lazily. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep state views faithful to the capture - Read key files, XML attributes, dconf groups and SQLite tables into dictionaries without inherited properties, so captured keys such as __proto__ stay data. - Version databases with their write-ahead log as of each checkpoint, and stage only digest-verified bytes for the SQLite and dconf readers. - Detect the format per version, report format changes, keep JSON null and empty containers as values, keep XML text and preserved whitespace, and mark truncated views (including XML depth) and comparisons of them. - Number versions and changes by checkpoint index, decide baseline presence from the captured type, and keep tagged SQLite integers, text bytes and blobs distinct. - Carry the capture's limitations and omitted content forward, match state directories by their home path, index changes once per capture, keep failed handle closes owned, and finish the CLI through the shared owner. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep coverage, logs and handles honest in the state view - Read coverage flags only from formats whose readers write them, so JSON fields named truncated stay data. - Fail a SQLite version whose write-ahead log existed but was not retained or does not match its digest, instead of reading the database alone. - Close retained content on every exit, including cancellation, and report a close that still fails together with the primary failure. - Finish the CLI with the canonical operation name. Co-Authored-By: Claude Opus 5.5 * fix(desktop): export only what the computer can install and seed - Refuse packages not from the Ubuntu 24.04 archive, since the computer installs by name from that archive. - Start the launch command in the captured working directory. - Respell the home path only in files the state view read as text. - Report files that were not retained instead of dropping them silently. - Load captured dconf keys for apps that own no files. - Keep home-level files as owned paths; rename reserved Python class names. - Mark truncated SQLite and text views in the class's get view, and read file types the way the state view does. - Release the output tree through a resource scope, and read seeds through an owner closed on every exit. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep staged folders owned, symlinked state roots and XML values - Stage database bytes in a private runtime root released through a resource scope, so a failed removal keeps its owner and is reported with the read's outcome. - Report cleanup facts and owners from desktop failures instead of wrapping them as a plain adapter error. - Record the state directories as walked, in their physical spelling, and match roles against both spellings, so files in a symlinked state directory are not mistaken for selected documents. - Keep whitespace-only XML leaf values; drop only formatting between elements. Co-Authored-By: Claude Opus 5.5 * fix(desktop): read XML in the exported app the way the state view does Keep whitespace-only leaf values and xml:space="preserve" text in the generated get view; drop only formatting between elements. Co-Authored-By: Claude Opus 5.5 * fix(desktop): release retained handles with their owners, read older captures - Release retained-content handles whose close failed through a resource scope on every exit; a release that fails again reports the exact handle owners. A read that finished keeps its bytes when only its close failed. - Accept captures written before state directories were recorded, and say in the state view's limitations that symlinked state directories may be misclassed for them. - Compute each XML element's whitespace context once instead of scanning siblings for every text run. Co-Authored-By: Claude Opus 5.5 * fix(desktop): boot without the working folder, bound XML, accept mirrors - Try the captured working directory only when an argument is relative, and start the app even when that folder is missing. - Cut the generated XML view at REA's node and depth limits, and keep one unreadable file from failing data/get. - Accept Ubuntu 24.04 mirrors, not only the canonical hosts; PPAs and other repositories are still refused. - Read seeds through the retained-content owner's new signature. Co-Authored-By: Claude Opus 5.5 * fix(desktop): state view review round - Stage a retained -journal beside its database so the SQLite worker accepts an inactive journal and refuses a hot one; a -wal or -journal that existed but was not retained (including one omitted at the baseline) makes the version unreadable. - Keep tables that read when another fails, and let cancellation and unfinished cleanup propagate instead of becoming view errors. - Read text as a key file only when every line parses as one. - Rank a file named in a window title or selected by the caller as a document even when the application holds it open; match titles on file-name boundaries; leave the display name out of dconf matching. - Bound JSON and key file views (4 MiB, 500 levels, exact numbers) and report too-large, missing and altered retained content separately. - Add no version or change while a database stays absent. - Read staged databases in place instead of copying them again. - Give Evidence the same limitations as the result. Co-Authored-By: Claude Opus 5.5 * fix(desktop): export review round - Load captured dconf keys without D-Bus: install dconf-cli, compile the existing and captured key files into the user database, and warn on failure instead of stopping the server. - Start relative-argument launches in the captured working directory, the home included, since the framework launches from the app folder; respell the home itself and option values under it. - Read views in a thread, skip files over 64 MiB, align the generated readers with the state view's, and close SQLite connections. - Never own the home or a shared base folder; map a symlinked state directory's files back to their home spelling. - Report omitted files and unretained opened folders. - Name the module so it cannot shadow another, and keep the class off the module's own names; derive names the bundle check accepts. - Write seeds as they are read, and remove an unpublished bundle on failure. Co-Authored-By: Claude Opus 5.5 * fix(desktop): capture review round - Ask the launch process group and token carriers to exit before killing them, on errors too; report incomplete cleanup with its facts. - Count descendants of the launch as started by the capture even when their environment cannot be read; treat a reused pid as a new process. - Follow symlinked desktop entries; name a flatpak's state after its application id; match flatpak windows by name only. - Stop on end of input; refuse empty, duplicate and reserved checkpoint names; require whitespace after the checkpoint keyword. - Stay on each walk root's device, and never walk or retain `/` or a directory containing the home; say so in the limitations. - Refuse a window manager that does not publish _NET_CLIENT_LIST. - Remove retained content when a capture fails before it is written. - Skip reading a file larger than the remaining content budget. - Record apt's archive origin for each package source. Co-Authored-By: Claude Opus 5.5 * fix(desktop): read an unset client list as no windows when supported xfwm4 leaves _NET_CLIENT_LIST unset while it manages no windows, so the capture refused an ordinary desktop with nothing open. Read _NET_SUPPORTED with the list: an unset list means no windows when the window manager supports it, and the capture is refused only when it does not. `xprop`'s "no such atom" output now reads as unset too. Co-Authored-By: Claude Opus 5.5 * fix(desktop): accept packages by apt's archive origin Accept a package source when apt reports its origin as Ubuntu for a noble suite, so mirrors count and PPAs or other repositories that merely have "ubuntu" in their URL do not. Captures made before origins were recorded fall back to Canonical's own archive hosts. Seed reads report why retained content could not be used. Co-Authored-By: Claude Opus 5.5 * feat(desktop): capture and export apps from vendor apt repositories A closed-source app installed from its vendor's apt repository, such as Typora, can now be captured and exported as a computer app. - The capture records each non-Ubuntu repository listing the installed version (uri, suite, components from apt's sources files, one-line or deb822) and the key that signs it: the signed-by key, or the trusted key that gpgv shows verifies the repository's signed index. The key is retained with the capture. - The capture records the packages owning the libraries the executable loads (DT_NEEDED with RUNPATH and the linker cache, through REA's ELF readers), since a vendor package may not depend on them. - The export declares a verified vendor repository as an AptSource with its key, installs a vendor package's library packages, and adds --no-sandbox to an Electron app's launch, since a container denies Chromium's sandbox. Co-Authored-By: Claude Opus 5.5 * fix(desktop): report a locked database at once in data/get A database the running app holds locked made each data/get wait sqlite's five-second busy timeout before reporting it. Record the Typora run in an agentenv computer in the verification. Co-Authored-By: Claude Opus 5.5 * perf(desktop): detect key files without splitting the whole text Read lines one at a time and stop at the first that a key file view would drop, instead of splitting a file of up to 64 MiB into arrays before its format is known. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep symlinked homes and state folders under the home in exports - Respell a path under its state folder's logical spelling only when that spelling is under the captured home, so a symlinked HOME keeps its files; pair walked and logical state folders only when the capture lists the same number, and match the innermost folder first. - When two files map to one home path, keep the one present before the capture and name the other in the limitations. - Read key files in the generated data/get with the state view's rule, checked against it by running the helper under python3. Co-Authored-By: Claude Opus 5.5 * fix(desktop): end captures on zombies and keep cleanup owners - A zombie or dead process no longer keeps the session open. - Remove only the partial capture file this run created. - Keep the exact owners of handles and processes whose cleanup failed on the returned error, so a retry can release them; test the published capture's cleanup-failure branch with mixed outcomes and retries. - Release the application once even when the exit checkpoint fails. - A cancellation before the capture is written removes what it retained. Co-Authored-By: Claude Opus 5.5 * fix(desktop): vendor package review round - Match apt sources through `apt-get indextargets`, which covers flat `./` and `/` suites and gives each release file and sources entry. - Accept a key whose signature verifies even when the release carries another signature without a key here; keep only the verifying key from a keyring holding several (OpenPGP packet splitter). - Install the package from its repository's release (`name/`), declare one source per URI and suite, support only the capture host's architecture, and refuse Ubuntu Pro repositories. - List only library packages the computer can install; walk libraries only for vendor installs, with RPATH inherited as ld.so does, normalized paths, and DT_NEEDED bounded while reading. - Put --no-sandbox after the program an env wrapper launches; keep the gpgv folder's owner when its removal fails; respect --max-file-bytes for keys; wait 0.25 s on a busy SQLite database in data/get. Co-Authored-By: Claude Opus 5.5 * docs(desktop): keep the vendor spec's gpgv command on one line A code span broken across lines left `` for VitePress to read as an unclosed HTML tag, so the docs build failed. Co-Authored-By: Claude Opus 5.5 * feat(desktop): add a validated task to exported desktop applications Each capture can now yield one agentenv task in its exported bundle. An agent writes the task (a prompt plus declarative checks over the app's data/get views); REA validates it and stays model-free. - `describe-desktop-application-task` summarises a capture's net change per file (role, format, views before and after) and drafts checks. - Checks: file presence, lines present/absent/counted, JSON-pointer values, SQLite rows and counts. REA keeps a task only if a check fails on the seed and every check passes on the captured end state; a file that was not retained at the baseline counts as absent, as the computer starts. - `export-desktop-application --task` writes tasks/NAME.json (deploy, agent, skills, recording, prompt, verifier), a compiled verify.py that reads data/get, and an eval. A boundary test runs the verifier under python3 and checks it agrees with REA's evaluator on every check kind. - Docs, spec and skill guidance, with the solve-rate recipe. Co-Authored-By: Claude Opus 5.5 * test(desktop): keep the seed test in its own block Co-Authored-By: Claude Opus 5.5 * docs(desktop): record the task verifier driven in an agentenv computer Co-Authored-By: Claude Opus 5.5 * docs(desktop): record the Typora task run in an agentenv computer Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep task checks bounded, complete and attributable - line_count counts lines holding a substring (contains) instead of matching a regular expression, which could stall the verifier on agent-written text and read differently in Python. - value_equals of a whole object or array is undecided on a truncated view. - The draft keeps a database whose rows changed under an unchanged hash, and a file whose truncated views match while it changed. - A task file that cannot be read keeps the reader's detail, and a failed cleanup keeps the read failure as its primary. Co-Authored-By: Claude Opus 5.5 * fix(desktop): leave a file with one version out of the task draft Co-Authored-By: Claude Opus 5.5 * docs(desktop): update the task verification's test counts Co-Authored-By: Claude Opus 5.5 * docs(desktop): record the task solve rate with Claude Code and Sonnet 5.5 Co-Authored-By: Claude Opus 5.5 * feat(desktop): capture and export apps installed from a downloaded .deb capture-desktop-application --package-file reads the file's control fields with dpkg-deb and records its SHA-256, size and dependencies when it holds the installed package; the file itself is never kept. The library walk now also runs for local installs. export-desktop-application --package-url declares agentenv's Download(url, sha256) for that file, its dependencies and library packages as apt packages, and a run script that installs it with dpkg. Co-Authored-By: Claude Opus 5.5 * fix(desktop): seed app-named dotfiles and respell UTF-16 seeds The capture now treats dotfiles in the home named after the application (.varicad-cfg.ini, .notesrc) as state roots, so a file the app only reads during the capture is still seeded. The exported seeding respells the captured home in UTF-16 text files with a byte-order mark, keeping the mark. Co-Authored-By: Claude Opus 5.5 * fix(desktop): hold readiness until the app's own window shows An exported app is ready only once a window of its class that the taskbar lists is shown, so a splash screen of the class, which skips the taskbar, no longer counts as the app being up. Co-Authored-By: Claude Opus 5.5 * docs(desktop): record VariCAD captured, exported and run in an agentenv computer Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep exported views strict JSON for infinite and inexact numbers The exported data/get spells infinite and negative-zero SQLite reals as REA's SQLite reader does ("Infinity", "-Infinity", "-0"), and refuses a JSON file holding a number that reading would change (1e999, integers past 2^53, -0, rounded fractions) with the state view's own error, instead of a view with an infinity in it. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep package-file facts tied to one file and URLs unechoed - A --package-url refusal never repeats the URL, and credentials are checked before the scheme. - The package file's identity must hold between dpkg-deb's read and the hash. - A failed close keeps its cleanup owner even when the capture is cancelled. - Readiness also ignores windows typed as splash screens. Co-Authored-By: Claude Opus 5.5 * fix(desktop): record database logs and keep state changes distinct - A selected path that is not a directory brings its -wal and -journal as content roots, so the capture records whether they exist and retains them. - The state view treats a file as absent only under recorded content roots and refuses a database whose log presence is unknown. - JSON changes list containers too, so an array that becomes an object with the same members is a change. - Key file entries escape the key as a JSON pointer segment, so [a/b] c and [a] b/c stay apart. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep watching through unread samples and late workers - An unreadable window list is unknown, not zero windows, and is named in the limitations. - Before ending the session, live run-token carriers no parent link reached are tracked; an unreadable process table leaves the exit unknown. - An env prefix's assignments find the program and name its XDG folders. - flatpak run options that take a value are skipped with it, -- is honoured and a /arch/branch suffix is dropped from the id. - State directories include the entry's full id. - Path options refuse empty values; roots count toward the entry limit. - A capture file whose close failed stays owned and is retried on discard. Co-Authored-By: Claude Opus 5.5 * fix(desktop): bound apt sources, keep older captures and cleanup owners - Sources files parse into one stanza per line or deb822 paragraph with its URIs and suites kept as lists, so no URI x suite product is built. - Repository architectures and release are optional, so captures written before they were recorded parse; the export treats them as unrestricted and unknown. - Library sources are matched against every index of a declared repository's URI, suite and components. - Two failures side by side keep both causes and both cleanup owners, and a host file whose close fails reports a lease on its handle. Co-Authored-By: Claude Opus 5.5 * fix(desktop): match escaped key file entries and suggest containers through members - The draft's key file lookup uses the state view's escaped `group/key`, so a key holding `/` or `~` still gets its suggestion. - JSON and XML changes now list containers too; the draft suggests an added or removed container through its listed members, and a container only when none of its members is listed. Co-Authored-By: Claude Opus 5.5 * fix(desktop): read relative PATH entries from the launch folder A program, or an empty or relative PATH entry, is resolved from the folder the program starts in, as the launch does, so `env PATH=. app` with the entry's Path still finds the app. Co-Authored-By: Claude Opus 5.5 * perf(desktop): compare JSON views on one shared path JSON and XML changes are found by walking both trees with one shared pointer path, building a pointer only for a listed value, so memory follows depth rather than every container's full path. A pointer over 4096 characters is counted but not listed. A 256-level document with 4000-character keys went from about 251 MiB of heap to under 1 MiB. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep JSON Lines as text in the exported reader The exported reader parses the whole text before an inexact number makes it JSON with an error, so JSON Lines with an integer above 2^53 stay text as in REA. Over-long integer spellings are unsafe without converting them. The guide no longer claims large integers keep their value. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep the state-path help free of dash-led words The CLI alias check reads `-wal` and `-journal` in help text as short flags, so the help names the write-ahead log and rollback journal instead. Co-Authored-By: Claude Opus 5.5 * fix(desktop): never record or export apt source credentials - Source URIs lose any user name and password as they are parsed, the way apt splits them, and so do index-target URIs and descriptions and the sources apt-cache policy lists, so matching works whether or not apt prints them. - A repository whose entry held credentials is marked `uri_credentials_removed`, and the export refuses it, or an older capture whose URI still holds them, with a reason that names no URL. Co-Authored-By: Claude Opus 5.5 * fix(desktop): look programs up without normalizing their paths Program paths and PATH entries are joined rather than resolved, so `..` after a symbolic link means what it means to the launch. The lookup moves to its own module. Co-Authored-By: Claude Opus 5.5 * fix(desktop): note inexact JSON numbers with a flag, not a list The exported reader only needs to know whether any number was inexact, so it keeps a boolean instead of every rejected spelling. Co-Authored-By: Claude Opus 5.5 * fix(desktop): refuse older captures that still hold apt credentials A capture written before credentials were removed counted the credentialed repository's libraries as installable, so the export refuses it whole and asks for a new capture rather than declaring packages no source offers. Recorded policy sources are redacted in export messages too. Co-Authored-By: Claude Opus 5.5 * test(desktop): keep the credential fixture on an example host A login on a real host reads as a secret to the commit scanner, so the older capture's fixture uses apt.example.com like the other credential tests. Co-Authored-By: Claude Opus 5.5 * fix(desktop): tidy looked-up paths and search relative entries only for env Empty and `.` segments are dropped from a looked-up program path, which keeps `/snap/bin/` entries recognisable as snaps, while `..` is still left for the kernel. Relative and empty PATH entries are searched only in the PATH an `env` prefix sets, which travels with the exported launch; the host's own PATH does not. Co-Authored-By: Claude Opus 5.5 * fix(desktop): keep excluded and unwalked database logs unknown The capture records the paths every walk left out, and the state view treats a log as absent only under content roots the baseline walked completely and outside those exclusions, so an excluded or unvisited log leaves the database unread rather than read without it. Co-Authored-By: Claude Opus 5.5 * fix(desktop): judge JSON numbers after duplicate keys settle The exported reader marks an inexact number instead of flagging it during the parse, then looks for marks in the parsed value, so a number a later duplicate key overwrote no longer refuses the file, as in REA. Co-Authored-By: Claude Opus 5.5 * fix(desktop): redact a repository's policy source when it names it An older capture whose policy source matched no sources entry kept that text whole, and the export named the repository by it; the export now names it without any user name or password. Co-Authored-By: Claude Opus 5.5 * fix(desktop): never signal the launch group on unknown membership When the current read cannot observe a carrier's process group, the carrier is signalled on its own instead of letting an older process-table row grant a group-wide SIGTERM. Co-Authored-By: Claude Opus 5.5 * fix(desktop): look programs up in the launch's order and refuse host-relative finds The lookup searches PATH as the launch does, an empty entry meaning the launch folder, so inspection and launch name the same program. A program found only through a relative or empty entry of the host's own PATH is refused, since the exported computer would not have it; an env prefix's PATH travels with the command. Co-Authored-By: Claude Opus 5.5 * fix(desktop): count unchecked paths and keep logs behind them unknown Each walk counts the paths it could not stat and the directories on another device it did not enter, and the checkpoint records it as `unchecked_paths`. The state view treats a log as absent only when the baseline left nothing unchecked, so a log whose stat failed stays unknown. Co-Authored-By: Claude Opus 5.5 * perf(desktop): scan parsed JSON with one iterator per level The check for marked numbers walks the parsed value with an iterator per nesting level instead of copying each container's children; a million-entry array went from about 15.7 MiB to 8.1 MiB of peak traced memory. Co-Authored-By: Claude Opus 5.5 * fix(desktop): start a relative program from its captured folder An exported launch whose program path is relative, such as `./app` from an entry's Path, now changes to the captured working directory first, as it already did for relative arguments. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- .../skills/reverse-engineer-anything/SKILL.md | 5 + .../references/environments.md | 78 +- docs/.vitepress/config.ts | 16 + docs/cli.md | 8 +- docs/desktop-application-capture.md | 305 ++++++++ docs/desktop-application-export.md | 223 ++++++ docs/desktop-application-state.md | 103 +++ docs/desktop-application-tasks.md | 164 ++++ docs/index.md | 4 + docs/specs/desktop-app-environments/README.md | 36 + .../desktop-app-environments/capture/plan.md | 15 + .../desktop-app-environments/capture/spec.md | 110 +++ .../desktop-app-environments/capture/tasks.md | 12 + .../capture/verification.md | 68 ++ .../desktop-app-environments/export/plan.md | 12 + .../desktop-app-environments/export/spec.md | 41 + .../desktop-app-environments/export/tasks.md | 9 + .../export/verification.md | 49 ++ .../state-view/plan.md | 12 + .../state-view/spec.md | 47 ++ .../state-view/tasks.md | 10 + .../state-view/verification.md | 52 ++ .../desktop-app-environments/tasks/plan.md | 15 + .../desktop-app-environments/tasks/spec.md | 67 ++ .../tasks/verification.md | 131 ++++ .../vendor-deb-files/plan.md | 24 + .../vendor-deb-files/spec.md | 93 +++ .../vendor-deb-files/verification.md | 163 ++++ .../vendor-packages/plan.md | 33 + .../vendor-packages/spec.md | 105 +++ .../vendor-packages/verification.md | 131 ++++ scripts/lib/test-projects.mjs | 2 +- .../desktop/DesktopApplicationCapture.test.ts | 576 ++++++++++++++ .../desktop/DesktopApplicationCapture.ts | 363 +++++++++ .../desktop/DesktopApplicationExport.test.ts | 574 ++++++++++++++ .../desktop/DesktopApplicationExport.ts | 511 +++++++++++++ .../desktop/DesktopApplicationState.test.ts | 510 +++++++++++++ .../desktop/DesktopApplicationState.ts | 488 ++++++++++++ .../desktop/DesktopCaptureFailure.test.ts | 268 +++++++ .../desktop/DesktopCaptureFailure.ts | 230 ++++++ .../desktop/DesktopCaptureOutput.test.ts | 105 +++ .../desktop/DesktopCaptureOutput.ts | 110 +++ .../desktop/DesktopCaptureRefusals.test.ts | 27 + .../desktop/DesktopCaptureRefusals.ts | 217 ++++++ .../desktop/DesktopCaptureScope.test.ts | 157 ++++ .../desktop/DesktopCaptureScope.ts | 117 +++ .../desktop/DesktopCheckpointRecorder.test.ts | 57 ++ .../desktop/DesktopCheckpointRecorder.ts | 144 ++++ .../desktop/DesktopExportInstall.ts | 70 ++ .../desktop/DesktopRetainedContent.test.ts | 42 ++ .../desktop/DesktopRetainedContent.ts | 127 ++++ .../desktop/DesktopSigningKeys.test.ts | 100 +++ src/application/desktop/DesktopSigningKeys.ts | 46 ++ .../desktop/DesktopSqliteState.test.ts | 121 +++ src/application/desktop/DesktopSqliteState.ts | 121 +++ .../desktop/DesktopStateVersions.ts | 184 +++++ src/application/desktop/DesktopTaskDraft.ts | 101 +++ .../desktop/DesktopTaskExport.test.ts | 257 +++++++ src/application/desktop/DesktopTaskExport.ts | 249 ++++++ .../desktop/desktopCapture.fixture.ts | 225 ++++++ src/cli.ts | 2 + src/cli/desktopCaptureCommands.ts | 322 ++++++++ src/cliCommandNames.ts | 4 + src/desktop/AptRepositoryInventory.ts | 281 +++++++ src/desktop/DconfDatabaseReader.test.ts | 28 + src/desktop/DconfDatabaseReader.ts | 54 ++ src/desktop/DebianPackageFile.test.ts | 202 +++++ src/desktop/DebianPackageFile.ts | 126 ++++ src/desktop/DebianPackageFileCleanup.test.ts | 50 ++ src/desktop/DebianPackageInventory.test.ts | 612 +++++++++++++++ src/desktop/DebianPackageInventory.ts | 226 ++++++ src/desktop/DesktopApplicationSession.test.ts | 114 +++ src/desktop/DesktopApplicationSession.ts | 359 +++++++++ src/desktop/DesktopContentStore.test.ts | 148 ++++ src/desktop/DesktopContentStore.ts | 215 ++++++ .../DesktopSessionObservations.test.ts | 237 ++++++ src/desktop/DesktopSessionObservations.ts | 379 ++++++++++ src/desktop/DesktopSessionRequests.test.ts | 61 ++ src/desktop/DesktopSessionRequests.ts | 53 ++ src/desktop/ExportedAppPython.test.ts | 86 +++ src/desktop/GpgvKeyCheck.test.ts | 97 +++ src/desktop/GpgvKeyCheck.ts | 145 ++++ src/desktop/HomeStateWalker.test.ts | 130 ++++ src/desktop/HomeStateWalker.ts | 167 ++++ src/desktop/HostFileRead.ts | 69 ++ src/desktop/HostQuery.ts | 51 ++ src/desktop/LibraryPackageInventory.test.ts | 43 ++ src/desktop/LibraryPackageInventory.ts | 245 ++++++ src/desktop/LinuxApplicationInventory.test.ts | 322 ++++++++ src/desktop/LinuxApplicationInventory.ts | 456 +++++++++++ src/desktop/ProgramLookup.ts | 49 ++ src/desktop/X11WindowObserver.ts | 117 +++ src/domain/desktop/appExport.test.ts | 290 +++++++ src/domain/desktop/appExport.ts | 481 ++++++++++++ src/domain/desktop/appExportInstall.test.ts | 711 ++++++++++++++++++ src/domain/desktop/appExportInstall.ts | 480 ++++++++++++ src/domain/desktop/appExportPython.ts | 438 +++++++++++ src/domain/desktop/appTask.ts | 182 +++++ src/domain/desktop/appTaskChecks.fixture.ts | 322 ++++++++ src/domain/desktop/appTaskChecks.test.ts | 197 +++++ src/domain/desktop/appTaskChecks.ts | 322 ++++++++ src/domain/desktop/appTaskDraft.test.ts | 351 +++++++++ src/domain/desktop/appTaskDraft.ts | 388 ++++++++++ src/domain/desktop/appTaskPython.ts | 264 +++++++ src/domain/desktop/aptKeys.fixture.ts | 73 ++ src/domain/desktop/aptRepositories.test.ts | 483 ++++++++++++ src/domain/desktop/aptRepositories.ts | 380 ++++++++++ .../desktop/debianPackageFile.fixture.ts | 54 ++ src/domain/desktop/debianPackageFile.test.ts | 102 +++ src/domain/desktop/debianPackageFile.ts | 87 +++ src/domain/desktop/desktopCapture.test.ts | 79 ++ src/domain/desktop/desktopCapture.ts | 296 ++++++++ src/domain/desktop/desktopEntry.test.ts | 179 +++++ src/domain/desktop/desktopEntry.ts | 250 ++++++ src/domain/desktop/desktopState.ts | 106 +++ src/domain/desktop/desktopStateDiff.test.ts | 150 ++++ src/domain/desktop/desktopStateDiff.ts | 119 +++ src/domain/desktop/elfDependencies.test.ts | 96 +++ src/domain/desktop/elfDependencies.ts | 242 ++++++ src/domain/desktop/elfImage.fixture.ts | 88 +++ src/domain/desktop/libraryPackages.test.ts | 225 ++++++ src/domain/desktop/libraryPackages.ts | 213 ++++++ src/domain/desktop/linuxPackages.test.ts | 230 ++++++ src/domain/desktop/linuxPackages.ts | 197 +++++ src/domain/desktop/openPgpKeys.test.ts | 127 ++++ src/domain/desktop/openPgpKeys.ts | 120 +++ src/domain/desktop/stateChanges.test.ts | 195 +++++ src/domain/desktop/stateChanges.ts | 290 +++++++ src/domain/desktop/stateFormats.test.ts | 149 ++++ src/domain/desktop/stateFormats.ts | 258 +++++++ src/domain/desktop/stateRoles.test.ts | 72 ++ src/domain/desktop/stateRoles.ts | 212 ++++++ src/domain/desktop/x11Windows.test.ts | 83 ++ src/domain/desktop/x11Windows.ts | 90 +++ src/process/ProcessRunTokenObservations.ts | 27 + src/process/TokenOwnedTermination.test.ts | 153 ++++ src/process/TokenOwnedTermination.ts | 128 ++++ src/process/capture/LinuxProcessSampling.ts | 4 + src/process/capture/ProcessSampling.ts | 1 + tests/boundary/cli/desktopCaptureCli.test.ts | 30 + .../providers/desktopExportPython.test.ts | 243 ++++++ .../providers/desktopInstallScript.test.ts | 80 ++ .../providers/desktopTaskPython.test.ts | 106 +++ 143 files changed, 24130 insertions(+), 3 deletions(-) create mode 100644 docs/desktop-application-capture.md create mode 100644 docs/desktop-application-export.md create mode 100644 docs/desktop-application-state.md create mode 100644 docs/desktop-application-tasks.md create mode 100644 docs/specs/desktop-app-environments/README.md create mode 100644 docs/specs/desktop-app-environments/capture/plan.md create mode 100644 docs/specs/desktop-app-environments/capture/spec.md create mode 100644 docs/specs/desktop-app-environments/capture/tasks.md create mode 100644 docs/specs/desktop-app-environments/capture/verification.md create mode 100644 docs/specs/desktop-app-environments/export/plan.md create mode 100644 docs/specs/desktop-app-environments/export/spec.md create mode 100644 docs/specs/desktop-app-environments/export/tasks.md create mode 100644 docs/specs/desktop-app-environments/export/verification.md create mode 100644 docs/specs/desktop-app-environments/state-view/plan.md create mode 100644 docs/specs/desktop-app-environments/state-view/spec.md create mode 100644 docs/specs/desktop-app-environments/state-view/tasks.md create mode 100644 docs/specs/desktop-app-environments/state-view/verification.md create mode 100644 docs/specs/desktop-app-environments/tasks/plan.md create mode 100644 docs/specs/desktop-app-environments/tasks/spec.md create mode 100644 docs/specs/desktop-app-environments/tasks/verification.md create mode 100644 docs/specs/desktop-app-environments/vendor-deb-files/plan.md create mode 100644 docs/specs/desktop-app-environments/vendor-deb-files/spec.md create mode 100644 docs/specs/desktop-app-environments/vendor-deb-files/verification.md create mode 100644 docs/specs/desktop-app-environments/vendor-packages/plan.md create mode 100644 docs/specs/desktop-app-environments/vendor-packages/spec.md create mode 100644 docs/specs/desktop-app-environments/vendor-packages/verification.md create mode 100644 src/application/desktop/DesktopApplicationCapture.test.ts create mode 100644 src/application/desktop/DesktopApplicationCapture.ts create mode 100644 src/application/desktop/DesktopApplicationExport.test.ts create mode 100644 src/application/desktop/DesktopApplicationExport.ts create mode 100644 src/application/desktop/DesktopApplicationState.test.ts create mode 100644 src/application/desktop/DesktopApplicationState.ts create mode 100644 src/application/desktop/DesktopCaptureFailure.test.ts create mode 100644 src/application/desktop/DesktopCaptureFailure.ts create mode 100644 src/application/desktop/DesktopCaptureOutput.test.ts create mode 100644 src/application/desktop/DesktopCaptureOutput.ts create mode 100644 src/application/desktop/DesktopCaptureRefusals.test.ts create mode 100644 src/application/desktop/DesktopCaptureRefusals.ts create mode 100644 src/application/desktop/DesktopCaptureScope.test.ts create mode 100644 src/application/desktop/DesktopCaptureScope.ts create mode 100644 src/application/desktop/DesktopCheckpointRecorder.test.ts create mode 100644 src/application/desktop/DesktopCheckpointRecorder.ts create mode 100644 src/application/desktop/DesktopExportInstall.ts create mode 100644 src/application/desktop/DesktopRetainedContent.test.ts create mode 100644 src/application/desktop/DesktopRetainedContent.ts create mode 100644 src/application/desktop/DesktopSigningKeys.test.ts create mode 100644 src/application/desktop/DesktopSigningKeys.ts create mode 100644 src/application/desktop/DesktopSqliteState.test.ts create mode 100644 src/application/desktop/DesktopSqliteState.ts create mode 100644 src/application/desktop/DesktopStateVersions.ts create mode 100644 src/application/desktop/DesktopTaskDraft.ts create mode 100644 src/application/desktop/DesktopTaskExport.test.ts create mode 100644 src/application/desktop/DesktopTaskExport.ts create mode 100644 src/application/desktop/desktopCapture.fixture.ts create mode 100644 src/cli/desktopCaptureCommands.ts create mode 100644 src/desktop/AptRepositoryInventory.ts create mode 100644 src/desktop/DconfDatabaseReader.test.ts create mode 100644 src/desktop/DconfDatabaseReader.ts create mode 100644 src/desktop/DebianPackageFile.test.ts create mode 100644 src/desktop/DebianPackageFile.ts create mode 100644 src/desktop/DebianPackageFileCleanup.test.ts create mode 100644 src/desktop/DebianPackageInventory.test.ts create mode 100644 src/desktop/DebianPackageInventory.ts create mode 100644 src/desktop/DesktopApplicationSession.test.ts create mode 100644 src/desktop/DesktopApplicationSession.ts create mode 100644 src/desktop/DesktopContentStore.test.ts create mode 100644 src/desktop/DesktopContentStore.ts create mode 100644 src/desktop/DesktopSessionObservations.test.ts create mode 100644 src/desktop/DesktopSessionObservations.ts create mode 100644 src/desktop/DesktopSessionRequests.test.ts create mode 100644 src/desktop/DesktopSessionRequests.ts create mode 100644 src/desktop/ExportedAppPython.test.ts create mode 100644 src/desktop/GpgvKeyCheck.test.ts create mode 100644 src/desktop/GpgvKeyCheck.ts create mode 100644 src/desktop/HomeStateWalker.test.ts create mode 100644 src/desktop/HomeStateWalker.ts create mode 100644 src/desktop/HostFileRead.ts create mode 100644 src/desktop/HostQuery.ts create mode 100644 src/desktop/LibraryPackageInventory.test.ts create mode 100644 src/desktop/LibraryPackageInventory.ts create mode 100644 src/desktop/LinuxApplicationInventory.test.ts create mode 100644 src/desktop/LinuxApplicationInventory.ts create mode 100644 src/desktop/ProgramLookup.ts create mode 100644 src/desktop/X11WindowObserver.ts create mode 100644 src/domain/desktop/appExport.test.ts create mode 100644 src/domain/desktop/appExport.ts create mode 100644 src/domain/desktop/appExportInstall.test.ts create mode 100644 src/domain/desktop/appExportInstall.ts create mode 100644 src/domain/desktop/appExportPython.ts create mode 100644 src/domain/desktop/appTask.ts create mode 100644 src/domain/desktop/appTaskChecks.fixture.ts create mode 100644 src/domain/desktop/appTaskChecks.test.ts create mode 100644 src/domain/desktop/appTaskChecks.ts create mode 100644 src/domain/desktop/appTaskDraft.test.ts create mode 100644 src/domain/desktop/appTaskDraft.ts create mode 100644 src/domain/desktop/appTaskPython.ts create mode 100644 src/domain/desktop/aptKeys.fixture.ts create mode 100644 src/domain/desktop/aptRepositories.test.ts create mode 100644 src/domain/desktop/aptRepositories.ts create mode 100644 src/domain/desktop/debianPackageFile.fixture.ts create mode 100644 src/domain/desktop/debianPackageFile.test.ts create mode 100644 src/domain/desktop/debianPackageFile.ts create mode 100644 src/domain/desktop/desktopCapture.test.ts create mode 100644 src/domain/desktop/desktopCapture.ts create mode 100644 src/domain/desktop/desktopEntry.test.ts create mode 100644 src/domain/desktop/desktopEntry.ts create mode 100644 src/domain/desktop/desktopState.ts create mode 100644 src/domain/desktop/desktopStateDiff.test.ts create mode 100644 src/domain/desktop/desktopStateDiff.ts create mode 100644 src/domain/desktop/elfDependencies.test.ts create mode 100644 src/domain/desktop/elfDependencies.ts create mode 100644 src/domain/desktop/elfImage.fixture.ts create mode 100644 src/domain/desktop/libraryPackages.test.ts create mode 100644 src/domain/desktop/libraryPackages.ts create mode 100644 src/domain/desktop/linuxPackages.test.ts create mode 100644 src/domain/desktop/linuxPackages.ts create mode 100644 src/domain/desktop/openPgpKeys.test.ts create mode 100644 src/domain/desktop/openPgpKeys.ts create mode 100644 src/domain/desktop/stateChanges.test.ts create mode 100644 src/domain/desktop/stateChanges.ts create mode 100644 src/domain/desktop/stateFormats.test.ts create mode 100644 src/domain/desktop/stateFormats.ts create mode 100644 src/domain/desktop/stateRoles.test.ts create mode 100644 src/domain/desktop/stateRoles.ts create mode 100644 src/domain/desktop/x11Windows.test.ts create mode 100644 src/domain/desktop/x11Windows.ts create mode 100644 src/process/TokenOwnedTermination.test.ts create mode 100644 src/process/TokenOwnedTermination.ts create mode 100644 tests/boundary/cli/desktopCaptureCli.test.ts create mode 100644 tests/boundary/providers/desktopExportPython.test.ts create mode 100644 tests/boundary/providers/desktopInstallScript.test.ts create mode 100644 tests/boundary/providers/desktopTaskPython.test.ts diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index a974b8c698..4784bfeabb 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -137,6 +137,9 @@ explicit path or endpoint and do not need it. - Agent environment (AgentEnv bundle) of a web application: read [environments](references/environments.md) before recording; it records with `capture_browser_scenario` and writes the bundle with `export_environment_bundle`. +- Task for a desktop application's exported environment, from a + `capture-desktop-application` folder: read + [desktop application tasks](references/environments.md#tasks-for-a-desktop-application-environment). - Fiddler Everywhere traffic or a native `.saz` capture: read the [Fiddler workflow](references/runtime-observation.md#fiddler-everywhere-traffic-debugging). Inspect saved SAZ with `inspect_web_network_capture` and `format: "saz"`; @@ -521,6 +524,8 @@ as complete while required questions remain open. - Agent environments (AgentEnv bundles) of a web application, recorded from a live instance or a HAR: [references/environments.md](references/environments.md) +- Tasks for a desktop application's exported environment: + [references/environments.md](references/environments.md#tasks-for-a-desktop-application-environment) ## Finish the task diff --git a/.agents/skills/reverse-engineer-anything/references/environments.md b/.agents/skills/reverse-engineer-anything/references/environments.md index 0be64fbab2..3157c81c99 100644 --- a/.agents/skills/reverse-engineer-anything/references/environments.md +++ b/.agents/skills/reverse-engineer-anything/references/environments.md @@ -8,7 +8,9 @@ confirmed. The workflow records the application once with again if a hint names a gap, then writes the bundle with `export_environment_bundle`. For a HAR the user already has, skip to [export](#export-and-hand-off) through `inspect_web_network_capture`, or use -`rea environment-build `. +`rea environment-build `. For a task in a desktop +application's exported environment, read +[its section](#tasks-for-a-desktop-application-environment). ## Before recording @@ -285,3 +287,77 @@ See [agent environments](https://github.com/morluto/rea/blob/main/docs/agent-environments.md) for what the state, replay and the bundle establish, and [evidence workflows](evidence-workflows.md) for the manifest and state views. + +## Tasks for a desktop application environment + +Use this when the user has a folder from `rea capture-desktop-application` and +wants a task for the environment `rea export-desktop-application` makes from +it: a prompt for a computer-use agent and checks that grade what the agent +left in the app's files. You write the prompt and the checks; REA validates +the task against the capture and exports it. These commands are CLI only. + +### Describe the change + +Run `rea describe-desktop-application-task CAPTURE --format json`. Read: + +- `owned_paths`: the only paths whose files the app serves, so the only files + a check can read. +- `files`: each file there whose state differs between the capture's first + and last version, with its `role`, the `baseline` and `final` versions and + the net `change`. Text changes list each added and removed line once, and + `line_counts` gives lines found at both ends whose count changed. +- `task`: a draft whose `checks` are suggestions from the net change of the + documents the user worked on, or of the app's own state when no document + changed. + +### Write the prompt + +Say what the user wants done, in their words and the app's terms: "Add buy +milk to my to-do list and save it". Never say how the file stores it: no +paths the user would not name, line formats, XML, keys or SQL, and never the +checks. Replace the draft's placeholder; the export refuses it. + +### Pick the checks + +Checks must express only the observed change: + +- Keep a suggestion only if it says what the user did. Drop those that follow + from how the app saves: a format rewrite, renumbered lines, counters. A + LibreCAD drawing gaining four holes was rewritten from DXF R12 to 2007, and + the draft suggests counts of `SECTION`, `ENDSEC` and `CIRCLE`; only + `CIRCLE` equals 9 is the user's change. +- Never check unchanged content, whole files, or values the app stamps: + times, ids it assigns, window sizes. REA refuses ISO 8601 date-times; the + rest is your judgment. +- Prefer a few robust checks: whole lines over positions, rows by meaningful + columns over generated ids, `at_least` when the user may add more. + +Kinds: `file_present` and `file_absent`; `lines_present` and `lines_absent` +(`lines`); `line_count` (`exact` or `contains`, and `equals`, `at_least` or +`at_most`); `value_equals` and `value_absent` (a JSON `pointer` into the view, +such as `/groups/GROUP/KEY` for a key file); `row_present` and `row_absent` +(`table`, `where` by column) and `row_count`. Each names its `file` as a path +under the home. The +[desktop application tasks](https://github.com/morluto/rea/blob/main/docs/desktop-application-tasks.md) +guide has the exact rules. + +### Export the task + +Write the task as JSON with a `name`, the `prompt` and the `checks`, then run +`rea export-desktop-application CAPTURE DIRECTORY --task TASK.json`. REA keeps +the task only when its checks fail on the seed and all pass on the captured +end state. A refusal names the check: one that fails on the end state does not +match what the user left; "every check already passes on the seed" means no +check captures the change. + +When the export refuses an application that no apt repository installs, ask +the user for the `.deb` file they installed it from and capture with +`--package-file`, then ask where the computer may download that file and pass +it as `--package-url`. Never fetch a vendor's download page yourself: such +sites often refuse automated downloads, and the copy and its licence are the +user's to choose. + +Tell the user the task's name, prompt and checks, and how to measure a solve +rate: run `agent-env run DIRECTORY --task NAME --agent claude-code` with +`--model MODEL` three times and count the runs reported `passed`. Do not run +agents unless asked. The bundle's seed is the user's real data: keep it local. diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index d644d593eb..d121d861e5 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -61,6 +61,22 @@ export default defineConfig({ { text: "Electron observation", link: "/electron-observation" }, { text: "Electron fuses", link: "/electron-fuses" }, { text: "Process capture", link: "/process-capture" }, + { + text: "Desktop application capture", + link: "/desktop-application-capture", + }, + { + text: "Desktop application state", + link: "/desktop-application-state", + }, + { + text: "Desktop application export", + link: "/desktop-application-export", + }, + { + text: "Desktop application tasks", + link: "/desktop-application-tasks", + }, { text: ".NET assemblies", link: "/managed-code-analysis" }, { text: "Python packaged applications", diff --git a/docs/cli.md b/docs/cli.md index db22675191..dedf9f638d 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -968,6 +968,10 @@ Choose the guide for the target and intended interaction: - [Electron fuses](electron-fuses.md): read packaged fuse bytes before expecting `--inspect` to reach an Electron main process. - [Node/Electron Inspector](javascript-runtime-observation.md): record script locations and execution contexts. - [Process capture](process-capture.md): run an executable and compare terminal, exit and filesystem observations. +- [Desktop application capture](desktop-application-capture.md): launch a Linux desktop application and record its windows and home-directory state at checkpoints while it is used. +- [Desktop application state](desktop-application-state.md): read a capture folder into the application's state files, typed views of each version and kinds of change. +- [Desktop application export](desktop-application-export.md): write a capture folder as an agentenv computer app bundle with its class, skill and seed. +- [Desktop application tasks](desktop-application-tasks.md): describe how a capture's files changed, then validate an agent-written task against the capture and export it with its verifier. Runtime requests name the target and actions. Launched targets run with your user permissions; consult the chosen guide for host requirements and effects. @@ -1319,7 +1323,9 @@ the command's signal handling. Commands routed through the shared `withCommandCancellation` handler in the CLI set status `130` for SIGINT or `143` for SIGTERM after their operation settles. Current source registrations use that handler for process capture and comparison (`capture-process`, -`compare-process-captures`), JavaScript application analysis (`analyze` when it +`compare-process-captures`), desktop application capture and state +(`capture-desktop-application`, `inspect-desktop-application-state`, +`export-desktop-application`, `describe-desktop-application-task`), JavaScript application analysis (`analyze` when it selects the JavaScript lane, and `analyze-javascript-application`), selected browser, web and Electron commands (including `analyze-web-bundle`, `observe-web-execution`, `capture-browser-scenario` and diff --git a/docs/desktop-application-capture.md b/docs/desktop-application-capture.md new file mode 100644 index 0000000000..b9f56a3db9 --- /dev/null +++ b/docs/desktop-application-capture.md @@ -0,0 +1,305 @@ +# Desktop application capture + +`rea capture-desktop-application` launches an installed Linux desktop +application, watches it while a person or a driver uses it, and records what +it is, which windows it opened and how the files under the home directory +changed. The result is the starting point for turning the application into an +agent environment: the inventory says how to install and launch it, the windows +say how to recognise it, and the checkpoints hold the state it read and wrote. + +The application runs with the current user's permissions, in the current +desktop session, against the user's real files. This is not a sandbox. The +capture folder keeps copies of real files from the home directory, so treat it +as private data and keep it local. + +## Host support + +Capture runs on Linux with an X11 display: `DISPLAY` must be set, and `xprop` +(from `x11-utils`) must be installed, or `REA_XPROP_COMMAND` must name it. +The window manager must support the EWMH client list (`_NET_CLIENT_LIST`, +listed in `_NET_SUPPORTED`), as xfwm4, Mutter, KWin and Openbox do; without it +the capture is refused, because no window could be seen. A supported list that +is unset, as xfwm4 leaves it while no window is open, means no windows. Applications that run under XWayland are visible; +native Wayland windows are not. Other hosts are refused before anything is +launched. + +## Capture an application + +Name the application by its desktop entry id, the entry's absolute path, or an +executable: + +```sh +rea capture-desktop-application --desktop-entry org.xfce.mousepad --output ./mousepad-capture +rea capture-desktop-application --executable gnucash --argument /home/ada/books.gnucash --output ./gnucash-capture +``` + +Desktop entries are looked up in `$XDG_DATA_HOME/applications` and then each +`$XDG_DATA_DIRS` directory; an entry installed as a symbolic link, as flatpak +and Nix do, is followed to its file. The entry's `Exec` line is expanded with no files; +`--argument` values are appended in its place, and the application runs in the +entry's `Path` or the home directory. When the line starts with +`env NAME=value`, the program is found on the `PATH` those assignments set, +with an empty or relative entry read from the folder the application runs +in, and their XDG folders name the state directories. Otherwise the program is +looked up on the host's `PATH` in the order the launch searches it, and one +found only through a relative or empty entry is refused, since the exported +computer would not have that entry. An `--executable` runs from +the current directory, as it would from a shell, and a relative path to it is +resolved there. + +Before writing anything the command refuses to start if a window of the +application is already open, or if the window list cannot be read. A launch +beside a running instance can hand off to it, and that instance's unsaved +state would be missing from the baseline. It then records the starting state, +launches the application in its own process group with a run token in its +environment, and reads commands from stdin, one per line: + +| Line | Effect | +| ------------------------------------------- | ----------------------------------------------------------- | +| empty, `checkpoint`, or `checkpoint ` | Record the current state as a checkpoint | +| any other text except `stop` | Record a checkpoint named by that text | +| `stop`, or the end of stdin | Record a final checkpoint, close the application and finish | + +An unnamed checkpoint is named `checkpoint-` after its time in the +session. A name that repeats an earlier checkpoint, or is one of the capture's +own names (`baseline`, `stop`, `time-limit`, `exit`), is refused with a +progress line on stderr; no checkpoint is taken and the session continues. + +The session also finishes when the launched process has exited and the +application's processes and windows are all gone in two consecutive samples, +or when `--max-duration-seconds` passes. Before deciding the processes are +gone, the capture looks for any process still carrying the run token, so a +worker whose launcher exited before it was sampled keeps the session open. A +sample whose window list could not be read is not taken as an empty one; the +limitations say how many there were. One queued line is handled between +samples, and reading stdin pauses while 16 lines wait. Progress lines on stderr say when +the window has opened and when each checkpoint is taken. + +When the session ends the capture sends SIGTERM to the launched process, its +process group and every process that carries the run token, waits up to five +seconds for them to exit, and then removes every process that still carries +the token, whether or not it stayed in the process group. If that cleanup +cannot be verified after a second attempt, the capture is still written, and +the command then fails, naming the run token and process group that may +remain. Interrupting the command (Ctrl-C) ends the session the same way +without a further checkpoint; the capture is still written with end reason +`cancelled`, and the command exits with status 130. Interrupted before the +application is launched, the command writes no capture and removes the +content it had retained. + +If the capture fails for any other reason before `capture.json` is written, +the application is closed the same way and the content already retained is +removed from the capture folder. The error names any process or file that +could not be removed. + +A person types these lines in the terminal. An agent can drive the +application with a tool such as `xdotool` and write the lines to a FIFO that +it passes as stdin. Because the end of stdin stops the session, the driver +must keep the FIFO open for writing between checkpoints, for example with +`exec 3>capture.fifo` in a shell, rather than open and close it for each +line. + +| Option | Default | Meaning | +| ------------------------ | ------- | ----------------------------------------------------------------------------------------- | +| `--state-path` | none | File or directory whose starting content is retained too (repeatable) | +| `--exclude` | none | Path left out of every checkpoint walk (repeatable); the output folder is always left out | +| `--max-duration-seconds` | 3600 | Stop the session after this long | +| `--max-entries` | 250,000 | Paths recorded per checkpoint walk, its roots included | +| `--max-file-bytes` | 64 MiB | Largest file whose content is retained | +| `--max-content-bytes` | 1 GiB | Total retained content | +| `--package-file` | none | The `.deb` file the application was installed from; its facts are recorded, not the file | + +`--state-path`, `--exclude` and `--output` refuse an empty value, which +would otherwise name the current directory. + +## The capture folder + +The output folder must be new, or empty and owned by the current user; the +command makes it private (mode 0700). It holds: + +- `capture.json`: the Evidence record, also printed on stdout. +- `content/`: the bytes of each retained file, stored once per + distinct content. + +The Evidence operation is `capture_desktop_application` and its predicate type +is `rea.desktop-application-capture`. Its result has four parts. + +**Application.** The desktop entry (id, path, name, `Exec`, icon, +`StartupWMClass`, categories, MIME types), the launch command and working +directory, the executable as found on `PATH` and after resolving links, and the +package that installed it. A Debian package records its name, version, +architecture and the apt sources that list the installed version, with the +origin (`o=`) apt reports for each source in `source_origins`, such as `Ubuntu` +for the Ubuntu archive and its mirrors or `LP-PPA-` for a PPA; an empty +source list means the version came from a local `.deb` or the package lists +are missing, which `package_lists_present` tells apart. + +Each source whose origin is not `Ubuntu` is also recorded in +`apt_repositories`. `apt-get indextargets` names the sources file entry apt +read it from and its release, so flat repositories match too, whether their +suite is `./` (Typora's `deb https://downloads.typora.io/linux ./`) or `/` +(pkgs.k8s.io's). The record holds that entry (`sources_file`, `uri`, `suite`, +`components`, none for a flat repository, and `architectures`, the entry's +`arch=` or `Architectures:` restriction, empty when it has none), `release`, +the release's Suite (apt's `a=`) or else its Codename, and the key that signs +it. A user name or password in an entry's URI (`https://user:password@host/…`) +is never recorded: REA removes it from every URI it reads, in the sources file +and in `apt-get indextargets` and `apt-cache policy` output alike, so entries +match whether or not apt prints it, and sets `uri_credentials_removed` on a repository whose URI held +one (a capture written before this was recorded omits the field). The key is +found by verification, not by name: the candidates are the +entry's `signed-by` keyrings, or else every `.gpg` and `.asc` key in +`/etc/apt/trusted.gpg.d` and `/etc/apt/trusted.gpg`, and each is checked with +`gpgv` against the release's `InRelease` (or `Release` with `Release.gpg`) +beside its index in apt's lists folder. A key counts when gpgv reports a +valid signature by it and no bad, expired, revoked or unverifiable signature +by it, whatever gpgv's exit status: a release also signed by a key the host +lacks makes gpgv fail. Each candidate is copied, an armored key decoded to a +binary keyring, into a private temporary folder that is removed afterwards; a +folder REA cannot remove fails the capture with the folder as its cleanup +resource. The key that verifies is recorded in `signing_key` with its path, +SHA-256, format (`binary` or `armored`) and primary fingerprint, and its bytes +are retained under `content/`. A key file holding one key is retained as it +is; from one holding several, such as a legacy `/etc/apt/trusted.gpg`, only +the transferable key with the verifying fingerprint is retained, in binary, so +the bundle trusts no other key. When no candidate verifies, `gpgv` is +missing, apt holds no signed release, no sources entry matches, the entry's +`Signed-By` holds the key itself, or the key is larger than +`--max-file-bytes` or the content budget, `signing_key` is null and +`signing_key_reason` says why. Ubuntu Pro's ESM repositories (origins such as +`UbuntuESMApps`, host `esm.ubuntu.com`) are not checked: apt reaches them only +with the host's Ubuntu Pro credentials, which a computer does not have. + +Unless the installed version comes from the Ubuntu 24.04 archive, so for a +vendor repository's package and for a local install that no apt source lists, +`library_packages` lists the packages that own the shared libraries the +executable loads at start; the walk runs beside the repository check, and not +at all for an archive package, whose export does not use it. REA reads +DT_NEEDED from the resolved executable and from each library it finds, and +looks each soname up as ld.so does: in the RPATH of the needing image and of +each image up its chain of loaders to the executable when the needing image +has no RUNPATH, then in its RUNPATH (with `$ORIGIN` expanded and paths +normalized), and then in the `ldconfig -p` cache entries for the executable's +machine. It asks `dpkg-query -S` for each library's package. Libraries in the +executable's own folder belong to the application: they are walked but not +listed, and neither is the application's own package. One batched +`apt-cache policy` then keeps only the packages the Ubuntu 24.04 archive or a +recorded vendor repository with a verified key offers, in any component its +sources entry lists, since the export declares them all; the rest, such as +packages from a PPA or a local `.deb`, are listed in +`uninstallable_library_packages`. Sonames found nowhere are listed in +`missing_libraries`, and libraries no package owns in `unpackaged_libraries`. +`library_packages_reason` says why the lists are incomplete: the executable is +not an ELF image (a launcher script, for example), `ldconfig`, `dpkg-query` or +`apt-cache` is missing, a library could not be read or names more than 512 +libraries, or the walk stopped after 512 libraries. + +A sources file, key or library whose handle REA cannot close fails the capture +with the file as its cleanup resource. When the repository check and the +library walk both fail, the capture reports both failures and every folder or +file either could not release. + +An application installed from a `.deb` file you downloaded, such as VariCAD +from varicad.com, has no apt source a computer could install it from. Name +that file with `--package-file` so the export can have the computer download +it instead. REA reads the file's control fields with +`dpkg-deb --field FILE Package Version Architecture Depends Pre-Depends`, and +when its Package, Version and Architecture are the installed package's, it +reads the file once more for its SHA-256 and size and records `package_file`: +the file's `name` without its folder, `sha256`, `size`, `package`, `version`, +`architecture`, and the `depends` and `pre_depends` entries as the file +declares them. Otherwise `package_file_reason` says why, for example that +`dpkg-deb` is not installed, that the file holds another version than the +installed one, or that the file changed between the two reads (its device, +inode, size and times are compared), since its fields and its hash could then +describe different files. The capture never copies the file: it is the vendor's, and only +these facts are kept. + +Snaps, flatpaks and +AppImages are recognised from their paths, and flatpaks also from a +`flatpak run` launch, whose application id is the first operand after +flatpak's options and their values. Electron applications are marked by +the fuse sentinel in their binary or a `resources/app.asar` beside it. State +directories are the XDG config, data, state and cache folders and the +dot-folder named after the package, the executable, the desktop entry's full +id and its last part, and the entry's name, with whether each existed at the +start. Dotfiles in the home named after the application (`.rc`, or +`.` followed by `-`, `.` or `_`, for a name of three or more characters) +that exist at the start join them, since an application may only read such a +file during the capture. A `flatpak run` launch is named after its application +id rather than the `flatpak` launcher, and adds `~/.var/app/`. + +**Session.** When it started, how long it ran and why it ended; the launched +process's exit status; every process seen in the application's tree; every +client window with its class pair, titles, owner pid and type, marked when it +belongs to the application; when the first application window appeared; the +most common class pair of the application's windows, for matching its window +later; files the application's processes held open under the walked roots; the first +8 KiB of the launched process's stderr; and whether cleanup was verified. Each +process records whether the capture started it: the launched process, a +process carrying the capture's run token, or a descendant of one, even when +its environment cannot be read, as for a zombie or a sandboxed renderer. +Processes are told apart by their start time, so a pid reused by another +process is not counted as the application's. + +A window belongs to the application when its owner pid is one of the +application's live processes or its class matches the desktop entry's +`StartupWMClass`, the executable's name, the entry's full id or its last part, +or a flatpak's application id. A flatpak's windows report pids from its +sandbox, so they are matched by class only. +Ownership is checked again at every sample, because a window can appear before +the process that owns it is found. A matching window's pid joins the +application's processes, so a launch handed to a process the capture did not +start, such as a D-Bus activated service, is still followed. That process does +not carry the run token, so the capture never signals it and leaves it +running. + +**State.** The walked roots (the home directory plus any content root +outside it), the content roots (state directories, `--state-path` values and +absolute `--argument` values, except `/` or a directory that contains the home +directory, which are named in the limitations instead; a selected path that is +not a directory brings the `-wal` and `-journal` paths beside it, so a database +selected on its own is captured with its logs), the state directories +as walked (their physical spelling, which differs from the inventory's when one +is a symlink), the paths every walk left out (the output folder and +`--exclude` values), and the checkpoints in order: `baseline` before +launch, each requested checkpoint, `stop` or `time-limit`, and `exit` after the +application has gone. Each checkpoint records how many paths its walk saw, +whether it was truncated, how many paths it could not read or directories on +another device it did not enter (`unchecked_paths`), which paths were created, modified, deleted or +changed type since the previous checkpoint, and which files it retained. A +path counts as created or deleted only where the walk listed its nearest +existing directory completely; differences under a directory that could not be +listed, or that a truncated walk did not reach, are counted as unconfirmed +instead. A walk does not enter a directory on another filesystem than its +root, such as an NFS, sshfs or other FUSE mount under the home directory; the +limitations list each such mount point. The +baseline retains every file under the content roots; later checkpoints retain +every changed file. A change records the file's type, size, mode, +modification time and, when known, the SHA-256 of its content before and +after, plus whether it lies in a state directory and whether the application +held it open. A file is retained only if it is still the version the walk +recorded and does not change while it is read. A file larger than the +remaining content budget is not read, unless stored content of the same size +could make it a duplicate. Files that could not be +retained are listed with the reason: too large, over the content budget, +changed since the walk, changed during the read, or unreadable. +Directories the application held files open in, but whose starting content was +not retained, are listed so a second capture can pass them with +`--state-path`. + +**Limitations.** What the capture could not establish, such as a truncated +walk or a missing window. + +## What the capture does not establish + +Changes are found by comparing walks, using each path's size, modification +time, inode and mode, so a change undone between two checkpoints is not seen +and a rewrite that keeps all four is missed. A change is not attributed to the +process that made it: other desktop services write under the home directory +too, such as the dconf database, the window manager's settings and GTK's +recently used list. The markers for state directories and open files are +evidence for attribution, not proof. + +Clicks and keystrokes are not recorded. Window titles are sampled every half +second, so short-lived titles can be missed. diff --git a/docs/desktop-application-export.md b/docs/desktop-application-export.md new file mode 100644 index 0000000000..ab687e849a --- /dev/null +++ b/docs/desktop-application-export.md @@ -0,0 +1,223 @@ +# Desktop application export + +`rea export-desktop-application ` turns a folder +written by [desktop application capture](desktop-application-capture.md) into an +agentenv bundle holding one computer app environment: the real application, +installed from the distribution archive, its vendor's apt repository or the +`.deb` file it was installed from on the `ubuntu-24.04` computer base, launched +at boot, seeded with the user's state and serving that state through agentenv's +data operations. + +```sh +rea export-desktop-application ./mousepad-capture ./mousepad-bundle +agent-env run ./mousepad-bundle --dry-run +``` + +The export reads the capture and its [state view](desktop-application-state.md); +it does not run the application. `--name` sets the environment name; otherwise +it comes from the desktop entry id or the program, with an `app-` prefix when +that name would not be a portable bundle name (for example all digits). The +output directory must be new or empty. `--task` adds a task an agent wrote for +the capture, once REA has validated it against the capture; see +[desktop application tasks](desktop-application-tasks.md). `--package-url` +says where the computer downloads the package of an application installed from +a `.deb` file; see [Applications from a .deb file](#applications-from-a-deb-file). + +The command returns Evidence with operation `export_desktop_application` and +predicate type `rea.desktop-application-export`, listing the environment name, +owned paths, seed files and written files. A failed export, cancellation +included, removes the bundle it had not yet published, so no partial copy of the +user's data remains. If the bundle was published but its output handles could +not be closed, the command reports a cleanup error that carries the export +Evidence; the bundle itself is complete. + +## The bundle + +| Path | Contents | +| -------------------------------- | -------------------------------------------------------------------------------------------------- | +| `envs//env.toml` | `type = "computer_app"`, the module and class, `computer = "ubuntu-24.04"`, `driver = "reference"` | +| `envs//.py` | The `ComputerAppEnvironment` class | +| `envs//SKILL.md` | What the app is, its window class and where its files live | +| `envs//seed/` | The application's files from before the capture, by home-relative path | +| `envs//apt-keys/` | The signing key of each vendor apt repository the app installs from | +| `envs//install-package.sh` | With `--package-url`: the run script that installs the downloaded package with dpkg | +| `tasks/smoke.json` | A task that deploys the app | +| `tasks/.json` | With `--task`: the agent's task, its verifier in `artifacts/` and the eval in `evals/` | +| `README.md` | What was exported, from which capture, and what to keep in mind | + +The class file is `_app.py` with hyphens as underscores, for example +`mousepad_app.py`: agentenv imports it under its stem, so a plain `.py` +could shadow a standard library module such as `calendar`. A class name that +would collide with a name the module imports or defines, such as `Path`, takes +an `App` suffix. + +The class declares: + +- `Packages(apt=[...])` with the package the capture found. A vendor package + is named `/`, its repository's release (the Suite apt + reports as `a=`, or else its Codename), so apt takes it from that repository + even when the Ubuntu archive offers a higher version or epoch, as noble's + transitional `firefox` does over packages.mozilla.org's; it is named alone + when the release names neither, or when the capture was written before + releases were recorded. Its library packages follow, so a package that + leaves out dependencies on the libraries it loads, as Typora's Electron + package does, still starts; library packages the capture found neither the + Ubuntu 24.04 archive nor a declared repository offers are left out, with a + limitation. An archive package's declared dependencies are trusted as they + are. The package must come from the Ubuntu 24.04 archive or a mirror of it, + or from a vendor apt repository whose signing key the capture verified and + retained. A source counts as the archive when apt reports its origin as + `Ubuntu` for a noble suite; a capture made before origins were recorded + counts only Canonical's own archive hosts. +- For a vendor repository, `sources=[AptSource(apt=..., suite=..., + components=[...], key="apt-keys/.gpg")]`, with the retained key copied + to that path (`.asc` when it is armored, which agentenv tells by its + content). apt takes one key per URI and suite, so sources that share them, + such as one repository listed for two architectures, are declared once, with + their components merged; one with a different key is left out. A repository + whose key the capture did not verify, an Ubuntu Pro (ESM) repository, one + whose sources entry's URI held a user name or password (which the capture + did not record and the export never writes; the reason names neither them + nor the URI), one not reached over http or https, or one whose `arch=` + restriction leaves out the capture host's architecture is refused with its + reason (a capture + written before restrictions were recorded counts as unrestricted); when + another usable repository lists the same version it is left out instead, + with a limitation. The export also refuses snaps, flatpaks, AppImages, and a + package no recorded repository lists unless it was installed from a `.deb` + file the capture recorded and `--package-url` is given, since the computer + could not install them by name, and a capture written before credentials + were removed whose repository URI still holds them, since it counted that + repository's libraries as installable; capture it again. +- `@computer_app(supports=...)` with both `amd64` and `arm64` for an archive + package, which the archive serves for both, and only the capture host's + architecture for a vendor package, since its repository may serve no other. + A vendor package captured on any other architecture is refused. A package + installed from its `.deb` file supports only that file's architecture. +- A launch at boot with the captured command, ready when a window of the + captured class appears and one of them is listed in the taskbar and is not + typed as a splash: a splash screen or a dialog of the class skips the + taskbar or says it is a splash, and an app whose splash shows first would + otherwise count as ready before its own window opens. Arguments that are the captured home or under it, and + `