feat(model): inspect GGUF and safetensors declarations (#5603)

This commit is contained in:
gensokyo-dev authored and GitHub committed 2026-10-21 00:50:50 +08:00
1 parent 5b676ba9f3
commit 98f4072f1f
60 files changed
+3835 -3

No files matched your search

@@ -171,6 +171,15 @@ explicit path or endpoint and do not need it.
absent metadata does not prove a binary is not Go, and unparsed text records
remain explicit. See the
[Go binary guide](https://github.com/morluto/rea/blob/main/docs/go-binaries.md).
- Bundled GGUF or safetensors model declarations: `inspect_model_artifact`
with the explicit local file. Default header hashing leaves full-file identity
unknown; `hash: "full"` additionally reads all weights for SHA-256. MCP returns
retained summaries; optional `view` selects one page in the initial call. Use
`inspect_analysis_view` kind `model-artifact` for
batched metadata keys, tensor names or type totals, selected fields and pages.
Preserve header-scoped indices and array preview omissions; do not infer model
behavior from declared architecture or tensor types. See the
[model file guide](https://github.com/morluto/rea/blob/main/docs/model-artifacts.md).
- Unity IL2CPP names, member owners and string literals from `global-metadata.dat`:
`inspect_il2cpp_metadata` with the explicit local file; MCP defaults to compact
counts. Query sections, images, types, methods, fields, properties, events,
@@ -38,6 +38,16 @@ unparsed module-text line. A missing record is an unknown, not proof of a
non-Go binary; observed metadata does not authenticate its claims. Function,
source and type recovery remain a separate native-analysis question.
## Model files
Use `inspect_model_artifact` (`inspect-model-artifact PATH`) for local GGUF v2/v3
and safetensors declarations without a model runtime. Header-only mode does not
read weights or establish full-file identity. `hash: "full"` explicitly streams
all bytes. Retained `model-artifact` views page `metadata`, `tensors` and derived
`types`, with exact batch filters and selected fields. Arrays retain typed counts
and eight-element previews; omitted values, unknown types and runtime behavior
stay explicit. See [model files](https://github.com/morluto/rea/blob/main/docs/model-artifacts.md).
## IL2CPP metadata
Use `inspect_il2cpp_metadata` for an explicit local Unity `global-metadata.dat`
+1
View File
@@ -180,6 +180,7 @@ Additional tools and host support depend on the target:
| Native binaries | Pseudocode, assembly, strings, symbols, calls and references | Hopper, Ghidra or IDA; [native analysis](docs/native-investigation.md) |
| Offline ELF layout | Sections, segments, original symbols/relocations and static mitigation candidates | Caller-supplied pwntools on Linux x64; [binary diagnostics](docs/binary-diagnostics.md) |
| Go binaries | Embedded compiler version, modules, replacements and build settings | Built-in static ELF/PE/thin Mach-O inspection; no Go installation required |
| Model artifacts | GGUF / safetensors metadata, tensor declarations and derived totals | Built-in header inspection; optional full-file hash; [model files](docs/model-artifacts.md) |
| IL2CPP metadata | Unity global-metadata.dat assembly images, type and method names and index ranges | Built-in static decoding of versions 24, 27, 29, 31 and 39; no Unity required |
| Godot PCK | Archive inventory and extraction, original header/directory facts and selected content identities | Built-in static formats 2, 3 and 4; no Godot required |
| Rust binaries | Embedded cargo-auditable crate names, versions, sources and dependency edges | Built-in static ELF/PE/thin Mach-O inspection; [Rust binaries](docs/rust-binaries.md) |
+1
View File
@@ -30,6 +30,7 @@ export default defineConfig({
{ text: "Native binaries", link: "/native-investigation" },
{ text: "Go build metadata", link: "/go-binaries" },
{ text: "IL2CPP metadata", link: "/il2cpp-metadata" },
{ text: "Model artifacts", link: "/model-artifacts" },
{ text: "Rust dependencies", link: "/rust-binaries" },
{
text: "JavaScript and Electron",
+13
View File
@@ -103,6 +103,19 @@ Formats 2, 3 and 4 retain header, directory and integrity facts. Compact deliver
keeps complete Evidence in the requested bundle. See [Godot PCK](godot-pck.md)
for embedded packs, batch queries and unknown encrypted coverage.
Inspect local model declarations without loading weights:
```bash
rea inspect-model-artifact /absolute/path/model.gguf --json
rea inspect-model-artifact /absolute/path/model.safetensors --hash full --json
```
Default hashing reads only headers and leaves full-file identity unknown.
`--hash full` streams the complete file for SHA-256. Both return typed metadata,
original tensor declarations and derived totals. MCP summaries retain complete
Evidence for filtered metadata, tensor and type-total pages; CLI compact delivery
uses `--detail summary --evidence_output FILE`. See [model files](model-artifacts.md).
Inspect Unity IL2CPP metadata names without Unity or a disassembler:
```bash
+145
View File
@@ -0,0 +1,145 @@
# Model file declarations
Use `inspect_model_artifact` (CLI: `inspect-model-artifact`) to inspect a local
GGUF v2/v3 or safetensors file. Detection uses content, including GGUF byte
order, rather than its extension. REA does not load a model, interpret weights,
run inference, import Python, open external-data references or use the network.
```bash
rea inspect-model-artifact /absolute/path/model.gguf --json
rea inspect-model-artifact /absolute/path/model.safetensors --hash full --json
```
The CLI defaults to complete Evidence. For compact delivery with portable access
to every retained observation:
```bash
rea inspect-model-artifact /absolute/path/model.gguf \
--detail summary --evidence_output /absolute/path/model-evidence.json --json
```
## Identity and reads
The default `hash: "header"` reads only the metadata and tensor directory. The
result records the exact selected path, file size, header length, header SHA-256,
bytes read and read scope. GGUF alignment padding is outside this prefix.
`artifact.sha256` and the Evidence subject are null: a header digest is not a
commitment to the weights. Different files can have identical headers and sizes.
`hash: "full"` additionally streams the remaining file bytes through SHA-256.
It returns a full-file digest and Evidence subject, while retaining the same
header digest. It uses bounded buffers and does not interpret tensor data, but
it does read the whole file. Cost therefore grows with file size. Both modes
check descriptor and selected-path identity and metadata before completion;
this detects observed replacement or modification, not adversarial modification
that defeats filesystem metadata checks.
## Facts and selected queries
GGUF reports version, byte order, alignment, typed metadata, and each tensor's
name, original type ID, dimensions, relative data offset, file offset and derived
byte length. Alignment accepts every positive UINT32 multiple of eight, including
non-powers of two. Metadata keys use ASCII lower-snake-case hierarchy components;
standard numeric components such as `general.base_model.0.name` are preserved.
Scalar metadata is complete. Arrays have declared element types,
counts and at most eight preview elements per array, with an explicit
`preview_complete` flag. This includes tokenizer vocabularies: omitted values
are not silently represented as a complete array. Strings are UTF-8; 64-bit
integer metadata and all shape dimensions use decimal strings without rounding.
Nonfinite floating metadata uses `NaN`, `+Infinity` or `-Infinity` with its
declared float type; negative zero is `"-0"`. Unknown tensor types retain their numeric IDs and null byte
lengths; affected totals and storage validation remain unknown.
Safetensors reports the header extent, exact string `__metadata__` values, dtype,
shape and declared data ranges. JSON duplicate keys, invalid UTF-8, out-of-file
ranges, overlaps, gaps, trailing unindexed data and known dtype/shape byte-length
contradictions are rejected with a typed `file_format_rejected` constraint (public code `target_unavailable`, category `unsupported_target`). Empty
tensors and scalar shapes are supported; packed sub-byte types require whole
byte storage. Unknown dtypes retain declared ranges and an explicit limitation
that shape-to-storage validation is unavailable. Each tensor records
`storage_validation: "consistent" | "unknown"`; `coverage.storage_sizes` remains
partial when that consistency cannot be checked. Tensor filters accept these
`storage_validation` states independently of dtype or name.
Parameter counts and per-type totals are **derived from declarations**, not
measurements of effective parameters or inference behavior. Shared or repeated
weights are not deduplicated. GGUF dimensions retain their original fastest-first
order; safetensors shapes retain JSON order. Safetensors tensor rows use name
order; GGUF rows use directory order. An index identifies a declaration within
its collection and header digest, not tensor contents or a cross-version match.
GGUF supports zero extents and scalar shapes, with at most four dimensions;
known quantized types still require complete blocks in the first dimension.
MCP defaults to a compact summary with counts, selected short headline metadata,
chat-template presence and derived totals. The full type histogram and all rows
remain in complete Evidence. Headline metadata uses short keys/types and scalar
values; long or compound values remain available through explicit metadata
queries. Use the returned `parent_evidence_id`:
```json
{
"source": {
"kind": "retained-evidence",
"evidence_id": "<parent_evidence_id>"
},
"view": {
"kind": "model-artifact",
"collection": "tensors",
"names": ["token_embd.weight", "output.weight"],
"fields": ["name", "type", "shape", "file_offset", "byte_length"],
"offset": 0,
"limit": 32
}
}
```
To answer a known question in the initial inspection, pass the same selection
without `kind` as `inspect_model_artifact`'s optional `view`, for example
`{"path":"/absolute/path/model.gguf","view":{"collection":"tensors","names":["token_embd.weight"],"fields":["type","shape","file_offset"]}}`.
This returns one selected page with compact model context and retains the full
source Evidence, avoiding another tool call. `view` requires summary delivery;
complete detail and a selection are rejected as conflicting choices.
CLI uses `--view '{"collection":"tensors","names":["token_embd.weight"]}'`
with `--detail summary --evidence_output FILE`.
Call `inspect_analysis_view` with the historical query object above for further pages. `collection: "metadata"` accepts
exact `keys`, indices, fields and paging; use it for `tokenizer.chat_template`
or tokenizer array previews. `collection: "types"` pages derived storage and
parameter totals and accepts exact `types`. Tensor pages also accept `types`.
Filters intersect; omitted filters mean all recorded rows and empty arrays mean
none. Every row retains its source-scoped index. Names, keys and type labels
follow field selection; `fields: []` returns only indices. Omitted fields return
all recorded columns. Compact warnings use the shared bounded view budget and
link to complete parent Evidence when messages are omitted. Page offsets count filtered
matches, while row indices retain their original scope. Each page carries source
identity, header coverage, array omissions, unknowns and parent Evidence links.
Pages read historical retained facts without reopening the model. Complete
Evidence can also be supplied inline or as a single authenticated Evidence JSON
record through `source: {"kind": "evidence-file", "path": "/absolute/path/model-record.json"}`.
CLI Evidence bundles contain a `records` array; import the bundle with
`import_evidence_bundle` before using retained queries, or save one complete
record as a standalone Evidence file. A bundle is not a single Evidence record.
The session owns validated indexes until that Evidence is released; there is no
live-file result cache or automatic freshness claim.
Imported reports must retain positional indices and unique names, keys and
type identities; malformed reports return a typed input error before indexing.
## Bounds and unsupported inputs
The reader admits one active inspection per provider. It enforces a 100,000,000
byte header ceiling (safetensors counts its declared JSON bytes separately from
the fixed eight-byte length prefix; GGUF counts the complete header) and a separate 32 MiB
retained representation budget before retaining rows or expanding strings.
Numeric products use exact integer arithmetic. GGUF follows its four-dimensional
tensor layout and limits metadata array nesting to 32 frames. Resource refusals
are typed, rather than returned as complete empty inventories. Cancellation
joins descriptor finalization; failed cleanup remains owned for retry.
PyTorch pickle and ZIP model deserialization are unsupported; convert using a
trusted external workflow and supply GGUF or safetensors. ONNX, TFLite, Core ML,
model execution and conversion are outside this operation.
Format references: [GGUF specification](https://github.com/ggml-org/ggml/blob/da8e2a54b919557b816b1c2beb432ccb766f7ce0/docs/gguf.md),
[ggml tensor layouts](https://github.com/ggml-org/ggml/blob/da8e2a54b919557b816b1c2beb432ccb766f7ce0/src/ggml-common.h),
[safetensors format](https://github.com/huggingface/safetensors/tree/e246a2560645b7525f5775669ed816eb57c5bcc8#format).
+1
View File
@@ -266,6 +266,7 @@ npx rea-agents@latest setup
- [التثبيت والإعداد](../installation.md): تسجيل الوكلاء، وتهيئة الموفّرين، والتحديثات، وإلغاء التثبيت.
- [الجاهزية واستكشاف الأخطاء](../installation.md#check-readiness-for-your-task): تشخيص وكيل أو محرك تحليل محدد.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI وEvidence](../cli.md): الأوامر، واختيار الموفّر، واللقطات، والاستيراد/التصدير، وحالات الخروج.
- [عقود MCP](../mcp-contracts.md) و[تعليمات الوكلاء](../mcp-prompts.md): نتائج الأدوات، والجلسات، والاستقصاءات الموجّهة.
- [كتالوج الأدوات](../mcp-contracts.md#generated-catalog): قائمة الأدوات والموفّرين وأوامر CLI المولّدة أثناء البناء.
+1
View File
@@ -264,6 +264,7 @@ Beginne mit den [praxisnahen Anleitungen](https://rea.tools/de/guides/) auf der
- [Installation und Einrichtung](../installation.md): Agentenregistrierung, Provider-Konfiguration, Updates und Deinstallation.
- [Bereitschaft und Fehlerbehebung](../installation.md#check-readiness-for-your-task): Diagnose eines bestimmten Agenten oder einer Analyse-Engine.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI und Evidence](../cli.md): Befehle, Provider-Auswahl, Snapshots, Import/Export und Exit-Status.
- [MCP-Kontrakte](../mcp-contracts.md) und [Agenten-Prompts](../mcp-prompts.md): Werkzeugergebnisse, Sitzungen und geführte Untersuchungen.
- [Werkzeugkatalog](../mcp-contracts.md#generated-catalog): Beim Build erzeugtes Inventar von Werkzeugen, Providern und CLI-Befehlen.
+1
View File
@@ -277,6 +277,7 @@ npx rea-agents@latest setup
- [Εγκατάσταση και ρύθμιση](../installation.md): καταχώριση agents, ρύθμιση παρόχων ανάλυσης, ενημερώσεις και απεγκατάσταση.
- [Έλεγχος ετοιμότητας και αντιμετώπιση προβλημάτων](../installation.md#check-readiness-for-your-task): διάγνωση ενός agent ή μιας μηχανής ανάλυσης.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI και Evidence](../cli.md): εντολές, επιλογή παρόχου, στιγμιότυπα, εισαγωγή/εξαγωγή και κωδικοί εξόδου.
- [Προδιαγραφές MCP](../mcp-contracts.md) και [prompts για agents](../mcp-prompts.md): αποτελέσματα εργαλείων, συνεδρίες και καθοδηγούμενες διερευνήσεις.
- [Κατάλογος εργαλείων](../mcp-contracts.md#generated-catalog): αυτόματα παραγόμενος κατάλογος εργαλείων, παρόχων και εντολών CLI.
+1
View File
@@ -264,6 +264,7 @@ Empieza con las [guías prácticas](https://rea.tools/es/guides/) del sitio web.
- [Instalación y configuración](../installation.md): registro de agentes, configuración de proveedores, actualizaciones y desinstalación.
- [Preparación y resolución de problemas](../installation.md#check-readiness-for-your-task): diagnóstico de un agente o motor de análisis concreto.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI y Evidence](../cli.md): comandos, selección de proveedores, instantáneas, importación/exportación y estados de salida.
- [Contratos de MCP](../mcp-contracts.md) y [prompts para agentes](../mcp-prompts.md): resultados de herramientas, sesiones e investigaciones guiadas.
- [Catálogo de herramientas](../mcp-contracts.md#generated-catalog): inventario de herramientas, proveedores y comandos de CLI generado durante la compilación.
+1
View File
@@ -249,6 +249,7 @@ npx rea-agents@latest setup
- [نصب و راه‌اندازی](../installation.md): ثبت ایجنت، پیکربندی ارائه‌دهنده، به‌روزرسانی و حذف نصب.
- [آمادگی و رفع اشکال](../installation.md#check-readiness-for-your-task): عیب‌یابی یک ایجنت یا موتور تحلیل.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI و شواهد](../cli.md): دستورات، انتخاب ارائه‌دهنده، snapshotها، ورود/خروج داده و وضعیت‌های خروج.
- [قراردادهای MCP](../mcp-contracts.md) و [پرامپت‌های ایجنت](../mcp-prompts.md): نتایج ابزارها، نشست‌ها و بررسی‌های هدایت‌شده.
- [فهرست ابزارها](../mcp-contracts.md#generated-catalog): فهرست تولیدشده هنگام build از ابزارها، ارائه‌دهندگان و دستورات CLI.
+1
View File
@@ -268,6 +268,7 @@ Commencez par les [guides pas à pas](https://rea.tools/fr/guides/) du site web.
- [Installation et configuration](../installation.md) : enregistrement des agents, configuration des fournisseurs, mises à jour et désinstallation.
- [Vérification et dépannage](../installation.md#check-readiness-for-your-task) : diagnostiquer un agent ou un moteur d'analyse donné.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI et Evidence](../cli.md) : commandes, choix du fournisseur, snapshots, import/export et codes de sortie.
- [Contrats MCP](../mcp-contracts.md) et [prompts pour agents](../mcp-prompts.md) : résultats des outils, sessions et investigations guidées.
- [Catalogue des outils](../mcp-contracts.md#generated-catalog) : inventaire des outils, fournisseurs et commandes CLI, généré lors du build.
+1
View File
@@ -289,6 +289,7 @@ Untuk mengetahui opsi, prasyarat, dan kontrak hasil secara tepat:
- [Instalasi dan penyiapan](../installation.md): pendaftaran agen, konfigurasi penyedia, pembaruan, dan penghapusan instalasi.
- [Kesiapan dan pemecahan masalah](../installation.md#check-readiness-for-your-task): mendiagnosis satu agen atau mesin analisis.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI dan Bukti](../cli.md): perintah, pemilihan penyedia, snapshot, impor/ekspor, dan status keluar.
- [Kontrak MCP](../mcp-contracts.md) dan [prompt agen](../mcp-prompts.md): hasil alat, sesi, dan investigasi terpandu.
- [Katalog alat](../mcp-contracts.md#generated-catalog): inventaris alat, penyedia, dan perintah CLI yang dibuat secara otomatis.
+1
View File
@@ -268,6 +268,7 @@ Inizia dalle [guide pratiche](https://rea.tools/it/guides/) del sito web. Per op
- [Installazione e configurazione](../installation.md): registrazione agente, configurazione provider, aggiornamenti e disinstallazione.
- [Pronto all'uso e risoluzione dei problemi](../installation.md#check-readiness-for-your-task): diagnostica un agente o un motore di analisi.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI ed Evidence](../cli.md): comandi, selezione provider, snapshot, import/export e stati di uscita.
- [Contratti MCP](../mcp-contracts.md) e [prompt agente](../mcp-prompts.md): risultati degli strumenti, sessioni e indagini guidate.
- [Catalogo strumenti](../mcp-contracts.md#generated-catalog): inventario generato di strumenti, provider e comandi CLI.
+1
View File
@@ -266,6 +266,7 @@ npx rea-agents@latest setup
- [インストールとセットアップ](../installation.md):エージェントの登録、プロバイダーの設定、更新、アンインストール。
- [準備状況の確認とトラブルシューティング](../installation.md#check-readiness-for-your-task):特定のエージェントや分析エンジンを診断。
- [GGUF / safetensors](../model-artifacts.md)
- [CLI と Evidence](../cli.md):コマンド、プロバイダーの選択、スナップショット、インポート/エクスポート、終了ステータス。
- [MCP 契約](../mcp-contracts.md)と[エージェント向けプロンプト](../mcp-prompts.md):ツールの結果、セッション、ガイド付き調査。
- [ツールカタログ](../mcp-contracts.md#generated-catalog):ビルド時に生成されるツール、プロバイダー、CLI コマンドの一覧。
+1
View File
@@ -279,6 +279,7 @@ npx rea-agents@latest setup
- [설치 및 설정](../installation.md): 에이전트 등록, 제공자 설정, 업데이트, 제거.
- [준비 상태 확인 및 문제 해결](../installation.md#check-readiness-for-your-task): 특정 에이전트 또는 분석 엔진 진단.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI 및 Evidence](../cli.md): 명령, 제공자 선택, 스냅샷, 가져오기/내보내기, 종료 상태.
- [MCP 계약](../mcp-contracts.md) 및 [에이전트 프롬프트](../mcp-prompts.md): 도구 결과, 세션, 안내에 따른 조사.
- [도구 카탈로그](../mcp-contracts.md#generated-catalog): 빌드 시 생성되는 도구, 제공자, CLI 명령 목록.
+1
View File
@@ -282,6 +282,7 @@ npx rea-agents@latest setup
- [Installation and setup](../installation.md): agent registration, provider configuration, updates, uninstall.
- [Readiness and troubleshooting](../installation.md#check-readiness-for-your-task): ഒരു agent-ന്റെയോ analysis engine-ന്റെയോ പ്രശ്നങ്ങൾ കണ്ടെത്തുക.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI and Evidence](../cli.md): commands, provider തിരഞ്ഞെടുക്കൽ, snapshots, import/export, exit statuses.
- [MCP contracts](../mcp-contracts.md), [agent prompts](../mcp-prompts.md): tool results, sessions, guided investigations.
- [Tool catalog](../mcp-contracts.md#generated-catalog): tools, providers, CLI commands എന്നിവയുടെ സ്വയമേവ സൃഷ്ടിച്ച പട്ടിക.
+1
View File
@@ -264,6 +264,7 @@ Zacznij od [przewodników z przykładami](https://rea.tools/pl/guides/) na stron
- [Instalacja i konfiguracja](../installation.md): rejestracja agentów, konfiguracja dostawców analizy, aktualizacje i odinstalowanie.
- [Gotowość i rozwiązywanie problemów](../installation.md#check-readiness-for-your-task): diagnostyka jednego agenta lub silnika analizy.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI i Evidence](../cli.md): polecenia, wybór dostawcy, migawki, import/eksport i kody zakończenia.
- [Kontrakty MCP](../mcp-contracts.md) i [prompty dla agentów](../mcp-prompts.md): wyniki narzędzi, sesje i prowadzone analizy.
- [Katalog narzędzi](../mcp-contracts.md#generated-catalog): generowany podczas kompilacji wykaz narzędzi, dostawców i poleceń CLI.
+1
View File
@@ -264,6 +264,7 @@ Comece pelos [guias práticos](https://rea.tools/pt-BR/guides/) do site. Para op
- [Instalação e configuração](../installation.md): registro de agentes, configuração de provedores, atualizações e desinstalação.
- [Prontidão e solução de problemas](../installation.md#check-readiness-for-your-task): diagnóstico de um agente ou motor de análise específico.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI e Evidence](../cli.md): comandos, seleção de provedores, snapshots, importação/exportação e status de saída.
- [Contratos MCP](../mcp-contracts.md) e [prompts para agentes](../mcp-prompts.md): resultados das ferramentas, sessões e investigações guiadas.
- [Catálogo de ferramentas](../mcp-contracts.md#generated-catalog): inventário de ferramentas, provedores e comandos da CLI gerado durante o build.
+1
View File
@@ -264,6 +264,7 @@ npx rea-agents@latest setup
- [Установка и настройка](../installation.md): регистрация агентов, настройка провайдеров, обновление и удаление.
- [Готовность и устранение неполадок](../installation.md#check-readiness-for-your-task): диагностика конкретного агента или движка анализа.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI и Evidence](../cli.md): команды, выбор провайдера, снимки, импорт/экспорт и коды завершения.
- [Контракты MCP](../mcp-contracts.md) и [промпты агентов](../mcp-prompts.md): результаты инструментов, сеансы и исследования с подсказками.
- [Каталог инструментов](../mcp-contracts.md#generated-catalog): создаваемый при сборке перечень инструментов, провайдеров и команд CLI.
+1
View File
@@ -264,6 +264,7 @@ npx rea-agents@latest setup
- [การติดตั้งและตั้งค่า](../installation.md): การลงทะเบียนเอเจนต์ การตั้งค่าผู้ให้บริการ การอัปเดต และการถอนการติดตั้ง
- [ความพร้อมใช้งานและการแก้ไขปัญหา](../installation.md#check-readiness-for-your-task): ตรวจวินิจฉัยเอเจนต์หรือเอนจินวิเคราะห์หนึ่งตัว
- [GGUF / safetensors](../model-artifacts.md)
- [CLI และ Evidence](../cli.md): คำสั่ง การเลือกผู้ให้บริการ สแนปช็อต การนำเข้า/ส่งออก และรหัสการสิ้นสุด
- [สัญญา MCP](../mcp-contracts.md) และ[พรอมป์ต์สำหรับเอเจนต์](../mcp-prompts.md): ผลลัพธ์เครื่องมือ เซสชัน และการตรวจสอบตามคำแนะนำ
- [แค็ตตาล็อกเครื่องมือ](../mcp-contracts.md#generated-catalog): รายการเครื่องมือ ผู้ให้บริการ และคำสั่ง CLI ที่สร้างระหว่างการบิลด์
+1
View File
@@ -269,6 +269,7 @@ Web sitesindeki [uygulamalı rehberlerden](https://rea.tools/tr/guides/) başlay
- [Yükleme ve kurulum](../installation.md): ajan kaydı, sağlayıcı yapılandırması, güncellemeler ve kaldırma.
- [Hazırlık ve sorun giderme](../installation.md#check-readiness-for-your-task): belirli bir ajanı veya analiz motorunu tanılama.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI ve Evidence](../cli.md): komutlar, sağlayıcı seçimi, anlık görüntüler, içe/dışa aktarma ve çıkış durumları.
- [MCP sözleşmeleri](../mcp-contracts.md) ve [ajan istemleri](../mcp-prompts.md): araç sonuçları, oturumlar ve yönlendirmeli araştırmalar.
- [Araç kataloğu](../mcp-contracts.md#generated-catalog): derleme sırasında oluşturulan araç, sağlayıcı ve CLI komutu envanteri.
+1
View File
@@ -264,6 +264,7 @@ npx rea-agents@latest setup
- [Встановлення й налаштування](../installation.md): реєстрація агентів, конфігурація провайдерів, оновлення та видалення.
- [Готовність та усунення неполадок](../installation.md#check-readiness-for-your-task): діагностика конкретного агента або рушія аналізу.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI та Evidence](../cli.md): команди, вибір провайдера, знімки, імпорт/експорт і статуси завершення.
- [Контракти MCP](../mcp-contracts.md) і [промпти агентів](../mcp-prompts.md): результати інструментів, сеанси й керовані дослідження.
- [Каталог інструментів](../mcp-contracts.md#generated-catalog): створюваний під час збірки перелік інструментів, провайдерів і команд CLI.
+1
View File
@@ -264,6 +264,7 @@ Bắt đầu với [hướng dẫn thực hành](https://rea.tools/vi/guides/) t
- [Cài đặt và thiết lập](../installation.md): đăng ký tác nhân, cấu hình trình phân tích, cập nhật và gỡ cài đặt.
- [Kiểm tra mức sẵn sàng và khắc phục sự cố](../installation.md#check-readiness-for-your-task): chẩn đoán một tác nhân hoặc bộ máy phân tích cụ thể.
- [GGUF / safetensors](../model-artifacts.md)
- [CLI và Evidence](../cli.md): lệnh, lựa chọn trình phân tích, ảnh chụp trạng thái, nhập/xuất và trạng thái thoát.
- [Hợp đồng MCP](../mcp-contracts.md) và [prompt cho tác nhân](../mcp-prompts.md): kết quả công cụ, phiên làm việc và điều tra có hướng dẫn.
- [Danh mục công cụ](../mcp-contracts.md#generated-catalog): danh sách công cụ, trình phân tích và lệnh CLI được tạo khi dựng dự án.
+1
View File
@@ -270,6 +270,7 @@ npx rea-agents@latest setup
- [安裝與設定](../installation.md):AI 代理註冊、提供者組態、更新和卸載。
- [就緒檢查與故障排查](../installation.md#check-readiness-for-your-task):診斷特定 AI 代理或分析引擎。
- [GGUF / safetensors](../model-artifacts.md)
- [CLI 與 Evidence](../cli.md):命令、提供者選擇、快照、匯入/匯出和退出狀態。
- [MCP 契約](../mcp-contracts.md)和[AI 代理提示詞](../mcp-prompts.md):工具結果、會話和引導式調查。
- [工具目錄](../mcp-contracts.md#generated-catalog):建置時生成的工具、提供者和 CLI 命令清單。
+1
View File
@@ -270,6 +270,7 @@ npx rea-agents@latest setup
- [安装与设置](../installation.md):智能体注册、提供方配置、更新和卸载。
- [就绪检查与故障排查](../installation.md#check-readiness-for-your-task):诊断特定智能体或分析引擎。
- [GGUF / safetensors](../model-artifacts.md)
- [CLI 与 Evidence](../cli.md):命令、提供方选择、快照、导入/导出和退出状态。
- [MCP 契约](../mcp-contracts.md)和[智能体提示词](../mcp-prompts.md):工具结果、会话和引导式调查。
- [工具目录](../mcp-contracts.md#generated-catalog):构建时生成的工具、提供方和 CLI 命令清单。
+5
View File
@@ -35,6 +35,7 @@ export const toolFamilyCatalog = (sources) => {
...sources.evmContracts.EVM_TOOL_CONTRACTS,
...sources.goContracts.GO_TOOL_CONTRACTS,
...sources.il2cppContracts.IL2CPP_TOOL_CONTRACTS,
...sources.modelContracts.MODEL_TOOL_CONTRACTS,
...sources.rustContracts.RUST_TOOL_CONTRACTS,
...sources.wasmContracts.WASM_TOOL_CONTRACTS,
...sources.sqliteContracts.SQLITE_TOOL_CONTRACTS,
@@ -271,6 +272,10 @@ export const providerCatalog = (sources) => {
identity: sources.il2cppProvider.IL2CPP_METADATA_PROVIDER_IDENTITY,
contracts: sources.il2cppContracts.IL2CPP_TOOL_CONTRACTS,
},
{
identity: sources.modelProvider.MODEL_ARTIFACT_PROVIDER_IDENTITY,
contracts: sources.modelContracts.MODEL_TOOL_CONTRACTS,
},
{
identity: sources.rustProvider.RUST_BINARY_PROVIDER_IDENTITY,
contracts: sources.rustContracts.RUST_TOOL_CONTRACTS,
+2
View File
@@ -69,6 +69,8 @@ const SOURCE_PATHS = {
evmProvider: "dist/evm/EvmoleRelease.js",
goContracts: "dist/contracts/go/goToolContracts.js",
goProvider: "dist/go/GoBinaryProvider.js",
modelContracts: "dist/contracts/model/modelToolContracts.js",
modelProvider: "dist/model/ModelArtifactProvider.js",
il2cppContracts: "dist/contracts/il2cpp/il2cppToolContracts.js",
il2cppProvider: "dist/il2cpp/Il2cppMetadataProvider.js",
rustContracts: "dist/contracts/rust/rustToolContracts.js",
+1
View File
@@ -9,6 +9,7 @@ const MIGRATED_PROVIDER_ROOTS = new Set([
"evm",
"go",
"il2cpp",
"model",
"rust",
"wasm",
"captures",
+1
View File
@@ -414,6 +414,7 @@ const workflowAvailabilityFor = ({
name === "inspect_desktop_application" ||
name === "inspect_go_binary" ||
name === "inspect_il2cpp_metadata" ||
name === "inspect_model_artifact" ||
name === "inspect_rust_binary" ||
name === "inspect_electron_fuses"
)
@@ -22,6 +22,7 @@ import {
inspectAnalysisViewInputSchema,
projectAnalysisView,
type AnalysisViewResult,
type AnalysisViewRequest,
type InspectAnalysisViewInput,
} from "../../domain/analysisView/analysisView.js";
import { jsonValueSchema } from "../../domain/jsonValue.js";
@@ -233,6 +234,14 @@ export const requireAnalysisViewRetention = (
export const summarizeRetainedAnalysis = (
evidence: Evidence,
retention: AnalysisViewRetention,
): Result<Evidence, AnalysisError> =>
projectRetainedAnalysis(evidence, retention, { kind: "summary" });
/** Retain the complete owner observation and deliver one explicitly selected view. */
export const projectRetainedAnalysis = (
evidence: Evidence,
retention: AnalysisViewRetention,
view: AnalysisViewRequest,
): Result<Evidence, AnalysisError> => {
const ready = requireAnalysisViewRetention(evidence.operation, retention);
if (!ready.ok) return ready;
@@ -241,7 +250,7 @@ export const summarizeRetainedAnalysis = (
return inspectAnalysisViewValidated(
{
source: { kind: "retained-evidence", evidence_id: evidence.evidence_id },
view: { kind: "summary" },
view,
},
ready.value.evidenceById,
);
@@ -0,0 +1,34 @@
import type { ProviderIdentity } from "../AnalysisProvider.js";
import type {
ModelArtifact,
ModelArtifactInput,
} from "../../domain/model/modelArtifact.js";
import { createEvidence, type Evidence } from "../../domain/evidence.js";
/** One Evidence interpretation for success and observations retained by file finalization failures. */
export const createModelArtifactEvidence = (
report: ModelArtifact,
provider: ProviderIdentity,
parameters: ModelArtifactInput,
): Evidence => {
const source = report.artifact;
return createEvidence(
source.sha256 === null
? undefined
: { path: source.path, sha256: source.sha256, format: "file" },
provider,
{
operation: "inspect_model_artifact",
parameters: { path: parameters.path, hash: parameters.hash },
result: report,
confidence: "observed",
subjectUnavailableReason:
"Full-file identity is unknown in header-only mode; normalized_result.artifact records a scoped header SHA-256 and exact source path/extent.",
locations: [
{ kind: "artifact-path", path: source.path },
{ kind: "file-offset-range", start: 0, end: source.header_length },
],
limitations: report.limitations,
},
);
};
@@ -0,0 +1,20 @@
import type {
ExecutionOptions,
ProviderIdentity,
} from "../AnalysisProvider.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import type {
ModelArtifact,
ModelArtifactInput,
} from "../../domain/model/modelArtifact.js";
import type { Result } from "../../domain/result.js";
/** Read-only, local model declarations; no model runtime, target execution or downloads. */
export interface ModelArtifactPort {
readonly identity: ProviderIdentity;
inspect(
input: ModelArtifactInput,
options?: ExecutionOptions,
): Promise<Result<ModelArtifact, AnalysisError>>;
close(): Promise<void>;
}
@@ -0,0 +1,99 @@
import { inspectModelArtifactInputSchema } from "../../domain/model/modelArtifactQuery.js";
import type { ModelArtifactPort } from "./ModelArtifactPort.js";
import type { ExecutionOptions } from "../AnalysisProvider.js";
import { modelArtifactSchema } from "../../domain/model/modelArtifact.js";
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import {
AnalysisCancelledError,
AnalysisOutputError,
} from "../../domain/analysisErrorCore.js";
import type { Evidence } from "../../domain/evidence.js";
import { createModelArtifactEvidence } from "./ModelArtifactEvidence.js";
import { err, ok, type Result } from "../../domain/result.js";
import {
requireAnalysisViewRetention,
projectRetainedAnalysis,
type AnalysisViewRetention,
} from "../analysisView/AnalysisViewService.js";
/** Shared input, Evidence and retained delivery boundary for CLI, MCP and SDK callers. */
export class ModelArtifactService {
constructor(readonly provider: ModelArtifactPort) {}
close(): Promise<void> {
return this.provider.close();
}
async inspect(
raw: unknown,
options?: ExecutionOptions,
retention: AnalysisViewRetention = {
evidenceById: undefined,
recordEvidence: undefined,
},
): Promise<Result<Evidence, AnalysisError>> {
const input = inspectModelArtifactInputSchema.safeParse(raw);
if (!input.success)
return err(
analysisInputErrorFromIssues(
"inspect_model_artifact",
input.error.issues,
raw,
{ cause: input.error },
),
);
if (options?.signal?.aborted)
return err(
new AnalysisCancelledError("inspect_model_artifact", {
cause: options.signal.reason,
}),
);
const { detail, view, ...parameters } = input.data;
if (detail === "summary") {
const ready = requireAnalysisViewRetention(
"inspect_model_artifact",
retention,
);
if (!ready.ok) return ready;
}
const inspected = await this.provider.inspect(parameters, options);
if (!inspected.ok) return inspected;
const checked = modelArtifactSchema.safeParse(inspected.value);
if (
!checked.success ||
checked.data.artifact.path !== parameters.path ||
checked.data.artifact.read_scope !==
(parameters.hash === "full" ? "full-file" : "header") ||
(parameters.hash === "header"
? checked.data.artifact.sha256 !== null
: checked.data.artifact.sha256 === null)
)
return err(
new AnalysisOutputError(
"inspect_model_artifact",
"Model reader changed the selected input or returned invalid facts",
),
);
const report = checked.data;
const evidence = createModelArtifactEvidence(
report,
this.provider.identity,
parameters,
);
if (options?.signal?.aborted)
return err(
new AnalysisCancelledError("inspect_model_artifact", {
cause: options.signal.reason,
partialObservation: evidence,
}),
);
return detail === "summary"
? projectRetainedAnalysis(
evidence,
retention,
view === undefined
? { kind: "summary" }
: { kind: "model-artifact", ...view },
)
: ok(evidence);
}
}
+2
View File
@@ -7,6 +7,7 @@ import { createDirectAnalysis } from "./composition/directAnalysis.js";
import { registerEvmCommands } from "./cli/evmCommands.js";
import { registerDesktopApplicationCommands } from "./cli/desktopApplicationCommands.js";
import { registerGoCommands } from "./cli/goCommands.js";
import { registerModelCommands } from "./cli/modelCommands.js";
import { registerIl2cppCommands } from "./cli/il2cppCommands.js";
import { registerRustCommands } from "./cli/rustCommands.js";
import { registerSqliteCommands } from "./cli/sqliteCommands.js";
@@ -122,6 +123,7 @@ export const createCli = (
registerDesktopApplicationCommands(cli, logger);
registerGoCommands(cli, logger);
registerIl2cppCommands(cli, logger);
registerModelCommands(cli, logger);
registerRustCommands(cli, logger);
registerSqliteCommands(cli, logger, environment);
registerArtifactCommands(cli, logger, analysis.runProviderAnalysis);
+4 -1
View File
@@ -1,3 +1,4 @@
import type { AnalysisViewRequest } from "../domain/analysisView/analysisView.js";
import { z } from "incur";
import { parseEvidence, type Evidence } from "../domain/evidence.js";
import type { JsonValue } from "../domain/jsonValue.js";
@@ -34,6 +35,7 @@ const isEvidence = (value: JsonValue): value is Evidence =>
"normalized_result" in value;
interface CliEvidenceDelivery {
readonly view?: AnalysisViewRequest;
readonly providerId: string;
readonly operation: string;
readonly signal: AbortSignal;
@@ -95,6 +97,7 @@ export const deliverCliAnalysisEvidence = async (
signal,
detail,
evidenceOutput,
...(input.view === undefined ? {} : { view: input.view }),
});
};
@@ -139,7 +142,7 @@ export const publishCliAnalysisEvidence = async (
return detail === "summary"
? inspectAnalysisView({
source: { kind: "inline", evidence: complete },
view: { kind: "summary" },
view: input.view ?? { kind: "summary" },
})
: { ok: true as const, value: complete };
},
+105
View File
@@ -0,0 +1,105 @@
import { inspectModelArtifactInputSchema } from "../domain/model/modelArtifactQuery.js";
import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js";
import { AnalysisInputError } from "../domain/analysisErrorCore.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import { z } from "incur";
import type { Logger } from "pino";
import type { CliInstance } from "./types.js";
import { absoluteCallerPath } from "../filesystem/callerPath.js";
import { createModelArtifactService } from "../composition/model.js";
import { CLI_COMMANDS } from "../cliCommandNames.js";
import { logCliCommand } from "../cliLogging.js";
import { withCommandCancellation } from "./commandCancellation.js";
import {
analysisEvidenceDeliveryOptions,
deliverCliAnalysisEvidence,
} from "./analysisEvidenceDelivery.js";
/** One-shot summaries export the same complete observations used by MCP retained pages. */
export const registerModelCommands = (
cli: CliInstance,
logger: Logger,
): void => {
cli.command(CLI_COMMANDS.inspectModelArtifact, {
description:
"Inspect GGUF or safetensors headers without loading model weights",
args: z.object({
path: z.string().describe("Local model file, detected by content"),
}),
options: z.object({
...analysisEvidenceDeliveryOptions,
view: z
.string()
.optional()
.describe(
"JSON model page selection; requires --detail summary and --evidence_output",
),
hash: z
.enum(["header", "full"])
.default("header")
.describe(
"header reads declarations only; full additionally streams all bytes for SHA-256",
),
}),
run: ({ args, options }) =>
withCommandCancellation((signal) =>
logCliCommand(logger, CLI_COMMANDS.inspectModelArtifact, async () => {
let view: unknown;
try {
view =
options.view === undefined ? undefined : JSON.parse(options.view);
} catch (cause) {
return projectAnalysisError(
new AnalysisInputError("inspect_model_artifact", { cause }, [
{
path: ["view"],
reason: "invalid_format",
message: "view must be a JSON model page selection",
},
]),
);
}
const selected = inspectModelArtifactInputSchema.safeParse({
path: absoluteCallerPath(args.path),
hash: options.hash,
detail: options.detail,
...(view === undefined ? {} : { view }),
});
if (!selected.success)
return projectAnalysisError(
analysisInputErrorFromIssues(
"inspect_model_artifact",
selected.error.issues,
{ view },
),
);
const service = createModelArtifactService();
return deliverCliAnalysisEvidence({
run: () =>
service.inspect(
{
path: absoluteCallerPath(args.path),
hash: options.hash,
detail: "complete",
},
{ signal },
),
service,
providerId: "model-artifact",
operation: "inspect_model_artifact",
signal,
detail: options.detail,
evidenceOutput: options.evidence_output,
...(selected.data.view === undefined
? {}
: {
view: {
kind: "model-artifact" as const,
...selected.data.view,
},
}),
});
}),
),
});
};
+1
View File
@@ -61,6 +61,7 @@ export const CLI_COMMANDS = Object.freeze({
inspectDesktopApplication: "inspect-desktop-application",
inspectGoBinary: "inspect-go-binary",
inspectIl2cppMetadata: "inspect-il2cpp-metadata",
inspectModelArtifact: "inspect-model-artifact",
inspectRustBinary: "inspect-rust-binary",
inspectPeResources: "inspect-pe-resources",
inspectAnalysisView: "inspect-analysis-view",
+6
View File
@@ -0,0 +1,6 @@
import { ModelArtifactService } from "../application/model/ModelArtifactService.js";
import { ModelArtifactProvider } from "../model/ModelArtifactProvider.js";
/** Construct the local reader without acquiring files, processes or network resources. */
export const createModelArtifactService = (): ModelArtifactService =>
new ModelArtifactService(new ModelArtifactProvider());
+1 -1
View File
@@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
...toolContractMetadata("inspect_analysis_view"),
kind: "application",
description:
'Inspect selected views of completed Go build metadata, binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot, inspect_il2cpp_metadata or inspect_web_network_capture Evidence without repeating analysis. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized non-entity paths written by POST/PUT/PATCH (item-shaped and delete-only paths stay recorded), and a render plan per read tool: item and list views reproduce their latest recordings from the state, and every other read tool is a recorded view with a reason. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies; its values keep the capture redaction markers, and a write whose URL it redacted ends the state of every type. For a retained record use source: {kind: "retained-evidence", evidence_id: "<parent_evidence_id>"}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: <complete Evidence>}. A completed Evidence JSON file on this host uses {kind: "evidence-file", path: "<absolute path>"}: REA reads and authenticates it, then projects it like retained Evidence, so a large capture is bounded by the file reader rather than by inline-input accounting. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/nLine truncated
'Inspect selected views of completed Go build metadata, binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot, inspect_il2cpp_metadata, inspect_model_artifact or inspect_web_network_capture Evidence without repeating analysis. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized non-entity paths written by POST/PUT/PATCH (item-shaped and delete-only paths stay recorded), and a render plan per read tool: item and list views reproduce their latest recordings from the state, and every other read tool is a recorded view with a reason. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies; its values keep the capture redaction markers, and a write whose URL it redacted ends the state of every type. For a retained record use source: {kind: "retained-evidence", evidence_id: "<parent_evidence_id>"}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: <complete Evidence>}. A completed Evidence JSON file on this host uses {kind: "evidence-file", path: "<absolute path>"}: REA reads and authenticates it, then projects it like retained Evidence, so a large capture is bounded by the file reader rather than by inline-input accounting. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, toolLine truncated
inputSchema: inspectAnalysisViewInputSchema,
outputSchema: evidenceResultOf(analysisViewResultSchema),
examples: [
+38
View File
@@ -0,0 +1,38 @@
import { inspectModelArtifactInputSchema } from "../../domain/model/modelArtifactQuery.js";
import { z } from "zod";
import { modelArtifactSchema } from "../../domain/model/modelArtifact.js";
import {
modelArtifactPageViewResultSchema,
modelArtifactSummaryViewResultSchema,
} from "../../domain/analysisView/analysisView.js";
import type { ToolContract } from "../toolContractTypes.js";
import { toolContractMetadata } from "../toolEffects.js";
import { evidenceResultOf } from "../toolOutputSchemaPrimitives.js";
export const MODEL_TOOL_CONTRACTS = [
{
name: "inspect_model_artifact",
...toolContractMetadata("inspect_model_artifact"),
kind: "artifact-provider",
description:
"Inspect local GGUF v2/v3 or safetensors declarations by content, without loading models or executing code. Default hash header reads only the header: full-file SHA-256 and Evidence subject are unknown; artifact.header_sha256 is a scoped prefix commitment, not weight identity. hash full additionally streams all file bytes for SHA-256. Optional view returns a selected metadata/tensor/type page plus model context in this call. Summary retains complete Evidence for inspect_analysis_view kind model-artifact, collection metadata (keys/indices/fields) or tensors (names/types/indices/fields), with stable header-scoped indices and paging. Metadata arrays retain eight-element typed previews; scalar metadata is complete. Tensor counts, shapes, types and offsets are observed; parameter/storage totals are derived from declarations. Unknown GGUF types retain IDs with unknown byte lengths. Tensor payload, model behavior and inference quality are not inspected. One active inspection, 100 MB header ceiling and 32 MiB retained-facts budget; resource refusals are explicit. No network, external files, Python, model runtime, extraction or deserialization of pickle.",
inputSchema: inspectModelArtifactInputSchema,
outputSchema: evidenceResultOf(
z.union([
modelArtifactSchema,
modelArtifactSummaryViewResultSchema,
modelArtifactPageViewResultSchema,
]),
),
examples: [
{
title: "Read model declarations without reading weights",
input: {
path: "/artifacts/model.gguf",
hash: "header",
detail: "summary",
},
},
],
},
] as const satisfies readonly ToolContract[];
+2
View File
@@ -4,6 +4,7 @@ import { WASM_TOOL_CONTRACTS } from "./wasm/wasmToolContracts.js";
import { EVM_TOOL_CONTRACTS } from "./evm/evmToolContracts.js";
import { DESKTOP_APPLICATION_TOOL_CONTRACTS } from "./desktopApplicationToolContracts.js";
import { GO_TOOL_CONTRACTS } from "./go/goToolContracts.js";
import { MODEL_TOOL_CONTRACTS } from "./model/modelToolContracts.js";
import { IL2CPP_TOOL_CONTRACTS } from "./il2cpp/il2cppToolContracts.js";
import { RUST_TOOL_CONTRACTS } from "./rust/rustToolContracts.js";
import { SQLITE_TOOL_CONTRACTS } from "./sqlite/sqliteToolContracts.js";
@@ -49,6 +50,7 @@ export const TOOL_CONTRACTS = [
...EVM_TOOL_CONTRACTS,
...GO_TOOL_CONTRACTS,
...IL2CPP_TOOL_CONTRACTS,
...MODEL_TOOL_CONTRACTS,
...RUST_TOOL_CONTRACTS,
...WASM_TOOL_CONTRACTS,
...SQLITE_TOOL_CONTRACTS,
+2
View File
@@ -66,6 +66,8 @@ const EXECUTION_WORDING_EXCEPTIONS: Readonly<Record<string, string>> = {
"inspects the operator-selected adb client; never starts its server",
list_adb_devices: "may start the local adb server, which REA does not own",
inspect_firmware_regions: "states there is no target execution",
inspect_model_artifact:
"states that model code and inference are never executed",
extract_firmware: "states that extracted code is never executed",
analyze_javascript_application: "reconstructs the graph without executing",
reconcile_javascript_runtime: "never invents execution claims",
+1
View File
@@ -128,6 +128,7 @@ export const TOOL_EFFECTS: Readonly<Record<string, ToolEffects>> = {
inspect_desktop_application: evidence,
inspect_go_binary: evidence,
inspect_il2cpp_metadata: evidence,
inspect_model_artifact: evidence,
inspect_rust_binary: evidence,
inspect_sqlite_database: effects({
mutatesSession: true,
+27
View File
@@ -1,3 +1,10 @@
import {
projectModelArtifactView,
modelArtifactSummarySchema,
modelArtifactViewRequestSchema,
modelArtifactViewItemsSchema,
modelArtifactViewContextSchema,
} from "./modelArtifactView.js";
import {
goBinarySummarySchema,
goBinaryViewContextSchema,
@@ -283,6 +290,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
pcapViewRequestSchema,
dartAotViewRequestSchema,
il2cppMetadataViewRequestSchema,
modelArtifactViewRequestSchema,
]);
// The application owns admission and authenticated Evidence parsing. SDK and
@@ -386,6 +394,7 @@ const viewResultBase = {
"inspect_pcap_capture",
"inspect_dart_aot",
"inspect_il2cpp_metadata",
"inspect_model_artifact",
...NATIVE_INVENTORY_OPERATIONS,
"match_native_functions",
]),
@@ -527,6 +536,21 @@ export const dartAotSummaryViewResultSchema = z.strictObject({
summary: dartAotSummarySchema,
});
export const modelArtifactSummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
parent_operation: z.literal("inspect_model_artifact"),
summary: modelArtifactSummarySchema,
});
export const modelArtifactPageViewResultSchema = z.strictObject({
kind: z.literal("model-artifact"),
view: modelArtifactViewRequestSchema,
items: modelArtifactViewItemsSchema,
...viewResultBase,
...modelArtifactViewContextSchema.shape,
parent_operation: z.literal("inspect_model_artifact"),
});
/** Metadata identity and row counts for selected IL2CPP delivery. */
export const il2cppMetadataSummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
@@ -610,6 +634,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
pcapCaptureSummaryViewResultSchema,
dartAotSummaryViewResultSchema,
il2cppMetadataSummaryViewResultSchema,
modelArtifactSummaryViewResultSchema,
nativeInventorySummaryViewResultSchema,
nativeFunctionMatchingSummaryViewResultSchema,
]),
@@ -756,6 +781,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
...dartAotViewContextSchema.shape,
parent_operation: z.literal("inspect_dart_aot"),
}),
modelArtifactPageViewResultSchema,
z.strictObject({
kind: z.literal("il2cpp-metadata"),
view: il2cppMetadataViewRequestSchema,
@@ -891,6 +917,7 @@ const retainedViewProjectors = new Map<string, typeof projectWasmArtifactView>([
["inspect_android_package", projectAndroidPackageView],
["inspect_dart_aot", projectDartAotView],
["inspect_il2cpp_metadata", projectIl2cppMetadataView],
["inspect_model_artifact", projectModelArtifactView],
["match_native_functions", projectNativeFunctionMatchingView],
["inspect_artifact", projectArtifactInventoryView],
["inventory_artifact", projectArtifactInventoryView],
@@ -0,0 +1,337 @@
import {
modelTypeRowSchema,
modelArtifactViewRequestSchema,
} from "../model/modelArtifactQuery.js";
export { modelArtifactViewRequestSchema } from "../model/modelArtifactQuery.js";
import { z } from "zod";
import {
modelArtifactSchema,
modelArtifactObjectSchema,
modelMetadataSchema,
modelTensorSchema,
modelSourceSchema,
type ModelArtifact,
} from "../model/modelArtifact.js";
import type { AnalysisError } from "../analysisErrorBase.js";
import { analysisViewInputError } from "./analysisViewErrors.js";
import { parseAnalysisViewResult } from "./analysisViewParent.js";
import { isImmutableJsonSnapshot } from "../immutableJson.js";
import type { JsonValue } from "../jsonValue.js";
import { err, ok, type Result } from "../result.js";
import type {
AnalysisViewParent,
AnalysisViewRequest,
UnsignedAnalysisView,
} from "./analysisView.js";
import {
completeWithinViewCoverage,
boundedViewMessages,
pageViewCoverage,
} from "./analysisViewCoverage.js";
const HEADLINE_TEXT_LIMIT = 512;
export const modelArtifactSummarySchema = modelArtifactObjectSchema
.omit({ metadata: true, tensors: true, totals: true })
.extend({
totals: modelArtifactObjectSchema.shape.totals.omit({ by_type: true }),
counts: z.strictObject({
metadata: z.number().int().nonnegative(),
tensors: z.number().int().nonnegative(),
types: z.number().int().nonnegative(),
}),
headlines: z.array(
modelMetadataSchema.omit({
source_offset: true,
source_length: true,
json_pointer: true,
}),
),
chat_template_present: z.boolean(),
});
const summarizeModel = (
report: ModelArtifact,
): z.infer<typeof modelArtifactSummarySchema> => {
const {
tensors,
metadata,
totals: { by_type, ...totals },
...identity
} = report;
const headlineKeys = new Set([
"general.architecture",
"general.name",
"general.file_type",
"tokenizer.ggml.model",
"format",
]);
const architecture = metadata.find(
(row) => row.key === "general.architecture",
)?.value;
if (
typeof architecture === "string" &&
architecture.length <= HEADLINE_TEXT_LIMIT - ".context_length".length
)
headlineKeys.add(`${architecture}.context_length`);
return {
...identity,
totals,
counts: {
tensors: tensors.length,
metadata: metadata.length,
types: by_type.length,
},
headlines: metadata
.filter(
(row) =>
headlineKeys.has(row.key) &&
row.element_count === null &&
row.element_type === null &&
row.type.length <= HEADLINE_TEXT_LIMIT &&
(row.value === null ||
typeof row.value === "boolean" ||
typeof row.value === "number" ||
(typeof row.value === "string" &&
row.value.length <= HEADLINE_TEXT_LIMIT)),
)
.map(
({
source_offset: _offset,
source_length: _length,
json_pointer: _pointer,
...row
}) => row,
),
chat_template_present: metadata.some(
(row) => row.key === "tokenizer.chat_template",
),
};
};
const modelContextSchema = modelArtifactSummarySchema.omit({
artifact: true,
coverage: true,
limitations: true,
});
export const modelArtifactViewItemsSchema = z.union([
z.array(modelTypeRowSchema.partial().required({ index: true })),
z.array(modelTensorSchema.partial().required({ index: true })),
z.array(modelMetadataSchema.partial().required({ index: true })),
]);
export const modelArtifactViewContextSchema = z.strictObject({
source: modelSourceSchema,
model: modelContextSchema,
source_coverage: modelArtifactObjectSchema.shape.coverage,
});
type Checked = {
report: ModelArtifact;
summary: z.infer<typeof modelArtifactSummarySchema>;
model: z.infer<typeof modelContextSchema>;
names: Map<string, number>;
keys: Map<string, number>;
typeRows: z.infer<typeof modelTypeRowSchema>[];
types: Map<string, number>;
};
const checked = new WeakMap<object, Checked>();
const read = (parent: AnalysisViewParent): Result<Checked, AnalysisError> => {
const key = parent.normalizedResult;
const reusable =
parent.sealedResult === true &&
key !== null &&
typeof key === "object" &&
isImmutableJsonSnapshot(key);
const cached = reusable ? checked.get(key) : undefined;
if (cached !== undefined) return ok(cached);
const parsed = parseAnalysisViewResult(modelArtifactSchema, parent);
if (!parsed.ok) return parsed;
const report = reusable ? (key as ModelArtifact) : parsed.value;
if (
parent.subject != null &&
(parent.subject.local_path !== report.artifact.path ||
parent.subject.digest.sha256 !== report.artifact.sha256)
)
return err(
analysisViewInputError([
{
path: ["source", "subject"],
reason: "invalid_value",
message:
"Model full-file subject does not match its recorded artifact",
},
]),
);
const summary = summarizeModel(report);
const {
artifact: _artifact,
coverage: _coverage,
limitations: _limitations,
...model
} = summary;
const value = {
report,
summary,
model,
typeRows: report.totals.by_type.map((row, index) => ({ index, ...row })),
types: new Map(
report.totals.by_type.map((row, index) => [row.type, index]),
),
names: new Map(report.tensors.map((row) => [row.name, row.index])),
keys: new Map(report.metadata.map((row) => [row.key, row.index])),
};
if (reusable) checked.set(key, value);
return ok(value);
};
/** Historical pages never reopen weights or report a scoped header digest as full-file identity. */
export const projectModelArtifactView = (
parent: AnalysisViewParent,
view: AnalysisViewRequest,
): Result<UnsignedAnalysisView, AnalysisError> => {
if (view.kind !== "summary" && view.kind !== "model-artifact")
return err(
analysisViewInputError([
{
path: ["view"],
reason: "invalid_value",
message:
"Model Evidence supports summary and model-artifact metadata/tensor pages",
},
]),
);
const parsed = read(parent);
if (!parsed.ok) return parsed;
const {
report,
summary,
model,
names,
keys,
types: typeIndices,
typeRows,
} = parsed.value;
const shared = {
parent_evidence_id: parent.evidenceId,
parent_operation: "inspect_model_artifact" as const,
parent_digest: parent.evidenceId.slice(3),
artifact: { path: report.artifact.path, sha256: report.artifact.sha256 },
limitations: boundedViewMessages(
parent.limitations,
parent.evidenceId,
"limitations",
),
unknowns: [
"Tensor values, runtime execution and inference quality were not inspected.",
...(report.artifact.sha256 === null
? [
"Full-file content identity is unknown; header_sha256 commits only to declarations.",
]
: []),
],
};
if (view.kind === "summary")
return ok({
kind: "summary",
view,
...shared,
summary: {
...summary,
limitations: boundedViewMessages(
summary.limitations,
parent.evidenceId,
"limitations",
),
},
coverage: completeWithinViewCoverage(1, 1),
});
const rows =
view.collection === "tensors"
? report.tensors
: view.collection === "metadata"
? report.metadata
: typeRows;
const selectedNames =
view.collection === "tensors"
? view.names
: view.collection === "metadata"
? view.keys
: view.types;
const lookup =
view.collection === "tensors"
? names
: view.collection === "metadata"
? keys
: typeIndices;
const indices =
view.indices === undefined ? undefined : new Set(view.indices);
const types =
view.collection === "tensors" && view.types !== undefined
? new Set(view.types)
: undefined;
const storageStates =
view.collection === "tensors" && view.storage_validation !== undefined
? new Set(view.storage_validation)
: undefined;
let selected: number[] | undefined;
if (selectedNames !== undefined)
selected = [
...new Set(
selectedNames.flatMap((name) => {
const index = lookup.get(name);
return index === undefined ? [] : [index];
}),
),
].sort((a, b) => a - b);
else if (indices !== undefined)
selected = [...indices]
.filter((index) => index < rows.length)
.sort((a, b) => a - b);
if (
indices !== undefined ||
types !== undefined ||
storageStates !== undefined
)
selected = (selected ?? rows.map((row) => row.index)).filter((index) => {
const row = rows[index]!;
return (
(indices === undefined || indices.has(index)) &&
(types === undefined || types.has(row.type)) &&
(storageStates === undefined ||
("storage_validation" in row &&
storageStates.has(row.storage_validation)))
);
});
const total = selected?.length ?? rows.length;
const pageIndices =
selected === undefined
? Array.from(
{
length: Math.max(
0,
Math.min(view.limit, rows.length - view.offset),
),
},
(_, index) => view.offset + index,
)
: selected.slice(view.offset, view.offset + view.limit);
const fields =
view.fields === undefined ? undefined : new Set<string>(view.fields);
const items = pageIndices.map((index) => {
const row = rows[index]!;
const result: Record<string, JsonValue> = { index };
for (const [name, value] of Object.entries(row))
if (fields === undefined || fields.has(name)) result[name] = value;
return result;
});
return ok({
kind: "model-artifact",
view,
...shared,
source: report.artifact,
model,
source_coverage: report.coverage,
items: modelArtifactViewItemsSchema.parse(items),
coverage: pageViewCoverage(view.offset, items.length, total),
});
};
+131
View File
@@ -0,0 +1,131 @@
import { z } from "zod";
import { localPathStringSchema } from "../localPath.js";
import { digestSchema } from "../digests.js";
import { jsonValueSchema } from "../jsonValue.js";
const count = z.number().int().nonnegative().max(Number.MAX_SAFE_INTEGER);
const decimal = z.string().regex(/^(?:0|[1-9][0-9]*)$/u);
export const modelSourceSchema = z.strictObject({
path: localPathStringSchema,
file_size: count,
header_sha256: digestSchema,
header_length: count,
sha256: digestSchema.nullable(),
bytes_read: count,
read_scope: z.enum(["header", "full-file"]),
});
const location = {
source_offset: count,
source_length: count,
json_pointer: z.string().nullable(),
};
/** Arrays retain declared types/counts and bounded observations, never an implicit full vocabulary. */
export const modelMetadataSchema = z.strictObject({
index: count,
key: z.string(),
type: z.string(),
value: jsonValueSchema,
element_type: z.string().nullable(),
element_count: count.nullable(),
preview_complete: z.boolean(),
...location,
});
export const modelTensorSchema = z.strictObject({
index: count,
name: z.string(),
type: z.string(),
type_id: count.nullable(),
shape: z.array(decimal),
data_offset: count,
file_offset: count,
byte_length: count.nullable(),
parameter_count: decimal,
byte_length_state: z.enum(["derived", "declared", "unknown"]),
storage_validation: z.enum(["consistent", "unknown"]),
...location,
});
export const modelArtifactObjectSchema = z.strictObject({
artifact: modelSourceSchema,
format: z.enum(["gguf", "safetensors"]),
version: count.nullable(),
byte_order: z.enum(["little", "big"]),
alignment: count.nullable(),
tensor_data_offset: count,
metadata: z.array(modelMetadataSchema),
tensors: z.array(modelTensorSchema),
totals: z.strictObject({
confidence: z.literal("derived"),
parameter_count: decimal,
tensor_bytes: count.nullable(),
by_type: z.array(
z.strictObject({
type: z.string(),
tensors: count,
parameter_count: decimal,
bytes: count.nullable(),
}),
),
}),
coverage: z.strictObject({
header: z.literal("complete"),
tensor_data: z.literal("not-interpreted"),
array_values: z.enum(["complete", "preview"]),
tensor_ranges: z.enum(["complete", "partial"]),
storage_sizes: z.enum(["complete", "partial"]),
}),
limitations: z.array(z.string()),
});
/** Lookup indexes are safe only after positional identities and uniqueness are proven. */
export const modelArtifactSchema = modelArtifactObjectSchema.superRefine(
(report, context) => {
const collections = [
{ name: "metadata", rows: report.metadata, identity: "key" },
{ name: "tensors", rows: report.tensors, identity: "name" },
{ name: "totals.by_type", rows: report.totals.by_type, identity: "type" },
] as const;
for (const { name, rows, identity } of collections) {
const seen = new Set<string>();
const prefix = name.split(".");
for (let index = 0; index < rows.length; index++) {
const row = rows[index]!;
if ("index" in row && row.index !== index) {
context.addIssue({
code: "custom",
path: [...prefix, index, "index"],
message: "Model row index must equal its source position",
});
return;
}
const value =
identity === "key" && "key" in row
? row.key
: identity === "name" && "name" in row
? row.name
: row.type;
if (seen.has(value)) {
context.addIssue({
code: "custom",
path: [...prefix, index, identity],
message: "Model collection identities must be unique",
});
return;
}
seen.add(value);
}
}
},
);
export type ModelArtifact = z.infer<typeof modelArtifactSchema>;
export type ModelTensor = z.infer<typeof modelTensorSchema>;
export type ModelMetadata = z.infer<typeof modelMetadataSchema>;
export interface ModelArtifactInput {
readonly path: string;
readonly hash: "header" | "full";
}
+82
View File
@@ -0,0 +1,82 @@
import { z } from "zod";
import { absoluteLocalPathStringSchema } from "../localPath.js";
import {
modelArtifactObjectSchema,
modelMetadataSchema,
modelTensorSchema,
} from "./modelArtifact.js";
export const modelTypeRowSchema =
modelArtifactObjectSchema.shape.totals.shape.by_type.element.extend({
index: z.number().int().nonnegative(),
});
const page = {
indices: z.array(z.number().int().nonnegative()).optional(),
offset: z.number().int().nonnegative().default(0),
limit: z.number().int().positive().default(32),
};
const tensorSelectionSchema = z.strictObject({
...page,
collection: z.literal("tensors"),
names: z.array(z.string()).optional(),
storage_validation: z
.array(modelTensorSchema.shape.storage_validation)
.optional(),
types: z.array(z.string()).optional(),
fields: z.array(modelTensorSchema.keyof()).optional(),
});
const typeSelectionSchema = z.strictObject({
...page,
collection: z.literal("types"),
types: z.array(z.string()).optional(),
fields: z.array(modelTypeRowSchema.keyof()).optional(),
});
const metadataSelectionSchema = z.strictObject({
...page,
collection: z.literal("metadata"),
keys: z.array(z.string()).optional(),
fields: z.array(modelMetadataSchema.keyof()).optional(),
});
export const modelArtifactSelectionSchema = z.discriminatedUnion("collection", [
tensorSelectionSchema,
typeSelectionSchema,
metadataSelectionSchema,
]);
export const modelArtifactViewRequestSchema = z.discriminatedUnion(
"collection",
[
tensorSelectionSchema.extend({ kind: z.literal("model-artifact") }),
typeSelectionSchema.extend({ kind: z.literal("model-artifact") }),
metadataSelectionSchema.extend({ kind: z.literal("model-artifact") }),
],
);
/** Selection changes delivery; full observations remain retained. */
export const inspectModelArtifactInputSchema = z
.strictObject({
path: absoluteLocalPathStringSchema,
hash: z
.enum(["header", "full"])
.default("header")
.describe(
"header reads declarations only; full streams all bytes for SHA-256",
),
detail: z.enum(["summary", "complete"]).default("summary"),
view: modelArtifactSelectionSchema
.optional()
.describe(
"Return this selected page with compact model context, retaining complete Evidence; requires summary detail",
),
})
.superRefine((value, context) => {
if (value.view !== undefined && value.detail === "complete")
context.addIssue({
code: "custom",
path: ["view"],
message:
"view selects summary delivery; omit view to request complete Evidence",
});
});
+363
View File
@@ -0,0 +1,363 @@
import { TextDecoder } from "node:util";
import { setImmediate } from "node:timers/promises";
import type { JsonValue } from "../domain/jsonValue.js";
import type {
ModelMetadata,
ModelTensor,
} from "../domain/model/modelArtifact.js";
import { ModelHeaderReader } from "./ModelHeaderReader.js";
import {
MODEL_ARRAY_PREVIEW,
ModelFormatFailure,
ModelReportBudget,
rejectModel,
} from "./ModelFormat.js";
import { ggmlType } from "./ModelTensorTypes.js";
const NAMES = [
"UINT8",
"INT8",
"UINT16",
"INT16",
"UINT32",
"INT32",
"FLOAT32",
"BOOL",
"STRING",
"ARRAY",
"UINT64",
"INT64",
"FLOAT64",
] as const;
const WIDTHS = [1, 1, 2, 2, 4, 4, 4, 1, 8, 12, 8, 8, 8];
/** Explicit-endian GGUF v2/v3 metadata and tensor-directory parser. */
export class GgufHeader {
#little = true;
readonly #budget = new ModelReportBudget();
constructor(readonly reader: ModelHeaderReader) {}
async #unsigned(width: 4 | 8): Promise<number> {
const b = await this.reader.take(width);
const value =
width === 4
? BigInt(this.#little ? b.readUInt32LE() : b.readUInt32BE())
: this.#little
? b.readBigUInt64LE()
: b.readBigUInt64BE();
if (value > BigInt(Number.MAX_SAFE_INTEGER))
throw new ModelFormatFailure(
"capacity",
"Header length/count exceeds exact host offset precision",
);
return Number(value);
}
async #text(retain: boolean, maximum?: number): Promise<string> {
const length = await this.#unsigned(8);
if (maximum !== undefined && length > maximum)
rejectModel("GGUF name exceeds its format byte limit");
if (retain) this.#budget.charge(6 * length + 2);
this.reader.guarantee(length);
const decoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
let result = "";
try {
for (let left = length; left > 0;) {
const n = Math.min(left, 65536);
const text = decoder.decode(await this.reader.take(n), {
stream: true,
});
if (retain) result += text;
left -= n;
}
const tail = decoder.decode();
if (retain) result += tail;
} catch (cause) {
if (cause instanceof TypeError)
rejectModel("GGUF string is not valid UTF-8");
throw cause;
}
return result;
}
async #value(
type: number,
retain: boolean,
depth = 0,
): Promise<{
value: JsonValue;
element_type: string | null;
element_count: number | null;
complete: boolean;
}> {
if (NAMES[type] === undefined)
rejectModel(`Unknown GGUF metadata type ${String(type)}`);
if (depth > 32)
throw new ModelFormatFailure(
"capacity",
"GGUF metadata nesting exceeds 32 array frames",
);
if (type === 8)
return {
value: await this.#text(retain),
element_type: null,
element_count: null,
complete: true,
};
if (type === 9) {
const element = await this.#unsigned(4);
const count = await this.#unsigned(8);
const minimum = WIDTHS[element];
if (minimum === undefined)
rejectModel(`Unknown GGUF array element type ${String(element)}`);
this.reader.guarantee(count * minimum);
const preview: JsonValue[] = [];
if (retain)
this.#budget.charge(128 + Math.min(count, MODEL_ARRAY_PREVIEW) * 64);
let complete = count <= MODEL_ARRAY_PREVIEW;
for (let index = 0; index < count; index++) {
if (index % 4096 === 4095) await setImmediate();
this.reader.guarantee((count - index) * minimum);
if (
element !== 8 &&
element !== 9 &&
(!retain || index >= MODEL_ARRAY_PREVIEW)
) {
let left = (count - index) * minimum;
while (left > 0) {
const size = Math.min(left, 65536);
const bytes = await this.reader.take(size);
if (element === 7 && bytes.some((byte) => byte !== 0 && byte !== 1))
rejectModel("GGUF boolean must be 0 or 1");
left -= size;
}
break;
}
const keep = retain && index < MODEL_ARRAY_PREVIEW;
const value = await this.#value(element, keep, depth + 1);
if (keep)
preview.push(
element === 9
? {
type: "ARRAY",
element_type: value.element_type,
element_count: value.element_count,
preview: value.value,
preview_complete: value.complete,
}
: value.value,
);
complete &&= value.complete;
}
return {
value: preview,
element_type: NAMES[element] ?? null,
element_count: count,
complete,
};
}
const b = await this.reader.take(WIDTHS[type] ?? 0);
let value: JsonValue;
switch (type) {
case 0:
value = b.readUInt8();
break;
case 1:
value = b.readInt8();
break;
case 2:
value = this.#little ? b.readUInt16LE() : b.readUInt16BE();
break;
case 3:
value = this.#little ? b.readInt16LE() : b.readInt16BE();
break;
case 4:
value = this.#little ? b.readUInt32LE() : b.readUInt32BE();
break;
case 5:
value = this.#little ? b.readInt32LE() : b.readInt32BE();
break;
case 6:
value = this.#little ? b.readFloatLE() : b.readFloatBE();
break;
case 7:
if (b[0] !== 0 && b[0] !== 1)
rejectModel("GGUF boolean must be 0 or 1");
value = b[0] === 1;
break;
case 10:
value = (
this.#little ? b.readBigUInt64LE() : b.readBigUInt64BE()
).toString();
break;
case 11:
value = (
this.#little ? b.readBigInt64LE() : b.readBigInt64BE()
).toString();
break;
case 12:
value = this.#little ? b.readDoubleLE() : b.readDoubleBE();
break;
default:
return rejectModel("Unsupported GGUF scalar");
}
if (Object.is(value, -0)) value = "-0";
if (typeof value === "number" && !Number.isFinite(value))
value = Number.isNaN(value)
? "NaN"
: value > 0
? "+Infinity"
: "-Infinity";
return {
value: retain ? value : null,
element_type: null,
element_count: null,
complete: true,
};
}
async read(): Promise<{
version: number;
byte_order: "little" | "big";
alignment: number;
tensor_data_offset: number;
metadata: ModelMetadata[];
tensors: ModelTensor[];
}> {
const versionBytes = await this.reader.take(4);
this.#little =
versionBytes.readUInt32LE() === 2 || versionBytes.readUInt32LE() === 3;
const version = this.#little
? versionBytes.readUInt32LE()
: versionBytes.readUInt32BE();
if (version !== 2 && version !== 3)
throw new ModelFormatFailure(
"unsupported",
`GGUF version ${String(version)} is unsupported; supported versions are 2 and 3`,
);
if (!this.#little && version !== 3)
rejectModel("Big-endian GGUF requires version 3");
const tensorCount = await this.#unsigned(8);
const metadataCount = await this.#unsigned(8);
this.#budget.charge((tensorCount + metadataCount) * 512);
this.reader.guarantee(tensorCount * 20 + metadataCount * 13);
const metadata: ModelMetadata[] = [];
const keys = new Set<string>();
let alignment = 32;
for (let index = 0; index < metadataCount; index++) {
this.reader.guarantee((metadataCount - index) * 13 + tensorCount * 20);
const start = this.reader.position;
const key = await this.#text(true, 65535);
// Numeric hierarchy components occur in standardized base_model.{id} keys.
if (
!/^[a-z0-9]+(?:_[a-z0-9]+)*(?:\.[a-z0-9]+(?:_[a-z0-9]+)*)*$/u.test(key)
)
rejectModel(
"GGUF metadata keys require ASCII lower-snake-case hierarchy components",
);
if (keys.has(key)) rejectModel(`Duplicate GGUF metadata key ${key}`);
keys.add(key);
const type = await this.#unsigned(4);
const parsed = await this.#value(type, true);
metadata.push({
index,
key,
type: NAMES[type] ?? String(type),
value: parsed.value,
element_type: parsed.element_type,
element_count: parsed.element_count,
preview_complete: parsed.complete,
source_offset: start,
source_length: this.reader.position - start,
json_pointer: null,
});
if (key === "general.alignment") {
if (
type !== 4 ||
typeof parsed.value !== "number" ||
parsed.value === 0 ||
parsed.value % 8 !== 0
)
rejectModel(
"GGUF general.alignment must be a positive UINT32 multiple of eight",
);
alignment = parsed.value;
}
}
const tensors: ModelTensor[] = [];
const names = new Set<string>();
for (let index = 0; index < tensorCount; index++) {
this.reader.guarantee((tensorCount - index) * 20);
const start = this.reader.position;
const name = await this.#text(true, 64);
if (names.has(name)) rejectModel(`Duplicate GGUF tensor name ${name}`);
names.add(name);
const dimensions = await this.#unsigned(4);
if (dimensions > 4)
rejectModel("GGUF tensor dimension count must be at most 4");
const shape: string[] = [];
let parameters = 1n;
for (let d = 0; d < dimensions; d++) {
const b = await this.reader.take(8);
const value = this.#little ? b.readBigUInt64LE() : b.readBigUInt64BE();
shape.push(value.toString());
parameters *= value;
}
const id = await this.#unsigned(4);
const offset = await this.#unsigned(8);
if (offset % alignment !== 0)
rejectModel(`GGUF tensor ${name} offset is not aligned`);
const type = ggmlType(id);
let length: number | null = null;
if (type !== undefined) {
if (BigInt(shape[0] ?? "1") % BigInt(type[1]) !== 0n)
rejectModel(
`GGUF tensor ${name} first dimension is not divisible by the quantization block size`,
);
const bytes = (parameters / BigInt(type[1])) * BigInt(type[2]);
if (bytes > BigInt(this.reader.size))
rejectModel(
`GGUF tensor ${name} declared byte length exceeds the file`,
);
length = Number(bytes);
}
tensors.push({
index,
name,
type: type?.[0] ?? `UNKNOWN_${String(id)}`,
type_id: id,
shape,
data_offset: offset,
file_offset: 0,
byte_length: length,
parameter_count: parameters.toString(),
byte_length_state: length === null ? "unknown" : "derived",
storage_validation: length === null ? "unknown" : "consistent",
source_offset: start,
source_length: this.reader.position - start,
json_pointer: null,
});
}
const tensor_data_offset =
tensorCount === 0
? this.reader.position
: Math.ceil(this.reader.position / alignment) * alignment;
for (const tensor of tensors) {
const offset = BigInt(tensor_data_offset) + BigInt(tensor.data_offset);
if (
offset > BigInt(this.reader.size) ||
offset + BigInt(tensor.byte_length ?? 0) > BigInt(this.reader.size)
)
rejectModel(`GGUF tensor ${tensor.name} range is outside the file`);
tensor.file_offset = Number(offset);
}
return {
version,
byte_order: this.#little ? "little" : "big",
alignment,
tensor_data_offset,
metadata,
tensors,
};
}
}
+378
View File
@@ -0,0 +1,378 @@
import { lstat, type FileHandle } from "node:fs/promises";
import type { ModelArtifactPort } from "../application/model/ModelArtifactPort.js";
import { createModelArtifactEvidence } from "../application/model/ModelArtifactEvidence.js";
import type { ExecutionOptions } from "../application/AnalysisProvider.js";
import type {
ModelArtifact,
ModelArtifactInput,
} from "../domain/model/modelArtifact.js";
import { modelArtifactSchema } from "../domain/model/modelArtifact.js";
import { AnalysisError } from "../domain/analysisErrorBase.js";
import {
AnalysisArtifactChangedError,
AnalysisAccessDeniedError,
AnalysisCancelledError,
AnalysisResourceConstraintError,
AnalysisUnsupportedTargetError,
} from "../domain/analysisErrorCore.js";
import { BinaryTargetError } from "../domain/configurationErrors.js";
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { err, ok, type Result } from "../domain/result.js";
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
import {
withArtifactRegularFile,
bindArtifactFileCleanup,
artifactFilePrimaryFailure,
artifactFileCancellationCause,
ArtifactFileCancelled,
ArtifactFileCleanupFailure,
} from "../artifacts/ArtifactFileOperation.js";
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
import { ModelHeaderReader } from "./ModelHeaderReader.js";
import { GgufHeader } from "./GgufHeader.js";
import { readSafetensorsHeader } from "./SafetensorsHeader.js";
import {
ModelFormatFailure,
rejectModel,
MODEL_REPORT_BYTES,
MODEL_HEADER_BYTES,
} from "./ModelFormat.js";
import { safetensorsBits } from "./ModelTensorTypes.js";
import { jsonValueByteLength } from "../domain/jsonValueByteLength.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import { sameRegularFileState } from "../filesystem/RegularFile.js";
export const MODEL_ARTIFACT_PROVIDER_IDENTITY = {
id: "model-artifact",
name: "REA model header reader",
version: "1",
} as const;
const OPERATION = "inspect_model_artifact";
/** One admitted descriptor and bounded report at a time; retained cleanup blocks new work. */
export class ModelArtifactProvider implements ModelArtifactPort {
readonly identity = MODEL_ARTIFACT_PROVIDER_IDENTITY;
readonly #resources = new ArtifactResourceScope();
#busy = false;
async close(): Promise<void> {
try {
await this.#resources.close();
} catch (cause) {
const error = new ProviderCleanupError(
this.identity.id,
this.#resources.cleanupResources,
{},
{ cause, operation: OPERATION },
);
bindArtifactFileCleanup(cause, error);
throw error;
}
}
async inspect(
input: ModelArtifactInput,
options?: ExecutionOptions,
): Promise<Result<ModelArtifact, AnalysisError>> {
if (this.#busy)
return err(
new AnalysisResourceConstraintError(
OPERATION,
"capacity",
"One model inspection is already active",
{ concurrent_inspections: 1 },
{ retryable: true },
),
);
this.#busy = true;
try {
return ok(
await this.#resources.run(() =>
withArtifactRegularFile(
input.path,
this.#resources,
options?.signal,
(file) => this.#read(file, input, options?.signal),
),
),
);
} catch (cause) {
const primary = artifactFilePrimaryFailure(cause);
let error: AnalysisError;
if (primary instanceof AnalysisError) error = primary;
else if (primary instanceof ModelFormatFailure) {
error =
primary.kind === "capacity"
? new AnalysisResourceConstraintError(
OPERATION,
"memory",
primary.message,
{
maximum_report_bytes: MODEL_REPORT_BYTES,
maximum_header_bytes: MODEL_HEADER_BYTES,
maximum_exact_offset: Number.MAX_SAFE_INTEGER,
},
{ cause, retryable: false },
)
: primary.kind === "unsupported"
? new AnalysisUnsupportedTargetError(
OPERATION,
input.path,
primary.message,
{ cause },
)
: new BinaryTargetError(input.path, primary.message, {
cause,
constraint: "file_format_rejected",
});
} else if (
primary instanceof ArtifactReaderFailure &&
primary.reason === "cancelled"
)
error = new AnalysisCancelledError(OPERATION, {
cause: artifactFileCancellationCause(primary),
});
else if (primary instanceof ArtifactFileCancelled)
error = new AnalysisCancelledError(OPERATION, {
cause: primary.fileCancellation.reason,
});
else if (
primary instanceof ArtifactReaderFailure &&
primary.reason === "integrity"
)
error = new AnalysisArtifactChangedError(
OPERATION,
input.path,
primary.message,
{ cause },
);
else if (
primary instanceof Error &&
"code" in primary &&
(primary.code === "EACCES" || primary.code === "EPERM")
)
error = new AnalysisAccessDeniedError(
OPERATION,
input.path,
primary.code,
{ cause: primary },
);
else if (
primary instanceof ArtifactReaderFailure &&
primary.reason === "limit"
)
error = new AnalysisResourceConstraintError(
OPERATION,
"memory",
primary.message,
{},
{ cause: primary },
);
else if (primary instanceof Error && "code" in primary)
error = new BinaryTargetError(input.path, primary.message, { cause });
else
error = new ProviderAdapterError(this.identity.id, OPERATION, {
cause,
diagnostics: {
path: input.path,
reason:
primary instanceof Error ? primary.message : String(primary),
},
});
if (
cause instanceof ArtifactReaderFailure &&
cause.cleanup !== undefined
) {
const cleanup = new ProviderCleanupError(
this.identity.id,
cause.cleanup.resources,
{ reason: cause.cleanup.reason },
{ cause, operation: OPERATION },
);
error = analysisErrorWithCleanupFailure(error, cleanup, OPERATION);
}
const completed =
cause instanceof ArtifactFileCleanupFailure &&
cause.fileCleanupFailure.outcome.kind === "completed"
? cause.fileCleanupFailure.outcome.value
: cause instanceof ArtifactFileCancelled
? cause.fileCancellation.outcome.value
: undefined;
if (completed !== undefined) {
// The descriptor owner retains observations even when its final close fails.
const report = modelArtifactSchema.safeParse(completed);
if (report.success)
error.retainPartialObservation(
createModelArtifactEvidence(report.data, this.identity, input),
);
}
bindArtifactFileCleanup(cause, error);
return err(error);
} finally {
this.#busy = false;
}
}
async #read(
file: FileHandle,
input: ModelArtifactInput,
signal?: AbortSignal,
): Promise<ModelArtifact> {
const initial = await file.stat({ bigint: true });
if (initial.size > BigInt(Number.MAX_SAFE_INTEGER))
throw new ModelFormatFailure(
"capacity",
"Model file exceeds exact host offset precision",
);
const reader = new ModelHeaderReader(file, Number(initial.size), signal);
const first = await reader.take(4);
if (
(first[0] === 0x80 && (first[1] ?? 0) >= 2 && (first[1] ?? 0) <= 5) ||
first.equals(Buffer.from([0x50, 0x4b, 3, 4]))
)
throw new ModelFormatFailure(
"unsupported",
"PyTorch pickle/ZIP model deserialization is unsupported; supply GGUF or safetensors instead",
);
const format = first.equals(Buffer.from("GGUF")) ? "gguf" : "safetensors";
const parsed =
format === "gguf"
? await new GgufHeader(reader).read()
: await readSafetensorsHeader(reader, first);
const intervals = parsed.tensors
.filter((row) => row.byte_length !== null)
.sort((a, b) => a.file_offset - b.file_offset);
let end = parsed.tensor_data_offset;
for (const row of intervals) {
if (row.file_offset < end && row.byte_length !== 0)
rejectModel(`Overlapping tensor byte ranges at ${row.name}`);
end = Math.max(end, row.file_offset + (row.byte_length ?? 0));
}
const header_length = reader.position;
const hashes = await reader.finish(input.hash === "full");
const after = await file.stat({ bigint: true });
const named = await lstat(input.path, { bigint: true }).catch(
(cause: unknown) => {
if (
cause instanceof Error &&
"code" in cause &&
["ENOENT", "ENOTDIR", "ELOOP"].includes(String(cause.code))
)
throw new ArtifactReaderFailure(
"integrity",
"Selected model path changed during inspection",
{ cause },
);
throw cause;
},
);
if ([after, named].some((stat) => !sameRegularFileState(initial, stat)))
throw new ArtifactReaderFailure(
"integrity",
"Model file changed during inspection",
);
const groups = new Map<
string,
{
type: string;
tensors: number;
parameter_count: bigint;
bytes: number | null;
}
>();
let parameters = 0n;
let bytes: number | null = 0;
for (const row of parsed.tensors) {
const count = BigInt(row.parameter_count);
parameters += count;
bytes =
bytes === null || row.byte_length === null
? null
: bytes + row.byte_length;
const group = groups.get(row.type) ?? {
type: row.type,
tensors: 0,
parameter_count: 0n,
bytes: 0,
};
group.tensors++;
group.parameter_count += count;
group.bytes =
group.bytes === null || row.byte_length === null
? null
: group.bytes + row.byte_length;
groups.set(row.type, group);
}
const unknownTypes = [
...new Set(
parsed.tensors
.filter(
(row) =>
row.byte_length === null ||
(format === "safetensors" &&
safetensorsBits(row.type) === undefined),
)
.map((row) => row.type),
),
];
const report = {
artifact: {
path: input.path,
file_size: Number(initial.size),
header_length,
...hashes,
read_scope:
input.hash === "full" ? ("full-file" as const) : ("header" as const),
},
format,
...parsed,
totals: {
confidence: "derived" as const,
parameter_count: parameters.toString(),
tensor_bytes: bytes,
by_type: [...groups.values()].map((group) => ({
...group,
parameter_count: group.parameter_count.toString(),
})),
},
coverage: {
header: "complete" as const,
tensor_data: "not-interpreted" as const,
array_values: parsed.metadata.every((row) => row.preview_complete)
? ("complete" as const)
: ("preview" as const),
storage_sizes: parsed.tensors.every(
(row) => row.storage_validation === "consistent",
)
? ("complete" as const)
: ("partial" as const),
tensor_ranges: parsed.tensors.every((row) => row.byte_length !== null)
? ("complete" as const)
: ("partial" as const),
},
limitations: [
"Facts are file declarations; model behavior, weight values and compatibility with an inference runtime were not verified.",
"Header SHA-256 commits only to the recorded prefix; file size is a separate observation. Different weights may share these declarations.",
...(input.hash === "header"
? ["Full-file SHA-256 is unknown because tensor data was not read."]
: [
"Full-file SHA-256 required streaming the entire file, including tensor data, without interpreting it.",
]),
...(parsed.metadata.some((row) => !row.preview_complete)
? [
"Metadata arrays retain at most eight elements per array; remaining values are omitted, with exact declared counts and source byte ranges.",
]
: []),
...(unknownTypes.length
? [
`Unknown tensor types prevent storage-size validation: ${unknownTypes.join(", ")}`,
]
: []),
],
};
if (jsonValueByteLength(report, MODEL_REPORT_BYTES) > MODEL_REPORT_BYTES)
throw new ModelFormatFailure("capacity", "Model report exceeds 32 MiB");
return modelArtifactSchema.parse(report);
}
}
+33
View File
@@ -0,0 +1,33 @@
/** Format failures stay separate from filesystem, capacity and cancellation failures. */
export class ModelFormatFailure extends Error {
constructor(
readonly kind: "format" | "unsupported" | "capacity",
message: string,
) {
super(message);
this.name = "ModelFormatFailure";
}
}
export function rejectModel(message: string): never {
throw new ModelFormatFailure("format", message);
}
// Safetensors' format ceiling is 100,000,000 bytes. The reader additionally
// bounds retained rows/text independently of that input ceiling.
export const MODEL_HEADER_BYTES = 100_000_000;
export const MODEL_REPORT_BYTES = 32 * 1024 * 1024;
export const MODEL_ARRAY_PREVIEW = 8;
/** Charge representation growth before retaining rows or decoding long strings. */
export class ModelReportBudget {
#remaining = MODEL_REPORT_BYTES;
charge(bytes: number): void {
if (!Number.isSafeInteger(bytes) || bytes < 0 || bytes > this.#remaining)
throw new ModelFormatFailure(
"capacity",
"Model facts exceed the 32 MiB retained representation budget",
);
this.#remaining -= bytes;
}
}
+138
View File
@@ -0,0 +1,138 @@
import { createHash } from "node:crypto";
import type { FileHandle } from "node:fs/promises";
import { assertArtifactReadActive } from "../artifacts/ArtifactFileOperation.js";
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
import {
ModelFormatFailure,
MODEL_HEADER_BYTES,
rejectModel,
} from "./ModelFormat.js";
/** Sequential prefix reader. Read-ahead is permitted only inside a parser-proven header extent. */
export class ModelHeaderReader {
position = 0;
#end = 0;
#guaranteedEnd = 0;
#buffer = Buffer.alloc(0);
#start = 0;
#headerPrefixBytes = 0;
readonly #hash = createHash("sha256");
constructor(
readonly file: FileHandle,
readonly size: number,
readonly signal?: AbortSignal,
) {}
/** Safetensors declares JSON bytes separately from its fixed length prefix. */
excludeSafetensorsLengthPrefix(): void {
if (this.position !== 8)
throw new TypeError(
"Safetensors header budget must follow its eight-byte length prefix",
);
this.#headerPrefixBytes = 8;
}
/** Minimum bytes implied by the remaining declared header records, never tensor payload. */
guarantee(bytes: number): void {
const end = this.position + bytes;
this.#validateEnd(end);
this.#guaranteedEnd = Math.max(this.#guaranteedEnd, end);
}
#validateEnd(end: number): void {
if (!Number.isSafeInteger(end) || end < 0 || end > this.size)
rejectModel("Declared header extent is outside the file");
if (end - this.#headerPrefixBytes > MODEL_HEADER_BYTES)
throw new ModelFormatFailure(
"capacity",
"Model header exceeds 100,000,000 bytes",
);
}
async take(length: number): Promise<Buffer> {
assertArtifactReadActive(this.signal);
const end = this.position + length;
this.#validateEnd(end);
if (!Number.isSafeInteger(length) || length < 0)
rejectModel("Invalid header field length");
if (end > this.#end) {
const readEnd = Math.max(
end,
Math.min(this.position + 65536, this.#guaranteedEnd),
);
const retained = this.#buffer.subarray(this.position - this.#start);
const next = Buffer.allocUnsafe(readEnd - this.position);
retained.copy(next);
let offset = retained.length;
while (offset < next.length) {
assertArtifactReadActive(this.signal);
const result = await this.file.read(
next,
offset,
next.length - offset,
this.position + offset,
);
if (result.bytesRead === 0)
throw new ArtifactReaderFailure(
"integrity",
"Model header ended before its admitted file extent",
);
this.#hash.update(next.subarray(offset, offset + result.bytesRead));
offset += result.bytesRead;
}
this.#start = this.position;
this.#end = readEnd;
this.#buffer = next;
}
const result = this.#buffer.subarray(
this.position - this.#start,
end - this.#start,
);
this.position = end;
return result;
}
async consume(length: number): Promise<void> {
while (length > 0) {
const size = Math.min(length, 65536);
await this.take(size);
length -= size;
}
}
/** Complete the exact prefix commitment, optionally streaming weights without retaining them. */
async finish(full: boolean): Promise<{
header_sha256: string;
sha256: string | null;
bytes_read: number;
}> {
if (this.position !== this.#end)
rejectModel("Parser left unconsumed header bytes");
const header_sha256 = this.#hash.copy().digest("hex");
if (!full)
return { header_sha256, sha256: null, bytes_read: this.position };
const buffer = Buffer.allocUnsafe(1024 * 1024);
let position = this.position;
while (position < this.size) {
assertArtifactReadActive(this.signal);
const { bytesRead } = await this.file.read(
buffer,
0,
Math.min(buffer.length, this.size - position),
position,
);
if (bytesRead === 0)
throw new ArtifactReaderFailure(
"integrity",
"Model ended before its admitted extent during full-file hashing",
);
this.#hash.update(buffer.subarray(0, bytesRead));
position += bytesRead;
}
return {
header_sha256,
sha256: this.#hash.digest("hex"),
bytes_read: position,
};
}
}
+74
View File
@@ -0,0 +1,74 @@
/** ggml da8e2a54: enum ggml_type and block sizeof assertions in ggml-common.h.
* Unknown future IDs retain their numeric identity and an unknown byte length.
*/
const GGML_TYPES: Readonly<Record<number, readonly [string, number, number]>> =
{
0: ["F32", 1, 4],
1: ["F16", 1, 2],
2: ["Q4_0", 32, 18],
3: ["Q4_1", 32, 20],
6: ["Q5_0", 32, 22],
7: ["Q5_1", 32, 24],
8: ["Q8_0", 32, 34],
9: ["Q8_1", 32, 36],
10: ["Q2_K", 256, 84],
11: ["Q3_K", 256, 110],
12: ["Q4_K", 256, 144],
13: ["Q5_K", 256, 176],
14: ["Q6_K", 256, 210],
15: ["Q8_K", 256, 292],
16: ["IQ2_XXS", 256, 66],
17: ["IQ2_XS", 256, 74],
18: ["IQ3_XXS", 256, 98],
19: ["IQ1_S", 256, 50],
20: ["IQ4_NL", 32, 18],
21: ["IQ3_S", 256, 110],
22: ["IQ2_S", 256, 82],
23: ["IQ4_XS", 256, 136],
24: ["I8", 1, 1],
25: ["I16", 1, 2],
26: ["I32", 1, 4],
27: ["I64", 1, 8],
28: ["F64", 1, 8],
29: ["IQ1_M", 256, 56],
30: ["BF16", 1, 2],
34: ["TQ1_0", 256, 54],
35: ["TQ2_0", 256, 66],
39: ["MXFP4", 32, 17],
40: ["NVFP4", 64, 36],
41: ["Q1_0", 128, 18],
42: ["Q2_0", 64, 18],
};
export const ggmlType = (
id: number,
): readonly [string, number, number] | undefined => GGML_TYPES[id];
/** safetensors e246a256 Dtype::bitsize, including packed sub-byte types. */
const SAFETENSORS_BITS: Readonly<Record<string, number>> = {
BOOL: 8,
F4: 4,
F6_E2M3: 6,
F6_E3M2: 6,
I8: 8,
U8: 8,
F8_E5M2: 8,
F8_E4M3: 8,
F8_E8M0: 8,
F8_E4M3FNUZ: 8,
F8_E5M2FNUZ: 8,
I16: 16,
U16: 16,
F16: 16,
BF16: 16,
I32: 32,
U32: 32,
F32: 32,
I64: 64,
U64: 64,
F64: 64,
C64: 64,
};
export const safetensorsBits = (type: string): number | undefined =>
Object.hasOwn(SAFETENSORS_BITS, type) ? SAFETENSORS_BITS[type] : undefined;
+298
View File
@@ -0,0 +1,298 @@
import { setImmediate } from "node:timers/promises";
import { assertArtifactReadActive } from "../artifacts/ArtifactFileOperation.js";
import { TextDecoder } from "node:util";
import type {
ModelMetadata,
ModelTensor,
} from "../domain/model/modelArtifact.js";
import { ModelHeaderReader } from "./ModelHeaderReader.js";
import {
ModelFormatFailure,
ModelReportBudget,
rejectModel,
} from "./ModelFormat.js";
import { safetensorsBits } from "./ModelTensorTypes.js";
type HeaderValue =
| string
| number
| HeaderValue[]
| { [key: string]: HeaderValue };
/** The format's restricted JSON header, with duplicate keys and exact integer checks at its owner. */
class HeaderJson {
#position = 0;
#nodes = 0;
constructor(
readonly text: string,
readonly budget: ModelReportBudget,
) {}
#space(): void {
while (" \t\r\n".includes(this.text[this.#position] ?? "x"))
this.#position++;
}
#string(): string {
const start = this.#position++;
while (this.#position < this.text.length) {
const c = this.text[this.#position++];
if (c === "\\") this.#position++;
else if (c === '"') {
const raw = this.text.slice(start, this.#position);
this.budget.charge(6 * raw.length + 2);
try {
const value: unknown = JSON.parse(raw);
if (typeof value !== "string")
return rejectModel("Invalid JSON string");
for (const unit of value) {
const code = unit.codePointAt(0) ?? 0;
if (code >= 0xd800 && code <= 0xdfff)
rejectModel("Safetensors JSON contains an unpaired surrogate");
}
return value;
} catch (cause) {
if (cause instanceof SyntaxError)
rejectModel("Invalid safetensors JSON string");
throw cause;
}
}
}
return rejectModel("Unterminated safetensors JSON string");
}
*#value(depth: number): Generator<void, HeaderValue> {
if (++this.#nodes % 4096 === 0) yield;
this.#space();
this.budget.charge(64);
if (depth > 4)
rejectModel("Safetensors header has an invalid nested structure");
const c = this.text[this.#position];
if (c === '"') return this.#string();
if (c === "{" || c === "[") {
this.#position++;
const object: { [key: string]: HeaderValue } = Object.create(null) as {
[key: string]: HeaderValue;
};
const array: HeaderValue[] = [];
const close = c === "{" ? "}" : "]";
this.#space();
if (this.text[this.#position] === close) {
this.#position++;
return c === "{" ? object : array;
}
while (true) {
this.#space();
if (c === "{") {
if (this.text[this.#position] !== '"')
rejectModel("Expected safetensors JSON object key");
const key = this.#string();
if (Object.hasOwn(object, key))
rejectModel(`Duplicate safetensors JSON key ${key}`);
this.#space();
if (this.text[this.#position++] !== ":")
rejectModel("Expected safetensors JSON colon");
object[key] = yield* this.#value(depth + 1);
} else array.push(yield* this.#value(depth + 1));
this.#space();
const next = this.text[this.#position++];
if (next === close) return c === "{" ? object : array;
if (next !== ",") rejectModel("Invalid safetensors JSON container");
}
}
const start = this.#position;
while (/^[0-9]$/u.test(this.text[this.#position] ?? "")) this.#position++;
const raw = this.text.slice(start, this.#position);
if (!/^(?:0|[1-9][0-9]*)$/u.test(raw))
return rejectModel(
"Safetensors header requires unsigned integer declarations",
);
const number = Number(raw);
if (!Number.isSafeInteger(number))
throw new ModelFormatFailure(
"capacity",
"Safetensors integer exceeds exact host offset precision",
);
return number;
}
*read(): Generator<void, HeaderValue> {
if (this.text[0] !== "{")
rejectModel("Safetensors header must begin with a JSON object");
const value = yield* this.#value(0);
this.#space();
if (this.#position !== this.text.length)
rejectModel("Trailing non-whitespace safetensors header bytes");
return value;
}
}
const objectOf = (
value: HeaderValue | undefined,
): { [key: string]: HeaderValue } => {
if (value === undefined || typeof value !== "object" || Array.isArray(value))
return rejectModel("Expected safetensors header object");
return value;
};
const integersOf = (value: HeaderValue | undefined): number[] => {
if (
!Array.isArray(value) ||
!value.every((item): item is number => typeof item === "number")
)
return rejectModel("Expected safetensors integer array");
return value;
};
const pointer = (value: string): string =>
value.replaceAll("~", "~0").replaceAll("/", "~1");
export const readSafetensorsHeader = async (
reader: ModelHeaderReader,
first: Buffer,
): Promise<{
version: null;
byte_order: "little";
alignment: null;
tensor_data_offset: number;
metadata: ModelMetadata[];
tensors: ModelTensor[];
}> => {
const budget = new ModelReportBudget();
const second = await reader.take(4);
reader.excludeSafetensorsLengthPrefix();
const length =
BigInt(first.readUInt32LE()) + (BigInt(second.readUInt32LE()) << 32n);
if (length === 0n || (await reader.take(1))[0] !== 0x7b)
rejectModel("File does not contain a GGUF or safetensors header");
if (length > BigInt(Number.MAX_SAFE_INTEGER))
throw new ModelFormatFailure(
"capacity",
"Safetensors header length exceeds exact host offset precision",
);
const size = Number(length);
reader.guarantee(size - 1);
// Native UTF-8 decode and the bounded JSON parser both retain header text.
budget.charge(2 * size);
let text: string;
try {
text =
"{" +
new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(
await reader.take(size - 1),
);
} catch (cause) {
if (cause instanceof TypeError)
rejectModel("Safetensors header is not valid UTF-8");
throw cause;
}
const steps = new HeaderJson(text, budget).read();
let step = steps.next();
while (!step.done) {
await setImmediate();
assertArtifactReadActive(reader.signal);
step = steps.next();
}
const root = objectOf(step.value);
const source = { source_offset: 8, source_length: size };
const metadata: ModelMetadata[] = [];
if (Object.hasOwn(root, "__metadata__")) {
for (const [key, value] of Object.entries(objectOf(root.__metadata__))) {
if (typeof value !== "string")
rejectModel("Safetensors __metadata__ values must be strings");
metadata.push({
index: metadata.length,
key,
type: "STRING",
value,
element_type: null,
element_count: null,
preview_complete: true,
...source,
json_pointer: `/__metadata__/${pointer(key)}`,
});
}
}
const tensors: ModelTensor[] = [];
const tensor_data_offset = 8 + size;
const names = Object.keys(root)
.filter((name) => name !== "__metadata__")
.sort();
budget.charge(names.length * 512);
for (const name of names) {
const entry = objectOf(root[name]);
if (
Object.keys(entry).length !== 3 ||
!Object.hasOwn(entry, "dtype") ||
!Object.hasOwn(entry, "shape") ||
!Object.hasOwn(entry, "data_offsets")
)
rejectModel(
`Safetensors tensor ${name} must declare dtype, shape and data_offsets`,
);
const type = entry.dtype;
if (typeof type !== "string")
rejectModel(`Safetensors tensor ${name} dtype must be a string`);
const shape = integersOf(entry.shape);
const offsets = integersOf(entry.data_offsets);
if (offsets.length !== 2)
rejectModel(`Safetensors tensor ${name} requires two data offsets`);
const start = offsets[0] ?? 0;
const end = offsets[1] ?? 0;
if (end < start || end > reader.size - tensor_data_offset)
rejectModel(
`Safetensors tensor ${name} range [${String(start)},${String(end)}) is outside the data buffer`,
);
let parameters = shape.includes(0) ? 0n : 1n;
if (parameters !== 0n)
for (const dimension of shape) {
parameters *= BigInt(dimension);
if (parameters > 0xffffffffffffffffn)
rejectModel(
`Safetensors tensor ${name} parameter count overflows uint64`,
);
}
const bits = safetensorsBits(type);
if (bits !== undefined) {
const expected = parameters * BigInt(bits);
if (expected % 8n !== 0n || expected / 8n !== BigInt(end - start))
rejectModel(
`Safetensors tensor ${name} byte length does not match dtype and shape`,
);
}
tensors.push({
index: tensors.length,
name,
type,
type_id: null,
shape: shape.map(String),
data_offset: start,
file_offset: tensor_data_offset + start,
byte_length: end - start,
parameter_count: parameters.toString(),
byte_length_state: "declared",
storage_validation: bits === undefined ? "unknown" : "consistent",
...source,
json_pointer: `/${pointer(name)}`,
});
}
const ordered = [...tensors].sort(
(a, b) =>
a.data_offset - b.data_offset ||
(a.byte_length ?? 0) - (b.byte_length ?? 0),
);
let end = 0;
for (const tensor of ordered) {
if (tensor.data_offset !== end)
rejectModel(
`Safetensors data ${tensor.data_offset < end ? "overlap" : "gap"} before ${tensor.name}: expected ${String(end)}, observed ${String(tensor.data_offset)}`,
);
end = tensor.data_offset + (tensor.byte_length ?? 0);
}
if (end !== reader.size - tensor_data_offset)
rejectModel(
`Safetensors unindexed trailing data: ${String(reader.size - tensor_data_offset - end)} bytes`,
);
return {
version: null,
byte_order: "little",
alignment: null,
tensor_data_offset,
metadata,
tensors,
};
};
+12
View File
@@ -27,6 +27,8 @@ import { createRecordedCrashService } from "../composition/binaryDiagnostics.js"
import { createDesktopApplicationService } from "../composition/desktop.js";
import { registerDesktopApplicationTool } from "./registerDesktopApplicationTool.js";
import { createGoBinaryService } from "../composition/go.js";
import { createModelArtifactService } from "../composition/model.js";
import { registerModelArtifactTools } from "./registerModelArtifactTools.js";
import { createIl2cppMetadataService } from "../composition/il2cpp.js";
import type { Il2cppMetadataService } from "../application/il2cpp/Il2cppMetadataService.js";
import type { GoBinaryService } from "../application/go/GoBinaryService.js";
@@ -329,6 +331,7 @@ export const createServer = (
const desktopApplication = createDesktopApplicationService();
const goBinary = options.goBinary ?? createGoBinaryService();
const rustBinary = options.rustBinary ?? createRustBinaryService();
const modelArtifact = createModelArtifactService();
const il2cppMetadata =
options.il2cppMetadata ?? createIl2cppMetadataService();
const peResources = createPeResourcesService();
@@ -416,6 +419,7 @@ export const createServer = (
goBinary.close(),
rustBinary.close(),
il2cppMetadata.close(),
modelArtifact.close(),
peResources.close(),
electronFuses.close(),
webNetworkCapture.close(),
@@ -491,6 +495,14 @@ export const createServer = (
analysisAdmission,
evidenceById,
);
registerModelArtifactTools(
server,
modelArtifact,
toolLogger,
recordEvidence,
analysisAdmission,
evidenceById,
);
registerBinaryAnalysisTools(toolContext);
registerReverseEngineeringTools(
server,
+34
View File
@@ -0,0 +1,34 @@
import type { EvidenceMcpServer } from "./EvidenceMcpServer.js";
import type { EvidenceLookup } from "../application/EvidenceInputResolver.js";
import type { ModelArtifactService } from "../application/model/ModelArtifactService.js";
import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js";
import type { Logger } from "pino";
import type { WithAdmittedAnalysis } from "./analysisAdmission.js";
import { toolContract } from "../contracts/toolContracts.js";
import { createEvidenceToolHandler } from "./contractToolHandler.js";
import { toolRegistrationOptions } from "./toolRegistrationOptions.js";
/** Bind the named model contract to the shared inspection and session Evidence owners. */
export const registerModelArtifactTools = (
server: EvidenceMcpServer,
service: ModelArtifactService,
logger: Logger,
recordEvidence?: EvidenceWriter["recordEvidence"],
withAdmittedAnalysis?: WithAdmittedAnalysis,
evidenceById?: EvidenceLookup,
): void => {
const contract = toolContract("inspect_model_artifact");
const handler = createEvidenceToolHandler(
server,
(input, options) =>
service.inspect(input, options, { recordEvidence, evidenceById }),
logger,
recordEvidence,
withAdmittedAnalysis,
);
server.registerTool(
contract.name,
toolRegistrationOptions(contract),
handler(contract),
);
};
+485
View File
@@ -0,0 +1,485 @@
import { createHash } from "node:crypto";
import { writeFile, unlink } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import { toolContract } from "../../src/contracts/toolContracts.js";
import {
modelArtifactSchema,
type ModelArtifact,
} from "../../src/domain/model/modelArtifact.js";
import {
modelArtifactSummaryViewResultSchema,
analysisViewResultSchema,
} from "../../src/domain/analysisView/analysisView.js";
import { createEvidence, parseEvidence } from "../../src/domain/evidence.js";
import { readEvidenceBundle } from "../../src/application/EvidenceBundleFiles.js";
import { connectLocalToolsMcp } from "../fixtures/localToolsMcp.js";
import { parseMcpToolError } from "../fixtures/mcpToolError.js";
import { ggufFixture, safetensorsFixture } from "../fixtures/model/model.js";
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
import { cliTest } from "../support/cli/cliFixture.js";
it("keeps generated headline keys and imported compound metadata out of compact summaries", async () => {
const root = await createTestTempDirectory("rea-model-headlines-");
const path = join(root, "model");
const architecture = "a".repeat(12000);
const { text, u32 } = ggufFixture();
await writeFile(
path,
ggufFixture({
tensors: [],
metadata: [
{ key: "general.architecture", type: 8, value: text(architecture) },
{ key: `${architecture}.context_length`, type: 4, value: u32(2048) },
],
}).bytes,
);
const { call } = await connectLocalToolsMcp();
const compact = await call("inspect_model_artifact", { path });
expect(compact.isError, JSON.stringify(compact)).not.toBe(true);
const summary = modelArtifactSummaryViewResultSchema.parse(
parseEvidence(compact.structuredContent).normalized_result,
);
expect(summary.summary.headlines).toEqual([]);
const full = parseEvidence(
(await call("inspect_model_artifact", { path, detail: "complete" }))
.structuredContent,
);
const report = modelArtifactSchema.parse(full.normalized_result);
expect(report.metadata.map(({ key, value }) => ({ key, value }))).toEqual([
{ key: "general.architecture", value: architecture },
{ key: `${architecture}.context_length`, value: 2048 },
]);
report.metadata[0]!.value = { future: architecture };
const imported = createEvidence(undefined, full.provider, {
operation: "inspect_model_artifact",
parameters: full.parameters,
result: report,
});
const page = await call("inspect_analysis_view", {
source: { kind: "inline", evidence: imported },
view: { kind: "summary" },
});
expect(page.isError, JSON.stringify(page)).not.toBe(true);
const importedSummary = modelArtifactSummaryViewResultSchema.parse(
parseEvidence(page.structuredContent).normalized_result,
);
expect(importedSummary.summary.headlines).toEqual([]);
});
it("rejects noncanonical imported model indexes and duplicate identities through public MCP", async () => {
const root = await createTestTempDirectory("rea-model-import-");
const path = join(root, "model");
await writeFile(path, ggufFixture().bytes);
const { call } = await connectLocalToolsMcp();
const response = await call("inspect_model_artifact", {
path,
detail: "complete",
});
const complete = parseEvidence(response.structuredContent);
const report = modelArtifactSchema.parse(complete.normalized_result);
const corruptions: ((value: ModelArtifact) => void)[] = [
(value) => {
value.tensors[0]!.index = 7;
},
(value) => {
value.metadata[0]!.index = 7;
},
(value) => {
value.tensors[1]!.name = value.tensors[0]!.name;
},
(value) => {
value.metadata[1]!.key = value.metadata[0]!.key;
},
(value) => {
value.totals.by_type[1]!.type = value.totals.by_type[0]!.type;
},
];
for (const corrupt of corruptions) {
const result = structuredClone(report);
corrupt(result);
const evidence = createEvidence(undefined, complete.provider, {
operation: "inspect_model_artifact",
parameters: complete.parameters,
result,
});
const failure = parseMcpToolError(
await call("inspect_analysis_view", {
source: { kind: "inline", evidence },
view: {
kind: "model-artifact",
collection: "tensors",
names: ["weight"],
},
}),
);
expect(failure.error).toMatchObject({ code: "invalid_request" });
}
await assertHistoricalModelPages(call, complete.evidence_id, report);
});
it("bounds compact model warnings and omits unselected long labels while retaining complete Evidence", async () => {
const root = await createTestTempDirectory("rea-model-long-labels-");
const path = join(root, "model");
const name = "tensor".repeat(4000);
const type = "FUTURE".repeat(4000);
const key = "metadata".repeat(3000);
await writeFile(
path,
safetensorsFixture(
JSON.stringify({
__metadata__: { [key]: "kept" },
[name]: { dtype: type, shape: [0], data_offsets: [0, 0] },
}),
Buffer.alloc(0),
).bytes,
);
const { call } = await connectLocalToolsMcp();
const response = await call("inspect_model_artifact", { path });
expect(response.isError, JSON.stringify(response)).not.toBe(true);
expect(JSON.stringify(response).length).toBeLessThan(16_000);
const summary = modelArtifactSummaryViewResultSchema.parse(
parseEvidence(response.structuredContent).normalized_result,
);
expect(summary.summary.coverage.storage_sizes).toBe("partial");
expect(summary.summary.limitations.join(" ")).toContain(
summary.parent_evidence_id,
);
const source = {
kind: "retained-evidence",
evidence_id: summary.parent_evidence_id,
};
for (const collection of ["tensors", "metadata", "types"] as const) {
const page = await call("inspect_analysis_view", {
source,
view: { kind: "model-artifact", collection, fields: [], limit: 1 },
});
expect(page.isError, JSON.stringify(page)).not.toBe(true);
expect(JSON.stringify(page).length).toBeLessThan(16_000);
const view = analysisViewResultSchema.parse(
parseEvidence(page.structuredContent).normalized_result,
);
if (view.kind !== "model-artifact") throw new Error("Incorrect view");
expect(view.items).toEqual([{ index: 0 }]);
expect(view.source_coverage.storage_sizes).toBe("partial");
expect(view.limitations.join(" ")).toContain(summary.parent_evidence_id);
}
const full = await call("inspect_model_artifact", {
path,
detail: "complete",
});
expect(full.isError, JSON.stringify(full)).not.toBe(true);
const report = modelArtifactSchema.parse(
parseEvidence(full.structuredContent).normalized_result,
);
expect(report.tensors[0]).toMatchObject({ name, type });
expect(report.metadata[0]).toMatchObject({ key, value: "kept" });
expect(report.limitations.join(" ")).toContain(type);
});
const assertHistoricalModelPages = async (
call: Awaited<ReturnType<typeof connectLocalToolsMcp>>["call"],
parentEvidenceId: string,
report: ModelArtifact,
): Promise<void> => {
const pages = [];
for (let offset = 0; offset < report.tensors.length; offset++) {
const page = await call("inspect_analysis_view", {
source: {
kind: "retained-evidence",
evidence_id: parentEvidenceId,
},
view: {
kind: "model-artifact",
collection: "tensors",
offset,
limit: 1,
fields: ["name", "type", "shape", "file_offset"],
},
});
expect(page.isError, JSON.stringify(page)).not.toBe(true);
const selected = analysisViewResultSchema.parse(
parseEvidence(page.structuredContent).normalized_result,
);
if (selected.kind !== "model-artifact") throw new Error("Incorrect view");
expect(selected.source.header_sha256).toBe(report.artifact.header_sha256);
expect(selected.source.sha256).toBeNull();
pages.push(...selected.items);
}
expect(pages).toEqual(
report.tensors.map(({ index, name, type, shape, file_offset }) => ({
index,
name,
type,
shape,
file_offset,
})),
);
const selected = await call("inspect_analysis_view", {
source: {
kind: "retained-evidence",
evidence_id: parentEvidenceId,
},
view: {
kind: "model-artifact",
collection: "tensors",
names: ["weight", "absent"],
indices: [0, 1],
fields: [],
},
});
const selectedView = analysisViewResultSchema.parse(
parseEvidence(selected.structuredContent).normalized_result,
);
if (selectedView.kind !== "model-artifact") throw new Error("Incorrect view");
expect(selectedView.items).toEqual(
report.tensors
.filter((row) => row.name === "weight")
.map(({ index }) => ({ index })),
);
const empty = await call("inspect_analysis_view", {
source: {
kind: "retained-evidence",
evidence_id: parentEvidenceId,
},
view: {
kind: "model-artifact",
collection: "metadata",
keys: [],
offset: 100,
},
});
expect(empty.isError, JSON.stringify(empty)).not.toBe(true);
const emptyView = analysisViewResultSchema.parse(
parseEvidence(empty.structuredContent).normalized_result,
);
expect(emptyView.coverage).toMatchObject({ total: 0, exhausted: true });
const unverified = await call("inspect_analysis_view", {
source: {
kind: "retained-evidence",
evidence_id: parentEvidenceId,
},
view: {
kind: "model-artifact",
collection: "tensors",
storage_validation: ["unknown"],
fields: ["name", "storage_validation", "byte_length"],
},
});
expect(unverified.isError, JSON.stringify(unverified)).not.toBe(true);
const unverifiedView = analysisViewResultSchema.parse(
parseEvidence(unverified.structuredContent).normalized_result,
);
if (unverifiedView.kind !== "model-artifact")
throw new Error("Incorrect storage selection");
expect(unverifiedView.items).toEqual(
report.tensors
.filter((row) => row.storage_validation === "unknown")
.map(({ index, name, storage_validation, byte_length }) => ({
index,
name,
storage_validation,
byte_length,
})),
);
const histogram = await call("inspect_analysis_view", {
source: {
kind: "retained-evidence",
evidence_id: parentEvidenceId,
},
view: {
kind: "model-artifact",
collection: "types",
types: ["F32"],
fields: ["type", "tensors", "parameter_count", "bytes"],
},
});
expect(histogram.isError, JSON.stringify(histogram)).not.toBe(true);
const histogramView = analysisViewResultSchema.parse(
parseEvidence(histogram.structuredContent).normalized_result,
);
if (histogramView.kind !== "model-artifact")
throw new Error("Incorrect histogram view");
expect(histogramView.items).toEqual(
report.totals.by_type.flatMap((row, index) =>
row.type === "F32" ? [{ index, ...row }] : [],
),
);
};
cliTest(
"shares scoped model facts through real CLI/MCP and pages retained metadata after source removal",
async ({ cli }) => {
const root = await createTestTempDirectory("rea-model-cli-");
const { call } = await connectLocalToolsMcp();
for (const [format, fixture] of [
["gguf", ggufFixture()],
["safetensors", safetensorsFixture()],
[
"safetensors",
safetensorsFixture(
'{"future":{"dtype":"FUTURE4","shape":[2],"data_offsets":[0,4]}}',
Buffer.alloc(4),
),
],
] as const) {
const path = join(root, `${format}.arbitrary`);
await writeFile(path, fixture.bytes);
const result = await cli.run({
arguments: [
"inspect-model-artifact",
path,
"--detail",
"complete",
"--json",
],
});
expect(result.exitCode, result.stderr).toBe(0);
const evidence = toolContract(
"inspect_model_artifact",
).outputSchema.parse(result.json);
const report = modelArtifactSchema.parse(evidence.normalized_result);
expect(report.format).toBe(format);
expect(report.artifact.sha256).toBeNull();
expect(evidence.subject).toBeNull();
const complete = await call("inspect_model_artifact", {
path,
detail: "complete",
});
expect(complete.isError, JSON.stringify(complete)).not.toBe(true);
expect(
parseEvidence(complete.structuredContent).normalized_result,
).toEqual(report);
const hashed = await call("inspect_model_artifact", {
path,
detail: "complete",
hash: "full",
});
expect(hashed.isError, JSON.stringify(hashed)).not.toBe(true);
expect(
parseEvidence(hashed.structuredContent).subject?.digest.sha256,
).toBe(createHash("sha256").update(fixture.bytes).digest("hex"));
const summary = await call("inspect_model_artifact", { path });
expect(summary.isError, JSON.stringify(summary)).not.toBe(true);
const retained = modelArtifactSummaryViewResultSchema.parse(
parseEvidence(summary.structuredContent).normalized_result,
);
expect(retained.summary.counts.tensors).toBe(report.tensors.length);
const direct = await call("inspect_model_artifact", {
path,
view: {
collection: "tensors",
names: ["weight"],
fields: ["name", "type", "shape", "file_offset"],
},
});
expect(direct.isError, JSON.stringify(direct)).not.toBe(true);
const directView = analysisViewResultSchema.parse(
parseEvidence(direct.structuredContent).normalized_result,
);
if (directView.kind !== "model-artifact")
throw new Error("Incorrect direct selection");
expect(directView.model.counts.tensors).toBe(report.tensors.length);
expect(directView.items).toEqual(
report.tensors
.filter((row) => row.name === "weight")
.map(({ index, name, type, shape, file_offset }) => ({
index,
name,
type,
shape,
file_offset,
})),
);
expect(directView.parent_evidence_id).toBe(
parseEvidence(complete.structuredContent).evidence_id,
);
await unlink(path);
await assertHistoricalModelPages(
call,
retained.parent_evidence_id,
report,
);
}
},
);
cliTest(
"exports complete model Evidence for portable paging and returns typed malformed errors",
async ({ cli }) => {
const root = await createTestTempDirectory("rea-model-export-");
const path = join(root, "model");
const output = join(root, "evidence.json");
await writeFile(path, ggufFixture().bytes);
const result = await cli.run({
arguments: [
"inspect-model-artifact",
path,
"--detail",
"summary",
"--view",
JSON.stringify({
collection: "metadata",
keys: ["tokenizer.chat_template"],
fields: ["key", "value"],
}),
"--evidence_output",
output,
"--json",
],
});
expect(result.exitCode, result.stderr).toBe(0);
const receipt = result.json as { summary: unknown };
const delivered = analysisViewResultSchema.parse(
parseEvidence(receipt.summary).normalized_result,
);
if (delivered.kind !== "model-artifact")
throw new Error("Missing CLI direct selection");
expect(delivered.items).toEqual([
{ index: 5, key: "tokenizer.chat_template", value: "{{ messages }}" },
]);
const exported = await readEvidenceBundle(output);
if (!exported.ok) throw exported.error;
const complete = exported.value.records.find(
(evidence) => evidence.operation === "inspect_model_artifact",
);
if (complete === undefined)
throw new Error("Missing complete exported model Evidence");
const recordPath = join(root, "model-record.json");
await writeFile(recordPath, JSON.stringify(complete));
const { call } = await connectLocalToolsMcp();
await unlink(path);
const page = await call("inspect_analysis_view", {
source: { kind: "evidence-file", path: recordPath },
view: {
kind: "model-artifact",
collection: "metadata",
keys: ["tokenizer.chat_template", "custom.exact"],
fields: ["key", "type", "value", "source_offset", "source_length"],
},
});
expect(page.isError, JSON.stringify(page)).not.toBe(true);
const selected = analysisViewResultSchema.parse(
parseEvidence(page.structuredContent).normalized_result,
);
if (selected.kind !== "model-artifact") throw new Error("Incorrect view");
expect(selected.items.map((row) => ("key" in row ? row.key : ""))).toEqual([
"custom.exact",
"tokenizer.chat_template",
]);
await writeFile(
path,
safetensorsFixture(
'{"x":{"dtype":"F32","shape":[3],"data_offsets":[4,16]}}',
).bytes,
);
const malformed = parseMcpToolError(
await call("inspect_model_artifact", { path }),
);
expect(malformed.error).toMatchObject({
code: "target_unavailable",
category: "unsupported_target",
details: { constraint: "file_format_rejected" },
});
},
);
+722
View File
@@ -0,0 +1,722 @@
import { createHash } from "node:crypto";
import { open, writeFile, symlink, truncate } from "node:fs/promises";
import { join } from "node:path";
import { expect, it, onTestFinished } from "vitest";
import { ModelArtifactProvider } from "../../../src/model/ModelArtifactProvider.js";
import { ModelHeaderReader } from "../../../src/model/ModelHeaderReader.js";
import { readSafetensorsHeader } from "../../../src/model/SafetensorsHeader.js";
import { GgufHeader } from "../../../src/model/GgufHeader.js";
import { createModelArtifactService } from "../../../src/composition/model.js";
import { modelArtifactSchema } from "../../../src/domain/model/modelArtifact.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { ggufFixture, safetensorsFixture } from "../../fixtures/model/model.js";
const hash = (bytes: Buffer) =>
createHash("sha256").update(bytes).digest("hex");
const providerAt = async (bytes: Buffer) => {
const root = await createTestTempDirectory("rea-model-reader-");
const path = join(root, "content.without-format-extension");
await writeFile(path, bytes);
const provider = new ModelArtifactProvider();
onTestFinished(() => provider.close());
return { provider, path, root };
};
it.each([false, true])(
"reads exact GGUF declarations, endian=%s, without hashing tensor payload",
async (bigEndian) => {
const fixture = ggufFixture({ bigEndian });
const { provider, path } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(result.value.artifact).toEqual({
path,
file_size: fixture.bytes.length,
header_sha256: hash(fixture.header),
header_length: fixture.header.length,
sha256: null,
bytes_read: fixture.header.length,
read_scope: "header",
});
expect(result.value.byte_order).toBe(bigEndian ? "big" : "little");
expect(
result.value.metadata.find((row) => row.key === "custom.exact")?.value,
).toBe("18446744073709551615");
expect(
result.value.metadata.find((row) => row.key === "tokenizer.ggml.tokens"),
).toMatchObject({
element_type: "STRING",
element_count: 10,
preview_complete: false,
value: Array.from({ length: 8 }, (_, n) => `token-${String(n)}`),
});
expect(
result.value.tensors.map(({ name, shape, byte_length, file_offset }) => ({
name,
shape,
byte_length,
file_offset,
})),
).toEqual([
{
name: "weight",
shape: ["2", "2"],
byte_length: 16,
file_offset: fixture.dataOffset,
},
{
name: "quant",
shape: ["32"],
byte_length: 18,
file_offset: fixture.dataOffset + 32,
},
]);
expect(result.value.totals).toMatchObject({
parameter_count: "36",
tensor_bytes: 34,
});
const full = await provider.inspect({ path, hash: "full" });
if (!full.ok) throw full.error;
expect(full.value.artifact.sha256).toBe(hash(fixture.bytes));
expect(full.value.artifact.header_sha256).toBe(
result.value.artifact.header_sha256,
);
},
);
it.each([false, true])(
"preserves GGUF empty extents, quantized empty rows and scalars, endian=%s",
async (bigEndian) => {
const declarations = [
{ name: "empty_first", type: 0, shape: [0n, 7n], offset: 0n, bytes: 0 },
{ name: "empty_later", type: 0, shape: [3n, 0n], offset: 0n, bytes: 0 },
{
name: "empty_quant_first",
type: 2,
shape: [0n, 32n],
offset: 0n,
bytes: 0,
},
{
name: "empty_quant_later",
type: 2,
shape: [32n, 0n],
offset: 0n,
bytes: 0,
},
{ name: "scalar", type: 0, shape: [], offset: 0n, bytes: 4 },
];
const fixture = ggufFixture({ bigEndian, tensors: declarations });
const { provider, path } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(
result.value.tensors.map(
({ name, shape, byte_length, parameter_count }) => ({
name,
shape,
byte_length,
parameter_count,
}),
),
).toEqual(
declarations.map(({ name, shape, bytes }) => ({
name,
shape: shape.map(String),
byte_length: bytes,
parameter_count: name === "scalar" ? "1" : "0",
})),
);
expect(result.value.totals).toMatchObject({
parameter_count: "1",
tensor_bytes: 4,
});
expect(result.value.artifact.bytes_read).toBe(fixture.header.length);
},
);
it.each([
{ name: "excess rank", type: 0, shape: [1n, 1n, 1n, 1n, 1n], bytes: 4 },
{ name: "quantized scalar", type: 2, shape: [], bytes: 0 },
])(
"rejects GGUF $name without relaxing representation constraints",
async (tensor) => {
const fixture = ggufFixture({ tensors: [{ ...tensor, offset: 0n }] });
const { provider, path } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
expect(result.ok).toBe(false);
if (result.ok) throw new Error("Expected format refusal");
expect(result.error).toMatchObject({
constraint: "file_format_rejected",
});
},
);
it.each([8, 24, 40])(
"accepts format-valid GGUF alignment %s and preserves exact offsets",
async (alignment) => {
const fixture = ggufFixture({ alignment });
const { path, provider } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(result.value.alignment).toBe(alignment);
expect(result.value.tensor_data_offset).toBe(fixture.dataOffset);
expect(
result.value.tensors.every((row) => row.file_offset % alignment === 0),
).toBe(true);
expect(result.value.tensors[1]?.file_offset).toBe(
fixture.dataOffset + Math.ceil(16 / alignment) * alignment,
);
},
);
it.each([0, 1, 2, 4, 12])(
"rejects GGUF alignment %s outside positive multiples of eight",
async (alignment) => {
const base = ggufFixture();
const fixture = ggufFixture({
metadata: [
{ key: "general.alignment", type: 4, value: base.u32(alignment) },
],
});
const { path, provider } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (result.ok) throw new Error("Invalid alignment accepted");
expect(result.error).toMatchObject({
_tag: "BinaryTargetError",
constraint: "file_format_rejected",
});
},
);
it.each([
"General.name",
"general.naïve",
"general..name",
"general/name",
"general.-name",
"general.name_",
"general.__name",
])(
"rejects invalid GGUF metadata key %s before publishing facts",
async (key) => {
const fixture = ggufFixture({
metadata: [{ key, type: 7, value: Buffer.from([1]) }],
});
const { path, provider } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (result.ok) throw new Error("Invalid metadata key accepted");
expect(result.error).toMatchObject({
_tag: "BinaryTargetError",
constraint: "file_format_rejected",
});
},
);
it("accepts standardized GGUF metadata keys containing numeric hierarchy components", async () => {
const base = ggufFixture();
const fixture = ggufFixture({
metadata: [
{ key: "general.base_model.0.name", type: 8, value: base.text("parent") },
],
});
const { path, provider } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(result.value.metadata[0]).toMatchObject({
key: "general.base_model.0.name",
value: "parent",
});
});
it.each([99_999_993, 100_000_000])(
"admits safetensors declared header length %s before its independent representation limit",
async (length) => {
const bytes = Buffer.alloc(9);
bytes.writeBigUInt64LE(BigInt(length));
bytes[8] = 0x7b;
const { path, provider } = await providerAt(bytes);
const file = await open(path, "r+");
try {
await file.truncate(8 + length);
} finally {
await file.close();
}
const result = await provider.inspect({ path, hash: "header" });
if (result.ok) throw new Error("Representation limit was bypassed");
expect(result.error).toMatchObject({
_tag: "AnalysisResourceConstraintError",
reason: expect.stringContaining("32 MiB"),
});
},
);
it("retains the absolute header budget for GGUF without allocating that extent", async () => {
const { path } = await providerAt(Buffer.alloc(0));
const file = await open(path, "r+");
onTestFinished(() => file.close());
await file.truncate(100_000_009);
const reader = new ModelHeaderReader(file, 100_000_009);
expect(() => reader.guarantee(100_000_000)).not.toThrow();
expect(() => reader.guarantee(100_000_001)).toThrow("100,000,000 bytes");
});
it("preserves safetensors scalar, empty and packed tensor declarations and raw string metadata", async () => {
const fixture = safetensorsFixture();
const { provider, path } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(result.value.artifact.header_sha256).toBe(hash(fixture.header));
expect(
result.value.tensors.map((row) => [row.name, row.shape, row.byte_length]),
).toEqual([
["empty", ["0", "7"], 0],
["weight", ["2", "2"], 16],
]);
expect(result.value.metadata.map((row) => [row.key, row.value])).toEqual([
["format", "pt"],
["license", "Apache-2.0"],
]);
const packed = safetensorsFixture(
JSON.stringify({
scalar: { dtype: "F4", shape: [2], data_offsets: [0, 1] },
empty: { dtype: "F32", shape: [0], data_offsets: [0, 0] },
}),
Buffer.alloc(1),
);
await writeFile(path, packed.bytes);
const checked = await provider.inspect({ path, hash: "full" });
if (!checked.ok) throw checked.error;
expect(checked.value.totals).toMatchObject({
parameter_count: "2",
tensor_bytes: 1,
});
});
it("returns unknown lengths for unknown GGUF types without inventing storage size", async () => {
const fixture = ggufFixture({
tensors: [{ name: "future", type: 999, shape: [1n], offset: 0n, bytes: 1 }],
});
const { provider, path } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(result.value.tensors[0]).toMatchObject({
type_id: 999,
type: "UNKNOWN_999",
byte_length: null,
byte_length_state: "unknown",
});
expect(result.value.coverage.tensor_ranges).toBe("partial");
expect(result.value.totals.tensor_bytes).toBeNull();
});
it("keeps unknown safetensors storage validation separate from observed valid byte ranges", async () => {
const fixture = safetensorsFixture(
'{"future":{"dtype":"FUTURE4","shape":[2],"data_offsets":[0,4]}}',
Buffer.alloc(4),
);
const { path, provider } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(result.value.tensors[0]).toMatchObject({
byte_length: 4,
byte_length_state: "declared",
storage_validation: "unknown",
});
expect(result.value.coverage).toMatchObject({
tensor_ranges: "complete",
storage_sizes: "partial",
});
});
it("preserves nonfinite floating metadata and negative zero without JSON coercion", async () => {
const values = [NaN, Infinity, -Infinity, -0];
const fixture = ggufFixture({
metadata: values.map((value, index) => {
const b = Buffer.alloc(8);
b.writeDoubleLE(value);
return { key: `custom.float_${String(index)}`, type: 12, value: b };
}),
});
const { path, provider } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(result.value.metadata.map((row) => [row.type, row.value])).toEqual(
["NaN", "+Infinity", "-Infinity", "-0"].map((value) => ["FLOAT64", value]),
);
});
it("validates booleans beyond an array preview without allocating the full value list", async () => {
const base = ggufFixture();
const values = Buffer.alloc(10000);
values[9999] = 2;
const fixture = ggufFixture({
metadata: [
{
key: "custom.flags",
type: 9,
value: Buffer.concat([base.u32(7), base.u64(10000n), values]),
},
],
});
const { path, provider } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (result.ok) throw new Error("Invalid omitted boolean was accepted");
expect(result.error).toMatchObject({
_tag: "BinaryTargetError",
constraint: "file_format_rejected",
});
});
it("refuses an oversized declared header before reading or expanding it", async () => {
const bytes = Buffer.alloc(9);
bytes.writeBigUInt64LE(100_000_001n);
bytes[8] = 0x7b;
const { path, provider } = await providerAt(bytes);
const file = await open(path, "r+");
try {
await file.truncate(100_000_009);
} finally {
await file.close();
}
const result = await provider.inspect({ path, hash: "header" });
if (result.ok) throw new Error("Oversized header was accepted");
expect(result.error).toMatchObject({
_tag: "AnalysisResourceConstraintError",
retryable: false,
reportedLimits: { maximum_header_bytes: 100_000_000 },
});
});
const malformed = [
["truncated GGUF", () => ggufFixture().bytes.subarray(0, 21)],
[
"invalid UTF8",
() => {
const f = ggufFixture({
metadata: [
{
key: "x",
type: 8,
value: Buffer.concat([
Buffer.from([1, 0, 0, 0, 0, 0, 0, 0]),
Buffer.from([0xff]),
]),
},
],
});
return f.bytes;
},
],
[
"duplicate GGUF metadata",
() =>
ggufFixture({
metadata: [
{ key: "x", type: 7, value: Buffer.from([0]) },
{ key: "x", type: 7, value: Buffer.from([1]) },
],
}).bytes,
],
[
"invalid boolean",
() =>
ggufFixture({
metadata: [{ key: "x", type: 7, value: Buffer.from([2]) }],
}).bytes,
],
[
"misaligned GGUF tensor",
() =>
ggufFixture({
tensors: [{ name: "x", type: 0, shape: [1n], offset: 1n, bytes: 4 }],
}).bytes,
],
[
"invalid quantization row",
() =>
ggufFixture({
tensors: [
{ name: "x", type: 2, shape: [16n, 2n], offset: 0n, bytes: 18 },
],
}).bytes,
],
[
"overlapping GGUF tensors",
() =>
ggufFixture({
tensors: [
{ name: "a", type: 0, shape: [1n], offset: 0n, bytes: 4 },
{ name: "b", type: 0, shape: [1n], offset: 0n, bytes: 4 },
],
}).bytes,
],
[
"duplicate JSON keys",
() =>
safetensorsFixture(
'{"x":{"dtype":"F32","dtype":"F16","shape":[2],"data_offsets":[0,4]}}',
Buffer.alloc(4),
).bytes,
],
[
"safetensors shape mismatch",
() =>
safetensorsFixture(
'{"x":{"dtype":"F32","shape":[3],"data_offsets":[0,16]}}',
).bytes,
],
[
"safetensors gap",
() =>
safetensorsFixture(
'{"x":{"dtype":"F32","shape":[3],"data_offsets":[4,16]}}',
).bytes,
],
[
"safetensors overlap",
() =>
safetensorsFixture(
'{"a":{"dtype":"F32","shape":[4],"data_offsets":[0,16]},"b":{"dtype":"F32","shape":[1],"data_offsets":[0,4]}}',
).bytes,
],
["safetensors trailing data", () => safetensorsFixture("{}").bytes],
[
"safetensors non-string metadata",
() => safetensorsFixture('{"__metadata__":{"x":1}}', Buffer.alloc(0)).bytes,
],
[
"safetensors unpaired surrogate",
() =>
safetensorsFixture('{"__metadata__":{"x":"\\ud800"}}', Buffer.alloc(0))
.bytes,
],
] as const;
it.each(malformed)(
"rejects %s with typed file_format_rejected",
async (_name, make) => {
const { provider, path } = await providerAt(make());
const result = await provider.inspect({ path, hash: "header" });
if (result.ok) throw new Error("Malformed model was accepted");
expect(result.error).toMatchObject({
_tag: "BinaryTargetError",
constraint: "file_format_rejected",
});
},
);
it("bounds expansion before allocation and distinguishes unsupported pickle, version and paths", async () => {
const fixture = ggufFixture();
fixture.bytes.writeBigUInt64LE(0xffffffffffffffffn, 8);
const { provider, path, root } = await providerAt(fixture.bytes);
const huge = await provider.inspect({ path, hash: "header" });
if (huge.ok) throw new Error("Huge count admitted");
expect(huge.error._tag).toBe("AnalysisResourceConstraintError");
await writeFile(path, Buffer.from([0x80, 4, 0, 0]));
const pickle = await provider.inspect({ path, hash: "header" });
if (pickle.ok) throw new Error("Pickle admitted");
expect(pickle.error._tag).toBe("AnalysisUnsupportedTargetError");
await writeFile(path, ggufFixture({ version: 999 }).bytes);
const version = await provider.inspect({ path, hash: "header" });
if (version.ok) throw new Error("Unknown version admitted");
expect(version.error._tag).toBe("AnalysisUnsupportedTargetError");
const alias = join(root, "link");
await symlink(path, alias);
const linked = await provider.inspect({ path: alias, hash: "header" });
expect(linked.ok).toBe(false);
const service = createModelArtifactService();
onTestFinished(() => service.close());
const invalid = await service.inspect({
path: "relative",
detail: "complete",
});
if (invalid.ok) throw new Error("Relative path admitted");
expect(invalid.error._tag).toBe("AnalysisInputError");
});
it("reads only the exact header, even when it ends in the middle of a normal buffer", async () => {
const fixture = safetensorsFixture();
const { path } = await providerAt(fixture.bytes);
const file = await open(path, "r");
onTestFinished(() => file.close());
const requests: [number, number][] = [];
const read = file.read.bind(file);
const boundary = {
read: async (
buffer: Buffer,
offset: number,
length: number,
position: number,
) => {
requests.push([position, length]);
expect(position + length).toBeLessThanOrEqual(fixture.header.length);
return read(buffer, offset, length, position);
},
} as Pick<typeof file, "read">;
const reader = new ModelHeaderReader(
boundary as typeof file,
fixture.bytes.length,
);
await readSafetensorsHeader(reader, await reader.take(4));
expect(await reader.finish(false)).toMatchObject({
bytes_read: fixture.header.length,
header_sha256: hash(fixture.header),
});
expect(requests.reduce((n, [, length]) => n + length, 0)).toBe(
fixture.header.length,
);
});
it.each(["header", "full"] as const)(
"classifies real truncation after admission during %s reading as an identity change",
async (mode) => {
const fixture = ggufFixture();
const { path } = await providerAt(fixture.bytes);
const file = await open(path, "r");
onTestFinished(() => file.close());
const admitted = await file.stat();
const reader = new ModelHeaderReader(file, admitted.size);
if (mode === "full") await reader.take(fixture.header.length);
await truncate(path, mode === "header" ? 0 : fixture.header.length);
const reading = mode === "header" ? reader.take(4) : reader.finish(true);
await expect(reading).rejects.toMatchObject({ reason: "integrity" });
},
);
it("keeps GGUF read-ahead inside the directory and hashes precisely that prefix", async () => {
const fixture = ggufFixture();
const { path } = await providerAt(fixture.bytes);
const file = await open(path, "r");
onTestFinished(() => file.close());
const read = file.read.bind(file);
let bytesRead = 0;
const boundary = {
read: async (
buffer: Buffer,
offset: number,
length: number,
position: number,
) => {
expect(position + length).toBeLessThanOrEqual(fixture.header.length);
const result = await read(buffer, offset, length, position);
bytesRead += result.bytesRead;
return result;
},
} as Pick<typeof file, "read">;
const reader = new ModelHeaderReader(
boundary as typeof file,
fixture.bytes.length,
);
await reader.take(4);
await new GgufHeader(reader).read();
await reader.finish(false);
expect(bytesRead).toBe(fixture.header.length);
});
it("inspects a 1 GiB sparse safetensors declaration using only its small header", async () => {
const dataBytes = 1024 ** 3;
const fixture = safetensorsFixture(
JSON.stringify({
weight: { dtype: "U8", shape: [dataBytes], data_offsets: [0, dataBytes] },
}),
Buffer.alloc(0),
);
const { path, provider } = await providerAt(fixture.header);
const file = await open(path, "r+");
try {
await file.truncate(fixture.header.length + dataBytes);
} finally {
await file.close();
}
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(result.value.artifact.bytes_read).toBe(fixture.header.length);
expect(result.value.tensors[0]).toMatchObject({
byte_length: dataBytes,
parameter_count: String(dataBytes),
});
});
it("supports true scalar shapes and prototype-named metadata without changing their spelling", async () => {
const fixture = safetensorsFixture(
'{"__metadata__":{"__proto__":"raw","constructor":"kept"},"scalar":{"dtype":"F32","shape":[],"data_offsets":[0,4]}}',
Buffer.alloc(4),
);
const { path, provider } = await providerAt(fixture.bytes);
const result = await provider.inspect({ path, hash: "header" });
if (!result.ok) throw result.error;
expect(result.value.tensors[0]).toMatchObject({
shape: [],
parameter_count: "1",
byte_length: 4,
});
expect(result.value.metadata.map(({ key, value }) => [key, value])).toEqual([
["__proto__", "raw"],
["constructor", "kept"],
]);
});
it("yields while validating a large metadata array, preserves cancellation and releases admission", async () => {
const f = ggufFixture();
const count = 1_000_000;
const fixture = ggufFixture({
metadata: [
{
key: "custom.bools",
type: 9,
value: Buffer.concat([
f.u32(7),
f.u64(BigInt(count)),
Buffer.alloc(count),
]),
},
],
});
const { path, provider } = await providerAt(fixture.bytes);
const controller = new AbortController();
const reason = new Error("stop large array");
const pending = provider.inspect(
{ path, hash: "header" },
{ signal: controller.signal },
);
setTimeout(() => controller.abort(reason), 0);
const result = await pending;
if (result.ok) throw new Error("Cancellation ignored");
expect(result.error).toMatchObject({
_tag: "AnalysisCancelledError",
cause: reason,
});
await writeFile(path, ggufFixture().bytes);
const recovered = await provider.inspect({ path, hash: "header" });
if (!recovered.ok) throw recovered.error;
expect(recovered.value.tensors).toHaveLength(2);
});
it("retains explicit scoped identity through shared service and preserves caller abort reasons", async () => {
const fixture = ggufFixture();
const { path } = await providerAt(fixture.bytes);
const service = createModelArtifactService();
onTestFinished(() => service.close());
const result = await service.inspect({ path, detail: "complete" });
if (!result.ok) throw result.error;
expect(result.value.subject).toBeNull();
expect(
modelArtifactSchema.parse(result.value.normalized_result).artifact.sha256,
).toBeNull();
const full = await service.inspect({
path,
hash: "full",
detail: "complete",
});
if (!full.ok) throw full.error;
expect(full.value.subject?.digest.sha256).toBe(hash(fixture.bytes));
const controller = new AbortController();
const reason = new Error("selected caller stop");
controller.abort(reason);
const stopped = await service.inspect(
{ path, detail: "complete" },
{ signal: controller.signal },
);
if (stopped.ok) throw new Error("Cancellation ignored");
expect(stopped.error.cause).toBe(reason);
});
+112
View File
@@ -0,0 +1,112 @@
/** Source-built declarations with independent expected byte layouts; no model loader. */
export const ggufFixture = (
options: {
bigEndian?: boolean;
version?: number;
alignment?: number;
metadata?: readonly { key: string; type: number; value: Buffer }[];
tensors?: readonly {
name: string;
type: number;
shape: readonly bigint[];
offset: bigint;
bytes: number;
}[];
} = {},
) => {
const big = options.bigEndian ?? false;
const alignment = options.alignment ?? 32;
const u32 = (n: number) => {
const b = Buffer.alloc(4);
if (big) b.writeUInt32BE(n);
else b.writeUInt32LE(n);
return b;
};
const u64 = (n: bigint) => {
const b = Buffer.alloc(8);
if (big) b.writeBigUInt64BE(n);
else b.writeBigUInt64LE(n);
return b;
};
const text = (s: string) => {
const b = Buffer.from(s);
return Buffer.concat([u64(BigInt(b.length)), b]);
};
const metadata = options.metadata ?? [
{ key: "general.architecture", type: 8, value: text("llama") },
{ key: "general.name", type: 8, value: text("Tiny 🦙") },
{ key: "llama.context_length", type: 10, value: u64(2048n) },
{ key: "custom.exact", type: 10, value: u64(0xffffffffffffffffn) },
{
key: "tokenizer.ggml.tokens",
type: 9,
value: Buffer.concat([
u32(8),
u64(10n),
...Array.from({ length: 10 }, (_, n) => text(`token-${String(n)}`)),
]),
},
{ key: "tokenizer.chat_template", type: 8, value: text("{{ messages }}") },
...(options.alignment === undefined
? []
: [{ key: "general.alignment", type: 4, value: u32(alignment) }]),
];
const tensors = options.tensors ?? [
{ name: "weight", type: 0, shape: [2n, 2n], offset: 0n, bytes: 16 },
{
name: "quant",
type: 2,
shape: [32n],
offset: BigInt(Math.ceil(16 / alignment) * alignment),
bytes: 18,
},
];
const header = Buffer.concat([
Buffer.from("GGUF"),
u32(options.version ?? 3),
u64(BigInt(tensors.length)),
u64(BigInt(metadata.length)),
...metadata.map((m) => Buffer.concat([text(m.key), u32(m.type), m.value])),
...tensors.map((t) =>
Buffer.concat([
text(t.name),
u32(t.shape.length),
...t.shape.map(u64),
u32(t.type),
u64(t.offset),
]),
),
]);
const dataOffset =
tensors.length === 0
? header.length
: Math.ceil(header.length / alignment) * alignment;
const length = Math.max(
dataOffset,
...tensors.map((t) => dataOffset + Number(t.offset) + t.bytes),
);
const bytes = Buffer.alloc(length, 0xa5);
header.copy(bytes);
bytes.fill(0, header.length, dataOffset);
return { bytes, header, dataOffset, u32, u64, text };
};
export const safetensorsFixture = (
header: string = JSON.stringify({
__metadata__: { format: "pt", license: "Apache-2.0" },
weight: { dtype: "F32", shape: [2, 2], data_offsets: [0, 16] },
empty: { dtype: "F16", shape: [0, 7], data_offsets: [16, 16] },
}),
data: Buffer = Buffer.alloc(16, 0xa5),
) => {
const text = Buffer.from(
header + " ".repeat((8 - (Buffer.byteLength(header) % 8)) % 8),
);
const length = Buffer.alloc(8);
length.writeBigUInt64LE(BigInt(text.length));
return {
bytes: Buffer.concat([length, text, data]),
header: Buffer.concat([length, text]),
dataOffset: 8 + text.length,
};
};
@@ -59,6 +59,8 @@ import { ArtifactReaderFailure } from "../../src/artifacts/ArtifactReader.js";
import { GoBinaryProvider } from "../../src/go/GoBinaryProvider.js";
import { PeResourcesProvider } from "../../src/native/pe/PeResourcesProvider.js";
import { Il2cppMetadataProvider } from "../../src/il2cpp/Il2cppMetadataProvider.js";
import { ModelArtifactProvider } from "../../src/model/ModelArtifactProvider.js";
import { ggufFixture } from "../fixtures/model/model.js";
import { inspectPeResourcesInputSchema } from "../../src/domain/native/peResources.js";
import { peResourceFixture } from "../../src/native/pe/PeResources.fixture.js";
import { createServer } from "../../src/server/createServer.js";
@@ -286,6 +288,80 @@ const selectedImage = async () => {
return path;
};
it.each([
"confirmed-close-cancel",
"failed-close",
"failed-close-cancel",
] as const)(
"retains complete scoped model observations through %s and retries the exact file owner",
async (mode) => {
const path = await selectedImage();
await writeFile(path, ggufFixture().bytes);
const provider = new ModelArtifactProvider();
const controller = new AbortController();
const reason = new Error("model close cancellation");
fault.blockNew = mode !== "confirmed-close-cancel";
if (mode !== "failed-close") fault.onClose = () => controller.abort(reason);
try {
const result = await provider.inspect(
{ path, hash: "header" },
{ signal: controller.signal },
);
if (result.ok) throw new Error("Expected finalization failure");
const evidence = parseEvidence(result.error.partialObservation);
expect(evidence.operation).toBe("inspect_model_artifact");
expect(evidence.subject).toBeNull();
expect(evidence.normalized_result).toMatchObject({
artifact: { path, sha256: null, read_scope: "header" },
tensors: [{ name: "weight" }, { name: "quant" }],
});
if (mode === "confirmed-close-cancel") {
expect(result.error).toBeInstanceOf(AnalysisCancelledError);
expect(result.error.cause).toBe(reason);
expect(fault.handles[0]?.fd).toBe(-1);
} else {
expect(result.error.cleanupIncomplete).toBe(true);
if (mode === "failed-close-cancel")
expect(result.error.cleanupPrimary?.cause).toBe(reason);
expect(fault.handles[0]?.fd).toBeGreaterThanOrEqual(0);
fault.blocked.clear();
await provider.close();
expect(result.error.cleanupIncomplete).toBe(false);
expect(fault.handles[0]?.fd).toBe(-1);
}
} finally {
fault.onClose = undefined;
fault.blocked.clear();
await provider.close();
}
},
);
it("preserves a model permission failure when cancellation arrives during close", async () => {
const path = await selectedImage();
const provider = new ModelArtifactProvider();
const denied = Object.assign(new Error("model read denied"), {
code: "EACCES",
});
fault.readFailure = { cause: denied };
fault.blockNew = false;
const controller = new AbortController();
fault.onClose = () => controller.abort(new Error("later stop"));
try {
const result = await provider.inspect(
{ path, hash: "header" },
{ signal: controller.signal },
);
if (result.ok) throw new Error("Read failure disappeared");
expect(result.error).toBeInstanceOf(AnalysisAccessDeniedError);
expect(result.error.cause).toBe(denied);
expect(result.error.partialObservation).toBeUndefined();
} finally {
fault.onClose = undefined;
await provider.close();
}
});
mcpTest(
"retains Rust descriptor cleanup and recovers in the same MCP connection",
async ({ mcp }) => {