mirror of
https://github.com/affaan-m/ECC.git
synced 2026-10-26 18:35:05 +00:00
fix(agents): list files the sanitizer cannot read
Step 1 skipped binaries while the role promised to verify the whole project, so a credential in a spreadsheet, a PDF or an embedded database got a clean PASS and never showed up in the report. Step 7 now lists every file that doesn't read as text, plus any directory it can't enter, and the pipeline stops on that list until a person confirms. On this repo that's a few dozen files, all images and video, which is why they're warnings and not failures. It doesn't try to extract or scan them: that needs nested-archive recursion and size bounds, and I'd rather list a file than half-scan it. Detection is a first-byte signature check plus a whole-file NUL/control-byte test, with file(1) only widening the list. If a detector fails, the file is listed as unverified. Filenames are untrusted, so Step 7 writes the paths to a manifest beside the project directory, for a person to open, and keeps them out of its stdout and its report section. Beside and not inside, so a re-run's text scans can't read it back. The script prints the count itself as its last line. A scan problem, a failed write, a walk that dies early or a run that never reaches that line gives an unknown count, and the gate treats unknown as a stop, not as zero. Step 1 no longer skips *.min.js. Tests in tests/skills/sanitizer-unreadable.test.js. Editing SKILL.md changes the skill registry digest, so skill-triggers@1.json carries the new value.
This commit is contained in:
1 parent
ef648e0189
commit
39450c9be2
4 files changed
+868
-7
No files matched your search
@@ -23,6 +23,7 @@ You are an independent auditor that verifies a forked project is fully sanitized
|
||||
- Scan every file for secret patterns, PII, and internal references
|
||||
- Audit git history for leaked credentials
|
||||
- Verify `.env.example` completeness
|
||||
- List files that cannot be read as text, for a human to open
|
||||
- Generate a detailed PASS/FAIL report
|
||||
- **Read-only** — you never modify files, only report
|
||||
|
||||
@@ -30,7 +31,9 @@ You are an independent auditor that verifies a forked project is fully sanitized
|
||||
|
||||
### Step 1: Secrets Scan (CRITICAL — any match = FAIL)
|
||||
|
||||
Scan every text file (excluding `node_modules`, `.git`, `__pycache__`, `*.min.js`, binaries):
|
||||
Scan every text file (excluding `node_modules`, `.git`, `__pycache__`). Minified bundles are not
|
||||
excluded — a hardcoded key hides in a `*.min.js` as readily as anywhere else. Binaries are not exempt
|
||||
from the audit either — Step 7 lists the files that don't read as text:
|
||||
|
||||
```
|
||||
# API keys
|
||||
@@ -136,6 +139,173 @@ git log --oneline | wc -l
|
||||
git log -p | grep -iE '(password|secret|api.?key|token)' | head -20
|
||||
```
|
||||
|
||||
### Step 7: Unreadable Files (WARNING)
|
||||
|
||||
Steps 1-3 only match text, so a credential sitting in a spreadsheet, a PDF or an embedded database
|
||||
is invisible to them. This pass lists the files that don't read as text, for a human to open. It
|
||||
does not extract or scan their contents.
|
||||
|
||||
The names of these files are untrusted, so this pass keeps them out of its stdout and out of its
|
||||
report section. The block writes one line per path to a manifest beside the project directory,
|
||||
`<project-dir>.UNREADABLE_FILES.txt`, where no scan step reaches it, and prints only a tally by
|
||||
type and a count. A person opens that file. Do not read it or copy anything from it.
|
||||
|
||||
Run it as one shell call from the project directory (`cd PROJECT_DIR`, as in Step 6). The outer
|
||||
part is plain sh and the two find passes call bash. The last line it prints is always
|
||||
`Unreadable count:`. A number means the walk finished. `unknown`, or no such line at all
|
||||
(a timeout, a killed run), means the scan is incomplete: report the count as unknown, never as
|
||||
zero. A few thousand files can take a minute or more, so if the call times out, re-run it with a
|
||||
longer timeout. A non-zero or unknown count means at least PASS WITH WARNINGS. Run by the
|
||||
pipeline, it also stops until a person confirms; run on its own, it just reports the section.
|
||||
|
||||
```bash
|
||||
manifest=$(pwd).UNREADABLE_FILES.txt
|
||||
tab=$(printf "\t")
|
||||
|
||||
# Classify by content, never by extension.
|
||||
# 1. First-byte signature: PK (docx/xlsx/pptx/odt/jar), OLE (legacy doc/xls), PDF, gzip, bzip2, zstd, 7z.
|
||||
# 2. Whole-file test with tr | cmp (grep -q exits on the first printable byte): a NUL or a control
|
||||
# byte other than tab, LF and CR is not plain text. Bytes 0x80-0xFF pass, since UTF-8 uses them.
|
||||
# This is the floor when file(1) is absent.
|
||||
# 3. file(1) only widens, against an opaque-binary allowlist (not "not text/*", which would flag
|
||||
# JSON and JavaScript), and fails closed on an error or on output that is not a mime type.
|
||||
# Over-listing (ANSI logs, form feeds, UTF-16) is the safe direction.
|
||||
#
|
||||
# The walk prints "label<TAB>path" records, "PROBLEM<TAB>reason" when the scan itself went wrong,
|
||||
# and END as its last record. A walk that dies before END gives an unknown count, not a short one.
|
||||
# LC_ALL=C printf %q keeps each path to one ASCII line, so a newline, a right-to-left override or a
|
||||
# zero-width name cannot pass for a second entry.
|
||||
step7_walk() {
|
||||
for t in dd od tr cmp; do
|
||||
command -v "$t" >/dev/null 2>&1 || printf "PROBLEM\t%s is missing, so files cannot be classified\n" "$t"
|
||||
done
|
||||
|
||||
# Pass 1: files.
|
||||
find . \( -name .git -o -name node_modules -o -name __pycache__ \) -prune -o -type f -exec bash -c '
|
||||
shape="^[A-Za-z0-9._+-]+/[A-Za-z0-9._+-]+; charset=[A-Za-z0-9._-]+\$"
|
||||
for f do
|
||||
[ -s "$f" ] || continue
|
||||
q=$(LC_ALL=C printf %q "$f")
|
||||
if ! [ -r "$f" ]; then
|
||||
printf "file not readable\t%s\n" "$q"
|
||||
continue
|
||||
fi
|
||||
label= ctl=
|
||||
sig=$(LC_ALL=C dd if="$f" bs=8 count=1 2>/dev/null | od -An -tx1 | tr -d " \n")
|
||||
case $sig in
|
||||
504b0304*|504b0506*|504b0708*) label="ZIP/OOXML container (PK: docx, xlsx, pptx, odt, jar)";;
|
||||
d0cf11e0a1b11ae1*) label="OLE compound document (legacy doc/xls/ppt/msi)";;
|
||||
25504446*) label="PDF";;
|
||||
1f8b*) label="gzip";;
|
||||
425a68*) label="bzip2";;
|
||||
28b52ffd*) label="zstd";;
|
||||
377abcaf271c*) label="7z";;
|
||||
esac
|
||||
LC_ALL=C tr -d "\000-\010\013\014\016-\037\177" < "$f" | cmp -s - "$f" || ctl=1
|
||||
if [ -z "$label" ] && [ -z "$sig" ]; then
|
||||
label="detector error, unverified"
|
||||
elif [ -z "$label" ] && [ -z "$ctl" ] && [ "${#sig}" -le 2 ]; then
|
||||
continue # one clean byte is text; file(1) calls every one-byte file binary
|
||||
elif [ -z "$label" ] && command -v file >/dev/null 2>&1; then
|
||||
out=$(file -b --mime-type --mime-encoding -- "$f" 2>/dev/null); rc=$?
|
||||
if [ "$rc" -ne 0 ] || ! [[ $out =~ $shape ]]; then
|
||||
[ -n "$ctl" ] || label="detector error, unverified"
|
||||
else
|
||||
mime=${out%%;*}
|
||||
case $mime in
|
||||
image/svg+xml) ;;
|
||||
image/*|audio/*|video/*|font/*|\
|
||||
application/pdf|application/zip|application/gzip|application/x-tar|\
|
||||
application/x-bzip2|application/x-xz|application/zstd|application/x-7z-compressed|\
|
||||
application/vnd.rar|application/x-sqlite3|application/vnd.sqlite3|\
|
||||
application/x-executable|application/x-pie-executable|application/x-sharedlib|\
|
||||
application/x-mach-binary|application/x-dosexec|application/wasm|\
|
||||
application/vnd.microsoft.portable-executable|\
|
||||
application/vnd.openxmlformats-officedocument.*|\
|
||||
application/vnd.oasis.opendocument.*|application/msword|\
|
||||
application/vnd.ms-excel|application/vnd.ms-powerpoint|\
|
||||
application/x-ole-storage|application/vnd.ms-office|application/octet-stream)
|
||||
label=$mime ;;
|
||||
esac
|
||||
if [ -z "$label" ]; then
|
||||
case ${out##*charset=} in
|
||||
binary) label="binary (file mime-encoding)" ;;
|
||||
unknown-8bit) label="unknown 8-bit encoding, unverified" ;;
|
||||
esac
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
if [ -z "$label" ]; then
|
||||
[ -n "$ctl" ] || continue
|
||||
label="contains NUL or control bytes (not plain text)"
|
||||
fi
|
||||
printf "%s\t%s\n" "$label" "$q"
|
||||
done' _ {} + 2>/dev/null ||
|
||||
printf "PROBLEM\tthe file walk reported an error, so some files may not have been checked\n"
|
||||
|
||||
# Pass 2: directories the walk cannot enter (an r/x test, since find error text differs between
|
||||
# GNU find, bfs and busybox).
|
||||
find . \( -name .git -o -name node_modules -o -name __pycache__ \) -prune -o -type d -exec bash -c '
|
||||
for d do
|
||||
{ [ -r "$d" ] && [ -x "$d" ]; } ||
|
||||
printf "directory not readable, contents unscanned\t%s\n" "$(LC_ALL=C printf %q "$d")"
|
||||
done' _ {} + 2>/dev/null ||
|
||||
printf "PROBLEM\tthe directory walk reported an error, so some directories may not have been checked\n"
|
||||
|
||||
printf "END\tof walk\n"
|
||||
}
|
||||
|
||||
# Paths go to the manifest, never to stdout. It sits beside the project, not inside it, so a
|
||||
# re-run of Steps 1-3 cannot grep it. Nothing here depends on a temp file or on sort: a failed
|
||||
# write or a scan problem turns the count into "unknown" instead of a short list.
|
||||
rm -f "$manifest" 2>/dev/null
|
||||
step7_walk | {
|
||||
n=0; problems=0; werr=; ended=
|
||||
while IFS="$tab" read -r label q; do
|
||||
if [ "$label" = PROBLEM ]; then
|
||||
problems=$((problems + 1)); echo "Scan problem: $q"
|
||||
elif [ "$label" = END ]; then
|
||||
ended=1
|
||||
else
|
||||
n=$((n + 1))
|
||||
{ printf "%s\t%s\n" "$q" "$label" >> "$manifest"; } 2>/dev/null || werr=1
|
||||
fi
|
||||
done
|
||||
if [ -n "$werr" ]; then
|
||||
problems=$((problems + 1)); echo "Scan problem: could not write every path to the manifest"
|
||||
fi
|
||||
if [ -z "$ended" ]; then
|
||||
problems=$((problems + 1)); echo "Scan problem: the walk ended early, so some files were not checked"
|
||||
fi
|
||||
# Tally by type, from the second column only.
|
||||
[ "$n" -eq 0 ] || { cut -f2 "$manifest" | LC_ALL=C sort | uniq -c; } 2>/dev/null
|
||||
if [ "$problems" -gt 0 ]; then
|
||||
echo "Unreadable count: unknown ($n listed, $problems scan problems)"
|
||||
else
|
||||
echo "Unreadable count: $n"
|
||||
fi
|
||||
}
|
||||
```
|
||||
|
||||
Each line of the manifest is a path quoted with `printf %q`, a tab, and the reason, so an odd
|
||||
name shows up as `$'...'` and cannot pass for two entries. Every path there needs a person to
|
||||
open the file and say what is in it — do not infer from the extension or the mime type. A count of
|
||||
zero is not proof of a clean repo.
|
||||
|
||||
Not listed, so not covered by this pass:
|
||||
|
||||
- `.git`, `node_modules` and `__pycache__` at any depth (`-name` matches a basename, files included),
|
||||
content reachable only through a symlink, and zero-byte files. Git history is out of scope too:
|
||||
Step 6's `git log -p` shows a removed binary only as `Binary files differ`.
|
||||
- SVG. It is text, so Steps 1-3 scan it.
|
||||
- What Steps 1-3 and Step 6 print. They don't consult this list, so a match in a listed file with
|
||||
no NUL byte still prints its line, and grep names a binary file that matches.
|
||||
- Payloads stored as printable text: base64 in an `.eml`, a Git LFS pointer, or a hand-built file
|
||||
with no NUL, control byte or known signature. file(1) catches part of that last kind, when it
|
||||
reports an unknown 8-bit encoding, and nothing does when it is absent. They read as text, and the
|
||||
Steps 1-3 patterns don't match them either. A minified bundle is scanned by Step 1, but a
|
||||
high-entropy value in the middle of its one long line can still slip the line-anchored entropy check.
|
||||
|
||||
## Output Format
|
||||
|
||||
Generate `SANITIZATION_REPORT.md` in the project directory:
|
||||
@@ -157,6 +327,7 @@ Generate `SANITIZATION_REPORT.md` in the project directory:
|
||||
| Dangerous Files | PASS/FAIL | {count} findings |
|
||||
| Config Completeness | PASS/WARN | {count} findings |
|
||||
| Git History | PASS/FAIL | {count} findings |
|
||||
| Unreadable Files | PASS/WARN | {count} findings |
|
||||
|
||||
## Critical Findings (Must Fix Before Release)
|
||||
|
||||
@@ -167,6 +338,21 @@ Generate `SANITIZATION_REPORT.md` in the project directory:
|
||||
|
||||
1. **[CONFIG]** `src/app.py:8` — Port 8080 hardcoded, should be configurable
|
||||
|
||||
## Unreadable Files
|
||||
|
||||
**Unreadable count: {the number, or `unknown`, from the last line Step 7 printed}**
|
||||
|
||||
- 5 image/png
|
||||
- 1 PDF
|
||||
- 1 ZIP/OOXML container (PK: docx, xlsx, pptx, odt, jar)
|
||||
|
||||
{One bullet per tally line Step 7 printed, as above, then each `Scan problem:` line it printed. If
|
||||
the count is unknown, or Step 7 printed no count, say the scan did not finish.}
|
||||
|
||||
The pipeline gate acts on this count, not on the verdict. The paths are in
|
||||
`{project-dir}.UNREADABLE_FILES.txt`, beside the project directory, and this section carries no
|
||||
filenames. A person must open each file before release.
|
||||
|
||||
## .env.example Audit
|
||||
|
||||
- Variables in code but NOT in .env.example: {list}
|
||||
@@ -176,21 +362,28 @@ Generate `SANITIZATION_REPORT.md` in the project directory:
|
||||
|
||||
{If FAIL: "Fix the {N} critical findings and re-run sanitizer."}
|
||||
{If PASS: "Project is clear for open-source release. Proceed to packager."}
|
||||
{If WARNINGS: "Project passes critical checks. Review {N} warnings before release."}
|
||||
{If WARNINGS: "Project passes critical checks. Review {N} warnings and open the {M} unreadable files before release."}
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example: Scan a sanitized Node.js project
|
||||
Input: `Verify project: /home/user/opensource-staging/my-api`
|
||||
Action: Runs all 6 scan categories across 47 files, checks git log (1 commit), verifies `.env.example` covers 5 variables found in code
|
||||
Action: Runs all 7 scan categories across 47 files, checks git log (1 commit), verifies `.env.example` covers 5 variables found in code
|
||||
Output: `SANITIZATION_REPORT.md` — PASS WITH WARNINGS (one hardcoded port in README)
|
||||
|
||||
## Rules
|
||||
|
||||
- **Never** display full secret values — truncate to first 4 chars + "..."
|
||||
- **Never** modify source files — only generate reports (SANITIZATION_REPORT.md)
|
||||
- **Never** modify source files — only generate reports (SANITIZATION_REPORT.md, and the Step 7
|
||||
manifest beside the project)
|
||||
- **Always** scan every text file, not just known extensions
|
||||
- **Always** run Step 7 and copy its count into the report's `## Unreadable Files` section; the
|
||||
pipeline gate reads that section, not stdout. A non-zero or unknown count is never a plain PASS.
|
||||
- **Never** read the `.UNREADABLE_FILES.txt` manifest or copy a path from it into the report or a
|
||||
reply
|
||||
- **Never** force a binary through the text patterns (`grep -a`, `--text`, `strings`). Step 7
|
||||
lists it for a person instead
|
||||
- **Always** check git history, even for fresh repos
|
||||
- **Be paranoid** — false positives are acceptable, false negatives are not
|
||||
- A single CRITICAL finding in any category = overall FAIL
|
||||
|
||||
Reference in new issue
Block a user