Files
4c28a7be9e ci: audit closed-issue delivery claims — cited PRs must actually be merged (#3522)
* ci: audit closed-issue delivery claims — cited PRs must actually be merged

Follow-up to #3223: an issue closed as 'fixed in #N' while #N was closed
without merging (or does not exist) leaves a completion claim that does not
hold. The weekly maintenance workflow now runs a read-only audit over issues
closed in the last 30 days, extracts delivery-claim PR references from issue
bodies and close-window comments, verifies each cited PR's merge state, and
reports findings to the step summary.

Design constraints:
- Read-only: no comments, no closes, no labels; findings are advisory and
  the run always stays green so the weekly audit never blocks maintenance.
- Unverifiable counts as unmerged: a 404 PR or an API failure fails the
  verdict, because an unverifiable claim is exactly the failure mode.
- Pure decision functions are exported for tests; a first live run against
  this repository surfaced one real finding (#3408 cites #3060, which does
  not resolve), included here as the motivating sample.

8 unit/contract tests: extraction, windowing (close comments included),
the #3223 unmerged-citation failure mode, the all-clear summary, workflow
contract, and a no-write guard.

* ci: rework audit evidence contract — claims vs context, four PR states, visible completeness

Revision per the #3522 review (reviewer ran the CLI through 19 controlled
cases plus a bounded live check of #3408):

- Bind findings to asserted delivery claims. Closing wording ('Delivered
  through #N'), claim verbs (fixed/shipped/landed/resolved/...), and bare
  'PR #N' inside closing comments are claims; historical mentions, quoted
  lines, fenced examples, negated claims, and tracking-only references are
  reported as contextual references and never produce findings. On the live
  #3408 the audit now recognizes the closing claim #3500 (merged) instead of
  flagging the historical #3060 text.
- Separate observed states: merged / open-unmerged / closed-unmerged /
  unknown (404, rate limit, transient). Unknown claims are advisory manual-
  verification items, not failed claims; each row carries the claim snippet,
  its source link, and the observation time. Checked claims are listed even
  when passing, so the report is traceable rather than an opaque all-clear.
- Paginate issue and comment collection (--paginate --slurp), bound comment
  requests (200) and PR checks (100), and surface skipped pages, rate limits,
  and retrieval failures as an explicit incomplete audit instead of an empty
  all-clear. A live-run maxBuffer overflow on the paginated issue list was
  caught and fixed (64MB); the audit reported it as incomplete before the fix.
- Scope the job to read access: job-level contents/issues/pull-requests read.

13 contract/unit tests including a CLI fixture (stub gh) that places the
relevant issue on page two and forces a comment-fetch failure.

* ci: report incomplete audit at PR check cap

---------

Co-authored-by: chenhz01 <chenhz01@users.noreply.github.com>
Co-authored-by: haelyra <49814733+haelyra@users.noreply.github.com>
2026-10-25 14:41:22 -04:00

76 lines
2.2 KiB
YAML

name: Scheduled Maintenance
on:
schedule:
- cron: '0 9 * * 1' # Weekly Monday 9am UTC
workflow_dispatch:
permissions:
contents: read
issues: write
pull-requests: write
jobs:
dependency-check:
name: Check Dependencies
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: '20.x'
- name: Check for outdated packages
run: npm outdated || true
security-audit:
name: Security Audit
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: '20.x'
- name: Run security audit
run: |
if [ -f package-lock.json ]; then
npm ci --ignore-scripts
npm audit signatures
npm audit --omit=dev --audit-level=high
else
echo "No package-lock.json found; skipping npm audit"
fi
stale:
name: Stale Issues/PRs
runs-on: ubuntu-latest
steps:
- uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0
with:
stale-issue-message: 'This issue is stale due to inactivity.'
stale-pr-message: 'This PR is stale due to inactivity.'
days-before-stale: 30
days-before-close: 7
closed-reference-audit:
name: Audit Closed-Issue PR References
runs-on: ubuntu-latest
permissions:
contents: read
issues: read
pull-requests: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: '20.x'
- name: Verify cited PRs in closed issues are actually merged
env:
GH_TOKEN: ${{ github.token }}
run: node scripts/ci/audit-closed-references.js