Validating a two-phase self-healing CI design

TL;DR

• Phase 1— deterministic regenerate-and-open-a-PR, never push to main — is strongly validated by industry precedent. Shopify Polaris, Chakra UI, and virtually every Changesets-based design system use exactly this “bot opens a PR, human merges” pattern. Adopt it with confidence.
• The single most important implementation decision is the bot's auth token: don't use the default GITHUB_TOKEN if you want your existing validate/Chromatic checks to run on the auto-generated PR. Use a GitHub App installation token— it's what Shopify's release bot does, and it's the documented fix in the Changesets docs.
• Phase 2 — an AI agent proposing fixes — is technically ready but has essentially no public precedent among the major design systems. Treat it as experimental, scope it tightly, and never let it touch generated artifacts that Phase 1 already owns.

Key findings

1. The “regenerate → open a PR → human merges, never auto-merge” pattern is the industry norm. Shopify Polaris uses the Changesets bot to open a “Version Packages” PR that a human reviews and merges — this directly validates a Phase 1 built the same way.
2. The default GITHUB_TOKEN loop-protection problem is real, well-documented, and hits exactly this use case. GitHub deliberately blocks workflows authenticated with GITHUB_TOKENfrom triggering further workflow runs, so a Phase 1 PR opened with it won't trigger staleness, Chromatic, or validate checks.
3. The recommended fix is a GitHub App token, not a PAT. GitHub's own docs, the Changesets docs, and the peter-evans/create-pull-requestmaintainer all converge on the same answer. Shopify's release bot is a GitHub App.
4. AI-agent-fixes-CI is available as an official pattern but unproven in major design systems. anthropics/claude-code-action ships example workflows including a CI-failure auto-fix. By default it commits to a new branch and opens a PR for human review — auto-merge requires explicitly granted elevated permissions.
5. pull_request_target combined with untrusted-code checkout is the dominant supply-chain footgun, and should be avoided entirely for a solo maintainer. Real RCEs have been found in Microsoft, Google, MITRE, and Splunk repos from exactly this misconfiguration.
6. Concurrency races between existing checks and a new regeneration job are a known problem with a standard fix — GitHub Actions concurrency groups.

How the major design systems handle automated artifact regeneration

Shopify Polaris is the closest confirmed analog to Phase 1. It uses Changesets: on merge to main, the Changesets action creates a changeset-release/main branch and opens a “Version Packages” PR carrying an up-to-date changeset version run — updated changelog and package versions. The PR stays current on every subsequent merge, and a human merges it to release. It's never auto-merged and never pushed directly to main. Commits and PRs are authored by shopify-github-actions-access[bot] — the [bot] suffix confirms a GitHub App installation identity, not a human PAT account.

Chakra UI runs the same Changesets pattern. Its published release workflow uses GITHUB_TOKEN plus npm OIDC trusted publishing, with a concurrency group keyed to workflow and ref. Chakra's author explicitly notes that PRs opened with GITHUB_TOKENdon't trigger other workflows, and that a PAT or App token is needed if CI should run on the version PR.

Carbon, Spectrum, Salesforce Lightning, and Atlaskitcould not be confirmed at the workflow level — Carbon relies on a merge queue and local regeneration with CI verification rather than a visible artifact-regeneration bot; Spectrum uses semantic-release, a different model from the changesets “open a PR” approach; Salesforce and Atlassian develop in internal monorepos, so their exact mechanism isn't publicly knowable. Treat statements about those four as unconfirmed, not as negative evidence.

Polaris is the one fully confirmable match, and it validates every element of a Phase 1 spec: regenerate derived artifacts, open a PR, keep it updated, human merges, and use a GitHub App bot rather than the default token.

The GITHUB_TOKEN problem

This is a documented, deliberate GitHub behavior, not a bug. Per GitHub's own docs: when the repository's GITHUB_TOKEN performs tasks, events it triggers won't create a new workflow run — workflow_dispatch and repository_dispatchare the only exceptions. The one partial carve-out for pull-request events is that any resulting run lands in an approval-required state, needing a human to click “Approve workflows to run.”

The Changesets docs say it directly: to always automatically run workflows for version PRs, use either a personal GitHub token or a GitHub App token, and pass it to the Changesets GitHub Action. peter-evans/create-pull-request — the most widely used generic PR-bot action — documents the same limitation, with the same two fixes: a repo-scoped PAT, or a GitHub App token.

The alternative is a metadata-predicate gate, the pattern NVIDIA-NeMo uses: CI runs automatically only when a maintainer with write access adds a “Run CICD” label or approves the run. It's viable but requires a human to be present to approve, which partly defeats the point of “self-healing.”

AI coding agents fixing CI in production

The capability exists officially. anthropics/claude-code-action ships example workflows including a CI-failure auto-fix and a test-failure-analysis flow. The example is tightly scoped by design: an explicit allow-list of tools — edit, read, grep, and scoped Bash(git:*), Bash(npm:*), Bash(gh:*) — rather than open-ended shell access.

Default behavior is PR-for-review, not auto-merge: the agent commits to a new branch and links to the PR creation page, and a human approves and merges. Automatic merging requires explicitly granting elevated permissions. The recommended trigger is workflow_runon the failed CI workflow's completion, scoped with the same tool allow-list and a turn/token budget to control cost and blast radius.

No major design system is publicly confirmed to run this in production. The closest general-purpose production tooling is CodeRabbit's “fix CI failures” feature — it clones the repo into a sandbox and deliberately excludes CI/build config, lock files, and other dependency manifests, surfacing those separately. That confirms the “agent opens a PR to fix CI, human reviews, and it's walled off from config and dependency files” pattern is being productized, just not yet inside a flagship design system. A documented Claude Code incident also underscores the need for tight tool scoping and a read-only-by-default posture when an agent has broad, ambiguous instructions in CI.

Security and trust for bot-authored PRs

Scope permissions minimally at the workflow or job level — contents: write and pull-requests: write, not the broad default. Avoid pull_request_target with untrusted-code checkout entirely; it's the single most exploited GitHub Actions pattern, with confirmed RCEs in Microsoft, Google, MITRE, and Splunk repos, including the ability to push code back into official branches. A solo maintainer's bots operate on their own repo's code, so plain pull_request, push, or workflow_run triggers are enough.

Apply branch protection equally to bot and human PRs — require the validate check to pass and require review, and don't add the bot to a bypass list. Restrict the regeneration bot to known generated-artifact globs so it can never touch source, ideally enforced with CODEOWNERS. Prefer GitHub App tokens for the auditable, signed-commit trail they produce, and pin third-party actions to a full commit SHA rather than a floating tag.

Race conditions between checks and the regeneration job

There's a real overlap risk: a push that triggers both an existing staleness check and a new regeneration job can race, or two regeneration runs on rapid pushes can both try to touch the same PR branch. A concurrency group keyed to the workflow and ref keeps only one regeneration run per branch executing at a time — but skip cancel-in-progress on main or release runs, where every run must complete.

The cleaner fix is separating the concern: run the staleness check on pull_request so stale artifacts fail the build, and run the regeneration bot on push to main (or on schedule) so it opens a PR — the two never contend for the same branch. create-pull-request is idempotent by design, re-running it updates the existing PR rather than opening duplicates, which combined with a concurrency group eliminates duplicate-PR races. Worth noting for later: concurrency plus cancel-in-progress can break a merge queue if one gets adopted down the line.

Recommendations

Stage 1 — ship Phase 1 with a GitHub App token first

1. Create a minimal GitHub App scoped to only Contents: write and Pull requests: write, installed on the repo. Mint its token in-workflow with actions/create-github-app-token and pass it to the PR-creation step.
2. Trigger the regeneration bot on push to main, plus workflow_dispatch for manual runs. Keep the staleness check on pull_request as a hard build failure.
3. Constrain the bot to commit only generated-artifact globs — never source files.
4. Add a concurrency block scoped to the regeneration workflow and ref.
5. Keep branch protection on main requiring the validate check and review, and confirm the bot can't bypass it.

Stage 2 — pilot Phase 2 in a sandbox, gated hard

1. Adopt anthropics/claude-code-action starting from its CI-failure auto-fix example, triggered via workflow_run on the CI workflow's failure conclusion.
2. Scope it tightly: contents: write and pull-requests: write only, a narrow tool allow-list, a turn/token budget, and an explicit instruction to fix source — never regenerate Phase-1-owned artifacts.
3. Force PR-for-review by default — don't grant merge or auto-merge permissions.
4. Start it on a narrow failure class, like lint or type-check, and on non-main branches only, expanding once the diffs earn trust.

What would change these recommendations: if downstream checks turn out to run on bot PRs while using GITHUB_TOKEN — worth testing explicitly — the App step can be skipped. If Phase 2's fix quality is low or it ever touches files outside its scope, revert it to comment-only analysis mode. And if a merge queue gets adopted later, remove cancel-in-progress from any workflow that feeds it.

Caveats

Only Shopify Polaris and Chakra UI could be confirmed at the workflow level. Carbon, Spectrum, Salesforce, Atlaskit, and Material may or may not regenerate artifacts via a PR bot — that couldn't be confirmed either way, and for Atlassian and Salesforce it may not be publicly knowable at all. The exact token-minting step in Polaris's release workflow couldn't be retrieved either — only that the identity is a GitHub App bot; the actions/create-github-app-tokenrecommendation comes from GitHub's and Changesets' own guidance, not a copy of Polaris's file.

No major design system is publicly confirmed to run an AI-agent-fixes-CI workflow in production — Phase 2 rests on the vendor's official examples and general engineering write-ups, plus CodeRabbit's productized fix-ci feature, not on a flagship precedent. It's an early-adopter bet, not a proven pattern. Some of the source material referenced dated future events — actions/checkout@v7 fork-PR behavior, a December 2025 pull_request_target default change — worth re-verifying against current GitHub behavior before relying on version-specific claims.

Sources

Changesets, Automating Changesets. CodeRabbit, Fix CI Failures. Anthropic, claude-code-action.