The Agentic Code Review Stack: Catching 'Almost Right' Bugs with Kiro CLI, GitHub Actions, and Sandboxed Agents

Every engineer has shipped an "almost right" fix. The code compiles. The unit tests pass. The PR looks clean. Three weeks later, a production incident trace points straight to the one branch you didn't test — the null-coalescing chain that silently swallowed an empty array, or the race condition that only appears when two jobs run in parallel. These are not logic errors; they are contextual errors, and they are precisely what conventional review tooling misses.

This tutorial builds a complete agentic code review stack that catches "almost right" bugs before they reach main. You'll combine Kiro CLI (AWS's agentic coding CLI) for autonomous analysis, GitHub Actions for orchestration and gating, and sandboxed autonomous agents (Docker-isolated) for safe, repeatable execution. By the end, you'll have a merge-gated pipeline that reviews every PR with an agent that can actually run your code, explore the call graph, and flag the subtle semantic drift that linters and human eyes skip.

Why this stack? Kiro CLI brings agentic reasoning to the terminal; GitHub Actions gives you a reliable trigger and a merge gate; the sandbox gives you the confidence to let an autonomous agent run untrusted code without burning down your infrastructure.

• 1. The "Almost Right" Bug Problem
• 2. Prerequisites & Stack Overview
• 3. Architecture Overview
• 4. Step 1 — Install & Configure Kiro CLI
• 5. Step 2 — Write the Sandboxed Agent
• 6. Step 3 — Wire Kiro into GitHub Actions
• 7. Step 4 — Build the Review Orchestrator
• 8. Step 5 — Harden the Sandbox
• 9. Step 6 — Gate the Merge
• 10. Production Best Practices
• 11. Edge Cases & Failure Modes
• 12. Frequently Asked Questions

1. The "Almost Right" Bug Problem

Let's define the target precisely. An "almost right" bug is one that satisfies all static and shallow checks but violates a runtime or contextual invariant. It's not a typo, and it's not a missing semicolon — the compiler is happy, the linter is silent, and the existing test suite is green.

The three flavors

Human reviewers are bad at these because they review diffs, not executions. A human reads the changed lines; an agentic reviewer can execute the code, trace the data flow, and compare the behavior against the module's own contract.

Why linters and static analysis don't catch them

Linters (ESLint, Ruff, etc.) operate on the AST. They catch patterns, not semantics. TypeScript catches type mismatches, but it can't know that Array.prototype.sort mutates in place and that your caller relied on the original order. These tools are necessary but insufficient.

The gap agentic review fills

An autonomous agent with a sandbox can:

• Run the test suite against the PR branch and compare to baseline.
• Mutate the input space — feed edge-case payloads (empty arrays, null, NaN, huge numbers) to the changed functions.
• Inspect the call graph to find downstream consumers who assumed the old contract.
• Synthesize a regression test that fails on the PR branch and passes on main.

That last capability — writing a failing test — is the killer feature. A bug that can be demonstrated by a failing test is a bug that cannot be argued away.

2. Prerequisites & Stack Overview

Before we write any code, make sure you have the following.

Prerequisites

• Kiro CLI installed and authenticated (see Step 1).
• A GitHub repository with Actions enabled.
• Docker available on the runner (use a self-hosted runner or a GitHub-hosted ubuntu-latest runner with Docker).
• Node.js 20+ if you're using the orchestrator script I'll provide.
• A GitHub token with pull-requests: write and checks: write permissions (for posting comments and creating status checks).

Stack components

3. Architecture Overview

Here's the end-to-end flow, as a diagram-in-words:

[PR opened / updated]
        │
        ▼
[GitHub Actions workflow: agentic-review]
        │
        ├── checkout PR branch
        ├── build project (to catch compile errors early)
        ├── pull sandbox image (cached)
        ├── run sandboxed Kiro agent  ──►  [Docker: --network none, --read-only, --memory 2g]
        │                                     │
        │                                     └── writes review-output.json
        ├── run orchestrator (policy layer)
        │       ├── parse findings
        │       ├── filter by severity / rule allowlist
        │       └── emit status: success | failure | neutral
        └── post PR comment + set status check

The pipeline has four distinct layers:

1. Trigger layer — GitHub Actions decides when to review.
2. Isolation layer — the sandbox decides how safely the agent runs.
3. Cognition layer — Kiro decides what to think about.
4. Policy layer — the orchestrator decides what to do about it.

Keeping these layers separate is the key to maintainability. If you want to swap Kiro for another agent (e.g., claude -p or aider), you only touch the cognition layer. If you want to change the merge policy, you only touch the policy layer.

4. Step 1 — Install & Configure Kiro CLI

Kiro CLI is the headless, CI-friendly interface to the Kiro agentic environment. It lets you run an autonomous coding agent from the command line, which is exactly what we need for a PR review pipeline.

4.1 Install

# Install globally via npm (recommended for CI)
npm install -g @aws/kiro-cli

# Or via the official installer script
curl -fsSL https://kiro.dev/install.sh | bash

# Verify
kiro --version

Note: Kiro CLI is under active development. Pin a specific version in your CI (npm install -g @aws/kiro-cli@<version>) so your pipeline doesn't break when a new release ships.

4.2 Authenticate

Kiro needs credentials to call the underlying model. In CI, use a secret:

export KIRO_API_KEY="${{ secrets.KIRO_API_KEY }}"
kiro auth login --api-key "$KIRO_API_KEY"

For local development, you can use the interactive login:

kiro auth login

4.3 Configure Kiro for review mode

Create a kiro.config.json at the repo root (or .kiro/config.json):

{
  "model": "kiro-1.5-pro",
  "mode": "review",
  "maxTokens": 8000,
  "temperature": 0.1,
  "timeoutMs": 120000,
  "review": {
    "scope": "diff",
    "includeTests": true,
    "severityThreshold": "warning",
    "outputFormat": "json"
  }
}

Key fields:

• "mode": "review" — tells Kiro to behave as a reviewer, not a code generator.
• "scope": "diff" — limits the agent's attention to the PR diff (critical for cost and latency).
• "outputFormat": "json" — gives us machine-readable findings for the orchestrator.

4.4 Smoke test

# Run Kiro against a single file to confirm it works
kiro run --prompt "Review src/utils/date.ts for edge-case bugs" --output review.json
cat review.json | jq '.findings | length'

If this prints a number (even 0), Kiro is working.

5. Step 2 — Write the Sandboxed Agent

Now we wrap Kiro in a Docker sandbox. The sandbox is what makes this stack safe — it means an autonomous agent can run arbitrary code from a PR without risking your CI infrastructure.

5.1 The Dockerfile

Create sandbox/Dockerfile:

FROM node:20-slim

# Minimal tooling: git for diffing, jq for JSON parsing
RUN apt-get update \
    && apt-get install -y --no-install-recommends git jq \
    && rm -rf /var/lib/apt/lists/*

# Install Kiro CLI inside the sandbox
RUN npm install -g @aws/kiro-cli@1.2.3

# The agent will run as a non-root user
RUN useradd -m -u 1000 agent
USER agent

WORKDIR /workspace

# Default entrypoint: the agent runner script
COPY --chown=agent:agent run-agent.sh /usr/local/bin/run-agent.sh
RUN chmod +x /usr/local/bin/run-agent.sh

ENTRYPOINT ["run-agent.sh"]

5.2 The agent runner script

Create sandbox/run-agent.sh:

#!/usr/bin/env bash
set -euo pipefail

# The PR diff is mounted at /workspace/diff.patch
# The repo is mounted read-only at /workspace/repo

export KIRO_API_KEY="${KIRO_API_KEY:?KIRO_API_KEY is required}"

cd /workspace/repo

# Generate the prompt from the diff
DIFF_SIZE=$(wc -l < /workspace/diff.patch)
PROMPT="You are a senior code reviewer. Review the changes in /workspace/diff.patch.
Focus ONLY on bugs that would pass existing tests but fail in production:
- semantic drift (e.g., null vs empty array)
- concurrency hazards (shared mutable state)
- contract violations (return type vs API contract)
- edge cases (empty inputs, boundary values)

For each finding, provide:
1. A severity (critical | warning | info)
2. A file path and line number
3. A one-sentence explanation
4. A concrete reproduction (a test that would fail)

Output your findings as JSON to /workspace/review-output.json.
Do not modify any source files. Do not run the test suite unless asked.
Diff has ${DIFF_SIZE} lines."

# Run Kiro in review mode
kiro run \
  --prompt "$PROMPT" \
  --config /workspace/kiro.config.json \
  --output /workspace/review-output.json

# Validate the output
if ! jq -e '.findings' /workspace/review-output.json > /dev/null 2>&1; then
  echo "ERROR: Kiro did not produce valid findings JSON" >&2
  exit 1
fi

echo "Review complete: $(jq '.findings | length' /workspace/review-output.json) findings"

5.3 Build the sandbox

docker build -t kiro-sandbox:latest ./sandbox

Why a separate Dockerfile? You could run Kiro directly on the runner, but then the agent has access to your CI secrets, your network, and your entire filesystem. The sandbox is the difference between "agentic review" and "agentic liability."

6. Step 3 — Wire Kiro into GitHub Actions

Now we create the GitHub Actions workflow that triggers on every PR and runs the sandboxed agent.

Create .github/workflows/agentic-review.yml:

name: Agentic Code Review

on:
  pull_request:
    types: [opened, synchronize, reopened]

permissions:
  contents: read
  pull-requests: write
  checks: write

jobs:
  agentic-review:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - name: Checkout PR branch
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Build project (early compile check)
        run: npm run build
        continue-on-error: true

      - name: Generate PR diff
        run: |
          git diff origin/${{ github.base_ref }}...HEAD > diff.patch
          echo "Diff lines: $(wc -l < diff.patch)"

      - name: Build sandbox image
        run: docker build -t kiro-sandbox:latest ./sandbox

      - name: Run sandboxed Kiro agent
        env:
          KIRO_API_KEY: ${{ secrets.KIRO_API_KEY }}
        run: |
          docker run --rm \
            --name kiro-review \
            --network none \
            --memory 2g \
            --cpus 2 \
            --read-only \
            --tmpfs /tmp \
            -v "$PWD/diff.patch:/workspace/diff.patch:ro" \
            -v "$PWD:/workspace/repo:ro" \
            -v "$PWD/kiro.config.json:/workspace/kiro.config.json:ro" \
            -v "$PWD/review-output.json:/workspace/review-output.json" \
            -e KIRO_API_KEY="$KIRO_API_KEY" \
            kiro-sandbox:latest
        continue-on-error: true

      - name: Run orchestrator
        run: node orchestrator.mjs

      - name: Post review comment
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const findings = JSON.parse(fs.readFileSync('review-output.json', 'utf8'));
            const summary = findings.findings || [];
            const critical = summary.filter(f => f.severity === 'critical');
            const warning = summary.filter(f => f.severity === 'warning');

            let body = `## 🤖 Agentic Review\n\n`;
            body += `**Findings:** ${summary.length} (${critical.length} critical, ${warning.length} warning)\n\n`;
            if (summary.length === 0) {
              body += `No issues found. ✅\n`;
            } else {
              body += `| Severity | File | Line | Finding |\n|---|---|---|---|\n`;
              for (const f of summary.slice(0, 20)) {
                body += `| ${f.severity} | \`${f.file}\` | ${f.line} | ${f.message} |\n`;
              }
            }

            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: body
            });

Key details:

• --network none — the agent cannot exfiltrate data or call external APIs.
• --read-only — the agent cannot modify the repo (it only writes to the mounted review-output.json).
• --memory 2g --cpus 2 — caps resource usage so a runaway agent can't kill the runner.
• continue-on-error: true — if Kiro crashes, we still run the orchestrator so the PR gets an "inconclusive" comment instead of a silent failure.

7. Step 4 — Build the Review Orchestrator

The orchestrator is the policy brain. It reads Kiro's JSON output, filters findings according to your rules, and emits a status check that GitHub branch protection can enforce.

Create orchestrator.mjs:

import fs from 'node:fs';
import path from 'node:path';

const REVIEW_OUTPUT = 'review-output.json';

// ---- Policy configuration ----
const POLICY = {
  // Severities that block the merge
  blocking: ['critical'],
  // Severities that are reported but do not block
  reportOnly: ['warning', 'info'],
  // Files to ignore (e.g., generated code, lockfiles)
  ignorePaths: [/\.lock$/, /dist\//, /node_modules\//],
  // Max findings to report before we consider the review "flooded"
  maxFindings: 50,
  // If Kiro crashed (no valid output), treat as inconclusive, not failure
  inconclusiveOnError: true,
};

function loadFindings() {
  if (!fs.existsSync(REVIEW_OUTPUT)) {
    return { findings: [], error: 'No review output produced' };
  }
  try {
    const raw = fs.readFileSync(REVIEW_OUTPUT, 'utf8');
    const parsed = JSON.parse(raw);
    return { findings: parsed.findings || [], error: null };
  } catch (e) {
    return { findings: [], error: `Failed to parse: ${e.message}` };
  }
}

function applyPolicy(findings) {
  return findings
    .filter((f) => !POLICY.ignorePaths.some((re) => re.test(f.file || '')))
    .slice(0, POLICY.maxFindings);
}

function classify(findings) {
  const blockers = findings.filter((f) => POLICY.blocking.includes(f.severity));
  const reports = findings.filter((f) => POLICY.reportOnly.includes(f.severity));
  return { blockers, reports };
}

function setStatusCheck(conclusion, summary) {
  // In GitHub Actions, write the status to a file that the workflow reads
  const payload = {
    conclusion,
    summary,
    blockers: summary.blockers,
    reports: summary.reports,
  };
  fs.writeFileSync('status-check.json', JSON.stringify(payload, null, 2));
  console.log(`Status: ${conclusion}`);
  console.log(`Blockers: ${summary.blockers.length}, Reports: ${summary.reports.length}`);
}

// ---- Main ----
const { findings, error } = loadFindings();

if (error && POLICY.inconclusiveOnError) {
  setStatusCheck('neutral', {
    blockers: [],
    reports: [],
    error,
    message: 'Agentic review was inconclusive — falling back to human review.',
  });
  process.exit(0);
}

const filtered = applyPolicy(findings);
const { blockers, reports } = classify(filtered);

if (blockers.length > 0) {
  setStatusCheck('failure', { blockers, reports });
  process.exit(1); // This makes the GitHub Actions job fail → blocks merge
} else {
  setStatusCheck('success', { blockers, reports });
  process.exit(0);
}

How the orchestrator enforces policy

• blocking: ['critical'] — only critical findings fail the build. Warnings are reported but don't block. This prevents "review fatigue" from a chatty agent.
• ignorePaths — skip generated code and lockfiles that always produce noise.
• inconclusiveOnError — if Kiro crashes (timeout, OOM), we mark the check as neutral rather than failure. A broken agent should not block a good PR.

8. Step 5 — Harden the Sandbox

The docker run command in the workflow is already fairly locked down, but for production you'll want to go further. Here's a hardened version:

docker run --rm \
  --name kiro-review \
  --network none \
  --memory 2g \
  --memory-swap 2g \
  --cpus 2 \
  --pids-limit 256 \
  --read-only \
  --tmpfs /tmp:size=64m \
  --tmpfs /run:size=16m \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --security-opt seccomp=default \
  -v "$PWD/diff.patch:/workspace/diff.patch:ro" \
  -v "$PWD:/workspace/repo:ro" \
  -v "$PWD/kiro.config.json:/workspace/kiro.config.json:ro" \
  -v "$PWD/review-output.json:/workspace/review-output.json" \
  -e KIRO_API_KEY="$KIRO_API_KEY" \
  kiro-sandbox:latest

What each flag does

Important: Never mount the CI secrets directory into the sandbox. Pass only the API key via environment variable, and even then, consider using a short-lived, scoped token that Kiro can't reuse after the job ends.

9. Step 6 — Gate the Merge

Now we make the review mandatory. GitHub branch protection rules are the enforcement mechanism.

9.1 Create a status check

The orchestrator writes status-check.json, but GitHub Actions needs a real check. The cleanest way is to use the actions/github-script step to create a check run, or simply let the job's exit code drive a required status check.

Add this to the workflow after the orchestrator step:

- name: Create status check
  if: always()
  run: |
    node -e "
      const fs = require('fs');
      const status = JSON.parse(fs.readFileSync('status-check.json', 'utf8'));
      // The job itself already exits 1 on blockers, so GitHub marks it failed
      console.log('Conclusion:', status.conclusion);
    "

Actually, the simplest robust approach: the orchestrator's process.exit(1) makes the GitHub Actions job fail. Then in branch protection, require the agentic-review check to pass.

9.2 Configure branch protection

In GitHub → Settings → Branches → Add rule:

• Branch name pattern: main
• Require status checks to pass before merging: ✅
  • Select the Agentic Code Review check (or the job name agentic-review).
• Require pull request reviews: ✅ (keep a human review as a second gate — see FAQ)

Now, if the agent finds a critical bug, the check fails, and the merge button is disabled.

10. Production Best Practices

10.1 Scope the review to the diff

Kiro's "scope": "diff" is your friend. Reviewing the entire repo on every PR is slow and expensive. For a 500-line PR, the diff is the only thing that matters.

10.2 Set a review budget

Cap tokens and time:

{
  "review": {
    "maxTokens": 8000,
    "timeoutMs": 120000,
    "maxFiles": 20,
    "maxDiffLines": 1000
  }
}

If the diff exceeds the budget, the agent should review a sample and mark the review as "partial" — better than timing out.

10.3 Cache the sandbox image

Building the Docker image on every PR is wasteful. Use docker/build-push-action with a cache, or push to a registry and pull it in CI:

- name: Pull sandbox image
  run: docker pull ghcr.io/your-org/kiro-sandbox:latest

10.4 Human-in-the-loop for critical paths

Agentic review should augment human review, not replace it. Configure the orchestrator so that if a finding touches a file in CODEOWNERS with a critical path (e.g., auth, payments), it always escalates to a human, even if the agent says it's fine.

10.5 Track false-positive rate

Log every finding to a metrics store. After a month, compute your false-positive rate (findings that a human dismissed). If it's above 30%, tighten the prompt or raise severityThreshold.

11. Edge Cases & Failure Modes

The "sandbox escape" scenario

The worst case is a PR that contains a Dockerfile or a script designed to break out of the container. The --cap-drop ALL, --security-opt no-new-privileges, and --network none flags make this very hard. But you should also ensure the host runner is itself ephemeral (GitHub-hosted runners are) so that even a successful escape only compromises a throwaway machine.

12. Frequently Asked Questions

Does this replace human code review?

No. It replaces the boring 80% — the mechanical "did you handle the empty array" checks. The human should focus on architecture, design intent, and API contracts. Configure branch protection to require both the agentic check and a human approval.

How do I avoid the agent hallucinating findings?

Three strategies:

1. Require a reproduction — the prompt demands a concrete failing test for every critical finding. If the agent can't produce one, the finding is downgraded.
2. Track false positives — log dismissals, and if a file keeps