←All Articles

Engineering Tactics · Part 3 of 7

AI Wrote Invalid JSON and Thought It Fixed a Bug It Didn't — How I Guard Against It

Feb 23, 2026

The AI said “Done. I’ve removed the unused permission from manifest.json.”

The JSON was valid. The build passed. The extension loaded. Everything looked fine.

The agent hadn’t removed one permission. It had removed the entire permissions key.

The Setup

This happened during a fix for my Chrome Web Store rejection. The reviewer had flagged two issues: a remotely hosted CDN URL (which I covered in the previous article) and an unused downloads permission that my extension declared but never used.

The downloads fix seemed trivial. I told the AI agent:

"Remove the 'downloads' permission from manifest.json. 
Don't change anything else."

One line to delete from a JSON array.

What Actually Happened

The agent used a text replacement operation on manifest.json. It targeted the permissions array to remove "downloads". But the replacement went wrong. Instead of removing one element from the array, it removed the entire permissions key and its value.

Before:

{
  "name": "ChatShuttle",
  "version": "1.0.0",
  "permissions": [
    "identity",
    "storage",
    "sidePanel",
    "downloads"
  ],
  "content_scripts": [...]
}

After the agent’s “fix”:

{
  "name": "ChatShuttle",
  "version": "1.0.0",
  "content_scripts": [...]
}

The entire permissions block was gone. Not just downloads, but identity, storage, sidePanel. Everything the extension needed to function.

The insidious part: the JSON was still valid. No syntax error. No build failure. npm run build passed. The extension loaded in Chrome without errors. It just couldn’t do anything. No Google sign-in (needs identity). No data persistence (needs storage). No side panel (needs sidePanel).

Why This Kind of Bug Is Dangerous

Code logic bugs crash loudly. Configuration bugs fail silently.

When an AI agent writes a function with a bug, you usually catch it: the test fails, the feature doesn’t work, you see an error. But when the agent damages a config file, especially one like manifest.json that defines capabilities rather than behavior, the symptoms might not appear until much later.

If I had submitted this to Chrome Web Store review, it might have passed. No remote code, no unused permissions, because there were no permissions at all. Then I would have published an extension thatcouldn’t sign users in. The debug cycle from that point would have been ugly.

What Caught It

One command:

$ git diff --stat
 manifest.json | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

I asked the agent to remove one line. The diff showed 10 deletions. That mismatch is the red flag.

Then the full diff confirmed it:

$ git diff manifest.json
-  "permissions": [
-    "identity",
-    "storage",
-    "sidePanel",
-    "downloads"
-  ],

The fix took 30 seconds: git checkout -- manifest.json, then I manually deleted the one line with "downloads". Done correctly in under a minute.

The Chain Reaction I Almost Triggered

Before I learned this lesson, my instinct was to ask the agent to fix its own mistake. That would have gone like this:

  1. Agent removes permissions key (mistake #1)
  2. I notice something is wrong, ask agent to add permissions back
  3. Agent adds permissions, but maybe the wrong list, maybe in the wrong format (mistake #2)
  4. I notice the format is wrong, ask agent to fix the format
  5. Agent restructures the JSON and breaks content_scripts in the process (mistake #3)

Three layers of fixes on top of a broken foundation. I’ve watched this exact chain reaction play out before. The correct move is always:rollback, start fresh.

The Three-Gate System

After this incident, I built a verification system for every AI-generated config change. Three gates, applied in order. The change must pass all three before I commit.

Gate 1: Structural Validation

Does the file still parse? Are all required keys present?

# For JSON files: validate structure
$ node -e "const m = require('./manifest.json'); \
  console.log('permissions:', m.permissions); \
  console.log('content_scripts:', m.content_scripts?.length);"

# Expected output should show all required fields
# If any field prints 'undefined', something was deleted

Gate 2: Content Verification

Does the diff match my intent? Did only the expected lines change?

# Check the scope of changes
$ git diff --stat
# Should show: 1 file changed, 0 insertions(+), 1 deletion(-)
# If the numbers don't match your intent → rollback

# Read the actual diff
$ git diff manifest.json
# Every changed line should be intentional

Gate 3: Functional Smoke Test

Does the thing still work end-to-end?

# Build
$ npm run build

# Load unpacked in chrome://extensions
# Click through: sign-in → import → restore → side panel
# Each feature exercises a different permission

Gate 1 catches structural damage (missing keys, syntax errors). Gate 2 catches scope creep (changed more than intended). Gate 3 catches functional regressions (the feature doesn’t work anymore).

In my case, Gate 2 alone would have caught the issue: 10 deletions when I expected 1.

The “Target Content Not Found” Problem

There’s a subtler version of this bug that I’ve seen multiple times. The AI agent tries to make a text replacement, but the target string doesn’t match exactly. Maybe whitespace is different, maybe a comma is missing. The replacement operation fails silently — the agent reports “target content not found” but then tells you it succeeded anyway.

This creates a different kind of false confidence. You ask the agent to fix something. It says done. But the file is unchanged. You test the old behavior, confirm it still works (because nothing changed), and assume everything is fine.

Gate 2 catches this too: git diff --stat shows zero files changed when you expected one.

Your Config Change Checklist

  1. Always diff first. git diff --stat before anything else.
  2. Count the lines. Deletions and insertions must match your intent.
  3. Validate structure. Parse the file and check required keys exist.
  4. Never chain AI fixes. If the first attempt went wrong, rollback completely.
  5. Treat config as fragile. Config changes need more scrutiny than code changes.
  6. Read “Done” with suspicion. The agent will report success even when the operation failed.

The Bigger Pattern

This incident changed how I think about AI pair programming. The agent is great at writing new code: functions, components, algorithms. It’s unreliable at editing existing structured files where the change needs to be surgically precise.

Config files, manifests, lock files, CI configs. These are the files where the difference between “remove one element” and “delete the entire block” is a one-character boundary. The agent doesn’t always respect that boundary.

The three-gate system (structural validation, content verification, functional smoke test) catches these failures before they compound. Takes about 60 seconds to run all three. A lot cheaper than debugging a broken submission.

For the full workflow that wraps around these gates, including when to scope changes and how to decide between commit and rollback, see the Checkpoint Loop in the first article of this series.

To see how ChatShuttle handles the complexity that generated all these debugging stories — the documentation walks through the architecture.