Why I tore down my agent platform

On August 27, 2026, my agent platform ran better than ever: three issues in roughly 87 minutes. Implementation, reviews by LLM judges, corrections, and merge, without my involvement. Then the fourth issue began. It ran longer than expected, OpenRouter responded with HTTP 429, and I faced a question the platform could not answer: keep waiting, change the routing, abort the run?

I have been working with AI-assisted programming privately for roughly two years. I spent seven active weeks on the agent platform. The OpenRouter bill for that came to around $340. My own time and the infrastructure were not included.

The bill did not end with tokens and VPS rent. Added to that was the software that sat idle in the meantime.

Seven weeks were enough to build plenty of automation, but not to build a platform whose self-operation paid off for my purpose.

The moment the telemetry helped: too late

I had restricted the routing to the three providers with the lowest measured latency. That had accelerated the previous runs. Over the restricted routing, 429 responses arrived repeatedly (routing settings, HTTP 429 as a rate limit). With the restriction, I had also reduced the fallback options.

The 429 responses were visible. What they meant for the active worker was hard for me to assess.

The platform did have several telemetry paths: a state.json at node boundaries, running Docker logs, a worker’s ReAct trajectory as a JSONL file, but only after it had finished. OpenTelemetry spans, but only at the end of the run. Metrics, but only after a successful cycle. ADR-0016 names this limitation explicitly as “Deferred Visibility”: the rebuilt export delivered no live view during the run. The separate metrics path (ADR-0029) did not close this gap.

The hidden precondition of this observability was a sufficiently orderly completion. A failure was precisely the case where that could not be relied upon. To understand the state transitions and checkpoints, I had an LLM analyze the code. The analysis was good. It only came too late for the operational question. Observability whose workings I first have to reconstruct through code analysis is hardly observability for operations.

And it was not only the visibility that was missing. What was missing was the connection between the existing signals and an action:

Signal → recognizable event → severity → notification → response

No notification reported a worker that had been active for an unusually long time. No rule distinguished a short series of 429s from a persistent provider problem. Even the end of the entire run through the time limit triggered no alert. The costs of completed reviews were traceable. Judge billing and KPI reporting per review are documented (ADR-0058). The progress of the active worker was not.

Only later was it fully clear what had ended the run. In the meantime, I had withdrawn the provider restriction and left the routing to OpenRouter. Then the Execute node’s maximum time budget took effect. I had previously raised it from 30 to 45 minutes. Even that was not enough. Before the issue was finished, the time limit had been exceeded and the Execute node was restarted. Whether the worker was making progress or had become entangled in a loop was not transparently visible.

What else the platform had to carry

The observability gap was no exception. It was one line on a longer list. The concrete solutions depend on the framework. LangChain, LangGraph, custom code. The operating questions do not disappear because of that. I had already built answers for many of them. The fourth issue showed which ones were missing.

flowchart TB
    subgraph plan["Planning, part of the platform"]
        human["Author: interviews and grilling sessions with the planner"]
        planner["agentic-planner-core: creates several dozen issues"]
        human <--> planner
    end
    subgraph start["Start, manual per run"]
        lauf["Author starts a run with several issues"]
    end
    subgraph own["Self-operation on its own VPS, autonomous up to the merge"]
        orch["agentic-developer-core: orchestrator, resume, retries, time budget"]
        worker["Worker on the same VPS"]
        ext["External services: OpenRouter with provider restriction and GitHub"]
        gates["Verification: deterministic quality gates and four LLM judges"]
        merge["Merge by the platform"]
        orch <--> worker
        orch <--> ext
        worker -->|"Pull Request"| gates
        gates -->|"Findings, correction loops in progress"| worker
        gates -->|"green"| merge
    end
    planner --> lauf
    lauf --> orch

Resume. After a restart, the platform picked up at the last saved node instead of starting the issue from scratch. That was useful and, unfortunately, coarser than I had initially assumed. A worker’s internal plan did not consist of individually saved execution steps. If it planned ten steps and was interrupted at the fourth, the Execute node started again. The working directory still contained the changes from the first steps, and the model could inspect them and build on them. Whether the new attempt picked up sensibly depended on how the model interpreted the existing changes. The platform resumed the state of the repository, not the agent’s line of thought.

Error handling. Layered retries with model fallback, loop detection (3 identical tool calls: redirect, 5: cancel), and a recursion limit as an abort safeguard offered sensible controls. They solved the wrong problems, however: a worker can block for a long time without repeating calls, and accumulated provider errors do not necessarily trip the recursion limit.

Security and isolation. An environment allowlist instead of a denylist, SSRF validation for URL fetches, process supervision, and a dedicated execution sandbox on a separate VPS. A server I had to operate, patch, and pay for.

Verification. Deterministic quality gates, four LLM judges over the PR diff, a feedback loop until the merge is green. The details stand with the rest of the list in the appendix.

The list has three entries without an ADR number. They exist only as plans:

• Finer resume: progress pointers and in-worker checkpoints, instead of starting over from the last saved node.
• Live visibility during the run: streaming tracing instead of telemetry only after completion.
• Alerting: rules that couple a signal to a response.

Each of these solutions would have justified the next sensible platform feature. All three remained planned and were never built.

Less platform, not a cheaper platform

All three points together would have meant permanent platform work: defining events, calibrating thresholds, operating notifications, testing aborts, hardening reruns. Meanwhile, the software I actually wanted to build with the platform was waiting.

The incident showed me what the next expansion would cost: more platform work. No longer the most sensible use of my time.

I describe the decision I made afterwards most precisely like this: I did not rebuild the same platform more cheaply. Before: implementation, reviews, corrections, and merge without my involvement. Today: start a session, set the agent-ready label, and merge myself. I reduced the autonomous scope and took the merge decision back into my own hands. The difference lies not in the cost line but in the kind of responsibility.

What remains is also not an operations-free state. A CLI harness orchestrates too. GitHub Actions and the model provider still handle operations. I run only one issue per session and check after roughly 60 minutes whether it is finished or needs steering. In about two out of ten issues I have to abort the session and restart it from the last work step. Not perfect, but the effort stays small. The honest comparison is not “operations versus no operations” but “less self-operation versus more self-operation”.

What runs today

flowchart TB
    subgraph plan["Planning, once per topic"]
        human["Author: interviews and grilling sessions with the planner"]
        planner["agentic-planner-core: creates several dozen issues"]
        human <--> planner
    end
    subgraph pick["Selection, manual per issue"]
        release["Author sets the agent-ready label"]
    end
    subgraph run["Execution and verification, autonomous up to the green pull request"]
        dcode["dcode CLI harness on a disposable VPS, one session per issue, policy layer"]
        ext["External services: OpenRouter with GLM 5.3 Flash, Tavily, and GitHub"]
        gates["quality-gates-toolkit: deterministic quality gates and LLM judges"]
        merge["Author: merge"]
        dcode <--> ext
        dcode -->|"Pull Request"| gates
        gates -->|"Findings, on average two correction loops"| dcode
        gates -->|"green"| merge
    end
    planner --> release
    release --> dcode

A CLI harness (dcode) uses GLM 5.3 Flash via OpenRouter, started per session with --yolo, one-time confirmation, after which the agent executes approved actions without further prompts. The policy layer restricts the intended network and filesystem tools: research via Tavily, data retrieval via read-only fetches. The filesystem scope of a run is the project checkout. It does not provide hard isolation of shell commands and subprocesses. By operating on a dedicated VPS as disposable infrastructure, I have not only reduced the self-operation, I also deliberately forgo a separate execution sandbox. The issues are prepared by agentic-planner-core. The verification is handled by my quality-gates-toolkit, as an individually deployable verification point, without the self-operation of the other components.

flowchart LR
    pr["Pull Request"] --> gates["Deterministic quality gates: Ruff, mypy, pytest, Coverage, Semgrep, pip-audit"]
    gates -->|"all green"| judges["Four LLM judges over the PR diff"]
    judges -->|"PASS"| merge["Merge approved"]
    judges -->|"FAIL"| fix["Correction and new push"]
    fix --> pr

The prompt for a complete run is one sentence: “Work the next agent-ready issue.” I have published the full skill definition, SKILL.md, both phase references, and the selection script as a Gist.

A complete run of this chain is publicly documented: in agentic-planner-core, issue #81 went through PR #82 to the merge. Four judge reviews, all PASS, token and cost figures per judge visible in the review.

What operations cost today is an order of magnitude, not proof of savings: over the three weeks up to September 13, the OpenRouter bill came to around $30 (exclusively for use of the new setup). A run from issue to merge-ready PR typically costs between $0.25 and $0.75. The reference quantity is the merge-ready PR: everything an issue consumes on the way there, such as judges, on average two correction loops, and failed runs, I attribute to the PR.

The test

Before the next longer unattended run, I would test two failure cases: a stuck worker and an aborted run. The test must answer three questions without retrospective code analysis:

• What did the worker last do?
• How do I find out about the standstill?
• What is executed again on restart?

As long as these answers are missing, I will not increase the autonomous runtime. The next sensible proof is not a fourth successful issue but an interruption that I detect in time and can fix in a controlled way.

Appendix: The full operations list

The operating problems described in the main text, my decisions on them, and the public sources at a glance:

Referenced repositories

• quality-gates-toolkit
  • public, MIT license, v1.7.0.
• agentic-planner-core
  • public, MIT license
• agentic-developer-core
  • public, MIT license
• issue-to-pr-skill