How I Built an Autonomous AI Coding Agent from Scratch

I didn't build this because I wanted another AI wrapper; I built it because once you move beyond toy repos and into real local projects with migrations, background jobs, generated types, shell scripts, env drift, and multiple apps living in one workspace, most AI coding tools stop behaving like engineers and start behaving like autocomplete with ambition. Cursor and Lovable are both useful, but in my experience they still struggle when the job requires persistent awareness of the local filesystem, disciplined command execution, and a reliable mental model of how one edit ripples across an entire codebase — which is exactly where complex production work begins.

Motivation

The core problem was never "can the model write code," because modern models can absolutely write code; the real problem was "can the system reason about my repo the way I do" — meaning it understands that touching a schema can break a serializer, that renaming a shared util can fan out into twenty imports, and that a seemingly harmless shell command can mutate generated output, invalidate caches, or wedge a dev server until you manually unwind the mess.

What I wanted was an agent that behaved less like a chat tab and more like a junior engineer with good instincts: inspect the repo before speaking, form a plan before editing, ask for permission before doing anything destructive, and keep enough state to know what it changed, why it changed it, and how to roll the change back if the next step failed.

That requirement immediately pushed me away from prompt-only architectures and toward a system with three hard guarantees: a live map of the codebase, a controlled execution loop for terminal work, and a memory layer that preserved task state across long sessions instead of forcing the model to rebuild context every few turns.

Architecture

The final shape was a Next.js front end for the operator UI, a Hono.js backend for the agent control plane, and OpenRouter as the model gateway — because Next.js is designed for full-stack React applications, Hono is a lightweight multi-runtime framework built on Web Standards, and OpenRouter exposes hundreds of models through a single API while supporting routing and fallback behavior behind that unified interface.

I used Drizzle ORM on top of Neon PostgreSQL for durable state, storing conversations, task runs, command receipts, patch metadata, retry counts, and rollback checkpoints. Drizzle's schema-first TypeScript approach made the data model readable inside the same codebase, and Neon pairs cleanly with serverless Postgres in TypeScript projects that also use Hono-style backends.

The interesting piece was the file-awareness layer: on startup the agent crawled the repo, normalized ignore rules, parsed supported files into AST-backed symbol maps, and built a graph of exports, imports, routes, migrations, config surfaces, and test files — so instead of shoving raw file blobs into context it could answer higher-order questions like "where is this type instantiated," "which API route consumes this schema," and "what is the smallest safe edit set for this request" before generating a single patch.

File Awareness

My rule was simple: the model never edits blind. Every change passed through a two-phase workflow where the planner proposed an intent, the repo map resolved the likely blast radius, and only then did the editor generate a patch targeted at exact symbols rather than vague text ranges.

That AST mapping paid for itself immediately — it let the agent distinguish between the string User in a comment and the User type exported from a shared module, prioritize files by dependency distance, and avoid the classic LLM mistake of fixing the visible surface while missing the schema, test, and config files that actually make the feature work end to end.

Terminal execution sat beside that graph as a first-class subsystem, not an afterthought: the agent could run bounded commands like lint, test, type-check, migration generation, and dev-server probes, but every command was tagged with a task ID, cwd, timeout, allowlist, captured stdout/stderr, and a post-run parser — so shell access became part of the reasoning loop instead of a chaotic side channel.

Multi-Modal Modes

Chat mode was the default operator surface. I streamed model output into the UI incrementally so the system felt conversational without giving up structure, using the model layer for planning and patch synthesis while the frontend handled progressive rendering and the backend emitted append-only events for status, tokens, command output, and edit proposals.

Voice mode used Deepgram on both sides of the loop, with speech-to-text handling live transcription and Aura text-to-speech speaking the agent response back in near real time. Deepgram's WebSocket-based streaming TTS forced me to chunk long replies and flush partial thoughts instead of waiting for the entire answer to finish.

Google Meet bot mode was the weirdest and most fun part: I used Puppeteer to automate a headful Chrome session that could join a meeting, monitor captions, and relay context back to the agent — though this mode was inherently more brittle because Meet automation depends on webpage structure, permissions, and UI elements that can change underneath you without warning.

Engineering Challenges

The most annoying frontend bug was stale closures in React, especially around long-lived callbacks for streaming tokens, microphone state, command events, and modal confirmations. Handlers would happily capture yesterday's state and then make today's UI look haunted. The fix was to stop trusting closure-captured values inside durable async flows, move volatile state behind refs or event-style handlers, and separate render state from execution state.

Streaming was the second major fight. Token transport, render updates, and side effects must be decoupled or you will create jank, duplicate messages, and race conditions. I switched to an append-only event log, batched token commits on a short interval, and let the UI render partial output through well-defined loading boundaries rather than treating every incoming chunk like a full state replacement.

The biggest backend safeguard was blast-radius analysis before edits: for every proposed patch I scored the likely impact by counting touched symbols, downstream import edges, adjacent config files, related tests, and whether the change crossed risk boundaries like db/, auth/, or deployment config — then forced a second confirmation for high-risk edits.

Lessons Learned

What worked was boring architecture: a small planner, a strict executor, typed state in Drizzle, durable logs in Postgres, explicit repo indexing, and model routing through OpenRouter instead of betting everything on one frontier model. The real performance win came from using the right model for the step, not from pretending a single model should plan, diff, summarize, and recover from shell failures equally well.

What didn't work was giving the agent too much freedom too early — raw "edit whatever you need" prompts, unrestricted shell access, giant file dumps, and optimistic UI updates that assumed the model's next chunk would always complete cleanly. Every one of those shortcuts looked fast in demos and turned into debugging debt once the system had to survive interrupted streams, failed patches, or a half-finished migration.

The practical targets that made the whole product feel trustworthy were simple and measurable: keep time-to-first-token under about two seconds, keep per-file index refresh below a quarter second after save, keep first-pass patch application above 95 percent, and never allow writes outside the repo root without an explicit override. Once an autonomous coding agent is fast, reversible, and blast-radius aware, developers stop treating it like a novelty and start trusting it with real work.