Learnings in agentic coding

8 months of side projects, in roughly the order they happened.

Euchre v1

When code agents began getting popular and we hadn’t gotten access at work yet, I started looking for any excuse to build a side project. I ended up starting with building a Euchre game because I wanted to teach Euchre to my friends and hadn’t played it in a while myself. I wanted to be able to play against LLMs, figuring it would also be an interesting way to learn about specific model behaviors. I spent a solid amount of time planning with Claude in a normal chat, planning a tech stack that was similar to what I was already familiar with at work. With our spec in hand, Claude blew my mind and one shot the full backend Euchre game engine. Then I started running headstrong into:

Providers had differing standards, I had to ensure my agent followed them correctly. It wasn’t intuitive to the agent that we’d face different standards which showed up in varying ways.
Complex UI and UX patterns where getting Claude to understand what I actually wanted was very tough. I was also figuring this part out as I went, as we had gotten past the initial spec. I found that it felt like Claude would “reward max”, where it would take a shortcut/do something I definitely didn’t want in order to finish the task.
The UI ended up being very clearly AI generated.

Learnings

Providing a theme for Claude to follow was very helpful
Using shorter chats helped me manage context and prevent context rot. It was very frustrating to have Claude go against an agreement we made 2 minutes before
- This forced me to work on handing off context between sessions, often just repeating things myself or starting from scratch in the next step
I should have stopped to re-spec and plan once I got past my initial spec

Vue Grab

I came across react-grab and thought it was really cool. I was doing frontend work with Vue.js at my job and found that the equivalent features in Cursor (the approved code agent at work) weren’t there yet. I also thought that it’d be useful to have Vue context. Instead of Cursor trying to investigate starting at a referenced HTML element, we could attach direct Vue component file references, route, store state and component state. This was interesting to me because I hadn’t written Chrome extensions or dug into Vue internals.

Learnings

Grilling my code agent until I understand its decisions and work helped me think about the big picture and what’s possible
Specs and huge upfront plans were less important
- Code is cheap to regenerate, so going back and rewriting things was a more important process to implement
  - I want to try Jujutsu as the VCS for a project in the future
Asking different models for thoughts helped get different perspectives and answers
Tailoring specific context to copy to the code agent was more important than I thought
- Even if it wasn’t a lot of tokens, adding information that wasn’t as necessary could lead agents to think about the wrong things
  - As a human, I would expect to just filter out irrelevant information, but agents interpreted information as important because I gave it to the agent
Having a second project at the same time was important so that I could dogfood with this devtooling project

Euchre V2

Vercel announced their AI Gateway Game Hackathon. I wanted to work on the model eval category of running different models against each other. I also had recently read OpenAI’s blog post about self-evolving agents. I thought revisiting Euchre could be interesting because rather than a simple “which model finishes the task fastest or scores the highest on the exam”, we would be asking:

How do models stack up against each other when they’re directly competing?
How do models compare with limited information?
How do models compare in their ability to self-evolve?

I took the original Euchre game engine, but restarted the rest from scratch. I used Stitch from Google for the design work this time, but the core problem stayed the same. I still struggled to get the agents to implement my intent on both design and code. Especially iterating on an image was tough. I’d ask Stitch to change one part of the design, and then another area of the design would get affected. Later on, I started playing with image generation in ComfyUI and InvokeAI, the concept of bounding boxes and masks seems like it would have helped a lot here.

Learnings:

Anthropic’s /frontend-design improved the design
Providing examples of designs I liked helped a lot, but I had to be careful to ensure the agent wasn’t directly copying and instead was taking what I wanted from the examples
Agent-browser MCP was very helpful for me to set defined goals and have Claude iterate on its own
Calibration/brier scoring
- Models (especially cheaper ones) were not good at rating their confidence and were not good at making decisions without help
  - Models had a poorer understanding of how to play Euchre than I expected, given how fast Claude was at building up the game engine
LLM-as-a-judge helped provide adversarial and equal reviews to the different agents
Self-evolution required a curated set of data about the agent’s previous decisions and outcomes
- I considered handing the agent a SQLite file or JSON it could query, but was weighing how often I wanted the agent to iterate on its prompt vs how many tokens to spend in each iteration
  - I ended up using a chain-of-draft approach with curated data containing their original decision, their reasoning, the outcome and an llm-judge’s feedback at the end of each hand.

Trap Planner

Then I started Trap-Planner. My friend was using Google Sheets to plan where to place players in a game, but the map was rotated and diamond-shaped, so sheets were harder to interpret. I wanted to try Replit out again now that it had matured a bit, so created a first version of a layout canvas planner with Replit. I very quickly ran out of credit on the first paid tier, so brought the project over to my Claude Code agent. I started using PRs more consistently in this repo, I was kind of back and forth on my previous projects, but found that just having PR descriptions or longer commit messages I could point the agent to helped a lot. Plus, it allowed me to open up Codex and ask for reviews with more context. In terms of prompting, rather than pasting in errors I started to just make the situation more accessible to Claude. For example, opening up the exact page and telling it to use agent-browser. I generated demo data by anonymizing and editing data from a few days of gameplay. This also really enabled me to run multiple agents at once locally and use cloud agents for a full stack service, because they could just spin up their docker containers.

CS2 Demo Review

My friends needed a substitute for their counter-strike team, and I hadn’t played the game much in over 5 years. I needed to catch up on the current strategy and lineups to throw utility and support the team. There are services like Leetify and Refrag, but they were mostly focused on long-term self analysis. Those were great to use while playing and trying to improve, but I needed a way to understand team-oriented play rather than individual level analysis. Some of their services were also paywalled, and I figured instead of spending $5 to unlock their feature, I could spend $50 in tokens to make the features myself. I chose to use Nuxt for this project so that I could also test out vue-grab here. At work I support an AI chat app, so vibe-coding an AI chat app let me play with concepts faster than I could at work. I saw an interesting X blog post from a finance startup (sorry I’m blanking on the name!) where they created separate sandboxed databases for each client. Then, their agent could go wild with SQL queries and analyze their data without risking leaking other clients’ data. Their theory was that LLMs are good at writing SQL, so why force them to use some custom API that won’t give them exactly what they need? I designed this project around those ideas. All data was stored in the browser in a SQLite db, the agent could query it as much as it wanted and the only data the user was sending to the internet was what was sent as part of LLM requests. I figured if a professional level team were using a project like this, they would be protective of their game recordings. Improving the agent turned into iterating on the pre-processing of the game recording’s data, map specific data, and tools that helped the agent understand the game better. Learnings:

Subject matter expertise was extremely important. Before adding much subject matter knowledge, the agent was next to useless. At the end, it gave us genuine insights about how our opponents played certain maps with ~2 mins of conversation. The insights actually helped us in game.
Models responded to prompting and context-rot differently. It was better to target a few models and curate towards them instead of trying to support every model.
I found that using cheaper but decent and fast models gave me better results and iteration than using big expensive models or super small super cheap models.
- Example: Even if Kimi K2.5 was only 75% as good as Opus, I could run it 3-4 times for the same price, so I could ask follow up questions more quickly and have a longer, deeper discussion for the same price.
  - Improved prompting/handholding also helped close the gap a bit.
Sub-agents would have been very helpful here, many analysis tasks were multi-part.
- Example: “How did Vitality take B site on their T side?”
  - The agent had to first identify which rounds Vitality would be on T side, then see where they took B site (which isn’t a clear cut yes/no analysis), then understand how far back in the game state to look and analyze many moments/events. The agent would also later need to compare these rounds against each other to look for trends, enemy adaptation and progression.
A key prompt for improving my agent was to ask it why it struggled to answer the way I wanted to at the end of chats, and then handing that feedback to Claude. I gave Claude the conversation context too, so essentially traces + “human eval”.

Browser automation started becoming a big trend. I saw a lot of automated QA projects (and worked on one at my company’s hackathon!), but didn’t see much in the way of accessibility. Automated tools only catch 30-80% of a11y issues. I figured that an agent could help close the gap because it can reason about things non-agentic automated tools cannot. I also wanted to automate screen reader testing, because automated tools often approximate what screen readers would say. We were adding more end-to-end tests at work, but a11y tests were very surface level.

This project started as figuring out how to create a VoiceOver CLI tool for agents like the agent-browser tool I had been using for browser automation. Claude found and helped wrap Guidepup. Writing up skills helped guide the agent to use the wrapper and walk through pages. Then, I figured with VoiceOver and agent-browser I had enough for agents to actually conduct most of an accessibility audit. After designing the tool library I wanted for the agent, I had Claude pull every WCAG criterion and we decided which tools to recommend for each one. I found public a11y tests that were strong evals: W3C ACT Rules which I used to test if the agent could detect/make a determination about specific rules and Mars commuter demo which was a page intentionally made inaccessible with answer key.

Once I started wanting to parallelize, I realized I needed Linux support. After a day or so I was able to work with Claude to copy our VoiceOver driver setup into a Linux compatible version:

Learning about Orca, screen reader for Linux
Xvfb for a virtual display
sprites.dev for sandbox environments

Learnings:

Had to seriously think about orchestration
Sub-agents and model choice were key for parallelization
- I used opus or sonnet for orchestration and sonnet or haiku for my sub-agents
- I used the Claude Agents SDK
Sandboxing was key
- Having checkpoints allowed me to easily reset
- Claude could drive 3-4 different browsers/screen readers at the same time to evaluate different pages or criteria
Adversarial code review gave me huge code quality improvements
- I had a higher tier Claude subscription and a low tier Codex subscription, so Claude would post PRs and watch for Codex to review them
  - If they got into too long of a review loop, it was a sign to back up
  - Prompting Claude to take Codex’s feedback and look for larger underlying issues rather than blindly resolving feedback helped to have better conversations about the overall direction of the project

The next steps I had envisioned to improve the agent skill were:

a web interface, which I used Claude Design to dream up
using Langfuse to develop evals and a golden dataset
adapting for easy use in Codespace environments
- I think this will be best as a new project where I take the learnings and use the original repository as a reference

technical deep-dive

Personal Site

I trimmed down my skills list to a few I really like
- /anti-ai-writing
  - Planned fork/addition: advising agents to avoid meta-commenting within documents when iterating
- /grill-me
- /codex:adversarial-review
- /deslop
  - I had Claude wire this up to a hook for commits
I pointed a11y-auditor at the page and fixed what it found.
I showed Claude examples of the style I wanted and had it generate a few variations to choose from/give feedback on.