OpenStation Blog: I Deleted 168,000 Lines of Comments. My AI Agent Didn’t Notice

Almost a third of our source code was comments. I deleted all of them and asked an AI agent to build the same features on both versions of the codebase, three times each. The code it wrote was just as good with or without comments.

If you use agents on a real codebase and wonder whether comments help them, slow them down, or don’t matter, here’s what I found.

Why I tried this

In this video (around 23:07), Lauren mentions that they don’t let their agents write comments in the code. Our codebase, is the opposite: it’s full of comments, and most of them were written by agents.

So I wondered: are all those comments helping the agent, or are they just extra text it has to read?

The experiment in one paragraph

I made a copy of the codebase with every comment removed, about 168,000 lines in total. Then I gave the same task to the same agent on both copies, three times on each, and checked the results the same way. I did this for two tasks, a small one and a bigger one.

The results

Every run passed every check: lint, type checks and the existing tests, plus, on the bigger task, a hidden test the agent never saw. A separate AI judge scored each result without knowing which copy it came from (more on that below).

Small task: tile all open windows (score out of 19)

Bigger task: keep a window on top (score out of 23)

The numbers are averages of three runs. The score differences are within noise, and so is the cost on the bigger task. On the small task, the agent used about 22% fewer tokens without comments, which makes sense: there was simply less to read.

The bigger task had a trap. The pinned state has to survive a page reload, and the PHP function that saves the window session drops any key it doesn’t know. Miss it and the feature looks fine until you reload. That’s exactly the kind of thing you’d expect a comment to help with. All six runs found it, with and without comments.

Why one run is not enough

My first plan was to run each branch once. I’m glad I didn’t.

On the bigger task, the three runs with comments used 8.21M, 8.79M and 10.17M tokens. The three runs without comments used 7.01M, 9.70M and 10.64M.

If I had picked one run from each branch, I could have written two opposite blog posts. Comparing 10.64M with 8.21M, removing comments makes the agent 30% more expensive. Comparing 7.01M with 10.17M, it makes the agent 31% cheaper. Same task, same code, same day.

Agents are not deterministic. If you’re measuring one, run it more than once.

How I ran it

• The agent: Claude Code with Opus 5.5 at high effort. Same model and settings for every run.
• The two codebases: our main branch, and a copy with the comments removed. I compiled both and checked that the resulting code is identical. The only comments left are the ones the tools need to work, such as lint directives and plugin headers, about 800 lines.
• The tasks: two real features, with the same prompt on both branches.
• The checks: the project’s own lint, type checks and tests. On the bigger task, also a hidden test I added after the agent finished, so it couldn’t write code just to pass it.
• The judge: a separate Claude session read each change and scored it against a checklist written before the runs: does it work, does it follow the project’s conventions, does it have tests, and so on. The changes were labelled with random names, so the judge didn’t know which branch each one came from.
• The runs: three per branch per task, alternating branches so that time of day affected both equally.

My conclusion

For the agent, code comments don’t make a difference to the quality of the code it writes. They don’t help, and they don’t hurt.

So keep a comment or delete it based on what it does for the people reading the code, not for the agent.

This doesn’t prove comments have zero effect. I tested one model, two tasks and three runs per branch, which is enough to rule out a big effect, not a small one.

Next: comments made no difference, but removing our docs/ folder and AGENTS.md did. That’s the next post.

Appendix: data and links

• The codebase with comments
• The codebase without comments
• Exactly what was removed (GitHub may only show part of it)
• Two of the small-task runs as pull requests: with comments, PR 974 and without comments, PR 975
• Full results, all runs: results page
• If you want to see the experiment script and the rubrics, they are all here
• Lines of code, measured with scc: