Green CI is not finished work for developers

The update for Flag Quiz had a green tick on every check. Unit tests passed, lint passed, the build job produced a signed bundle without complaint. I uploaded it to the internal testing track, installed it over the existing version on my own slightly battered phone, and opened it to make sure the new question pack showed up.

It did. My streak and best score had also gone back to zero.

It took about an hour to find. The app saved player stats as a small JSON blob, using Gson, and the release build had code shrinking turned on. R8 had quietly renamed the fields on the stats class, so the new version was looking for short obfuscated names in JSON that had been written with the original ones. Nothing matched, so everything came back as defaults. The tests ran against the debug variant, where nothing was renamed, and the old saved data never existed in a test at all. Every check had told the truth about what it checked. None of them checked what a person updating the app would see.

The fix was two annotations and a keep rule. The lesson took longer to sink in: a green pipeline had told me the code was fine, and I had been very ready to believe that meant the release was fine.

What a green badge actually tells you

Green CI means one snapshot of the code survived the automated checks someone thought to write. That's useful. It's also much less than it feels like when the tick turns green after a long afternoon.

I find it helps to think of the path to done as a series of checkpoints, each of which proves something specific and leaves something else unproven:

The Flag Quiz bug lived in the gap between the second and fourth rows. CI never built the variant I was shipping, and nothing in the pipeline had an old install to upgrade from. Most of the "it was green, how did this break?" moments I've seen live in one of those gaps.

Where green dies quietly

The same handful of places come up again and again, on my own apps and on client projects.

Build variants. Debug and release are different programs. Minification, different signing, different config values, logging stripped out. If CI only builds one of them, it is only vouching for that one.

Real data. Test databases are clean. Production has duplicate rows from a bug four years ago, a customer with an apostrophe in their surname, and dates stored in two formats because of a migration nobody finished. An idempotency check that's perfect in tests can fall over on the first real row it meets.

Background processes. Anything that happens later, on another process, tends to be invisible to tests. Queues, scheduled jobs, webhooks from a third party.

Flags and release trains. "Merged behind a flag" is finished for nobody until the flag flips. A merged change waiting for next week's release is still inventory, however green it is.

The client version of this was a small online shop where I'd moved order confirmation emails onto a queue so checkout would feel faster. The tests ran with the queue set to process jobs immediately, which is normal for tests. Everything was green. I merged it, deployed it and told the client it was done. What I hadn't checked was that the production server, which had recently been moved to a new host, had no queue worker running. Jobs went into the table and sat there. The client found out the next afternoon when two customers emailed asking whether their order had gone through.

Nothing in the pipeline could have caught that. A thirty-second check after deploy would have: place a test order, see the email arrive. I'd skipped it because green felt like permission to stop.

Green branches waiting for review are inventory

There's a quieter version of this problem that doesn't involve anything breaking. On one remote team I was on, it was normal to have four or five green pull requests open per person at any time, waiting for review. Each of them felt like done work. Standups were full of "that's ready, just needs a review".

But a green branch that sits for three days isn't finished. Main moves underneath it. Conflicts appear. The author forgets why they made the odd choice on line 140, and the reviewer has to ask, and the answer takes half a day to come back across time zones. We were producing green faster than we could merge it, and calling the pile throughput.

What helped was boring: a limit on open pull requests per person, and review time planned into the day rather than squeezed between features. Plan a developer day with review and feature work covers how I split that. More green without merge capacity only makes the pile taller.

How I say "done" now

The words matter more than I expected. When I said "CI's green" at standup, people heard "finished", and so, if I'm honest, did I. So I changed the sentence.

Instead of the state of the pipeline, I describe where the work actually is. Compare:

• Weak: "Branch is green, should be good to go."
• Better: "Merged. Staging verified with orders 4412 and 4413. Release ticket 901 is open. Flag tax_service_v1 is still off."

The second one is longer and much more useful. Anyone reading it knows what's true and what isn't, and nobody has to guess whether "good to go" means merged, deployed or released. If the honest answer is "green, not merged", I say that. It's a perfectly normal state for work to be in, and it stops the green branch masquerading as shipped work.

The question I try to answer instead of "do the tests pass?" is "what can someone do today that they couldn't yesterday?" For user-facing work that's easy to check. For internal work, I answer with something observable: "Main no longer calls the deprecated endpoint, and the old deploy step is removed." If the honest answer is "nothing yet", I'm describing motion, which is fine as long as I don't call it finished. Busy versus finishing work has more on that difference.

Clients don't care about the pipeline

For freelance work, green CI means even less to the person paying. A client hears "tests pass" and reasonably assumes the job is done and the invoice is coming. Then they open the site and the thing they asked for isn't there, because it's on a branch, or on staging, or behind a setting.

I now write into the proposal what done means for that job. Usually it's something like: deployed to staging, client has checked it against the agreed list, then deployed to production. What the client actually wants is a link they can click and steps they can follow. "Go to the booking page, choose next Tuesday, you should see the new time slots." That is the deliverable. The test suite is how I make sure the link keeps working, and they never need to see it.

Red and green aren't moods

The other half matters too. A red build means the checks did their job, and it says nothing about the person who pushed. Teams that treat red as embarrassing and green as a win, without asking anything about merge or verification, slowly teach people to game the checks: skip the flaky test, shrink the coverage, avoid writing the awkward integration scenario that keeps failing. That makes the badge greener and the product worse.

Green is permission to move to the next checkpoint. Nothing more.

Green is permission to move to the next checkpoint.

The done footer template I paste into tickets

Done when:
- [ ] CI green on the merge commit
- [ ] Merged to main (or waiting on: NAME / DATE)
- [ ] Release build checked on a real device or real env
- [ ] Staging verified: steps ____ / accounts ____ / flags ____
- [ ] Rollback: ____
- [ ] Support or runbook note: updated / not needed
- [ ] What a user or operator can now see: ____

I fill the blanks before I start coding, which is the important part. Writing "what a user can now see" at the start forces me to decide how I'll check it, and that's usually where the gap shows up. For Flag Quiz, the release build line is now "install release over the current store version on a real phone, check stats survive". For the shop, it's "place a test order on production, see the email arrive within a minute".

Before I stop for the day, I update which boxes are actually true, and that becomes my status. If only the first box is ticked, the work isn't finished, and I say so in those words. Not every ticket needs every line; I delete the ones that don't apply instead of ticking them for show.

For reviewers, the footer gives you a good question to ask: which of these boxes are still open, and who closes them? Approving a pull request with no verify plan is how green work piles up while users keep hitting the old behaviour. If meetings are what's stopping things getting merged, that's a scheduling problem worth fixing on its own, and fewer, better remote meetings is where I'd start.

I still like a green tick. I just stopped treating it as the end of anything. The work is finished when someone outside my editor can rely on it, and the only way I've found to know that is to go and look, on the build I'm shipping, in the place where it runs. The broader version of this idea, applied to a whole working week, is in developer productivity without fake busyness.