A team I advised last year had 4,000 automated tests and a release pipeline that took 90 minutes to go red. The tests were green almost all the time — which sounds like a win, until you notice that the ones catching real bugs were the same 200 they'd written in the first six months. The other 3,800 were maintenance tax. Every UI tweak broke a hundred of them, engineers "fixed" failures by loosening assertions, and confidence in the suite quietly went to zero.
That is the real failure mode of test automation. Not "too little coverage" — the wrong coverage, automated at the wrong layer, defended as if quantity were the goal.
Start with the cost of being wrong, not the ease of writing a test
The first question I ask isn't "can we automate this?" It's "what does it cost us when this breaks in production, and how often does the underlying logic change?" Those two axes decide almost everything.
A payment calculation that changes twice a year and loses money silently when wrong is the ideal automation target: high blast radius, stable logic. A marketing landing page that the growth team reshapes every sprint is close to the worst — low cost of failure, constant churn. Automating the second one buys you a test that fails for the right reason maybe 5% of the time and for cosmetic reasons the rest.
Automate what is expensive to get wrong and slow to verify by hand. Leave what is cheap to break and cheaper to eyeball.
What earns automation
- Business-critical logic. Pricing, tax, permissions, money movement, anything with a compliance or safety edge. These deserve dense unit and integration coverage.
- Regression-prone paths. The bug that came back twice already. Write the test the moment you fix it the second time — you now have evidence it's fragile.
- Boring, repetitive verification. API contracts, data transformations, permutations of inputs a human would never patiently check. Machines are genuinely better here.
- The critical user journeys — a handful of them. Signup, checkout, login, the one flow that generates revenue. Five to ten end-to-end tests that actually exercise the product, not five hundred.
What to skip, or do differently
Skipping is a strategy, not an admission of laziness. Some things cost more to automate than they'll ever save.
- Pixel-perfect UI and layout. A human sees a broken layout in two seconds; a snapshot test flags a legitimate redesign as a failure every time. Use visual-diff tooling deliberately, on a few key screens — not as a reflex on every component.
- Throwaway or exploratory features. If you're not sure the feature will survive the quarter, heavy automation is a bet on code you may delete. Test the risky core by hand, ship, then automate what proves durable.
- Rarely-run, low-stakes admin paths. The internal tool three people use once a month rarely justifies an E2E suite. A short manual checklist is honest and cheaper.
- Everything through the UI. The most common mistake I see: teams push all their testing to the slowest, flakiest layer. If logic can be verified with an API or unit test, it should be. Reserve the browser for what genuinely requires a browser.
The pyramid still holds — most inversions are self-inflicted
The shape hasn't changed: a wide base of fast unit tests, a thinner band of integration tests, a small cap of end-to-end tests. Teams invert it because UI tests feel like they prove more. They prove less, more slowly, and break more often. A 30-second unit test that fails on the exact broken function beats a 6-minute E2E test that tells you only that "checkout is sad."
Flaky tests are worse than no tests
A test that fails randomly 1 in 20 runs will, at scale, fail somewhere on nearly every pipeline run. Your team learns to hit retry. Once retry becomes reflex, a real failure gets retried away too — and now your suite is actively hiding bugs. Quarantine flaky tests fast, fix or delete them, and treat flakiness as a Sev-2, not a nuisance. A smaller suite you trust beats a large one you don't.
Where to start on Monday
Don't launch an automation initiative. Do this instead: pull the last 20 production incidents and ask which ones a test would have caught cheaply. That list is your real priority queue — grounded in what actually hurt you, not in coverage percentages.
Coverage is a number. Confidence is the goal — and the two are only loosely related. What would you delete from your suite tomorrow if no one measured coverage?