How failing tests can be green

Imagine this conversation:

Friend: “We have an integration system where we deploy every two weeks all commits and run hundreds of end to end tests on it.”

Me: “And if everything passed it goes into production?”

Friend: “Yes.”

Me: “How often do they fail?”

Friend: “Uhmm, well like every night. But we just take the failing tests and run them again and after that or the next run they will be green.”

Me: “And then it goes into production?”

Friend: “Yes.”

Me: “Uh?”

After some further digging it turns out the tests where not green in the first run for some time (like months). Nobody seem to care. Everybody thought they have just an unstable test system and no time for fixing it.

“Slow is the new fast”.

At some point somebody took the time to investigate one of the failing tests. He found that this was a real bug also happning in production and he opened a bug ticket for it. Yes, the system was unstable and they had flickering tests but this is no reason to put the (un)tested code into production. It took a team of 5 people three weeks to stabilize the system. No new features were deployed during that time.

If you accept that red tests are ok you are on the way to hell. You have no clue if the system behaves like you think it should.

No flickering tests are allowed. The build has to be always green. Do not accept any excuses telling you that this is ok and everything is under control.

It is not.

Don’t do it.

(Traffic Light Image from psdblast.)