Chris Lucian

Recently I was asked "How much effort should we really put into automated tests and refactoring?"

For some context here is our testing and refactoring process. Note for every new feature we build Automated Acceptance Tests, Unit Tests, and Refactor afterward.

On top of that we value automation so much we incorporate an evaluation for automation in all ticket or bug tasks.

Here is a great chart on a 5 year time savings of automating the task. (Source: https://xkcd.com/1205/ ) 

Using this chart we can extrapolate time saving that unit testing, automated acceptance tests, and refactoring.

If we take this quote as a basis for the argument:

“Indeed, the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code. ...[Therefore,] making it easy to read makes it easier to write.”

― Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship

For example unit testing, we spend a portion of all of our work creating unit tests. We have found that we can get a very low bug rate and drive down the cost of development through this practice. 

The following is a short article on the benefits of unit testing: https://dzone.com/articles/top-8-benefits-of-unit-testing

Sections 5,6, and 8 directly relate to the time spent reading code. If our unit tests save us only 6 hours a week, according to the above graph over a 5 year period we would spend 2 months (per person) less in debugging time as long as we have good code coverage. In previous environments I had seen bugs eat up many hours a day, in those environments we could have seen over 5 months or more time savings just by unit testing. Since unit tests provide documentation, expedite the debugging process and increases quality we should be focusing on making this a strong skill.

The following is an article on the top 10 benefits of automated acceptance testing: https://saucelabs.com/blog/top-10-benefits-of-automated-testing 

Companies that need to do manual integration tests have to have people dedicated to this task. This means we save entire headcount in this area by automating integration level  testing. These tests also help developers quickly gain context as well. These tests take more time to maintain so we cannot lose the unit tests by using only automated acceptance tests.

One last part of this is my analogy to herd immunity. Bugs can still creep into the code with the above practices however when enough of the code is tested in this way we will isolate the side effects of a bug to a recent change rather than many changes over time. If most of the code is tested then the bugs that do come up are easy to solve, however the fewer tests the less efficient we will be at debugging.

source: https://en.wikipedia.org/wiki/Herd_immunity 

So the amount of time we invest in writing tests and refactoring has an enormous payoff in the long term. Especially if you take into account the lifetime of projects that had lasted 10 years or more.

If any of these numbers seem small in any way we couple that with technical debt and things become much worse. We spend time refactoring because we want to keep the cost of change down. Here is a chart on how technical debt can get out of hand: 

source: https://www.planetgeek.ch/2014/06/12/effective-teams-always-releasable/ 

Finally there is an argument against this which is the time to market, this is where the "design payoff line" comes into play. In this blog: (https://martinfowler.com/bliki/DesignStaminaHypothesis.html) Martin Fowler states that the design payoff line is hit within “weeks not months”. (let alone years)

Based on the evidence I have and the axioms I hold to be true, the level of testing we ask of our developers is the right one. I hope this information is helpful.

What about the cost of the tests? 

A followup asked was a question about the increase in cost in maintaining the tests. As for the tests being more code to maintain, I think you have to consider the refactor step in "Red, Green, Refactor". The team is asked to always refactor after the tests go green. This means removing duplication of code and creating single points in the application of where to modify it when the task is completed. Ultimately this means less, and easier to read code. We can only refactor safely if there are unit tests around all of the code. Those are the ideas presented in the Clean Code and Refactoring to patterns books. Ideally you will end up with less code when you write many unit tests. This means that the team needs to be really good at refactoring and unit testing. 

Feedback and questions always wanted. Please leave comments!