How regression tests can save you time and money – Part 1

How regression tests can save you time and money – Part 1

By Alexandra Martinez

In my last post, Why regression tests are so important, I explained what regression testing is and why you should implement sets of tests for all your development projects. As a super quick recap, regression tests are a type of software testing used to make sure a new change to the code did not break existing functionality.

For this post, I’m going to focus on how regression tests can save you a ton of development and testing time. Then, in my next post, I’ll cover how they can also save you money on all your software development projects.

Let’s get into it!

If you don’t have any kind of automated testing (tests that run automatically by a program) in place for your project, you’re probably relying on manual testing. Having testers manually review your application or API is not a bad choice (it’s better than not having any testing at all), but it can lead to a lot of human mistakes and will definitely not save you time. In fact, you will likely have to add more time in order to test properly.

Manual vs. automated testing timeline

Imagine this: a manual tester takes roughly 1-2 days to review a very simple mobile application, and so every time that there is a change to the functionality, or the code, the tester has to spend time re-testing everything to make sure it’s all still working properly. This means that in addition to development time you’ll also be adding 1-2 days of manual testing per change to the application.

Depending on the project, you could be facing 4-7 changes every two weeks, which would add another 4-14 days for manual testing! And that’s assuming the changes worked properly and didn’t break anything. If the added functionality didn’t work, the code goes back to the developer to be fixed and then back to the tester again (adding yet another 1-2 days).

Instead of using manual testing, if you invest in automated regression testing, these numbers can be drastically reduced. For example, a very complex API could be entirely tested in only 10-15 minutes! Manual testing would likely take days, if we want the same detailed inspection and testing.

Let’s break it down visually:

As you can see, there’s massive time savings with automated regression testing. Even for the best-case scenarios, why would you spend a whole day on manual testing when you can invest in automated testing and spend only one minute for the rest of the project?

Real-life scenario

I can give you some real numbers on how much time we saved on a project by using automated regression testing for every change in the functionality.

At first, we were performing the regression testing manually. Every time there was a change in the code, we would have to go back to our set of tests and review that every single call is working as it’s supposed to. This would take us at least four hours to review, but not nearly as thorough as it was needed.

I would say that, on average, we spent eight hours per feature just on manual regression testing. We would get assigned five features per sprint (a two week period), which equaled testing for roughly 40 out of those 80 hours. We spent half our time coding and the other half testing!

We started implementing automated regression testing and our numbers improved dramatically. Now, these tests would take 10 minutes to run and would be much more thorough. Now, instead of spending eight hours testing each feature, we would spend between 30 and 50 minutes. We now spent only 3.5 hours out of the 80 hours per sprint. That’s less than 5% of the total time!

With this new approach, because human error was no longer a factor, we were rarely introducing new bugs to the project. This meant that the number of bugs was descending with every fix (since no new bugs were being created), our code quality and our time to deliver was improved, and our client was getting more results than before.

This was just one scenario, but going from 50% of the sprint coding the solution and 50% testing it manually, to spending less than 5% on testing with automated regression tests, I think it’s worth it to invest in regression tests.

Now that you learned about saving time, I’m sure you’re wondering if and how regression tests can also save you money. Check out part 2 here!

About the author

Alexandra is a MuleSoft Developer where she’s responsible for leading and mentoring teams, always ensuring the best quality for project deliveries. Made in Mexico, she enjoys learning new technologies and is always looking for new challenges. She loves statistics because she’s bad at them, so they keep her alive (an eternal challenge). Her natural habitat, however, involves watching Netflix and playing video games.