How long does it take to run your Regression Testing Set?

1 Comment

Regression Testing

I want to dedicate this blog to a specific problem that we have during this iteration.

This post is for people who have the same problem, therefore can benefit from it and get some ideas from what we are trying to do.

Our feedback time is giant (Too long to know if we broke anything after committing code). After performing a quick value stream map for the development task we found out that we take between 24h to 48h depending on commited time and working hours.

Scenario I = 48 hours

Commit at 9:00 AM
Automatic build ready to be tested at 00:00 AM
Results at 6:00 PM (It takes 18h to run all the test cases, at this time developers are out from office)
Developers are back at 9:00 AM

Scenario II = 24 hours

Commit at 06:00 PM
Automatic build ready to be tested at 00:00 AM
Results at 6:00 PM (It takes 18h to run all the test cases, developers are still at the office)

Currently the team has 2 Test Sets, first one for smoke testing, second Full regression testing set; The smoke testing covers just the basic part, all new test cases for new features are added to the Full Regression testing set.

New Approach:
The team decided to create a third test set called “Release Test Set”, this test set will contain the test cases for features developed during the ongoing release, this means, all the automated test cases created by the team during the iterations will end on this “Release Test Set”.

When the product is released all these test cases will be moved to the “Regression Testing Set”. With this approach the team will save around 17h of waiting time.

We will continue to run the Regression Testing Set on daily basis but this time the team has the possibility to get quicker feedback about their actions during the iteration.