Why is it difficult to have a Green Build?
Hi guys, in my last post: Why is it important to have a “Green Build” I did explain how important is to have a Green Build. In this post I will explain several reasons why it is so complicated to have a green build
Before I start to enumerate several possible reasons I want to highlight the background of our system. We have a long way to go until we are truly agile. We have all the necessary skills within the Feature Squad to release a product but the organization is structured in different departments (System Build, Development, QA, etc.). This set up is responsible for the presence of silos. We are working toward an organization where silos will not exist but this is how organization works. On top of this the developer must wait several hours to get some feedback about what he did. He only knows if he broke something several hours later. Now with this information you will understand better the problems we are facing.
There are four different areas that I want to touch:
- Code changes
- Tests changes
- Build System
- Test System
I don’t have the perfect solution for all these problems, I have my beliefs and my opinions but I don’t know how to solve all issues in a short time. Since I do not want to finish this blog without any ideas I will give an example of another team we have where the results are extremely positive.
This small team has the ownership of everything. They own the build system, testing system, production system and whatever is needed for the work. They practice TDD, and there are no departments saying how they should work. They solve their own problems on daily basis, they do whatever is needed to make it “green”. They commit their code every couple of hours, they don’t go home if the code does not compile and all the test are green. I believe the biggest advantage of this team is the fact they know why a green build is so important. After failing so many times they understood why a green build is mandatory.
All organizations should aim for continuous improvement. However I truly believe the first step is to understand where we are and where we want to go. So if your organization has some problems, bigger or smaller, does not matter. I believe first you should realize which problems you have and define a plan where you want to go. Then just work with the rest of the organization to achieve those goals. Do not forget to empower your teams, they know what to do much better than you.
Please let me know what do you think.
I would love to get a star rating for this post:
I believe that not being able to keep a green build up, is related to how a green build is perceived. To me having a green build is a measure of the maximal speed a project can proceed with. You start slow, and then stakeholders start press for “faster, sooner”. The project leader/coordinator/manager ..needs to make it clear to them that the project has a maximal speed, and that speed is reached when the the build is no longer consistently green. That is the speed where things can not go any faster, without making hard to determine long term damage, and without building up technical debt.
Shorter said, I see it related to communication, and I mean real communication, that involves more than just sending the message (also involves making sure the message is understood clearly ..)
Another point of view that I see is that of how the green build is linked to some sort of incentive or performance metric. I would tie red builds to black balls, or to a lack of white balls. All those involved should be made aware that a project is a common endeavor, and they are all depending on each-other. In such an environment, when the build is not green, everybody should feel the same urge to correct it, because it would affect all, and if one person can not pull the brake by one-self, than others will also pull the brake, as it affects them also.
Hi Lucian, thanks for your feedback, I agree with many of your points but to be honest communication has little to do with understanding of a green build. I truly believe that goes deeper than that, I believe that you must understand software development really well, after that communication can help, but if you don’t understand software development it will be quite hard :)
As always, great post. I’m currently leading a project as ScrumMaster and Architect with a very small team (4 people in total), 4 months, quite complex stuff in there. New technology for most of us (Rails) and mostly junior people. And it’s going pretty good. And we do have a green build most of the time.
How we’re doing it:
- Physical isolation from other teams (although there’s play time during lunch!)
- Full empowerment: everyone knows how many hours they work. I’m against over-time, by the way. Everyone has an equal say about everything. I do have to lead on some items, but overall everyone feels it’s THEIR product.
- Accountability: whoever takes ownership of a story, is responsible for delivering it working as expected. Everyone is encouraged to ask for help when needed to other members of the project or the company, and websites like stackoverflow.
- Good tools: bought RailsCasts subcriptions, good IDEs and fast computers.
- Load of sticky notes and whiteboards around the room.
- Also, EVERYONE in the team takes part in the Sprint Demo with the client.
This is working, and I believe it’s mostly due to being VERY lucky with the 4 elements comprising the team.
As you’ve said, and also in the above comments by Juan and Wim, I believe it’s very important to remove dependencies on other teams and to keep the team as small as possible. Remember the book “Rework”: hire only when it hurts! The same applies to growing the team size.
I know this can be hard on complex projects, but an effort should be made to keep the team small and self-sufficient. IMHO.
Cheers,
Ricardo
Thanks for your answer Ricardo… I truly agree with what you said. The scenario that I had in mind was much more complex with several teams in a big complex product. In any case I think if we start small and make it work we can always scale it. In your case if you need to scale you will be able to do it, or at least i think so since you got the basics up and running.
I think you already listed the solutions to the above problems:
- remove silos: have multifunctional teams that dont depend on “other” teams.
- clarity: clearly show if and when the build fails. Make some sort of flow analysis to understand how long it lasts until team gets to know that the build has failed (especially in the case you described above, where developers dont control the build systema and QA department doesnt get to know about the latest changes done by Devs)
- encorage good unit testing and code reviews that could prevent the build from failing
- and finally, Devs should understand how important how important is to make the build green (like the team that wouldnt go home until it works).
my 2 cents.
Thanks :) so now I just need to make it work :)
I like this article. Keeping a build ‘green’ is very difficult an in some case even impossible. The more complex your product and environment grows, the harder it gets.
The teams may know the importance of green build, and the difficulties of keeping it green, but it is very important for the stakeholders (i.e. decision makers) to be aware that there IS a problem of keeping the build green. Awareness brings part of the solution, at least it opens a mindset where you can openly look for it.
When you have an product organization which is structured in different departments, I believe it is also recommendable to have a (small) team which is responsible of looking at the ‘whole picture’, and closely monitors how the work of the different teams has ended up (or not) into a ‘green’ end product. They could have the power to decide whether or not the build is green, and should have the competence to give independent feedback to the teams why a build is not green. And all of this needs to happen with the blessing of the stakeholders, of course.
I don’t think it would be that difficult to keep it green if everyone involved understands the concept and the importance. I see much more problems related with organizations that are structured around departments, these organizations typically incentive behavior like “my job is done” its red but i did my job… I think this is the true problem… We need to break these barriers and create cross functional teams, give full empowerment to the teams to do what they think they should do and let them deliver… I truly believe this is the way to go ;)