A week or so ago I went on a scrum bootcamp with my team at work. The course facilitator proposed, when asked about the disruptions that the team is exposed to for production fixes, the pattern of having a separate team for production defect fixes. I have been thinking about this and, although there are some valid arguments for this approach, I do not agree for the following reasons.
I feel that the development team should be striving to deliver software features that are defect free and operationally efficient in a cost effective way. The support desk and operations guys are the ones that traditionally bare the brunt of suboptimal deliverables.
If defect fixes are done outside of the delivery team, the delivery team ultimately does not experience the consequences of poor quality. If a developer is going to be woken up at all hours of the night, or spend their days fixing bugs, then I am really confident that they will spend extra effort to make sure their deliverable is defect free or come up with ways to improve the process. As developers we all want to be working on new features, not fixing defects.
If the team fixing defects is separate to the project team then the consequences for the developer are not as dire if they get it wrong. It basically becomes someone else’s problem when the code hits production, which is wrong.
I argue that, second and third line support should be the responsibility of the delivery team. If it is not, the delivery team can become separated from reality when they are not involved in investigating and fixing defects. I feel it is really important for developers to be aware of the production defects that are being logged and the types of issues that their clients are experiencing. Getting this information firsthand allows for reality to be used in future enhancements to improve the system and client experience.
This, one would argue, adds noise into the process of delivering new features and enhancements. And it most definitely will. However, if the team is motivated to do what they enjoy, that is delivering new features and enhancements, then it should only be for a short period until the issues in the process are resolved. But if the team is not involved in the second and third line support, and feeling the pain, then they will continue to deliver in the same manner and there will be no urgency to change. So the noise should only be for a short period until the team is delivering features that are defect free.
Without this type of team structure, opportunities to learn and improve are minimised. Feedback and learning are critical to enable continual improvement. Learning happens on a number of fronts from supporting defect fixes. Firstly you are getting first hand knowledge of how the system is being used and how it is performing. Secondly, it is an opportunity for the team to provide each other feedback.
The learning should lead to continuos improvement in development practices. As developers we should be continually looking for areas to improve our practice and without this type of direct feedback continual improvement is not as urgent. If defects are getting into production we need to ask ourselves why? What did we miss in our unit tests? Was there a gap in the requirements? Should we have done a code review and if we did not do a code review, why not? In my experience it has been the smallest, most inconsequential changes that often result in defects in production.
As a team after fixing a defect we should be striving to get this feedback to understand the cause and the effect. Doing the Five whys as a team is a good way to achieve this. This is an incredibly powerful tool but really difficult to implement. You need to really think outside of the box when doing a five why’s, but also it must be done in a safe space. And as a delivery team and a defect fixing team there is the chance of finger pointing. Difficult questions need to be asked, and doing one of these sessions in a hostile environment is not productive.
In closing, I am a firm believer that the team creating the new features should also be the team that is fixing the defects they create. Otherwise it simply becomes a continual circle of delivering sub-optimal code containing defects. The urgency needs to be created to change this cycle, by feeling the pain and continually striving to find out how features can be delivered that are defect free.