Succeeding with Agile Software Development - Mike Cohn

Why transitioning is hard

1. Successful Transition is neither top down or bottom up ONLY. 
1. Successful change is both top down and bottom up. "Change is best introduced bottom up with support at appropriate points from the management - both local and at a higher level". An organization attempting transition without top management support will encounter resistance that cannot be overcome from below (happens when the process begins to affect areas outside the original team). Without bottom up engagement, the transition will feel like sitting under a ceiling fan in an open air restaurant in Mexico; just a bunch of hot air blowing down from above. Individuals resist being told what to do.  
2. The end state is unpredictable. 
1. "Having a chance to change or personalize a process (or a framework or practices -- XP, Scrum, or another) to fit themselves seems to be a critical success factor for a team to adopt a process." 
2. You may have a clear vision of what "doing Scrum" means to you, and you may get others to buy into exactly that same vision, but where the organization ends up is likely to be somewhat different. In fact, to even refer to end states in a Scrum transition is incorrect; there can be no end state in a process that calls for continuous improvement. 
3. This creates a problem for an organization that wants to transition to Scrum thru a traditional change approach that relies on gap analysis and then on closing the identified gap. If we cannot anticipate the end state of a Scrum Transition, we cannot identify all of the gaps between there and the current state. So, a gap-analysis driven approach will not work. The closest we can come is to identify gaps between where we are now and an improved, intermediate state.
4. Organizations are living systems. We can never direct a living system, only disturb it and wait to see the response. We can't know all the forces shaping an organization we wish to change, so all we can do is provoke the system in some way by experimenting with a force we think might have some impact, then watch to see what happens -- Christopher Avery's views on Organization as a Living System.
5. Transition to Scrum cannot be a process that "articulates and defines the entire change process required to bridge the gap between "as is" and "to be" and creates tactical plans." Creating such a plan would require leaping two impossible hurdles - first knowing exactly where we all want to end up; and second, knowing exactly the steps to get there. Because we cannot overcome these impossibilities, the best we can do is adopt a "Provoke and Observe" approach in which we try something, see if it moves us close to an intermediate improved state, and if so do more of it. These pokings and proddings of the organization are not random. They are carefully selected based on experience, wisdom and intuition to drive a successful transition to Scrum.
3. Scrum is Pervasive:
1. Affect an Individual at a very fundamental level. A change like, say, "introducing an operational readiness check" is an isolated change. This is not a pervasive change; even if the dev team has to go for it, the change is unlikely to affect them fundamentally in how the accomplish their daily work -- they can still continue doing the majority of their work unscathed. As a comparison, a developer transitioning to scrum will need to change herself fundamentally -- work on smaller pieces of work, do TDD, write automated tests, remove off headphones while doing pair programming, and a lot more. These changes are not something to be relegated to a few hours a week like code inspections. Resistance in such cases will be a lot more.
2. Affect has implications beyond the team adopting it. For example, introducing operational readiness check may not have impact on finance, sales and other departments, but introducing Scrum will definitely impact those departments. 
4. Scrum is dramatically different.
1. On Scrum, members need to unlearn their past behaviors (testers testing to compliance, programmers trained to analyze problem in depth for a perfect solution, etc). They need to adjust to the concept of emergent design. 
5. Change is coming more quickly than ever before. 
1. Change is coming too fast, too often and teams are mostly not prepared for change. Teams are asked to do more with fewer people. Outsourcing and distributed teams are now more common. Constantly accelerating change of technology. 
6. Adopting best practices 
1. Is dangerous for Scrum - what works for one team doesn't work for another.