Running a “big bang” datacenter migration is a lot like conducting a symphony: A cross-functional group of highly-trained performers – each with different expertise, perspectives, and parts to play – must all execute together to produce the desired result. This requires the right team, careful planning, great communication, and a good information system. When this is done well, the result is much greater than the sum of its parts.
The scope of a datacenter migration can be either a continual slow migration of data and applications or a “big bang” project in which all data and all applications from one datacenter move to another datacenter in a weekend. The latter is much more financially attractive in many cases.
However, this approach requires intense coordination. The team for such an event must include various infrastructure teams (both internal and 3rd party teams such as telecom teams), specialized line-of-business application development and operations teams, business analysts, IT communications teams; the list goes on and the team structure can be very complex.
Each team has its own priorities in the migration, but to be successful, each team member must understand both the larger composition as well as the part he or she plays in that whole. Furthermore, teams must collaborate effectively during a compressed timeframe and be flexible when the inevitable contingencies arise.
In short, the organization must become a “team of teams” to achieve success. This goes beyond simply having a cross-functional team assigned; one must ensure the teams work as an ensemble. At Trexin, we call this state of dynamic alignment in an organization Aligned Execution. To achieve such alignment in a cross-functional team environment, team members must understand with their heads, engage with their hearts, and execute in unison with their hands.
It was recently our honor to help an operating entity of one of the largest financial services companies in the world achieve this alignment during their “big bang” production datacenter migration. Here’s how we did it.
Composing the “Music”: The Cutover Plan
A symphony orchestra could not perform a symphony without music, and a datacenter migration team should not attempt a datacenter migration without a commonly understood cutover plan.
The overall cutover plan is absolutely critical to success. It must unite detailed plans from each team in a common framework and be documented early so that risks and gaps become apparent to the team. The cutover plan becomes the basis for the communications which will align both execution teams and stakeholders to a common vision.
This is the “music” the team will follow. As a cross-functional document, it is a common framework with clearly defined milestones and expected timings. It sets the order, rhythm, and cadence of tasks. It outlines dependencies – especially cross-team dependencies – and it does so in a way that is flexible enough to adjust to any contingencies which may arise. Speaking of contingencies, a well-thought out risk plan or rollback plan is critical. If the worst happens and the migration must be aborted, every team must execute that plan efficiently.
Approximately 90% of the work for a Datacenter Migration is in the planning. IT Leaders and, to some degree, business stakeholders, should know what activities are required – day by day and hour by hour. If this is not the case, the risk of highly visible failures increases dramatically.
Finding the “Ensemble”: The Right People
One cannot perform a symphony with only the horn section or only the violin section. Similarly, a datacenter migration is not only an infrastructure project or an applications team project; clearly, both teams play a critical role. It’s an ensemble piece in a way that few other IT-led projects are.
This is not the time for prima donnas. There is plenty of room for great leadership and great individual performances. But a well-executed data center migration requires intense cross-functional communication and teaming. One cannot afford those leaders who do not understand the ensemble nature of this work.
Hiring the “Conductor”: The Right Communication
Any sufficiently complex human activity requires a central point of control and communication. For the symphony orchestra, this is the conductor, an individual whose sole focus is extracting the best performance from a team.
The reason a conductor exists is simple – this leader manages the “big picture” view of the event. The conductor serves to rehearse the greater team and to dynamically coordinate critical decisions during the performance. She is in full control of shaping the performance and recovering from any mistakes along the way.
“Big-Bang” datacenter migrations require a strong command center, led by an event controller. The command center comprises key representatives from infrastructure and applications teams as well as testing, issue management, and communications teams. It is a central point of decision-making and coordination for all teams involved in the event. The event controller owns the overall execution of the cutover plan. When contingencies arise, it is the event controller who pulls the appropriate levers to get back on track.
Building the “Information System”: The Right Tools
The information system for the conductor is simple: the baton and his ears. The conductor uses a baton to communicate start times and stop times, cadence, volume, balance and emotional tone. Assuming he is not deaf, he has built-in feedback loop to understand what’s happening. Furthermore, all the performers are more than likely together in one place and maintain a high degree of situational awareness.
In a datacenter migration, there is no built-in information system. The command center must communicate and receive status information from teams that are likely not located in one place. By assigning representatives from each execution team to sit with the command center, we lay the foundation of a critical information pipeline. For this pattern to be effective, information must flow in sufficient detail and with sufficient data currency to support effective decision-making, yet not burden execution teams with reporting overhead. This critical balance cannot be defined; it must be found through practice. In our experience, specific tools and automation are highly desirable.
At key points in the migration, the status reporting must be communicated in business context not simply in technology context. As an example, when making a “Ready for Business” or an “Abort” decision, it is insufficient to know that something on the Mainframe isn’t working; the decision-maker must understand that customers will be unable to access the claims processing system as a result. This also drives communication to stakeholders, another key command center function.
Governance and Sponsorship
Even the Conductor has a boss: the business sponsors of the event. This is also the case for a datacenter migration, or for that matter any program of sufficient size. The function of this governance function is to guide the teams and make the really tough and high-risk decisions. This executive committee makes key decisions such as “Go/No-Go” or “Rollback” decisions. If the conditions are not right for a datacenter migration and it needs to be rescheduled, it is the governance committee which makes this decision and guides the communication.
However, it must be noted that the governance committee must not interfere with execution unnecessarily. Once the show has begun, it is the conductor who makes the key execution decisions. It is the same with the event controller during a datacenter migration.
Critical to success is an active, engaged sponsor willing to make the tough calls when necessary, but also get out of the way of the emerging high-performance team wherever possible.
Practice, Practice, Practice
No team – from a college football team to a symphony orchestra to the team that put a man on the moon – would be very successful without intense, repeated practice as a team. Practice makes the team.
Let’s assume that we have a documented and socialized plan with estimated timings, a team of collaborators, a strong event controller, and an information system to support decision-making. But understanding the plan with your head and knowing the plan with your heart are two different things. The latter is only possible through shared experience and practice.
Even assuming perfection of the technology solution, the “team of teams” needs opportunities to build experience and trust in a lower risk environment. For performing artists, this is the rehearsal period. They start small by breaking up activities into easy chunks before they stitch the entire composition together. They build in a feedback loop so that feedback from critical perspectives are incorporated.
For your datacenter migration team, be sure to carve out opportunities to run the cutover plan. This may be executing the plan for a non-production environment first, completing a paper walk-through with all teams are present or conducting a dry run in an isolated environment. Make it a learning activity. Track actual vs. planned activity timings. If activities were missed or problems were encountered, these should be recorded. “Lessons learned” sessions should be held to find optimization opportunities.
If this is done well, then the execution team becomes a learning organization which can – over time – optimize the plan to the point that the datacenter migration appears seamless.
Careful Orchestration Drives Results
The specific practices outlined in this article help build the high-performance “team of teams” required to make a datacenter migration – or any other complex transition – a success. These include: documenting an end-to-end cutover plan, crafting a team from the right ensemble performers, designating an event controller and appropriate governance structures, and including opportunities for practice in the overall plan. Building a motivated “team of teams” in this way can be the difference between an “IT-created disaster” and an “IT tour de force.”
Please reach out to us if you need assistance with your datacenter migration strategy or with building your “team of teams.” Trexin Consulting’s experts are not only veteran consultants, we have also walked in your shoes as leaders in industry. Whether you’re looking at a complex datacenter migration, a migration to the Cloud, or a complex product launch, our strategy services can help your teams align and execute at peak performance.