Cloud deployments and SaaS offerings have moved many workload deployments out of the data center. However, the local data center still hosts workloads and data that can be legally restricted, too important, too sensitive or otherwise impractical to trust with outside vendors. That means a lot rides on successful data center upgrades.
Upgrades remain the primary means of advancing the data center’s IT capabilities. They build resilience, boost performance, lower operating costs, enhance security and rein in downtime by replacing aging or underperforming IT assets. But upgrades always carry an element of risk: Oversights and mistakes can lead to unexpected downtime, issues with workload availability, performance disruptions and unacceptable management or security gaps. It’s important to nail upgrades successfully the first time through or, at least, have the capacity to fall back effectively if things go wrong.
The key to smooth, successful and meaningful data center upgrades is preparation. Know the goals and the path to each goal, and meet the roadblocks along the way to bring huge benefits to the upgrade process.
1. Understand your business needs
Some of the costliest and riskiest mistakes occur when business leaders opt for unnecessary technology investments or when they lack clear criteria for ROI. Upgrades shouldn’t serve as a showcase for new technologies but should help the business operate more efficiently and competitively. They should focus on the following:
- expanding vital resources, such as storage or compute;
- enabling new workloads for more, new or updated applications;
- supporting additional employees, users or transactions;
- lowering the cost per user or transaction; and
- building IT capabilities, such as workload resilience, business continuity and disaster recovery.
For example, a server upgrade might support consolidating multiple workloads onto one system, reducing costs such as power, space and system maintenance. Adding storage to the infrastructure can be as simple as adding new disks to the existing storage server. Include a redundant, load-balanced server to enhance the availability of critical workloads.
2. Identify the upgrade targets
With business needs clearly defined, determine the scope of the upgrade project, and identify the hardware, software, services, policies, processes and workflows affected by the upgrade.
Upgrades are never an all-or-nothing decision. Upgrade scope can be incredibly narrow and specific, so clearly determine the scope in advance to prevent potentially disastrous oversights, such as unanticipated hardware or software version incompatibility.
Consider an aging legacy workload and workload-dependent hardware. You could rework the vital workload into an updated software product, deploy that updated software on capable new server hardware and retire the aging hardware. In this case, the principal upgrade targets would involve both the legacy server and its legacy application.
Also, consider the secondary scope, such as dependencies. For example, if that aging application relies on an older database, must you upgrade that database and its hardware as well? Practices such as application or systems management are considered dependencies, and you should include them in the scope.
3. Create the upgrade roadmap
An upgrade project or goal usually involves multiple discrete upgrade tasks. For example, an upgrade might involve adding two racks of servers to the infrastructure, but that might also require the addition of power and cooling capacity upgrades or enhancements to WAN access.
Even something simple like a server upgrade can involve supporting work. For example, to add a 2U server to a full rack, you might need to relocate one or more other servers first to make space. Such considerations can seem like minutiae, but overlooking even small details can delay an upgrade.
Examine the intended goal and its dependencies, and target the dependencies first. This might be a matter of simply relocating hardware to make appropriate rack space or upgrading supporting platforms — such as a database and its server — before rolling out the intended upgrade.
4. Identify the upgrade technologies
Identifying new products for an upgrade might be as simple as selecting new disks for a storage upgrade or choosing the make, model and loadout for new servers. Larger upgrades demand more consideration. For example, if it’s time to update a group of related servers, evaluate the potential of advanced technology platforms, such as hyper-converged infrastructure.
Going into an upgrade with unfamiliar hardware and software is a recipe for disaster, so test products before committing to them when possible. Use hands-on evaluations and proof-of-principle projects to validate new hardware and software products and learn their deployment, setup, configuration and interoperability considerations.
5. Clean up or enhance existing infrastructure
Everything in the data center has a lifecycle, and aging infrastructure can pose an impediment to the business. As you review infrastructure to determine the scope and requirements for an upgrade, take additional time to consider tasks or projects that might not directly impact the intended upgrade, but can still benefit the business and infrastructure in a broader sense, including the following:
- adding redundancy to a critical application;
- removing redundancy from applications that no longer need it;
- updating aging cabling to facilitate greater network bandwidth;
- updating power distribution and power backup, such as power distribution units and uninterruptible power supply subsystems;
- repositioning racks and closing open hardware gaps to enhance cooling airflow; and
- reviewing system management tools and capabilities.
Such tangential or secondary upgrades can enhance the data center’s performance, reliability and efficiency.
6. Prepare documentation
Most data center hardware and software require an assortment of details for deployment. Details include default network addressing, licensing data such as activation codes, and detailed instruction for setup and configuration, such as product manuals and user guides.
Organize this data, and ensure the availability of everything before launching an upgrade task. Otherwise, the business risks unacceptable delays and unwanted cancellations because of missing details. For example, a new software product might require a license number or activation code. IT staff don’t want to scramble for this information when performing an upgrade at 2 a.m. when the vendor’s sales and support staff might not be available for a prompt response.
7. Back up and be ready to restore
Today’s virtualized data center environments make it relatively easy to move workloads to other servers, which enables you to replace and configure an emptied server. At the same time, you can replicate workloads running inside VMs using copy or snapshot technologies. Still, mistakes and unintended consequences happen, and even seemingly straightforward upgrades can go sideways in a hurry. Before and during an upgrade, prepare for the worst with simple guidelines, such as the following:
- performing a complete backup of any server applications or storage systems involved in an upgrade process;
- testing the restoration process and ensuring that the restoration works;
- ensuring that all IT staff involved in the upgrade task can execute a restoration or rollback successfully;
- documenting the current configuration of existing hardware and software — even if you plan to retire those products — so that you can restore them to a known-working state if necessary; and
- documenting all changes that take place during the upgrade — such as changing an IP address or moving a VM to a different physical server — and ensuring those changes are reflected in systems or change management tools.
8. Keep stakeholders informed
Upgrades can disrupt regular business and keep important workloads unavailable for prolonged periods. This affects employees, business partners, customers and business leaders alike. Data center upgrades should always loop stakeholders into the process.
An upgrade project usually involves numerous tasks. So, treat each task individually, and communicate that to stakeholders. Common communication includes the following:
- telling stakeholders what upgrades are planned, why the work is necessary, what changes or new capabilities should be available and when the work takes place;
- reminding stakeholders about upgrade schedules — usually including several reminders leading up to the actual upgrade task;
- sending an update to stakeholders if you encounter problems or delays; and
- sending an all-clear to stakeholders once the upgrade task is complete and normal functionality resumes.
Also, provide accurate contact information for support or help desk access if problems arise. For example, if a software upgrade comes with new features and functions, train the support personnel on those changes in advance so that they can effectively address user questions and problems.
9. Validate the deployment
Once you complete the upgrade task, test and validate that the hardware or software deployment works properly. For example, a new server should run well and have secure configuration. When you reinstall or migrate workloads back to the server, those workloads should become accessible across the local data center network. At this stage, IT staff must benchmark and measure performance, troubleshoot and remediate upgrade problems, or execute a rollback if needed. Only once you know everything works as intended should you open the resources for general use across the business LAN or internet.
Upgrades can disrupt systems or application management tools. Capture any changes involved in an upgrade task in all data center systems and security management tools, or prepare to enter changes manually using the documentation you collect. Some management platforms require additional installation of agents or drivers before the management tool can oversee new assets properly.
10. Roll out a deployment systematically if necessary
Roll out upgrade projects that involve high levels of uncertainty or risk in stages. For example, updating a mission-critical legacy application and its associated hardware involves deploying new assets simultaneously — in parallel with current assets — and opening segments of the environment for beta testing, rather than simply replacing the application and hardware in an upgrade. This minimizes the impact of unforeseen problems after the upgrade goes live because the original hardware and software remain available and operational.
If the new assets operate as intended, you can systemically migrate additional user groups to the new assets. When you have migrated all users successfully and thoroughly proven the new assets, you can then retire and remove any old assets as a smaller cleanup upgrade task.
11. Consider data center standards
There are no established requirements for data center design, implementation, performance or availability. However, governments, stakeholders, partners and customers have come to expect a level of adherence to certain standards. Upgrade projects are an excellent opportunity to examine data center standards and consider the value of adopting an existing standard.
Numerous code and best practice standards can validate your data center, including Uptime Institute, Telecommunications Industry Association 942, American National Standards Institute/BICSI 002-2014, EN 50600 and a range of optional standards, such as Leadership in Energy and Environmental Design, Green Globes and Energy Star. Understand the role of each standard, evaluate its importance or benefit to your business and plan to implement desired standards as part of ongoing data center upgrade projects.