Throughout my career I’ve been involved in several data migration projects, ranging from a CMS with a decade’s worth of content to the engine powering an analytics dashboard with hundreds of millions of timestamped data points.
Despite often being neglected and overlooked, data migrations are critical for growing organizations, as they enable the transition to new systems when the old ones don’t work well anymore. However, migrations are also high risk projects; a failed migration can lead to data loss, financial and reputational damage, and a blow to team morale.
In my view, the planning and execution skills needed to complete a migration successfully distinguish great engineers from the rest.
In this article, I’ll share a few ideas and lessons I learned while working on data migration projects.
Deciding the migration target
Data migrations are typically a last resort, initiated only after every ounce of performance has been squeezed from an existing system.
The first big decision is choosing where to move the data. Sometimes, the choice is clear due to policy, compliance or legal reasons. Other times, there are a few options to choose from, and we need to think about them carefully.
For example, in one project, our team had to choose between a technology that (in theory) was perfect for our use case but new to us, and a general-purpose technology that we already were familiar with. We built a series of ‘spikes’ - exploratory prototypes - to assess the feasibility of each option. Factors like scalability potential, the team’s learning curve, support from platform/infrastructure teams and long-term productivity were all part of our evaulation.
During the decision process we captured our context, reasoning and trade-offs in an Architecture Decision Record (ADR). This documentation helped us to clearly formulate the goals of the new system while recording our decision-making process for future maintainers.
Migration steps
Building a prototype for the new system is just the beginning. Next comes the migration plan, which needs to consider both technical constraints and business expectations. For example, it is a very common requirement that the service must remain fully operational during the migration.
A typical migration plan includes:
- Provisioning the infrastructure, platform or data storage solution for the migration target.
- Creating a robust migration script that reads data from the old system, performs any data transformation required, and writes data to the new system.
- Testing the migration script with a diverse subset of data first, to find and fix any problems. Pro tip: Examine all the edge cases present in your data.
- Implementing feature toggles or an API gateway to gradually introduce the new system in a controlled way and allow revert if errors are detected. I guarantee that you’ll need this sooner or later.
- Monitoring health metrics in the new system, to ensure that performance matches expectations.
- Scheduling the migration in batches to avoid overloading the system. It is also handy to have a way to measure the progress being made towards completion so the status can be communicated easily to stakeholders.
- Withdrawing the old system and cleaning up any remaining migration artifacts.
Finally, don’t forget to celebrate! If the migration was successful, you’ll end up with a more robust and scalable system, and a strong foundation for future growth.