Data migration process (Risk & strategy)

Risk management should be part of every project, especially a large scale migration project. Risks related to schedule, costs and scope should be clearly identified, documented and a plan must be put together to mitigate them.

In my experience, risks specific to data migration come from a number of contributors like

(1) data quality

(2) extraction, transformation and load tool complexity

(3) performance of systems (extraction and target system persistence)

(4) coordination of project activities as related to testing, UAT, system preparation

(5) resource constraints

Data quality:

Without a thorough understanding of the business rules and logic related to how the information is stored within the source and target system all migration projects are at risk.

Even if you understand the rules from a system perspective, try and get a handle on how the business uses this information. In some cases, even after a thorough mapping exercise, gaps could arise due to data issues…in such an event additional cycles will be required to re-map and re-run the extraction, transform and load routines. Don’t underestimate the work effort required to cleanse the data…in some cases, the risk mitigation steps involve leaving the source system as-is and including additional logic

Extraction, transformation and load tool complexity

As business systems add more functionality and modules, data migration becomes an extremely challenging exercise. In the past, data migration of metadata could be easily accomplished by writing data directly into the database(s) by using simple commands and flat files at the database layer. As business systems have matured, most of the times the database layer doesn’t contain the hierarchy and relationship information…these are stored and managed within the application layer. Extraction and loads have to use the application APIs and in some cases, these are not conducive to support extraction and loads in a mass manner. Often, schedules are adversely impacted due to added cycles of development to find/develop routines and software to support the extraction, transformation and load.

I would highly recommend contacting the software vendor and request reference information and talk to other customers/users that have been through a similar exercise and learn from their experience. Benchmark exercises like this will help in setting a timeline for such activities… Risk management could involve adding additional resources to drive closure of risk items. If additional cycles from software vendor are required, engage your vendor management team and actively manage your contacts at the software vendor. Escalations and jeopardy’s should be raised so that executives within your organization and the software vendor are informed about potential impacts and risks…

Performance of systems

Bad performance of business systems could lead to delays and inability to complete a load in the allotted time. We have talked about the complexity of the extraction, transformation and load routines…with added complexity it will be no surprise that most systems are not setup properly to support mass migrations…in fact different settings are required within application /business systems to support migrations and these could be totally different from what is required for day to day operations…

Thorough testing with multiple wash-rinse and repeat cycles are essential to clearly identify performance issues…in some cases, because of a n – tiered architecture, you might run into bottle necks at the application front ends or at the database layers…ensure your best performance people are monitoring the load process and can tune the systems properly.

From an database perspective (Oracle or others…), you might need to increase the memory allocated along with updating indices on a frequent basis as well as maybe even turning off archiving…another thing to do is maybe turn off search indexing within business systems to speed up the load process and perform this activity as a post go-live activity.

The key is to able to load all the data within the allotted time frame, some times risk mitigations might involve loading over a prolonged period, added front ends, additional memory and CPU’s, database/business system tuning.

Project coordination

Most often data migration projects are part of a larger project to implement a new business system. In these large scale implementations, there are a number of activities which must be completed for e.g. system design, configuration, process design, data mapping, unit tests, user acceptance test etc. data migration tests need to be performed so that all elements of testing can be done. This requires a high degree of coordination and any slips in code development, design configuration or testing cycles will impact the overall schedule and time allocated for data migration.

Risk mitigation could involve procuring a separate environment to conduct all data migration tests. In addition, ensure that all project activities which could impact migration are clearly identified and have the necessary dependencies captured within the project plan. Having a project review similar to scrum reviews can be beneficial.

Resource constraints

In an ideal world, all projects would be funded and resourced well with proper resource leveling. In the real world this rarely happens, the same is true with most of the migration projects I have managed. Typically the roles for project manager, business analyst, data experts, process owners and code developers are all merged and melded together and these resources have to wear multiple hats through out the implementation of the business system. In order to mitigate any risks to schedule and costs, clearly identify the resource requirement and secure the resources required ahead of time. The later you start; the chances of securing resources will be limited. Contract resources for execution of the extractions and loads can be viable option but requires clear documentation of the process and thorough knowledge of what needs to be done. If you have the right resource and have a plan, migrations can be handled with a set of contract resources…