Data Migration Services (DMS)

As new e-business, integration and package solutions are put into production in hundreds of companies, the old issue of Data Migration continues to pose a challenge. Data migration needs arise in migrations from legacy systems to package solutions. It is required when web based systems are put in place that need access to back office data. It becomes a necessary activity when systems consolidate or are spun off.

Data migration, on the surface, appears to be a simple and straightforward task. Yet practitioner know that it can present formidable challenges. Careful methods have to be adopted to ensure that your valuable data is properly transferred. Transformations have to be done correctly. Integrity and other relationships between data have to be preserved. The semantics of the business objects and the related data on the source and target sides often have subtle differences that can make migration rules complex. Data migration not done properly can expose your operations to risk or at least delay the new systems from going into production. Today, applications have hundreds of tables, business rules and database objects that need to be brought across. The volume of data that needs to be transferred can also be daunting requiring adequate resources and careful planning.

To ensure a smooth, complete and accurate Data migration, Kanrad recommends
- a sound migration methodology
- adequate project planning
- use of appropriate tools for extraction, transformation and loading
- proper test suites to test the migration
- sound audit procedures and documentation
- good issue resolution mechanisms
- thorough cutover planning.

Kanrad's DMS addresses all the above issues in a comprehensive and integrated fashion. Processes are well laid out, work breakdown units identified and risk managed throughout the process.

The Kanrad Data migration team consists of Project Managers, Data Architects, Data Migration Analysts, Development Engineers, SysAdmin/DBAs and Testers.

The Migration process

Kanrad's data migration process generally follows the following steps:

Migration Requirements Analysis
This is a key initial step in understanding the work involved. Kanrad's Data Migration Analysts work with your data modelers, analysts, developers and DBAs. They identify the applications and the scope of the data that needs to be migrated. Source and target data structure information is collected and mapped. Business rules are collected and studied to arrive at proper transformation logic. A proper inventory of source and target data is made.

Issues encountered at this stage are often related to
- data semantics,
- lack of availability of required data,
- quality of data,
- differing integrity rules
- changing targets (or sources)
- simply lack of adequate information about the source that leaves many data elements unidentified or ambiguously defined or used.

These are serious issues and Kanrad's issue resolution process is used to resolve them expeditiously in close cooperation with the client. Where the issues cannot be resolved immediately appropriate modifications to the project risk model are made and management kept informed.

Based on the requirements, the Acceptance Criteria are defined and a Test Strategy developed to verify the correctness of the migration after it is done.

The deliverables of this stage are
- an inventory document
- a mapping document (with issues identified)
- a project plan (using PERT)
- a detailed acceptance criteria and test strategy..
An incremental approach is suggested so that the migration effort is undertaken in manageable portions.

Migration Platform & Tools
Depending on the requirements a suitable migration platform is set up. The efficiency of this platform is key to productivity of the migration especially if the data volumes are very large. Multiple platforms are setup - for trial and production runs.

Custom data migration programs are developed as per the transformation requirements using appropriate programming languages and tools.

Detailed test plans are developed. Audit trails and logging routines are set up. Where possible data source identification is retained or recorded. A revert policy is established in case the production load needs to be undone.

Reverse Engineering
If adequate documentation or information is not available about the source data then a Reverse Engineering activity is undertaken. The source code, meta data and actual data are used to attempt to extract the rules and relationships. These rules and relationships are validated with the Business Analysts and Users. Sometimes a Best Guess or a compromise solution will have to be accepted.

Trial Runs
Trial runs are important. Trial runs of the programs often reveal the need for changes to accommodate missed conditions or further tuning. Various kinds of errors are identified requiring further analysis and clarifications. Such iterations are taken a few times till the test data show acceptable results.

Data Scrubbing
Poor or unacceptable Data Quality at the source is an often encountered problem and Data Scrubbing becomes an important activity in data migration. The Scrubbing rules are established during the Analysis phase and the corrections made are logged. Sometimes this activity can severely impact the time line because cleaning the data might involve research and investigation of non-electronic records, etc.

Production Load
This is the actual load of data. Execution is carefully planned and monitored. Sometimes production loads require the dropping of certain integrity constraints on the target side. Care is taken to ensure that the original state is restored once the load is completed.

Testing
Once the data has been loaded stringent tests are performed to ensure that the migration was successful and accurate. These test include the running of test scripts, random checks, the inspection of logs and audit trails, cross validation of data between source and target as well as any application level tests on the target side.

Repository
All documentation and routines associated with the migration are left behind as part of the Repository. This repository is particularly useful for other similar migrations and for learning from the experience.