Column - 16 June 2004

How to passively migrate transitory data

Summary

Rather than migrating transitory data during a conversion event, it can be moved gradually using a passive approach. The core migration process can actively migrate the persistent data between old and new IT applications, and passively let the transitory data be drained from the old application afterwards. This approach can reduce the outage time required during migration, and the processing infrastructure required to support the high volume / peak load of a migration event.

Actions

• Evaluate whether the data to be migrated from your old application is persistent or transitory
• If transitory, identify whether a passive migration approach can be taken alongside the active migration of its companion persistent data
• Identify the criteria that upstream interfaces will use to redirect transactions to the new application
• Evaluate whether the transitory data can be merged from the old and new applications into a consolidated processing stream
• Identify the elapsed time a passive migration approach will require to 'empty' the old application (assuming no transaction replenishment)
• Evaluate opportunities for performance and process tuning of the new application based on a gradual, passive data migration.

What is 'transitory data' ?

Transitory data is usually transactional and uses the information contained in the transaction for processing rather than information stored elsewhere (e.g. a customer database). As well, transitory data's processing does not rely on what has been processed in the past, nor what will be processed in the future. That is, little or no historical information is retained - processing is performed and then forgotten. The processing of transitory data may be one step in an end-to-end process that stores persistent (customer) data downstream.

Persistent data is the opposite of transitory data. Persistent data includes customers' details, their past purchase history, specific preferences, orders performed and their invoices. This information is referenced by business staff, auditors and perhaps customers themselves - it must be retained intact for use and reference in future processing. Persistent data may contain historical transactions, but is usually viewed and migrated together to retain its collective integrity.

The following (simplified) example illustrates how the passive migration of a mail processing system (i.e. for letters with stamps) from the old to the new might work without the need to migrate the transitory data (i.e. mail) between systems. The delivery of your regular mail today concerns the current transaction (letter) and not what was delivered to you last week or what will be sent next month. Once a letter has been delivered the delivery details (for unregistered mail) can be forgotten.

Christmas mail can generate a delivery backlog with customers experiencing some delays until the backlog is addressed. If a new mail processing system was deployed over the Christmas period, the new system could accept and process all post-Christmas (i.e. new) deliveries whilst the old system gradually processed the delayed Christmas backlog. Mail would be output from both systems for delivery as normal by existing staff to businesses and residences. Once the old mail system had finished processing the Christmas mail backlog, it could be decommissioned with the new mail system processing all future mail (including all of next year's Christmas load).

Adopting this approach enables the 'mail system' migration to be approached passively for mail already accepted by the postal service, and avoids the need to physically reprocess all the Christmas mail through the new system.

A key enabler of this passive approach is the redirection of the upstream transaction source. In the postal example above, new mail is processed by the new mail system rather than added to the Christmas backlog. This means that the postal workers of the old system can ignore new mail processing and focus on clearing the backlog. When the Christmas backlog is finished there is no new workload on the old system. This will have been taken up by the new mail system that started empty and began to receive the mail from the cutover point. The cutover process can be gradual (e.g. based on postal regions) or absolute (i.e. 'all' mail after the cutover date is processed by the new system).

Handling interim processing

If a small amount of persistent data (used in transitory processing) is stored within the old application, there can be issues when transitory data is now stored across two locations. The persistent data needed to perform transitory processing may require a view across two data repositories. For example, a volume-based pricing approach in billing can't be performed accurately without a complete transaction history (or at least their summary). The solution to this is to deem one system as the master data repository - the old system until the persistent data is migrated, and then the new system from that point going forward.

During a passive migration's transition period, customer transactions can be stored in two locations. When addressing a customer or business inquiry, users may need to research in two locations (and use two application interfaces) to determine their answers. This extra work is temporary during the transition phase, and may be avoided if more about the customer and their migration status (e.g. their postal region) is known. If specific criteria are known that indicate that migration is finished or not started (i.e. for the northern postal region), then staff can determine where they need to look quickly.

When customers are granted access to their transactions (e.g. unbilled phone calls), consolidated views may be needed to avoid confusion. Data extracted from both old and new systems will need processing to present it in a consistent way. Additional complexity can then arise if customers update their data, or perform processing that operates across the two systems. In these cases the transactions' original source must be captured or re-determined.

At the completion of the transition period...

At the end of the transition period, when the bulk of transaction processing has been completed, the remaining errored and recycled transactions must be addressed to finalise the migration. This effort is finite since no new transactions will be received.

Sometimes business judgment may be used to eliminate / write-off some transactions. That is, the effort / cost required to correct them is outweighed by the business benefit to be gained. This is especially true for the last transactions still located on an application about to be decommissioned. This approach will not be suitable for all occasions - it depends on the application involved and the data being migrated.

Once the passive migration's transition period is completed, and the persistent data has been migrated, there is likely to be little or no information left to archive or migrate. With no new transactions and all the old ones processed, the old application can now be decommissioned.

A passive migration timeline

• Establish downstream mechanisms to integrate feeds from both the old and new applications: They will initially receive nothing from the new application and the existing processing load from the old application.
• Redirect part of the upstream transactions from the old application to the new application: Downstream processing will begin to receive transactions from both applications. The old application will process to a lower base-load with the remainder being processed by the new application. The new application can be tuned using this initial workload.
• Further migration events are performed: The old application workload is diminished whilst the new application is asked to perform a greater percentage of the work. Due to the (parallel) sharing of the total workload, the elapsed processing time may be reduced.
• The last migration is performed: All transactions are directed to the new application which must process the entire workload. The processing load of the old application reduces to zero as transactions are directed downstream.
• Old errors are addressed: The transactions on the old application are cleared until no transitory data remains.
• Old application is decommissioned: This will include the removal of downstream processing that accepted transactions from the old and new applications.

Migration opportunities

• Allow the tuning of the new system based on a gradual increase in its workload
• Early reductions in the old system's hardware based on reduced processing loads (i.e. cost savings)
• Interim processing may be faster with parallel extraction performed at less than peak load on two systems. This may reduce the risks associated with tuning the performance of the new application.
• Reduction in migration costs by avoiding the migration effort of transitory data during the migration event (weekend)
• Risk reduction by allowing the migration to be slowed if the processing load build-up on the new application cannot be handled (i.e. performance issues)

Tags: Migration, Data migration

[ Share with others ]

Post this page to a social bookmarking site:

delicious	Digg it	Furl	Google
reddit	StumbleUpon	Technorati	Yahoo MyWeb

Other 'purebill' columns

• Reduce migration impacts by merging outbound interfaces - Hiding upstream change from downstream applications reduces the overall impact, cost and complexity of a migration.

Previous column: How ready is your billing data?

Next column: Reusing the processes of billing migration

All previous purebill columns can be found in the archive section.

Recent Updates

• Column - Defend revenue in depth - Credit Assessment, Fraud and Credit Management (23 Jan 2010)
• Note 64: Report Distribution Access (09 Jan 2010)
• Note 63: Reporting / Business Support (04 Jan 2010)
• Column - Managing the 'case' of alphanumeric keys and identifiers (20 Jul 2009)
• Column - Stretch Key Dimensions to See What Breaks (14 Mar 2009)

Sign up to receive a brief text email when a new purebill column is published.