Migrating static and dynamic customer data separately

Summary

Customer data changes at different speeds - from relatively static details (customer name), to slowly changing information (address, billing hierarchies, banking details), through to ever changing dynamic transactions (phone calls, purchases) that may occur up until a migration begins.

The work per customer required during a conversion / migration event (weekend) can be reduced if selected, less dynamic data is pre-migrated from the legacy application ahead of its related dynamic data.

Increasing the customer data that can be moved per migration event brings benefits including:

More customers converted per migration (entire customer segments moved as a group)
Fewer migration events (lowered migration costs)
Shorter outages during migration (higher system availability)
Increased scope to fix unexpected problems during a migration event (lowered risk of a 'No go' migration decision)

How fast does customer data change?

Businesses store a wide range of information about their customers, each with its own propensity to change. All customer data falls somewhere on a continuum from information that is fixed and never changed, to details that change only slowly, through to the remainder that can change at any time.

Examples within each of these categories includes:

Static: Customer name (with changes usually only based on marriage, divorce or deed poll), details of credit checks/scores, archived information (invoice details, old disputes, inquiries, contracts)
Slowly changing: Relationships between entities (billable service to account, account to customer, customer to customer), address details, service ownership (churn)
Dynamic: Subscriptions, one-time charges, new bills, payments, adjustments, purchases, disputes, pre-paid balances

The type and quantity (volume) of data that falls into each group will differ by business and industry. When a new application is deployed, and data from the legacy application it replaces must be migrated, the business must decide which data will be migrated and the timing of its movement. This holds for higher volume business functions such as billing (e.g. telecommunications) through to relatively low volume examples such as (say) a computer application tracking a local government's town planning requests.

Temporary deferral of customer data maintenance

During a migration event, the customer data selected for migration must not be updated (on the legacy application) once the migration has begun, or the updates will not be reflected in the target application. To support this goal, maintenance of those customers about to be migrated must be deferred for a short, pre-migration period.

If customers' data is migrated in stages (e.g. from static to dynamic), the maintenance of the static or slowly changing data (migrated ahead of its related dynamic data) must be deferred since an 'accurate' copy now resides in the target application's databases. To reduce the system complexity and customer impact of such a staged migration, the elapsed duration of each migration should be measured in terms of a few days or weeks, not months.

Customers are usually not aware of a business' data migrations, and nor should they care. This means that data migrations involving higher transaction and customer volumes have fewer opportunities to modify the business' existing customer service approach, and may only support short maintenance deferrals.

For example, in the case of telecommunication billing, phone calls and new customer connections must be processed until shortly before each migration starts. A local government's town planning department with a smaller 'customer' population and receiving relatively few new requests per day can delay new town planning requests (i.e. 'new' customers and transactions) as they are received until after a data migration is completed.

Tracking migration 'status' to redirect regular processing

A larger application's migration will usually be staged across a number of events (weekends) to reduce the risk of a failed migration, and minimise the impact if a reversal must be performed. Smaller applications may be migrated using a 'big bang' approach, but most others will be broken up into two or more events, staggered over time.

Dividing up the work into manageable segments allows each segment to progress independently along its own migration timetable / path. The speed with which each segment moves down its path can then be based on the quality of its own legacy data, the training status of impacted staff, and the impacts of other events such as public holidays, seasonal workloads and related software upgrades.

A migration register that stores each migration segment's status can be used to delay or redirect regular processing. Segments that have not started the migration process can be flagged for unchanged processing. Segments that have begun to migrate can be flagged with specific behaviours such as processing delays (e.g. through recycling), forced erroring (for later reprocessing) or have their business rules modified to include short-term migration variations.

Once a segment has been migrated, specific field values (e.g. account numbers, phone numbers, market segments) stored in the register can redirect inbound transactions to the correct application (legacy or target). During a migration, some services / customers may error and not migrate, remaining on the legacy application. Details of these services / customers stored on the register can continue to direct inbound transactions to the legacy application.

The register can also indicate whether a migration segment is partially migrated, with static data on the target application and dynamic data continuing to be processed by the legacy application. When a segment is 'partially' migrated, processing on both the legacy and target applications must be sensitive to its status. Processing on the target application must ignore the pre-migrated data in its processing (e.g. extracts, reports) since the legacy application remains the application (database) of record; the legacy application must avoid some processing where it might alter data already migrated.

The static data's pre-migration may be performed concurrent with regular processing on the target application by using the migration register as an exclusion filter for the target application's processing. If the pre-migration is unsuccessful its data can be deleted (removed) with minimal penalty; if pre-migration processing is successful, the static data's maintenance on the legacy system can begin to be deferred / restricted. If the pre-migration is performed alongside the target application's regular processing, a full database backout will not be possible (risk), with the trade-off being a lack of customer outage (reward).

Once all migrations have completed, the migration register and its related processing should be removed from the target application. Assuming the legacy application is decommissioned, its migration-related processing will also be removed. This avoids ongoing, redundant processing complicating or constraining the new application's future operation.

Holding transactions to avoid 'double processing'

Transactions processed in real-time, or near real-time (asynchronously), are reflected immediately against customers' data, and for some applications (e.g. pre-paid billing) must be performed in that way to satisfy their functional purpose. Other processing is performed in batch, perhaps once or twice a day, or more frequently such as (say) every quarter hour to process high volume transaction loads.

During a migration, transactions performed in real-time must either be disabled at their source to stop new processing, or have the details captured offline for later reprocessing once the migration is complete. The decision will depend on the risk / reward of stopping all processing (with its resultant customer impact) versus the developments of a post-migration reprocessing mechanism (with its own risks and development costs).

Where transactions are performed in batch, the (files of) transactions can often be 'held' (upstream) until after the migration event is completed. They can then be released, possibly in stages, and monitored for correct processing. The (now) updated migration register will reflect customers' new location, allowing the held transactions to flow to the appropriate application. For newly migrated customers, this will result in a redirected transaction flow.

Holding batched transactions avoids them being processed first by the legacy application, before being reprocessed by a migration process into the target application's preferred format. This additional (migration) workload would increase the time required to complete a migration event. (The same migration development will be required since transactions captured 'to date' must still be migrated. Holding transactions upstream just reduces the volume that must be processed during a migration event.)

Tags: Data, Migration, System, Application