Note 34: Data Manipulation

Posted: 19 June 2007

Thirteen examples of common data manipulations

The data manipulations performed in mediation are limited only by the biller's creativity, the information supplied by the network and the demands of the downstream systems (including billing). Manipulations may be applied individually or in combination. A selection of different data manipulation steps includes:

Classification: Mediation can use business rules to classify transactions into standard categories allowing downstream systems to treat transactions consistently. For example, phone calls can be classified as being local, long-distance, or international.
Standardisation: A network can consist of different vendors' equipment that supplies the same transactions in different formats. Mediation can standardise the information passed downstream hiding the exact nature of the transaction's source. For example, fixed phone exchanges are provided by multiple vendors, but provide the same service to customers. Mediation allows phone call details to be supplied to billing in a consistent, vendor-neutral form.
Correlation: A dispersed network can record different parts of a single transaction in multiple locations. For example, tollways and mobile phone networks record a transaction in multiple places as a journey or mobile phone call is made. The tollway biller must correlate segments of a journey such as the entry/exit points (e.g. New Jersey Turnpike), or derive the journey based on which measurement points were passed (e.g. CityLink in Melbourne, Australia). The mobile phone biller must collect all the fragments of a phone call and correlate them into one billable transaction passed downstream to billing.
Correction: Network information can contain known systemic errors that are cheaper to correct in mediation than through a network software release. Examples of such errors include transposed phone number digits, time formats or record layouts.
Enhancement: The raw network information can be supplemented with additional fixed or derived information for downstream systems (including billing). These additional fields can be derived from keyed information available within the network transaction. For example, the phone exchange identified in the phone call record could be used in isolation, or in combination with other fields, to derive additional fields such as geographic location (e.g. state), technology type, time zone, or point-to-point distance.
Modification: Fields in a transaction can be modified in a known and repeatable manner for later processing. One example is time zone standardisation. This modification allows downstream systems to operate to a standard time zone without each having to calculate the standard time correctly.
Filtering: The network may produce records that are useful for some downstream systems (e.g. monitoring network load) but not for others (e.g. billing). Mediation can remove records completely, or filter specific records from particular downstream interfaces.
Summarisation: Networks producing interim usage measurements can have those measurements summarised into a consolidated record. For example, interim accounting records measuring data downloaded from the internet may be summarised into one record before it is passed to an ISP's billing system.
Duplicate checking: The biller may choose to validate current network usage against historical usage to check whether it has been processed before. Duplicates can occur due to a network fault or another system resending information accidentally. Duplicate checking can also be performed in the billing system, but can be performed more beneficially in mediation before transactions are sent to multiple downstream systems. A key factor in duplicate checking is the length of time against which transactions are compared. The longer the history, the more transaction data must be held. Duplicate checking slows mediation's throughput, though this can be moderated by only checking specific transaction types.
Sorting: Downstream systems may require or desire transactions in a specific order. To avoid delaying all transactions whilst sorts are performed, post-mediation processing should be considered if only some downstream systems require their data sorted, or different systems require different sort orders.
Duplication: Each transaction may be passed to multiple downstream systems. Each system may need different processing. Mediation can provide each downstream interface with the specific information and format it requires.
Redirection: Mediation may supply records from one network to multiple billing systems based on their customer segmentation (e.g. consumer, corporate, wholesale). Alternatively, transactions may be redirected as part of a migration from a legacy application to its replacement.
Formatting: Outbound interfaces may require fields be supplied in different coding schemes such as EBCDIC, ASCII, CSV, or XML. Mediation can take the same source field value and output it in the appropriate coding scheme.

The correction and standardisation performed centrally in mediation reduces the maintenance effort expended downstream, and the integration effort required when new transactions (and networks) are introduced. The standardised transaction format(s) generated by the collection / mediation function can be reused by other business functions without them needing to understand the idiosyncrasies of the network. e.g. fraud detection, regulatory reporting, network load analysis.

Tags: Billing, Data Manipulation