Withhold and filter transactions to minimise operational impacts

Summary

For business applications receiving transactions across interfaces, a (temporary) withholding of transactions can be beneficial in situations such as:

When upstream systems produce invalid field values or superfluous events (data errors)
When (processing) defects are found in downstream applications (software errors)
When sales of new products are made before the matching business processing is ready (timing issues)
During migration, to reduce the customer-related data that must be converted (minimising the reprocessing workload)

By deploying a withholding / filtering function, business and operational impacts can be addressed whilst minimising change to upstream and downstream applications.

Filtering transactions: What's involved?

The need to filter transactions stems from the operational fact that things go wrong, and it can be easier to remove transactions temporarily from processing whilst system / application issues are worked through. In some cases, the removed / filtered transactions may be unprocessable and must be removed permanently and written off due to timing or data issues.

A filtering mechanism can be introduced in many ways. One is to place the filter mechanism at the beginning of an application's processing (as a preprocessor) to prevent transactions from being accepted into the application. This allows the filter functionality to be 'shared' across a diverse range of upstream transaction sources. An alternative location is at the end of an application's processing before transactions are distributed downstream (to possibly many systems). This location can also allow transactions to be held, delayed or removed depending on the processing context.

Filters may be introduced for the long-term, or to address short-term needs before being removed (e.g. during a system/application migration). Four areas to be considered when introducing a transaction filtering mechanism include:

Selection Criteria
Transaction Reporting
Write-off and Removal
Release Mechanisms

Selection Criteria: Since not all transactions will be removed from a processing stream / interface, the criteria must be versatile enough to target only specific transactions and allow the remainder to pass through unhindered. The selection criteria will revolve around identifying aspects of a transaction's 'who', 'what' or 'when'. 'Who' can describe a customer segmentation impacted by a downstream problem or processing constraint. 'What' can outline the particular transaction types to be targeted. 'When' allows only transactions for a particular date range, day of week or time of day to be targeted. The three criteria can be combined to identify filtering for particular transactions on particular customers.

When an application processing problem occur it may impact only a segment of the customer population ('who'), impact only specific transactions ('what') or be a problem only for a specific time period ('when'). The specific criteria will be very sensitive to each problem's context, and will probably require knowledgeable staff to define the correct and minimal criteria. Minimal criteria are important to avoid delaying unaffected transactions and the consequential impact to customers and a business' revenue flow. If a filter is being used during a system migration, the criteria may be well defined, predetermined and aligned to the migration's segmentation instead of a particular system problem.

The selection criteria must select transactions in a manner that minimises impact on the application's end-to-end performance. This concern may affect whether the criteria are implemented via a general purpose solution or a custom solution that is suited for one context only (e.g. during migration).

Reporting: Reporting on filtered transactions helps business and operational staff understand how many transactions have been filtered (impact), how long they have been held (delay), their financial value (for estimating the billing impact or an EOM accrual allowance), how quickly they are accumulating, and against which selection criteria ('filter rule') they are being held.

This raw information can help management prioritise how each selection criteria's situation will be addressed. For example, if there are a number of software defects downstream, high-value and/or high-volume problems may receive an increased resolution priority, problems with older transactions may be processed earlier (to avoid write-offs associated with transaction aging and minimise customer impacts), and migration-related filtering can be assessed for its revenue and cash-flow impacts.

Write-off and Removal: Some transactions will not or cannot be released for processing and must be removed and discarded. This step must be considered carefully since transactions represent revenue that will never be recognised and/or customer related processing that will never be performed.

If transactions represent revenue, reporting is desirable so that the financial impact can be understood and noted. If transactions represent customer requests, then the customer may need to be notified or staff made aware of the impact so that inquiries can be answered appropriately.

Examples of why transactions may be removed include their instigation for internal use (testing products before public release), age (delayed and now too old to bill), data problems that cannot be corrected (invalid field values, unknown customers), upstream system problems (duplicate transactions generated in error), and marketing / sales errors (product withdrawn from sale, product trials unsuccessful).

Release: Eventually transactions will be released once the initial circumstance that caused them to be held are resolved. When a high-volume of transactions must be released, consideration should be given to the impact on downstream processing of a surge of pent-up workload.

A staged release of transactions will allow normal processing to continue along with the released workload. The order in which transactions are released can use different criteria to increase the business benefit of the release using an approach such as the oldest first (to avoid aging write-offs) or release those 'about to bill' (to maximise the cash flow impact).

Transactions may be held in a database (e.g. RDBMS table) or stored in files that are recycled (with new transactions added as they appear). The release of transactions may be performed by a staged removal of selection criteria, or, where that will generate excessive workload downstream, a gradual relaxing of the criteria will release held transactions in a managed way.

Examples where Filtering can be applied

The implementation of a filter in the circumstances outlined below may be reactive to unexpected problems or proactive as part of a planned deployment (e.g. migration processing).

Data Errors: An upstream system, or its upstream predecessor, can produce malformed transactions with invalid field values, or generate unexpected and superfluous transactions that shouldn't be processed. Field values may be considered in error because planned updates have yet to be applied downstream, a software change had unplanned effects, or equipment has begun malfunctioning.

This application of filtering is reactive and used as a 'harm minimisation' technique. Often some transactions have already been passed to the downstream application causing problems and the filter is placed to reduce the ongoing impact of the data errors. If these transactions continued to be passed downstream they might cause system crashes, overwhelming error processing, produce misleading reports, or cause unexpected processing results.

Some transaction data errors may be correctable and downstream applications can be updated to receive the new field values before filtered transactions are released. Others transactions will be selected and removed permanently because they are either unwanted (duplicated transactions) or too damaged to recover.

Software errors: When processing defects are found in an application, the problem might be limited to specific customers or products. The transactions themselves are accurate and well formed, but when processed by downstream processing, undesirable processing takes place. This application of filtering is also reactive and used as a 'harm minimisation' technique. Some defective processing has already taken place and will require remediation, but by reducing new errors (i.e. additional transactions), the 'clean up' effort can be minimised.

Once the software defect has been resolved, the filtered transactions can be released and normal processing can resume. Depending on the defect's impact, the filtered transactions may be released after the software is corrected but before the errored transactions are fully remediated.

Timing issues: New products may be developed and released for public sale ahead of the business system changes that support them. This approach allows a business to capture early sales in the market, or react to competitors faster than the reacting business can change their internal systems.

Transactions once sold will begin to flow towards the impacted application(s), which can filter them aside until the necessary functionality has been updated and is ready to process them. This approach is sometimes used when new products are offered 'free' for an introductory period before being charged at full price. During the 'free' period updates to the business' systems may be performed and finalised before being used to charge end customers.

Minimising migration workload: Often during migration, a customer's (unbilled) transaction data must be converted to the new system's format. During a migration event this can create a substantial processing workload in industries such as telecommunications.

An alternative that reduces the volume of customer-related data to be converted is to hold the transactions for 'soon to be converted' customers upstream and only release them once the migration has been performed. The converted customers' released transactions then flow downstream to their new target application, are created in their new format (i.e. the holding system can create either old or new format depending on the destination), and processed without the need for a 'conversion'.

Archive column: How to passively migrate transitory data

Tags: Billing, Filter, Withhold, Transactions, Release, Migration, Manage, Problem Resolution