Using A/B testing in billing processes

Summary

Billers can use A/B testing to improve their billing processes by measuring the actual behaviour of their customer base. By using actual rather than guessed customer behaviours, changes can be validated to achieve improved results.

As well as ongoing A/B testing on new functional / customer interfaces changes, a biller's customer population can also change invalidating historical results. A/B testing measures customer behaviour and as a business becomes more successful, customers' behaviour can change along with the demographics. For example, a customer base comprised of 'tech-savvy' early adopters may behave differently to a later snapshot containing a broader cross-section of the general population.

Updated 09 Nov 2005: Web magazine 'A List Apart' provides a specific A/B testing example and its results. (Link to text at end of column.)

Updated 16 Dec 2006: Google's example of A/B testing in advertising. (Link to text at end of column.)

What is A/B testing?

A/B testing, also referred to as champion / challenger testing, divides a customer (user) population into two groups and assesses their response to two similar, but different, interfaces, questions, tasks, offerings, or products (i.e. 'A' or 'B'). Each customer is randomly assigned to one of the groups and their reaction / behaviour is measured. Whilst any one individual's responses is of low importance in the testing, aggregating the responses of a 'large' customer population can indicate a statistically significant trend.

The benefit (or not) of a new approach ('B') over the incumbent approach ('A') can then be assessed. If the new approach is 'better' then it may become the new incumbent, and the process can be repeated.

A/B testing can provide a statistically valid measure of which approach works better, or that both work just as well. The testing does not rely on someone's pet theory or an individual's best guess - it is based on the hard facts of customers' actual actions.

Jakob Nielsen's column highlights how A/B testing can test website usability, and improve customer responses to actions such as: purchasing products, signing up for newsletters and providing contact details for subsequent use by a sales department. Historically, A/B testing has been used in direct mail advertising to improve and continuously test the effectiveness of the messages advertisers send to their customers.

How can A/B testing be used in billing?

Billing has many processes with measurable results that can benefit from continuous improvement including:

Credit Management
Customer Churn Reduction
Online Advertising
Bad Debt Recovery

Multiple tests can be performed at once by segmenting the customer base into different market segment / customer profile groups and testing each group concurrently. For example, a market segment may identify customer sub-sets such as VIP, new customer (i.e. with little historical data), long-term good customer, late-paying customer, and customers with multiple network services. A biller's segmentation will depend on what makes sense for their specific customer base, market segmentation and industry.

The ability to test concurrently will depend on a biller's ability to design new approaches (i.e. new 'B' options), deploy the tests across their customer base, monitor tests as they are performed (in case of a strong, adverse reaction), and interpret the results once testing is complete. Testing can be performed across multiple customer segment, profiles and business processes concurrently. The specialised skills used in designing statistically 'valid' tests and assessing the results could be shared across a biller's different processes. These (mathematical, statistical) skills are different to those of interface, useability, collection and product design.

Credit Management - Customers who do not pay their bills by the due date will be contacted to remind, hint at and then threaten disconnection before finally disconnecting service when payments are not made. The measurable goals in credit management could include: the elapsed time until full payment is made (an important one), the number of customer contacts required (costs, recovery time), customers' future payment behaviour (better or worse), and the total cost to receive payments (comparing different combinations of contacts).

The collection actions and their timing can be varied (tested) to find the best combination (by customer profile) that encourages customers to pay on time (best) or shortly after being contacted (next best). The cost of each contact must also be included when determining the 'best' combination.

For example, the timing of an early reminder notice could be delayed by one or more days for prompt payers to see if customers will pay on their own, or advanced by a day or more for habitual late payers to encourage their payment. Billers will save money when redundant actions are avoided (e.g. a forgotten payment by prompt payers), and will reduce / avoid bad debt when payment will never be made by quickly moving through the entire credit management process.

Customer Churn Reduction - A biller's analysis of their customers' behaviour may generate predictions of which customers are more likely to churn. Billers may use these predictions to contact their customers and try to retain their business. The contact methods used (mail, phone call, email, SMS), timing, the sales messages given, and product offers used could all be subjected to an A/B testing approach.

By segmenting their customers and approaching them directly, billers may uncover less costly customer retention solutions. The biller's existing approach could be tested against multiple options ('B', 'C', ...) if the customer population was large enough to support statistically valid results.

Online Advertising - Customers performing their own customer care / provisioning through the biller's online portal may view advertising for the biller's other products. A/B testing could evaluate the sales effectiveness of the advertisements' specific products, style (text, picture), page location and price points.

Products: Which products do customers respond to (buy)? Does their response vary depending on their existing or historical purchases? Does 'location' and customer segmentation change the results?
Style: The biller could test whether advertisements based on text, static pictures, or moving images were best.
Content: Which advertising words do customers respond to best?
Location: Which locations on the web page achieve the best results? If presented beside a customer's online bill, does the success vary by page type (service summary, charge details, capturing an online payment)? Do different products work in different locations, or are some locations always poor performers?
Offers: What are the price points and product combinations that attract customer purchases? Does the timing of an offer (morning, night, weekend, EOM) change the success rate?

Bad Debt Recovery - A customer who does not pay their outstanding debt will eventually be disconnected. The biller may then pursue the outstanding debt, either performing the recovery themselves or through specialist debt collection agencies. A/B testing could be applied to test the allocation methods used to direct recovery cases (with different customer profiles) to collection teams / agencies.

Testing could assess which specific teams / agencies recovered the most debt, their recovery time, recovery costs, and whether these factors changed over time (through a competitive testing program of the incumbent solution).

-- :: --

:: Updated (09 Nov 2005): The web design magazine 'A List Apart' provides another example of A/B testing. In the article, Nick Usborne describes how two new versions of an existing website were developed and tested. The reader is asked to guess the outcome of the testing before the results are revealed. If the testing had not been performed, the website in question might have deployed the 'preferred' version and incurred a substantial drop in sales...

:: Updated (16 Dec 2006): Google outlines how to use A/B testing to improve the results of advertising performed through its publisher and search networks. A javascript template is included so that two ad units can be compared for effectiveness based on a random allocation of ads per visit.

Tags: Billing, Testing, Champion, Challenger