Full text RSS feed - © Stephen Jones 2010

Column - 31 March 2012

Data or query - sequence-neutral processing for fraud

When a fraudulent customer is identified, it can't be assumed their next attempt at fraud begins once they are caught. When a (fraud performing) person's or organisation's business revolves around the establishment and exploitation of a biller, the customer provisioning / setup phase can be as important as the exploitation phase.

The fraudulent lifecycle should not be assumed as:

SCENARIO A

• Setup up customer 1
• Exploit customer 1 for fraud
• Get detected and terminated by the biller
• Setup up customer 2
• Exploit customer 2 for fraud
• Get detected and terminated by the biller
• etc.

but could instead be:

SCENARIO B

• Setup up customer 1
• Setup up customer 2
• Setup up customer 3
• Setup up customer 4
• etc.
• Exploit customer 1 for fraud
• Get detected and terminated by the biller
• Exploit customer 2 for fraud
• Get detected and terminated by the biller
• Exploit customer 3 for fraud
• Get detected and terminated by the biller
• Exploit customer 4 for fraud
• Get detected and terminated by the biller
• etc.

New customer setup / provisioning can be performed ahead of time and/or independent of the customers' subsequent exploitation for fraud.

Re-evaluating old data and queries when new data arrives

Instead, what if the identification of a fraudulent customer caused a reevaluation of 'recent' customer setup / provisioning looking for 'similar' customers for proactive review ahead of their (possibly fraudulent) exploitation. This process would require that the order in which a 'new customer' was credit / fraud checked and the identification of a new fraud were sequence neutral. If a customer check is performed in Scenario A it relies on the check occurring after the previous fraud to identify a candidate for review by fraud analysts. This breaks down under scenario B when for customers 2, 3, 4, etc. no fraud has (yet) occurred and so no review can be triggered.

An alternative solution would be for the fraud on customer 1 to trigger a review of 'similar' customers 2, 3 4, etc. despite them (as yet) showing no signs of fraud. They could be identified in the context of customer 1 and flagged for monitoring (as fraudulent), as having a higher risk of fraud, or as valid non-fraudulent customers requiring no further action. Such a review could be performed against customers provisioned for the previous period of weeks or months (i.e. whatever metrics indicate is the appropriate period for this activity).

Data retention and reevaluation

The raw data to support such sequence neutral processing would be generated by existing business applications, but the post-processing to perform the sequence neutral checks might best be performed 'offline' from a business application's critical path processing (but still executed 'quickly' to provide value). Any late landing results that triggered 'hits' could be fed to offline staff for near-term review, supported by business processes that take action with any actionable data that it found.

The customer and query data stored for sequence neutral post-processing could be (re)structured to optimise the 'offline' processing performance, rather than the performance of the source applications from which it is drawn.

Further reading - Jeff Jonas

Jeff Jonas identified the idea of 'sequence neutrality' and how it could be applied to data and subsequent queries against that data. He wrote an essay suggesting how the gradual accumulation of data might have triggered a review of the 'Christmas Day' (2009) bomber's US visa. When the bomber's US visa check was performed initially there was no reason to deny him a visa (e.g. query checking for prior fraud), and so a visa was issued (e.g. a service/customer was provisioned by the biller).

Later, additional information was obtained from various sources (e.g. a fraud was committed by a 'similar' customer), that suggested him to be a security risk. The bomber's visa had already been issued, and there was no provision for late landing information to trigger a review or flag the visa for further investigation. The query (visa check) and information (security risk) were not sequence neutral.

Tags: Fraud, Credit, Data, Review

Sign up to receive a brief text email when new purebill postings are published.

Originally posted on purebill.com - Link to column

Note: This should only be appearing on the www.purebill.com website - all other postings of this on the internet are displaying copyrighted material inappropriately.

Full text RSS feed - © Stephen Jones 2010

Column - 23 January 2010

Defend revenue in depth - Credit Assessment, Fraud and Credit Management

Summary

Billers lose revenue by selling to customers who don't pay their debts. This outcome can be through customers' lack of financial means, customers who incur debts they have no intention of paying (fraud), or by allowing customers who run into financial difficulties the opportunity to continue their purchasing. At each point in the billing timeline, specialised systems can assess the financial risk posed by a customer and alert billing staff to take action when problems or 'riskiness' appear.

The revenue impact is doubly important for products where the biller incurs a direct external expense with each customer sale. For example, revenue sharing with third parties for digital purchases (e.g. content settlement for song and movie downloads), and inter-network telecom transactions (e.g. interconnect termination settlements for voice calls and SMS).

Credit assessment is performed at the beginning of the billing relationship, fraud management is performed continuously assessing new transaction between customer bills, and credit management is performed at the end of the billing lifecycle defending the payment of the billed transactions.

Credit Assessment

A biller signing up a new service or customer must assess the 'financial risk' the customer represents. Questions such as:

• How sure is the biller that the new customer will pay their bills?
• Do they have a prior history of bad debt or non-payment (internally at the same biller or noted externally at an agency)?
• What products are they seeking to purchase? How quickly can substantial charges be incurred for those products? Are third-party revenue settlements (costs) involved?
• What (if any) initial limits should be placed on a service/customer? e.g. deposits, billing frequency, spending limits

In addition, the biller may also credit assess an existing customer if their desired new product or service is expensive and therefore presents an increased financial (revenue) risk.

The assessment can performed using a comparison against the biller's previous experience of similar customers (e.g. demographic similarity, product mix, relationship length), and using external organisations such as credit bureaux and reference agencies. By avoiding or placing spending limits on 'bad risk' customers the biller minimises their exposure to incurred revenue (transactions) that will not be paid by the customer.

Fraud Management

Once customers are established, they will start performing transactions that will be collected and placed on their bill. Fraud management assesses customers that perform transactions with a 'normal' count/volume range will be assessed as having a 'low risk' of fraud. However, customers that perform unusually high transaction volumes, or specific high-risk transactions, will be assessed as having a 'high risk' of fraud, and will be escalated for manual review by the biller's fraud analysts.

Examples of high risk / low risk assessments might be:

• a telecom biller may assess a dozen SMS transactions in a day as 'low risk', but one hundred in an hour as 'high risk'
• a telecom biller may assess calls to the USA as 'low risk', but calls to the Caribbean as 'high risk'
• a content provider may assess up to three movie downloads per day as 'low risk', but greater than six downloads in a day as 'high risk'

The assessment of 'fraud risk' is a spectrum from very low to very high. An assessment model that provides a 'risk score' to the biller enables prioritised review by the biller's fraud analysts. With fraud management being a process of continuous assessment, analysts can focus their attention on the riskiest customers (based on their 'risk score') and know they are most likely to find fraud (where it exists), and confirm acceptable behaviour by other customers deemed 'risky'. The assessment model can gain a better idea of an individual customer's normal behaviour the longer the billing relationship exists, and for this reason new customers are often assessed as higher risk due to their lack of prior history.

If a customer is assessed by fraud analysts as having a 'high risk' of fraud, the biller can suspend further transactions and/or request interim payments for debt (revenue) incurred to-date. If the customer is not performing fraud they can explain their transaction pattern and/or make an interim payment. If the customer is performing fraud, their ability to incur further costs to the biller, such as interconnect payments or revenue sharing, will be stopped.

Examples of detected fraud can be updated back into the fraud assessment model so that examples of similar transaction behaviour can be assessed as risky and escalated earlier for manual review.

Credit Management

Once a customer has 'passed' their credit assessment and established their billing arrangement, they will be billed for the products, access and services they have consumed. Assuming a post-paid billing relationship, the customer must pay each bill's debt, preferably by the due date. In a normal billing relationship this cycle is repeated in an ongoing manner for the duration of the customer's relationship with the biller. This repeating process finalises and formalises the recognition of the revenue.

The biller's risk is that a customer will not pay their incurred debt (bill) either intentionally (fraud) or due to poor or changed financial circumstances. There is also the financial impact of delayed customer payments (debt repayment) that places borrowing costs on the biller until customers eventually pay.

Credit management examines customers over time assessing their 'bad debt risk', and modifies how the customer is reminded of their outstanding debt. Reminders for payment sent to each customer can be adjusted to reward (and forgive) an occasional late payment, and to hasten the 'next steps' for customers who habitually pay late.

The riskiest customers, such as new customers who are yet to pay their first bill, or those that incur high bills and delay their payments, can be assessed and reviewed aggressively, whilst customers who pay on time can have their 'next steps' delayed on the presumption that payment will be received (based on historical payment evidence).

Tags: Fraud, Credit, Risk, Revenue

Sign up to receive a brief text email when new purebill postings are published.

Originally posted on purebill.com - Link to column

Note: This should only be appearing on the www.purebill.com website - all other postings of this on the internet are displaying copyrighted material inappropriately.

Full text RSS feed - © Stephen Jones 2010

Note 64: Report Distribution and Access

Posted: 09 January 2010

Independent of their content, reports have common issues relating to their size, timeliness, distribution, retention and access. Each of these issues only grows over time since reports are seldom stopped, new problems will be identified, customer numbers (hopefully) increase, and products and services proliferate. Billers must find reporting solutions appropriate to their size, industry and cost structure that can support and grow with their needs.

Size

Audit reports as a 'group' tend to be large, and increase in proportion to both the number of transactions performed and the number of customers. For larger billers, exception reports can also be large. The size of 'reports' affects the processing performed at all points in their lifecycle. Larger reports take longer to extract from databases, execute report-specific calculations, occupy more storage when generated (and if archived), take longer to print and, if transmitted, distribute.

Each report may be small and simple when taken in isolation, but, taken together, reports can consume a large amount of storage, computer resources, time and money. Reports must also be considered in light of schedules that generate fresh reports each day, week, and / or month.

Timeliness

A report's content and intended use determines when its data becomes out of date. Since reports are historical and reflect a point-in-time, a report once generated degrades in its usefulness as the customer and transaction details it describes change. Each report's retention and usefulness will drive its individual archive and access requirements.

The needs of some report recipients will be more urgent than others. For example, the identification of possible fraud will require prompt delivery, whilst a profitability analysis can be distributed with a short delay.

Exception reports are used to highlight specific events for review and possible response. These reports usually have a short useful period and need to be actioned promptly. Batched processing can make it difficult to obtain reports more frequently than daily because data cannot be 'highlighted' until it has been processed into the billing system. Billing systems that operate in real-time may support reports generated throughout the day, though to be useful staff must be available to action the reports once delivered.

By their nature, audit reports have longer useful periods and can tolerate short delays in their distribution. Financial reports need to be generated 'within days' of the calendar end-of-month, and may be retained as an audit record for years.

Adhoc reports are the least likely to be retained since they often satisfy a transitory, short-term information need. The 'use by' date of their information will depend on the specific circumstances that initiated their generation.

Distribution

A report's need for distribution depends on what its users use it for. Audit reports are usually larger with their users interested in specific sections that cannot be determined ahead of time. Audit reports of financial transactions may have little need for distribution, substituting access to the centralised report data for access at the user's location. Interface files will be distributed according to the technology, format and timetables included in their interface definitions.

Exception reports are more likely to need prompt and complex distribution to ensure information is delivered to where it can be actioned. Prompt access ensures that the report's content is still accurate and any actions timely. These reports can be sent to specific printers at the review team's location, and/or made available online, so that actionable details are available as soon as they are generated. If required, additional copies may be sourced from a central repository.

Adhoc reports are short term and are more likely to be obtained by interested staff from a central repository. Depending on the distribution system used, the effort (cost) of distribution, rather than self-service, is likely to outweigh the benefits.

Reports may contain information used by a team in one location, or they may contain information for multiple teams (e.g. per region, branch, state, product). Reports used by a single team can be sent in their entirety to one location. Reports used by multiple teams must be disaggregated into subsets relevant to each location. This reduces the quantity of report data that is distributed, printed and reviewed, and ensures the details received are relevant. The same disaggregation may be performed when reports are reviewed online by their various users.

Access

Economies of scale often reduce a business' costs. Where a standard technology solution can be implemented across a business, the costs associated with report access, storage, archiving and retrieval will be reduced. A limited choice of access methods can address most staff and regulatory needs. These choices include:

• Access with no distribution: Reports are located centrally where users can access the reports they require. Report sections can be extracted and printed locally.
• Access with distribution: Reports are distributed (and printed or available online) on a scheduled basis to fixed locations, but recipients and other users can access the reports from a central repository.
• Distribution by physical media: Reports held centrally are distributed on physical media such as CD or DVD. Reports, especially larger reports, can be sent to customers without using quantities of paper. Interface files may be exchanged in a similar manner.

Centralised storage ensures that storage costs are minimised (economies of scale), common retention policies are implemented, and authorised users can obtain reports without needing to work through the local gatekeepers of disparate systems.

Internet and intranet based solutions have broadened the available options for accessing and distributing reports. Report information can be retrieved from a central repository using:

• Email: Reports are generated and emailed to intended recipients. Reports can be varied by recipient, and include standard formats such as PDF, HTML, XML and/or CSV. Extensive distribution can be performed using email, including information sent outside the corporate boundary (possibly compressed and encrypted). Distribution changes can be performed centrally and reflected in the next reporting cycle. Report users can print the reports locally (if desired), or view them electronically. Size restrictions on an email's attachments may limit the reports that can be distributed in this way, but this can be addressed using hybrid solutions that download files from internet / intranet / extranet webpages.
• Webpages: Generated reports are made available from a centralised internet / intranet portal. Authorised staff access the portal and download their reports using a web browser. Users may download files that contain the reports (as per email), or view them (in HTML) directly. Large report files may be distributed by sending a report's URL by email, with staff using it to download the report to their location.

Information security becomes more difficult as reports become digital/electronic. Emailed and downloaded reports can be emailed onward to additional recipients. This problem is a matter of degree and ease since reports and business information have always been open to copying and distribution - only the technologies have changed.

Data Warehouses

Statically defined reports provide billers with the information they require, but do not support any further analysis. If a biller's staff observe an interesting trend, they cannot investigate it without considerable effort and reference back to the original billing databases.

Data warehouses are specialised data repositories that hold an 'offline' copy of the biller's customer and transaction data in a format that supports analysis (rather than transaction processing). Billing systems are designed to perform small transactions quickly (e.g. select the billing address of customer A). Data warehouses are designed to support complicated queries efficiently across large data volumes (e.g. select the billing addresses of customers who purchased product C on a public holiday).

Data warehouses allow billers to investigate customer and transaction details using the data's different dimensions (determinants). Additional data attributes can also be applied. For example, a transaction date can be further identified as a Monday, public holiday, in summer, and during the school holidays. These additional attributes allow reports and biller investigations to be performed using a wider range of criteria than available in static reports.

Data warehouse repositories are the end result of specialised processing that extracts customer and transaction data from a biller's different systems, transforms it into standard layouts and loads it ready for analysis. A biller with multiple billing systems can consolidate their reporting information into one location and analyse customers across all their products and services. Analysis can be used to identify correlations, trends and, in more sophisticated use, begin to identify predictive indicators of future customer actions (e.g. churn, termination, bad debt, fraud).

Tags: Billing, Reports, Size, Distribution, Access, Timeliness

Sign up to receive a brief text email when new purebill postings are published.

Originally posted on purebill.com - Link to note

Note: This should only be appearing on the www.purebill.com website - all other postings of this on the internet are displaying copyrighted material inappropriately.

Full text RSS feed - © Stephen Jones 2010

Note 63: Reporting / Business Support

Posted: 04 January 2010

Reports are point-in-time historical views of the billing databases. They can describe transactions performed, highlight important customer statuses, show the results of business analysis, list transactions exhaustively, or highlight specific activities (whether good or bad) that require further investigation. Reports layouts may be generated in a format for human use, or in files suitable for automated processing.

The important steps in the reporting process are:

• Use selection criteria to target specific transactions (e.g. charges, customers, exceptions)
• Extract details about the selected transactions
• Sort the transaction details into a preferred order (optional)
• Total and/or aggregate transactions (optional)
• Format the results for manual review, into an interface file, or format for database storage
• Distribute the results to the intended audience(s)
• Store the results for reference, and possible archiving (optional)

Billers may use specialised reporting systems to perform some or all of these steps. Multiple reports may be extracted and generated at once, and/or separate processes may generate individual reports.

Auditing Reports

Auditing reports list all the transactions performed within a specific domain. By their nature, these reports can be substantial and may not be reviewed in detail by the biller's staff. When a problem occurs, staff may use these reports to investigate transactions of a certain type. Due to their volume, these reports are more likely to be viewed electronically than printed on paper.

Financial reports: These reports justify the biller's financial performance to external groups such as the tax department, stock exchange, auditors and shareholders. Financial reports allow billed details to be traced to summarised financial postings, and each summarised financial posting to be traceable back to its constituent sources.

Details may be aggregated for some report / files where the individual customer details are not required. Aggregated reports and files are faster to process, easier to understand and can be distributed more easily. Examples of financial reports include:

General Ledger: The daily postings of every financial transaction performed in billing including details of new bills generated by the billing cycle, adjustments performed against historical bills and debt written off as uncollectable. These reports will describe both the biller's debt, financial liabilities (e.g. amounts billed in advance) and its revenue.

End-of-Month: Aggregated financial totals calculated during end-of-month processing including a biller's outstanding liabilities, billed revenue, accrued revenue (e.g. charges incurred (phone calls) but not yet billed), and outstanding debt (nett of new bills, adjustments and payments).

Aged Debt: - An analysis of customer debt (credit) broken up into time buckets (e.g. credit balance, not yet overdue (current), 30 / 60 / 90 / 120 / 120+ days overdue). This report identifies and quantifies 'old' debt that can be pursued for payment.

Tax: Tax amounts will be calculated for billed transactions and these amounts may be totaled by tax type, currency, taxed products and services, and taxable jurisdictions. The results of these reports can be used to remit the correct tax amounts.

Interface file generation: All records fulfilling specific criteria may be formed into interface files passed to internal business systems or external groups for further processing. Examples of such files are:

Content settlement: The revenue from billed content may be shared with a biller's business partners. The specific settlement arrangements between the biller and each content provider is combined with the billed transaction data to determine the financial settlements paid. Chapter 28 discusses this topic in further detail.

Customer reports: To win their business, billers may offer their larger customers additional benefits that include reporting. Customer-specific reports need to be handled carefully to ensure that each customer receives only their own reports. Secure report distribution is also required to retain confidentiality. Customisation of customers' reporting increases the running and maintenance costs of billing, but this may be offset by increased revenue and customer lock-in.

Regulatory reporting: Industry or competition regulators may monitor billers by requiring the supply of transaction pricing, service provisioning and service restoration details. These reports may generate industry-wide statistics, monitor code-of-conduct compliance, and/or confirm that a biller's pricing is in accordance with competition policy. Access restrictions may be required to avoid exposing private customer details (if included) and a biller's competitive intelligence.

Exception Reports

Exception reports identify small subsets of transactions, customers or situations that need closer, individual review. Their transaction volumes are intended to be smaller than auditing reports (i.e. exceptional) and targeted at situations 'of interest'. Whilst these reports can highlight positive situations, they are more likely to be used defensively to highlight negative situations. The biller's staff can then focus their (finite) attention on these negative situations that would otherwise be difficult to detect. Billers with customer bases numbering in the millions can only use an exception-based approach since manual inspection is not a cost-effective option.

Depending on each report's purpose, some transactions will be reviewed as acceptable requiring no further action. For the remainder, the biller's staff can initiate an appropriate business process to correct or terminate the identified negative situation (or alternatively reward the positive circumstance).

If too many records are selected for review and subsequently turn out to require no action (i.e. false positives), the selection criteria can be adjusted to better target a more appropriate subset. Examples of exception reports include:

Unpaid accounts: Customers may receive a courtesy phone call, email or SMS to remind them of their outstanding debt and ask for payment. The 'days outstanding' selection criteria may differ by customer segment, or the report may identify the biller's premium customers only.

Unusual high spending: Customers that incur charges beyond a credit limit or above their normal spending pattern may be contacted or monitored. The biller can make the customer aware of their increased spending and evaluate whether fraud is involved to minimise uncollectable debt.

Business accounts to be disconnected: The disconnection process can differ by customer segment to reflect the business impact of incorrect actions. Disconnection of business accounts may be initiated by report to place a 'speed bump' of a manual review before any action is performed. For example, the disconnection of a residential customer may inconvenience one family with relatively modest spending, but the automated disconnection of a major corporation could lose the biller substantial revenue, incur legal fees and compensation claims, and be difficult to reverse manually. A single home service can be reconnected in minutes, but a corporation's 10,000 services would take much longer.

Fraud: Specific criteria may be used to identify activity suspected as fraudulent. These situations can be sent to a specialised group for timely review and actioning. Customers the biller suspects of fraud may have continuous monitoring by report.

Accounts unbilled for longer than X days: This report may be used to ensure that all accounts are billed every (say) quarter. If accounts have been excluded from billing, this report will draw them to the biller's attention to ensure that the cause of billing delay is determined and promptly removed.

Adhoc Reports

The majority of reporting will be performed on a scheduled basis using a static selection of criteria, formatting and distribution. In addition to this base workload, the biller may require the capability to generate additional one-time or short-term reports on an adhoc basis. Often, these adhoc reports will extract information on transactions, totals or customers that cannot be obtained easily using online screens (especially in bulk).

Report proliferation and retention are two of the dangers inherent in adhoc reporting. Proliferation can result when user groups and individuals each generate customised reports that almost, but don't quite, align. Localisation of a common corporate policy can be a source of adhoc reporting used to support the (now) different reporting needs. The biller's business must support the additional operational costs and the confusion of misaligned totals. Unintended report development occurs when adhoc reports intended for the short term are extended in scope and duration until they are retained permanently as long-term reports.

Billers can address each of these dangers by developing processes that either curtail the execution of an adhoc report or moves it to the biller's scheduled reporting suite (along with its purpose, supporting documentation, operational costs, identified business use and ongoing support). Examples of adhoc reports include:

Problem investigation: Reports used to scope problems found within IT systems or processes. Once a problem's cause has been fixed, adhoc reports may be used to monitor the remaining errors as they are fixed and confirm that the solution was correct (i.e. no new examples of the problem are found).

Revenue Assurance: Reports on individual customers may be used to confirm a biller's conformance to its contract terms. This confirmation may be performed after new contracts have been provisioned, when major changes are made in product or service definitions, or when customers raise complaints. The pricing of products and services across all customers may be confirmed through a similar process that compares the prices billed to customers against market offering definitions. Chapter 36 discusses this topic in more detail.

Tags: Billing, Reports, Auditing, Exception, Adhoc

Sign up to receive a brief text email when new purebill postings are published.

Originally posted on purebill.com - Link to note

Note: This should only be appearing on the www.purebill.com website - all other postings of this on the internet are displaying copyrighted material inappropriately.

Full text RSS feed - © Stephen Jones 2010

Column - 20 July 2009

Managing the 'case' of alphanumeric keys and identifiers

Summary

Key alphanumeric identifiers such as email addresses, user names and hardware references (e.g. MAC addresses) are often used as indexes in database tables, and can have problems when their case, that is uppercase, lowercase or mixed case, impacts the application's retrieval of or access to the identifiers. For example, without prior processing, identifiers loaded as their uppercase equivalent values will not be found when searched for using an exact match against a lowercase equivalent. This complication is compounded when identifiers are passed to external/downstream systems, not just retained and used internally.

Three approaches to addressing this situation are:

• Force one case for all stored identifiers, for example by uppercasing them, and search for the values using the same forced case, but at the cost of losing the ability to pass the correct 'case' information to downstream systems.
• Do nothing to the identifier, storing it in its original format passed into the application from upstream, and wear the risk of missed searches due to incorrect case formatting
• Store the identifier in both its original and case forced forms (in a separate (additional) field), supporting both its original case and a standardised form such as uppercased (e.g. used when searching)

Situation

Identifiers such as a user name can be entered in one case, for example mixed case, in the form of 'StephenJones', but later processing might retrieve the same user name using a different case, for example using a user entered lower case form of 'stephenjones'. Users performing searches and lookups are likely to type the full range of upper, lower and mixed case entries, will varying what they type over time, and will expect their data is found reliably.

In addition, identifiers and keys passing through upstream applications may have their case modified without consultation, and these unasked for modifications should not cause the local application's processing to fail.

Problems

When identifiers are passed from upstream into your application's local processing and must be passed downstream or returned to the originating system with their case preserved, your local application must not modify the identifiers' case or the identifiers will not match the originating system's stored values.

Of course, not all up and downstream systems will be case sensitive, but many are and the defensive design presumption should be that future interfaces will be. The problems that appear with case occur when not all systems in an domain ecosystem handle carefully the case of their identifiers.

An identifier's entire ecosystem must preserve and take the same approach to case preservation, or all systems must treat the case as nothing special. If any one system stops preserving case, it is likely to pass on the incorrect case, for example an upper cased identifier to other (downstream) systems, which may then have a problem matching against their stored identifiers.

This is a Yoda situation - do or do not, there is no try. Preserve case completely, or preserve it not at all.

Timing

Case preservation must start from a database's initial population, and generally cannot be retrofitted once field population has commenced. There may be some such circumstances where it is known that all existing identifiers are lowercase or of a certain construction by identifier type, but it is unwise to rely on this being the case across all fields across time.

If database population has commenced, a refresh of the database from an external point of truth may be possible, allowing fields that were initially stored with an incorrect case to be stored correctly. This approach presumes that a consistent and canonical version of the identifier is available elsewhere, is easily passed again to the downstream application, and that the act of receiving such a refresh will not adversely impact the receiving application (e.g. performance degradation, increased storage requirements, network congestion as data is retransmitted, duplicate records presented to end customers or staff).

Searching

An example is a username key stored away as 'StephenJones', but when a user later searches for it simply as 'stephenjones' a simple string search or lookup will fail as the two strings do not match, and the search will therefore be seen as unsuccessful. This will return the 'false' result to the user's search that 'StephenJones' / 'stephenjones' is not present.

An extension of this is to retain both the identifier's original form ('as passed'), such as 'StephenJones', and also store it in a separate field/columns in (say) an uppercased form for use in searching. When retrieved, the original form can used in processing, preserving the original case, and ensuring that identifiers match across different applications storing the same identifier.

By pre-standardising the search identifier, a search will find a string even when entered as 'StephenJones', 'stephenjones', or 'STEPHENJONES'. Where different cased strings for the same identifier type are passed legitimately to an application and must be stored separately, the list of possible matches can be returned and the user can select from them (or an additional field is required to distinguish between them).

Performance Impacts

Whilst command options and special functions can perform case insensitive matching, this functionality can generate substantial and unnecessarily repeated workloads, which will become a bigger problem as database sizes or transaction loads increase. The approach outlined above incurs a small one-time performance hit (case conversion) at the point of insertion, and a modest additional data storage cost, but the long-term processing cost results in an equivalent retrieval cost to the regular lookup and case conversions are avoided. This is a trade-off of space (storage of additional columns and their indexes), for CPU/time (required for case conversion).

The three alternatives outlined at the beginning of this text all perform direct string matches with no additional processing per access, but they have different application / functional trade-offs such as missed matches, or loss to downstream applications of an identifier's original case.

Tags: Case, Identifier, Key, Index, Search

Sign up to receive a brief text email when new purebill postings are published.

Originally posted on purebill.com - Link to column

Note: This should only be appearing on the www.purebill.com website - all other postings of this on the internet are displaying copyrighted material inappropriately.

Full text RSS feed - © Stephen Jones 2010

Column - 14 March 2009

Stretch key dimensions to see what breaks

Summary

An application's design is outlined initially based on the specified business requirements, selected or existing technologies, performance envelope, expected data volumes and the financial resources available to build, deploy and operate it. The solution, whatever it is, will meet or exceed what is asked of it in the contemporary environment and is expected to run successfully (or it is not yet a solution).

Now take this solution and stretch the key dimensions to see what breaks.

This examination looks for limits in the design that will occur when, for example, the system becomes wildly popular and more customers use it, the products being processed increase their transaction counts per day, or six months of data must now be retained rather than the initially specified week. Dimensions are stretched individually and then in combination to tease out the unseen limits that might lie hidden in the initial design.

Stretching key dimensions allows an architect to validate a solution by:

• Understanding whether the planned infrastructure accommodates these increases, and where the limits are. If the infrastructure will break it identifies where it will break which can be highlighted to the application's owner, or the planned infrastructure can be purchased with specific upgrade paths in mind.
• Confirming there are sufficient hours in the day to perform the processing at the expected throughput, with head room to accommodate 'busy days' or 'catch up' after an outage. A solution that cannot process a day's processing in a day and relies on the weekend when things are quieter has no long-term future.
• Validating the data access choices that were made are still valid as the system scales. What might work for when a week's data is held, may be unusable with six month's data loaded.
• Confirming how the application's increased workload will be scaled across additional hardware (if required), and the transition path as the load increases. Working through the transition before the application is deployed can influence the data stored and its structure.
• Confirming the application can still be recovered if the data volumes are increased and/or the data is now split amongst an increased infrastructure.

Based on this examination, elements of the design may be recognised as problems requiring redesign. The redesign will be cheaper whilst the design is still virtual, technical choices are not locked-in and the business data has yet to be stored in the repositories.

by Stephen Jones - This essay was originally posted by myself at the 97 Things wiki and is now available at bookstores including Amazon - 97 Things Every Software Architect Should Know.

This work is licensed under Creative Commons Attribution 3

Tags: Operational, Design, Performance, Scale, Recovery

Sign up to receive a brief text email when new purebill postings are published.

Originally posted on purebill.com - Link to column

Note: This should only be appearing on the www.purebill.com website - all other postings of this on the internet are displaying copyrighted material inappropriately.

Full text RSS feed - © Stephen Jones 2010

Note 62: Maintenance Lifecycle

Posted: 09 March 2009

Each product and service purchased by a customer will undergo a maintenance lifecycle during the lifetime of the biller / customer relationship. Before the customer establishes a billing relationship there will be no billing details stored, after the customer's billing relationship has commenced (many) products and services will be provisioned, and when the relationship ends entries will be end-dated and remain though they will no longer be used by billing.

Maintenance may be initiated using the billing application's own screens, from sub-screens embedded within Customer Relationship Management (CRM) applications, reflect a biller's change to the customer's status (e.g. disconnection for non-payment), or execute remote requests issued by an external customer's systems. The source of maintenance transactions is less important than their actions, and across the lifetime of their relationship with a biller, a customer is likely to be maintained by different mechanisms.

Regardless of source, pre-validation of maintenance requests is performed to confirm that the requested changes are permitted for the customer, that the product combinations are allowed, and that the network can support the changes (where applicable). This validation may be performed by the applications capturing the details or by the billing / provisioning systems when the maintenance request is received. If errors or problems are detected these will be identified allowing the staff or customer to change the offending details and resubmit. Where pre-validation is performed outside the billing / provisioning systems (e.g. in CRM systems), the same validation rules must be employed to avoid valid maintenance from being rejected, and to avoid the staff / customer frustration of maintenance requests that pass external pre-validation but are then rejected when passed to billing / provisioning.

Maintaining a customer's product and service data is tightly aligned with the provisioning process. New connections, maintenance and terminations / cancellations may be reflected in the network and/or in billing. The complications of provisioning are also reflected in billing's maintenance lifecycle, including the need to keep the details of the network aligned with customers' billings.

The majority of the maintenance lifecycle is concerned with the commencement of recurring charges, rate/discount plans and other long-term customer structures. Non-recurring (one-time) charges may also be provisioned as part of this process (e.g. installation charges). Usage charges will be based around actual network use rather than customer provisioning / maintenance.

Capturing customer details

The establishment of new customers must capture information that is independent of the products and services sold. Examples include the customer's identity (name, identity numbers (where applicable)), where to send their bills (postal address, email details), which media is preferred (paper, internet portal), bill format (detailed, summary, sections), customer segmentation (generated by the biller), their initial credit rating/score (if applicable), payment preferences (direct debit, pre-paid, post-paid), and language preferences (Braille, Large Font, French).

Existing customers may have specific details confirmed (e.g. contact details) when new purchases are made, though credit checks may be performed before high-value items are purchased to reconfirm the customer's ongoing creditworthiness.

The details stored for individuals will be different to those stored for companies. Date-of-birth and details of personal ID numbers (e.g. US social security number (SSN), drivers licence number) may be used to identify an individual during the identification process. Individuals may also act as an authorised representative for another customer. For example, a customer may have their own phone service, and also be an authorised representative for their elderly parent's account.

Companies will have different means of identifying themselves (e.g. Australian Business Number (ABN)) and may have corporate relationships between entities that must also be captured. Details of related entities can be used when assessing a corporation's total purchasing with the biller, or, if their bills remain unpaid, which entities must have their services suspended or terminated.

Add / Install / Provision

Each of the customer's billable products and services must be established in the biller's network and / or added to the billing system. There may be service features installed in the network that are not charged for (e.g. call waiting), and there may be billing-centric constructions (e.g. discount plans) that do not affect the network. Biller-specific business rules determine whether the network and/or billing is updated.

The installation of bundled offerings presents additional challenges since different products and services, using possibly unrelated networks, must be installed concurrently. The biller must determine the billing rules that apply when a bundle is only partially provisioned.

The challenge of correctly provisioning services is often greater for the network than for billing. In these cases, successful network completion can be used as the trigger to forward installation details to billing. The order of processing can also be reversed to use the successful update of the billing system to drive provisioning in the network.

New network connections are likely to be sent to both the network and billing systems. The information included in new service connections can include parameters (determinants) used when determining the charges' correct rates. For example, a customer may be charged a different rate depending on a new communication line's maximum speed.

Billing-centric constructions, such as discount and negotiated rate plans, are sent to the billing system because they affect how the customer is charged for their network access and use, rather than the operation of the network connection itself. In their respective networks, phone calls will still be connected and water will still flow regardless of the prices charged for each.

The division between these two areas can blur when billing functions are performed in the network. Billers offering pre-paid services, such as mobile phones, may require rating (and taxation) be performed prior to network use. Billers may perform these billing functions within their network to achieve the required performance and availability, and avoid the revenue losses that can occur with delayed balance adjustments. In these cases, new customer rate plans can require change within the 'network' to update entries held within the pre-paid billing functions.

Delays in providing details to billing can result in back-dated processing in billing. Whilst the biller's systems may be capable of handling billing maintenance in real-time using the current date, and most transactions may be performed in this manner, delays by external parties, internal system problems or error processing can generate entries with 'historical' dates. If accurate commencement dates are important to the biller, their systems must be able to handle backdated transactions, even if the date to be processed is only yesterday. Some billing systems, built around an assumption of real-time processing, can find it difficult to perform processing for transactions in the 'past'.

Non-recurring (one-time) charges are candidates for such historical processing. Charges may be incurred by business partners without real-time access to the biller's billing platforms. The charges passed are correctly dated when they occur, but may be in the past on their arrival at billing.

Update / Change / Modify

Once established, products, services and customer details will need to be modified to reflect ongoing changes in the customer's needs. For example, customer details such as their mailing address, billing hierarchy or negotiated rates may need adjustment without canceling the underlying network services.

The biller may segment the staff who perform different types of customer maintenance. Changes to the customer's billing address may be performed by a wide range of (authorised) staff, whilst rate and other complex changes may be limited to specialist groups.

The methods and limitations on maintaining customer details will differ by biller, and possibly by billing system where more than one exists within a biller. Whilst most changes may be effective dated to provide an audit trail, the biller may decide that some customer details can reflect only the latest value. For example, the customer's outstanding (debt/credit) balance may be stored without effective dating.

Maintenance performed on some customer data can indirectly modify processing performed elsewhere. For example, if a customer uses 'inheritance' down through a billing hierarchy to specify a rate plan to services within the hierarchy, changing 'where' a service is located in such a hierarchy may change which rate plan it inherits and uses to calculate prices (rating), and without changing any other property of the service itself.

Cancel / Terminate

All that has been provisioned against the customer will one day be canceled or terminated. Each product, service, discount plan, and rate plan will be end-dated reflecting its individual end-of-life, or the end of the customer / biller relationship. This is the case for the long-term, recurring structures rather than usage or non-recurring charges which are one-time and billed as they occur.

When special discount plans and negotiated rate plans are canceled whilst retaining an active network service, the customer may revert to the biller's standard rates. Such a cancellation is billing-centric and may not be reflected to the network. However, the cancellation of a network service will generate changes in billing to end-date the corresponding services entry and end any associated recurring charges. The cancellation of the service in billing may generate a cascade of cancellations 'as-of' the service's end-date for related pricing structures such as discount and rate plans.

Non-recurring charges may be generated by the cancellation process where pricing structures such as bundle and contract offerings contain early termination penalties, or where levies are charged for biller-initiated disconnections. The dates of these additional charges will need to be chosen to ensure they fall within the effective (start and end) dates used to extract charges for the bill cycle.

The accumulated cancellation of products and services can lead to the situation where no active services exist on an account, and / or no active accounts exist for a customer. When a customer's account has no active services remaining (and no source of future charges), the biller may invoke business rules to cancel and finalise the account, generating a final bill. This housekeeping activity reflects the reality of no future activity on the account, enables any outstanding debt or credit to be finalised, and allows the biller to avoid the ongoing costs associated with generating empty, zero value bills.

Tags: Billing, Customer, Service, Product, Add, Change, Delete

Sign up to receive a brief text email when new purebill postings are published.

Originally posted on purebill.com - Link to note

Note: This should only be appearing on the www.purebill.com website - all other postings of this on the internet are displaying copyrighted material inappropriately.