Provide operational statistics to business users and support staff

Summary

By building a mechanism for reporting operational performance, business users can monitor their application's operational status themselves, and validate any performance and operational defects they uncover. This enables the business to understand the tempo of their application's 'normal' operations on a 'first' not 'third hand' basis, helping to uncover abnormalities early.

The same statistics and reporting will be useful for operational and technical support staff as they monitor and modify the application on behalf of their business customers. Common reporting provides uncontested evidence that can be discussed and evaluated to confirm defect detection and subsequent resolution.

Why provide business users with operational statistics?

Business users use their application (e.g. billing) in an intensive manner throughout their working day and are the people most likely to detect when it is behaving slowly or badly. However, these people are (generally) several degrees removed from direct operational observation of an application's execution. When the system is running slowly, or users believe it is running slowly, they have no direct mechanism through which they can assess their observation. In addition, when there has been an outage, business users often have no mechanism for understanding how close their application is to recovering back to status quo / business as usual processing.

Questions that business users may ask themselves:

Is processing running slowly now or today? (point in time assessment)
Is processing running slower over time? (trend assessment)
Is processing running slower since the last data migration? (trend assessment)
Are my business statistics improving, or declining over time? (trend assessment)
Did the application tuning performed recently improve performance as expected? (trend assessment)
What else was running when there were problems? (problem diagnosis)
Are there any unexplained gaps in processing? (business continuity / assurance)
Does my application's performance varied by hour, day, day of week? (planning / problem diagnosis / consequential impact)

Enable self monitoring by business users

From direct observation or by combing sources such as log files, direct statistics of the running application can be collected on a regular basis, such as intra-day, on a daily/nightly basis, or weekly. Some observations are, by their nature, most useful when reported immediately on an intra-day basis (e.g. work backlog yet to be processed), whilst others may be better calculated over a longer period such as daily (e.g. throughput per hour or day per processing stage).

By presenting the collected details via (say) a web-based interface, users can assess the current operational state of their application from their browser without waiting for their technical or support staff to respond. The details captured and reported will be specific to each system and user community, and can include areas identified as central to the application and/or prone to problems historically. An iterative approach can start with an initial set of measurements, which are then built upon as experience in measuring the application grows.

Enabling business users also enables the application's support staff. Instead of looking across a disparate set of tools, the technical and other support staff can utilise the same reporting and statistics to monitor their application. Their proactive use of these tools can preempt problems before they are identified by business users, but at the very least when the business reports a problem or defect they can share a common and understandable data source in their discussions.

Data spread across multiple systems can be consolidated to form a common view, often a business view. By employing computers to mechanically gather information, support and technical staff can avoid the distraction of these repetitive tasks, and more accurate information will be captured (i.e. subject to less human error) and maintained over time (i.e. as an automated and scheduled activity).

Perform business analysis from the now consolidated source data

Once the business has collected such data in one place, users can extract data subsets, for example as CSV text files, and import them into other tools such as spreadsheets and lightweight databases. Manipulation for reporting, statistical analysis and presentation can then be performed, in addition to trend and 'what if' analysis. By providing the raw data in a reliable format and storing it consistently, business users and operational staff can analyse an application's operations over time.

By also designing an ability to provide continuity of results across planned outages and unplanned downtimes, users will understand the present status and can perform historical analysis without gaps in the data.

Present key fields, but store additional details

Data presented in operational reporting represents selected key fields extracted from a broad pool of available data. Related fields not presented 'on screen' can still be extracted from their source, stored and made available for business and operational users in their analysis efforts. Although this information will require additional effort to access (since it is not presented 'on screen'), the information is captured, consistent and stored in a central accessible location.

The original files used in loading statistics are retained for a defined period of time, so that problems in the data presented can be investigated using the source material, and further analysis can be performed (if required) using the fields not loaded or presented.

Ease-of-use

These operational report screens should be included as part of an application, and be as easy to access and use as the core application's screens, although access to these screens may be limited to certain roles or individuals. By presenting the screens within, and using the same technologies as, the core application, the same mechanisms that allow or disallow access can be employed. By providing information in a format users are familiar with, less training and learning will be required, and the application's technical complexity will be reduced by minimising the number of technologies involved.

When designing a new application, or renovating an existing application to include new functionality, reporting screens that monitor operational performance should be considered. A similar level of usability should be assumed with UI designers, with business users running through scenarios of use to understand how errors, poor performance and normal behaviour would appear, and how the reporting will be impacted by any outages.

During screen design, consideration should be given to the software business users use when performing their subsequent analysis. This approach may influence the formatting of CSV files, using for example pipe '|' instead of comma delimiters. The screens may include related transaction identifiers not used directly by the business, but useful when talking with technical and support staff who do use these additional, domain specific identifiers.

The screen details may be trialled through 'quick and dirty' examples generated manually in formats unsuitable for long-term use, but these can validate that the information to be presented is suitable, available and useful to business and support staff. Initial trials can be performed on an adhoc or daily basis by support staff and emailed to interested parties, but once deemed useful, the development of an automated delivery 'to screen' eliminates the manual effort, staff distraction and additional technology stack required to deliver useful usable information dependably.

Tags: Operational, Statistics, Reporting, Analysis, Screen