Managing outage processing through alternate landing zones

Summary

Interfaces files from upstream applications are commonly received in directories until the receiving application can process them. The generated results of an application's processing are also sent to the next application's equivalent receiving location. These up and downstream landing zones buffer and decouple the data exchange between applications to address planned system outages and peak volume processing.

By creating alternate landing zones, the primary landing zones can be employed by 'business as usual' processing, with the alternates employed during an outage to manage file processing. During an outage, an application's focus can be changed to its alternate landing zones processing only the files placed in its input and producing output to its alternate location, minimising impact to downstream applications during planned outages.

Primary 'Business as Usual' Interface Processing

When an interface file is sent between applications (or indeed across the boundary between organisations), the destination is commonly a specific directory on a nominated server. This achieves the information transfer without needing to identify how or when the file will be used, and allows access limits to be applied on the target server.

The interface files can accumulate in the receiving application's directory (input landing zone) until the application is ready to process them. Similar processing can be performed with individual transactions logged by an application prior to them being processed, though for simplicity, file based processing will be assumed in this description.

When the receiving application has processed its incoming data flows it will commonly produce files for a subsequent downstream application, and in this case will send the output files to the downstream application's receiving directory (downstream landing zone) where they can remain until that application is ready to process them.

Using Alternate Landing Zones during outages

When an outage is performed (product defect fix, project release), the application can be adjusted (e.g. by a software switch) to read files from, and produce files to, alternate upstream and downstream landing zones instead of the 'business as usual' landing zones. All the same application processing is performed, just the source and destination of the application's information flow is changed.

New files will still be received in the primary 'business as usual' landing zones, but now the production team can manage which of, and when, the files are processed by the application through their managed introduction into the upstream alternate landing zone.

With the alternate upstream landing zone starting empty, the processing application will have 'no work' to perform until a file is introduced into its alternate upstream landing zone. Once introduced it will process the file, before waiting until the next file is introduced. The processing of each file can be monitored and evaluated individually allowing for adjustment and business / technical confirmation without being swamped in high-volume or concurrent processing.

After processing each incoming file, the downstream landing zone will now hold the results of processing rather than forwarding them on to the next downstream application (as would occur under 'business as usual' processing). This allows their correctness to be assessed, and for their passage to the downstream application to be performed in a managed way with validation and review.

Similar processing could be performed with transactions that are logged / captured before being processed by an application. A mechanism would need to be built to transfer specific transactions between 'landing zones' in the same manner that individual files can be copied.

Benefits of Decoupling

Upstream files continue to arrive without concerns of them being lost (if 'not backed up') during a change / release deployment. During an outage, only previously 'backed-up' files are introduced from the primary into the alternate upstream landing zone.

Since most batch processing only runs when a file is available in the (primary / alternate) upstream landing zone, this approach can reduce the manual intervention required to manage which processing takes place, and when it is performed.

Management of an outage is simplified by controlling the processing performed and its timing. This can reduce resource contention, and allow business and technical reviews to focus on specific processing as it is performed.

Implementation Mechanism

Step 1: Create 'alternate' upstream and corresponding downstream (if appropriate) landing zones for each upstream interface.

Step 2: Nominate a 'switch' mechanism to drive processing from either the primary or alternate landing zones. This could be as simple as a zero-byte 'trigger' file whose presence indicates that alternate landing zone processing should be employed.

Step 3: Update scripts or other processing to check the switch mechanism (trigger file) and modify where to look for upstream files, or place output files for downstream system. By simply adding the trigger file, 'business as usual' processing will switch immediately to 'alternate' processing, which can be reversed by removing the same file. Nuanced processing could add separate trigger files per interface allowing fine-grained control of which interfaces were impacted.

Operational Processing Flow

In the lead up to an outage, switch to 'alternate' processing.
Perform backup steps to create a restore point should a backout be required.
Backup all upstream files used in managed (validation) processing allowing them to be restored if required.
Manage the release of upstream files from the primary landing zone to the corresponding alternate landing zone. This should trigger the consequential processing and produce any output interface files.
Review the processing performed, intermediate files, databases updated and the output files produced and assess whether to continue or backout.
If a backout is required, restore to the prepared backup point including any files processed during the outage.
If the change is accepted, ensure the downstream alternate downstream landing zone has been copied to the primary downstream landing zone, ensuring all alternate landing zones are empty.
Once the outage has completed, switch back to 'primary' processing through the reversal of the trigger file settings.
'Business as usual' processing will now see any file backlog and resume normal application processing (all other conditions being acceptable).

Tags: Release Management, Planned Outage, Interface