Control Effectiveness
Rating control effectiveness helps to highlight areas of strength and weakness within a bowtie model. This information can then be used to support matrix-based risk assessment and prioritisation.
Results are typically displayed using a colour scale (for example, red for very poor through to green for good). This visual representation makes interpretation straightforward, even for those unfamiliar with the methodology.
When designing an effectiveness scale, consider carefully whether allocating a rating such as “average” adds meaningful value. What does an average control indicate, and how does it help prioritise improvement?
Key Considerations When Rating Control Effectiveness
Adequacy describes the extent to which a properly functioning control will interrupt a specific scenario.
For example, a handheld fire extinguisher may be highly effective in tackling a small galley oven fire, yet largely ineffective in controlling a large fuel spill fire.
Care should therefore be taken when copying a control into another bowtie or scenario, as its adequacy may differ. Other terms sometimes used include validity or impact.
A control must not only be adequate, but also reliable in practice. Assurance refers to the level of confidence that the control will function as intended when required.
Other terms sometimes used include availability or reliability.
Assessment of adequacy and assurance should take escalation factors into account. Before assigning an effectiveness rating, consider:
- The significance of associated escalation factors.
- How effectively those escalation factors are managed.
This structured approach enables a well-informed judgement on overall control effectiveness.
Control Function and Criticality
Control Function
Controls can be categorised according to their function within the bowtie. This provides clarity for users and helps illustrate where operational effort is concentrated (for example, whether most controls are preventative rather than mitigative).
Control Criticality
Not all controls are of equal importance in managing a specific threat. Classifying controls as either standard or critical supports prioritisation.
- Focuses stakeholder attention on key risk controls.
- Highlights where deeper escalation factor analysis may be required.
- Supports targeted assurance and oversight activities.
Standard controls remain essential. Multiple standard control failures in sequence may be just as significant as the failure of a single critical control.
Considerations for Determining Criticality
- If the control were absent or rated “very poor”, would operations need to cease?
- Would its absence likely result in a significant regulatory finding?
- If technical, would its unserviceability represent a ‘no-go’ condition?
These questions assist decision-making but do not mandate classification. Professional judgement remains essential.
Control Ownership and Type
Control Ownership
Allocating accountability strengthens integrated safety management. Each control should be linked to the post holder responsible for ensuring it is properly resourced and maintained.
- Supports integrated safety accountability within the SMS.
- Identifies potential common failure modes across multiple controls.
- Enhances communication of safety responsibilities.
- Encourages collaborative working and cross-functional awareness.
Avoid simply assigning accountability at the highest executive level where possible; identifying operationally responsible post holders adds greater value.
Control Type
Categorising controls by type (for example, technical, procedural, training, supervisory) helps identify potential over-reliance in certain areas. An over-dependence on training controls, for instance, may indicate systemic weakness.
Threat Exposure
Not all threats carry the same level of operational exposure. Differentiating threats by exposure highlights areas requiring greater attention.
A frequently encountered threat may justify higher prioritisation than a rare one, even if both could lead to serious consequences. Stakeholder perspectives may also influence perceived significance.
Example – Manufacturing Facility
- Low exposure threat: Major power failure caused by severe weather. Infrequent, but potentially disruptive.
- High exposure threat: Operator fatigue during routine shifts. Continuous operational exposure.
While both threats warrant management, fatigue may demand greater operational focus due to its ongoing exposure. Conversely, owners or investors may prioritise resilience to power disruption due to financial impact.
Evaluating threat exposure ensures that resources and management attention are directed towards the areas of greatest operational risk.