Risk Matrices and Heatmap FUD

There is much fear, uncertainty, and doubt (FUD) regarding risk matrices that express risk attributes as High, Medium, Low, which are assigned values like 1, 2, 3. The detractors contend that it constitutes an invalid, nonsensical method of measuring risk. Related FUD discourages the use of heatmaps, claiming ambiguous results further proving their case that these methods should be wiped from existence. If that sounds like a strong-armed sales pitch, you are not alone.

First we have to contend with a handful of academic papers asserting that risk matrices are “worse than useless” and that a matrix approach leads to random decisions. These papers and the blogs that cite them establish a very narrow scope and specific assumptions, which is the first clue that the results will be qually narrow and assumptive. These scope limits and assumptions may or may not exist in the real world applications. Chief among these assumptions is that the values assigned in a risk matrix are arbitrary and subjective. Perhaps, but no more so than any other method of eliciting expert data. And the papers that rail against risk matrices as being based in subjectivity never take the same critical view of other expert elicitation methods. Meanwhile,reputable sources like NASA, the DoD, and the nuclear industry offer time-tested proof that risk matrices are effective tools to identify, manage, and reduce risk.  Like all tools, when properly used they can be very effective.  Misused or fed bad input data, all tools can lead to disaster.  

Unfortunately an echo chamber has formed in the cyber risk management community that continues to cite these anti-matrix papers as irrefutable fact. And trashing risk matrix models as a way to demonstrate the superiority of allegedly pure quantitative approaches is all the rage in some circles.  As marketing tactics go, this echo chamber is a major win.  But for advancing the field of cybersecurity or making cyber risk analysis accessible to a wider audience, it serves no purpose.

Another pervasive argument against risk matrices is rooted in the rules of descriptive statistics which states that categorical, nominal data can be counted, but higher math such as multiplication and division is not allowed. The argument goes:

  1. High, Medium, Low are categories.
  2. Assigning 1, 2, 3 in their place just swaps a number for a word, so it is still categorical.
  3. The rules of descriptive statistics states that math, except for counting, is not allowed on categorical data.
  4. Conclusion: The methodology is nonsensical.

This all seems quite logical. But the devil is in the up-front assumptions. In this case, steps 1 and 2 are where the assertion is made that Arabic numbers are being used as category labels. But in most cases they are being used exactly as they were intended, which is to represent an actual numeric measure. And you may recall that Arabic numbers were invented as a shorthand notation for sets and set math. So, on the whole, a risk matrix is no better or worse than using, for example, a Beta-PERT[1] as short-hand for many probability distributions. When it comes to subjectivity neither approach offers any safeguards against garbage-in garbage-out. The truth is that each method has pros and cons. And it should be noted that many modified Beta-PERT implementations even provide pre-set lambda values labeled as High, Medium and Low as inputs.

Rest assured that High, Medium, Low labels work just fine in the real world when they represent set math. And the equivalence property offers yet another shortcut to expressing probabilities as ratios, decimals, percentages, and even integers for easy math. As mentioned previously, Arabic numbers are nothing more than symbols for an underlying set. The set of one thing { | } is 1 and the set of three things { |, |, | } is 3, and so on. But the choice of symbols, be it Arabic numerals, binary, hexadecimal, or the word “High makes no difference.

It is important to note that the underlying values of High, Medium, Low are quite often more complex than the integer and decimal examples given here. Often, they are very precise ranges taken from engineering, finance, etc. For brevity, the examples that follow stick to simple releases, though we provide an example of using ranges when building our heatmap legend.

Let’s examine the mechanics of the math behind a relative assessment of risk, in which the goal is to rank order a set of risks relative to one another. To begin, here is the symbolic representation of ten coin toss observations using Arabic numbers and sets.

Sample Set

p(Heads)

p(Tails)

Set

Integer

Integer

Set

Integer

Set

 | | | |   | | | | 

10

1

|

9

| | | | | | | |

 | | | |   | | | | 

10

2

| |

8

| | | | | | | |

 | | | |   | | | | 

10

3

| | |

7

| | | | | | |

Now let’s look at set and symbolic representations of p(Heads | Tails) with decimal, integer, and percentage equivalence, which are shown below. For simplicity we are only defining three bands. But there could be 10 or 100, if you so chose.

Symbols

Equivalence

Symbols

Set Ratio

Numeric Ratio

Decimal

Decimal X 10

(Integer)

Decimal X 100

(Percentage)

Word Label

| :  | | | |   | | | | 

1 : 10

0.1

1

10%

Low

| | :  | | | |   | | | | 

2 : 10

0.2

2

20%

Medium

| | | :  | | | |   | | | | 

3 : 10

0.3

3

30%

High

And here are set and symbolic representations of Consequence.

Set

Integer

Word Label

|

1

Low

| |

2

Medium

| | |

3

High

And here are the set and symbolic representations of a heatmap legend, again depicting equivalence. We use compex ranges rather than simple values. Nothing wrong with that.

Set Ratio

Integer

Decimal

Word Label

⊆ | | |

<= 3

< 0.3

Green

⊄  | | |  ⊂  | | | |  | |

> 3 or < 7

> 0.3 or < 0.7

Yellow

⊇   | | | |  | |

>= 7

>= 0.7

Red

Set notation: ⊂ subset, ⊆ subset or equal, ⊄ not a subset, ⊇ superset or equal

And here is the underlying math with Arabic integers, decimals, percentages and sets. All valid. All arriving at the same relative result due to equivalence.

Risk

Likelihood

Consequence

1

Green

Low

High

2

3

=

1

X

3

3

0.3

=

0.1

X

3

4

0.3

=

10%

X

3

5

 | | |

=

|

X

| | |

Here is another example. When comparing this risk with the one above, we clearly see it is larger and we can reliably rank order an entire list of risks using this approach. And, because of equivalency, it doesn’t matter if we use integers, decimals, percentages, or sets to arrive at the answer.

Risk A

Risk

Likelihood

Consequence

1

Red

High

High

2

9

=

3

X

3

3

0.9

=

0.3

X

3

4

0.9

=

30%

X

3

5

 | | | |   | | | |

=

| | |

X

| | |

Let’s do the math again for Risk B

Risk

Likelihood

Consequence

1

Yellow

Medium

High

2

6

=

2

X

3

3

0.6

=

0.2

X

3

4

0.6

=

20%

X

3

5

 | | | |  |

=

|

X

| | |

And again for Risk C

Risk

Likelihood

Consequence

1

Yellow

High

Medium

2

6

=

3

X

2

3

0.6

=

3

X

0.2

4

0.6

=

30%

X

2

5

 | | | |  |

=

| | |

X

| |

But, wait! How can those last two calculations convey anything of meaning? Surely, two completely different risks, one with high likelihood and low consequence and another with exactly the opposite characteristics, should not be the same. This is another commonly encountered argument against risk matrices that deserves explanation.

When analyzing relative risk, the results make complete sense. Likelihood, in this case, is a measure of frequency. A higher frequency event that produces lots of small losses presents the exact same investment problem as a low frequency event that produces a large loss. Time-value of money issues aside and using dollars as our measure of consequence, the risks are financially equivalent.

But the detractors are doing an apples to oranges comparison. They are examining relative results in the context of an absolute risk assessment. Unlike relative risk, absolute risk is the measure of a single risk that is independent from that of other risks.

Absolute risk lies at the heart of consequence analysis and vulnerability analysis. Consequence analysis assumes the probability of occurrence is 1.0 or 100%. Using a more sophisticated expression of probability, as opposed to a simple binomial distribution, easily depicted as red and green, is mathematical overkill. The value of risk will always equal the consequence because the assumption is that the probability distribution collapsed to 1.0.  If we rework the math of the last two examples, we find that a matrix approach correctly differentiates between them.

Absolute measure of Risk B

Risk

Likelihood

Consequence

3

=

1

X

3

Absolute measure of Risk C

Risk

Likelihood

Consequence

2

=

1

X

2

In contrast to consequence analysis, vulnerability analysis compares two absolute measures of the same risk to one another. The goal this time is to identify which of two mitigation alternatives should be selected. So let’s step through a simple example using a risk matrix.

Mitigation A reduces the likelihood of an attack by one third.

Risk

Likelihood

Consequence

Yellow

High

Medium

1

6

=

3

X

2

2

Mitigation

-1

+0

Yellow

Medium

Medium

3

4

=

2

X

2

Mitigation B reduces the consequence of an attack by one half.

Risk

Likelihood

Consequence

Yellow

=

High

X

Medium

1

6

=

3

X

2

2

Mitigation

+0

-1

Green

High

Low

3

3

=

3

X

1

Comparing the outcome of mitigation A and mitigation B, the better choice is B. No math violations occurred. It is easy to understand and easy to communicate.

In summary, complex probability distributions, Beta-PERTs, and elaborate Monte Carlo simulations certainly have their place in analyzing risk, but so too does the simple risk matrix. And, as a tool for quickly communicating risk posture, it is hard to beat a simple risk matrix and heatmap. Let’s not be too quick to dismiss them from our tool set … unless, of course, you are selling an elaborate model. 😉