Decision Design: How do you create the infrastructure to make decisions that people trust?
What’s the best way of improving decisions? If you had asked me about a year ago my main answer would have been “get more information”! The better informed you are, the more likely you are able to effectively weigh trade offs involved in a decision and come to a better result.
However, over the last year one event really highlighted to me that more information is rarely enough to ensure good decision making.
It took place on 30th September of last year.
Liverpool and Tottenham were playing a Premier League game against each other. Liverpool scored a crucial early goal that could have been a clear turning point in the game. However, after consulting the Video Assistant Referee (VAR), a group of referees who sit in a separate room and watch video footage to help the on pitch ref, the goal was disallowed because the player was offside.
Now, I’m a Liverpool fan and as is the case with every supporter, I was outraged by this (obviously incorrect) decision made unfairly against my team.
However, this time was a little different. When they replayed the footage again and again, a friend sitting next to me (who was a Tottenham fan) also seemed confused at the decision. You could see people around me in the pub were confused at the decision. You could hear the commentators confused at the decision.
Universal confusion at a refereeing decision is a fairly rare phenomenon in football. In general fans are outraged when decisions don’t go their way and the decisions are deemed as “obvious” when they do. As a result almost every decision leaves half the fans content and half the fans upset. Rarely is everyone confused. In this event, universal confusion signalled that the decision was pretty clearly and objectively incorrect.
After the match finished, the heat surrounding the decision to disallow the goal didn’t die down. Shortly after the game the PGMOL (the Premier League refereeing body) released an apology - “PGMOL acknowledge a significant human error occurred during the first half of Tottenham Hotspur v Liverpool. The goal by Luiz Díaz was disallowed for offside by the on-field team of match officials. This was a clear and obvious factual error and should have resulted in the goal being awarded through VAR intervention, however, the VAR failed to intervene. PGMOL will conduct a full review into the circumstances which led to the error.”
This statement led to more questions than answers - the whole point of VAR was to prevent clear and obvious factual errors from taking place. The apology came quickly and was remarkably full throated, which was not typical of the PGMOL.
Over the next few days, PGMOL were forced into releasing the full audio of what went behind the scene of the game. You can read the full transcript and audio here - but here are three main takeaways:
There was a miscommunication where the Video Assistant Referee thought the decision on the pitch was to allow the goal - which they confirmed as correct.
The on-pitch referee was checking the position of the wrong player.
After the erroneous decision was incorrectly confirmed and the VAR team realised what happened, the VAR is heard swearing in the audio, before saying they "can't do anything" as they've "restarted the game".
Anyway, this isn’t a football newsletter. I’m neither qualified to write in detail about the decision made nor do I suspect my readers care very much about the impact on Liverpool’s otherwise unbeaten record so far this season.
What makes a good decision?
What I do want to draw attention to is the decision-making process. The entire debacle was frustrating for fans of football because all the information was right there in front of the decision makers. Yet they came to the wrong conclusion because they rushed the decision, communicated poorly with each other and their process didn’t adequately take into account the fact that the decision was irreversible. The bigger point is that better information is not sufficient to drive better decisions.
So how do you actually come up with better decisions - and what do we actually mean by “better” decisions. Ultimately good decisions are decisions which feel fair and justified at the time, and in retrospect hold up to scrutiny (even if they were incorrect with hindsight). This is really a way of saying the decision process is trustworthy. When conducting research at UCL last year, I developed a framework for trust centred on three aspects of trustworthiness in a process:
Output Trust - the use of inputs makes the outputs as accurate & reliable as possible
Process Trust - the process for making decisions is reasonable & transparent
Motivation Trust - the intention behind the decision making is clear & aligned
A parallel example outside the worlds of sport and business is in the judicial system. The degree of trust you have in the legal system as a whole is a function of how accurate the guilty verdicts are, how effective and fair the process is for bringing people to trial and whether you actually agree with the morals that underpin the laws themselves. Cumulatively the system works best if you are confident in the trustworthiness of all three of these pillars. If any one of these pillars starts to be questioned, it contributes to a significant erosion in trust.
Similarly in football, fans typically question the “Motivation Trust” component in decision making. In particular, referees are accused of bias and favouritism (esp in relation to big clubs). Sometimes fans also question the output quality of the decisions made by refs - especially when it comes to very subjective refereeing calls. However, the example above between Liverpool and Tottenham was a failure of process. During the game, there were several process failures and the process itself was opaque until PGMOL was forced to reveal details. This process failure led to a further erosion of Output and Motivation trust.
In start-ups and scale-ups, leaders can face a similarly high stakes and ambiguous decision making environment. This is often made more challenging by commercial pressures, a need to act quickly and data quality being very noisy.
So for the sake of referees, judges, and company leaders, it’s worth think about how we make better decisions. Or at least decisions which we consider to be more trustworthy. To help do that, I’m going to set out a few evidence-based principles for each of the pillars of trustworthy processes that I outlined earlier.
Output Trust
Building Output Trust is about increasing the reliability and reducing the bias of the outcomes of decisions. Ultimately, as much as I questioned “better information” in the VAR example, higher quality inputs do increase the reliability and accuracy of decisions. You just have to make sure you’re using the information effectively. In practice this looks like:
Triangulating different sources of information to get more information and reduce the overall “noise” from your inputs
Understanding what biases you may have which influence how the inputs are interpreted and decisions are made. Here are three that particularly impact business decision making:
First, I’ve written in the past about our tendency to ignore data and instead stick with the status quo. This “data blindness” is influenced by who sits around the decision making table and how well they know each other. In both cases, it’s important to value those who may be “disagreeable givers” - people who are willing to be direct and tell the truth, even if it may not be nice.
Second, when coming up with solutions or answers to problems we have a bias to “add things” rather than cut existing processes down, which is covered well in Leidy Klotz’s book Subtract.
Third, when we are making decisions on projects we’ve already started, we have a tendency to exhibit escalating commitment to our previous decision even if it looks less favourable in retrospect (which I wrote about this post).
Understand when more information is difficult and may not even be contributing to a better decision. This is particularly the case in situations which are very ambiguous and unknowable. There is often a tendency to wait for evidence one way or another in these ambiguous situations but definitive information is unlikely to arrive. In these environments you might have to make decisions based on “conviction without strong evidence” in order to unblock the decision. It helps to understand when you face this kind of environment and be explicit about “conviction” driven decisions, as they will inevitably be less reliable than decisions made with more information.
Process Trust
In addition to controlling the inputs and how they are used, you can also improve the quality of your process which brings together those inputs and the way that outputs are used.
The best way to think about this is across the 3 stages of the decision making process:
Before a decision is made you need to identify what type of decision you’re facing
During a decision your focus should be on providing clarity to stakeholders and decision makers
After a decision is made a strong process will set the outcome up for success
1. Before a decision is made, you need to be able to adjust your process to the type of decision you face.
This is because the “process” used will inevitably have an impact on outcomes and one process does not necessarily fit all types of decisions. In particular, here are three things to think about when it comes to the type of decision you face:
Is the outcome likely to be a 1-way-door or 2-way-door?
1-way-doors are decisions that are difficult (or even impossible) to reverse, whereas 2-way-doors are decisions that can be quite easily reversed. In general 1-way-door decisions are higher stakes and require more consideration before being made. These high stakes may mean you want more “hard” information before committing to a decision, and may result in including more stakeholders and setting out the rationale more clearly than you otherwise would. 2-way-doors require less upfront conviction/certainty, but organisations should be wary about “flip-flopping” in an unproductive way. As a result, you can be more experimental about 2-way-doors but you should spend more time setting upfront “success” or “kill” criteria so that you don’t move the goal posts once the decision has been made.
Is the need for a decision urgent or non-urgent?
Not making a decision is an interim decision. Sometimes there is a higher cost to waiting to make a final call than at other times. Equally moving very fast can result in less data being gathered or may add unnecessary pressure which contributes to more emotional decisions. Understanding the level of urgency can shift the time budget you allow when it comes to a final decision - if things are very urgent you can take a “fast and frugal” approach, but if you have more time you can afford to be more considerate.
Is this a topic that needs high buy-in or would a more directive process be better?
A key consideration in building Process Trust is who is involved in the decision and what is their role in that process. If a decision needs to have deep buy-in among stakeholders, it can be beneficial for them to be involved early in the discussion so they can feel like they were able to contribute to and feel ownership over the outcomes. However, some decisions don’t need as much buy-in into the process and instead suit a more directive form of decision-making. I would rather not have my doctor ask what kind of prescription I should receive, instead I’d rather he just told me what decision he thinks is best. Directive processes are better suited when decisions are technical in nature, where there is an overwhelming need for clarity or where stakeholders have deep trust in the decision makers to act effectively on their behalf (see below section on Motivation). In the case of our doctor’s prescription example, all three of those are satisfied. Directive processes can afford to be more “closed off” while high buy-in processes may need more stakeholder engagement and input.
2. While the decision-making process is ongoing, clarity becomes a primary process objective.
This is because misunderstandings can lead to errors in outcomes and generate post-hoc frustration. There are a few different types of clarity that are helpful:
Clarity about where a decision is in the process.
It always takes time for a decision to be made. People take time to escalate issues, gather evidence, discuss the options, consider the right path, make the decision then implement the decision. However, for many people delivering on the ground at a company, that process is a bit of a black box. This impacts people’s confidence in the process and the uncertainty hinders their ability to deliver effectively day-to-day. E.g., if you’re waiting for a decision which won’t be made for a long time you’re going to be held up, or equally if you push ahead on work that will be impacted by a later decision you could be wasting time & resources that could be used elsewhere.
Clarity about what people are trying to communicate when discussing a decision.
A key part of a decision process is weighing up the options with the other decision owners and discussing the relative benefits and challenges of each path. During this discussion, it’s likely that people will come from very different starting points and ways of thinking. As a result, miscommunication can be super damaging. You can see this from the audio transcript from the Liverpool game - the VAR doesn’t even know what the on-field decision was, so when he says “check complete” the on-field ref mistakes that for validation that the previous decision was correct. This is a perfect example of where shared language (in this case “check complete”) does not reliably indicate shared understanding. Jeff Patton talks about this a lot in his excellent book about User Story Mapping but effectively surface level conversations without clear explanation can lead to unresolved (and sometimes unknown) ambiguity in what’s being communicated. There are a range of ways you can help mitigate this from simply “repeating back the final decision and explaining the rationale” all the way through to more structural process changes around how you document decisions and include a “cooling period” before a decision goes live. But the key take-away is that it’s better to be overly explicit in what decision you think is best and why than being implicit (even if it takes a bit longer).
Clarity about who has what role in the decision process.
All decisions have stakeholders, especially those made by company leadership. As noted in the above section on “types of decisions”, in some cases you will need buy-in from the stakeholders during the decision making process. In general however, even if you don’t need buy-in, you may still want stakeholder involvement (e.g., to provide information to decision-makers or to describe the problem that an organisation is facing). However, including lots of stakeholders as equal partners in the discussion may slow down decision making processes and may be counterproductive at helping land on a clear outcome. It’s helpful to have an understanding of the different kinds of roles in a decision and think about who fits into each type of role. I tend to like the OVIS framework developed by my old employer BCG, which has 4 separate types of stakeholders:
Owners: people who are ultimately accountable for the decision
Veto Holders: people who have the right to block decisions
Influencers: people able to contribute to the discussion, either because they will be involved in implementation of the decision or because they have expertise in the subject area
Supporters: people who can provide information & assistance to the decision owners
In our refereeing example, the VAR is the owner, the on-field ref is a veto holder, the assistant VAR is an influencer and the technical person doing the replays is a supporter. Having clearer roles enables people to understand how they can best support the decision making process, while also increasing the Process Trust in the decision to outsiders.
3. After you’ve made the decision, adjust other things in your control to give the decision the best chance of success.
Decision processes don’t end at the point when you announce the original decision. Decisions have to be implemented, they have impacts on stakeholders and often they can be revisited (esp if the facts on the ground change). As a result, Process Trust extends beyond the point of making the decision. There are two things which can increase the probability of long term success:
First, a decision can’t be seen in isolation to the broader system it’s taking place within. The most obvious example from a work context is when a decision implies a new priority or a higher priority for an existing piece of work. Given the system constraints, new priorities effectively mean other things are being deprioritised. This can be done implicitly, which will generate confusion and misalignment; or it can be explicitly where the decision makers internalise the system effects and explicitly also decide what has to be deprioritised. When the decision makers don’t internalise the system effects like this, the success of the implementation of the decision largely falls out of their hands and into the hands of the people driving the changes on the ground. This can be fine if they are aligned and understand the rationale for the decision very clearly, however, it’s more likely to go wrong and thus render a good decision ineffective.
Second, time plays tricks with what we consider to be success. I’ve written about this at length but we are really biassed when it comes down to retrospectively assessing whether something was successful or not. This is due to a range of factors, such as our tendency to exhibit what psychologist and poker player Amy Duke describes as “resulting”. There are a couple of ways to get around this which I cover in that previous post, but the two notable ones are to:
Set upfront success and kill criteria for decisions - objectively measurable ways of assessing whether the ideas were successful or not
Create a system of accountability for decisions by introducing partners who as less close to the decision to offer counsel and provide scrutiny
Motivation Trust
Ultimately when people make decisions on behalf of a wider stakeholder group, there is always a feeling of lost control among those who aren’t directly involved in the process. Building a good process and coming to reliable outputs can help produce positive outcomes which justify the decision makers taking control, however, it’s worth considering what happens when things go wrong. Even in a perfect process, assumptions made about the world may need to be revised and new information can render previously-seen-as-sensible decisions as ineffective. The best protective factor towards this retrospective judgement is having clear Motivation Trust among the stakeholders. Having a high degree of alignment in this area can also improve the probability of decisions being implemented effectively. There are some common characteristics of decision processes with high Motivation Trust:
Decision owners are upfront about the rationale for a decision.
In communicating the outcomes of a decision, owners should also be upfront about the rationale. Offering more of the “why” has a range of benefits: it helps stakeholders understand the reasoning which makes them more likely to agree or at least tolerate the decision, it provides a shared mental model for how decision owners think with the wider group and it forces the group of decision owners to align on a shared understanding of the motivation for the decision. It also forces alignment about values and strategy. Sometimes this will feel challenging especially in the “conviction led” decisions we spoke about earlier, however, providing transparency about when decisions are conviction driven can help to build psychological safety and trust with stakeholders.
Everyone respects the decision making process.
This is easy to do when you agree with the decision but is harder when either decisions go against what you think is best or if you are managing high amounts of uncertainty because of the decision. Respecting the process involves two main pieces; first, participants have to bring decisions “into the process” instead of trying to deal with them informally; second, when decisions are made, decision owners should commit to the decisions (disagreement should be voiced before a decision is made but minimised in the immediate aftermath). On this latter point, “disagree and commit” is a principle which several successful companies have adopted to improve their decision architecture including Intel and Amazon. Having decision owners muddy messaging and clarity after a decision is made tends to erode confidence in the decision process and makes effective delivery more difficult - it’s also just bad for motivation.
Decision owners are upfront about what might change their minds and over what time horizon.
Finally, even though we are asking you to be clear about rationale and to “disagree and commit” - it’s also important to be clear about what kind of information or evidence would help change your mind about a decision. Asking questions like “what would we have to see for this to be the wrong decision” and setting a review date some distance into the future can help increase the confidence in the quality of the decision and the responsiveness of the decision makers. However, you should be a bit careful to put the review date sufficiently far ahead to avoid chopping and changing too soon before a decision fully has time to embed.
Final thoughts
Just to wrap up, it's worth highlighting how hard it is to make good decisions: you almost never have enough time or information, decisions always feel more personal than we would like and retrospective reassessment can set an unreasonable bar for forecasting. Setting up a good decision process and good decision architecture isn’t a catch-all solution, but it is a protective step against some of the common pitfalls of poor decision making. Hopefully, some of the suggestions in this (admittedly longer than planned) post will help people in your organisation avoid the pain felt by all Liverpool fans back in September.
