Many efforts to apply machine learning get stuck due to concerns about the “black box” — that is, the lack of transparency around why a system does what it does. Sometimes this is because people want to understand why some prediction was made before they take life-altering actions, as when a computer vision system indicates a 95% likelihood of cancer from an x-ray of a patient’s lung. Sometimes it’s because technical teams need to identify and resolve bugs without disrupting the entire system. And now that the General Data Protection Regulation (GDPR) is in effect, businesses that handle consumer data are required to explain how automated systems make decisions, especially those that significantly affect individual lives, like allocating credit or hiring a candidate for a job. While GDPR only applies in Europe, businesses around the world anticipate that similar changes are coming and so are revisiting governance efforts.
When Is It Important for an Algorithm to Explain Itself?
Many efforts to apply machine learning get stuck due to concerns about the black box — that is, the lack of transparency around why a system does what it does. Sometimes this is because people want to understand why some prediction was made before they take life-altering actions, as when a computer vision system indicates a 95% likelihood of cancer from an x-ray of a patient’s lung. Sometimes it’s because technical teams need to identify and resolve bugs without disrupting the entire system. And now that the General Data Protection Regulation (GDPR) is in effect, businesses that handle consumer data are required to explain how automated systems make decisions, especially those that significantly affect individual lives, like allocating credit or hiring a candidate for a job. To consider explainability, you have to decide whether you need only be able to explain what procedures you’ll be using – for example, the types of data and types of models – or whether you want to be able to explain the inner workings of a mathematical model. That distinction is important because different machine learning algorithms are more and less easy to explain. A final challenge in explainability is to make it clear what the model actually optimizes for. This may seem daunting, but if the right people ask the right questions at the right time to inform a series of judgment calls and decisions, things become tractable.
HBR Learning