“I model for a living”. If I had said that to my friends, I suspect it would be greeted by laughter. The closest I’ve been to a catwalk, is seeing, well a cat walk. Obviously, when I say, model, I mean quantitatively model. Without any sort of context, most people would likely interpret it from the fashion meaning. I could give many similar examples, where interpretation of language can be imprecise. Just as with language, interpretation of models (the quantitative type!) can be tricky. Last week, I wrote about a few of my takeaways from NIPS machine conference and also a summary of the time series workshop at NIPS. One big topic at NIPS, was the subject of interpretation, which I’ll try to write about in more detail here. We use quantitative models to give us an answer based upon inputs. Whilst, it’s not expected that we should precisely be able to replicate a model output in our head, we often want to understand its output, and if possible why it has given us a certain output. The reason why interpretation is such a big topic in machine learning, is that typically machine learning derived models tend to be quite complicated. The question of “why” has driven humans through the ages. Why should that stop now, just because we have machines?
Jenn Wortman Vaughan (Microsoft) gave a talk on “manipulating and measuring model interpretability”. There were several approaches to aiding interpretability, one was creating a simple model. Indeed the idea of using a simple model to mimic a complex model, for reasons of interpretability, was suggested by a number of speakers at NIPS. Here the link between input and output is easier to fathom. She also asked the question of what was interpretability? For one it is driven by legal necessity, take for example the new GDPR regulations from the EU. Interpretability could be considered as a latent variable. It is not purely a computational problem and it needs an interdisciplinary approach.
Many of her points were echoed by a discussion panel. Interpretability is a property of a model, yet we don’t know how to optimise a model for interpretability, said Bolei Zhou (MIT). We could say broadly that, said Rich Caruana (Microsoft), in information theory there was often a trade off between accuracy and interpretability. He suggested that whilst simple models could be used to proxy a complex model, maybe it wouldn’t give a faithful representation. We can tell a lot from a mimic model, but we also have the question of data distribution. Kiri Wagstaff (JPL) noted that models are sensitive to their trained distributions. The bigger problem she added, training on finite distribution which might not be representative of future. This is of course, a problem we are all to familiar with in finance (admittedly, we’re not the only ones)! It’s no use backtesting a model purely in 2008, the returns there often look very different to any other year in the past 20 years, you wish to pick.
The idea of interpretation also needs to deal with the fact that humans face two types of problems, Suchi Saria (John Hopkins) said. First, there are tasks which humans understand, and we just get machines to do them faster (eg. image recognition). Second, there are instances where humans don’t know the answer and we expect the machine to tell us. We might want to include trading in this class. Humans don’t know in advance what is the best trade to put on, with much accuracy. At the same time, we don’t need massive accuracy, we just need to be right most of the most of time. Personally, in the first instance, I could accept the argument, that interpretability of what a model is doing may not be quite as important in the first case.
There was another interpretability panel, this time structured as a debate with a motion of whether interpretation was important. Yann LeCun, director of AI research at Facebook and one of the most well known figures in machine learning made was for the motion that interpretation was not always important. He told a story of when he had worked on a project with some economists to forecast real estate prices in the past. The result was two models, one quite simple and another much more complicated black box model. Whilst the simple one proved less accurate, the economists ended up using that in their publications. Despite that, they were keen to create a startup with the complex model! The moral is that if a model is more accurate, people are willing to use it, regardless of whether they can really understand it. The flip side is of course, that people do very much care when models go wrong and they don’t understand them. It’s probably one of the main reasons that certain domains haven’t quite embraced machine learning quite as much, notably areas such as finance and health. If a trader starts losing a lot of money on a massively black box strategy, it is challenging to keep running it.
Interpretability is also important in the learning process, not purely in judging the output. Vickoriya Krakovna (DeepMind) made the point that in a reinforcement learning exercise, what if the algorithm ended up exploiting a loophole in its reward function? Interpretability might not important if a system was 100% robust (but in practice, well, 100% might be a bit optimistic, I’d think!). It might also be possible to include humans as part of the learning process, to check a model was not exploring unsafe areas, said John Platt (Google) on his fusion power talk. He was working on a project using machine learning to develop fusion power. Exploring “unsafe” areas of the search space, could end up destroying an actual physical machine, which was being used to generate fusion power (no, they haven’t cracked the problem yet!)
Ali Rahimi (pictured above) and Benjamin Recht were awarded the test of time award for their research paper from nearly a decade ago. In Rahimi’s acceptance speech, he alluded to the topic of interpretability and the lack of understanding more broadly being a risk in machine learning. There was some hyperbole around AI and machine learning, he said, quoting the line “AI is the new electricity”. He drew a parallel with alchemy. Machine learning had become like alchemy. We want systems built on rigour, rather than alchemy. Yes, alchemy can be successful. The first alchemists made many scientific discoveries in their (unsuccessful) quest for gold. However, at the same time, similar people were responsible for using leeches in all sorts of unsuccessful ways in medicine. In machine learning, we can end up applying brittle optimisation techniques to loss function we don’t understand. We need to move machine learning from a state where of alchemy to electricity, he concluded.
We haven’t yet cracked the problem of interpreting complicated models, such as those used in machine learning, however, it seems clear from NIPS that it is an area which is attracting a lot of research from the machine learning community. For machine learning to be embraced more broadly, maybe it’s not just all about how number crunching we do, but more about whether we can be happy with their outputs and understanding the methods used.