Superforecasting: The Art and Science of Prediction

I really enjoyed this book a few years ago, and I have come back to offer a review based on my notes at the time and how the insights have settled for me over time. I took away many key concepts for successfully forecasting uncertain events and also some areas I noted for further exploration. Many of the following notes are structured from the authors' insight into the demonstrated practices of repeatedly successful forecasters. The book mentions repeatedly the importance of measurement for assessment and revising forecasts and programs. Many people simply don't create any metrics of anything when they make unverifiable and chronologically ambiguous declarations. The book emphasizes the importance of receiving this feedback on predictions that measurement allows, as there is a studied gap between confidence and skill in judgment. We have a tendency to be uninterested in accumulating counterfactuals, but we must know when we fail to learn from it. If forecasts are either not made or not quantified and ambiguous, we can't receive clear feedback, so the thought process that led to the forecasts can't be improved upon. Feedback, however, allows for the psychological trap of hindsight bias. This is that when we know the outcome, that knowledge of the outcome skews our perception of what we thought at the time of the prediction and before we knew the outcome. The main qualities for successful forecasting are being open-minded, careful, and undertaking self-critical thinking with focus, which is not effortless. Commitment to self-improvement is the strongest predictor of long-term performance in measured forecasting. This can basically be considered as equivalent to the popular concept of grit. Studies show that individuals with fixed mindsets do not pay attention to new information that could improve their future predictions. Similarly, forecasts tend to improve when more probabilistic thinking is embraced rather than fatalistic thinking in regards to the perspective that certain events are inevitable. A few interesting findings that the authors expand upon in more detail in the book: experience is important to have the tacit knowledge essential to the practice of forecasting, and that grit, or perseverance, towards making great forecasts is three times as important as intelligence. Practices to undertake when forecasting are to create a breakdown of components to the question that you can distinguish and scrutinize your assumptions; develop backwards thinking as answering the questions of what you would need to know to answer the question, and then making appropriate numerical estimations for those questions; practice developing an outside view, which is starting with an anchored view from past experience of others, at first downplaying the problem's uniqueness; explore other potential views regarding the question; and express all aspects and perspectives into a single number that can be manipulated and updated. Psychological traps to be aware of discussed in the book include confirmation bias, which is a willingness to seek out information that confirms your hypothesis and not seek out information that may contradict it, which is the opposite of discovering counterfactuals; belief perseverance, also known as cognitive dissonance, in which individuals can be incapable of updating their belief in the face of new evidence by rationalization in order to not have their belief upset; scope insensitivity, which is not properly factoring in an important aspect of applicability of scope, such as timeframe, properly into the forecast; and thought type replacement, which is replacing a hard question in analysis with a similar question that's not equivalent but which is much easier to answer. Researched qualities to strive for as a forecaster: cautious, humble, nondeterministic, actively open-minded, reflective, numerate, pragmatic, analytical, probabilistic, belief updaters, intuitive psychologists, growth mindset. The authors then delve into a bit of another practical perspective on forecasting, which involves teams. Psychological traps for teams include the known phenomenon known as groupthink, which is that small cohesive groups tend to unconsciously develop shared illusions and norms that are often biased in favor of the group, which interfere with critical thinking regarding objective reality. There is also a tendency for members of the group to leave the hard work of critical thinking to others on the team instead of sharing this work optimally, which when combined with groupthink, leads the group towards tending to feel a sense of completion upon reaching a level of agreement. One idea to keep in mind for management of a group is that the group's collective thinking can be described as a product of the communication of the group itself and not the sum of the thinking of the individual members of a group. There are some common perceived problems with forecasting, which receive attention in the book: the wrong side of maybe fallacy, which is the thinking that a forecast was bad because the forecast was greater than 50% but the event didn't occur, which can lead to forecasters not willing to be vulnerable with their forecasts; publishing forecasts for all to see, where research shows that public posting of forecasts, with one's name associated with the forecast, creates more open-mindedness and increased performance; and the fallacy that because many factors are unquantifiable due their real complexity, the use of numbers in forecasting is therefore not useful. Some concepts that I took note of for further research from the book were: Bayesian-based application for belief updating, which is basically a mathematical way of comparing how powerful your past belief was relative to some specific new information, chaos theory, game theory, Monte Carlo methods, and systematic intake of news media. These are concepts that I was particularly interested in from the book based on my own interests and that I have continued to explore. This book was very valuable for cohesively bringing together the above concepts in the context of a compelling story, based on the DARPA research project which was compellingly won by the author's team as a product of the research that led to this groundbreaking book.

This book tells a great story about how a group of unpaid volunteers were able to defeat the most credentialed experts in a forecasting tournament and it goes through the techniques they used to do it. It's a harrowing underdog story. The author does a good job on showing how to predict the future when it comes to financial and socio-political forecasts but he doesn't go far enough in explaining how we could use these techniques in our daily life when it comes to everyday things like whether to save or spend money, and how much, where to go to school, what career to stay in, whether a relationship will last, how long a given business will stay afloat. After all, we make these big decisions based on future forecasts! The author does state that in the beginning of the book that we make forecasts all the time in our lives but I'm not sure to what degree we're able to consciously apply forecasting principles to every-day life situations. He could've given more practical examples if that were the case. He does say "Just as you can't learn to ride a bicycle by reading a physics textbook, you can't become a superforecaster by reading training manuals. Learning requires doing, with good feedback that leaves no ambiguity about whether you are succeeding. " So going off that you can't just expect to automatically become a good forecaster by reading this book. You have to getting out make a lot of forecasts, get feedback, and revise the way you do things accordingly. The problem is I'm not sure how many people reading this book would be motivated to go out of their way to do this. Still I don't want to detract you from reading this book because it truly was a good read. Just reading about the way these superforecasters would think and go about things should inspire us to do the same. They didn't see their views as "treasures to be guarded but as hypotheses to be tested." They were able to look at multiple perspectives and handle the cognitive dissonance (most ideologically driven people could not bear to do like-wise.) They would seek "active open-mindedness" which means they would go out of their way to have other falsify their views so they can sharpen their perspective. They would tap into the "Wisdom of the Crowds" by getting in lengthy internet discussions with other forecasters where they would "disagree without being disagreeable". They had the "growth mindset" which means they treat every failure not as a blow to their ego but as a learning opportunity as they would have lengthy postmortems on their failed predictions. They had the intellectual humility to recognize that reality is complex, but the confidence in their abilities to execute their task in a determined way.....And so on.... So regardless of whether or not you are able to successful apply these principles to your everyday life, this is still an interesting story and we could use the way these superforecasters think as a model to how we should approach our beliefs about the outside world.

Just like how Wall Street experts correctly forecast a market crash. If we predict that the stock market will decline by 20% every six months, we’ll eventually be right, but we don’t know exactly when. The authors’ research, funded by IARPA (Intelligence Advanced Research Projects Activity), found that most experts are barely better than random guessing when it comes to predicting world events. Still, a small subset of people, known as superforecasters, consistently outperform both experts and chance in forecasting outcomes, even without access to classified information or elite resources. Their findings show that superforecasting is not a talent, but rather a skill that can be learned. He summarized the composite portrait of a model superforecaster in chapter 8 of his book. A superforecaster tends to be: CAUTIOUS: Nothing is certain HUMBLE: Reality is infinitely complex NONDETERMINISTIC: What happens is not meant to be and does not have to happen In their abilities and thinking styles, they tend to be: ACTIVELY OPEN-MINDED: Beliefs are hypotheses to be tested, not treasures to be protected INTELLIGENT AND KNOWLEDGEABLE WITH A “NEED FOR COGNITION”: Intellectually curious, enjoy puzzles and mental challenges REFLECTIVE: Introspective and self-critical NUMERATE: Comfortable with numbers In their methods of forecasting, they tend to be: PRAGMATIC: Not wedded to any idea or agenda ANALYTICAL: Capable of stepping back from the top-of-your-nose perspective and considering other views DRAGONFLY-EYED: Value diverse views and synthesize them into their own PROBABILISTIC: Judge using many grades of maybe THOUGHTFUL UPDATERS: When facts change, they change their minds GOOD INTUITIVE PSYCHOLOGISTS: Aware of the value of checking for cognitive and emotional biases In their work ethics, they tend to have: A GROWTH MINDSET: Believe it’s possible to get better GRIT: Determined to keep at it, however long it takes Not every superforecaster has every attribute, but the odds are better when the superforecasters work together as a group and complement each other. This is a great book for both strategic planning and making informed business decisions.

I’ve been reading about Philip Tetlock’s work on forecasting for years and I was impressed. But somehow Superforecasting: The Art and Science of Prediction kept slipping down my “next read” list. That was my loss. I wish I’d read this book years ago. Superforecasting will give you insight into much more than forecasting. You’ll learn a lot about how we make decisions and the role that cognitive biases play. You’ll discover how to lead more effectively. You’ll also discover how we’re improving the way we make forecasts and decisions. Philip Tetlock and Dan Gardner compare the current state of forecasting and decision-making to the state of medicine in the 19th century. Here’s how they phrase it. “All too often, forecasting in the twenty-first century looks too much like nineteenth-century medicine. There are theories, assertions, and arguments. There are famous figures, as confident as they are well compensated. But there is little experimentation, or anything that could be called science, so we know much less than most people realize. And we pay the price. Although bad forecasting rarely leads as obviously to harm as does bad medicine, it steers us subtly toward bad decisions and all that flows from them—including monetary losses, missed opportunities, unnecessary suffering, even war and death.” That sounds dreadful. But the authors think you can improve your forecasting and decision-making. You can learn from what superforecasters do. That’s what Superforecasting is about. Start by paying attention to the process. Increase the number of your information inputs. Learn how to ask pointed questions. Watch out for cognitive biases and what the authors called “bait and switch.” Here’s Philip Tetlock’s description of “bait and switch.” “Formally, it’s called attribute substitution, but I call it bait and switch: when faced with a hard question, we often surreptitiously replace it with an easy one.” Personally, that was one of my powerful takeaways from this book. I’ve become acutely aware of how often I do a bait and switch when I’m analyzing information. Make precise forecasts. Replace the equivalent of, “I think it might rain” with “I think there’s a 70% possibility of rain before 5:00 PM.” Once you’ve done the hard work of developing a preliminary forecast adjust it as you gather more data and insight. Superforecasters adjust their forecasts frequently and in small increments. There’s one more thing you need to do. You need to review your forecasting performance. As with learning and mastering any other skill, you need good feedback and reflection. There’s one more big insight in this book. You’ll make better forecasts if you combine the practices of superforecasters with the practices of people the authors call “super questioners.” That covers the basics of the book, but it doesn’t give you an idea of how rich the material is. Several things in Superforecasting are worth the price of the book all by themselves. The leadership chapter is excellent. There’s a lot of good material about both making good leadership decisions and conveying those decisions to others. The book gives you an excellent discussion of Daniel Kahneman’s systems 1 and 2. As you read the book, you’re also reading an excellent review of cognitive biases. I loved the many historical examples. I learned a lot from analyses of the Bay of Pigs and the Cuban Missile Crises, even though I’ve read a lot about both. The authors tell the story of the CIA analysis of the weapons of mass destruction in Iraq. In a Nutshell Superforecasting: The Art and Science of Prediction will give you insight into much more than forecasting. If you apply what you learn from this book, you will make better forecasts and better decisions. You’ll also be able to improve your leadership and help create more effective teams.

I have been, and continue to be, a pundit. Upon reading this book I discovered that one of the reasons I'm somewhat good at punditry is because I am what Tetlock calls a "hedgehog." Hedgehogs tell tight, simple, clear stories that grab and hold audiences. Hedgehogs are confident. We organise our thoughts around "Big Ideas" and then we squeeze complex problems into our preferred cause-effect templates. Let's face it--hedgehogs make good pundits. The problem is that hedgehogs make terrible forecasters. I've noticed this myself. When predicting how the Supreme Court will decide a case, what a jury will decide, or how the public may respond to something, my forecasts are notoriously inaccurate. Reading Forecasting may not change my tactics in giving interviews (why mess with success) but I think it will affect how I write in the future. Concepts like Fermi estimates, outside vs. inside views, confirmation biases, and questioning basic, emotionally charged beliefs are not in my toolbox, but they will be now. If I am going to offer predictions I owe it to my readers to sharpen these skills. Easy reading? Not really. However, worth the effort.

I had high hopes for "Superforecasting" given all the buzz around it, but I came away with mixed feelings. Tetlock and Gardner present some fascinating ideas about prediction and decision-making, and the concept of superforecasters is intriguing. The book does a good job of explaining how some people are consistently better at making predictions than others, and it offers insights into their thought processes. However, I found the book to be somewhat repetitive and, at times, dry. While the anecdotes and examples were interesting, I felt like the authors could have condensed their main points into a much shorter book. Additionally, I was hoping for more practical advice on how to apply these concepts in everyday life or business settings, but the book fell short in this area. The writing style is accessible, which is a plus, but I sometimes felt like the authors were stretching their material to fill pages. Some chapters seemed to meander without adding much new information. On the positive side, the book does make you think critically about how we make predictions and judgments. It challenges some common misconceptions about forecasting and emphasizes the importance of constantly updating our beliefs based on new information. Overall, "Superforecasting" is worth a read if you're deeply interested in the topic of prediction and decision-making. However, if you're looking for a practical guide to improving your own forecasting skills, you might find this book somewhat lacking. It's more of an exploration of the subject than a how-to manual.

TOGETHER We have learned a lot about superforecasters, from their lives to their test scores to their work habits. Taking stock, we can now sketch a rough composite portrait of the modal superforecaster. In philosophic outlook, they tend to be: CAUTIOUS: Nothing is certain HUMBLE: Reality is infinitely complex NONDETERMINISTIC: What happens is not meant to be and does not have to happen In their abilities and thinking styles, they tend to be: ACTIVELY OPEN-MINDED: Beliefs are hypotheses to be tested, not treasures to be protected INTELLIGENT AND KNOWLEDGEABLE, WITH A “NEED FOR COGNITION”: Intellectually curious, enjoy puzzles and mental challenges REFLECTIVE: Introspective and self-critical NUMERATE: Comfortable with numbers In their methods of forecasting they tend to be: PRAGMATIC: Not wedded to any idea or agenda ANALYTICAL: Capable of stepping back from the tip-of-your-nose perspective and considering other views DRAGONFLY-EYED: Value diverse views and synthesize them into their own PROBABILISTIC: Judge using many grades of maybe THOUGHTFUL UPDATERS: When facts change, they change their minds GOOD INTUITIVE PSYCHOLOGISTS: Aware of the value of checking thinking for cognitive and emotional biases In their work ethic, they tend to have: A GROWTH MINDSET: Believe it’s possible to get better GRIT: Determined to keep at it however long it takes What more to say? “But ultimately, as with intelligence, this has less to do with traits someone possesses and more to do with behavior. A brilliant puzzle solver may have the raw material for forecasting, but if he doesn’t also have an appetite for questioning basic, emotionally charged beliefs he will often be at a disadvantage relative to a less intelligent person who has a greater capacity for self-critical thinking. It’s not the raw crunching power you have that matters most. It’s what you do with it.’’ ‘Questioning emotional beliefs’! Who does that! These ‘super-forecasters’. What else ? “Yet these are ordinary people. Forecasting is their hobby. Their only reward is a gift certificate and bragging rights on Facebook. Why do they put so much into it? One answer is it’s fun. “Need for cognition” is the psychological term for the tendency to engage in and enjoy hard mental slogs. People high in need for cognition are the sort who like crosswords and Sudoku puzzles, the harder, the better—and superforecasters score high in need-for-cognition tests.’’ ‘Need for cognition’. I didn’t know there was a term for this problem. I’ve worked out that’s my situation. Interesting. Another gem . . . “That was deeply perceptive. People often assume that when a decision is followed by a good outcome, the decision was good, which isn’t always true, and can be dangerous if it blinds us to the flaws in our thinking.’’ Great! CONTENTS 1. An Optimistic Skeptic 2. Illusions of Knowledge 3. Keeping Score 4. Superforecasters 5. Supersmart? 6. Superquants? 7. Supernewsjunkies? 8. Perpetual Beta 9. Superteams 10. The Leader’s Dilemma 11. Are They Really So Super? 12. What’s Next? All-in-all, an enlightening and informative explanation. One key theme, intelligence not controlled and harnessed to training is just . . . wasted. Recommended. Dozens and dozens of notes (some linked) No index No photographs

In a multi-year research study, authors Philip Tetlock and Dan Gardner identified volunteers, only paid $250 per year, to regularly try to predict questions about current events in a competition. A certain number of them have achieved the level of a "superforecaster" where they outperform even the federal intelligence community in their predictions. Obviously, these individuals demand further examination so that we all can learn from their "secret sauce." What makes these individuals tick in their work? And are they just lucky? Or super-smart? Do they work well in teams? If so, how? The authors deal with each concern and others, chapter by chapter, in their analysis of what it takes to accurately predict the future. Of course, no one - even the best superforecasters - have perfect prescience. They do have self-confidence in their abilities but also humility vis-a-vis the task at hand. They come from many walks of life but usually have an insatiable curiosity for continual improvement and a willingness to flex their concepts of rightness and wrongness in light of new data. The authors simply state that their goal is to do for the task of pundits what evidence-based medicine did for physicians. They want the language of probability to better engage with our civil dialogue and for talking heads to be accountable for their predictions. Perhaps then, they will state their positions with more precision and testability instead of vague pronouncements. And perhaps that, they suggest, can help society escape an endless cycle of partisan discord based on tribalism instead of accuracy. I, for one, wish them success.

This is an amazing book. I had read Thinking Fast and Slow before this. I thought these books pose juxtaposing ideas. Great insights into how human beings make decisions and how to consciously employ system 2.

O livro de Tetlock foi escrito em parceria com um jornalista (Gardner), que, nas palavras de Tetlock, tornou a escrita fluida e a exposição de conceitos complexos, didática e objetiva. O livro percorre várias correntes do pensamento contemporâneo, como a análise de diversas heurísticas, segundo a concepção de Amos Tversky e Daniel Kahneman (tendo o Autor dialogado ao longo da elaboração do livro com este último, em interação definidas como "socráticas", em suas própria palavras). Para não tornar esta resenha maçante, vou me deter em outros dois aspectos apenas. O primeiro é a ideia, inicialmente formulada por Fermi, de decompor problemas complexos em elementos mais simples, permitindo, com isso, estimar fenômenos com grande precisão. Uma versão mais moderna do mesmo método é detalhada por Lawrence Weinstein (em seus artigos e nos livros Guesstimation e Guesstimation 2.0), que não são citados, por razões que desconheço. O segundo é o debate com Nassim Taleb, com quem o Autor inclusive escreveu um artigo em colaboração. Longe da retórica inflamada de Taleb, ele reconhece que fenômenos de previsibilidade nula e a ocorrência de fatos inteiramente inesperados, são parte da vida e do mundo, e têm consequências matemáticas e estatísticas corretamente definidas por Taleb (distribuições com caudas sobremaneira longas ou pesadas), mas que isso não invalida a capacidade de previsão em um mundo onde tais fenômenos, de fato, podem ocorrer (os famosos "cisnes negros"), mas não constituem uma ameaça a qualquer previsibilidade, como argumentado por Taleb. Os conceitos extremos de Taleb tornariam o mundo, de tal modo, inapreensível, a ponto de flutuarmos na pura estocasticidade. Tendo a me alinhar com Tetlock, pois, a despeito da validade das críticas de Taleb, meu próprio trabalho cotidiano é identificar padrões e analisá-los. Obviamente, em um mundo carente de qualquer padrão discernível, tal exercício é inteiramente inútil.

A great book that shows what differentiates superforecasters - those who score a 60% greater degree of accuracy than ordinary forecasters, compared to, well, ordinary forecasters. The author structures each of the attributes into its own chapter with well known historic examples, such as why the intelligence community got it so wrong with Iraq's WMD program, to how they later made substantial changes and how that was put into effect on the killing of Osama bin Laden. The book also makes two very important points about forecasting, that most predictions that we encounter on TV or from pundits are either ambiguous with no specific actions, or that the timeframes are unclear, bringing it to the level where it's basically impossible to refute the initial claim. Setting clear results and timeframes is an important prerequisite to becoming a better forecasters, as these predictions can later be turned into feedback. The second point is that forecasting is definitely not a trait that only a selected few geniuses are able to perform, in fact the author shows that most superforecasters are actually ordinary people that follow a set of methods which anyone can learn and master through practising. I recommend anyone who is interested in improving their prediction skills to have a read of this book.

Aunque el libro se enfoca en hacer forecasts, en realidad se trata de cómo tomar decisiones, hacia donde mirar y qué tomar en cuenta. Es algo tedioso y técnico en ciertos puntos, pero termina siendo un librazo con mucha investigación detras.

Sehr gründliche Recherche, Einbeziehung der wichtigsten Theorien, plausible Schlussfolgerungen. Absolut empfehlenswert, insbesondere für Auswahl und Bewertung von Managern und Politikern