business career entrepreneur success

Making Mistakes Based on Evidence

People often have a hard time interpreting information about sample size. Is not that they think sample size is an important we all know that more evidence is better than less evidence. Instead, the challenge for decision-making comes when quality of evidence and sample size come into conflict.

A sort of diminishing returns for sample size

Suppose that I have two points: I tell you truthfully that one of the coins is weighted on one side so that it will flip that same side about two thirds of the time. The other point is just a normal fair coin you need to guess which is which. I pull one coin chosen randomly out of my pocket and I flip it three times, it comes up heads each time. I pull the other coin in my pocket I flip it over and over, after a couple of minutes the tally is 20 heads and 15 tails.

When surveyed most people guess that the first point is the biased one it always comes up heads but even a fair coin will flip the same way three times in a row pretty often, about 25% of time in fact. The second coin has a much larger sample size and it actually is slightly more likely to be that biased coin. I want to offer one important caveat however: for many sorts of decisions there’s a sort of diminishing returns for sample size.

A large pool sometimes doesn’t do any good

A good example comes from polling and nationwide elections sometimes will see criticism of a national poll that takes the following form: they only surveyed 500 people! How can they draw conclusions about more than 300 million Americans? If you want to find out what candidate a majority of Americans prefers for president, you don’t need to survey millions of people.

A survey of a few hundred to about a thousand people will provide a relatively small margin for error on the order of about 4%. Increasing the sample size beyond that point wouldn’t make the pole much better. What matters most for polling is that the sample is representative, there shouldn’t be any systematic bias in who participated. A large pool doesn’t do any good if the people’s opinions aren’t independent from each other, as can happen if a poll samples too many people from one geographic area or from the same age range.

How evidence integration actually works

It’s usually better to have a sample of 500 people who are broadly representative of the population the poll of 10,000 people who are unrepresentative. So even though I’m emphasizing the counterintuitive point that we can often do better with more low-quality evidence once sample size gets large enough then there are cases like polling for which quality becomes more important.

So far I told you that we integrate evidence to help us reach decisions and that we don’t always integrate evidence the right way, but I haven’t yet told you how evidence integration actually works, how psychological or neurobiological processes take different source of information and integrate them into a decision. There’s a good reason I haven’t told you this: we don’t know!

Evidence is accumulated over time

We only have a few tantalizing hints about the process of evidence integration and why it sometimes goes awry at least for the complex real-world decisions I discussed in this lecture. Those hints come from research on much simpler sort of decisions, those only involving two known outcomes with no uncertainty. The basic idea is that evidence is accumulated over time and a decision is reached when the evidence reaches some criteria.

Let me illustrate this with an analogy: suppose that you are sitting at an intersection in a major city. You watch the crossing traffic flowing left and right, if you just glance the street promote you can’t tell right away which direction had more traffic to the left or right. But after you watch for a few seconds you might have a guess and after a minute or so you can reach a judgment: left there are more cars going left.

The threshold value

Researchers have used experimental paradigms with this sort of flavor: a human participant or animal watches a display with moving dots and their task is to decide whether more dots are going left or going right. For now let’s consider two conditions one which there are 5% more dots to the right and the other in which there are 25% more dots moving to the right.

When researchers record activity and neuron sought to be involved with evidence integration an interesting pattern emerges: in both of those conditions 5% percent and 25% the activity of the neuron increases over time, but increases much faster in a 25% condition, where the evidence is stronger. The neurons activity increases continuously until it reaches some threshold value.

We’re also more prone to mistakes

It might take a bit more or less time to reach that threshold depending on the quality of the evidence. It’s faster when there’s 25% bias then when there’s 5% bias, but once it reaches the threshold the decision is made. The current best models assume that evidence accumulates continuously until we reach a decision.

If the evidence seems to be very high quality than the rate of accumulation is faster and we decide more quickly, but we’re also more prone to mistakes. If there’s a lot of low-quality evidence the rate of accumulation is slower and our decisions are slower but they’re also potentially more accurate.