Statistical guesswork and incomplete information

Can a simple probability experiment provide a credible basis for more complex decisions?

The key problem: In the absence of essential information, can a simple coin toss be a reasonable approximation for more complex probability experiments like finding the answers to a multiple choice questions? 

Question: Will the number 3 come up when a fair die is thrown? 
Answer: There is a (1/6) probability of the number 3 coming up. How do we know? We arrive at this number for the probability based on a significant amount of background information available to us. This can be summarized as: 

1. The die has 6 faces in total 
2. One of the 6 faces must come up when the die is thrown
3. The number 3 appears on one of these faces and hence, is 1 of the 6 possibilities 
4. Since the die is fair, each face is equally likely to come up 

Alternative scenario 

But what happens if we must answer the question with no background information available to us? Since the question is in “Will…?” form, we can legally respond with a Yes or No and not explain our answer any further. The probability for each answer can be notionally taken as (1/2) in the absence of any further information regarding the possible events. 

Another instance: “Will Spain beat Germany in the football match today?” A football fan (or a researcher with access to historical information) may be able to give a prediction based on how good each team is and how they have played against each other in the past. But for someone who doesn’t follow football at all, a Yes or a No are both equally acceptable answer. The only knowledge he/she needs is that football is a game played between two teams and that one team will win. Neglecting the chance of a draw, for simplicity, both teams have equal probability of winning. This is the same probability as Heads or Tails appearing in a coin toss. It is not uncommon to see people base important decisions on the result of a coin toss, even when the probability of each outcome of those decision is much more complex. 

For example, a Bollywood/Hollywood villain may use a coin toss to decide whether to kill the hero or not. Even though there is a 50% chance of the coin telling him to kill the hero or spare him, the actual decision he needs to take is much more complex. Can we retain the probabilistic simplicity of the coin toss, while applying the principle to more complex statistical problems? Consider the following example. 

Example: A real life situation 

A student is appearing for an examination, where each question offers five choices of answer - A, B, C, D and E. He is unprepared, has no knowledge of the subject and must rely entirely on guesswork to answer the questions. He finds a (fair) coin in his pocket and decides to use it as a tool to get some help from fate. Since he has 5 choices to deal with and the coin has only 2 sides, he devises a simple algorithm. 

1. Is the answer A? Heads: Yes. Stop tossing | Tails: No. Proceed to next step 
2. Is the answer B? Heads: Yes. Stop tossing | Tails: No. Proceed to next step 
3. Is the answer C? Heads: Yes. Stop tossing | Tails: No. Proceed to next step 
4. Is the answer D? Heads: Yes. Stop tossing | Tails: No. Proceed to next step 
5. Is the answer E? If not A, B, C nor D, it must be E 

How likely is he to do well in the examination? Here is a simple analysis. Initially, let us assume that each of the five choices have equal probability to be the right answer to a particular question. This can be reasonably demonstrated by examining the solution of a large number of questions. If 100 questions are taken, 20 each should be A, B, C, D and E. That is the extent of information that is available with the student. 

• Number of choices per answer = 5 
• Number of tosses = 5 – 1 = 4 (according to the algorithm) 

Probability analysis 

Outcome of coin toss				Result	Random Variable (C)	Probability
1	2	3	4
H	-	-	-	A	1	½
T	H	-	-	B	2	¼
T	T	H	-	C	3	1/8
T	T	T	H	D	4	1/16
T	T	T	T	E	5	1/16

• Expected value of C = E(C) = {(1/2) x 1} + {(1/4) x 2} + {(1/8) x 3} + {(1/16) x 4} + {(1/16) x 5} = 1.9375 à 2
• Approximating this as C = 2, this tends to choice B in the sample space
• Variance of C = V(C) = E(C²) – {E(C)}² = S²(C) = 1.4355
• Standard deviation of C = S(C) = 1.1981

Reliability of the coin toss as an estimator 

Intuitively, the above expectation indicates that if the same method is used a large number of times, the student is most likely to get answer B to most questions. Unless a test-creator consciously takes the decision to make each option equally likely, the inherent bias in each human being can be expected to result in an uneven distribution in the solution. 

If a student is very familiar with a professor and his/her past examinations, it is possible to work out a pattern and guess which answer is most likely. Let’s say that a particular professor tends to have 15 As, 20 Bs, 50 Cs, 10 Ds and 5 Es in his solution. In that case, the student may choose to adjust the algorithm to include this incomplete information he has. The simplest way to do this is to re-assign the random variables this: C:1, B:2, A:3, D:4 and E:5. 

Therefore, the probability of getting a C answer using his algorithm is very high.