Conditional Probability and Bayes

one of the most important concepts in all of probability theory — that of conditional probability.

– Sheldon M. Ross

Topics

• Association Rules
• Conditional Probability
• Naive Bayes

Association Rules

Association Rules

$$ \text{Milk} \Rightarrow \text{Croissant} [ \text{support} = 2/5, \text{confidence} = 2/3 ] $$

• A measure of co-occurrence

Prize in Boxes

Three boxes M, N, O:

• Only one of them refers to a prize.
• The participant claims one of them, e.g., M.
• The host removes one of the empty boxes (N) from the other two boxes (N, O).
• Now we have only two candidate boxes for the participants, M, O.
• The participant is asked to reclaim a box.

Question:

• Should the participant switch?

Frequentist vs Bayesian

Frequentists:

• Probability is based on the repetition of events.
• Without repetition, the probability is unknown.
• Events are random.
• Make predictions based on probabilities.
• Relies on NHST to validate our models.

Bayesian:

• Probability is objective (educated guess). It is a conceptual tool to describe our degree of certainty.
• Probability is not necessarily a one-to-one map of occurrences of events.
• Data (such as a previous reoccurring event) is used to update our beliefs.
• Parameters of models are random.

Joint Probability

Joint Probability $$ P(A\cap B) $$ also denoted as $P(A, B)$

Examples of joint probabilities:

• $P(A, B)=0$: Event $A$ and event $B$ are so different that they will never happen together. In this case, $P(A\cup B) = P(A) + P(B)$.
• $P(A, B) = P(A)P(B)$: $A$ and $B$ are independent of each other.
• $P(A) + P(B) = 1$.

Conditional Probability and Bayes’s Theorem

$P(A\vert B)$: the probability of event $A$ if event $B$ occurred.

• It’s a rescaling/(re)normailization of $P(A)$: $P(A\cap B) = P(A\vert B)P(B)$;
• We didn’t specify $A$ and $B$: $P(B\cap A) = P(B\vert A)P(A)$ also holds;
• Apply $P(A\cap B) = P(B \cap A)$: $P(A\vert B)P(B) = P(B\vert A)P(A)$

In

$$ \begin{equation}P(A\vert B) = \frac{P(B\vert A)P(A)}{P(B)}\end{equation} $$

• $P(A)$: prior
• $P(A\vert B)$: posterior
• $P(B\vert A)$: likelihood

$A\to H$ (A theory) & $B\to D$ (some data points)

• $P(H\vert D) = \frac{P(D\vert H)P(H)}{P(D)}$

Solutions to the Prize in Boxes Problem

• $P(\text{Win}\vert \text{Switch}, \text{Wrong Box}) = 1$: The participant made a wrong choice at the first attempt, then switched.
• $P(\text{Win}\vert \text{Switch}, \text{Right Box}) = 0$
• $P(\text{Win}\vert \text{NoSwitch}, \text{Wrong Box})=1$
• $P(\text{Wrong Box}) = 2/3$
• $P(\text{Right Box}) = 1/3$
• $P(\text{Win}\vert \text{Switch})$: We will solve this.

Applying Bayes’ theorem,

$$ \begin{align} P(\text{Win}\vert \text{Switch}) =& P(\text{Win}\vert \text{Switch}, \text{Wrong Box}) P(\text{Wrong Box}) + P(\text{Win}\vert \text{Switch}, \text{Right Box}) P(\text{Right Box}) \\ =& 2/3 \nonumber \end{align} $$

$$ \begin{align} P(\text{Win}\vert \text{NoSwitch}) =& P(\text{Win}\vert \text{NoSwitch}, \text{Wrong Box}) P(\text{Wrong Box}) + P(\text{Win}\vert \text{NoSwitch}, \text{Right Box}) P(\text{Right Box}) \\ =& 1/3 \nonumber \end{align} $$

We update our probability perception when we have new data.

Rare Disease

We are about to test for a rare disease:

• Prevalence: $P(D) = 0.001$
• Test method: $P(+\vert D) = 0.99$
• Test method: $P(+\vert H) = 0.005$

The positive rate for a random person is

$$ P(+) = P(+\vert D)P(D) + P(+\vert H) P(H) = 0.006. $$

If a test is positive, the probability of the person being test has the disease is

$$ P(D\vert +) = \frac{P(+\vert D) P(D)}{P(+)} = 0.99\times 0.001/0.006 = 0.165. $$

$$ P(H\vert +) = \frac{P(+\vert H) P(H)}{P(+)} = 0.005\times (1-0.001)/0.006 = 0.8325. $$

Naive Bayes

• Chapter 16 of Introduction to Probability and Statistics for Engineers and Scientists (6th ed.) by Sheldon Ross.
• Naive Bayes