Probability

Definition

• Statistical experiment: Any procedure that produces data or observations
• Sample space: Denoted $S$ , the set of all possible outcomes of a statistical experiment
• Sample point: An outcome (element) in $S$
• Event: Subset of $S$, or a set of sample points
  • $S$ is also an event, called a sure event
  • Event that contains no elements, $\varnothing$ is called a null event
• Mutually exclusive / disjoint events: have no element in common
• Contained: event $A$ is contained in $B$ if all elements of $A$ are also elements of $B$. $A\subset B$
  • If $A\subset B$ and $B\subset A$, $A=B$ : $A$ and $B$ are equivalent

Event Operations

• $A\cap A^{\prime }=\varnothing$
• $A\cap \varnothing =\varnothing$
• $A\cup A^{\prime }=S$
• $(A^{\prime }{)}^{\prime }=A$
• $A\cup (B\cap C)=(A\cup B)\cap (A\cup C)$
• $A\cap (B\cup C)=(A\cap B)\cup (A\cap C)$
• $A\cup B=A\cup (B\cap A^{\prime })$
• $A=(A\cap B)\cup (A\cap B^{\prime })$
• De Morgans
  • $(A_1\cup A_2\cup ...\cup A_n{)}^{\prime }=A_1^{\prime }\cap A_2^{\prime }\cap ...\cap A_n^{\prime }$
  • $(A_1\cap A_2\cap ...\cap A_n{)}^{\prime }=A_1^{\prime }\cup A_2^{\prime }\cup ...\cup A_n^{\prime }$

Counting Methods

• Multiplication principle
  • When a series of events happen (one after another?)
  • To find the total number of final outcomes, multiply the number of ways for each event to occur
  • E.g. Store sells 2 types of sandwiches and 3 types of drinks. Total number of ways to choose a sandwich and a drink is $2\times 3=6$
• Addition principle
  • When choosing between mutually exclusive events (if you choose one option, you can’t choose the other at the same time)
  • To find the total number of final outcomes, add the number of ways for each event to occur
  • E.g. Same store as above, Total number of ways to choose a sandwich or a drink is $2+3=5$
• Permutation
  • A permutation is a selection and arrangement of $r$ objects out of $n$ (Order is taken into consideration)
  • $P_r^n=\frac{n!}{(n-r)!}=n(n-1)(n-2)...(n-r+1)$
  • Derived from multiplication principle, seen as choosing $r$ objects sequentially: $n$ ways to choose first object, $n-2$ to choose second etc.
  • E.g. The number of possible four-letter alphabet strings in which all letters are different $=P_4^{26}$
• Combination
  • A combination is a selection of $r$ objects out of $n$ without regard for order
  • $C_r^n$ or $\left(\genfrac{}{}{0ex}{}{n}{r}\right)=\frac{n!}{r!(n-r)!}$
  • Derivation of formula:
    • $P_r^n$ can be seen as two steps, 1: $C_n^r$ and 2: arrange items, $r!$
    • By multiplication principle, $P_r^n=C_n^r\times r!$ , solve to get formula
  • E.g. From 4 women and 3 men, how many possible groups with 2 w and 3 m: select 2 women and select 1 man. By multiplication rule, ways $=\left(\genfrac{}{}{0ex}{}{4}{2}\right)\times \left(\genfrac{}{}{0ex}{}{3}{1}\right)$

Probability

• The chance of an event happening
• Relative frequency of event $A$ after $n$ repetitions of experiment $f_A=\frac{n_A}{n}$. As $n\to \infty ,f_A\to P(A)$
• Axioms of probability
  1. For event , $0\le P(A)\le 1$
  2. For sample space $S$, $P(S)=1$
  3. For mutually exclusive $A$ and $B$ ($A\cap B=\varnothing$), $P(A\cup B)=P(A)+P(B)$

Properties of probability

• $P(\varnothing )=0$
  • $\varnothing \cap \varnothing =\varnothing$ and $\varnothing \cup \varnothing =\varnothing$
  • Applying axiom 3, $P(\varnothing )=P(\varnothing \cup \varnothing )=P(\varnothing )+P(\varnothing )=2P(\varnothing )$, implies $P(\varnothing )=0$
• For mutually exclusive $A_1,A_2,\cdots ,A_n$ ($A_i\cap A_j=\varnothing$ for any $i\ne j$), $P(A_1\cup A_2\cup \cdots \cup A_n)=P(A_1)+P(A_2)+\cdots +P(A_n)$
  • Use induction on axiom 3
• For event $A$, $P(A’)=1-P(A)$
  • $S=A\cup A’$ and $A\cap A’=\varnothing$, from axiom 2 and 3,
  • $1=P(S)=P(A\cup A’)=P(A)+P(A’)$
• For any events $A$ and $B$, $P(A)=P(A\cap B)+P(A\cap B’)$
  • Based on $A=(A\cap B)\cup (A\cap B^{\prime })$ and $(A\cap B)\cap (A\cap B^{\prime })=\varnothing$ and axiom 3
• For any events $A$ and $B$, $P(A\cup B)=P(A)+P(B)-P(A\cap B)$
  • From $A\cup B=B\cup (A\cap B^{\prime })$ and $B\cap (A\cap B^{\prime })=\varnothing$, and the previous property
  • $P(A\cup B)=P(B)+P(A\cap B^{\prime })=P(B)+P(A)-P(A\cap B)$
• If $A\subset B$, then $P(A)\le P(B)$
  • Since $A\subset B$ , then $A\cup B=B$
  • $A\cup B=A\cup (B\cap A^{\prime })$ and $A\cap (B\cap A^{\prime })=\varnothing$
  • $P(B)=P(A\cup B)=P(A\cup (B\cap A^{\prime }))=P(A)+P(B\cap A^{\prime })\ge P(A)$
• Finite sample space with equally likely outcomes
  • $S=\{a_1,a_2,\cdots ,a_n\}$ , where all $a$ are equally likely, $P(a_1)=P(a_2)=\cdots$
  • Then for event $A\subset S$, $P(A)=\text{no. of sample points in }A/\text{no. of sample points in }S$

Conditional Probability

• Probability that event $A$ happens, given that we have the information that “event $B$ has occurred”, or probability of $A$ given $B$ $P(A|B)=\frac{P(A\cup B)}{P(B)}$
• Kinda changing the sample space to be B, so now the sample points where $A$ happens is $A\cup B$
• Multiplication rule
  • $P(A\cap B)=P(A)P(B|A)$ if $P(A)\ne 0$
  • $P(A\cap B)=P(B)P(A|B)$ if $P(B)\ne 0$
• Inverse probability formula: $P(A|B)=\frac{P(A)P(B|A)}{P(B)}$

Independence

• Events $A$ and $B$ are independent, $A\perp B$ iff $P(A\cap B)=P(A)P(B)$
• If $P(A)\ne 0$, $A\perp B$ iff $P(A|B)=P(A)$
  • Knowledge of $A$ does not change the probability of $B$
• $A\perp B⟺P(B|A)=\frac{P(A\cap B)}{P(A)}=\frac{P(A)P(B)}{P(A)}=P(B)$
• Venn diagram can’t show independence, only can show mutually exclusive (they are totally different)

Law of Total Probability

• Partition: If $A_1,A_2,\cdots$ are mutually exclusive events, and ${\cup }_{i=1}^n{A}_i=S$ , $A_1,A_2,\cdots$ is a partition of $S$
• LoTP: For a partition $A_1,A_2,\cdots$ of $S$ and any event $B$, $P(B)={\sum }_{i=1}^{n}P(B\cap A_i)={\sum }_{i=1}^{n}P(A_i)P(B|A_i)$
  • For any events $A$ and $B$, $P(B)=P(A)P(B|A)+P(A’)P(B|A’)$ (special case where the partition is $A$ and $A’$)

Baye’s Theorem

• For a partition $A_1,A_2,\cdots$ of $S$ and any event $B$ and $k=1,2,\cdots$ . $P(A_k|B)=\frac{P(A_k)P(B|A_k)}{{\sum }_{i=1}^{n}P(A_i)P(B|A_i)}$
• Derived from conditional probability, multiplication rule and law of total probability: $P(A_k|B)=\frac{P(A_k\cap B)}{P(B)}=\frac{P(A_k)P(B|A_k)}{{\sum }_{i=1}^{n}P(B\cap A_i)}=\frac{P(A_k)P(B|A_k)}{{\sum }_{i=1}^{n}P(A_i)P(B|A_i)}$
• Or derived from $P(A\cap B)=P(A)P(A|B)=P(B)P(B|A)$ → $P(A|B)=\frac{P(B)P(B|A)}{P(A)}$
• Special case where $n=2$, partition is $\{A,A’\}$, $P(A|B)=\frac{P(A)P(B|A)}{P(A)P(B|A)+P(A^{\prime })P(B|A^{\prime })}$
  • $P(A|B)$ (Posterior Probability): The updated probability of $A$ occurring given that $B$ has occurred. This is the value you are solving for
  • $P(A)$ (Prior Probability): The initial probability of $A$ occurring before any new information ($B$) is taken into account
  • $P(B|A)$ (Likelihood): The probability of $B$ occurring given that $A$ is true. This is a measure of how likely the evidence ($B$) is under the hypothesis ($A$)
  • $P(A^{\prime })$ (Complementary Prior Probability): The probability of the opposite of $A$ occurring
  • $P(B|A^{\prime })$ (Likelihood of the Alternative): The probability of $B$ occurring given that $A$ is false (i.e., $A^{\prime }$ is true). This accounts for false positives or alternative causes of the evidence.