Probability: Difference between revisions

(10 intermediate revisions by the same user not shown)

Line 12:

* For all events <math>A</math> and <math>B</math>, <math>A \subset B \implies P(A) \leq P(B)</math>

===Conditional Probability===

<math>P(A|B)</math> is the probability of event A given event B.

Mathematically, this is defined as <math>P(A|B) = P(A,B) / P(B)</math>.

Note that this can also be written as <math>P(A|B)P(B) = P(A, B)</math>

With some additional substitution, we get '''Baye's Theorem''':

<math>

P(A|B) = \frac{P(B|A)P(A)}{P(B)}

</math>

==Random Variables==

A random variable is a variable which takes on a distribution rather than a value.

===PMF, PDF, CDF===

Line 18:

Line 30:

The comulative distribution function (CDF) is <math>F(x) = P(X \leq x)</math>.

The CDF is the prefix sum of the PMF or the integral of the PDF. Likewise, the PDF is the derivative of the CDF.

===Joint Random Variables===

Two random variables are independant iff <math>f_{X,Y}(x,y) = f_X(x) f_Y(y)</math>.

Otherwise, the marginal distribution is <math>f_X(x) = \int f_{X,Y}(x,y) dy</math>.

===Change of variables===

Let <math>g</math> be a monotonic increasing function and <math>Y = g(X)</math>.

Then <math>F_Y(y) = P(Y \leq y) = P(X \leq g^{-1}(y)) = F_X(g^{-1}(y))</math>.

And <math>f_Y(y) = \frac{d}{dy} F_Y(y) = \frac{d}{dy} F_X(g^{-1}(y)) = f_X(g^{-1}(y)) \frac{d}{dy}g^{-1}(y)</math>

Hence:

f_Y(y) = f_x(g^{-1}(y)) \frac{d}{dy} g^{-1}(y)

</math>

==Expectation and Variance==

Line 67:

Line 92:

* <math>\bar{X} \sim N(\mu, \sigma^2 / n)</math>

* <math>(n-1)S^2 / \sigma^2 \sim \chi^2(n-1)</math>

===Jensen's Inequality===

Let g be a convex function (i.e. second derivative is positive).

Then <math>g(E(x)) \leq E(g(x))</math>.

==Moments and Moment Generating Functions==

Line 79:

Line 109:

===Moment Generating Functions===

To compute moments, we can use a moment generating function (MGF):

With the MGF, we can get any order moments by taking n derivatives and setting <math display="inline">t=0</math>.

; Notes

* The MGF, if it exists, uniquely defines the distribution.

* The MGF of <math>X+Y</math> is <math>MGF_{X+Y}(t) = E(e^{t(X+Y)})=E(e^{tX})E(e^{tY}) = MGF_X(t) * MGF_Y(t)</math>

===Characteristic function===

Line 127:

Line 158:

Then the order statistics are <math>X_{(1)}, ..., X_{(n)}</math> where <math>X_{(i)}</math> represents the i'th smallest number.

;Min and Max

===Min and Max===

The easiest to reason about are the minimum and maximum order statistics:

;Joint PDF

===Joint PDF===

If <math>X_i</math> has pdf <math>f</math>, the joint pdf of <math>X_{(1)}, ..., X_{(n)}</math> is:

<math>

Line 139:

Line 170:

since there are n! ways perform a change of variables.

;Individual PDF

===Individual PDF===

<math>

f_{X(i)}(x) = \frac{n!}{(i-1)!(n-i)!} F(x)^{i-1} f(x) [1-F(x)]^{n-1}

</math>

==Inequalities and Limit Theorems==

Line 166:

Line 200:

Apply Markov's inequality:

Let <math>Y = |X - \mu|</math>

Then <math>P(|X - \mu| \geq k) = P(Y \geq k) = = P(Y^2 \geq k^2) \leq \frac{E(Y^2)}{k^2} = \frac{E((X - \mu)^2)}{k^2}</math>

Then:

<math>

\begin{aligned}

P(|X - \mu| \geq k) &= P(Y \geq k) \\

&= P(Y^2 \geq k^2) \\

&\leq \frac{E(Y^2)}{k^2} \\

&= \frac{E((X - \mu)^2)}{k^2}

\end{aligned}

</math>

}}

* Usually used to prove convergence in probability

Line 229:

Line 271:

==Textbooks==

* [https://smile.amazon.com/dp/032179477X Sheldon Ross' A First Course in Probability]

** This is a very good textbook that is standard across many universities. However, it only covers one semester of content.

~~The books below cover both introductory probability as well as statistics.~~

* [https://smile.amazon.com/dp/0321795431 Hogg and Craig's Mathematical Statistics]

* [https://smile.amazon.com/dp/0534243126 Casella and Burger's Statistical Inference]

@@ Line 12: / Line 12: @@
 * For all events <math>A</math> and <math>B</math>, <math>A \subset B \implies P(A) \leq P(B)</math>
 {{hidden | Proof | }}
+===Conditional Probability===
+<math>P(A|B)</math> is the probability of event A given event B.<br>
+Mathematically, this is defined as <math>P(A|B) = P(A,B) / P(B)</math>.<br>
+Note that this can also be written as <math>P(A|B)P(B) = P(A, B)</math>
+With some additional substitution, we get '''Baye's Theorem''':
+<math>
+P(A|B) = \frac{P(B|A)P(A)}{P(B)}
+</math>
+==Random Variables==
+A random variable is a variable which takes on a distribution rather than a value.
 ===PMF, PDF, CDF===
@@ Line 18: / Line 30: @@
 The comulative distribution function (CDF) is <math>F(x) = P(X \leq x)</math>.<br>
 The CDF is the prefix sum of the PMF or the integral of the PDF. Likewise, the PDF is the derivative of the CDF.
+===Joint Random Variables===
+Two random variables are independant iff <math>f_{X,Y}(x,y) = f_X(x) f_Y(y)</math>.<br>
+Otherwise, the marginal distribution is <math>f_X(x) = \int f_{X,Y}(x,y) dy</math>.
+===Change of variables===
+Let <math>g</math> be a monotonic increasing function and <math>Y = g(X)</math>.<br>
+Then <math>F_Y(y) = P(Y \leq y) = P(X \leq g^{-1}(y)) = F_X(g^{-1}(y))</math>.<br>
+And <math>f_Y(y) = \frac{d}{dy} F_Y(y) = \frac{d}{dy} F_X(g^{-1}(y)) = f_X(g^{-1}(y)) \frac{d}{dy}g^{-1}(y)</math><br>
+Hence:
+<math display="block">
+  f_Y(y) = f_x(g^{-1}(y)) \frac{d}{dy} g^{-1}(y)
+</math>
 ==Expectation and Variance==
@@ Line 67: / Line 92: @@
 * <math>\bar{X} \sim N(\mu, \sigma^2 / n)</math>
 * <math>(n-1)S^2 / \sigma^2 \sim \chi^2(n-1)</math>
+===Jensen's Inequality===
+{{main | Wikipedia: Jensen's inequality}}
+Let g be a convex function (i.e. second derivative is positive).
+Then <math>g(E(x)) \leq E(g(x))</math>.
 ==Moments and Moment Generating Functions==
@@ Line 79: / Line 109: @@
 ===Moment Generating Functions===
 To compute moments, we can use a moment generating function (MGF):
-<math>M_X(t) = E(e^{tX})</math>
+<math display="block">M_X(t) = E(e^{tX})</math>
 With the MGF, we can get any order moments by taking n derivatives and setting <math display="inline">t=0</math>.
 ; Notes
 * The MGF, if it exists, uniquely defines the distribution.
 * The MGF of <math>X+Y</math> is <math>MGF_{X+Y}(t) = E(e^{t(X+Y)})=E(e^{tX})E(e^{tY}) = MGF_X(t) * MGF_Y(t)</math>
 ===Characteristic function===
@@ Line 127: / Line 158: @@
 Then the order statistics are <math>X_{(1)}, ..., X_{(n)}</math> where <math>X_{(i)}</math> represents the i'th smallest number.
-;Min and Max
+===Min and Max===
 The easiest to reason about are the minimum and maximum order statistics:
 <math>P(X_{(1)} <= x) = P(\text{min}(X_i) <= x) = 1 - P(X_1 > x, ..., X_n > x)</math>
 <math>P(X_{(n)} <= x) = P(\text{max}(X_i) <= x) = P(X_1 <= x, ..., X_n <= x)</math>
-;Joint PDF
+===Joint PDF===
 If <math>X_i</math> has pdf <math>f</math>, the joint pdf of <math>X_{(1)}, ..., X_{(n)}</math> is:
 <math>
@@ Line 139: / Line 170: @@
 since there are n! ways perform a change of variables.
-;Individual PDF
+===Individual PDF===
+<math>
+f_{X(i)}(x) = \frac{n!}{(i-1)!(n-i)!} F(x)^{i-1} f(x) [1-F(x)]^{n-1}
+</math>
 ==Inequalities and Limit Theorems==
@@ Line 166: / Line 200: @@
 Apply Markov's inequality:<br>
 Let <math>Y = |X - \mu|</math><br>
-Then <math>P(|X - \mu| \geq k) = P(Y \geq k) = = P(Y^2 \geq k^2) \leq \frac{E(Y^2)}{k^2} = \frac{E((X - \mu)^2)}{k^2}</math>
+Then:<br>
+<math>
+\begin{aligned}
+P(|X - \mu| \geq k) &= P(Y \geq k) \\
+&= P(Y^2 \geq k^2) \\
+&\leq \frac{E(Y^2)}{k^2} \\
+&= \frac{E((X - \mu)^2)}{k^2}
+\end{aligned}
+</math>
 }}
 * Usually used to prove convergence in probability
@@ Line 229: / Line 271: @@
 ==Textbooks==
 * [https://smile.amazon.com/dp/032179477X Sheldon Ross' A First Course in Probability]
-** This is a very good textbook that is standard across many universities. However, it only covers one semester of content.
-The books below cover both introductory probability as well as statistics.
 * [https://smile.amazon.com/dp/0321795431 Hogg and Craig's Mathematical Statistics]
 * [https://smile.amazon.com/dp/0534243126 Casella and Burger's Statistical Inference]