Discrete Multivariate R.V.s

Definition (Joint CDF) The Joint CDf of r.v.s $X$ and $Y$ is the function $F_{X,Y}$ given by

$$F_{X,Y}(x,y) = P(X\leq x,Y\leq y)$$

Definition (Joint PMF) The Joint PMF of discrete r.v.s $X$ and $Y$ is the function $p_{X,Y}$ given by

$$p_{X,Y}(x,y) = P(X=x,Y=y)$$

Definition (Marginal PMF) For discrete r.v.s $X$ and $Y$, Marginal PMF of $X$ is

$$P(X=x) = \sum_y P(X=x,Y=y)$$

Definition (Conditional PMF) For discrete r.v.s $X$ and $Y$, the Conditional PMF of $X$ given $Y=y$ is

$$P_{X|Y}(x|y) = P(X=x|Y=y)=\frac{P(X=x,Y=y)}{P(Y=y)}$$

Definition (Independence of Discrete R.V.s) Random variables $X$ and $Y$ are independent if for all x and y

$$F_{X,Y}(x,y) = F_X(x) F_Y(y)$$

for all x and y also equivalent to the condition

$$P(Y=y|X=x) = P(Y=y)$$

---

Continuous Multivariate R.V.s

Definition (Joint PDF) If $X$ and $Y$ are continuous with joint CDF $F_{X,Y}$ then

$$f_{X,Y}(x,y) = \frac{\partial^2}{\partial x \partial y} F_{X,Y}(x,y)$$

Definition (Marginal PDF) If $X$ and $Y$ are continuous with joint PDF $f_{X,Y}$ then

$$f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy$$

Definition (Conditional PDF) For continuous r.v.s. $X$ and $Y$ with joint PDF $f_{X,Y}$ the Conditional PDF of $Y$ given $X=x$ is

$$f_{Y|X}(y|x)= \frac{f_{X,Y}(x,y)}{f_X(x)}$$

Definition (Independence of Continuous R.V.s) Random variables $X$ and $Y$ are independent if for all x and y

$$F_{X,Y}(x,y) = F_X(x)F_Y(y)$$

If $X$ and $Y$ are continuous with joint PDF $f_{X,Y}$

$$f_{X,Y}(x,y) = f_X(x) f_Y(y)$$

Theorem (2D LOTUS) Let g be a function from $R^2$ to $R$

If $X$ and $Y$ are discrete

$$E(g(X,Y)) = \sum_x \sum_y g(x,y) P(X=x,Y=y)$$

If $X$ and $Y$ are continuous

$$E(g(X,Y)) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g(x,y) f_{X,Y}(x,y) dxdy$$

General Bayes’ Rule

---

Convariance and Correlation

Covariance

• Measure a tendency of two r.v.s $X\&Y$ to go up or down together
• Positive Covariance: $X$ go up, $Y$ tends go up
• Negative Covariance: $X$ go up, $Y$ tends go down

Definition (Covariance) The covariance between r.v.s $X$ and $Y$ is

$$Cov(X,Y) = E((X-EX)(Y-EY))=E(XY)-E(X)E(Y)$$

Theorem (Uncorrelated) If $X$ and $Y$ are independent, then they are Uncorrelated($Cov(X,Y)=0$)

Properties of Covariance

• $Cov(X,X) = Var(X)$
• $Cov(X,Y) = Cov(Y,X)$
• $Cov(X,c) = 0$
• $Cov(a\cdot X,Y) = a\cdot Cov(X,Y)$
• $Cov(X+Y,Z) = Cov(X,Z)+Cov(Y,Z)$
• $Cov(X+Y,W+Z) = Cov(X,Z)+Cov(X,W)+Cov(Y,Z)+Cov(Y,W)$
• $Var(X+Y) = Var(X)+Var(Y) + 2Cov(X,Y)$
• For n r.v.s $X_1,\dotsb ,X_n$ $Var(X_1+\dotsb +X_n)=Var(X_a)+\dotsb+Var(X_n)+2\sum_{i<j}Cov(X_i,Y_j)$

Definition (Correlation) The Correlation between r.v.s $X$ and $Y$ is

$$Corr(X,Y) = \frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}$$

Shifting and Scaling $X$ and $Y$ has no effect on correlation

Theorem (Correlation Bounds) For any r.v.s $X$ and $Y$

$$-1\leq Corr(X,Y) \leq 1$$

---

Change of Variables

Theorem (Change of Variables in One Dimension) Let $X$ be a continuous r.v. with PDF $f_X$, and let $Y = g(X)$, where $g$ is differentiable and strictly increasing. Then the PDF of $Y$ is given by

$$f_Y(y) = f_X(x) \left| \frac{dx}{dy} \right|$$

where $x = g^{-1}(y)$

Proof:

$$F_Y(y) = P(Y\leq y)=P(g(X)\leq y)=P(X\leq g^{-1}(y))=F_X(g^{-1}(y))=F_X(x)$$

Then result obtained By the chain rule

Theorem (Change of Variables) Let $X = (X_1,…,X_n)$ be a continuous random vector with joint PDF $f_X(x)$ and $Y=g(X)$, $g$ is an invertible function from $R^n$ to $R^n$ then $\frac{\partial \mathbf{x}}{\partial \mathbf{y}}$ form a Jacobian matrix

$$\frac{\partial \mathbf{x}}{\partial \mathbf{y}} = \left( \begin{array}{cccc} \frac{\partial x_1}{\partial y_1} & \frac{\partial x_1}{\partial y_2} & \dotsb & \frac{\partial x_1}{\partial y_n} \\ \vdots & \vdots & & \vdots \\ \frac{\partial x_n}{\partial y_1} & \frac{\partial x_n}{\partial y_2} & \dotsb & \frac{\partial x_n}{\partial y_n} \\ \end{array} \right)$$

Then the joint PDF of $Y$ is

$$f_Y(y) = f_X(x) \left| \frac{\partial \mathbf{x}}{\partial \mathbf{y}} \right|$$

---

Convolutions

Theorem (Convolution Sums and Integrals)
If $X$ and $Y$ are independent discrete r.v.s, then the PMF of their sum $T = X+Y$ is

$$\begin{align} P(T=t) =& \sum_x P(Y = t-x) P(X=x) \\ =& \sum_y P(X= t-y) P(Y=y) \end{align}$$

If $X$ and $Y$ are independent continuous r.v.s, then the PMF of their sum $T = X+Y$ is

$$\begin{align} f_T(t) =& \int_{-\infty}^{\infty} f_Y(t-x) f_X(x) dx \\ =& \int_{-\infty}^{\infty} f_X(t-y) f_Y(y) dy \end{align}$$

---

Order Statistics

Definition (Order Statistics) For r.v.s $X_1,X_2,…,X_n$ the order statistics sre the random variables $X_{(1)},…,X_{(2)}$, where

• $X_{(1)} = min (X_1,…,X_n)$
• $X_{(2)}$ is the $2^{nd}$ of $X_1,…,X_n$
• $\vdots$
• $X_{n} = max(X_1,…,X_n)$

The order statistics are dependent, for example , if $X_{(1)} = 100$, then $X_{(n)}$ is forced to be $\geq 100$

We foucs on the case $X_1,…,X_n$ are i.i.d continuous r.v.s, with CDF $F$ and PDF $f$

Theorem (CDF of Order Statistics) Let $X_1,…,X_n$ be i.i.d continuous r.v.s with CDF F, Then the CDF of the $j^{th}$ order statistic $X_{(j)}$ is

$$P(X_{(j)}\leq x)=\sum_{k=j}^n \left( \begin{array}{c} n\\k \end{array} \right) F(x)^k(1-F(x))^{n-k}$$

Proof:

---

Let’s start with a specical case when $j=n, X_{(n)}=max(X_1,…,X_n)$:

$$\begin{align} F_{X_{(n)}} (x) &= P[max(X_1,...,X_n)\leq x] \\ &=P(X_1\leq x)\dotsb P(X_n\leq x) \\ &=[F(x)]^n \end{align}$$

---

Then, consider another special case when $j=1, X_{(1)} = min(X_1,…,X_n)$:

$$\begin{align} F_{X_{(1)}} (x) &= P[min(X_1,...,X_n)\leq x] \\ &=1 - P(X_1> x)\dotsb P(X_n> x) \\ &=1-[1-F(x)]^n \end{align}$$

The result here can be rewrite as $\sum_{k=1}^n\left( \begin{array}{c} n\\k \end{array} \right) F(x)^k (1-F(x))^{n-k}$

This result can be obtained by expand $[F(x) + 1 -F(x)]^n$

---

Finally, let’s consider more general case where $1<j<n, X_{(j)}\leq x$, this means at least $j$ of $\{X_i \}$ fall to the left of $x$

Denote $N$ as the nunber of $X_i$ landing to the left of $x$. $X_i$ lands to the left of $x$ w.p. $P(X_i\leq x) = F(x)$. Then $N\sim Bin(n,F(x))$

$$P(X_{(j)}\leq x) = P(N\geq j=\sum_{k=j}^n \left( \begin{array}{c} n\\k \end{array} \right) F(x)^k(1-F(x))^{n-k}$$

---

Theorem (PDF of Order Statistic) Let $X_1,…,X_n$ be i.i.d. continuous r.v.s with CDF $F$ and PDF $f$. Then the marginal PDF of $j^{th}$ order statistic $X_{(j)}$ is

$$f_{X_{(j)}} (x) = n \left( \begin{array}{c} n-1\\j-1 \end{array} \right) f(x) F(x)^{j-1} (1-F(x))^{n-j}$$

Theorem (Joint PDF) Let $X_1,…,X_n$ be i.i.d. continuous r.v.s with PDF $f$, Then the joint PDF of all order statistics is

$$f_{X_{(1)},...,X_{(n)}}(x_1,...,x_n) = n! \prod_{i=1}^n f(x_i), x_1<x_2<\dotsb <x_n$$

Example 1(Order Statistics of Uniforms) $U_1,U_2,…,U_n$ are i.i.d. $Unif(0,1)$ r.v.s with CDF F and PDF $f$

For $0\leq x\leq 1$ ,$f(x) = 1$ , $F(x) = x$, Then

$$f_{U_{(j)}} = n \left( \begin{array}{c} n-1\\j-1 \end{array} \right) x^{j-1} (1-x)^{n-j}$$ $$F_{U_{(j)}}(x) = \sum_{k=j}^n \left( \begin{array}{c} n\\k \end{array} \right)x^k (1-x)^{n-k} = \int_0^x f_{U_{(j)}} (t) dt = \frac{n!}{(j-1)!(n-j)!}\int_0^x t^{j-1}(1-t)^{n-j}dt$$