Discrete Multivariate R.V.s

Definition (Joint CDF) The Joint CDf of r.v.s $X$ and $Y$ is the function $F_{X,Y}$ given by

F_{X, Y} (x, y) = P (X \leq x, Y \leq y)

$F_{X,Y}(x,y) = P(X\leq x,Y\leq y)$

Definition (Joint PMF) The Joint PMF of discrete r.v.s $X$ and $Y$ is the function $p_{X,Y}$ given by

p_{X, Y} (x, y) = P (X = x, Y = y)

$p_{X,Y}(x,y) = P(X=x,Y=y)$

Definition (Marginal PMF) For discrete r.v.s $X$ and $Y$ , Marginal PMF of $X$ is

P (X = x) = \sum_{y} P (X = x, Y = y)

$P(X=x) = \sum_y P(X=x,Y=y)$

Definition (Conditional PMF) For discrete r.v.s $X$ and $Y$ , the Conditional PMF of $X$ given $Y=y$ is

P_{X | Y} (x | y) = P (X = x | Y = y) = \frac{P (X = x, Y = y)}{P (Y = y)}

$P_{X|Y}(x|y) = P(X=x|Y=y)=\frac{P(X=x,Y=y)}{P(Y=y)}$

Definition (Independence of Discrete R.V.s) Random variables $X$ and $Y$ are independent if for all x and y

F_{X, Y} (x, y) = F_{X} (x) F_{Y} (y)

$F_{X,Y}(x,y) = F_X(x) F_Y(y)$

for all x and y also equivalent to the condition

P (Y = y | X = x) = P (Y = y)

$P(Y=y|X=x) = P(Y=y)$

Continuous Multivariate R.V.s

Definition (Joint PDF) If $X$ and $Y$ are continuous with joint CDF $F_{X,Y}$ then

f_{X, Y} (x, y) = \frac{\partial^{2}}{\partial x \partial y} F_{X, Y} (x, y)

$f_{X,Y}(x,y) = \frac{\partial^2}{\partial x \partial y} F_{X,Y}(x,y)$

Definition (Marginal PDF) If $X$ and $Y$ are continuous with joint PDF $f_{X,Y}$ then

f_{X} (x) = \int_{- \infty}^{\infty} f_{X, Y} (x, y) d y

$f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy$

Definition (Conditional PDF) For continuous r.v.s. $X$ and $Y$ with joint PDF $f_{X,Y}$ the Conditional PDF of $Y$ given $X=x$ is

f_{Y | X} (y | x) = \frac{f_{X, Y} (x, y)}{f_{X} (x)}

$f_{Y|X}(y|x)= \frac{f_{X,Y}(x,y)}{f_X(x)}$

Definition (Independence of Continuous R.V.s) Random variables $X$ and $Y$ are independent if for all x and y

F_{X, Y} (x, y) = F_{X} (x) F_{Y} (y)

$F_{X,Y}(x,y) = F_X(x)F_Y(y)$

If $X$ and $Y$ are continuous with joint PDF $f_{X,Y}$

f_{X, Y} (x, y) = f_{X} (x) f_{Y} (y)

$f_{X,Y}(x,y) = f_X(x) f_Y(y)$

Theorem (2D LOTUS) Let g be a function from $R^2$ to $R$

If $X$ and $Y$ are discrete

E (g (X, Y)) = \sum_{x} \sum_{y} g (x, y) P (X = x, Y = y)

$E(g(X,Y)) = \sum_x \sum_y g(x,y) P(X=x,Y=y)$

If $X$ and $Y$ are continuous

E (g (X, Y)) = \int_{- \infty}^{\infty} \int_{- \infty}^{\infty} g (x, y) f_{X, Y} (x, y) d x d y

$E(g(X,Y)) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g(x,y) f_{X,Y}(x,y) dxdy$

General Bayes’ Rule

Convariance and Correlation

Covariance

Measure a tendency of two r.v.s $X\&Y$ to go up or down together
Positive Covariance: $X$ go up, $Y$ tends go up
Negative Covariance: $X$ go up, $Y$ tends go down

Definition (Covariance) The covariance between r.v.s $X$ and $Y$ is

C o v (X, Y) = E ((X - E X) (Y - E Y)) = E (X Y) - E (X) E (Y)

$Cov(X,Y) = E((X-EX)(Y-EY))=E(XY)-E(X)E(Y)$

Theorem (Uncorrelated) If $X$ and $Y$ are independent, then they are Uncorrelated( $Cov(X,Y)=0$ )

Properties of Covariance

$Cov(X,X) = Var(X)$
$Cov(X,Y) = Cov(Y,X)$
$Cov(X,c) = 0$
$Cov(a\cdot X,Y) = a\cdot Cov(X,Y)$
$Cov(X+Y,Z) = Cov(X,Z)+Cov(Y,Z)$
$Cov(X+Y,W+Z) = Cov(X,Z)+Cov(X,W)+Cov(Y,Z)+Cov(Y,W)$
$Var(X+Y) = Var(X)+Var(Y) + 2Cov(X,Y)$
For n r.v.s $X_1,\dotsb ,X_n$ $Var(X_1+\dotsb +X_n)=Var(X_a)+\dotsb+Var(X_n)+2\sum_{i<j}Cov(X_i,Y_j)$

Definition (Correlation) The Correlation between r.v.s $X$ and $Y$ is

C o r r (X, Y) = \frac{C o v (X, Y)}{\sqrt{V a r (X) V a r (Y)}}

$Corr(X,Y) = \frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}$

Shifting and Scaling $X$ and $Y$ has no effect on correlation

Theorem (Correlation Bounds) For any r.v.s $X$ and $Y$

- 1 \leq C o r r (X, Y) \leq 1

$-1\leq Corr(X,Y) \leq 1$

Change of Variables

Theorem (Change of Variables in One Dimension) Let $X$ be a continuous r.v. with PDF $f_X$ , and let $Y = g(X)$ , where $g$ is differentiable and strictly increasing. Then the PDF of $Y$ is given by

f_{Y} (y) = f_{X} (x) | \frac{d x}{d y} |

$f_Y(y) = f_X(x) \left| \frac{dx}{dy} \right|$

where $x = g^{-1}(y)$

Proof:

F_{Y} (y) = P (Y \leq y) = P (g (X) \leq y) = P (X \leq g^{- 1} (y)) = F_{X} (g^{- 1} (y)) = F_{X} (x)

$F_Y(y) = P(Y\leq y)=P(g(X)\leq y)=P(X\leq g^{-1}(y))=F_X(g^{-1}(y))=F_X(x)$

Then result obtained By the chain rule

Theorem (Change of Variables) Let $X = (X_1,…,X_n)$ be a continuous random vector with joint PDF $f_X(x)$ and $Y=g(X)$ , $g$ is an invertible function from $R^n$ to $R^n$ then $\frac{\partial \mathbf{x}}{\partial \mathbf{y}}$ form a Jacobian matrix

\frac{\partial x}{\partial y} = (\begin{array}{cccc} \frac{\partial x_{1}}{\partial y_{1}} & \frac{\partial x_{1}}{\partial y_{2}} & \dots & \frac{\partial x_{1}}{\partial y_{n}} \\ ⋮ & ⋮ & ⋮ \\ \frac{\partial x_{n}}{\partial y_{1}} & \frac{\partial x_{n}}{\partial y_{2}} & \dots & \frac{\partial x_{n}}{\partial y_{n}} \end{array})

$\frac{\partial \mathbf{x}}{\partial \mathbf{y}} = \left( \begin{array}{cccc} \frac{\partial x_1}{\partial y_1} & \frac{\partial x_1}{\partial y_2} & \dotsb & \frac{\partial x_1}{\partial y_n} \\ \vdots & \vdots & & \vdots \\ \frac{\partial x_n}{\partial y_1} & \frac{\partial x_n}{\partial y_2} & \dotsb & \frac{\partial x_n}{\partial y_n} \\ \end{array} \right)$

Then the joint PDF of $Y$ is

f_{Y} (y) = f_{X} (x) | \frac{\partial x}{\partial y} |

$f_Y(y) = f_X(x) \left| \frac{\partial \mathbf{x}}{\partial \mathbf{y}} \right|$

Convolutions

Theorem (Convolution Sums and Integrals)
If $X$ and $Y$ are independent discrete r.v.s, then the PMF of their sum $T = X+Y$ is

\begin{aligned} P (T = t) = & \sum_{x} P (Y = t - x) P (X = x) \\ = & \sum_{y} P (X = t - y) P (Y = y) \end{aligned}

$\begin{align} P(T=t) =& \sum_x P(Y = t-x) P(X=x) \\ =& \sum_y P(X= t-y) P(Y=y) \end{align}$

If $X$ and $Y$ are independent continuous r.v.s, then the PMF of their sum $T = X+Y$ is

\begin{aligned} f_{T} (t) = & \int_{- \infty}^{\infty} f_{Y} (t - x) f_{X} (x) d x \\ = & \int_{- \infty}^{\infty} f_{X} (t - y) f_{Y} (y) d y \end{aligned}

$\begin{align} f_T(t) =& \int_{-\infty}^{\infty} f_Y(t-x) f_X(x) dx \\ =& \int_{-\infty}^{\infty} f_X(t-y) f_Y(y) dy \end{align}$

Order Statistics

Definition (Order Statistics) For r.v.s $X_1,X_2,…,X_n$ the order statistics sre the random variables $X_{(1)},…,X_{(2)}$ , where

$X_{(1)} = min (X_1,…,X_n)$
$X_{(2)}$ is the $2^{nd}$ of $X_1,…,X_n$
$\vdots$
$X_{n} = max(X_1,…,X_n)$

The order statistics are dependent, for example , if $X_{(1)} = 100$ , then $X_{(n)}$ is forced to be $\geq 100$

We foucs on the case $X_1,…,X_n$ are i.i.d continuous r.v.s, with CDF $F$ and PDF $f$

Theorem (CDF of Order Statistics) Let $X_1,…,X_n$ be i.i.d continuous r.v.s with CDF F, Then the CDF of the $j^{th}$ order statistic $X_{(j)}$ is

P (X_{(j)} \leq x) = \sum_{k = j}^{n} (\begin{matrix} n \\ k \end{matrix}) F (x)^{k} (1 - F (x))^{n - k}

$P(X_{(j)}\leq x)=\sum_{k=j}^n \left( \begin{array}{c} n\\k \end{array} \right) F(x)^k(1-F(x))^{n-k}$

Proof:

Let’s start with a specical case when $j=n, X_{(n)}=max(X_1,…,X_n)$ :

\begin{aligned} F_{X_{(n)}} (x) & = P [m a x (X_{1}, . . ., X_{n}) \leq x] \\ = P (X_{1} \leq x) \dots P (X_{n} \leq x) \\ = [F (x)]^{n} \end{aligned}

$\begin{align} F_{X_{(n)}} (x) &= P[max(X_1,...,X_n)\leq x] \\ &=P(X_1\leq x)\dotsb P(X_n\leq x) \\ &=[F(x)]^n \end{align}$

Then, consider another special case when $j=1, X_{(1)} = min(X_1,…,X_n)$ :

\begin{aligned} F_{X_{(1)}} (x) & = P [m i n (X_{1}, . . ., X_{n}) \leq x] \\ = 1 - P (X_{1} > x) \dots P (X_{n} > x) \\ = 1 - [1 - F (x)]^{n} \end{aligned}

$\begin{align} F_{X_{(1)}} (x) &= P[min(X_1,...,X_n)\leq x] \\ &=1 - P(X_1> x)\dotsb P(X_n> x) \\ &=1-[1-F(x)]^n \end{align}$

The result here can be rewrite as $\sum_{k=1}^n\left( \begin{array}{c} n\\k \end{array} \right) F(x)^k (1-F(x))^{n-k}$

This result can be obtained by expand $[F(x) + 1 -F(x)]^n$

Finally, let’s consider more general case where $1<j<n, X_{(j)}\leq x$ , this means at least $j$ of $\{X_i \}$ fall to the left of $x$

Denote $N$ as the nunber of $X_i$ landing to the left of $x$ . $X_i$ lands to the left of $x$ w.p. $P(X_i\leq x) = F(x)$ . Then $N\sim Bin(n,F(x))$

P (X_{(j)} \leq x) = P (N \geq j = \sum_{k = j}^{n} (\begin{matrix} n \\ k \end{matrix}) F (x)^{k} (1 - F (x))^{n - k}

$P(X_{(j)}\leq x) = P(N\geq j=\sum_{k=j}^n \left( \begin{array}{c} n\\k \end{array} \right) F(x)^k(1-F(x))^{n-k}$

Theorem (PDF of Order Statistic) Let $X_1,…,X_n$ be i.i.d. continuous r.v.s with CDF $F$ and PDF $f$ . Then the marginal PDF of $j^{th}$ order statistic $X_{(j)}$ is

f_{X_{(j)}} (x) = n (\begin{matrix} n - 1 \\ j - 1 \end{matrix}) f (x) F (x)^{j - 1} (1 - F (x))^{n - j}

$f_{X_{(j)}} (x) = n \left( \begin{array}{c} n-1\\j-1 \end{array} \right) f(x) F(x)^{j-1} (1-F(x))^{n-j}$

Theorem (Joint PDF) Let $X_1,…,X_n$ be i.i.d. continuous r.v.s with PDF $f$ , Then the joint PDF of all order statistics is

f_{X_{(1)}, . . ., X_{(n)}} (x_{1}, . . ., x_{n}) = n! \prod_{i = 1}^{n} f (x_{i}), x_{1} < x_{2} < \dots < x_{n}

$f_{X_{(1)},...,X_{(n)}}(x_1,...,x_n) = n! \prod_{i=1}^n f(x_i), x_1<x_2<\dotsb <x_n$

Example 1(Order Statistics of Uniforms) $U_1,U_2,…,U_n$ are i.i.d. $Unif(0,1)$ r.v.s with CDF F and PDF $f$

For $0\leq x\leq 1$ , $f(x) = 1$ , $F(x) = x$ , Then

f_{U_{(j)}} = n (\begin{matrix} n - 1 \\ j - 1 \end{matrix}) x^{j - 1} (1 - x)^{n - j}

$f_{U_{(j)}} = n \left( \begin{array}{c} n-1\\j-1 \end{array} \right) x^{j-1} (1-x)^{n-j}$

F_{U_{(j)}} (x) = \sum_{k = j}^{n} (\begin{matrix} n \\ k \end{matrix}) x^{k} (1 - x)^{n - k} = \int_{0}^{x} f_{U_{(j)}} (t) d t = \frac{n!}{(j - 1)! (n - j)!} \int_{0}^{x} t^{j - 1} (1 - t)^{n - j} d t

$F_{U_{(j)}}(x) = \sum_{k=j}^n \left( \begin{array}{c} n\\k \end{array} \right)x^k (1-x)^{n-k} = \int_0^x f_{U_{(j)}} (t) dt = \frac{n!}{(j-1)!(n-j)!}\int_0^x t^{j-1}(1-t)^{n-j}dt$

Stochastic-Process (Multivariate)

2018-07-29

Stochastic-Process (Multivariate)

Discrete Multivariate R.V.s

Continuous Multivariate R.V.s

General Bayes’ Rule

Convariance and Correlation

Properties of Covariance

Change of Variables

Convolutions

Order Statistics

Stochastic-Process (Multivariate)

Discrete Multivariate R.V.s

Continuous Multivariate R.V.s

General Bayes’ Rule

Convariance and Correlation

Properties of Covariance

Change of Variables

Convolutions

Order Statistics

thanks~