Conditional Expectation Given An Event

If $Y$ is a discrete r.v.

$$E(Y|A) = \sum_y P(Y=y|A)$$

If $Y$ is continuous r.v.

$$E(Y|A) = \int_{-\infty}^{\infty} yf(y|A)dy$$

Approximation

Image a large number of $n$ of replication of experiments $y_1,…,y_n$

$$E(Y)\approx \frac{1}{n}\sum_{j=1}^n y_j$$

If $I_j$ is the indicator of $A$ occurring

$$E(Y|A) \approx \frac{\sum_{j=1}^n y_j I_j}{\sum_{j=1}^n I_j}$$

Example (Life Expectation)

Yang is 24 years old, he hear average life expectancy is $80$, Should he conclude he has 50 years of life left ?

Of Course not, cause he already live $24$ years and some people may die less than $24$

$$E(T) < E(T|T\geq 30)$$

Law of Total Expectation

Let $A_1,…,A_n$ be partition of a sample space, $Y$ be a random variable on sample space. Then

$$E(Y) = \sum_{i=1}^n E(Y|A_i) P(A_i)$$

Example (Geometric Expectation Redux)

Let $X\sim Geom(p)$, as the number of Tails before the first Heads in a sequence of coin flips with p. $p$ of head. To get $E(X)$ from sum of series, it also can be obtained in another way. We condition on the outcome of the first toss: if it lands heads, then $X$ is $0$ and we’re done ; if it lands Tails, then we wasted one toss and back to where we started by memorylessness Therefore

$$\begin{align} E(X) &= E(X|\text{first toss }H)\cdot p + E(X|\text{first toss }T)\cdot q \\ &= 0 \cdot p + (1+E(X)) \cdot q \end{align}$$

which gives $E(X) = q/p$

Example (Time until HH vs. HT)

You toss a fair coin repeatedly. What is the expected number of tosses until the pattern HT/HH appears for the first times ?

Times until HT

• $W_{HT}$: number of tosses untill HT appears
• $W_1$: waiting time for first H

• $W_2$: additional waiting time for the first T

  Then

  $W_1\sim Fs(\frac{1}{2}),E[W_1] = 2$

  $W_2\sim Fs(\frac{1}{2}),E[W_2] = 2$

  $$E[W_{HT}] = E[W_1+W_2]=E[W_1] + E[W_2] = 4$$

Times until HH

$$E[W_{HH}] = E[W_HH|\text{first toss }H]\cdot \frac{1}{2} + E[W_HH|\text{first toss }T]\cdot \frac{1}{2}$$

where

$$E[W_{HH}|\text{first toss }T] = 1 + E[W_{HH}]$$

and

$$\begin{align} E[W_{HH}|\text{first toss }H] =& E[W_{HH}|\text{first toss }H, \text{second toss }H]\cdot \frac{1}{2} \\ &+ E[W_{HH}|\text{first toss }H, \text{second toss }T]\cdot \frac{1}{2} \\ =& 2\cdot \frac{1}{2} + (E[W_{HH}]+2)\cdot \frac{1}{2} \end{align}$$

Thus we get $E[W_{HH}] = 6$

As we can see the above example use the memorylessness property of the Conditional Expectation of distribution, and construct target in both side to calculate the target

---

Conditional Expectation Given An R.V.

Let $g(x) = E(Y|X=x)$ Then the conditional expectation of $Y$ given $X$, denoted $E(Y|E)$ is defined to be the random variable $g(X)$

Example (Stick Length)

Suppose we have a stick of length $1$ and break the stick at a point $X$ chosen uniformly at random. Given that $X=x$, we then choose another breakpoint $Y$ uniformly on the interval $[0,x]$, find $E(Y|X)$, and its mean and variance

$$E(Y|X) = X/2$$ $$E(E(Y|X)) = E(X/2) = \frac{1}{4}$$ $$Var(E(Y|X)) = Var(X/2) = \frac{1}{48}$$

---

Properties of Conditional Expectation

Theorem (Dropping independent)

If $X$ and $Y$ are independent, then $E(Y|X)=E(Y)$

Taking Out What’s Known

$$E(h(X)Y|X) = h(X) E(Y|X)$$

Theorem (Linearity)

$$E(Y_1+Y_2|X) = E(Y_1|X) + E(Y_2|X)$$

Theorem (Adam’s Law)

For any r.v.s $X$ and $Y$

$$E(E(Y|X)) = E(Y)$$

Proof by LOTP

For $X$ discrete

$$E(Y) = \sum_x E(Y|X=x) P(X=x)$$

We let $E(Y|X=x) = g(x)$, then

$$E(E(Y|X)) = E(g(X)) = \sum_x g(x) P(X=x) = \sum_x E(Y|X=x) P(X=x)$$

So

$$E(E(Y|X)) = E(Y)$$

Theorem (Adam’s Law with Extra Conditioning)

For any r.v.s $X,Y,Z$

$$E(E(Y|X,Z)|Z) = E(Y|Z)$$ $$E(E(X|Z,Y)|Y) = E(X|Y)$$

Definition (Conditional Variance)

$$Var(Y|X) = E((Y-E(Y|X))^2|X)$$

this equivalent to

$$Var(Y|X) = E(Y^2|X) - (E(Y|X))^2$$

Theorem (Eve’s Low)

$$Var(Y) = E(Var(Y|X)) + Var(E(Y|X))$$

Example (Random Sum)

A store receives $N$ customers a day, $N$ is an r.v. with finite mean and variance. Let $X_j$ be the amount spend by the $j^{th}$ customer, $X_j$ has the mean $\mu$ and variance $\sigma^2$, $N$ and $X_j$ are independent of one another. Find the random sum $X = \sum_{j=1}^N X_j$ in terms of $\mu,\sigma^2,E(N),Var(N)$

For E(X)

$$E(X|N) = E\left( \sum_{j=1}^N X_j|N \right) = \sum_{j=1}^N E(X_j|N)= \sum_{j=1}^N E(X_j) = N\mu$$

Finally, by Adam’s Law

$$E(X) = E(E(X|N)) = E(N\mu) =\mu E(N)$$

For Var(X)

We conditon on $N$ get $Var(X|N)$

$$Var(X|N) = Var\left( \sum_{j=1}^N X_j|N \right) = \sum_{j=1}^N Var(X_j|N) = \sum_{j=1}^N Var(X_j) = N\sigma^2$$

Eve’s Law give the unconditional variance of $X$

$$\begin{align} Var(X) =& E(Var(X|N)) + Var(E(X|N))\\ =& E(N\sigma^2) + Var(N\mu) \\ =& \sigma^2 E(N) + \mu^2 Var(N) \end{align}$$

---

Prediction and Estimation

Theorem (Projection Interpretation)

For any function $h$, the r.v. $Y-E(Y|X)$ is Uncorrelated with $h(X)$: $Cov(Y-E(Y|X),h(X)) = 0$, equivalently

$$E((Y-E(Y|X))h(X)) = 0$$

.