Generating Function

Three kinds of generating functions

• Probability Generating Function (PGF) : related to Z-transform
• Moment Generating Function (MGF) : related to Laplace transform
• Characteristic Function (CF) : related to Fourier transform

Motivation

• PGF: handling non-negative integral random variables
• MGF: handling general random variables
• CF: equally useful with MGF

Application

• Easy to characterizing the distribution of the sum of independent random variables
• Play a central role in the study of branching processes
• Provide a bridge between complex analysis and probability
• ……

---

Moment Generating Function

Definition (Moment Generating Function) MGF of an r.v. $X$ is $M(t) = E(e^{tX})$, as a function of $t$ (different t denote different valued moment) and this must finite on some open interval (-a,a) containing 0 or dont exist.

Why we need MGF

• MGF encodes the moments of an r.v.
• MGF of an r.v. Determines its distribution, like CDF and PMF/PDF
• MFG make it easy to find the distribution of a sum of i.r.v.s.

Theorem (Moments via Derivatives of the MGF) $E(X^n) = M^{(n)}(0)$
Using Taylor expansion of $M(t)$ at 0

$$M(t) = \sum_{n=0}^{\infty} M^{(n)}(0) \frac{t^n}{n!}$$

Using Taylor expansion of $E(X)$

$$M(t) = E(e^{tX}) = E\left( \sum_{n=0}^{\infty} X^n \frac{t^n}{n!} \right)=\sum_{n=0}^{\infty} E(X^n) \frac{t^n}{n!}$$

Matching the coefficients of two expansions, we get $E(X^n) = M^{(n)}(0)$

MGF of Distribution

Theorem (MGF Determines the Distribution) Two r.v. have the same MGF have the same distribution, more strictly, if there is even a tiny interval containing 0 on which the MGF are equal, the the r.v.s must have same distribution.

Example 1 (Bernoulli MGF) MGF of $X\sim Bern(p)$
$e^{tX}=e^t$ with probability $p$, and $1$ with probability $q$, so $M(t) = E(e^{tX})=pe^t + q$

Example 2 (Geometric MGF) MGF of $X\sim Geom(p)$

$$M(t) = E(e^{tX})=\sum_{k=0}^{\infty} e^{tk}q^kp=p\sum_{k=0}^{\infty} (qe^t)^k=\frac{p}{1-qe^t}$$

t in $(-\infty, log(1/p))$

Example 3 (Uniform MGF) MGF of $U\sim Unif(a,b)$

$$M(t) = E(e^{tU}) = \int_a^b e^{tu}\frac{1}{b-a} du = \frac{e^{tb}-e^{ta}}{t(b-a)}$$

and $M(0) = 1$

Example 4 (Binomial MGF) $Bin(n,p)$

$$M(t) = (pe^t+q)^n$$

Example 5 (Negative Binomial) $NBin(r,p)$

$$M(t) = \left( \frac{p}{1-qe^t} \right)^r$$

Theorem (MGF of Location-Scale Transformation) If $X$ has MGF $M(t)$, then MGF of $a+bX$ is

$$E(e^{t(a+bX)})=e^{at}E(e^{btX})=e^{at}M(bt)$$

Example 6 (Normal MGF) MGF of $(X = \mu + \sigma Z) \sim N(\mu,\sigma^2)$

$$M_Z(t) = E(e^{tZ})=\int_{-\infty}^{\infty}e^{tz}\frac{1}{\sqrt{2\pi}}e^{-z^2/2}dz=e^{t^2/2}$$

Use the Theorem above then

$$M_X(t) = e^{\mu t}M_Z(\sigma t) = e^{\mu t}e^{(\sigma t)^2/2} = e^{\mu t + \frac{1}{2} \sigma^2 t^2}$$

Sum of Independent Distributions

Theorem (MGF of A Sum of Independent R.V.s) If $X$ and $Y$ are independent, Then

$$M_{X+Y} (t) = M_X(t) M_Y(t)$$

Example 1 (Sum of Poissons) $X\sim Pois(\lambda), Y\sim Pois(\mu)$, $X$ and $Y$ are independent. Then $X+Y \sim Pois(\lambda + \mu)$
The MGF of $X$ is

$$E(e^{tX}) = \sum_{k=0}^{\infty}e^{tk}\frac{e^{-\lambda}\lambda^k}{k!}=e^{-\lambda}\sum_{k=0}^{\infty}\frac{(\lambda e^t)^k}{k!}=e^{-\lambda}e^{\lambda e^t}=e^{\lambda(e^t-1)}$$

The MGF of $X+Y$ is

$$E(e^{tX})E(e^{tY}) = e^{\lambda (e^t-1)} e^{\mu (e^t-1)} = e^{(\lambda + \mu)(e^t-1)}$$

Which is the $Pois(\lambda + \mu)$, so $X+Y\sum Pois(\lambda+\mu)$

Example 2 (Sum of Normals) $X_1\sim N(\mu_1,\sigma_1^2)$ and $X_2 \sim N(\mu_2,\sigma_2^2)$, $X_1+X_2 = ?$
MGF of $X_1+X_2$ is

$$M_{X_1+X_2}(t)= M_{X_1}(t)M_{X_2}(t)= e^{\mu_1t+\frac{1}{2}\sigma^2_1t^2}\cdot e^{\mu_2 t+\frac{1}{2}\sigma_2^2 t^2}=e^{(\mu_1+\mu_2)t+\frac{1}{2}(\sigma_1^2+\sigma_2^2)t^2}$$

Which is the N(\mu_1 + \mu_2, \sigma_2^2 + \sigma_1^2) MGF.

---

Probability Generating Function

Definition (PGF) PGF of a nonnegative integer-valued r.v. $X$ with PMF $p_k = P(X=k)$ is the generating function of the PMF, By LOTUS , this is

$$E(t^X) = \sum_{k=0}^{\infty} p_k t^k$$

Example 1 (Generating Dice Probabilities) Let $X$ be the sum from rolling 6 pair dice, $X_1,…,X_6$ be the individual rolls, what is $P(X=18)$ ?
The PGF of $X_1$ is

$$E(t^{X_1}) = \frac{1}{6}(t+t^2+\dotsb+t^6)$$

The PGF of $X$ is

$$E(t^X) = E(t^{X_1}\dotsb t^{X_6}) = E(t^{X_1})\dotsb E(t^{X_6})=\frac{t^6}{6^6}(1+t+\dotsb +t^5)^6$$

The coefficient of $t^{18}$ in the PGF is $P(X=18)$, so

$$P(X=18) = \frac{3421}{6^6}$$

Theorem (PMF \& PGF)

$$P(X=k) = \frac{g_{X}^{(k)}(0)}{k!}$$

---

Characteristic Function

Definition CF The Characteristic function of a random variable $X$ is the function $\phi : R \rightarrow C$ defined by

$$\phi(t) = E(e^{itX}), i = \sqrt{-1}$$