The Exponential Family

random variable $\mathbf{X}$ is in the exponential family

$$\begin{align} P(\mathbf{X}=x;\eta) &= h(x)\mathop{exp}\{ \eta^T\mathbf{T}(x) - A(\eta) \} \\ &= \frac{1}{Z(\eta)} h(x) exp \{ \eta^T T(x) \} \end{align}$$

• $\eta$ : vector of natural parameters
• $\mathbf{T}$ : vector of sufficient statistics
• $\mathbf{A}$ : log partition function
• $log Z(\eta) = A(\eta)$

Example

Multivariate Gaussian

$$\begin{align} P(\mathbf{x};\mu;\Sigma) &= \frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}} \text{exp}\left( -\frac{1}{2} (\mathbf{x}-\mu)^T \Sigma^{-1} (\mathbf{x}-\mu) \right) \\ &= \frac{1}{(2\pi)^{p/2}} \text{exp} \left( -\frac{1}{2}(\text{tr } \mathbf{x}^T\Sigma^{-1} \mathbf{x} +\mu^T \Sigma^{-1} \mu - 2 \mu^T \Sigma^{-1} \mathbf{x} +ln |\Sigma| )\right) \\ &= \underbrace{ \frac{1}{(2\pi)^{p/2}} }_{h(\mathbf{x})} \text{exp} \left( -\frac{1}{2} \underbrace{\text{tr } \Sigma^{-1}\mathbf{x}\mathbf{x}^T}_{\text{vec}(\Sigma^{-1})^T \text{vec}(\mathbf{x}\mathbf{x}^T)} + \mu^T\Sigma^{-1}\mathbf{x} - \underbrace{\frac{1}{2} \mu^T \Sigma^{-1} \mu - \frac{1}{2} \text{ln}|\Sigma| }_{A(\eta)} \right) \end{align}$$

This implies that:

• $\eta = \left( \Sigma^{-1} \mu, -\frac{1}{2} vec (\Sigma^{-1}) \right)$
• $\mathbf{T}(\mathbf{x}) = (\mathbf{x}, vec(\mathbf{x}\mathbf{x}^T))$
• $A(\eta) = \frac{1}{2} (\mu^T \Sigma^{-1} \mu + \text{ln} |\Sigma|)$
• $h(\mathbf{x}) = \frac{1}{(2\pi)^{p/2}}$

Bernoulli

$$\begin{align} P(x;p) &= p^x (1-p)^{1-x} \\ \text{ln} P(x;p) &= x\text{ln}(p)+(1−x)\text{ln}(1−p) \\ &= x\text{ln}(p)−x\text{ln}(1−p)+\text{ln}(1−p) \\ &= x(\text{ln}(p)−\text{ln}(1−p))+\text{ln}(1−p) \\ &= x\text{ln} (\frac{p}{1-p}) + \text{ln} (1-p) \\ \text{exp} (\text{ln} P(x;p)) &= \text{exp} \left( x\text{ln} (\frac{p}{1-p}) + \text{ln} (1-p) \right) \end{align}$$

This implies that:

• $\eta = \text{ln} (\frac{p}{1-p})$
• $T(x) = x$
• $A(\eta) = -\text{ln}(1-p)$
• $h(x) = 1$

Others

the univariate Gaussian, Poisson, gamma, multinomial, linear regression, Ising model, restricted Boltzmann machines, and conditional random fields (CRFs) are all in the exponential family

Why Exponential Family

Moment generating property

$$\begin{align} \int P(x,\eta) dx = \int h(x) e^{\eta^T T(x) - A(\eta)} dx &= 1 \\ \int h(x) e^{\eta^T T(x)} &= Z(\eta) \end{align}$$

-

$$\begin{align} \frac{dA}{d\eta} &= \frac{d}{d\eta} log (Z(\eta)) = \frac{1}{Z(\eta)} \frac{d}{d\eta} Z(\eta) \\ &= \frac{1}{Z(\eta)} \frac{d}{d\eta} \int h(x) e^{\eta^T T(x)}dx \\ &= \int T(x) \frac{h(x) e^{\eta^T T(x)}}{Z(\eta)} = E[T(x)] \end{align}$$

-

$$\begin{align} \frac{d^2 A}{d^2 \eta} &= \int T^2(x) \frac{h(x) e^{\eta^T T(x)}}{Z(\eta)} dx - \int T(x) \frac{h(x) e^{\eta^T T(x)}}{Z(\eta)} dx \frac{1}{Z(\eta)} \frac{d}{d \eta} Z(\eta) \\ &= E[T^2(x)] - E^2[T(x)] \\ &= Var[T(x)] \end{align}$$

• $A(\eta)$ is convex since

  $$\frac{d^2 A(\eta)}{d \eta^2} = Var[T(x)] > 0$$

• specific $\eta$ map to mean $\mu$, so we define an invert $\psi (\mu) = \eta$

.