PML Exercise 3 Solutions

Exercise 3.1 [Uncorrelated does not imply independent †]

Let $X \sim U(-1, 1)$ and $Y = X^2$. Clearly $Y$ is dependent on $X$ (in fact, $Y$ is uniquely determined by $X$). However, show that $\rho(X,Y) = 0$. Hint: if $X \sim U(a, b)$ then $\mathbb{E}[X] = (a+b)/2$ and $\mathbb{V}[X] = (b-a)^2/12$.

解答：

$$\begin{aligned} \rho&= \mathrm{corr}[X,Y]\triangleq \frac{\mathrm{Cov}[X,Y]}{\sqrt{\mathbb{V}[X]\mathbb{V}[Y]}}\\ &=\frac{\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]}{\sqrt{\frac{1}{3}\mathbb{V}[Y]}}=\sqrt3\frac{\mathbb{E}[X^3]-\mathbb{E}[X]\mathbb{E}[X^2]}{\sqrt{\mathbb{E}[X^4]-\mathbb{E}^2[X^2]}} \end{aligned}$$

易知 $\mathbb{E}[X^2]=\mathbb{V}[X]+\mathbb{E}^2[X]=\frac{1}{3}$，$\mathbb{E}[X^3]=\int_{-1}^1 p(x)x^3\mathrm{d}x=0$，$\mathbb{E}[X^4]=\int_{-1}^1p(x)x^4\mathrm{d}x=\frac{1}{5}$，带入上式可得 $\rho=0$。

Exercise 3.2 [Correlation coefficient is between -1 and +1]

Prove that $-1 \le \rho(X,Y) \le 1$.

$$\begin{aligned} \rho &= \frac{\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]}{\sqrt{(\mathbb{E}[X^2]-\mathbb{E}^2[X])(\mathbb{E}[Y^2]-\mathbb{E}^2[Y])}}\\ \rho^2 &=\frac{(\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y])^2}{(\mathbb{E}[X^2]-\mathbb{E}^2[X])(\mathbb{E}[Y^2]-\mathbb{E}^2[Y])} \end{aligned}$$

解答： 令 $\tilde{X}=X-\mathbb{E}[X], \tilde{Y}=Y-\mathbb{E}[Y]$，根据定义有 $\mathbb{V}[X]=\mathbb{E}[\tilde{X}^2],\mathbb{V}[Y]=\mathbb{E}[\tilde{Y}^2], \mathrm{Cov}[X,Y]=\mathbb{E}[\tilde{X}\tilde{Y}]$，引入任意实数 $t$，有

$$\begin{aligned} \mathbb{E}[(t\tilde{X}+\tilde{Y})^2]&\geq 0\\ t^2\mathbb{E}[\tilde{X}^2]+2t\mathbb{E}[\tilde{X}\tilde{Y}]+\mathbb{E}[\tilde{Y}^2]&\geq0\\ \end{aligned}$$

因为关于 $t$ 的一元二次函数上式恒成立，说明其判别式必须小于等于0，所以 $\Delta=(2\mathbb{E}[\tilde{X}\tilde{Y}])^2-4\mathbb{E}[\tilde{X}^2]\mathbb{E}[\tilde{Y}^2]\leq 0$，即 $\mathrm{Cov}^2[X,Y]\leq\mathbb{V}[X]\mathbb{V}[Y]$。

Exercise 3.3 [Correlation coefficient for linearly related variables is ±1 †]

Show that, if $Y = aX + b$ for some parameters $a > 0$ and $b$, then $\rho(X,Y) = 1$. Similarly show that if $a < 0$, then $\rho(X,Y) = -1$.

解答：

$$\rho=\frac{\mathbb{E}[aX^2+bX]-\mathbb{E}[X]\cdot (a\mathbb{E}[X]+b)}{\sqrt{a^2\mathbb{V}^2[X]}}=\frac{a(\mathbb{E}[X^2]-\mathbb{E}^2[X])}{a\mathbb{V}[X]}=1$$

类似易得-1。

Exercise 3.4 [Linear combinations of random variables]

Let $x$ be a random vector with mean $m$ and covariance matrix $\Sigma$. Let $A$ and $B$ be matrices.

• a. Derive the covariance matrix of $Ax$.
• b. Show that $\text{tr}(AB) = \text{tr}(BA)$.
• c. Derive an expression for $\mathbb{E}[x^TAx]$.

解答：

• a. 令 $x\in \mathbb{R}^N, A\in\mathbb{R}^{M\times N}, B\in \mathbb{R}^{N \times M}$

$$\begin{aligned} Ax &=[\sum_{m=1} A_{1,m}x_m, \cdots, \sum_{m=1}A_{N,m}x_{m}]^{\top}\\ \Sigma_{Ax}&=\begin{bmatrix} \sum_{n_1=1}\sum_{n_2=1} A_{1,n_1} A_{1,n_2} \Sigma_{n_1,n_2} & \cdots & \\ \vdots & \ddots & \vdots \\ & \cdots & \end{bmatrix}\\ &=A\Sigma A^{\top} \end{aligned}$$

• b.

$$\begin{aligned} \mathrm{tr}(AB) &= \sum_{m=1} \sum_{n=1} A_{m,n} B_{n,m} = \sum_{n=1} \sum_{m=1} B_{n,m} A_{m,n} = \mathrm{tr}(BA) \end{aligned}$$

• c. $A\in \mathbb{R}^{N\times N}$

$$\begin{aligned} \mathbb{E}[x^{\top}Ax] &= \sum_{n_1=1}\sum_{n_2=1} A_{n_1,n_2} \mathbb{E}[x_{n_1}x_{n_2}] \end{aligned}$$

⚠️ 完善：

• a. $\mathbb{E}[Ax]=A\mathbb{E}[x]=Am$，根据协方差的定义

$$\begin{aligned} \Sigma_{y=Ax}&=\mathrm{Cov}[Ax]=\mathbb{E}[(Ax-\mathbb{E}[Ax])(Ax-\mathbb{E}[Ax])^{\top}]\\ &=\mathbb{E}[A(x-m)(x-m)^{\top}A^{\top}]\\ &=A\mathbb{E}[(x-m)(x-m)^{\top}]A^{\top}\\ &=A\Sigma A^{\top} \end{aligned}$$

• c. 首先由标量的迹等于其本身 $x^{\top}Ax=\mathrm{tr}(x^{\top}Ax)$，再由 $\mathrm{tr}({AB})=\mathrm{tr}({BA})$可得 $\mathrm{tr}(x^{\top}Ax)=\mathrm{tr}((Ax)x^{\top})=\mathrm{tr}(Axx^{\top})$。 两边同时取期望，由于迹运算本质是线性求和，因此可以与期望交换顺序得

$$\mathbb{E}[x^{\top}Ax]=\mathbb{E}[\mathrm{tr}(Axx^{\top})]=\mathrm{tr}(\mathbb{E}[Axx^{\top}])=\mathrm{tr}(A\mathbb{E}[xx^{\top}])$$

又有协方差的定义 $\Sigma=\mathbb{E}[xx^{\top}]-\mathbb{E}[x]\mathbb{E}[x]^{\top}=\mathbb{E}[xx^{\top}]-mm^{\top}$，可得

$$\mathbb{E}[x^{\top}Ax]=\mathrm{tr}[A(\Sigma+mm^{\top})]=\mathrm{tr}(A\Sigma)+m^{\top}Am$$

Exercise 3.5 [Gaussian vs jointly Gaussian]

Let $X \sim \mathcal{N}(0, 1)$ and $Y = WX$, where $p(W = -1) = p(W = 1) = 0.5$. It is clear that $X$ and $Y$ are not independent, since $Y$ is a function of $X$.

• a. Show $Y \sim \mathcal{N}(0, 1)$.
• b. Show $\text{Cov}[X,Y] = 0$. Thus $X$ and $Y$ are uncorrelated but dependent, even though they are Gaussian. Hint: use the definition of covariance $\text{Cov}[X,Y] = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y]$ and the rule of iterated expectation $\mathbb{E}[XY] = \mathbb{E}[\mathbb{E}[XY|W]]$.

解答：

• a. 从 PDF 出发

$$\begin{aligned} p(Y=y)&=p(WX=y)\\ &=p(-X=y\mid W=-1)p(W=-1)+ p(X=y\mid W=1)p(W=1)\\ &=\frac{1}{2}f(-y)+\frac{1}{2}f(y)\\ &=f(y)=\mathcal{N}(0,1) \end{aligned}$$

• b. 是否能认为 W、X独立（需要此假设）

$$\begin{aligned} \mathrm{Cov}[X,Y]&=\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]\\ &=\mathbb{E}[XY]\\ &=\mathbb{E}[\mathbb{E}[XY\mid W]]\\ &=\sum_{W}p(W)p(XY\mid W)\\ &=\frac{1}{2}p(X^2)+\frac{1}{2}p(-X^2)\\ &=0 \end{aligned}$$

⚠️ 纠错：

• b.

$$\begin{aligned} \mathbb{E}_W[\mathbb{E}[XY\mid W]]&=p(W=1)\mathbb{E}[XY\mid W=1]+p(W=-1)\mathbb{E}[XY\mid W=-1]\\ &=\frac{1}{2}\mathbb{E}[X^2]+\frac{1}{2}\mathbb{E}[-X^2]\\ &=0 \end{aligned}$$

Exercise 3.6 [Normalization constant for a multidimensional Gaussian]

Prove that the normalization constant for a $d$-dimensional Gaussian is given by

$$(2\pi)^{d/2}|\Sigma|^{1/2} = \int \exp\left(-\frac{1}{2}(x- \mu)^T\Sigma^{-1}(x- \mu)\right)dx$$

Hint: diagonalize $\Sigma$ and use the fact that $|\Sigma| = \prod_i \lambda_i$ to write the joint pdf as a product of $d$ one-dimensional Gaussians in a transformed coordinate system. (You will need the change of variables formula.) Finally, use the normalization constant for univariate Gaussians.

解答：

$$\begin{aligned} p(\mathbf{x})&=\frac{1}{(2\pi)^{\frac{d}{2}}\prod_{i=1}^d\lambda_i^{\frac{1}{2}}}\exp{\left\{-\sum_{i=1}^d\frac{(x_i -\mu_i)^2}{2\lambda_i} \right\}}\\ \end{aligned}$$

⚠️ 纠错：

$$\begin{aligned} I&=\int_{\mathbb{R}^d} \exp{\left(-\frac{1}{2}(x-\mu)^{\top}\Sigma^{-1}(x-\mu) \right)}\mathrm{d}x\\ &=\int_{\mathbb{R}^d} \exp{\left(-\frac{1}{2}y^{\top}\Sigma^{-1}y \right)}\mathrm{d}y \quad \text{substitute }y=x-\mu\\ \end{aligned}$$

因为协方差矩阵是一个对称正定矩阵，它可以被正交对角化 $\Sigma = U \Lambda U^T$，其中 $U$ 是正交矩阵（满足 $U^TU = UU^T = I$），$\Lambda = \text{diag}(\lambda_1, \lambda_2, \dots, \lambda_d)$是由其特征值组成的对角矩阵，因此 $\Sigma^{-1} = U \Lambda^{-1} U^T$，由此可得

$$y^T\Sigma^{-1}y = y^T (U \Lambda^{-1} U^T) y = (U^Ty)^T \Lambda^{-1} (U^Ty)$$

引入 $z=U^{\top}y$，变量代换的雅可比矩阵就是 $U(\left|\det(U)\right|=1)$，因此微元代换关系为 $\mathrm{d}y = |\det(U)| \mathrm{d}z = \mathrm{d}z$。

$$\begin{aligned} I &= \int_{\mathbb{R}^d} \exp\left(-\frac{1}{2}z^T \Lambda^{-1} z\right) \mathrm{d}z\\ &= \prod_{i=1}^d \int_{-\infty}^{\infty} \exp\left(-\frac{z_i^2}{2\lambda_i}\right) \mathrm{d}z_i \quad \text{using } z^T \Lambda^{-1} z = \sum_{i=1}^d \frac{z_i^2}{\lambda_i}\\ &=\int_{-\infty}^{\infty} \exp\left(-\frac{z_i^2}{2\lambda_i}\right) \mathrm{d}z_i = (2\pi\lambda_i)^{1/2} \quad \text{using } \int_{-\infty}^{\infty} \exp(-\frac{x^2}{2\sigma^2})dx = \sqrt{2\pi\sigma^2} \\ &=I = \prod_{i=1}^d (2\pi)^{1/2} \lambda_i^{1/2} = (2\pi)^{d/2} \left( \prod_{i=1}^d \lambda_i \right)^{1/2} \end{aligned}$$

Exercise 3.7 [Sensor fusion with known variances in 1d]

Suppose we have two sensors with known (and different) variances $v_1$ and $v_2$, but unknown (and the same) mean $\mu$. Suppose we observe $n_1$ observations $y^{(1)}_i \sim \mathcal{N}(\mu, v_1)$ from the first sensor and $n_2$ observations $y^{(2)}_i \sim \mathcal{N}(\mu, v_2)$ from the second sensor. (For example, suppose $\mu$ is the true temperature outside, and sensor 1 is a precise (low variance) digital thermosensing device, and sensor 2 is an imprecise (high variance) mercury thermometer.) Let $D$ represent all the data from both sensors. What is the posterior $p(\mu|D)$, assuming a non-informative prior for $\mu$ (which we can simulate using a Gaussian with a precision of 0)? Give an explicit expression for the posterior mean and variance.

解答： 假设 $\mu \sim \mathcal{N}(m_0, \tau_0^{-1}), \tau_0 \rightarrow 0^{+}$，

$$\begin{aligned} p(\mu\mid D)&\propto p(D\mid \mu)p(\mu)\propto p(D\mid \mu)\\ &=p(\{y_i^1\}_{i=1}^{n_1}\mid \mu, v_1) p(\{y_j^2\}_{j=1}^{n_2}\mid \mu, v_2)\\ &\propto \exp{\left\{\sum_{i=1}^{n_1}-\frac{(y_i^1 - \mu)^2}{2v_1} +\sum_{j=1}^{n_2}-\frac{(\mu-y_j^2)^2}{2v_2} \right\}}\\ &\propto \exp{\left(-\frac{\mu^2 -2(v_2\sum_{i=1}^{n_1}y_i^1+v_1\sum_{j=1}^{n_2}y_j^2)\mu}{2v_1v_2 / (n_1v_2+n_2v_1)}\right)}\\ &=\mathcal{N}(\frac{v_2\sum_{i=1}^{n_1}y_i^1 + v_1\sum_{j=1}^{n_2}y_j^2}{n_1v_2+n_2v_1}, \frac{v_1v_2}{n_1v_2+n_2v_1}) \end{aligned}$$

Exercise 3.8 [Show that the Student distribution can be written as a Gaussian scale mixture]

Show that a Student distribution can be written as a Gaussian scale mixture, where we use a Gamma mixing distribution on the precision $\alpha$, i.e.

$$p(x|\mu, a, b) = \int_0^\infty \mathcal{N}(x|\mu, \alpha^{-1})\text{Ga}(\alpha|a, b)d\alpha$$

This can be viewed as an infinite mixture of Gaussians, with different precisions.

解答：

$$\begin{aligned} p(x\mid \mu,a,b) &= \int_{0}^{\infty}\frac{\alpha^{\frac{1}{2}}}{\sqrt{2\pi}} \exp{\left\{-\frac{\alpha(x-\mu)^2}{2} \right\}} \frac{b^a}{\Gamma(a)}\alpha^{a-1}\exp{\left\{-b\alpha \right\}} \mathrm{d}\alpha\\ &=\frac{b^a}{\sqrt{2\pi} \Gamma(a)}\int_{0}^{\infty}\alpha^{a-\frac{1}{2}}\exp{\left\{-\frac{(x-\mu)^2+2b}{2}\alpha \right\}}\mathrm{d}\alpha\\ &=\frac{b^a}{\sqrt{2\pi} \Gamma(a)}\cdot \left[\frac{2}{(x-\mu)^2 + 2b} \right]^{a+\frac{1}{2}}\int_{0}^{\infty}t^{a-\frac{1}{2}}\exp{\{-t\}}\mathrm{d}t \qquad \text{substitue } t=\frac{(x-\mu)^2+2b}{2}\alpha \\ &=\frac{b^a \Gamma(a+\frac{1}{2})}{\sqrt{2\pi} \Gamma(a)}\left[\frac{2}{(x-\mu)^2 + 2b} \right]^{a+\frac{1}{2}} \end{aligned}$$

⚠️ 完善： 利用 $\frac{2}{(x-\mu)^2 + 2b} = \frac{2}{2b \left(1 + \frac{(x-\mu)^2}{2b}\right)} = \frac{1}{b} \left( 1 + \frac{(x-\mu)^2}{2b} \right)^{-1}$得到

$$\begin{aligned} p(x\mid \mu,a,b)&= \frac{b^a \Gamma(a+\frac{1}{2})}{\sqrt{2\pi} \Gamma(a)} \cdot b^{-(a+\frac{1}{2})} \left[ 1 + \frac{(x-\mu)^2}{2b} \right]^{-(a+\frac{1}{2})}\\ &= \frac{\Gamma(a+\frac{1}{2})}{\Gamma(a)\sqrt{2\pi b}} \left[ 1 + \frac{1}{2b}(x-\mu)^2 \right]^{-(a+\frac{1}{2})} \end{aligned}$$

一个具有自由度 $\nu$（degrees of freedom）、位置参数 $\mu$ 和精度参数 $\lambda$ 的标准学生 t 分布的 PDF 为：

$$\text{St}(x\mid \mu, \lambda, \nu) = \frac{\Gamma(\frac{\nu+1}{2})}{\Gamma(\frac{\nu}{2})} \left(\frac{\lambda}{\nu \pi}\right)^{\frac{1}{2}} \left[ 1 + \frac{\lambda}{\nu}(x-\mu)^2 \right]^{-\frac{\nu+1}{2}}$$

令自由度 $\nu = 2a$，令精度参数 $\lambda = \frac{a}{b}$ 即可得到上式