PML Exercise 2 Solutions

Exercise 2.1 [Conditional independence †]

• a. Let $H \in \{1, \dots, K\}$ be a discrete random variable, and let $e_1$ and $e_2$ be the observed values of two other random variables $E_1$ and $E_2$. Suppose we wish to calculate the vector

$$\vec{P}(H|e_1, e_2) = (P(H=1|e_1, e_2), \dots, P(H=K|e_1, e_2))$$

Which of the following sets of numbers are sufficient for the calculation?

• i. $P(e_1, e_2), P(H), P(e_1|H), P(e_2|H)$

• ii. $P(e_1, e_2), P(H), P(e_1, e_2|H)$

• iii. $P(e_1|H), P(e_2|H), P(H)$

• b. Now suppose we now assume $E_1 \perp E_2|H$ (i.e., $E_1$ and $E_2$ are conditionally independent given $H$). Which of the above 3 sets are sufficient now? Show your calculations as well as giving the final result. Hint: use Bayes rule.

解答：

• a. 选项 ii 是计算该式的充分条件，$P(H\mid e_1, e_2) = \frac{P(H)P(e_1, e_2\mid H)}{P(e_1, e_2)}$
• b. $P(e_1, e_2\mid H) = P(e_1\mid H) P(e_2\mid H)$，则 i 也是充分条件

⚠️ 纠错： 选项 iii 也是充分条件，因为 $P(e_1, e_2) = \sum_{h=1}^K P(e_1\mid H=h) P(e_2\mid H=h) P(H=h)$

Exercise 2.2 [Pairwise independence does not imply mutual independence]

We say that two random variables are pairwise independent if

$$p(X_2|X_1) = p(X_2)$$

and hence

$$p(X_2, X_1) = p(X_1)p(X_2|X_1) = p(X_1)p(X_2)$$

We say that $n$ random variables are mutually independent if

$$p(X_i|X_S) = p(X_i) \quad \forall S \subseteq \{1, \dots, n\} \setminus \{i\}$$

and hence

$$p(X_{1:n}) = \prod_{i=1}^n p(X_i)$$

Show that pairwise independence between all pairs of variables does not necessarily imply mutual independence. It suffices to give a counter example.

解答： 注意 $S$ 是一个集合，存在反例：$X_1\sim U(0,1), X_2\sim U(0, 1), X_3\sim U(X_1, X_2)$

⚠️ 纠错： 上例表示不正确且不是两两独立，标准反例是 $X_1$ 和 $X_2$ 为独立的二元随机变量，$X_3=X_1\oplus X_2$，其中 $\oplus$ 为异或

Exercise 2.3 [Conditional independence iff joint factorizes †]

In the text we said $X \perp Y |Z$ iff

$$p(x, y|z) = p(x|z)p(y|z)$$

for all $x, y, z$ such that $p(z) > 0$. Now prove the following alternative definition: $X \perp Y |Z$ iff there exist functions $g$ and $h$ such that

$$p(x, y|z) = g(x, z)h(y, z)$$

for all $x, y, z$ such that $p(z) > 0$.

解答：

• $\Rightarrow$: 令 $g(x,z)=p(x\mid z), h(y,z)=p(y\mid z)$
• $\Leftarrow$: 两边对 $y$ （或 $x$） 积分（或求和）得 $\int_y p(x,y\mid z)= p(x\mid z) = g(x,z)\int_y h(y, z)$，$\int_x p(x,y\mid z)=p(y\mid z)=h(y,z)\int_x g(x,z)$，又因为 $\int_{x,y}p(x,y\mid z)=1=\int_x g(x,z) \int_y h(y,z)$，所以 $p(x\mid z)p(y\mid z)=g(x,z)h(y,z)\int_y h(y,z) \int_x g(x,z)=g(x,z)h(y,z)=p(x,y\mid z)$

Exercise 2.4 [Convolution of two Gaussians is a Gaussian]

Show that the convolution of two Gaussians is a Gaussian, i.e.,

$$p(y) = \mathcal{N}(x_1|\mu_1, \sigma_1^2) \otimes \mathcal{N}(x_2|\mu_2, \sigma_2^2) = \mathcal{N}(y|\mu_1 + \mu_2, \sigma_1^2 + \sigma_2^2)$$

where $y = x_1 + x_2$, $x_1 \sim \mathcal{N}(\mu_1, \sigma_1^2)$ and $x_2 \sim \mathcal{N}(\mu_2, \sigma_2^2)$ are independent rv’s.

解答： 因为

$$P_y(y^*)=\int_{-\infty}^{\infty}p_1(x_1)\left[\int_{-\infty}^{y^*-x_1} p_2(x_2)\mathrm{d}x_2 \right]\mathrm{d}x_1, \quad p(y)=\left[\frac{\mathrm{d}}{\mathrm{d}y^*} P_y(y^*) \right]_{y^*=y}=\int p_1(x_1) p_2(y - x_1)\mathrm{d}x_1$$

求解

$$\begin{aligned} \int_{-\infty}^{\infty} p_1(x_1) p_2(y - x_1)\mathrm{d}x_1&=\int_{-\infty}^{\infty} \frac{1}{2 \pi \sigma_1 \sigma_2} \exp{\left\{-\frac{1}{2} \left[\frac{(x_1 -\mu_1)^2}{\sigma_1^2}+\frac{(y-x_1 -\mu_2)^2}{\sigma_2^2}\right] \right\}}\mathrm{d}x_1 \\ &=\text{norm}_y \exp{\left\{ -\frac{y^2 - 2(\mu_1+\mu_2)y}{2(\sigma_1^2+\sigma_2^2)} \right\}} \end{aligned}$$

Exercise 2.5 [Expected value of the minimum of two rv’s †]

Suppose $X,Y$ are two points sampled independently and uniformly at random from the interval $[0, 1]$. What is the expected location of the leftmost point?

解答： 令 $Z=\min \{X, Y\}$，$\int_{0}^1 p_Z(Z=z) z \mathrm{d}z = \int_0^1 (p(X<Y)p_X(X=z) + p(X\geq Y) p_Y(Y=z)) z \mathrm{d} z=\int_0^1 z \mathrm{d}z = \frac{1}{2}$

⚠️ 纠错： 因为知道 $X<Y$ 时，$X$ 就不服从原先的均匀分布了。 从 CDF 法出发，首先 $P(Z>z)=P(\min{\{X,Y\}}>z)$，利用两者的独立性 $P(Z>z)=P(X>z, Y>z)=P(X>z)P(Y>z)=(1-z)(1-z)$，由此得到 CDF 再求导，最后期望得 $\frac{1}{3}$

Exercise 2.6 [Variance of a sum]

Show that the variance of a sum is

$$\mathbb{V}[X + Y] = \mathbb{V}[X] + \mathbb{V}[Y] + 2\text{Cov}[X,Y]$$

where $\text{Cov}[X,Y]$ is the covariance between $X$ and $Y$.

解答：

$$\begin{aligned} \mathbb{V}[X+Y] &= \mathbb{E}[X^2 + Y^2 + 2XY]-\mathbb{E}^2[X+Y]\\ &= \mathbb{E}[X^2]+\mathbb{E}[Y^2]+2\mathbb{E}[XY]-\mathbb{E}^2[X+Y]\\ &= \mathbb{E}[X^2]+\mathbb{E}[Y^2]+2\mathbb{E}[XY]-(\mathbb{E}[X]+\mathbb{E}[Y])^2\\ &=\mathbb{V}[X]+\mathbb{V}[Y]+2\mathrm{Cov}[X,Y] \end{aligned}$$

Exercise 2.7 [Deriving the inverse gamma density †]

Let $X \sim \text{Ga}(a, b)$, and $Y = 1/X$. Derive the distribution of $Y$.

解答： $P(y)=P(1/X\leq y)=P(X\geq 1/y)=1-P(X\leq 1/y)=1-\int_{0}^{1/y} \frac{b^a}{\Gamma(a)}x^{a-1}\mathrm{e}^{-bx}\mathrm{d}x$，则 $p(y)=\mathrm{d P(y)}{\mathrm{d} y}=-\frac{b^a}{\Gamma(a)}y^{1-a}\mathrm{e}^{-b/y}\cdot -\frac{1}{y^2}=\frac{b^a}{\Gamma(a)}y^{-1-a}\mathrm{e}^{-b/y}$

⚠️ 改进： 可以用变量变换公式 $f_Y(y)=f_X(x)\left|\frac{\mathrm{d}x}{\mathrm{d}y} \right|$ 更直接，同时注意定义区间

Exercise 2.8 [Mean, mode, variance for the beta distribution]

Suppose $\theta \sim \text{Beta}(a, b)$. Show that the mean, mode and variance are given by

$$\mathbb{E}[\theta] = \frac{a}{a+b}$$ $$\mathbb{V}[\theta] = \frac{ab}{(a+b)^2(a+b+1)}$$ $$\text{mode}[\theta] = \frac{a-1}{a+b-2}$$

解答：

$$\begin{aligned} \mathbb{E}[\theta]&=\int_0^1 \frac{1}{B(a,b)}x^a (1-x)^{b-1}\mathrm{d}x\\ &=\frac{1}{B(a,b)}\int_{0}^1 x^a(1-x)^{b-1}\mathrm{d}x\\ &=\frac{B(a+1, b)}{B(a, b)}\\ &=\frac{\Gamma(a+1)\Gamma(a+b)}{\Gamma(a)\Gamma(a+b+1)} \text{ using recursive property } \Gamma(a+1)=a\Gamma(a)\\ &=\frac{a}{a+b} \end{aligned}$$

求二阶矩，与期望结合可得方差

$$\begin{aligned} \mathbb{E}[\theta^2]&= \frac{B(a+2, b)}{B(a, b)}\\ &=\frac{\Gamma(a+2)\Gamma(a+b)}{\Gamma(a)\Gamma(a+b+2)}\\ &=\frac{(a+1)a}{(a+b+1)(a+b)} \end{aligned}$$

众数是使概率密度函数 $p(\theta)$ 取到最大值的 $\theta$ 值（连续情况下），先取对数处理

$$\ln p(\theta)=\ln \left(\frac{1}{B(a,b)} \right)+(a-1)\ln\theta + (b-1)\ln (1-\theta)$$

对 $\theta$ 求一阶导数并令其为0

$$\frac{\mathrm{d}}{\mathrm{d}\theta}\ln p(\theta) = \frac{a-1}{\theta}-\frac{b-1}{1-\theta}=0$$

求解可得 $\theta = \frac{a-1}{a+b-2}$

Exercise 2.9 [Bayes rule for medical diagnosis †]

After your yearly checkup, the doctor has bad news and good news. The bad news is that you tested positive for a serious disease, and that the test is 99% accurate (i.e., the probability of testing positive given that you have the disease is 0.99, as is the probability of testing negative given that you don’t have the disease). The good news is that this is a rare disease, striking only one in 10,000 people. What are the chances that you actually have the disease? (Show your calculations as well as giving the final result.)

解答： 令X为是否患病，Y为是否诊断患病，则

$$\begin{aligned} &p(Y=1\mid X=1)=0.99, p(Y=0\mid X=0)=0.99, p(X=1)=1/10000 \\ &p(X=1\mid Y=1)=\frac{p(Y=1\mid X=1) p(X=1)}{p(Y=1\mid X=0)p(X=0) + p(Y=1\mid X=1)p(X=1)}=\frac{11}{1122} \end{aligned}$$

Exercise 2.10 [Legal reasoning]

Suppose a crime has been committed. Blood is found at the scene for which there is no innocent explanation. It is of a type which is present in 1% of the population.

• a. The prosecutor claims: “There is a 1% chance that the defendant would have the crime blood type if he were innocent. Thus there is a 99% chance that he is guilty”. This is known as the prosecutor’s fallacy. What is wrong with this argument?
• b. The defender claims: “The crime occurred in a city of 800,000 people. The blood type would be found in approximately 8000 people. The evidence has provided a probability of just 1 in 8000 that the defendant is guilty, and thus has no relevance.” This is known as the defender’s fallacy. What is wrong with this argument?

解答：

• a. 令X为是此人否该血型，Y为此人是否犯罪

$$p(X=1)=1, p(X=1\mid Y=1)=1, p(Y=1\mid X=1)=\frac{p(X=1\mid Y=1)p(Y=1)}{p(X=1)}=p(Y=1)$$

• b. 与人数无关

⚠️ 纠错：

• a. 检方将 $p(X=1\mid Y=0)=0.01$ 当成了 $p(Y=0\mid X=1)=0.01$
• b. 辩护人的谬误在于只关注后验概率的绝对值。实际上，该证据使嫌疑人的犯罪概率由先验的 $\frac{1}{800000}$ 上升到了 $\frac{1}{8000}$，增加了 100 倍，因此是高度相关的证据。

Exercise 2.11 [Probabilities are sensitive to the form of the question that was used to generate the answer †]

My neighbor has two children. Assuming that the gender of a child is like a coin flip, it is most likely, a priori, that my neighbor has one boy and one girl, with probability 1/2. The other possibilities—two boys or two girls—have probabilities 1/4 and 1/4.

• a. Suppose I ask him whether he has any boys, and he says yes. What is the probability that one child is a girl?
• b. Suppose instead that I happen to see one of his children run by, and it is a boy. What is the probability that the other child is a girl?

解答：

• a. $p(\text{only one boy} \mid \text{have boy})=\frac{p(\text{only one boy})}{p(\text{have boy})}=\frac{1/2}{3/4}=2/3$
• b. $p(\text{the other is a girl}\mid \text{see a boy})=1/2$

Exercise 2.12 [Normalization constant for a 1D Gaussian]

The normalization constant for a zero-mean Gaussian is given by

$$Z = \int_a^b \exp\left(-\frac{x^2}{2\sigma^2}\right) dx$$

where $a = -\infty$ and $b = \infty$. To compute this, consider its square

$$Z^2 = \int_a^b \int_a^b \exp\left(-\frac{x^2+y^2}{2\sigma^2}\right) dxdy$$

Let us change variables from cartesian $(x, y)$ to polar $(r, \theta)$ using $x = r \cos \theta$ and $y = r \sin \theta$. Since $dxdy = rdrd\theta$, and $\cos^2\theta + \sin^2 \theta = 1$, we have

$$Z^2 = \int_0^{2\pi} \int_0^\infty r \exp\left(-\frac{r^2}{2\sigma^2}\right) drd\theta$$

Evaluate this integral and hence show $Z = \sqrt{2\pi\sigma^2}$. Hint 1: separate the integral into a product of two terms, the first of which (involving $d\theta$) is constant, so is easy. Hint 2: if $u = e^{-r^2/2\sigma^2}$ then $du/dr = -\frac{1}{\sigma^2} r e^{-r^2/2\sigma^2}$, so the second integral is also easy (since $\int u'(r)dr = u(r)$).

解答：

$$\begin{aligned} Z^2 &= \int_{0}^{2\pi} 1\mathrm{d}\theta \cdot \int_{0}^{\infty} r\exp{\left(-\frac{r^2}{2\sigma^2} \right)}\mathrm{d}r \\ & = 2\pi\int_{0}^{\infty}\frac{1}{2}\exp{\left(-\frac{r^2}{2\sigma^2} \right)} \mathrm{d}r^2\\ & = \pi \cdot \left. -2\sigma^2 \exp{\left(-\frac{r^2}{2\sigma^2} \right)}\right |_{0}^{\infty}\\ & = 2\pi \sigma^2 \end{aligned}$$