变分推断

2. 变分推断

Reference：[变分推断PPT]

2.1 变分推断介绍

变分推断（Variational Inference，VI）是贝叶斯学习中常用的、含有隐变量模型的学习和推断方法。变分推断和马尔科夫链蒙特卡洛法（MCMC）属于不同的技巧：

• MCMC通过随机抽样的方法近似地计算模型的后验概率（采样），适合小数据集以及精确度更重要的场景
• 变分推断通过解析的方法计算模型的后验概率的近似值（优化），适合大数据集以及想快速测试多种模型的场景

为什么关心后验概率 $P(\theta\mid X)$ ？

1. 推断（Beyesian Inference）：后验分布 $P(\theta \mid X)$ 包含了模型的重要信息，描述了数据样本产生的过程，例如从用户的观影历史评分信息 $Y$ 中推断用户的偏好模型 $\theta$
2. 决策（Beyesian Dicision Theory）：对于新样本 $\tilde{x}$，求 $P(\tilde{x}\mid X)$

$$\begin{aligned} P(\tilde{x} \mid X)=& \int_{\theta} P(\tilde{x},\theta \mid X) \ \mathrm{d}\theta \\ =&\int_{\theta} P(\tilde{x} \mid \theta) P(\theta \mid X) \ \mathrm{d} \theta \\ =& E_{\theta \mid X} [P(\tilde{x}\mid \theta)] \\ \end{aligned}\tag{62}$$

被称为后验预测分布（Posterior predictive distribution），例如根据用户的历史评分信息 $X$ 预测用户对于新电影 $\tilde{x}$ 的评分

2.2 变分推断推导

贝叶斯参数学习问题的描述：

• $X$ 观测数据
• $Z$ 隐变量+参数
• $\theta$ 超参数 注意，这里的符号表示和 EM 算法中的表述有区别，贝叶斯参数学习需要推断的是 $Z$ 中的参数，及学习后验分布 $P(Z\mid \theta)$

首先是 evidence 的分解

$$\begin{aligned} \underbrace{\log P(X\mid \theta)}_{\mathrm{evidence}} &= \underbrace{\int_{Z} q(Z) \log \frac{P(X,Z\mid \theta)}{q(Z)} \ \mathrm{d}Z}_{\mathrm{ELBO}}+\underbrace{\int_Z q(Z)\log \frac{q(Z)}{P(Z\mid X,\theta)} \ \mathrm{d}Z}_{\mathrm{KL}(q(Z)\mid\mid P(Z\mid X,\theta))}\\ \end{aligned}\tag{63}$$

当我们知道超参数 $\theta$ 时，上式中 evidence 应是固定的，因为 $\log P(X\mid \theta) = \log \sum_Z P(X, Z\mid \theta)$，虽然这个值通常求不出来。

变分推断的目标是通过最小化 $\mathrm{KL}(q(Z) \mid \mid P(Z\mid X,\theta))$ 来寻找与后验分布 $P(Z\mid X,\theta)$ 最相似的变分分布 $q(Z)$。 q(Z)^* = \arg \min_{q(Z)} \mathrm{KL}(q(Z) \mid \mid P(Z\mid X,\theta))\tag{64} 后验分布 $P(Z\mid X,\theta)$ 太复杂，直接估计其密度很困难，但利用 KL 散度和 ELBO 的和为常数，可以转而求

$$\begin{aligned} q(Z)^* =& \arg \min_{q(Z)} \mathrm{KL}(q(Z)\mid \mid P(Z\mid X,\theta)) \\ =&\arg \max_{q(Z)} \mathrm{ELBO}\\ =&\arg \max_{q(Z)} \int_Z q(Z)\log \frac{P(X,Z\mid \theta)}{q(Z)}\ \mathrm{d} Z\\ =&\arg \max_{q(Z)} \int_Z q(Z)\log P(X,Z\mid \theta) \ \mathrm{d}Z - \int_Z q(Z)\log q(Z) \ \mathrm{d} Z\\ =&\arg \max_{q(Z)} E_{q(Z)}[\log P(X,Z\mid \theta)]-E_{q(Z)}[\log q(Z)] \end{aligned}\tag{65}$$

变分分布 $q(Z)$ 有多种参数化方法，要求参数化后的 $q(Z)$ 使得上述优化问题容易求解，一种常用的方法是假设 $q(Z)$ 对 $Z=(Z_1,Z_2,\dots,Z_d)$ 的所有分量 $Z_j$ 都是相互独立的（实际是条件独立于参数），即满足 q(Z) = q(Z_1)\cdot q(Z_2)\cdots q(Z_d)\tag{66} 这时的变分分布被称为满足平均场（mean field）假设。

KL 散度的最小化或 ELBO 的最大化实际是在平均场的集合，即满足独立假设的分布集合 $Q=\{q(Z) \mid q(Z) =\prod_{j=1}^d q(Z_j)\}$ 之中进行的 q(Z)^* = \arg \max_{q(Z)\in Q} E_{q(Z)}[\log P(X,Z\mid \theta)]-E_{q(Z)}[\log q(Z)] \tag{67}Reference：[intermediate_vb, PRML chapter 10] 现在我们将目标函数重新写为

$$\begin{aligned} \mathrm{ELBO}=&\int q_{\phi}(\mathbf{z}) \log \left(\frac{p(\mathbf{x},\mathbf{z})}{q_{\phi}(\mathbf{z})}\right)\mathrm{d}\mathbf{z}\\ =&\int q_{\phi}(\mathbf{z}) \log (p(\mathbf x, \mathbf z))\mathrm d \mathbf z - \int q_{\phi}(\mathbf z)\log(q_{\phi}(\mathbf z))\mathrm d \mathbf z\\ =&\underbrace{\int \prod_{i=1}^M q_i(z_i)\log(p(\mathbf x, \mathbf z))\mathrm d \mathbf z}_{-H(q, p)} + \underbrace{\left(-\int \prod_{j=1}^M q_j (z_j) \sum_{i=1}^M \log (q_i(z_i))\mathrm d \mathbf z\right)}_{H(q)} \end{aligned}\tag{68}$$

首先考虑第一部分 $-H(q,p)$

$$\begin{aligned} -H(q,p)=&\int \prod_{i=1}^M q_i (z_i)\log (p(\mathbf x, \mathbf z)) \mathrm d \mathbf z \\ =&\int_{Z_1}\int_{Z_2}\cdots\int_{Z_M} \prod_{i=1}^M q_i(z_i)\log(p(\mathbf x, \mathbf z))\mathrm d \mathbf{z}_1 \mathrm d \mathbf{z}_2 \cdots \mathrm d \mathbf{z}_M \end{aligned} \tag{69}$$

只考虑其中一项 $q_j(z_j)$

$$\begin{aligned} -H(q,p)_j = &\underset{Z_j}{\int} q_j(z_j)\left(\underset{Z_{i\neq j}}{\int \cdots \int} \prod_{i\neq j}^M q_i(z_i) \log (p(\mathbf x, \mathbf z)) \prod_{i\neq j}^M \mathrm d z_i\right)\mathrm d z_j \\ =&\underset{Z_j}{\int} q_j(z_j) \mathbb{E}_{\mathbf{z} \backslash z_j}[\log (p(\mathbf x, \mathbf z))] \mathrm{d} z_j \end{aligned} \tag{70}$$

再考虑第二部分 $H(q)$

$$\begin{aligned} H(q)=&-\int \left(\prod_{i=1}^M q_i (z_i)\right) \sum_{i=1}^M \log (q_i(z_i))\mathrm d \mathbf z\\ =&\sum_{i=1}^M \left(-\underset{Z_i}{\int} q_i(z_i) \log (q_i(z_i)) \mathrm{d} z_i\right)\\ =& \sum_{i=1}^M H(q_i(z_i)) \end{aligned}\tag{71}$$

仅考虑其中一项 $q_j(z_j)$

$$\begin{aligned} H(q)_j =& -\underset{Z_j}{\int} q_j(z_j) \log (q_j(z_j))\mathrm{d} z_j + \mathrm{Const.}\\ =&H(q_j(z_j))+\mathrm{Const.} \end{aligned}\tag{72}$$

针对 ELBO 只考虑优化 $q_j$

$$\begin{aligned} \mathrm{ELBO}(q_j)=& -H(q,p)_j + H(q)_j\\ =&\underset{Z_j}{\int} q_j(z_j) \mathbb{E}_{\mathbf{z} \backslash z_j}[\log (p(\mathbf{x},\mathbf{z}))]\mathrm{d} z_j-\underset{Z_j}{\int} q_j(z_j) \log (q_j(z_j))\mathrm{d} z_j +\mathrm{Const. 1} \\ =& \underset{Z_j}{\int} q_j(z_j) \log{\tilde{p}(\mathbf{x},z_j)} \mathrm{d} z_j - \underset{Z_j}{\int} q_j(z_j) \log (q_j(z_j))\mathrm{d} z_j +\mathrm{Const. 2}\\ =& \underset{Z_j}{\int} q_j(z_j)\log \left[\frac{\tilde{p}(\mathbf{x},z_j)}{q_j(z_j)}\right]\mathrm{d} z_j + \mathrm{Const. 2} \\ =& -\mathbb{KL}\left(q_j(z_j) \mid \mid \tilde{p}(\mathbf{x},z_j) \right) + \mathrm{Const. 2} \end{aligned} \tag{73}$$

这里我们定义了一个新分布 $\tilde{p}(\mathbf{x},z_j)$ \log \tilde{p}(\mathbf{x},z_j) = \mathbb{E}_{\mathbf{z} \backslash z_j}[\log (p(\mathbf{x},\mathbf{z}))] + \mathrm{Const.} \tag{74} 因此我们可以通过最小化下述 KL 散度来最大化 ELBO，而 KL 散度的性质可知其值为零时最小，即 $q_j^* = \tilde{p}(\mathbf{x},z_j)$ \log q_j^*(z_j) = \mathbb{E}_{\mathbf{z}\backslash z_j}[\log(p(\mathbf{x},\mathbf{z}))] + \mathrm{Const.} \tag{75}注意此处的 $\exp(\mathbb{E}_{\mathbf{z}\backslash z_j}[\log(p(\mathbf{x},\mathbf{z}))])$ 是伪概率分布（pseudo distribution），只能满足概率分布的非负性而不能保证具有归一性，常数项为归一化常数 $\int \exp(\mathbb{E}_{\mathbf{z}\backslash z_j}[\log(p(\mathbf{x},\mathbf{z}))]) \mathrm{d} z_j$，保证 $\tilde{p}$ 的归一性和非负性，因此有 q_j^*(z_j)=\frac{\exp(\mathbb{E}_{\mathbf{z}\backslash z_j}[\log(p(\mathbf{x},\mathbf{z}))])}{\int \exp(\mathbb{E}_{\mathbf{z}\backslash z_j}[\log(p(\mathbf{x},\mathbf{z}))]) \mathrm{d} z_j} \tag{76}

2.3 Gaussian-Gamma

可观测变量为 $\mathcal{D = \{x_1,\dots, x_n\}}$，似然为

$$\begin{aligned} p(\mathcal{D}\mid \mu, \tau) &= \prod_{i=1}^n \left(\frac{\tau}{2\pi}\right)^{\frac{1}{2}}\exp \left(\frac{-\tau}{2}(x_i - \mu)^2\right)\\ &= \left(\frac{\tau}{2\pi}\right)^{\frac{n}{2}} \exp \left(\frac{-\tau}{2} \sum_{i=1}^n (x_i - \mu)^2\right) \end{aligned}\tag{77}$$

假设先验为

$$\begin{aligned} p(\mu\mid \tau) &= \mathcal{N}(\mu_0, (\lambda_0 \tau)^{-1})\propto \exp \left(\frac{-\lambda_0 \tau}{2}(\mu-\mu_0)^2\right)\\ p(\tau) &= \mathrm{Gamma}(\tau\mid a_0,b_0) \propto \tau^{a_0-1}\exp(-b_0 \tau) \end{aligned}\tag{78}$$

利用共轭性质可以计算解析后验（在2.4节介绍）

$$\begin{aligned} p(\mu,\tau \mid \mathcal{D}) \propto& p(\mathcal{D}\mid \mu, \tau)p(\mu \mid \tau) p(\tau)\\ =& \mathcal{N} (\mu_n, (\lambda_n \tau)^{-1}) \mathrm{Gamma}(\tau \mid a_n, b_n)\\ \end{aligned}\tag{79}$$

此处

$$\begin{aligned} \mu_n &= \frac{\lambda_0 \mu_0 + n\bar x}{\lambda_0 + n}\\ \lambda_n &= \lambda_0 +n \\ a_n &= a_0 + \frac{n}{2}\\ b_n &= b_0 + \frac{1}{2} \sum_{i=1}^n (x_i - \bar x)^2 + \frac{\lambda_0 n (\bar x -\mu_0)^2}{2(\lambda_0 + n)} \end{aligned} \tag{80}$$

但是如果我们不能计算其解析后验，可用变分推断来近似其后验。假设变分分布 $q(\mathbf{z})$ 为

$$\begin{aligned} q(\mu, \tau) &= q_{\mu}(\mu) q_{\tau} (\tau) \end{aligned}\tag{81}$$

利用式（75）得出的结论

$$\begin{aligned} \log q_{\mu}^*(\mu)&= \mathbb{E}_{q_{\tau}(\tau)} \left[\log p(\mu, \tau, \mathcal{D})\right]\\ &= \mathbb{E}_{q_{\tau}(\tau)} \left[\log p(\mathcal{D} \mid \mu, \tau) + \log p(\mu\mid \tau)\right] + \mathrm{Const.}\\ &= \mathbb{E}_{q_{\tau}(\tau)} \left[\frac{n}{2} \log \tau - \frac{\tau}{2} \sum_{i=1}^n (x_i - \mu)^2 - \frac{\lambda_0 \tau}{2} (\mu- \mu_0)^2\right] + \mathrm{Const.}\\ &= -\frac{1}{2} \mathbb{E}_{q_{\tau}}[\tau] \underbrace{\left[\sum_{i=1}^n (x_i - \mu)^2 + \lambda_0 (\mu-\mu_0)^2 \right]}_{\text{terms taking out of } \tau} + \mathrm{Const.} \end{aligned} \tag{82}$$

将中括号内式子展开，形成高斯分布 $\mathcal{N}(\mu; \mu^*, \tau^*)$ 的形式（如果一个连续随机变量的对数概率密度函数是关于该变量的严格负二次函数，并且该函数可以被正规化（即对应的密度可积且积分有限），那么该随机变量必定服从高斯分布）

$$\begin{aligned} \sum_{i=1}^n\left(x_i-\mu\right)^2+\lambda_0\left(\mu-\mu_0\right)^2 & =n \mu^2-2 n \mu \bar{\mathbf{x}}+\lambda_0 \mu^2-2 \lambda_0 \mu_0 \mu+\mathrm { Const. } \\ & =\left(n+\lambda_0\right) \mu^2-2 \mu\left(n \overline{\mathbf{x}}+\lambda_0 \mu_0\right) + \mathrm { Const. }\\ & =\left(n+\lambda_0\right)\left(\mu^2-\frac{2 \mu\left(n \overline{\mathbf{x}}+\lambda_0 \mu_0\right)}{n+\lambda_0}\right)+ \mathrm { Const. }\\ & =\left(n+\lambda_0\right)\left(\mu-\frac{n \overline{\mathbf{x}}+\lambda_0 \mu_0}{n+\lambda_0}\right)^2+\mathrm { Const. } \end{aligned}\tag{83}$$

因此我们有

$$\begin{aligned} \log q_\mu^*(\mu) & =-\frac{\mathbb{E}_{q_\tau}[\tau]}{2}\left[\sum_{i=1}^n\left(x_i-\mu\right)^2+\lambda_0\left(\mu-\mu_0\right)^2\right]+\mathrm { Const. } \\ & =-\frac{\mathbb{E}_{q_\tau}[\tau]\left(n+\lambda_0\right)}{2}\left(\mu-\frac{n \overline{\mathbf{x}}+\lambda_0 \mu_0}{n+\lambda_0}\right)^2+\mathrm { Const. }\\ & =-\frac{1}{2} \underbrace{\mathbb{E}_{q_\tau}[\tau]\left(n+\lambda_0\right)}_{\tau^*}\left(\mu-\underbrace{\frac{n \overline{\mathbf{x}}+\lambda_0 \mu_0}{n+\lambda_0}}_{\mu^*}\right)^2+\mathrm { Const. } \\ \Longrightarrow q_\mu^*(\mu) & =\mathcal{N}\left(\frac{n \overline{\mathbf{x}}+\lambda_0 \mu_0}{n+\lambda_0}, \mathbb{E}_{q_\tau}[\tau]\left(n+\lambda_0\right)\right) \quad \because-\frac{\tau}{2}(x-\mu)^2 \end{aligned}\tag{84}$$

利用式（82），去掉期望符号 $\mathbb{E}_{q_{\tau}}[\cdot]$，我们还可以得到 $p(\mu \mid \mathcal{D}, \tau)$（注意，删掉期望值就是原分布的后验，因为 $p(\mathcal{D}, \tau)$ 在常数项里）

$$\begin{aligned} \log p(\mathcal{D} \mid \mu, \tau)+\log p(\mu \mid \gamma) & =\underbrace{-\frac{\tau}{2} \sum_{i=1}^n\left(x_i-\mu\right)^2}_{\log (p(\mathcal{D} \mid \mu, \tau))} \underbrace{-\frac{\lambda_0 \tau}{2}\left(\mu-\mu_0\right)^2}_{\log p(\mu \mid \gamma)}+\mathrm { Const. } \\ & =-\frac{\tau}{2}\left[\sum_{i=1}^n\left(x_i-\mu\right)^2+\lambda_0\left(\mu-\mu_0\right)^2\right]+\mathrm { Const. } \\ & =-\frac{\tau\left(n+\lambda_0\right)}{2}\left(\mu-\frac{n \overline{\mathbf{x}}+\lambda_0 \mu_0}{n+\lambda_0}\right)^2+\mathrm { Const. }\\ \Longrightarrow p(\mu \mid \mathcal{D}, \tau) & =\mathcal{N}\left(\frac{n \overline{\mathbf{x}}+\lambda_0 \mu_0}{n+\lambda_0}, \tau\left(n+\lambda_0\right)\right) \end{aligned} \tag{85}$$

同理我们可以计算 $\log q_{\tau}^* (\tau)$

$$\begin{aligned} \log q_\tau^*(\tau) & =\mathbb{E}_{q_\mu}[\log p(\mu, \tau, \mathcal{D})] \\ & =\mathbb{E}_{q_\mu}[\log p(\mathcal{D} \mid \mu, \tau)+\log p(\mu \mid \tau)+\log p(\tau)]+\mathrm { Const. } \\ & =\mathbb{E}_{q_\mu}[\underbrace{\frac{n}{2} \log (\tau)-\frac{\tau}{2} \sum_{i=1}^n\left(x_i-\mu\right)^2}_{\log p(\mathcal{D} \mid \mu, \tau)} \underbrace{-\frac{\lambda_0 \tau}{2}\left(\mu-\mu_0\right)^2}_{\log p(\mu \mid \gamma)} \underbrace{+\left(a_0-1\right) \log (\tau)-b_0 \tau}_{\log p(\tau)}]+\mathrm { Const. }\\ & = \frac{n}{2}\log \tau + (a_0-1)\log \tau - b_0 \tau -\frac{\tau}{2}\mathbb{E}_{q_{\mu}(\mu)}\left[\sum_{i=1}^n (x_i -\mu)^2 + \lambda_0(\mu-\mu_0)^2\right] + \mathrm{Const.} \\ & = \left(\underbrace{\frac{n}{2}+ a_0}_{a_n} -1\right) \log \tau - \tau \left(\underbrace{b_0 + \frac{1}{2} \mathbb{E}_{q_{\mu}(\mu)}\left[\sum_{i=1}^n (x_i - \mu)^2 + \lambda_0 (\mu - \mu_0)^2 \right]}_{b_n} \right) + \mathrm{Const.} \\ &\Longrightarrow q_{\tau}^* (\tau) = \mathrm{Gamma}(a_n, b_n) \end{aligned}\tag{86}$$

可以将 $b_n$ 展开写为

$$\begin{aligned} b_n & =b_0+\frac{1}{2} \mathbb{E}_{q_\mu}\left[\sum_{i=1}^n\left(x_i-\mu\right)^2+\lambda_0\left(\mu-\mu_0\right)^2\right] \\ & =b_0+\frac{1}{2} \mathbb{E}_{q_\mu}\left[-2 \mu n \bar{x}+n \mu^2+\lambda_0 \mu^2-2 \lambda_0 \mu_0 \mu\right]+\sum_{i=1}^nx_i^2+\lambda_0 \mu_0^2 \\ & =b_0+\frac{1}{2}\left[\left(n+\lambda_0\right) \mathbb{E}_{q_\mu}\left[\mu^2\right]-2\left(n \bar{x}+\lambda_0 \mu_0\right) \mathbb{E}_{q_\mu}[\mu]+\sum_{i=1}^nx_i^2+\lambda_0 \mu_0^2\right] \end{aligned}\tag{87}$$

因为前面已经知道 $q_{\mu}(\mu)$ ，可以计算这里的 $\mathbb{E}_{q_{\mu}}[\mu^2]$ 和 $\mathbb{E}_{q_{\mu}}[\mu]$。

同样地，也可以轻易地获得原分布的后验 $p(\tau\mid \mathcal{D},\mu)$

$$\begin{aligned} & \log p(\tau \mid \mathcal{D}, \mu)= \log (p(\mathcal{D} \mid \mu, \tau))+\log p(\mu \mid \tau)+\log p(\tau)+\text { Const. } \\ &= \underbrace{\frac{n}{2} \log (\tau)-\frac{\tau}{2} \sum_{i=1}^n\left(x_i-\mu\right)^2}_{\log (p(\mathcal{D} \mid \mu, \tau))} \underbrace{-\frac{\lambda_0 \tau}{2}\left(\mu-\mu_0\right)^2}_{\log p(\mu \mid \gamma)} \underbrace{+\left(a_0-1\right) \log (\tau)-b_0 \tau}_{\log p(\tau)}+\text { Const. } \\ &=\underbrace{\left(\frac{n}{2}+a_0\right.}_{a_n}-1) \log (\tau)-\tau(\underbrace{b_0+\frac{1}{2} \sum_{i=1}^n\left(x_i-\mu\right)^2+\lambda_0\left(\mu-\mu_0\right)^2}_{b_n})+\text { Const. } \\ & \Longrightarrow p(\tau \mid \mathcal{D}, \mu)= \operatorname{Gamma}\left(a_n, b_n\right) \\ & \qquad \qquad \qquad \qquad \begin{cases}a_n & =\frac{n}{2}+a_0 \\ b_n & =b_0+\frac{1}{2} \sum_{i=1}^n\left(x_i-\mu\right)^2+\lambda_0\left(\mu-\mu_0\right)^2\end{cases} \end{aligned}\tag{88}$$

2.4 指数族分布

2.4.1 概览

给定先验和似然都是指数族分布，则他们形成一个共轭对，则变分推断（平均场近似）有下列更新公式

$$\eta_j = \mathbb{E}_{q(\mathbf{z}\backslash z_j\mid \cdot)}[\eta_{\mathrm{post}}(\mathbf{z}\backslash z_j)]\tag{89}$$

这里的 $\eta_{\mathrm{post}}(\mathbf{z}\backslash z_j)$ 是和后验分布 $p(z_j \mid \cdot)$ 相关的自然参数。

和通用的更新公式相比 \log q_i^*(z_i)=\mathbb{E}_{i\neq j}[\log p(\mathbf{x}, \mathbf{z})]\tag{90} 使用指数族更新公式更加直接方便。

2.4.2 指数族

指数族分布通常用自然参数 $\eta$ 表示为下列形式

$$\begin{aligned} &h(x)\exp(T(x)^{\top}\eta - A(\eta)) \\\\ &= \underbrace{\exp(-A(\eta))}_{\text{normalization}} h(x)\exp(T(x)^{\top}\eta)\\ &\Longrightarrow \exp(-A(\eta)){\int}h(x)\exp(T(x)^{\top}\eta)\mathrm{d}x = 1\\ &\Longrightarrow {\int}h(x)\exp(T(x)^{\top}\eta) \mathrm{d}x= \exp(A(\eta))\\ &\Longrightarrow A(\eta) =\log{\int}h(x) \exp{T(x)^{\top}\eta} \mathrm{d}x \end{aligned}\tag{91}$$

• $\eta$-自然参数（natural parameter）
• $T(x)$-充分统计量（sufficient statistic）
• $A(\eta)$-对数配分函数（log-partition function）：主要作用是归一化，为 $\eta$ 的凸函数
• $h(x)$-基测度（base measure）：不依赖于参数 $\eta$ 的部分，用于调整使得积分有意义

指数族分布具有易求最大似然估计的性质（可用下列高斯分布验证，其求解易于原参数形式）

$$\begin{aligned} &\arg \max_{\eta} \left[\log p(X\mid \eta) \right]\\ &= \arg \max_{\eta} \left[\log \prod_{i=1}^n p(x_i \mid \eta) \right]\\ &= \arg \max_{\eta} \left[\log \left\{\prod_{i=1}^n h(x_i) \exp\left(\sum_{i=1}^n T(x_i)^{\top} \eta - n A(\eta)\right)\right\}\right]\\ &= \arg \max_{\eta} \underbrace{\left[\sum_{i=1}^n T(x_i)^{\top} \eta - n A(\eta) \right]}_{\mathcal{L}(\eta)}\\ &\Longrightarrow \frac{\partial \mathcal{L}(\eta)}{\partial \eta}= \sum_{i=1}^n T(x_i) - n A'(\eta) =0\\ &\Longrightarrow A'(\eta) =\sum_{i=1}^n \frac{T(x_i)}{n} \end{aligned}\tag{92}$$

从另一个角度来看，指数分布族具有性质：对数规范化因子 $A(\eta)$ 对自然参数 $\eta$ 的导数等于充分统计量 $T(x)$ 的数学期望，这是任何情况都成立的

$$\begin{aligned} \frac{\mathrm{d}}{\mathrm{~d} \eta} A(\eta) & =\frac{\mathrm{d}}{\mathrm{~d} \eta} \log \int h(x) \exp \left\{\eta^{\mathrm{T}} T(x)\right\} \mathrm{d} x \\ & =\frac{\int T(x) \exp \left\{\eta^{\mathrm{T}} T(x)\right\} h(x) \mathrm{d} x}{\underbrace{\int h(x) \exp \left\{\eta^{\mathrm{T}} T(x)\right\} \mathrm{d} x}_{A(\eta)\text{ 见式(91)}}} \text{(交换积分与微分的顺序)}\\ & =\int T(x) \exp \left\{\eta^{\mathrm{T}} T(x)-A(\eta)\right\} h(x) \mathrm{d} x \\ & =\int T(x) p(x \mid \eta) \mathrm{d} x \\ & =\mathbb{E}[T(x)] \end{aligned}\tag{93}$$

再对一阶导数求导，得到Hessian矩阵

$$\begin{aligned} \frac{\mathrm{d}A^2(\eta)}{\mathrm{d}^2 \eta} &= \frac{\mathrm{d} \int T(x) p(x\mid \eta)\mathrm{d}x}{\mathrm{d}\eta}\\ &=\int T(x)\triangledown_{\eta} p(x\mid \eta)\mathrm{d}x \end{aligned}\tag{94}$$

计算

$$\begin{aligned} \triangledown_{\eta}p(x\mid \eta) &= \triangledown_{\eta}(h(x)\exp{(\eta^{\top}T(x)-A(\eta) )})\\ &=p(x\mid \eta){(T(x) - \triangledown_{\eta}A(\eta))}\\ &=p(x\mid \eta)(T(x)-\mathbb{E}[T(x)]) \text{ 由式 (93)} \end{aligned}\tag{95}$$

代入式 (94)可得

$$\begin{aligned} \frac{\mathrm{d}A^2(\eta)}{\mathrm{d}^2 \eta} &= \int T(x)p(x\mid \eta)(T(x)-\mathbb{E}[T(x)])^{\top}\mathrm{d}x\\ &= \int (T(x)T(x)^{\top} - T(x)\mathbb{E}[T(x)]^{\top})p(x\mid \eta)\mathrm{d}x\\ &=\mathbb{E}[T(x)T(x)^{\top}] - \mathbb{E}[T(x)]\mathbb{E}[T(x)]^{\top} \end{aligned}\tag{96}$$

这就是协方差矩阵的定义公式，即 $A(\eta)$ 的二阶导数是充分统计量的协方差矩阵。进而可知，对任意非零向量 $v$，协方差矩阵 $\Sigma=\text{Cov}(T(x))$ 满足

$$\begin{aligned} v^{\top}\Sigma v &= v^{\top} \mathbb{E}[(T(x)-\mathbb{E}[T(x)]) (T(x)-\mathbb{E}[T(x)])^{\top}]v\\ &= \mathbb{E}[v^{\top} (T(x)-\mathbb{E}[T(x)]) (T(x)-\mathbb{E}[T(x)])^{\top}v]\\ &= \mathbb{E}[(v^{\top} (T(x) - \mathbb{E}[T(x)]))^2] \geq 0 \end{aligned}$$

由此可知 $\frac{\mathrm{d}A^2(\eta)}{\mathrm{d}^2 \eta}$ 为半正定矩阵，则 $A(\eta)$ 为凸函数。当使用最大似然估计来求解指数族分布的参数时，需要最小化负对数似然函数（NLL）：

$$\begin{aligned} \text{NLL}(\eta) &= -\eta^{\top} T(x)+A(\eta) - \log{h(x)} \end{aligned}\tag{97}$$

由于 $A(\eta)$ 是凸函数，且 $-\eta^T T(x)$ 是线性函数（也是凸函数），所以 NLL 也是凸函数。这意味着：指数族分布的最大似然估计问题，保证存在全局最优解，没有局部最优陷阱（如果优化的是常规参数 ordinary parameter 不一定能保证）。 这就是为什么逻辑回归、线性回归等模型训练如此稳定且高效的根本原因。

例如一维高斯分布，可以将其写为指数族分布的形式

$$\begin{aligned} \mathcal{N}\left(x ; \mu, \sigma^2\right) & =\left(2 \pi \sigma^2\right)^{-1 / 2} \exp \left(-\frac{(x-\mu)^2}{2 \sigma^2}\right) \\ & =\exp \left(-\frac{x^2-2 x \mu+\mu^2}{2 \sigma^2}-\frac{1}{2} \log \left(2 \pi \sigma^2\right)\right) \\ & =\exp \left(-\frac{1}{2 \sigma^2} x^2+\frac{\mu}{\sigma^2} x-\frac{\mu^2}{2 \sigma^2}-\frac{1}{2} \log \left(2 \pi \sigma^2\right)\right) \\ & =\exp \left(\left[\begin{array}{ll} x & x^2 \end{array}\right]\left[\begin{array}{ll} \frac{\mu}{\sigma^2} & -\frac{1}{2 \sigma^2} \end{array}\right]^{\top}-\frac{\mu^2}{2 \sigma^2}-\frac{1}{2} \log \left(2 \pi \sigma^2\right)\right) \end{aligned}\tag{98}$$

其中

$$\begin{aligned} T(x) &= \left[\begin{array}{ll} x & x^2 \end{array}\right] \\ \boldsymbol{\eta} &= \left[\begin{array}{ll} \eta_1 & \eta_2 \end{array}\right]\\ &= \left[\begin{array}{ll} \frac{\mu}{\sigma^2} & -\frac{1}{2\sigma^2} \end{array}\right]\\ \theta &= \left[\begin{array}{ll} \mu \\ \sigma^2\end{array}\right] = \left[\begin{array}{ll} \frac{-\eta_1}{2\eta_2} \\ \frac{-1}{2\eta_2}\end{array}\right] \end{aligned}\tag{99}$$

现在我们可以移除 $\mu$ 和 $\sigma$ 得到一维高斯分布的指数族分布形式

$$\begin{aligned} \mathcal{N}_{\text {nat}}(x, \boldsymbol{\eta}) & =\exp \left(\left[\begin{array}{ll} x & x^2 \end{array}\right]\left[\begin{array}{ll} \eta_1 & \eta_2 \end{array}\right]^{\top}-\frac{\mu^2}{2 \sigma^2}-\frac{1}{2} \log \left(2 \pi \sigma^2\right)\right) \\ & =\exp \left(\left[\begin{array}{ll} x & x^2 \end{array}\right]\left[\begin{array}{ll} \eta_1 & \eta_2 \end{array}\right]^{\top}-\frac{\left(\frac{-\eta_1}{2 \eta_2}\right)^2}{2\left(\frac{-1}{2 \eta_2}\right)}-\frac{1}{2} \log \left(2 \pi\left(\frac{-1}{2 \eta_2}\right)\right)\right) \\ & =\exp \left(T(x)^{\top} \boldsymbol{\eta}+\frac{\eta_1^2}{4 \eta_2}-\frac{1}{2} \log \left(\frac{2 \pi}{-2 \eta_2}\right)\right) \\ & =\exp \left(T(x)^{\top} \boldsymbol{\eta}+\underbrace{\frac{\eta_1^2}{4 \eta_2}+\frac{1}{2} \log \left(-2 \eta_2\right)-\frac{1}{2} \log (2 \pi)}_{A(\boldsymbol{\eta})}\right) \end{aligned}\tag{100}$$

2.4.3 共轭概率

共轭表示先验和后验是同种形式的概率分布，例如 \underbrace{p_{\eta_{\text{post}}}(\theta\mid \mathbf{x})}_{\text{same type}}\propto p(\mathbf{x}\mid \theta)\underbrace{p_{\eta_{\text{prior}}}(\theta)}_{\text{same type}}\tag{101} 使用指数族分布表示时，共轭的含义是先验和后验有相同的充分统计量 $T(\theta)$ 和 $h(\theta)$ （注意这里的 $\theta$ 是变量），不同的自然参数，即 $\eta_{\text{post}}, \eta_{\text{prior}}$ 以及不同的对数归一化因子。

证明：一个指数族分布的先验 $p(\theta | \alpha, \nu) = \frac{1}{Z(\alpha, \nu)} \exp\left( \alpha^T \theta - \nu A(\theta) \right)$ 必定有对应的似然使其拥有一个共轭的后验

$$\begin{aligned} p(\theta\mid x) &\propto p(x\mid \theta)p(\theta \mid \alpha,\nu)\\ &= \prod_{i=1}^N \left[h(x_i)\exp{(\theta^{\top} T(x_i)-A(\theta))}\right] \cdot \frac{1}{Z(\alpha, \nu)} \exp\left( \alpha^T \theta - \nu A(\theta) \right) \\ &\propto \exp\left( \theta^T \sum_{i=1}^N T(x_i) - N \cdot A(\theta) \right) \cdot \exp\left( \alpha^T \theta - \nu A(\theta) \right) \\ &= \exp\left( (\underbrace{\alpha + \sum_{i=1}^N T(x_i)}_{\alpha_{\text{new}}})^T \theta - (\underbrace{\nu + N}_{\nu_{\text{new}}}) A(\theta) \right) \end{aligned}\tag{102}$$

观察最后的结果，会发现后验分布的形式与先验分布完全一致，只是参数发生了变化，$\alpha' = \alpha + \sum_{i=1}^N T(x_i)$，$\nu' = \nu + N$。

2.4.4 变分推断

隐变量 $\beta$ 的后验分布，注意下式中的 $h(\beta)$ 和 $T(\beta)$ 是相同的，因为设置变分分布是和真实条件后验分布同一种分布，这是同分布的结果

$$\begin{aligned} &p(\beta \mid z, x)\\ &= h(\beta) \exp \left(T(\beta)^{\top} \eta (z, x) - A_g(\eta(z,x))\right)\\ &\approx q(\beta \mid \lambda) =h(\beta)\exp\left(T(\beta)^{\top}\lambda - A_g(\lambda) \right) \end{aligned}\tag{103}$$

隐变量 $z$ 的后验分布

$$\begin{aligned} &p(z\mid \beta, x)\\ &= h(z) \exp\left(T(z)^{\top} \eta(\beta,x) - A_l(\eta(\beta,x))\right)\\ &\approx q(z\mid \phi) =h(z)\exp\left(T(z)^{\top} \phi - A_l(\phi)\right) \end{aligned}\tag{104}$$

固定 $\phi$ ，优化 $\lambda$ ，每一步去除无关项（损失函数见式（65））

$$\begin{aligned}\\ &\mathcal{L}(\lambda, \phi)\\ &= E_{q(z,\beta)}[\log p(x,z,\beta)] - E_{q(z,\beta)}[\log q(z,\beta)]\\ &= E_{q(z,\beta)}[\log p(\beta \mid x, z)+\log p(z,x)] - E_{q(z,\beta)} [\log q(\beta)] - E_{q(z,\beta)} [\log q(z)] \\ &= E_{q(z,\beta)} [\log p(\beta \mid x, z)] - E_{q(z,\beta)} [\log q(\beta)]\\ &= E_{q(z,\beta)} [\log h(\beta)] + E_{q(z,\beta)}[T(\beta)^{\top} \eta(z,x)] - E_{q(z,\beta)} [A_g(\eta(x,z))] \\ &\quad - E_{q(z,\beta)} [\log h(\beta)] - E_{q(z,\beta)} [T(\beta)^{\top} \lambda] + E_{q(z,\beta)} [A_g(\lambda)]\\ &= E_{q(\beta)}[T(\beta)^{\top}] E_{q(z)}[\eta(z,x)] - E_{q(z)}[A_g(\eta(x,z))] - E_{q(\beta)} [ T(\beta)^{\top}\lambda] + A_g(\lambda) \quad \text{using } \frac{\partial A(\eta)}{\partial \eta} = E_{p(x\mid \eta)}[T(x)] \\ &= A_g'(\lambda)^{\top} E_{q(z)}[\eta (z,x)] - \lambda A_g' (\lambda)^{\top} + A_g(\lambda) \quad \text{taking partial derivative with respect to } \lambda\\ &\Rightarrow A_g''(\lambda)^{\top} E_{q(z)}[\eta(z,x)] - A_g'(\lambda)^{\top} - \lambda A_g''(\lambda)^{\top} + A_g' (\lambda)=0\\ &\Rightarrow A_g'' (\lambda)^{\top} (E_{q(z)}[\eta(x,z)]-\lambda) =0\\ &\Longrightarrow \lambda =E_{q(z)}[\eta(x,z)] \qquad \text{where } q(z) =q(z\mid \phi) \end{aligned}\tag{105}$$

同理，固定 $\lambda$ ，优化 $\phi$，可得到更新公式 $\phi=E_{q(\beta \mid \lambda)}[\eta(\beta, x)]$

2.5 基于变分推断的LDA参数学习

![[Pasted image 20241129221401.png|700]]

For each topic $k$ :

$$\boldsymbol{\beta}_k \sim \operatorname{Dir}(\xi, \ldots, \xi) \quad \text{for } k \in\{1, \ldots, K\}\tag{106}$$

For each document $d$ :

$$\boldsymbol{\theta}_{\boldsymbol{d}} \sim \operatorname{Dir}(\alpha, \ldots, \alpha)\tag{107}$$

For each word $w \in\{1, \ldots, N\}$ :

$$\begin{aligned} & z_{d,n} \sim \operatorname{Mult}\left(\boldsymbol{\theta}_d\right) \\ & w_{d,n} \sim \operatorname{Mult}\left(\beta_{z_{d,n}}\right) \end{aligned}\tag{108}$$

因为先验和似然是共轭的，而选取变分分布时应该选取和条件后验相同的分布有利于计算，所以此时选取和先验同样的分布（可以选择不同分布，但是会带来计算复杂性） q(\boldsymbol{\beta}_k)=\mathrm{Dir}(\boldsymbol{\lambda}_k), \quad q(\boldsymbol{\theta}_d)=\mathrm{Dir}(\boldsymbol{\gamma}_d), \quad q(z_{d,n})= \mathrm{Mult}(\boldsymbol{\phi}_{d,n})\tag{109} $z_{d,n}$ 的变分分布选取基于两个理由，一是离散的、有限类别的变量，Multinomial（或 Categorical）分布是唯一合理的描述方式；二是由式（75）可知

$$\begin{aligned} &\ln q^*(z_{d,n}) = \mathbb{E}_{q(\theta, \beta)}[\ln p(z_{d,n}|\theta_d) + \ln p(w_{d,n}|z_{d,n}, \beta)] + C\\ & \Rightarrow \ln q^*(z_{d,n} = k) = \mathbb{E}[\ln \theta_{d,k}] + \mathbb{E}[\ln \beta_{k, w_{d,n}}] + C\\ & \Rightarrow q^*(z_{d,n} = k) \propto \exp\left( \mathbb{E}[\ln \theta_{d,k}] + \mathbb{E}[\ln \beta_{k, w_{d,n}}] \right) \end{aligned}\tag{110}$$

这就完全符合 Multinomial 分布 的定义。

2.5.1 基于指数族分布的变分推断

2.5.1.1 更新 $\boldsymbol{\phi}_{d,n}$

首先找到后验 $p(z_{d,n}=k\mid \boldsymbol{\theta}_d,\boldsymbol{\varphi}_k,w_{d,n})$ 的自然参数

$$\begin{aligned} p(z_{d,n}=k \mid \boldsymbol{\theta}_d,\boldsymbol{\beta}_{1:K},w_{d,n}) &\propto p(z_{d,n}=k \mid \boldsymbol{\theta}_d) \cdot p(w_{d,n}\mid z_{d,n}=k,\boldsymbol{\beta}_{1:K}) \\ &= \theta_{d,k} \cdot \beta_{k,w_{d,n}} \\ &= \exp{\left(\underbrace{(\log \theta_{d,k} + \log \beta_{k,w_{d,n}})}_{\eta_l(\boldsymbol{\theta}_{d,k}, \boldsymbol{\beta}_{1:K},w_{d,n})} \cdot \underbrace{1}_{T(z_{d,n})} \right)} \end{aligned}\tag{111}$$

使用正常的多项式分布可以表达为 p(z_{d,n}\mid \boldsymbol{\theta}_d,\boldsymbol{\beta}_{1:K},w_{d,n}) = \mathrm{Mult}(\theta_{d,1}\cdot\beta_{1,w_{d,n}},\dots,\theta_{d,k}\cdot\beta_{k,w_{d,n}})\tag{112} 利用更新公式可知

$$\begin{aligned} \eta(\phi_{d,n}^k)&=\log (\phi_{d,n}^k) \quad \text{多项式分布的自然参数形式}\\ &= E_{q(\boldsymbol{\theta}_{d},\boldsymbol{\beta}_{k})}[\eta_l(\boldsymbol{\theta}_d, \boldsymbol{\beta}_{1:K},w_{d,n})] + \mathrm{Const.} \quad \text{由式（75）的最优更新公式} \\ &= E_{q(\boldsymbol{\theta}_d)}[\log (\theta_{d,k})]+E_{q(\boldsymbol{\beta}_k)}[\log (\beta_{k,w_{d,n}})] + \mathrm{Const.}\\ &= \Psi(\gamma_{d,k})-\Psi\left(\sum_{l=1}^K \gamma_{d,l}\right)+\Psi(\lambda_{k,w_{d,n}})-\Psi\left(\sum_{v=1}^V \lambda_{k,v}\right) + \mathrm{Const.}\\ &\Longrightarrow \phi_{d,n}^k \propto \exp{\left[ \Psi(\gamma_{d,k})-\Psi\left(\sum_{l=1}^K \gamma_{d,l}\right)+\Psi(\lambda_{k,w_{d,n}})-\Psi\left(\sum_{v=1}^V \lambda_{k,v}\right)\right]}\\ &\quad \quad \quad \quad \propto \exp{\left[ \Psi(\gamma_{d,k})+\Psi(\lambda_{k,w_{d,n}})-\Psi\left(\sum_{v=1}^V \lambda_{k,v}\right)\right]}\\ \end{aligned}\tag{113}$$

2.5.1.2 更新 $\boldsymbol{\gamma}_d$

同样先推导后验 $p(\boldsymbol{\theta}_d \mid \mathbf{z}_d)$ 的表达式

$$\begin{aligned} p(\boldsymbol{\theta}_d \mid \mathbf{z}_d) &= p(\boldsymbol{\theta}_d \mid \boldsymbol{\alpha}) \prod_{n=1}^N p(z_{d,n} \mid \boldsymbol{\theta}_d)\\ &= \prod_{k=1}^K \left(\theta_{d,k}^{\alpha_k-1} \prod_{n=1}^N \theta_{d,k}^{\mathbb{1}(z_{d,n}=k)} \right)\\ &= \exp \left[\log \left(\prod_{k=1}^K \left(\theta_{d,k}^{\alpha_k -1} \prod_{n=1}^N \theta_{d,k}^{\mathbb{1}(z_{d,n}=k)} \right)\right)\right]\\ &= \exp \left[\sum_{k=1}^K \log \left(\theta_{d,k}^{\alpha_k-1}\prod_{n=1}^N \theta_{d,k}^{\mathbb{1}(z_{d,n}=k)} \right) \right]\\ &= \exp \left[\sum_{k=1}^K \left(\log \theta_{d,k}^{\alpha_k -1}+ \sum_{n=1}^N \log \left(\theta_{d,k}^{\mathbb{1}(z_{d,n}=k)} \right)\right) \right]\\ &= \exp \left[\sum_{k=1}^K \left((\alpha_k -1) \log \theta_{d,k} + \sum_{n=1}^N \mathbb{1}(z_{d,n}=k) \log \theta_{d,k} \right) \right]\\ &= \exp \left[\sum_{k=1}^K \left(\alpha_k -1 + \sum_{n=1}^N \mathbb{1}(z_{d,n}=k) \right) \log \theta_{d,k} \right]\\ &= \exp \left(\underbrace{\left[\begin{array}{ccc} \alpha_1 - 1 + n_1 \\ \cdots \\ \alpha_K - 1 + n_K \end{array}\right]^{\top}}_{\eta_l(\boldsymbol{\alpha}, \mathbf{z}_d)} \underbrace{\left[\begin{array}{ccc} \log \theta_{d,1} \\ \cdots \\ \log \theta_{d,K} \end{array}\right]}_{T(\boldsymbol{\theta}_d)} \right) \quad \text{by letting } n_k=\sum_{n=1}^N \mathbb{1}(z_{d,n}=k)\\ &= \mathrm{Dir} (\alpha_1+n_1, \dots, \alpha_K + n_K) \end{aligned}\tag{114}$$

接下来用变分分布 $q(\eta(\boldsymbol{\gamma}_d))= \mathrm{Dir}(\eta(\boldsymbol{\gamma}_d))$ 来近似 $p(\boldsymbol{\theta}_d \mid \mathbf{z}_d)$ ，利用更新公式

$$\begin{aligned} \eta(\boldsymbol{\gamma}_d)&= E_{q(\mathbf{z}_{d}\mid \boldsymbol{\phi}_{d})} [\eta_l (\boldsymbol{\alpha},\mathbf{z}_d)] \quad \text{狄利克雷分布的自然参数形式}\\ &= E_{q(\mathbf{z}_{d}\mid \boldsymbol{\phi}_{d})}[(\alpha_1-1+n_1) \dots (\alpha_K - 1+ n_K)] \end{aligned}\tag{115}$$

计算这个期望

$$\begin{aligned} E_{q(\mathbf{z}_{d}\mid \boldsymbol{\phi}_{d})}\left[\sum_{n=1}^N \mathbb{1}(z_{d,n}=k) \right] &= \sum_{n=1}^N E_{q(\mathbf{z}_{d}\mid \boldsymbol{\phi}_{d})}[\mathbb{1}(z_{d,n}=k)]\\ &= \sum_{n=1}^N q(z_{d,n}=k)\\ &= \sum_{n=1}^N \phi_{d,n}^k \end{aligned}\tag{116}$$

因此有

$$\begin{aligned} \eta(\boldsymbol{\gamma}_d) &= \left[\left(\alpha_1-1+\sum_{n=1}^N \phi_{d,n}^1\right) \dots \left(\alpha_K - 1 + \sum_{n=1}^N \phi_{d,n}^K\right) \right]\\ &\Longrightarrow \boldsymbol{\gamma}_d = \left[\left(\alpha_1 + \sum_{n=1}^N \phi_{d,n}^1\right) \dots \left(\alpha_K + \sum_{n=1}^N \phi_{d,n}^K\right)\right] \quad \text{（迪利克雷分布的自然参数为 } \eta_i = \gamma_i -1 \text{）}\\ &\quad \quad\quad= \boldsymbol{\alpha}+\sum_{n=1}^N \boldsymbol{\phi}_{d,n} \end{aligned}\tag{117}$$

2.5.1.3 更新 $\boldsymbol{\lambda}_k$

与 $\boldsymbol{\gamma}_d$ 更新公式类似，有 \boldsymbol{\lambda}_k = \xi + \sum_{d=1}^D \sum_{n=1}^N w_{d,n} \cdot \phi_{d,n}^k\tag{118}

2.5.2 基于展开式的推断

当超参数给定时，$\log p(X\mid \theta)$ 是常数，因此

$$\begin{aligned} q(Z)^* &= \arg\min_{q(Z)} \mathrm{KL}(q(Z)\|p(Z \mid X, \theta)) \\ &= \arg\max_{q(Z)} \mathrm{ELBO} \\ &= \arg\max_{q(Z)} \int_Z q(Z) \log \frac{p(X, Z \mid \theta)}{q(Z)} \, \mathrm{d}Z \\ &= \arg\max_{q(Z)} \int_Z q(Z) \log p(X, Z \mid \theta) \, \mathrm{d}Z - \int_Z q(Z) \log q(Z) \, \mathrm{d}Z \\ &= \arg\max_{q(Z)} E_{q(Z)}[\log p(X, Z \mid \theta)] - E_{q(Z)}[\log q(Z)] \end{aligned} \tag{119}$$

KL 散度的最小化或证据下界的最大化实际是在平均场的集合，即满足独立假设的分布集合 $Q = \{q(Z) \mid q(Z) = \prod_{j=1}^d q(Z_j) \}$ 之中进行的

$$\begin{aligned} &\log p\left( \mathbf{w}, \mathbf{z}, \varphi_{1:K}, \theta_{1:M} \mid \alpha, \beta \right)\\ &= \log \left\{ \left[ \prod_{m=1}^M p(\theta_m \mid \alpha) \right] \left[ \prod_{k=1}^K p(\varphi_k \mid \beta) \right] \left[ \prod_{m=1}^M \prod_{n=1}^{N_m} p(z_{mn} \mid \theta_m) \right] \left[ \prod_{m=1}^M \prod_{n=1}^{N_m} p(w_{mn} \mid \varphi_{1:K}, z_{mn}) \right] \right\} \\ &= \sum_{m=1}^M \log p(\theta_m \mid \alpha) + \sum_{k=1}^K \log p(\varphi_k \mid \beta) + \sum_{m=1}^M \sum_{n=1}^{N_m} \log p(z_{mn} \mid \theta_m) + \sum_{m=1}^M \sum_{n=1}^{N_m} \log p(w_{mn} \mid \varphi_{1:K}, z_{mn}) \end{aligned}\tag{120}$$

定义基于平均场的变分分布

$$\begin{aligned} q(\mathbf{z}, \varphi_{1:K}, \theta_{1:M} \mid \mu_{1:K}, \gamma_{1:M}, \{\eta_{mn}\}) &= \prod_{k=1}^{K} q(\varphi_k \mid \mu_k) \prod_{m=1}^{M} q(\theta_m \mid \gamma_m) \prod_{m=1}^{M} \prod_{n=1}^{N_m} q(z_{mn} \mid \eta_{mn}) \\ &= \prod_{k=1}^{K} \text{Dir}(\varphi_k \mid \mu_k) \prod_{m=1}^{M} \text{Dir}(\theta_m \mid \gamma_m) \prod_{m=1}^{M} \prod_{n=1}^{N_m} \text{Mult}(z_{mn} \mid \eta_{mn}) \end{aligned}\tag{121}$$

展开证据下界

$$\begin{aligned} \text{ELBO} &= E_{q(\mathbf{z}, \varphi_{1:K}, \theta_{1:M} \mid \mu_{1:K}, \gamma_{1:M}, \{ \eta_{mn} \})} \left[ \log p(\mathbf{w}, \mathbf{z}, \varphi_{1:K}, \theta_{1:M} \mid \alpha, \beta) \right] \\ &\quad - E_{q(\mathbf{z}, \varphi_{1:K}, \theta_{1:M} \mid \mu_{1:K}, \gamma_{1:M}, \{\eta_{mn} \})} \left[ \log q(\mathbf{z}, \varphi_{1:K}, \theta_{1:M} \mid \mu_{1:K}, \gamma_{1:M}, \{\eta_{mn}\}) \right] \\ &= \sum_{m=1}^{M} E_{q(\theta_m \mid \gamma_m)} \left[ \log p(\theta_m \mid \alpha) \right] + \sum_{k=1}^{K} E_{q(\varphi_k \mid \mu_k)} \left[ \log p(\varphi_k \mid \beta) \right] + \sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q(z_{mn}, \theta_m \mid \eta_{mn},\gamma_m)} \left[ \log p(z_{mn} \mid \theta_m) \right] \\ &\quad + \sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q(\varphi_{1:K}, z_{mn} \mid \mu_{1:K}, \eta_{mn})} \left[ \log p(w_{mn} \mid \varphi_{1:K}, z_{mn}) \right] - \sum_{k=1}^{K} E_{q(\varphi_k \mid \mu_k)} \left[ \log q(\varphi_k \mid \mu_k) \right] \\ &\quad - \sum_{m=1}^{M} E_{q(\theta_m \mid \gamma_m)} \left[ \log q(\theta_m \mid \gamma_m) \right] - \sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q(z_{mn} \mid \eta_{mn})} \left[ \log q(z_{mn} \mid \eta_{mn}) \right] \end{aligned}\tag{122}$$

第一项：

$$\begin{aligned} &\sum_{m=1}^{M} E_{q(\theta_m | \gamma_m)} \left[ \log p(\theta_m \mid \alpha) \right] \\ &= \sum_{m=1}^{M} E_{q(\theta_m | \gamma_m)} \left[ \log \left( \frac{\Gamma\left(\sum_{k=1}^{K} \alpha_k\right)}{\prod_{k=1}^{K} \Gamma(\alpha_k)} \prod_{k=1}^{K} \theta_{mk}^{\alpha_k - 1} \right) \right] \\ &= \sum_{m=1}^{M} \mathbb{E}_{q(\theta_m | \gamma_m)} \left[ \log \Gamma\left( \sum_{k=1}^{K} \alpha_k \right) - \sum_{k=1}^{K} \log \Gamma(\alpha_k) + \sum_{k=1}^{K} (\alpha_k - 1) \log \theta_{mk} \right] \\ &= \sum_{m=1}^{M} \log \Gamma\left( \sum_{k=1}^{K} \alpha_k \right) - \sum_{m=1}^{M} \sum_{k=1}^{K} \log \Gamma(\alpha_k) + \sum_{m=1}^{M} \sum_{k=1}^{K} (\alpha_k - 1) E_{q(\theta_m | \gamma_m)} \left[ \log \theta_{mk} \right] \\ &= \sum_{m=1}^{M} \log \Gamma\left( \sum_{k=1}^{K} \alpha_k \right) - \sum_{m=1}^{M} \sum_{k=1}^{K} \log \Gamma(\alpha_k) + \sum_{m=1}^{M} \sum_{k=1}^{K} (\alpha_k - 1) \left[ \psi(\gamma_{mk}) - \psi\left( \sum_{l=1}^{K} \gamma_{ml} \right) \right] \end{aligned}\tag{123}$$

此处用到迪利克雷分布作为指数族分布的性质：对数规范化因子对自然参数的导数等于充分统计量的数学期望，$\psi$ 是 digamma 函数，即对数伽马函数的一阶导数。 第二项：

$$\begin{aligned} &\sum_{k=1}^{K} E_{q\left(\varphi_{k} \mid \mu_{k}\right)}\left[\log p\left(\varphi_{k} \mid \boldsymbol{\beta}\right)\right] \\ &=\sum_{k=1}^{K} \log \Gamma\left(\sum_{v=1}^{V} \beta_{v}\right)-\sum_{k=1}^{K} \sum_{v=1}^{V} \log \Gamma\left(\beta_{v}\right)+\sum_{k=1}^{K} \sum_{v=1}^{V}\left(\beta_{v}-1\right)\left[\psi\left(\mu_{k v}\right)-\psi\left(\sum_{s=1}^{V} \mu_{k s}\right)\right] \end{aligned}\tag{124}$$

第三项：

$$\begin{aligned} &\sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q(z_{mn}, \theta_m \mid \eta_{mn}, \gamma_m)} [\log p(z_{mn} \mid \theta_m)] \\ &= \sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q(z_{mn}, \theta_m \mid \eta_{mn}, \gamma_m)} \left[ \log \prod_{k=1}^{K} (\theta_{mk})^{\mathbb{I}(z_{mn}=k)} \right] \\ &= \sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q(z_{mn}, \theta_m \mid \eta_{mn}, \gamma_m)} \left[ \sum_{k=1}^{K} \mathbb{I}(z_{mn}=k) \log \theta_{mk} \right] \\ &= \sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} E_{q(z_{mn}, \theta_m \mid \eta_{mn}, \gamma_m)} [\mathbb{I}(z_{mn}=k) \log \theta_{mk}] \\ &= \sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} E_{q(z_{mn} \mid \eta_{mn})} [\mathbb{I}(z_{mn}=k)] E_{q(\theta_m \mid \gamma_m)} [\log \theta_{mk}] \\ &= \sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk} \left[ \psi(\gamma_{mk}) - \psi \left( \sum_{l=1}^{K} \gamma_{ml} \right) \right] \end{aligned}\tag{125}$$

第四项：

$$\begin{aligned} &\sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q\left(\boldsymbol{\varphi}_{1: K}, z_{mn} \mid {\mu}_{1: K}, {\eta}_{mn}\right)}\left[\log p\left(w_{mn} \mid {\varphi}_{1: K}, {z}_{mn}\right)\right] \\ &=\sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q\left({\varphi}_{1: K}, {z}_{mn} \mid {\mu}_{1: K}, {\eta}_{mn}\right)}\left[\log \prod_{k=1}^{K} \varphi_{k, i\left(w_{mn}\right)}^{\mathbb{I}\left(z_{mn}=k\right)}\right] \\ &=\sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q\left({\varphi}_{1: K}, {z}_{mn} \mid {\mu}_{1: K}, {\eta}_{mn}\right)}\left[\sum_{k=1}^{K} \mathbb{I}\left(z_{mn}=k\right) \log \varphi_{k, i\left(w_{mn}\right)}\right] \\ &=\sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} E_{q\left(\varphi_{k}, z_{mn} \mid \mu_{k}, \eta_{mn}\right)}\left[\mathbb{I}\left(z_{mn}=k\right) \log \varphi_{k, i\left(w_{mn}\right)}\right] \\ &=\sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} E_{q\left(z_{mn} \mid \eta_{mn}\right)}\left[\mathbb{I}\left(z_{mn}=k\right)\right] E_{q\left(\varphi_{k} \mid \mu_{k}\right)}\left[\log \varphi_{k, i\left(w_{mn}\right)}\right] \\ &=\sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk}\left[\psi\left(\mu_{k, i\left(w_{mn}\right)}\right)-\psi\left(\sum_{s=1}^{V} \mu_{k s}\right)\right] \end{aligned}\tag{126}$$

式中 $i(w_{mn})\in \{1, \dots, V\}$ 表示单词 $w_{mn}$ 的索引。 第五项：

$$\begin{aligned} &-\sum_{k=1}^{K} E_{q\left(\varphi_{k} \mid \mu_{k}\right)}\left[\log q\left(\varphi_{k} \mid \mu_{k}\right)\right] \\ &=-\sum_{k=1}^{K} E_{q\left(\varphi_{k} \mid \mu_{k}\right)}\left[\log \left(\frac{\Gamma\left(\sum_{v=1}^{V} \mu_{k v}\right)}{\prod_{v=1}^{V} \Gamma\left(\mu_{k v}\right)} \prod_{v=1}^{V} \varphi_{k v}^{\mu_{k v}-1}\right)\right] \\ &=-\sum_{k=1}^{K} E_{q\left(\varphi_{k} \mid \mu_{k}\right)}\left[\log \Gamma\left(\sum_{v=1}^{V} \mu_{k v}\right)-\sum_{v=1}^{V} \log \Gamma\left(\mu_{k v}\right)+\sum_{v=1}^{V}\left(\mu_{k v}-1\right) \log \varphi_{k v}\right] \\ &=-\sum_{k=1}^{K} \log \Gamma\left(\sum_{v=1}^{V} \mu_{k v}\right)+\sum_{k=1}^{K} \sum_{v=1}^{V} \log \Gamma\left(\mu_{k v}\right)-\sum_{k=1}^{K} \sum_{v=1}^{V}\left(\mu_{k v}-1\right) E_{q\left(\varphi_{k} \mid \mu_{k}\right)}\left[\log \varphi_{k v}\right] \\ &=-\sum_{k=1}^{K} \log \Gamma\left(\sum_{v=1}^{V} \mu_{k v}\right)+\sum_{k=1}^{K} \sum_{v=1}^{V} \log \Gamma\left(\mu_{k v}\right)-\sum_{k=1}^{K} \sum_{v=1}^{V}\left(\mu_{k v}-1\right)\left[\psi\left(\mu_{k v}\right)-\psi\left(\sum_{s=1}^{V} \mu_{k s}\right)\right] \end{aligned}\tag{127}$$

第六项：

$$\begin{aligned} &-\sum_{m=1}^{M} E_{q\left(\theta_{m} \mid \gamma_{m}\right)}\left[\log q\left(\theta_{m} \mid \gamma_{m}\right)\right] \\ &=-\sum_{m=1}^{M} \log \Gamma\left(\sum_{k=1}^{K} \gamma_{m k}\right)+\sum_{m=1}^{M} \sum_{k=1}^{K} \log \Gamma\left(\gamma_{m k}\right)-\sum_{m=1}^{M} \sum_{k=1}^{K}\left(\gamma_{m k}-1\right)\left[\psi\left(\gamma_{m k}\right)-\psi\left(\sum_{l=1}^{K} \gamma_{m l}\right)\right] \end{aligned}\tag{128}$$

第七项：

$$\begin{aligned} &-\sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q(z_{mn} \mid \eta_{mn})} [\log q(z_{mn} \mid \eta_{mn})] \\ &= -\sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q(z_{mn} \mid \eta_{mn})} \left[ \log \prod_{k=1}^{K} \eta_{mnk}^{\mathbb{I}(z_{mn}=k)} \right] \\ &= -\sum_{m=1}^{M} \sum_{n=1}^{N_m} E_{q(z_{mn} \mid \eta_{mn})} \left[ \sum_{k=1}^{K} \mathbb{I}(z_{mn}=k) \log \eta_{mnk} \right] \\ &= -\sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} E_{q(z_{mn} \mid \eta_{mn})} [\mathbb{I}(z_{mn}=k)] \cdot \log \eta_{mnk} \\ &= -\sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk} \log \eta_{mnk} \end{aligned}\tag{129}$$

上述七项合并得到

$$\begin{aligned} &\text{ELBO}({\mu}_{1: K}, {\gamma}_{1: M} , \{\eta_{mn}\}, \alpha, {\beta}) \\ &=\mathcal{L}({\mu}_{1: K}, {\gamma}_{1: M} , \{\eta_{mn}\}, \alpha, {\beta}) \\ &=\sum_{m=1}^{M} \log \Gamma\left(\sum_{k=1}^{K} \alpha_{k}\right)-\sum_{m=1}^{M} \sum_{k=1}^{K} \log \Gamma\left(\alpha_{k}\right)+\sum_{m=1}^{M} \sum_{k=1}^{K}\left(\alpha_{k}-1\right)\left[\psi\left(\gamma_{m k}\right)-\psi\left(\sum_{l=1}^{K} \gamma_{m l}\right)\right] \\ &+\sum_{k=1}^{K} \log \Gamma\left(\sum_{v=1}^{V} \beta_{v}\right)-\sum_{k=1}^{K} \sum_{v=1}^{V} \log \Gamma\left(\beta_{v}\right)+\sum_{k=1}^{K} \sum_{v=1}^{V}\left(\beta_{v}-1\right)\left[\psi\left(\mu_{k v}\right)-\psi\left(\sum_{s=1}^{V} \mu_{k s}\right)\right] \\ &+\sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk}\left[\psi\left(\gamma_{m k}\right)-\psi\left(\sum_{l=1}^{K} \gamma_{m l}\right)\right] \\ &+\sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk}\left[\psi\left(\mu_{k, i\left(w_{mn}\right)}\right)-\psi\left(\sum_{s=1}^{V} \mu_{k s}\right)\right] \\ &-\sum_{k=1}^{K} \log \Gamma\left(\sum_{v=1}^{V} \mu_{k v}\right)+\sum_{k=1}^{K} \sum_{v=1}^{V} \log \Gamma\left(\mu_{k v}\right)-\sum_{k=1}^{K} \sum_{v=1}^{V}\left(\mu_{k v}-1\right)\left[\psi\left(\mu_{k v}\right)-\psi\left(\sum_{s=1}^{V} \mu_{k s}\right)\right] \\ &-\sum_{m=1}^{M} \log \Gamma\left(\sum_{k=1}^{K} \gamma_{m k}\right)+\sum_{m=1}^{M} \sum_{k=1}^{K} \log \Gamma\left(\gamma_{m k}\right)-\sum_{m=1}^{M} \sum_{k=1}^{K}\left(\gamma_{m k}-1\right)\left[\psi\left(\gamma_{m k}\right)-\psi\left(\sum_{l=1}^{K} \gamma_{m l}\right)\right] \\ &-\sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk} \log \eta_{mnk} \end{aligned}\tag{130}$$

目标函数 $\mathcal{L}(\mu_{1:K}, \gamma_{1:M}, \{\eta_{mn}\}, \alpha,\beta)$ 中关于 $\mu_k$ 的部分：

$$\begin{aligned} \mathcal{L}_{[{\mu}_k]} &= \sum_{v=1}^{V} (\beta_v - 1) \left[ \psi(\mu_{kv}) - \psi\left(\sum_{s=1}^{V} \mu_{ks}\right) \right] + \sum_{m=1}^{M} \sum_{n=1}^{N_m} \eta_{mnk} \left[ \psi(\mu_{k, i(w_{mn})}) - \psi\left(\sum_{s=1}^{V} \mu_{ks}\right) \right] \\ & \quad - \log \Gamma\left(\sum_{v=1}^{V} \mu_{kv}\right) + \sum_{v=1}^{V} \log \Gamma(\mu_{kv}) - \sum_{v=1}^{V} (\mu_{kv} - 1) \left[ \psi(\mu_{kv}) - \psi\left(\sum_{s=1}^{V} \mu_{ks}\right) \right] \\ &= \sum_{v=1}^{V} \left[ \psi(\mu_{kv}) - \psi\left(\sum_{s=1}^{V} \mu_{ks}\right) \right] \left( \beta_v + \sum_{m=1}^{M} \sum_{n=1}^{N_m} \eta_{mnk} \mathbb{I}(i(w_{mn}) = v) - \mu_{kv} \right) \\ & \quad - \log \Gamma\left(\sum_{v=1}^{V} \mu_{kv}\right) + \sum_{v=1}^{V} \log \Gamma(\mu_{kv}) \end{aligned}\tag{131}$$

分别关于 $\mu_{kv}$，$v=1,\dots,V$ 求偏导，得到 \left[\sum_{m=1}^M \sum_{n=1}^{N_m} \mathbb{I}(i(w_{mn})=v)\cdot \eta_{mnk}+\beta_v -\mu_{kv} \right]\cdot \psi'(\mu_{kv}) + \left[-\sum_{m=1}^M \sum_{n=1}^{N_m} \eta_{mnk}+\sum_{s=1}^V (\mu_{ks}-\beta_s) \right] \cdot \psi'(\sum_{s=1}^V \mu_{ks}) \tag{132} 因为 KL 散度为 $q(z)$ 的凸函数，而对单个变分参数的优化等价于优化一个 KL 散度加一个常数（见式（73）），所以令偏导数为零，得到 $\mu_{kv}$ 的更新公式 \mu_{kv} = \beta_v +\sum_{m=1}^M \sum_{n=1}^{N_m} \eta_{mnk} \mathbb{I}(i(w_{mn}=v))\tag{133} 目标函数 $\mathcal{L}(\mu_{1:K}, \gamma_{1:M}, \{\eta_{mn}\}, \alpha,\beta)$ 中关于 $\gamma_m$ 的部分：

$$\begin{aligned} \mathcal{L}_{[\boldsymbol{\gamma}_m]} &= \sum_{k=1}^{K} (\alpha_k - 1) \left[ \psi(\gamma_{mk}) - \psi \left( \sum_{l=1}^{K} \gamma_{ml} \right) \right] + \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk} \left[ \psi(\gamma_{mk}) - \psi \left( \sum_{l=1}^{K} \gamma_{ml} \right) \right] \\ & \quad - \log \Gamma \left( \sum_{k=1}^{K} \gamma_{mk} \right) + \sum_{k=1}^{K} \log \Gamma (\gamma_{mk}) - \sum_{k=1}^{K} (\gamma_{mk} - 1) \left[ \psi(\gamma_{mk}) - \psi \left( \sum_{l=1}^{K} \gamma_{ml} \right) \right] \\ &= \sum_{k=1}^{K} \left[ \psi(\gamma_{mk}) - \psi \left( \sum_{l=1}^{K} \gamma_{ml} \right) \right] \left( \alpha_k + \sum_{n=1}^{N_m} \eta_{mnk} - \gamma_{mk} \right) \\ & \quad - \log \Gamma \left( \sum_{k=1}^{K} \gamma_{mk} \right) + \sum_{k=1}^{K} \log \Gamma (\gamma_{mk}) \end{aligned}\tag{134}$$

分别关于 $\gamma_{mk}$，$k=1,\dots, K$ 求偏导，得到 \left[\sum_{n=1}^{N_m} \eta_{mnk}+\alpha_k-\gamma_{mk} \right]\cdot \psi'(\gamma_{mk}) + \left[-\sum_{n=1}^{N_m} \sum_{l=1}^K \eta_{mnl} -\sum_{l=1}^{K}(\alpha_l-1) + \sum_{l=1}^K (\gamma_{ml}-1) \right] \cdot \psi'(\sum_{l=1}^K \gamma_{ml}) \tag{135} 令偏导数为零，得到 $\gamma_{mk}$ 的更新公式 \gamma_{mk} = \alpha_k + \sum_{n=1}^{N_m} \eta_{mnk} \tag{136} 目标函数中关于 $\eta_{mn}$ 的部分：

$$\begin{aligned} \mathcal{L}_{\{\eta_{mn}\}} &= \sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk} \left[ \psi(\gamma_{mk}) - \psi \left( \sum_{l=1}^{K} \gamma_{ml} \right) \right] + \sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk} \left[ \psi(\mu_{k, i(w_{mn})}) - \psi \left( \sum_{s=1}^{V} \mu_{ks} \right) \right] \\ &- \sum_{m=1}^{M} \sum_{n=1}^{N_m} \sum_{k=1}^{K} \eta_{mnk} \log \eta_{mnk} \end{aligned}\tag{137}$$

考虑约束 $\sum_{l=1}^K \eta_{mnl}=1$，构造约束优化问题的拉格朗日函数，并分别关于 $\eta_{mnk}$，$k=1,\dots,K$ 求偏导，得到 \psi(\gamma_{mk})-\psi(\sum_{l=1}^K \gamma_{ml})+\psi(\mu_{k,i(w_{mn})}) - \psi(\sum_{s=1}^V \mu_{ks})-\log \eta_{mnk}-1+\lambda \tag{138} 令偏导数为零，得到 $\eta_{mnk}$ 的更新公式 \eta_{mnk}=\frac{\exp \left\{ \psi(\gamma_{mk}) - \psi(\sum_{l=1}^K \gamma_{ml}) + \psi(\mu_{k,i(w_{mn})})-\psi(\sum_{s=1}^V \mu_{ks}) \right\}}{\sum_{t=1}^K \left(\exp \left\{ \psi(\gamma_{mt}) - \psi(\sum_{l=1}^K \gamma_{ml}) + \psi(\mu_{t,i(w_{mn})})-\psi(\sum_{s=1}^V \mu_{ts}) \right\}\right)}\tag{139} 目标函数中关于 $\alpha$ 的部分： \mathcal{L}_{[{\alpha}]} = \sum_{m=1}^{M} \log \Gamma\left(\sum_{k=1}^{K} \alpha_{k}\right) - \sum_{m=1}^{M} \sum_{k=1}^{K} \log \Gamma\left(\alpha_{k}\right) + \sum_{m=1}^{M} \sum_{k=1}^{K} (\alpha_{k} - 1) \left[ \psi(\gamma_{mk}) - \psi \left( \sum_{l=1}^{K} \gamma_{ml} \right) \right] \tag{140} 分别关于 $\alpha_k$，$k=1,\dots,K$ 求一阶和二阶偏导，得到

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial \alpha_k} &= M \left[ \psi\left(\sum_{l=1}^{K} \alpha_l\right) - \psi(\alpha_k) \right] + \sum_{m=1}^{M} \left[ \psi(\gamma_{mk}) - \psi \left( \sum_{l=1}^{K} \gamma_{ml} \right) \right] \\ \frac{\partial^2 \mathcal{L}}{\partial \alpha_k \partial \alpha_t} &= M \left[ \psi'\left(\sum_{l=1}^{K} \alpha_l\right) - \mathbb{I}(k=t) \psi'(\alpha_k) \right] \end{aligned}\tag{141}$$

由此得到目标函数关于 $\alpha$ 的梯度 $g(\alpha)$ 和 Hessian 矩阵 $H(\alpha)$，应用牛顿法求目标函数关于 $\alpha$ 的最大化，根据以下公式迭代 \alpha_{\text{new}} = \alpha_{\text{old}} - H(\alpha_{\text{old}})^{-1} g(\alpha_{\text{old}}) \tag{142} 目标函数中关于 $\beta$ 的部分： \mathcal{L}_{[{\beta}]} = \sum_{k=1}^{K} \log \Gamma\left(\sum_{v=1}^{V} \beta_{v}\right) - \sum_{k=1}^{K} \sum_{v=1}^{V} \log \Gamma(\beta_{v}) + \sum_{k=1}^{K} \sum_{v=1}^{V} (\beta_{v} - 1) \left[ \psi(\mu_{kv}) - \psi\left(\sum_{s=1}^{V} \mu_{ks}\right) \right] \tag{143} 分别关于 $\beta_v$，$v=1,\dots, V$ 求一阶和二阶偏导，得到

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial \beta_v} &= K \left[ \psi\left(\sum_{s=1}^{V} \beta_s\right) - \psi(\beta_v) \right] + \sum_{k=1}^{K} \left[ \psi(\mu_{kv}) - \psi\left(\sum_{s=1}^{V} \mu_{ks}\right) \right] \\ \frac{\partial^2 \mathcal{L}}{\partial \beta_v \partial \beta_l} &= K \left[ \psi'\left(\sum_{s=1}^{V} \beta_s\right) - \mathbb{I}(v=l) \psi'(\beta_v) \right] \end{aligned}\tag{144}$$

由此得到目标函数关于 $\beta$ 的梯度 $g(\beta)$ 和 Hessian 矩阵 $H(\beta)$，应用牛顿法求目标函数关于 $\beta$ 的最大化，根据以下公式迭代 \beta_{\text{new}} = \beta_{\text{old}} - H(\beta_{\text{old}})^{-1} g(\beta_{\text{old}}) \tag{145} 注意：超参数可以不进行更新，以及论文中推荐先更新局部参数至收敛再更新全局参数[2003 Latent dirichlet allocation]。

2.6 随机变分推断

Reference: [2013 Stochastic Variational Inference] 利用指数族分布的良好性质，采用自然梯度代替一般梯度，解决参数向量之间的欧氏距离与它们所代表的概率分布之间的实际统计差异不成比例的问题，并使用批量更新提高效率。

2.7 结构化随机变分推断

Reference: [2015 Structured Stochastic Variational Inference] 放松平均场近似，以允许全局参数和局部隐藏变量之间的任意依赖关系，通过减少偏差、对局部最优解的敏感性和对超参数的敏感性来产生更好的参数估计。