机器学习之极大似然，最大后验

从独立同分布的数据估计参数

假设我们要从独立同分布的随机变量$D=\{x_1,\dots ,x_n\}$来估计出一个$\theta$ 最大似然估计: $$\begin{gathered} L(\theta )=P(D;\theta )=P(x_1,\dots ,x_n;\theta ) \\ =P(x_1;\theta )P(x_2;\theta )\dots P(x_n;\theta ) \\ ={\Pi }_i^{N}P(x_i;\theta ) \end{gathered}$$ $\hat{\theta }=\underset{\theta }{\mathrm{argmax}}L(\theta )=\underset{\theta }{\mathrm{argmax}}log(L(\theta ))$

例1：伯努利模型 $P(x)={\theta }^x(1-\theta {)}^{1-x}$ 那么$L(\theta )=\Pi P(x_i;\theta )=\Pi ({\theta }^{x_i}(1-\theta {)}^{1-x_i})={\theta }^{\sum _{i=1}^N{x}_i}(1-\theta {)}^{\sum _{i=1}^N(1-x_i)}$ $\log L(\theta )=\log {\theta }^{\sum _{i=1}^N{x}_i}(1-\theta {)}^{\sum _{i=1}^N(1-x_i)}=n_h\log \theta +(N-n_h)\log (1-\theta )$ $\frac{\partial \log L(\theta )}{\partial \theta }=\frac{n_h}{\theta }-\frac{N-n_h}{1-\theta }=0$ $(1-\theta )n_h=\theta (N-n_h)\Rightarrow {\hat{\theta }}_{MLE}=\frac{n_h}{N}$

例2：单变量正态分布模型 $P(x)=(2\pi {\sigma }^2{)}^{-\frac{1}{2}}\exp \{-(x-\mu {)}^2/2{\sigma }^2\}$ $L(\theta )=\Pi P(x_i)=\Pi (2\pi {\sigma }^2{)}^{-\frac{1}{2}}\exp \{-(x_i-\mu {)}^2/2{\sigma }^2\}$ $\log L(\theta )=-\frac{N}{2}\log 2\pi {\sigma }^2-\frac{1}{2{\sigma }^2}\sum _{i=1}^N(x_i-\mu {)}^2$ $\frac{\partial \log L(\theta )}{\partial \mu }=\frac{1}{{\sigma }^2}\sum _{i=1}^N(x_i-\mu )=0\Rightarrow {\mu }_{MLE}=\frac{\sum _{i=1}^N{x}_i}{N}$ $\frac{\partial \log L(\theta )}{\partial {\sigma }^2}=-\frac{N}{2{\sigma }^2}+\frac{1}{2{\sigma }^4}\sum _{i=1}^N(x_i-\mu {)}^2=0\Rightarrow {\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^2$

为了防止过拟合（上面求解出来的太过于精确了）我们就会用${\hat{\theta }}_{ML}=\frac{n_h}{n_h+n_t}$

贝叶斯学习

$P(\theta |D)=\frac{P(D|\theta )P(\theta )}{P(D)}$，其中$P(\theta |D)$就是后验，$P(D|\theta )$就是上面求的似然估计，$P(\theta )$就是$\theta$的一个先验,$P(D)$就是相当于是一个归一项、常熟。 那么，$P(\theta |D)\propto P(D|\theta )P(\theta )$。 其中，最大后验概率就是${\hat{\theta }}_{MAP}=\underset{\theta }{\mathrm{argmax}}P(\theta |D)$。 当$\theta$的先验是一个均匀分布时，那么$P(\theta )$也是一个常熟，那么$P(\theta |D)\propto P(D|\theta )$即MLE=MAP

例1：Beta分布，似然是伯努利分布 给$\theta$一个先验是Beta分布，$P(\theta )=\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}{\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}$ 那么$\theta$的后验$P(\theta |D)\propto P(D|\theta )P(\theta )\propto {\theta }^{n_h}(1-\theta {)}^{N-n_h}\times {\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}={\theta }^{n_h+\alpha -1}(1-\theta {)}^{N-n_h+\beta -1}\propto Beta(\alpha +n_h,\beta +n_t)$ 也就是先验和后验共轭了 例2：Beta分布，似然是二项分布 给$\theta$一个先验是Beta分布，$P(\theta )=\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}{\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}$ $P(D|\theta )=C_n^k{\theta }^k(1-\theta {)}^{n-k}$ 那么后验概率$P(\theta |D)\propto P(D|\theta )P(\theta )\propto {\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}{\theta }^k(1-\theta {)}^{n-k}\propto Beta(\alpha +k,\beta +n-k)$ 例3： 多项分布：$P(x_1,\dots ,x_k;n,p_1,\dots ,p_k)=\frac{n!}{x_1!\dots x_n!}{p}_1^{x_1}\dots p_k^{x_k}$ 狄利克雷分布：$P({\theta }_1,\dots ,{\theta }_K)=\frac{1}{B(\alpha )}{\Pi }_i^K{\theta }_i^{{\alpha }_i-1}$

参考：哈工大机器学习PPT