{"id":557,"date":"2017-11-29T16:43:27","date_gmt":"2017-11-29T08:43:27","guid":{"rendered":"http:\/\/taoteam.xiaobaidonghui.cn\/?p=557"},"modified":"2017-11-29T16:43:27","modified_gmt":"2017-11-29T08:43:27","slug":"%e5%8f%98%e5%88%86%e8%b4%9d%e5%8f%b6%e6%96%af%e7%ae%97%e6%b3%95%e7%90%86%e8%a7%a3%e4%b8%8e%e6%8e%a8%e5%af%bc-variational-bayes","status":"publish","type":"post","link":"http:\/\/www.taoteam.tech\/?p=557","title":{"rendered":"\u53d8\u5206\u8d1d\u53f6\u65af\u7b97\u6cd5\u7406\u89e3\u4e0e\u63a8\u5bfc Variational Bayes"},"content":{"rendered":"

\u539f\u6587\u5730\u5740\uff1a http:\/\/blog.huajh7.com\/2013\/03\/06\/variational-bayes\/<\/p>\n

\u524d\u8a00
\n\u4e0a\u4e16\u7eaa90\u5e74\u4ee3\uff0c\u53d8\u5206\u63a8\u65ad\u5728\u6982\u7387\u6a21\u578b\u4e0a\u5f97\u5230\u8fc5\u901f\u53d1\u5c55\uff0c\u5728\u8d1d\u53f6\u65af\u6846\u67b6\u4e0b\u4e00\u822c\u7684\u53d8\u5206\u6cd5\u7531Attias\u7684\u4e24\u7bc7\u6587\u7ae0\u7ed9\u51fa\u3002Matthew J.Beal\u7684\u535a\u58eb\u8bba\u6587\u300aVariational Algorithms for Approximate Bayesian Inference\u300b\u4e2d\u6709\u6bd4\u8f83\u5145\u5206\u5730\u8bba\u8ff0\uff0c\u4f5c\u8005\u5c06\u5176\u5e94\u7528\u4e8e\u9690\u9a6c\u5c14\u79d1\u592b\u6a21\u578b\uff0c\u6df7\u5408\u56e0\u5b50\u5206\u6790\uff0c\u7ebf\u6027\u52a8\u529b\u5b66\uff0c\u56fe\u6a21\u578b\u7b49\u3002\u53d8\u5206\u8d1d\u53f6\u65af\u662f\u4e00\u7c7b\u7528\u4e8e\u8d1d\u53f6\u65af\u4f30\u8ba1\u548c\u673a\u5668\u5b66\u4e60\u9886\u57df\u4e2d\u8fd1\u4f3c\u8ba1\u7b97\u590d\u6742\uff08intractable\uff09\u79ef\u5206\u7684\u6280\u672f\u3002\u5b83\u4e3b\u8981\u5e94\u7528\u4e8e\u590d\u6742\u7684\u7edf\u8ba1\u6a21\u578b\u4e2d\uff0c\u8fd9\u79cd\u6a21\u578b\u4e00\u822c\u5305\u62ec\u4e09\u7c7b\u53d8\u91cf\uff1a\u89c2\u6d4b\u53d8\u91cf(observed variables, data)\uff0c\u672a\u77e5\u53c2\u6570\uff08parameters\uff09\u548c\u6f5c\u53d8\u91cf\uff08latent variables\uff09\u3002\u5728\u8d1d\u53f6\u65af\u63a8\u65ad\u4e2d\uff0c\u53c2\u6570\u548c\u6f5c\u53d8\u91cf\u7edf\u79f0\u4e3a\u4e0d\u53ef\u89c2\u6d4b\u53d8\u91cf(unobserved variables)\u3002\u53d8\u5206\u8d1d\u53f6\u65af\u65b9\u6cd5\u4e3b\u8981\u662f\u4e24\u4e2a\u76ee\u7684:<\/p>\n

\u8fd1\u4f3c\u4e0d\u53ef\u89c2\u6d4b\u53d8\u91cf\u7684\u540e\u9a8c\u6982\u7387\uff0c\u4ee5\u4fbf\u901a\u8fc7\u8fd9\u4e9b\u53d8\u91cf\u4f5c\u51fa\u7edf\u8ba1\u63a8\u65ad\u3002
\n\u5bf9\u4e00\u4e2a\u7279\u5b9a\u7684\u6a21\u578b\uff0c\u7ed9\u51fa\u89c2\u6d4b\u53d8\u91cf\u7684\u8fb9\u7f18\u4f3c\u7136\u51fd\u6570\uff08\u6216\u79f0\u4e3a\u8bc1\u636e\uff0cevidence\uff09\u7684\u4e0b\u754c\u3002\u4e3b\u8981\u7528\u4e8e\u6a21\u578b\u7684\u9009\u62e9\uff0c\u8ba4\u4e3a\u6a21\u578b\u7684\u8fb9\u7f18\u4f3c\u7136\u503c\u8d8a\u9ad8\uff0c\u5219\u6a21\u578b\u5bf9\u6570\u636e\u62df\u5408\u7a0b\u5ea6\u8d8a\u597d\uff0c\u8be5\u6a21\u578b\u4ea7\u751fData\u7684\u6982\u7387\u4e5f\u8d8a\u9ad8\u3002
\n\u5bf9\u4e8e\u7b2c\u4e00\u4e2a\u76ee\u7684\uff0c\u8499\u7279\u5361\u6d1b\u6a21\u62df\uff0c\u7279\u522b\u662f\u7528Gibbs\u53d6\u6837\u7684MCMC\u65b9\u6cd5\uff0c\u53ef\u4ee5\u8fd1\u4f3c\u8ba1\u7b97\u590d\u6742\u7684\u540e\u9a8c\u5206\u5e03\uff0c\u80fd\u5f88\u597d\u5730\u5e94\u7528\u5230\u8d1d\u53f6\u65af\u7edf\u8ba1\u63a8\u65ad\u3002\u6b64\u65b9\u6cd5\u901a\u8fc7\u5927\u91cf\u7684\u6837\u672c\u4f30\u8ba1\u771f\u5b9e\u7684\u540e\u9a8c\uff0c\u56e0\u800c\u8fd1\u4f3c\u7ed3\u679c\u5e26\u6709\u4e00\u5b9a\u7684\u968f\u673a\u6027\u3002\u4e0e\u6b64\u4e0d\u540c\u7684\u662f\uff0c\u53d8\u5206\u8d1d\u53f6\u65af\u65b9\u6cd5\u63d0\u4f9b\u4e00\u79cd\u5c40\u90e8\u6700\u4f18\uff0c\u4f46\u5177\u6709\u786e\u5b9a\u89e3\u7684\u8fd1\u4f3c\u540e\u9a8c\u65b9\u6cd5\u3002<\/p>\n

\u4ece\u67d0\u79cd\u89d2\u5ea6\u770b\uff0c\u53d8\u5206\u8d1d\u53f6\u65af\u53ef\u4ee5\u770b\u505a\u662fEM\u7b97\u6cd5\u7684\u6269\u5c55\uff0c\u56e0\u4e3a\u5b83\u4e5f\u662f\u91c7\u7528\u6781\u5927\u540e\u9a8c\u4f30\u8ba1(MAP)\uff0c\u5373\u7528\u5355\u4e2a\u6700\u6709\u53ef\u80fd\u7684\u53c2\u6570\u503c\u6765\u4ee3\u66ff\u5b8c\u5168\u8d1d\u53f6\u65af\u4f30\u8ba1\u3002\u53e6\u5916\uff0c\u53d8\u5206\u8d1d\u53f6\u65af\u4e5f\u901a\u8fc7\u4e00\u7ec4\u76f8\u4e92\u4f9d\u7136\uff08mutually dependent\uff09\u7684\u7b49\u5f0f\u8fdb\u884c\u4e0d\u65ad\u7684\u8fed\u4ee3\u6765\u83b7\u5f97\u6700\u4f18\u89e3\u3002<\/p>\n

\u95ee\u9898\u63cf\u8ff0
\n\u91cd\u65b0\u8003\u8651\u4e00\u4e2a\u95ee\u9898\uff1a1\uff09\u6709\u4e00\u7ec4\u89c2\u6d4b\u6570\u636eDD\uff0c\u5e76\u4e14\u5df2\u77e5\u6a21\u578b\u7684\u5f62\u5f0f\uff0c\u6c42\u53c2\u6570\u4e0e\u6f5c\u53d8\u91cf\uff08\u6216\u4e0d\u53ef\u89c2\u6d4b\u53d8\u91cf\uff09Z={Z1,…,Zn}Z={Z1,…,Zn} \u7684\u540e\u9a8c\u5206\u5e03: P(Z|D)P(Z|D)\u3002<\/p>\n

\u6b63\u5982\u4e0a\u6587\u6240\u63cf\u8ff0\u7684\u540e\u9a8c\u6982\u7387\u7684\u5f62\u5f0f\u901a\u5e38\u662f\u5f88\u590d\u6742(Intractable)\u7684,\u5bf9\u4e8e\u4e00\u79cd\u7b97\u6cd5\u5982\u679c\u4e0d\u80fd\u5728\u591a\u9879\u5f0f\u65f6\u95f4\u5185\u6c42\u89e3\uff0c\u5f80\u5f80\u4e0d\u662f\u6211\u4eec\u6240\u8003\u8651\u7684\u3002\u56e0\u800c\u6211\u4eec\u60f3\u80fd\u4e0d\u80fd\u5728\u8bef\u5dee\u5141\u8bb8\u7684\u8303\u56f4\u5185\uff0c\u7528\u66f4\u7b80\u5355\u3001\u5bb9\u6613\u7406\u89e3(tractable)\u7684\u6570\u5b66\u5f62\u5f0fQ(Z)\u6765\u8fd1\u4f3cP(Z|D)P(Z|D),\u5373 P(Z|D)\u2248Q(Z)P(Z|D)\u2248Q(Z)\u3002<\/p>\n

\u7531\u6b64\u5f15\u51fa\u5982\u4e0b\u4e24\u4e2a\u95ee\u9898\uff1a<\/p>\n

\u5047\u8bbe\u5b58\u5728\u8fd9\u6837\u7684Q(Z)Q(Z),\u90a3\u4e48\u5982\u4f55\u5ea6\u91cfQ(Z)Q(Z)\u4e0eP(Z|D)P(Z|D)\u4e4b\u95f4\u7684\u5dee\u5f02\u6027 (dissimilarity).
\n\u5982\u4f55\u5f97\u5230\u7b80\u5355\u7684Q(Z)Q(Z)?
\n\u5bf9\u4e8e\u95ee\u9898\u4e00\uff0c\u5e78\u8fd0\u7684\u662f\uff0c\u6211\u4eec\u4e0d\u9700\u8981\u91cd\u65b0\u5b9a\u4e49\u4e00\u4e2a\u5ea6\u91cf\u6307\u6807\u3002\u5728\u4fe1\u606f\u8bba\u4e2d\uff0c\u5df2\u7ecf\u5b58\u5728\u63cf\u8ff0\u4e24\u4e2a\u968f\u673a\u5206\u5e03\u4e4b\u95f4\u8ddd\u79bb\u7684\u5ea6\u91cf\uff0c\u5373\u76f8\u5bf9\u71b5\uff0c\u6216\u8005\u79f0\u4e3aKullback-Leibler\u6563\u5ea6\u3002<\/p>\n

\u5bf9\u4e8e\u95ee\u9898\u4e8c\uff0c\u663e\u7136\u6211\u4eec\u53ef\u4ee5\u81ea\u4e3b\u51b3\u5b9aQ(Z)Q(Z)\u7684\u5206\u5e03\uff0c\u53ea\u8981\u5b83\u8db3\u591f\u7b80\u5355\uff0c\u4e14\u4e0eP(Z|D)P(Z|D)\u63a5\u8fd1\u3002\u7136\u800c\u4e0d\u53ef\u80fd\u6bcf\u6b21\u90fd\u624b\u5de5\u7ed9\u51fa\u4e00\u4e2a\u4e0eP(Z|D)P(Z|D)\u63a5\u8fd1\u4e14\u7b80\u5355\u7684Q(Z)Q(Z)\uff0c\u5176\u65b9\u6cd5\u672c\u8eab\u5df2\u7ecf\u4e0d\u5177\u5907\u53ef\u64cd\u4f5c\u6027\u3002\u6240\u4ee5\u9700\u8981\u4e00\u79cd\u901a\u7528\u7684\u5f62\u5f0f\u5e2e\u52a9\u7b80\u5316\u95ee\u9898\u3002\u90a3\u4e48\u6570\u5b66\u5f62\u5f0f\u590d\u6742\u7684\u539f\u56e0\u662f\u4ec0\u4e48\uff1f\u5728\u201c\u6a21\u578b\u7684\u9009\u62e9\u201d\u90e8\u5206\uff0c\u66fe\u63d0\u5230Occam\u2019s razor\uff0c\u8ba4\u4e3a\u4e00\u4e2a\u6a21\u578b\u7684\u53c2\u6570\u4e2a\u6570\u8d8a\u591a\uff0c\u90a3\u4e48\u6a21\u578b\u590d\u6742\u7684\u6982\u7387\u8d8a\u5927;\u6b64\u5916\uff0c\u5982\u679c\u53c2\u6570\u4e4b\u95f4\u5177\u6709\u76f8\u4e92\u4f9d\u8d56\u5173\u7cfb(mutually dependent)\uff0c\u90a3\u4e48\u901a\u5e38\u5f88\u96be\u5bf9\u53c2\u6570\u7684\u8fb9\u7f18\u6982\u7387\u7cbe\u786e\u6c42\u89e3\u3002<\/p>\n

\u5e78\u8fd0\u7684\u662f\uff0c\u7edf\u8ba1\u7269\u7406\u5b66\u754c\u5f88\u65e9\u5c31\u5173\u6ce8\u4e86\u9ad8\u7ef4\u6982\u7387\u51fd\u6570\u4e0e\u5b83\u7684\u7b80\u5355\u5f62\u5f0f\uff0c\u5e76\u53d1\u5c55\u4e86\u5e73\u5747\u573a\u7406\u8bba\u3002\u7b80\u5355\u8bb2\u5c31\u662f\uff1a\u7cfb\u7edf\u4e2d\u4e2a\u4f53\u7684\u5c40\u90e8\u76f8\u4e92\u4f5c\u7528\u53ef\u4ee5\u4ea7\u751f\u5b8f\u89c2\u5c42\u9762\u8f83\u4e3a\u7a33\u5b9a\u7684\u884c\u4e3a\u3002\u4e8e\u662f\u6211\u4eec\u53ef\u4ee5\u4f5c\u51fa\u540e\u9a8c\u6761\u4ef6\u72ec\u7acb\uff08posterior independence\uff09\u7684\u5047\u8bbe\u3002\u5373\uff0c\u2200i,p(Z|D)=p(Zi|D)p(Z\u2212i|D)\u2200i,p(Z|D)=p(Zi|D)p(Z\u2212i|D)
\nKullback-Leibler\u6563\u5ea6
\n\u5728\u7edf\u8ba1\u5b66\u4e2d\uff0c\u76f8\u5bf9\u71b5\u5bf9\u5e94\u7684\u662f\u4f3c\u7136\u6bd4\u7684\u5bf9\u6570\u671f\u671b\uff0c\u76f8\u5bf9\u71b5 D(p||q)D(p||q) \u5ea6\u91cf\u5f53\u771f\u5b9e\u5206\u5e03\u4e3a P\u800c\u5047\u5b9a\u5206\u5e03\u4e3aQ\u65f6\u7684\u65e0\u6548\u6027\u3002<\/p>\n

\u5b9a\u4e49 \u4e24\u4e2a\u6982\u7387\u5bc6\u5ea6\u51fd\u6570\u4e3ap(x)p(x)\u548cq(x)q(x)\u4e4b\u95f4\u7684\u76f8\u5bf9\u71b5\u5b9a\u4e49\u4e3a<\/p>\n

DKL(p||q)=\u2211xp(x)logp(x)q(x)DKL(p||q)=\u2211xp(x)log\u2061p(x)q(x).<\/p>\n

KL\u6563\u5ea6\u6709\u5982\u4e0b\u6027\u8d28\uff1a<\/p>\n

DKL(p||q)\u2260DKL(q||p)DKL(p||q)\u2260DKL(q||p);
\nDKL(p||q)\u22650DKL(p||q)\u22650 \uff0c\u5f53\u4e14\u4ec5\u5f53p=qp=q\u65f6\u4e3a\u96f6\uff1b
\n\u4e0d\u6ee1\u8db3\u4e09\u89d2\u4e0d\u7b49\u5f0f\u3002
\nQ\u5206\u5e03\u4e0eP\u5206\u5e03\u7684KL\u6563\u5ea6\u4e3a\uff1a<\/p>\n

DKL(Q||P)=\u2211ZQ(Z)logQ(Z)P(Z|D)=\u2211ZQ(Z)logQ(Z)P(Z,D)+logP(D)
\nDKL(Q||P)=\u2211ZQ(Z)log\u2061Q(Z)P(Z|D)=\u2211ZQ(Z)log\u2061Q(Z)P(Z,D)+log\u2061P(D)
\n\u6216\u8005<\/p>\n

logP(D)=DKL(Q||P)\u2212\u2211ZQ(Z)logQ(Z)P(Z,D)=DKL(Q||P)+L(Q).
\nlog\u2061P(D)=DKL(Q||P)\u2212\u2211ZQ(Z)log\u2061Q(Z)P(Z,D)=DKL(Q||P)+L(Q).
\n\u7531\u4e8e\u5bf9\u6570\u8bc1\u636elogP(D)logP(D)\u88ab\u76f8\u5e94\u7684Q\u6240\u56fa\u5b9a\uff0c\u4e3a\u4e86\u4f7fKL\u6563\u5ea6\u6700\u5c0f\uff0c\u5219\u53ea\u8981\u6781\u5927\u5316L(Q)L(Q)\u3002\u901a\u8fc7\u9009\u62e9\u5408\u9002\u7684Q\uff0c\u4f7fL(Q)L(Q)\u4fbf\u4e8e\u8ba1\u7b97\u548c\u6c42\u6781\u503c\u3002\u8fd9\u6837\u5c31\u53ef\u4ee5\u5f97\u5230\u540e\u9a8cP(Z|D)P(Z|D)\u7684\u8fd1\u4f3c\u89e3\u6790\u8868\u8fbe\u5f0f\u548c\u8bc1\u636e\uff08log evidence\uff09\u7684\u4e0b\u754cL(Q)L(Q)\uff0c\u53c8\u79f0\u4e3a\u53d8\u5206\u81ea\u7531\u80fd\uff08variational free energy\uff09\uff1a<\/p>\n

L(Q)=\u2211ZQ(Z)logP(Z,D)\u2212\u2211ZQ(Z)logQ(Z)=EQ[logP(Z,D)]+H(Q)
\nL(Q)=\u2211ZQ(Z)log\u2061P(Z,D)\u2212\u2211ZQ(Z)log\u2061Q(Z)=EQ[log\u2061P(Z,D)]+H(Q)
\nvb1<\/p>\n

\u5e73\u5747\u573a\u7406\u8bba\uff08Mean Field Method\uff09
\n\u6570\u5b66\u4e0a\u8bf4\uff0c\u5e73\u5747\u573a\u7684\u9002\u7528\u8303\u56f4\u53ea\u80fd\u662f\u5b8c\u5168\u56fe\uff0c\u6216\u8005\u8bf4\u7cfb\u7edf\u7ed3\u6784\u662fwell-mixed\uff0c\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\uff0c\u7cfb\u7edf\u4e2d\u7684\u4efb\u4f55\u4e00\u4e2a\u4e2a\u4f53\u4ee5\u7b49\u53ef\u80fd\u63a5\u89e6\u5176\u4ed6\u4e2a\u4f53\u3002\u53cd\u89c2\u7269\u7406\uff0c\u5e73\u5747\u573a\u4e0e\u5176\u8bf4\u662f\u4e00\u79cd\u65b9\u6cd5\uff0c\u4e0d\u5982\u8bf4\u662f\u4e00\u79cd\u601d\u60f3\u3002\u5176\u5b9e\u7edf\u8ba1\u7269\u7406\u7684\u7814\u7a76\u76ee\u7684\u5c31\u662f\u671f\u671b\u5bf9\u5b8f\u89c2\u7684\u70ed\u529b\u5b66\u73b0\u8c61\u7ed9\u4e88\u5408\u7406\u7684\u5fae\u89c2\u7406\u8bba\u3002\u7269\u7406\u5b66\u5bb6\u575a\u4fe1\uff0c\u5373\u4fbf\u4e0d\u6ee1\u8db3\u5b8c\u5168\u56fe\u7684\u5047\u8bbe\uff0c\u4f46\u65e2\u7136\u8fd9\u79cd\u201c\u5c40\u90e8\u201d\u5230\u201c\u6574\u4f53\u201d\u7684\u4f5c\u7528\u5f97\u4ee5\u5b9e\u73b0\uff0c\u90a3\u4e48\u4e2a\u4f53\u4e4b\u95f4\u7684\u5c40\u90e8\u4f5c\u7528\u76f8\u8f83\u4e8e\u201c\u5168\u5c40\u201d\u7684\u4f5c\u7528\u662f\u53ef\u4ee5\u5ffd\u7565\u4e0d\u8ba1\u7684\u3002<\/p>\n

\u6839\u636e\u5e73\u5747\u573a\u7406\u8bba\uff0c\u53d8\u5206\u5206\u5e03Q(Z)\u53ef\u4ee5\u901a\u8fc7\u53c2\u6570\u548c\u6f5c\u5728\u53d8\u91cf\u7684\u5212\u5206\uff08partition\uff09\u56e0\u5f0f\u5206\u89e3\uff0c\u6bd4\u5982\u5c06ZZ\u5212\u5206\u4e3aZ1\u2026ZMZ1\u2026ZM
\nQ(Z)=\u220fi=1Mq(Zi|D)
\nQ(Z)=\u220fi=1Mq(Zi|D)
\n\u6ce8\u610f\u8fd9\u91cc\u5e76\u975e\u4e00\u4e2a\u4e0d\u53ef\u89c2\u6d4b\u53d8\u91cf\u4e00\u4e2a\u5212\u5206\uff0c\u800c\u5e94\u8be5\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u505a\u51b3\u5b9a\u3002\u5f53\u7136\u4f60\u4e5f\u53ef\u4ee5\u8fd9\u4e48\u505a\uff0c\u4f46\u662f\u6709\u65f6\u5019\uff0c\u5c06\u51e0\u4e2a\u6f5c\u53d8\u91cf\u653e\u5728\u4e00\u8d77\u4f1a\u66f4\u5bb9\u6613\u5904\u7406\u3002<\/p>\n

\u5e73\u5747\u573a\u65b9\u6cd5\u7684\u5408\u7406\u6027
\n\u5728\u91cf\u5b50\u591a\u4f53\u95ee\u9898\u4e2d\uff0c\u7528\u4e00\u4e2a\uff08\u5355\u4f53\uff09\u6709\u6548\u573a\u6765\u4ee3\u66ff\u7535\u5b50\u6240\u53d7\u5230\u7684\u5176\u4ed6\u7535\u5b50\u7684\u5e93\u4ed1\u76f8\u4e92\u4f5c\u7528\u3002\u8fd9\u4e2a\u6709\u6548\u573a\u5305\u542b\u6240\u6709\u5176\u4ed6\u7535\u53d7\u5230\u7684\u5176\u4ed6\u7535\u5b50\u7684\u5e93\u4ed1\u76f8\u4e92\u4f5c\u7528\u3002\u8fd9\u4e2a\u6709\u6548\u573a\u5305\u542b\u4e86\u6240\u6709\u5176\u4ed6\u7535\u5b50\u5bf9\u8be5\u7535\u5b50\u7684\u76f8\u4e92\u4f5c\u7528\u3002\u5229\u7528\u6709\u6548\u573a\u53d6\u4ee3\u7535\u5b50\u4e4b\u95f4\u7684\u5e93\u4ed1\u76f8\u4e92\u4f5c\u7528\u4e4b\u540e\uff0c\u6bcf\u4e00\u4e2a\u7535\u5b50\u5728\u4e00\u4e2a\u6709\u6548\u573a\u4e2d\u8fd0\u52a8\uff0c\u7535\u5b50\u4e0e\u7535\u5b50\u4e4b\u95f4\u7684\u8fd0\u52a8\u662f\u72ec\u7acb\u7684(\u9664\u4e86\u9700\u8981\u8003\u8651\u6ce1\u5229\u4e0d\u76f8\u5bb9\u539f\u7406)\uff0c\u539f\u6765\u7684\u591a\u4f53\u95ee\u9898\u8f6c\u5316\u4e3a\u5355\u4f53\u95ee\u9898\u3002<\/p>\n

\u540c\u6837\u5728\u53d8\u5206\u5206\u5e03Q(Z)\u8fd9\u4e2a\u7cfb\u7edf\u4e2d\uff0c\u6211\u4eec\u4e5f\u53ef\u4ee5\u5c06\u6bcf\u4e00\u4e2a\u6f5c\u53d8\u91cf\u5212\u5206\u770b\u6210\u662f\u4e00\u4e2a\u5355\u4f53\uff0c\u5176\u4ed6\u5212\u5206\u5bf9\u5176\u7684\u5f71\u54cd\u90fd\u53ef\u4ee5\u7528\u4e00\u4e2a\u770b\u505a\u662f\u5176\u81ea\u8eab\u7684\u4f5c\u7528\u3002\u91c7\u7528\u7684\u529e\u6cd5\u662f\u8fed\u4ee3(Iterative VB(IVB) algorithm)\u3002\u8fd9\u662f\u7531\u4e8e\u5f53\u53d8\u5206\u81ea\u7531\u80fd\u53d6\u5f97\u6700\u5927\u503c\u7684\u65f6\u5019\uff0c\u5212\u5206ZiZi\u4e0e\u5b83\u7684\u4e92\u65a5\u96c6Z\u2212iZ\u2212i(\u6216\u8005\u66f4\u8fdb\u4e00\u6b65\uff0c\u9a6c\u5c14\u79d1\u592b\u6bef(Markov blanket), mb(Zi))mb(Zi))\u5177\u6709\u4e00\u4e2a\u7b80\u5355\u7684\u5173\u7cfb\uff1a<\/p>\n

Q(Zi)\u221d1Cexp\u27e8lnP(Zi,Z\u2212i,D)\u27e9Q(Z\u2212i)orQ(mb(Zi))
\nQ(Zi)\u221d1Cexp\u2061\u27e8ln\u2061P(Zi,Z\u2212i,D)\u27e9Q(Z\u2212i)orQ(mb(Zi))
\n\uff08\u4e3a\u4fdd\u6301\u6587\u7ae0\u7684\u8fde\u8d2f\u6027\uff0c\u6b64\u5904\u5148\u4e0d\u8bc1\u660e\uff0c\u4e0b\u6587\u5c06\u8be6\u7ec6\u8bf4\u660e\uff09<\/p>\n

\u4e8e\u662f\uff0c\u5bf9\u4e8e\u67d0\u4e2a\u5212\u5206ZiZi,\u6211\u4eec\u53ef\u4ee5\u5148\u4fdd\u6301\u5176\u4ed6\u5212\u5206Z\u2212iZ\u2212i\u4e0d\u53d8\uff0c\u7136\u540e\u7528\u4ee5\u4e0a\u5173\u7cfb\u5f0f\u66f4\u65b0ZiZi\u3002\u76f8\u540c\u6b65\u9aa4\u5e94\u7528\u4e8e\u5176\u4ed6\u5212\u5206\u7684\u66f4\u65b0\uff0c\u4f7f\u5f97\u6bcf\u4e2a\u5212\u5206\u4e4b\u95f4\u5145\u5206\u76f8\u4e92\u4f5c\u7528\uff0c\u6700\u7ec8\u8fbe\u5230\u7a33\u5b9a\u503c\u3002<\/p>\n

\u5177\u4f53\u66f4\u65b0\u8fb9\u7f18\u6982\u7387\uff08VB-marginal\uff09\u6b65\u9aa4\u5982\u4e0b\uff1a<\/p>\n

\u521d\u59cb\u5316Q(0)(Zi)Q(0)(Zi)\uff0c\u53ef\u968f\u673a\u53d6\uff1b
\n\u5728\u7b2ck\u6b65\uff0c\u8ba1\u7b97Z\u2212iZ\u2212i\u7684\u8fb9\u7f18\u5bc6\u5ea6Qk<\/a>\u221dexp\u222bZ\u2217iQk\u22121<\/a>logP(Zi,Z\u2212i,D)dZiQk<\/a>\u221dexp\u2061\u222bZi\u2217Qk\u22121<\/a>log\u2061P(Zi,Z\u2212i,D)dZi
\n\u8ba1\u7b97ZiZi\u7684\u8fb9\u7f18\u5bc6\u5ea6Qk<\/a>\u221dexp\u222bZ\u2217\u2212iQk<\/a>logP(Zi,Z\u2212i,D)dZ\u2212iQk<\/a>\u221dexp\u2061\u222bZ\u2212i\u2217Qk<\/a>log\u2061P(Zi,Z\u2212i,D)dZ\u2212i
\n\u7406\u8bba\u4e0aQ\u221e<\/a>Q\u221e<\/a>\u5c06\u4f1a\u6536\u655b\uff0c\u5219\u53cd\u590d\u6267\u884c(2), (3)\u76f4\u5230Q(Zi)Q(Zi),Q(Z\u2212i)Q(Z\u2212i)\u7a33\u5b9a\uff0c\u6216\u7a33\u5b9a\u5728\u67d0\u4e2a\u5c0f\u8303\u56f4\u5185\u3002
\n\u6700\u540e\uff0c\u5f97 Q(Z)=Q(Zi|D)Q(Z\u2212i|D)Q(Z)=Q(Zi|D)Q(Z\u2212i|D)
\n\u5e73\u5747\u573a\u4f30\u8ba1\u4e0b\u8fb9\u7f18\u6982\u7387\u7684\u65e0\u610f\u4e49\u6027 (VB-marginals)
\n\u6ce8\u610f\u5230Q(Z)\u4f30\u8ba1\u7684\u662f\u8054\u5408\u6982\u7387\u5bc6\u5ea6\uff0c\u800c\u5bf9\u4e8e\u6bcf\u4e00\u4e2aQi(Zi)Qi(Zi)\uff0c\u5176\u4e0e\u771f\u5b9e\u7684\u8fb9\u7f18\u6982\u7387\u5bc6\u5ea6Pi(Zi)Pi(Zi)\u7684\u5dee\u522b\u53ef\u80fd\u662f\u5f88\u5927\u7684\u3002\u4e0d\u5e94\u8be5\u7528Qi(Zi)Qi(Zi)\u6765\u4f30\u8ba1\u771f\u5b9e\u7684\u8fb9\u7f18\u5bc6\u5ea6\uff0c\u6bd4\u5982\u5728\u4e00\u4e2a\u8d1d\u53f6\u65af\u7f51\u7edc\u4e2d\uff0c\u4f60\u4e0d\u5e94\u8be5\u7528\u5b83\u6765\u63a8\u6d4b\u67d0\u4e2a\u8282\u70b9\u7684\u72b6\u6001\u3002\u800c\u8fd9\u5176\u5b9e\u662f\u5f88\u7cdf\u7cd5\u7684\uff0c\u76f8\u6bd4\u4e8e\u5176\u4ed6\u80fd\u591f\u4f7f\u7528\u8282\u70b9\u72b6\u6001\u4fe1\u606f\u6765\u8fdb\u884c\u5c40\u90e8\u63a8\u6d4b\u7684\u7b97\u6cd5\uff0c\u53d8\u5206\u8d1d\u53f6\u65af\u65b9\u6cd5\u66f4\u4e0d\u5229\u4e8e\u8c03\u8bd5\u3002<\/p>\n

\u6bd4\u5982\u4e00\u4e2a\u6807\u51c6\u7684\u9ad8\u65af\u8054\u5408\u5206\u5e03P(\u03bc,x)P(\u03bc,x)\u548c\u6700\u4f18\u7684\u5e73\u5747\u573a\u9ad8\u65af\u4f30\u8ba1Q(\u03bc,x)Q(\u03bc,x)\u3002Q\u9009\u62e9\u4e86\u5728\u5b83\u81ea\u5df1\u4f5c\u7528\u57df\u4e2d\u7684\u9ad8\u65af\u5206\u5e03\uff0c\u56e0\u800c\u53d8\u5f97\u5f88\u7a84\u3002\u6b64\u65f6\u8fb9\u7f18\u5bc6\u5ea6Qx(x)Qx(x)\u53d8\u5f97\u975e\u5e38\u5c0f\uff0c\u5b8c\u5168\u4e0ePx(x)Px(x)\u4e0d\u540c\u3002<\/p>\n

vb2<\/p>\n

\u6cdb\u51fd\u7684\u6982\u5ff5
\n\u4e0a\u6587\u5df2\u7ecf\u63d0\u5230\u6211\u4eec\u8981\u627e\u5230\u4e00\u4e2a\u66f4\u52a0\u7b80\u5355\u7684\u51fd\u6570D(Z)D(Z)\u6765\u8fd1\u4f3cP(Z|D)P(Z|D)\uff0c\u540c\u65f6\u95ee\u9898\u8f6c\u5316\u4e3a\u6c42\u89e3\u8bc1\u636elogP(Z)logP(Z)\u7684\u4e0b\u754cL(Q)L(Q)\uff0c\u6216\u8005L(Q(Z))L(Q(Z))\u3002\u5e94\u8be5\u6ce8\u610f\u5230L(Q)L(Q)\u5e76\u975e\u666e\u901a\u7684\u51fd\u6570\uff0c\u800c\u662f\u4ee5\u6574\u4e2a\u51fd\u6570\u4e3a\u81ea\u53d8\u91cf\u7684\u51fd\u6570\uff0c\u8fd9\u4fbf\u662f\u6cdb\u51fd\u3002\u6211\u4eec\u5148\u4ecb\u7ecd\u4e00\u4e0b\u4ec0\u4e48\u662f\u6cdb\u51fd\uff0c\u4ee5\u53ca\u6cdb\u51fd\u53d6\u5f97\u6781\u503c\u7684\u5fc5\u8981\u6761\u4ef6\u3002<\/p>\n

\u6cdb\u51fd<\/p>\n

\u8bbe\u5bf9\u4e8e(\u67d0\u4e00\u51fd\u6570\u96c6\u5408\u5185\u7684)\u4efb\u610f\u4e00\u4e2a\u51fd\u6570y(x)y(x)\uff0c\u6709\u53e6\u4e00\u4e2a\u6570J[y]J[y]\u4e0e\u4e4b\u5bf9\u5e94\uff0c\u5219\u79f0J[y]J[y]\u4e3ay(x)y(x)\u7684\u6cdb\u51fd\u3002\u6cdb\u51fd\u53ef\u4ee5\u770b\u6210\u662f\u51fd\u6570\u6982\u5ff5\u7684\u63a8\u5e7f\u3002 \u8fd9\u91cc\u7684\u51fd\u6570\u96c6\u5408\uff0c\u5373\u6cdb\u51fd\u7684\u5b9a\u4e49\u57df\uff0c\u901a\u5e38\u8981\u6c42y(x)y(x) \u6ee1\u8db3\u4e00\u5b9a\u7684\u8fb9\u754c\u6761\u4ef6\uff0c\u5e76\u4e14\u5177\u6709\u8fde\u7eed\u7684\u4e8c\u9636\u5bfc\u6570\uff0e\u8fd9\u6837\u7684y(x)y(x)\u79f0\u4e3a\u53ef\u53d6\u51fd\u6570\u3002
\n\u6cdb\u51fd\u4e0d\u540c\u4e8e\u590d\u5408\u51fd\u6570\uff0c<\/p>\n

\u4f8b\u5982g=g(f(x))g=g(f(x)); \u5bf9\u4e8e\u540e\u8005\uff0c\u7ed9\u5b9a\u4e00\u4e2axx\u503c\uff0c\u4ecd\u7136\u662f\u6709\u4e00\u4e2agg\u503c\u4e0e\u4e4b\u5bf9\u5e94\uff1b \u5bf9\u4e8e\u524d\u8005\uff0c\u5219\u5fc5\u987b\u7ed9\u51fa\u67d0\u4e00\u533a\u95f4\u4e0a\u7684\u51fd\u6570y(x)y(x)\uff0c\u624d\u80fd\u5f97\u5230\u4e00\u4e2a\u6cdb\u51fd\u503cJ[y]J[y]\u3002(\u5b9a\u4e49\u5728\u540c\u4e00\u533a\u95f4\u4e0a\u7684)\u51fd\u6570\u4e0d\u540c\uff0c\u6cdb\u51fd\u503c\u5f53\u7136\u4e0d\u540c\uff0c \u4e3a\u4e86\u5f3a\u8c03\u6cdb\u51fd\u503cJ[y]J[y]\u4e0e\u51fd\u6570y(x)y(x)\u4e4b\u95f4\u7684\u4f9d\u8d56\u5173\u7cfb\uff0c\u5e38\u5e38\u53c8\u628a\u51fd\u6570y(x)y(x)\u79f0\u4e3a\u53d8\u91cf\u51fd\u6570\u3002
\n\u6cdb\u51fd\u7684\u5f62\u5f0f\u591a\u79cd\u591a\u6837\uff0c\u901a\u5e38\u53ef\u4ee5\u79ef\u5206\u5f62\u5f0f\uff1aJ[y]=\u222bx1x0F(x,y,y\u2032)dxJ[y]=\u222bx0x1F(x,y,y\u2032)dx
\n\u6cdb\u51fd\u53d6\u6781\u503c\u7684\u5fc5\u8981\u6761\u4ef6
\n\u6cdb\u51fd\u7684\u6781\u503c<\/p>\n

\u201c\u5f53\u53d8\u91cf\u51fd\u6570\u4e3ay(x)y(x)\u65f6\uff0c\u6cdb\u51fdJ[y]J[y]\u53d6\u6781\u5927\u503c\u201d\u7684\u542b\u4e49\u5c31\u662f\uff1a\u5bf9\u4e8e\u6781\u503c\u51fd\u6570y(x)y(x)\u53ca\u5176\u201c\u9644\u8fd1\u201d\u7684\u53d8\u91cf\u51fd\u6570y(x)+\u03b4y(x)y(x)+\u03b4y(x)\uff0c\u6052\u6709J[y+\u03b4y]\u2264J[y]J[y+\u03b4y]\u2264J[y];<\/p>\n

\u6240\u8c13\u51fd\u6570y(x)+\u03b4y(x)y(x)+\u03b4y(x)\u5728\u53e6\u4e00\u4e2a\u51fd\u6570y(x)y(x)\u7684\u201c\u9644\u8fd1\u201d\uff0c\u6307\u7684\u662f\uff1a<\/p>\n

|\u03b4y(x)|<\u03b5|\u03b4y(x)|<\u03b5;
\n\u6709\u65f6\u8fd8\u8981\u6c42|(\u03b4y)\u2032(x)|<\u03b5|(\u03b4y)\u2032(x)|<\u03b5.
\n\u8fd9\u91cc\u7684\u03b4y(x)\u03b4y(x)\u79f0\u4e3a\u51fd\u6570y(x)y(x)\u7684\u53d8\u5206\u3002<\/p>\n

Euler\u2013Lagrange\u65b9\u7a0b<\/p>\n

\u53ef\u4ee5\u4eff\u9020\u51fd\u6570\u6781\u503c\u5fc5\u8981\u6761\u4ef6\u7684\u5bfc\u51fa\u529e\u6cd5\uff0c\u5bfc\u51fa\u6cdb\u51fd\u53d6\u6781\u503c\u7684\u5fc5\u8981\u6761\u4ef6\uff0c\u8fd9\u91cc\u4e0d\u505a\u4e25\u683c\u7684\u8bc1\u660e\uff0c\u76f4\u63a5\u7ed9\u51fa\u3002 \u6cdb\u51fdJ[y]J[y]\u53d6\u5230\u6781\u5927\u503c\u7684\u5fc5\u8981\u6761\u4ef6\u662f\u4e00\u7ea7\u53d8\u5206\u03b4J[y]\u03b4J[y]\u4e3a0\uff0c\u5176\u5fae\u5206\u5f62\u5f0f\u4e00\u822c\u4e3a\u4e8c\u9636\u5e38\u5fae\u5206\u65b9\u7a0b\uff0c\u5373Euler-Largange\u65b9\u7a0b\uff1a<\/p>\n

\u2202F\u2202y\u2212ddx\u2202F\u2202y\u2032=0
\n\u2202F\u2202y\u2212ddx\u2202F\u2202y\u2032=0
\n\u6cdb\u51fd\u7684\u6761\u4ef6\u6781\u503c<\/p>\n

\u5728\u7ea6\u675f\u6761\u4ef6 \u4e0b\u6c42\u51fd\u6570J[y]J[y]\u7684\u6781\u503c\uff0c\u53ef\u4ee5\u5f15\u5165Largange\u4e58\u5b50\u03bb\u03bb\uff0c\u4ece\u800c\u5b9a\u4e49\u4e00\u4e2a\u65b0\u7684\u6cdb\u51fd\uff0c J~[y]=J[y]\u2212\u03bbJ0[y]J~[y]=J[y]\u2212\u03bbJ0[y]\u3002\u4ecd\u5c06\u03b4y\u03b4y\u770b\u6210\u662f\u72ec\u7acb\u7684\uff0c\u5219\u6cdb\u51fdJ~[y]J~[y]\u5728\u8fb9\u754c\u6761\u4ef6\u4e0b\u53d6\u6781\u503c\u7684\u5fc5\u8981\u6761\u4ef6\u5c31\u662f\uff0c<\/p>\n

(\u2202\u2202y\u2212ddx\u2202\u2202y\u2032)(F\u2212\u03bbG)=0
\n(\u2202\u2202y\u2212ddx\u2202\u2202y\u2032)(F\u2212\u03bbG)=0
\n\u95ee\u9898\u6c42\u89e3
\n\u5bf9\u4e8eL(Q(Z))=EQ(Z)[lnP(Z,D)]+H(Q(Z))L(Q(Z))=EQ(Z)[ln\u2061P(Z,D)]+H(Q(Z))\uff0c\u5c06\u53f3\u5f0f\u7b2c\u4e00\u9879\u5b9a\u4e49\u4e3a\u80fd\u91cf(Energy)\uff0c\u7b2c\u4e8c\u9879\u770b\u505a\u662f\u4fe1\u606f\u71b5(Shannon entropy)\u3002\u6211\u4eec\u53ea\u8003\u8651\u81ea\u7136\u5bf9\u6570\u7684\u5f62\u5f0f\uff0c\u56e0\u4e3a\u5bf9\u4e8e\u4efb\u4f55\u5e95\u6570\u7684\u5bf9\u6570\u603b\u662f\u53ef\u4ee5\u901a\u8fc7\u6362\u5e95\u516c\u5f0f\u5c06\u5176\u5199\u6210\u81ea\u7136\u5bf9\u6570\u4e0e\u4e00\u4e2a\u5e38\u91cf\u7684\u4e58\u79ef\u5f62\u5f0f\u3002\u53e6\u5916\u6839\u636e\u5e73\u5747\u573a\u5047\u8bbe\u53ef\u4ee5\u5f97\u5230\u5982\u4e0b\u79ef\u5206\u5f62\u5f0f\uff0c<\/p>\n

L(Q(Z))=\u222b(\u220fiQi(Zi))lnP(Z,D)dZ\u2212\u222b(\u220fkQk(Zk))\u2211ilnQi(Zi)dZ
\nL(Q(Z))=\u222b(\u220fiQi(Zi))ln\u2061P(Z,D)dZ\u2212\u222b(\u220fkQk(Zk))\u2211iln\u2061Qi(Zi)dZ
\n\u5176\u4e2dQ(Z)=\u220fiQi(Zi)Q(Z)=\u220fiQi(Zi)\uff0c\u4e14\u6ee1\u8db3 \u2200i.\u222bQi(Zi)dZi=1\u2200i.\u222bQi(Zi)dZi=1
\n\u8003\u8651\u5212\u5206Z={Zi,Z\u2212i}Z={Zi,Z\u2212i}\uff0c\u5176\u4e2dZ\u2212i=Z\u2216ZiZ\u2212i=Z\u2216Zi\uff0c\u5148\u8003\u8651\u80fd\u91cf\u9879(Energy)\uff08\u7b2c\u4e00\u9879\uff09\uff0c<\/p>\n

EQ(Z)[lnP(Z,D)]=\u222b(\u220fiQi(Zi))lnP(Z,D)dZ=\u222bQi(Zi)dZi\u222bQ\u2212i(Z\u2212i)lnP(Z,D)dZ\u2212i=\u222bQi(Zi)\u27e8lnP(Z,D)\u27e9Q\u2212i(Z\u2212i)dZi=\u222bQi(Zi)lnexp\u27e8lnP(Z,D)\u27e9Q\u2212i(Z\u2212i)dZi=\u222bQi(Zi)lnQ\u2217i(Zi)dZi+lnC
\nEQ(Z)[ln\u2061P(Z,D)]=\u222b(\u220fiQi(Zi))ln\u2061P(Z,D)dZ=\u222bQi(Zi)dZi\u222bQ\u2212i(Z\u2212i)ln\u2061P(Z,D)dZ\u2212i=\u222bQi(Zi)\u27e8ln\u2061P(Z,D)\u27e9Q\u2212i(Z\u2212i)dZi=\u222bQi(Zi)ln\u2061exp\u2061\u27e8ln\u2061P(Z,D)\u27e9Q\u2212i(Z\u2212i)dZi=\u222bQi(Zi)ln\u2061Qi\u2217(Zi)dZi+ln\u2061C
\n\u5176\u4e2d\u5b9a\u4e49Q\u2217i(Zi)=1Cexp\u27e8lnP(Z,D)\u27e9Q\u2212i(Z\u2212i)Qi\u2217(Zi)=1Cexp\u2061\u27e8ln\u2061P(Z,D)\u27e9Q\u2212i(Z\u2212i)\uff0cCC\u4e3a\u7684\u5f52\u4e00\u5316\u5e38\u6570\u3002\u518d\u8003\u8651\u71b5\u91cf(entropy)\uff08\u7b2c\u4e8c\u9879\uff09\uff0c<\/p>\n

H(Q(Z))=\u2212\u2211i\u222b(\u220fkQk(Zk))lnQi(Zi)dZ=\u2212\u2211i\u222b\u222bQi(Zi)Q\u2212i(Z\u2212i)lnQi(ZidZidZ\u2212i=\u2212\u2211i\u27e8\u222bQi(Zi)lnQi(Zi)dZi\u27e9Q\u2212i(Z\u2212i)=\u2212\u2211i\u222bQi(Zi)lnQi(Zi)dZi
\nH(Q(Z))=\u2212\u2211i\u222b(\u220fkQk(Zk))ln\u2061Qi(Zi)dZ=\u2212\u2211i\u222b\u222bQi(Zi)Q\u2212i(Z\u2212i)ln\u2061Qi(ZidZidZ\u2212i=\u2212\u2211i\u27e8\u222bQi(Zi)ln\u2061Qi(Zi)dZi\u27e9Q\u2212i(Z\u2212i)=\u2212\u2211i\u222bQi(Zi)ln\u2061Qi(Zi)dZi
\n\u6b64\u65f6\u5f97\u5230\u6cdb\u51fd\uff0c<\/p>\n

L(Q(Z))=\u222bQi(Zi)lnQ\u2217i(Zi)dZi\u2212\u2211i\u222bQi(Zi)lnQi(Zi)dZi+lnC=(\u222bQi(Zi)lnQ\u2217i(Zi)dZi\u2212\u222bQi(Zi)lnQi(Zi)dZi)\u2212\u2211k\u2260i\u222bQk(Zk)lnQk(Zk)dZk+lnC=\u222bQi(Zi)lnQ\u2217i(Zi)Qi(Zi)dZi\u2212\u2211k\u2260i\u222bQk(Zk)lnQk(Zk)dZk+lnC=\u2212DKL(Qi(Zi)||Q\u2217i(Zi))+H[Q\u2212i(Z\u2212i)]+lnC
\nL(Q(Z))=\u222bQi(Zi)ln\u2061Qi\u2217(Zi)dZi\u2212\u2211i\u222bQi(Zi)ln\u2061Qi(Zi)dZi+lnC=(\u222bQi(Zi)ln\u2061Qi\u2217(Zi)dZi\u2212\u222bQi(Zi)ln\u2061Qi(Zi)dZi)\u2212\u2211k\u2260i\u222bQk(Zk)ln\u2061Qk(Zk)dZk+lnC=\u222bQi(Zi)ln\u2061Qi\u2217(Zi)Qi(Zi)dZi\u2212\u2211k\u2260i\u222bQk(Zk)ln\u2061Qk(Zk)dZk+ln\u2061C=\u2212DKL(Qi(Zi)||Qi\u2217(Zi))+H[Q\u2212i(Z\u2212i)]+ln\u2061C
\n\u6ce8\u610f\u5230L(Q(Z))L(Q(Z)) \u5e76\u975e\u53ea\u6709\u4e00\u4e2a\u7b49\u5f0f\uff0c\u5982\u679c\u4e0d\u53ef\u89c2\u6d4b\u53d8\u91cf\u6709M\u4e2a\u5212\u5206\u3002 \u90a3\u4e48\u5c06\u6709M\u4e2a\u65b9\u7a0b\u3002 \u4e3a\u4e86\u4f7f\u5f97L(Q(Z))L(Q(Z))\u8fbe\u5230\u6700\u5927\u503c, \u540c\u65f6\u6ce8\u610f\u5230\u7ea6\u675f\u6761\u4ef6, \u6839\u636e\u6cdb\u51fd\u6c42\u6761\u4ef6\u6781\u503c\u7684\u5fc5\u8981\u6761\u4ef6, \u5f97,<\/p>\n

\u2200i.\u2202\u2202Qi(Zi){\u2212DKL[Qi(Zi)||Q\u2217i(Zi)]\u2212\u03bbi(\u222bQi(Zi)dZi\u22121)}:=0
\n\u2200i.\u2202\u2202Qi(Zi){\u2212DKL[Qi(Zi)||Qi\u2217(Zi)]\u2212\u03bbi(\u222bQi(Zi)dZi\u22121)}:=0
\n\u76f4\u63a5\u6c42\u89e3\u5c06\u5f97\u5230Gibbs\u5206\u5e03\uff0c\u7565\u663e\u590d\u6742;\u5b9e\u9645\u4e0a\uff0c\u6ce8\u610f\u5230KL\u6563\u5ea6\uff0c\u6211\u4eec\u53ef\u4ee5\u76f4\u63a5\u5f97\u5230KL\u6563\u5ea6\u7b49\u4e8e0\u7684\u65f6\u5019\uff0cL(D)L(D)\u8fbe\u5230\u6700\u5927\u503c\uff0c\u6700\u7ec8\u5f97\u5230<\/p>\n

Qi(Zi)=Q\u2217i(Zi)=1Cexp\u27e8lnP(Zi,Z\u2212i,D)\u27e9Q\u2212i(Z\u2212i)
\nQi(Zi)=Qi\u2217(Zi)=1Cexp\u2061\u27e8ln\u2061P(Zi,Z\u2212i,D)\u27e9Q\u2212i(Z\u2212i)
\nC\u4e3a\u5f52\u4e00\u5316\u5e38\u6570C=\u222bexp\u27e8ln(Zi,Z\u2212i,D)\u27e9Q\u2212i(Z\u2212i)dZ\u2212iC=\u222bexp\u2061\u27e8ln\u2061(Zi,Z\u2212i,D)\u27e9Q\u2212i(Z\u2212i)dZ\u2212i , Q(Zi)Q(Zi)\u4e3a\u8054\u5408\u6982\u7387\u51fd\u6570\u5728\u9664ZiZi\u672c\u8eab\u5916\u7684\u5176\u4ed6\u5212\u5206\u4e0b\u7684\u5bf9\u6570\u671f\u671b\u3002\u53c8\u53ef\u4ee5\u5199\u4e3a lnQi(Zi)=\u27e8lnP(Zi,Z\u2212i,D)\u27e9Q\u2212i(Z\u2212i)+constln\u2061Qi(Zi)=\u27e8ln\u2061P(Zi,Z\u2212i,D)\u27e9Q\u2212i(Z\u2212i)+const.<\/p>\n

\u53c2\u8003\u6587\u732e
\n[1] Sm\u00eddl, V\u00e1clav, and Anthony Quinn. The variational Bayes method in signal processing. Springer, 2006.<\/p>\n

[2] Beal, Matthew James. Variational algorithms for approximate Bayesian inference. Diss. University of London, 2003.<\/p>\n

[3] Fox, Charles W., and Stephen J. Roberts. \u201cA tutorial on variational Bayesian inference.\u201d Artificial Intelligence Review 38.2 (2012): 85-95.<\/p>\n

[4] Attias, Hagai. \u201cInferring parameters and structure of latent variable models by variational Bayes.\u201d Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc., 1999.<\/p>\n

[5] Attias, Hagai. \u201cA variational Bayesian framework for graphical models.\u201dAdvances in neural information processing systems 12.1-2 (2000): 209-215.<\/p>\n","protected":false},"excerpt":{"rendered":"

\u539f\u6587\u5730\u5740\uff1a http:\/\/blog.huajh7.com\/2013\/03\/06\/variational-bay […]<\/p>\n","protected":false},"author":18,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[4],"tags":[],"_links":{"self":[{"href":"http:\/\/www.taoteam.tech\/index.php?rest_route=\/wp\/v2\/posts\/557"}],"collection":[{"href":"http:\/\/www.taoteam.tech\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.taoteam.tech\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.taoteam.tech\/index.php?rest_route=\/wp\/v2\/users\/18"}],"replies":[{"embeddable":true,"href":"http:\/\/www.taoteam.tech\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=557"}],"version-history":[{"count":0,"href":"http:\/\/www.taoteam.tech\/index.php?rest_route=\/wp\/v2\/posts\/557\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.taoteam.tech\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=557"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.taoteam.tech\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=557"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.taoteam.tech\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=557"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}