{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "# Exercise - Variational Mean Field Approximation for Univariate Gaussian" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "## Table of Contents\n", "* [Introduction](#Introduction)\n", "* [Requirements](#Requirements) \n", " * [Knowledge](#Knowledge)\n", " * [Modules](#Python-Modules)\n", "* [Data](#Data)\n", "* [Exercises](#Exercises)\n", " * [Exercise - Mean Field Approximation of the Posterior](Exercise---Mean-Field-Approximation-of-the-Posterior)\n", " * [Exercise - Proof](#Proof)\n", " * [Exercise - Implementation of the Mean Field Approximation](#Exercise---Implementation-of-the-Mean-Field-Approximation)\n", "* [Literature](#Literature)\n", "* [Licenses](#Licenses)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "## Introduction\n", "\n", "**[TODO]**\n", "\n", "In order to detect errors in your own code, execute the notebook cells containing assert or assert_almost_equal." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "## Requirements\n", "\n", "### Knowledge\n", "\n", "**[TODO]**" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "### Python Modules" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import numpy as np\n", "from matplotlib import pyplot as plt\n", "import scipy.stats" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "np.random.seed(40)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data\n", "\n", "Data:\n", "$$\n", " X \\sim \\mathcal N(\\mu, \\frac{1}{\\tau})\n", "$$\n", "\n", "\n", "Probability Density Function:\n", "$$\n", "p(X \\mid \\mu, \\tau) = \\sqrt{\\frac{\\tau}{2\\pi}} \\exp\\left( -\\frac{\\tau (X-\\mu)^2 }{2} \\right)\n", "$$\n", "\n", "with \n", "- $\\mu$: mean\n", "- $\\sigma^2$: variance\n", "- $\\tau =\\frac{1}{\\sigma^2}$ : precision" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# generation of observed data\n", "N = 10\n", "mu = 10.\n", "sigma = 2.\n", "X = np.random.normal(mu, sigma, N)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "x = np.arange(3,18,0.01)\n", "p_x = scipy.stats.norm.pdf(x, loc=mu, scale=sigma)\n", "plt.plot(x, p_x, label=\"true Gaussian\")\n", "plt.plot(X, np.zeros_like(X), \"ro\", label=\"Data\")\n", "plt.title(\"\")\n", "plt.xlabel(\"x\")\n", "plt.ylabel(\"p(x)\")\n", "plt.legend();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercises\n", "\n", "### Exercise - Mean Field Approximation of the Posterior\n", "\n", "**Task:**\n", "\n", "Find the mean field approximation of the posterior:\n", "\n", " $p(\\mu, \\tau \\mid X) \\approx q(\\mu)q(\\tau)$.\n", "\n", "- The observed data was sampled from a Gaussian distribution: $X \\sim \\mathcal N(\\mu, \\frac{1}{\\tau})$.\n", "- Use a constant prior for the mean and the precision $\\tau = \\frac{1}{\\sigma^2}$:\n", "$$\n", "p(\\mu, \\tau) = const. \\quad \\text{ for } \\tau > 0 \n", "$$\n", "\n", "$\\theta = (\\theta_0, \\theta_1) = (\\mu, \\tau)$\n", "\n", "Note: Typically the mean-field approximation in closed form is done with the conjugate distributions. But here we use\n", "the constant (improper) prior because it's a little bit easier. " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "#### Recap: Mean field approximation\n", "\n", "Loop until convergence:\n", "$$\n", "\\log q({\\theta}_k) = \\mathbb E_{q_{-k}} \\left[ \\log{\\hat p( {\\theta} \\mid {\\mathcal D} )} \\right] + const.\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "**Hint:**\n", "\n", "Use the \"Sum of difference of the mean\":\n", "$$\n", "\\sum_{i=1}^n (x_i-\\mu)^2 = \\sum_{i=1}^n(x_i-\\bar{x})^2 + n(\\bar{x} -\\mu)^2\n", "$$\n", "with\n", "$$\\bar{x} = \\frac{1}{n}\\sum_{i=1}^n x_i$$\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise - Proof\n", "\n", "**Task:**\n", "\n", "Show that:\n", "$$\n", "q(\\mu) = \\mathcal N(\\bar{X}, \\frac{1}{\\gamma_1})\n", "$$\n", "with $\\gamma_1 = m \\mathbb E_{q_{\\tau}}[\\tau]$\n", "\n", "and\n", "\n", "$$\n", "q(\\tau) = \\text{Gamma}(\\frac{m}{2}+1, \\frac{2}{\\gamma_2})\n", "$$\n", "\n", "with \n", "$$\n", "\\gamma_2= \\sum_{i=1}^m \\left(X_i^2\\right)-m\\bar X^2 + m\\text{var}_{q_{\\mu}}(\\mu)\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "#### Gamma distribution\n", "$$\n", "\\text{Gamma}(k, \\theta') = p(x) = x^{k-1}\\frac{e^{-x/\\theta'}}{\\theta'^k\\Gamma(k)}\n", "$$\n", "- $k$ is the shape \n", "- $\\theta'$ the scale, \n", "- $\\Gamma(.)$ is the Gamma function" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise - Implementation of the Mean Field Approximation \n", "\n", "**Task:**\n", "\n", "Implement the mean field approximation. Like always, you are free to implement as many helper functions as you want.\n", "\n", "If everything is correct, executing the cells below should plot figures similar to these:\n", "\n", "\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "-" } }, "outputs": [], "source": [ "def optim(X=X, mean_mu=1, sigma_quare_mu=1, loc_tau=1., scale_tau=1.):\n", " raise NotImplementedError()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "mean_mu, sigma_quare_mu, loc_tau, scale_tau = optim(X)\n", "mean_mu, sigma_quare_mu, loc_tau, scale_tau" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.figure(figsize=(12,4))\n", "\n", "x = np.arange(5,15,0.01)\n", "p_mu = scipy.stats.norm.pdf(x, loc=mean_mu, scale=np.sqrt(sigma_quare_mu))\n", "ax = plt.subplot(121)\n", "ax.plot(x, p_mu)\n", "ax.set_xlabel(\"$\\mu$\")\n", "ax.set_ylabel(\"q($\\mu$)\")\n", "ax.set_title(\"Mean: q($\\\\mu$)\")\n", "print(\"true mu: \", mu)\n", "\n", "x = np.arange(0,1,0.01)\n", "p_tau = scipy.stats.gamma.pdf(x, a=loc_tau, scale=scale_tau)\n", "ax = plt.subplot(122)\n", "ax.plot(x, p_tau)\n", "ax.set_xlabel(\"$\\\\tau$\")\n", "ax.set_ylabel(\"q($\\\\tau$)\")\n", "ax.set_title(\"Precision: q($\\\\tau$)\")\n", "print(\"true tau: \", 1/sigma**2)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "## Literature\n", "\n", "**[TODO]**" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "## Licenses\n", "\n", "### Notebook License (CC-BY-SA 4.0)\n", "\n", "*The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).*\n", "\n", "Exercise - Variational Mean Field Approximation for Univariate Gaussian
\n", "by Christian Herta