WTF Is Marginal Likelihood

As if "likelihood" wasn't confusing enough now we have "marginal likelihood" (or "integrated likelihood") WTF It's also sometimes called "model evidence" or "evidence". Why?

wikipedia's definition is $$ p(\mathbb{X}|\alpha) = \int_\theta p(\mathbb{X}|\theta) \, p(\theta|\alpha)\ \operatorname{d}\!\theta $$

Isn't this just the bottom of Bayes' formula? So it's the probability of the observations? But why call it these strange names?

Wait that's not quite right. You can't actually know the probability of an observation. You need the assumption of what generated the observations.

So it's the probability of the observations under the chosen model. But this is still too simple because you don't actually have this probability either. A model has parameters ... how can you know the probability of these observations without a parametrized model?

So it's expressed as the marginalization of the likelihood since the likelihood can be calculated directly. No we have the probability of the observations under this model is the probability of these observations under this model with some parameters theta (likelihood) times the prior probability of theta integrated over all possible parameters theta. And since this number is how probable whatever model you chose is, it's also called the evidence of this model.

Matrix Inversion Lemma

This equation is used everywhere.


Matrix Inversion Lemma: $$\mathbf{(A^{-1} + B^{-1})^{-1} = A - A(A+B)^{-1} A = B-B(A+B)^{-1}B}$$
I saw this and went WTF? But let's not panic. Let's take this in the one dimensional case and see if it holds.
Working backwards ... $$ \begin{align*} a-a(a+b)^{-1} a &= a-\frac{a^2}{a+b} \\ &= \frac{a(a+b)}{a+b} - \frac{a^2}{a+b} \\ &= \frac{a^2 + ab}{a+b} - \frac{a^2}{a+b} \\ &= \frac{ab}{a+b} \end{align*} $$
Oh hey that's not too bad. The last equation is easily recognizable as $$\frac{1}{\frac{1}{a} + \frac{1}{b}}$$ which when written as $$(a^{-1} + b^{-1})^{-1}$$ is our original equation as expected.
OK so just repeat it with matrices! $$ \begin{align*} \mathbf{(A^{-1} + B^{-1})^{-1}} &= \mathbf{ (A^{-1} (BB^{-1}) + (A^{-1}A)B^{-1})^{-1} } \\ &= \mathbf{(A^{-1} (A+B) B^{-1})^{-1}} \\ &= \mathbf{ (B (A+B)^{-1} A ) } \\ &= \mathbf{ ((A+B - A) (A+B)^{-1} A ) } \\ &= \mathbf{(A+B)(A+B)^{-1}A - A (A+B)^{-1}A} \\ &= \mathbf{A - A(A+B)^{-1}A} \end{align*} $$ Note: This is also known as the Woodbury matrix identity.