vignettes/factor_model.Rmd
factor_model.RmdThe Gaussian factor model for \({\boldsymbol{X}}= (X_1, \ldots, X_q)\) is given by \[\begin{equation} {\boldsymbol{X}}_{q\times 1} = {\boldsymbol{\Lambda}}_{q \times k} {\boldsymbol{Z}}_{k \times 1} + {\boldsymbol{\varepsilon}}_{q\times 1}, \tag{1} \end{equation}\] where \({\boldsymbol{Z}}\sim \mathcal{N}({\boldsymbol{0}},{\boldsymbol{I}}_k)\) for \(k \le q\), \({\boldsymbol{\Lambda}}\) is a (deterministic) loading matrix, and \({\boldsymbol{\varepsilon}}\sim \mathcal{N}({\boldsymbol{0}}, {\boldsymbol{\Psi}})\), where \({\boldsymbol{\Psi}}= \operatorname{diag}({\boldsymbol{\sigma}}^2) = \operatorname{diag}(\sigma_1^2, \ldots, \sigma_q^2)\). The distribution of \({\boldsymbol{X}}\) is thus multivariate normal, \[ {\boldsymbol{X}}\sim \mathcal{N}({\boldsymbol{0}}, {\boldsymbol{\Lambda}}{\boldsymbol{\Lambda}}' + {\boldsymbol{\Psi}}). \]
The log-density of the Gaussian factor model (1) is \[ \log p({\boldsymbol{X}}\mid {\boldsymbol{\Lambda}}, {\boldsymbol{\sigma}}) = \tfrac 1 2 \left\{{\boldsymbol{X}}' {\boldsymbol{\Sigma}}^{-1} {\boldsymbol{X}}+ \log |{\boldsymbol{\Sigma}}| \right\}, \qquad {\boldsymbol{\Sigma}}= {\boldsymbol{\Lambda}}{\boldsymbol{\Lambda}}' + {\boldsymbol{\Psi}}. \] If calculated directly, the computational cost is \(\operatorname{\mathcal{O}}(q^3)\) as incurred by the matrix inversion. However, the Wodbury formula allows the inverse to be computed as \[ \begin{aligned} ({\boldsymbol{\Psi}}+ {\boldsymbol{\Lambda}}{\boldsymbol{\Lambda}}')^{-1} = {\boldsymbol{\Psi}}^{-1} - {\boldsymbol{\Psi}}^{-1}{\boldsymbol{\Lambda}}{\boldsymbol{\Omega}}^{-1}{\boldsymbol{\Lambda}}'{\boldsymbol{\Psi}}^{-1}, \qquad {\boldsymbol{\Omega}}= {\boldsymbol{I}}_k + {\boldsymbol{\Lambda}}'{\boldsymbol{\Psi}}^{-1}{\boldsymbol{\Lambda}}. \end{aligned} \] The cost of the inversion is thus \(\operatorname{\mathcal{O}}(k^4q)\), which can be much smaller than \(\operatorname{\mathcal{O}}(q^3)\) when \(k \ll q\). Similarly for the log-determinant, by the matrix determinant lemma we have \[ \log |{\boldsymbol{\Psi}}+ {\boldsymbol{\Lambda}}{\boldsymbol{\Lambda}}'| = \log |{\boldsymbol{\Psi}}| + \log|{\boldsymbol{\Omega}}|. \]