Covariance and Correlation Matrices

The symmetric and positive definite covariance matrix \[ \Sigma = \begin{pmatrix} \sigma_1^2 & \sigma_{12} & \dots & \dots & \sigma_{1m} \\ \sigma_{21} & \sigma_2^2 & \sigma_{23} & \dots & \sigma_{2m}\\ \vdots & & \ddots & & \vdots \\ \vdots & & & \ddots & \vdots \\ \sigma_{m1} & \dots & \dots & \sigma_{m,m-1} & \sigma_m^2 \end{pmatrix} \] is parametrized by a vector of variance parameters \(\theta = (\theta_1, \dotsc, \theta_k)^\top\). The meaning and the number (\(k\)) of variance parameters is different for each covariance structure.

In many covariance structures we use the decomposition \[ \Sigma = DPD \] where the diagonal standard deviation matrix is \[ D = \begin{pmatrix} \sigma_1 & 0 & \cdots & & 0 \\ 0 & \sigma_2 & 0 & \cdots & 0 \\ \vdots & & \ddots & \ddots & \vdots \\ & & & \sigma_{m-1} & 0 \\ 0 & \cdots & & 0 & \sigma_m \end{pmatrix} \] with entries \(\sigma_i > 0\), and the symmetric correlation matrix \(P\) is \[ P = \begin{pmatrix} 1 & \rho_{12} & \rho_{13} & \cdots & \cdots & \rho_{1,m-1} \\ \rho_{12} & 1 & \rho_{23} & \ddots & & \vdots \\ \rho_{13} & \rho_{23} & \ddots & \ddots & \ddots & \vdots \\ \vdots & \ddots & \ddots & \ddots & \rho_{m-2,m-1} & \rho_{m-2,m} \\ \vdots & & \ddots & \rho_{m-2,m-1} & 1 & \rho_{m-1,m} \\ \rho_{1, m-1} & \cdots & \cdots & \rho_{m-2,m} & \rho_{m-1,m} & 1 \end{pmatrix} \] with entries \(\rho_{ij} \in (-1, 1)\). Since these covariance structures assume different variances for each time point they are called “heterogeneous” covariance structures. Assuming a constant \(\sigma = \sigma_1 = \dotsb = \sigma_m\) gives a “homogeneous” covariance structure instead.

Transformation to Variance Parameters

For standard deviation parameters \(\sigma\) we use the natural logarithm \(\log(\sigma)\) to map them to \(\mathbb{R}\). For correlation parameters \(\rho\) we consistently use the transformation \[ \theta = f(\rho) = \mathop{\mathrm{sign}}(\rho) \sqrt{\frac{\rho^2}{1 - \rho^2}} \] which maps the correlation parameter to \(\theta \in \mathbb{R}\). It has the inverse \[ \rho = f^{-1}(\theta) = \frac{\theta}{\sqrt{1 + \theta^2}}. \] This is important because the resulting variance parameters \(\theta\) can be optimized without constraints over the whole of \(\mathbb{R}\).

Unstructured (us)

Any covariance matrix can be represented by this saturated correlation structure. Here \(k = m (m+1) / 2\). See the algorithm vignette for details.

Homogeneous (ad) and Heterogeneous Ante-dependence (adh)

The ante-dependence correlation structure (Gabriel 1962) is useful for balanced designs where the observations are not necessarily equally spaced in time. Here we use an order one ante-dependence model, where the correlation matrix \(P\) has elements \[ \rho_{ij} = \prod_{k=i}^{j-1} \rho_k. \] So we have correlation parameters \(\rho_k\), \(k = 1, \dotsc, m-1\).

We use a heterogeneous covariance structure to allow for different within subject variances. So here we can identify \[ \theta = (\log(\sigma_1), \dotsc, \log(\sigma_m), f(\rho_1), \dotsc, f(\rho_{m-1})) \] and we have in total \(2m - 1\) variance parameters. Assuming a constant variance yields a homogeneous ante-dependence covariance structure with total \(m\) variance parameters.

Note our naming convention for the homogeneous and heterogeneous covariance structures that a suffix h is used to denote the heterogeneous version, e.g, ad for homogeneous and adh for heterogeneous ante-dependence. This is different from the name used in SAS for ante-dependence covariance structure, where ANTE(1) refers to heterogeneous ante-dependence covariance structure and a homogeneous version is not provided in SAS.

Homogeneous (toep) and Heterogeneous Toeplitz (toeph)

Toeplitz matrices (Toeplitz 1911) are diagonal-constant matrices. Here we can model the correlation matrix as a Toeplitz matrix: \[ P = \begin{pmatrix} 1 & \rho_1 & \rho_2 & \cdots & \cdots & \rho_{m-1} \\ \rho_1 & 1 & \rho_1 & \ddots & & \vdots \\ \rho_2 & \rho_1 & \ddots & \ddots & \ddots & \vdots \\ \vdots & \ddots & \ddots & \ddots & \rho_1 & \rho_2 \\ \vdots & & \ddots & \rho_1 & 1 & \rho_1 \\ \rho_{m-1} & \cdots & \cdots & \rho_2 & \rho_1 & 1 \end{pmatrix} \] This means that the correlation between two time points only depends on the distance between them, i.e. \[ \rho_{ij} = \rho_{\left\vert i - j\right\vert} \] and we have correlation parameters \(\rho_k\), \(k = 1, \dotsc, m-1\).

We use a heterogeneous covariance structure to allow for different within subject variances. So here we can identify \[ \theta = (\log(\sigma_1), \dotsc, \log(\sigma_m), f(\rho_1), \dotsc, f(\rho_{m-1})) \] and we have in total \(2m - 1\) variance parameters. This is similar to the heterogeneous ante-dependence structure, but the correlation parameters are used differently in the construction of \(P\). Assuming a constant variance yields a homogeneous Toeplitz covariance structure with total \(m\) variance parameters.

Homogeneous (ar1) and Heterogeneous (ar1h) Autoregressive

The autoregressive covariance structure can be motivated by the corresponding state-space equation \[ y_{it} = \varphi y_{i,t-1} + \varepsilon_t \] where the white noise \(\varepsilon_t\) has a normal distribution with mean zero and a constant variance. It can be shown that this gives correlations \[ \rho_{ij} = \rho^{\left\vert i - j\right\vert}. \] where \(\rho\) is related to \(\varphi\) and the variance and is the single correlation parameter here.

Assuming a constant variance in the state-space equation yields a homogeneous autoregressive covariance structure with total only \(k=2\) variance parameters, otherwise we obtain the heterogeneous autoregressive covariance structure with \(k = m + 1\) variance parameters.

Homogeneous (cs) and Heterogeneous (csh) Compound Symmetry

The compound symmetry covariance structures assume a constant correlation between different time points: \[ \rho_{ij} = \rho \] where \(\rho\) is the single correlation parameter here.

Assuming a constant variance in the state-space equation yields a homogeneous compound symmetry covariance structure with total only \(k=2\) variance parameters, otherwise we obtain the heterogeneous compound symmetry covariance structure with \(k = m + 1\) variance parameters.

Spatial exponential (sp_exp)

For spatial exponential, the covariance structure is defined as follows:

\[ \rho_{ij} = \rho^{d_{ij}} \]

where \(d_{ij}\) is the distance between time point \(i\) and time point \(j\),

A total number of parameters \(k = 2\) is needed:

The parameterization for \(\theta\) is a little different from previous examples. In previous examples, \(\rho\) can take values from -1 to 1, but here we need to restrict \(\rho\) to (0, 1). Hence we have the following parametrization form.

\[ \theta = (\log(\sigma), \mathop{\mathrm{logit}}(\rho)) \]