--- title: "Get Started with forecastdom" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Get Started with forecastdom} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 4 ) ``` ## Overview **forecastdom** collects forecast evaluation tests under the taxonomy of Li, Liao, and Quaedvlieg (2022): | | **Equal accuracy** | **Superior accuracy** | |---|---|---| | **Unconditional** | `dm_test()` | `spa_test()` | | **Conditional** | `gw_test()` | `cspa_test()` | It also bundles the tests usually paired with these in applied work: nested-model comparisons (`cw_test()`, `enc_new()`, `mse_f_test()`), return predictability (`ivx_wald()`), and parameter instability (`qll_hat()`). ```{r setup} library(forecastdom) set.seed(42) ``` ## Pairwise Forecast Comparison ### Diebold-Mariano test Compares two sets of forecast errors and asks whether the average loss differential is zero. ```{r dm} e1 <- rnorm(200) e2 <- rnorm(200, mean = 0.15) dm_test(e1, e2) ``` The default `correction = TRUE` applies the Harvey, Leybourne, and Newbold (1997) small-sample correction, which uses a $t$ distribution with $T - 1$ degrees of freedom in place of the asymptotic normal. ### Clark-West test The standard DM test is undersized when the two models are nested. The Clark-West (2007) MSFE-adjusted statistic fixes this by adding a correction term for the extra parameters in the larger model and is approximately $N(0,1)$ under the null. ```{r cw} actual <- rnorm(200) f1 <- actual + rnorm(200, sd = 0.5) # benchmark (restricted) f2 <- actual + rnorm(200, sd = 0.4) # alternative (unrestricted) cw_test(actual - f1, actual - f2, f1, f2) ``` ### Giacomini-White test Asks whether two methods have equal predictive ability *given* the information available to the forecaster. The default instruments are a constant and the lagged loss differential. ```{r gw} gw_test(e1, e2) ``` ## Multiple Forecast Comparison ### Hansen's SPA test Asks whether the benchmark has the lowest expected loss among all competitors. The null is that no competitor beats the benchmark on average; the bootstrap controls family-wise error across the $J$ comparisons. ```{r spa} sim <- do_sim(J = 3, n = 250, a = 1, c = 0, rho_u = 0.4) spa_test(sim$Y, level = 0.05, B = 1000) ``` ### CSPA test Li, Liao, and Quaedvlieg (2022) ask the same question conditional on a state variable $X_t$. The null is $$H_0:\ \min_{j}\, \mathbb{E}[L_{0,t} - L_{j,t} \mid X_t = x] \le 0 \quad \text{for all } x,$$ i.e. there is no competitor and no state value where the benchmark is strictly worse. The test combines a Legendre-polynomial series estimate of the conditional mean differentials $h_j(x)$ with an HAC-based Gaussian-process critical value. ```{r cspa-null} # Under the null (a = 1) the benchmark is weakly dominant sim_null <- do_sim(J = 3, n = 500, a = 1, c = 0, rho_u = 0.4) cspa_test(sim_null$Y, sim_null$X, level = 0.05, trim = 2) ``` ```{r cspa-alt} # Under the alternative (a = 1.5) one competitor beats the benchmark in some states sim_alt <- do_sim(J = 3, n = 500, a = 1.5, c = 0, rho_u = 0.4) result <- cspa_test(sim_alt$Y, sim_alt$X, level = 0.05, trim = 2) result ``` ### Visualization `cspa_test_plot()` draws the estimated conditional means $\hat h_j(x)$, their lower envelope, and the upper confidence bound on that envelope. The CSPA null is rejected when the dashed bound drops below zero anywhere on the support of $x$. ```{r cspa-plot} cspa_test_plot(result) ``` ### CSMS: confidence set for the most superior If no model is singled out as the benchmark, `csms()` runs the CSPA test once per candidate and keeps the candidates that are *not* rejected. The resulting set is asymptotically a level-$(1-\alpha)$ confidence set for the conditionally most-superior method. ```{r csms, eval = FALSE} csms(losses, X, level = 0.05, trim = 2, method_names = c("AR1", "HAR", "HARQ", "LASSO")) ``` ## Predictive Regressions ### IVX-Wald test Kostakis, Magdalinos, and Stamatogiannis (2015) build an instrument from a near-stationary transformation of the regressor so that the predictability test has a standard $\chi^2$ distribution even when the predictor is near-unit-root. ```{r ivx} n <- 300 x <- cumsum(rnorm(n)) y <- 0.02 * x + rnorm(n) ivx_wald(y, as.matrix(x), K = 1, M_n = floor(n^(1/3))) ``` ### Elliott-Muller test Tests $H_0:\ \beta_t = \beta$ for all $t$ against the alternative that the regression coefficient drifts. The statistic $\widehat{qLL}$ has a non-standard limit; critical values are tabulated in Elliott and Muller (2006, Table 1). ```{r qll} X <- matrix(rnorm(n * 2), n, 2) y2 <- X %*% c(0.5, -0.3) + rnorm(n) qll_hat(y2, X, L = floor(n ^ (1 / 3))) ``` ## References - Clark, T.E. and McCracken, M.W. (2001). Tests of Equal Forecast Accuracy and Encompassing for Nested Models. *Journal of Econometrics*, 105(1), 85-110. - Clark, T.E. and West, K.D. (2007). Approximately Normal Tests for Equal Predictive Accuracy in Nested Models. *Journal of Econometrics*, 138(1), 291-311. - Diebold, F.X. and Mariano, R.S. (1995). Comparing Predictive Accuracy. *Journal of Business & Economic Statistics*, 13(3), 253-263. - Elliott, G. and Muller, U.K. (2006). Efficient Tests for General Persistent Time Variation in Regression Coefficients. *Review of Economic Studies*, 73(4), 907-940. - Giacomini, R. and White, H. (2006). Tests of Conditional Predictive Ability. *Econometrica*, 74(6), 1545-1578. - Hansen, P.R. (2005). A Test for Superior Predictive Ability. *Journal of Business & Economic Statistics*, 23(4), 365-380. - Harvey, D., Leybourne, S., and Newbold, P. (1997). Testing the Equality of Prediction Mean Squared Errors. *International Journal of Forecasting*, 13(2), 281-291. - Kostakis, A., Magdalinos, T., and Stamatogiannis, M.P. (2015). Robust Econometric Inference for Stock Return Predictability. *Review of Financial Studies*, 28(5), 1506-1553. - Li, J., Liao, Z., and Quaedvlieg, R. (2022). Conditional Superior Predictive Ability. *Review of Economic Studies*, 89(2), 843-875.