The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

policy_data

This vignette is a guide to policy_data(). As the name suggests, the function creates a policy_data object with a specific data structure making it easy to use in combination with policy_def(), policy_learn(), and policy_eval(). The vignette is also a guide to some of the associated S3 functions which transform or access parts of the data, see ?policy_data and methods(class="policy_data").

We will start by looking at a simple single-stage example, then consider a fixed two-stage example with varying actions sets and data in wide format, and finally we will look at an example with a stochastic number of stages and data in long format.

Single-stage: wide data

Consider a simple single-stage problem with covariates/state variables \((Z, L, B)\), binary action variable \(A\), and utility outcome \(U\). We use sim_single_stage() to simulate data:

(d <- sim_single_stage(n = 5e2, seed=1)) |> head()
#>            Z          L B A          U
#> 1  1.2879704 -1.4795962 0 1 -0.9337648
#> 2  1.6184181  1.2966436 0 1  6.7506026
#> 3  1.2710352 -1.0431352 0 1 -0.3377580
#> 4 -0.2157605  0.1198224 1 0  1.4993427
#> 5 -1.0671588 -1.3663727 0 1 -9.1718727
#> 6 -1.4469746 -0.4018530 0 0 -2.6692961

We give instructions to policy_data() which variables define the action, the state covariates, and the utility variable:

pd <- policy_data(d, action="A", covariates=list("Z", "B", "L"), utility="U")
pd
#> Policy data with n = 500 observations and maximal K = 1 stages.
#> 
#>      action
#> stage   0   1   n
#>     1 278 222 500
#> 
#> Baseline covariates:
#> State covariates: Z, B, L
#> Average utility: -0.98

In the single-stage case the history \(H\) is just \((B, Z, L)\). We access the history and actions using get_history():

get_history(pd)$H |> head()
#> Key: <id, stage>
#>       id stage          Z     B          L
#>    <int> <int>      <num> <num>      <num>
#> 1:     1     1  1.2879704     0 -1.4795962
#> 2:     2     1  1.6184181     0  1.2966436
#> 3:     3     1  1.2710352     0 -1.0431352
#> 4:     4     1 -0.2157605     1  0.1198224
#> 5:     5     1 -1.0671588     0 -1.3663727
#> 6:     6     1 -1.4469746     0 -0.4018530

get_history(pd)$A |> head()
#> Key: <id, stage>
#>       id stage      A
#>    <int> <int> <char>
#> 1:     1     1      1
#> 2:     2     1      1
#> 3:     3     1      1
#> 4:     4     1      0
#> 5:     5     1      1
#> 6:     6     1      0

Similarly, we access the utility outcomes \(U\):

get_utility(pd) |> head()
#> Key: <id>
#>       id          U
#>    <int>      <num>
#> 1:     1 -0.9337648
#> 2:     2  6.7506026
#> 3:     3 -0.3377580
#> 4:     4  1.4993427
#> 5:     5 -9.1718727
#> 6:     6 -2.6692961

Two-stage: wide data

Consider a two-stage problem with observations \(O = (B, BB, L_{1}, C_{1}, U_{1}, A_1, L_2, C_{2}, U_{2}, A_2, U_{3})\). Following the general notation introduced in Section 3.1 of (Nordland and Holst 2023), \((B,BB)\) are the baseline covariates, \(S_k =(L_{k, C_{k}})\) are the state covariates at stage k, \(A_{k}\) is the action at stage k, and \(U_k\) is the reward at stage \(k\). The utility is the sum of the rewards \(U=U_{1}+U_{2}+U_{3}\).

We use sim_two_stage_multi_actions() to simulate data:

d <- sim_two_stage_multi_actions(n=2e3, seed = 1)
colnames(d)
#>  [1] "B"   "BB"  "L_1" "C_1" "A_1" "L_2" "C_2" "A_2" "L_3" "U_1" "U_2" "U_3"

Note that the data is in wide format. The data is transformed using policy_data() with instructions on which variables define the actions, baseline covariates, state covariates, and the rewards:

pd <- policy_data(d,
                  action = c("A_1", "A_2"),
                  baseline = c("B", "BB"),
                  covariates = list(L = c("L_1", "L_2"),
                                    C = c("C_1", "C_2")),
                  utility = c("U_1", "U_2", "U_3"))
pd
#> Policy data with n = 2000 observations and maximal K = 2 stages.
#> 
#>      action
#> stage default   no  yes    n
#>     1       0 1017  983 2000
#>     2     769  826  405 2000
#> 
#> Baseline covariates: B, BB
#> State covariates: L, C
#> Average utility: 0.39

The length of the character vector action determines the number of stages K (in this case 2). If the number of stages is 2 or more, the covariates argument must be a named list. Each element must be a character vector with length equal to the number of stages. If a covariate is not available at a given stage we insert an NA value, e.g., L = c(NA, "L_2").

Finally, the utility argument must be a single character string (the utility is observed after stage K) or a character vector of length K+1 with the names of the rewards.

In this example, the observed action sets vary for each stage. get_action_set() returns the global action set and get_stage_action_sets() returns the action set for each stage:

get_action_set(pd)
#> [1] "default" "no"      "yes"

get_stage_action_sets(pd)
#> $stage_1
#> [1] "no"  "yes"
#> 
#> $stage_2
#> [1] "default" "no"      "yes"

The full histories \(H_1 = (B, BB, L_{1}, C_{1})\) and \(H_2=(B, BB, L_{1}, C_{1}, A_{1}, L_{2}, C_{2})\) are available using get_history() and full_history = TRUE:

get_history(pd, stage = 1, full_history = TRUE)$H |> head()
#> Key: <id, stage>
#>       id stage        L_1        C_1          B     BB
#>    <int> <num>      <num>      <num>      <num> <char>
#> 1:     1     1  0.9696772  1.7112790 -0.6264538 group2
#> 2:     2     1 -2.1994065 -2.6431237  0.1836433 group1
#> 3:     3     1  1.9480938  2.0619342 -0.8356286 group2
#> 4:     4     1  0.1798532  1.0066957  1.5952808 group2
#> 5:     5     1  0.4150568  0.1538534  0.3295078 group2
#> 6:     6     1  0.6468405 -0.0982121 -0.8204684 group3

get_history(pd, stage = 2, full_history = TRUE)$H |> head()
#> Key: <id, stage>
#>       id stage    A_1        L_1        L_2        C_1        C_2          B
#>    <int> <num> <char>      <num>      <num>      <num>      <num>      <num>
#> 1:     1     2    yes  0.9696772 -0.7393434  1.7112790  2.4243702 -0.6264538
#> 2:     2     2     no -2.1994065  0.4828756 -2.6431237 -2.6647281  0.1836433
#> 3:     3     2     no  1.9480938  0.4803055  2.0619342  2.4747615 -0.8356286
#> 4:     4     2    yes  0.1798532 -0.3574497  1.0066957  2.0571959  1.5952808
#> 5:     5     2     no  0.4150568  2.0473541  0.1538534 -0.9649004  0.3295078
#> 6:     6     2    yes  0.6468405 -2.3701135 -0.0982121  1.0989523 -0.8204684
#>        BB
#>    <char>
#> 1: group2
#> 2: group1
#> 3: group2
#> 4: group2
#> 5: group2
#> 6: group3

Similarly, we access the associated actions at each stage via list element A:

get_history(pd, stage = 1, full_history = TRUE)$A |> head()
#> Key: <id, stage>
#>       id stage    A_1
#>    <int> <num> <char>
#> 1:     1     1    yes
#> 2:     2     1     no
#> 3:     3     1     no
#> 4:     4     1    yes
#> 5:     5     1     no
#> 6:     6     1    yes

get_history(pd, stage = 2, full_history = TRUE)$A |> head()
#> Key: <id, stage>
#>       id stage     A_2
#>    <int> <num>  <char>
#> 1:     1     2      no
#> 2:     2     2      no
#> 3:     3     2 default
#> 4:     4     2     yes
#> 5:     5     2     yes
#> 6:     6     2      no

Alternatively, the state/Markov type history and actions are available using full_history = FALSE:

get_history(pd, full_history = FALSE)$H |> head()
#> Key: <id, stage>
#>       id stage          L         C          B     BB
#>    <int> <int>      <num>     <num>      <num> <char>
#> 1:     1     1  0.9696772  1.711279 -0.6264538 group2
#> 2:     1     2 -0.7393434  2.424370 -0.6264538 group2
#> 3:     2     1 -2.1994065 -2.643124  0.1836433 group1
#> 4:     2     2  0.4828756 -2.664728  0.1836433 group1
#> 5:     3     1  1.9480938  2.061934 -0.8356286 group2
#> 6:     3     2  0.4803055  2.474761 -0.8356286 group2

get_history(pd, full_history = FALSE)$A |> head()
#> Key: <id, stage>
#>       id stage       A
#>    <int> <int>  <char>
#> 1:     1     1     yes
#> 2:     1     2      no
#> 3:     2     1      no
#> 4:     2     2      no
#> 5:     3     1      no
#> 6:     3     2 default

Note that policy_data() overrides the action variable names to A_1, A_2, … in the full history case and A in the state/Markov history case.

As in the single-stage case we access the utility, i.e. the sum of the rewards, using get_utility():

get_utility(pd) |> head()
#> Key: <id>
#>       id         U
#>    <int>     <num>
#> 1:     1  1.110369
#> 2:     2 -1.788041
#> 3:     3  2.836251
#> 4:     4  3.173743
#> 5:     5  1.891312
#> 6:     6 -1.120837

Multi-stage: long data

In this example we illustrate how polle handles decision processes with a stochastic number of stages, see Section 3.5 in (Nordland and Holst 2023). The data is simulated using sim_multi_stage(). Detailed information on the simulation is available in ?sim_multi_stage. We simulate data from 2000 iid subjects:

d <- sim_multi_stage(2e3, seed = 1)

As described, the stage data is in long format:

d$stage_data[, -(9:10)] |> head()
#>       id stage event        t      A          X     X_lead         U
#>    <num> <num> <num>    <num> <char>      <num>      <num>     <num>
#> 1:     1     1     0 0.000000      1  1.3297993  0.0000000 0.0000000
#> 2:     1     2     0 1.686561      1 -0.7926711  1.3297993 0.3567621
#> 3:     1     3     0 3.071768      0  3.5246509 -0.7926711 2.1778778
#> 4:     1     4     1 3.071768   <NA>         NA         NA 0.0000000
#> 5:     2     1     0 0.000000      1  0.7635935  0.0000000 0.0000000
#> 6:     2     2     0 1.297336      1 -0.5441694  0.7635935 0.5337427

The id variable is important for identifying which rows belong to each subjects. The baseline data uses the same id variable:

d$baseline_data |> head()
#>       id     B
#>    <num> <int>
#> 1:     1     0
#> 2:     2     0
#> 3:     3     1
#> 4:     4     1
#> 5:     5     1
#> 6:     6     0

The data is transformed using policy_data() with type = "long". The names of the id, stage, event, action, and utility variables must be specified. The event variable, inspired by the event variable in survival::Surv(), is 0 whenever an action occur and 1 for a terminal event.

pd <- policy_data(data = d$stage_data,
                  baseline_data = d$baseline_data,
                  type = "long",
                  id = "id",
                  stage = "stage",
                  event = "event",
                  action = "A",
                  utility = "U")
pd
#> Policy data with n = 2000 observations and maximal K = 4 stages.
#> 
#>      action
#> stage    0    1    n
#>     1  113 1887 2000
#>     2  844 1039 1883
#>     3  956   74 1030
#>     4   72    0   72
#> 
#> Baseline covariates: B
#> State covariates: t, X, X_lead
#> Average utility: 2.46

In some cases we are only interested in analyzing a subset of the decision stages. partial() trims the maximum number of decision stages:

pd3 <- partial(pd, K = 3)
pd3
#> Policy data with n = 2000 observations and maximal K = 3 stages.
#> 
#>      action
#> stage    0    1    n
#>     1  113 1887 2000
#>     2  844 1039 1883
#>     3  956   74 1030
#> 
#> Baseline covariates: B
#> State covariates: t, X, X_lead
#> Average utility: 2.46

SessionInfo

sessionInfo()
#> R version 4.4.1 (2024-06-14)
#> Platform: aarch64-apple-darwin23.5.0
#> Running under: macOS Sonoma 14.6.1
#> 
#> Matrix products: default
#> BLAS:   /Users/oano/.asdf/installs/R/4.4.1/lib/R/lib/libRblas.dylib 
#> LAPACK: /Users/oano/.asdf/installs/R/4.4.1/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.0
#> 
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> time zone: Europe/Copenhagen
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] splines   stats     graphics  grDevices utils     datasets  methods  
#> [8] base     
#> 
#> other attached packages:
#> [1] ggplot2_3.5.1       data.table_1.15.4   polle_1.5          
#> [4] SuperLearner_2.0-29 gam_1.22-4          foreach_1.5.2      
#> [7] nnls_1.5           
#> 
#> loaded via a namespace (and not attached):
#>  [1] sass_0.4.9          utf8_1.2.4          future_1.33.2      
#>  [4] lattice_0.22-6      listenv_0.9.1       digest_0.6.36      
#>  [7] magrittr_2.0.3      evaluate_0.24.0     grid_4.4.1         
#> [10] iterators_1.0.14    mvtnorm_1.2-5       policytree_1.2.3   
#> [13] fastmap_1.2.0       jsonlite_1.8.8      Matrix_1.7-0       
#> [16] survival_3.6-4      fansi_1.0.6         scales_1.3.0       
#> [19] numDeriv_2016.8-1.1 codetools_0.2-20    jquerylib_0.1.4    
#> [22] lava_1.8.0          cli_3.6.3           rlang_1.1.4        
#> [25] mets_1.3.4          parallelly_1.37.1   future.apply_1.11.2
#> [28] munsell_0.5.1       withr_3.0.0         cachem_1.1.0       
#> [31] yaml_2.3.8          tools_4.4.1         parallel_4.4.1     
#> [34] colorspace_2.1-0    globals_0.16.3      vctrs_0.6.5        
#> [37] R6_2.5.1            lifecycle_1.0.4     pkgconfig_2.0.3    
#> [40] timereg_2.0.5       progressr_0.14.0    bslib_0.7.0        
#> [43] pillar_1.9.0        gtable_0.3.5        Rcpp_1.0.13        
#> [46] glue_1.7.0          xfun_0.45           tibble_3.2.1       
#> [49] highr_0.11          knitr_1.47          farver_2.1.2       
#> [52] htmltools_0.5.8.1   rmarkdown_2.27      labeling_0.4.3     
#> [55] compiler_4.4.1

References

Nordland, Andreas, and Klaus K. Holst. 2023. “Policy Learning with the Polle Package.” https://doi.org/10.48550/arXiv.2212.02335.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.