Further considerations

Normalized Entropy

Golan et al. [1] defined normalized entropy for the signal, \(\mathbf{X}\boldsymbol{\beta}\), in the GME framework, as

\[\begin{align} \qquad \qquad \qquad \qquad \qquad S(\mathbf{\widehat{p}})=\frac{-\mathbf{\widehat{p}}' \text{ln}\mathbf{\widehat{p}}}{(K+1)\text{ln} M} \qquad \qquad \qquad \qquad \qquad (3) \end{align}\] where \(S(\mathbf{\widehat{p}})\in[0,1]\) and \(S(\mathbf{\widehat{p}})=1\) indicates perfect uncertainty, and \(S(\mathbf{\widehat{p}})=0\) indicates no uncertainty.

In the GCE framework it can be defined as

\[\begin{align} \qquad \qquad \qquad \qquad \qquad S(\mathbf{\widehat{p}})=\frac{-\mathbf{\widehat{p}}' \text{ln}\mathbf{\widehat{p}}}{-\mathbf{\widehat{q}}' \text{ln}\mathbf{\widehat{q}}} \qquad \qquad \qquad \qquad \qquad (4) \end{align}\]

but in this case the we can no longer state that \(S(\mathbf{\widehat{p}})\in[0,1]\).

GCEstim package reports normalized entropies but it uses always the definition in (3) independently of the framework used.

Consider dataGCE (see “Generalized Maximum Entropy framework” and Generalized Cross Entropy framework”).

The GME estimation can be obtained, for instance, with

res.lmgce.100.GME <-
  GCEstim::lmgce(
    y ~ .,
    data = dataGCE,
    cv = TRUE,
    cv.nfolds = 5,
    support.signal = c(-100, 100),
    support.signal.points = 5,
    twosteps.n = 0,
    seed = 230676
  )

and the GCE estimation with

res.lmgce.100.GCE <-
  GCEstim::lmgce(
    y ~ .,
    data = dataGCE,
    cv = TRUE,
    cv.nfolds = 5,
    support.signal = c(-100, 100),
    support.signal.points =
      matrix(
        c(
          rep(1 / 5, 5),
          c(0.1, 0.1, 0.6, 0.1, 0.1),
          c(0.1, 0.1, 0.6, 0.1, 0.1),
          rep(1 / 5, 5),
          rep(1 / 5, 5),
          rep(1 / 5, 5)
        ),
        ncol = 5,
        byrow = TRUE
      ),
    twosteps.n = 0,
    seed = 230676
  )

The NormEnt extracts the normalized entropy from the models by default (model=TRUE).

NormEnt(res.lmgce.100.GME)
#> [1] 0.9977441
NormEnt(res.lmgce.100.GCE)
#> [1] 0.9187735

Each estimate has its own normalized entropy associated (model=FALSE)

NormEnt(res.lmgce.100.GME, model = FALSE)
#> (Intercept)        X001        X002        X003        X004        X005 
#>   0.9999346   0.9999985   0.9997937   0.9993152   0.9956162   0.9918064

NormEnt(res.lmgce.100.GCE, model = FALSE)
#> (Intercept)        X001        X002        X003        X004        X005 
#>   0.9999345   0.7627071   0.7627323   0.9993525   0.9958272   0.9920873

References

Acknowledgements

This work was supported by Fundação para a Ciência e Tecnologia (FCT) through CIDMA and projects https://doi.org/10.54499/UIDB/04106/2020 and https://doi.org/10.54499/UIDP/04106/2020.