Common pitfalls / FAQ

1. “impute() stopped because my input is fully observed”

Symptom. You call impute(df, tree) on a fully-observed dataset and it stops before training.

Why this happens. impute() only imputes cells that are NA in matched target rows. A fully observed input now stops before training and directs you to cross_validate() for predictive assessment. Supply a real missing target cell to use the imputation path. A complete data set is for predictive assessment, not an imputation result.

Diagnose.

library(pigauto)
data(avonet300, tree300)
df <- avonet300
rownames(df) <- df$Species_Key
df$Species_Key <- NULL
sum(is.na(df))                  # if 0, there's nothing for impute() to do

Fix. Mask some cells before calling impute(), then evaluate predictions only on the held-out cells:

set.seed(1L)
hide   <- sample(which(!is.na(df$Mass)), 30L)
df_obs <- df
df_obs$Mass[hide] <- NA          # hide 30 mass values

result <- impute(df_obs, tree300)

result$completed$Mass[hide]      # pigauto's imputations
df$Mass[hide]                    # held-out truth, for comparison
sum(result$imputed_mask[, "Mass"])  # 30

For your own data with real NAs, the imputed values you actually care about are result$completed[result$imputed_mask], not result$prediction$imputed.

See also. ?impute (“What gets imputed (read this first)”), issue #67.

2. “My ordinal trait predicted 100 % majority class”

Symptom. You impute an ordinal trait and the prediction is the majority class for every species. For example, on avonet300$Migration (K = 3 ordinal: Resident / Partial / Full), 300/0/0.

Why this happens. Two things compound:

1. If your input has no NAs in that column, impute() stops before fitting; use cross_validate() instead (see Pitfall 1).
2. At default settings (n_imputations = 1L, pool_method = "median"), a small ordinal trait whose marginal distribution is heavily skewed (AVONET Migration is ~78 % Resident / 14 % Partial / 8 % Full at n = 300) can have its calibrated gate snap to a corner that returns the majority class for every species.

Diagnose.

library(pigauto)
data(avonet300, tree300)
df <- avonet300; rownames(df) <- df$Species_Key; df$Species_Key <- NULL

table(df$Migration)              # check the marginal distribution
hide <- sample(which(!is.na(df$Migration)), 30L)
df$Migration[hide] <- NA
result <- impute(df, tree300, verbose = FALSE)
table(result$completed$Migration[hide])

Fix. Compare candidate settings with cross_validate() on your own trait table and report the held-out result. For an imbalanced ordinal trait, one candidate is more draws with pool_method = "mode"; it is not a package-wide recommendation.

set.seed(1L)
hide   <- sample(which(!is.na(df$Migration)), 30L)
df_obs <- df
df_obs$Migration[hide] <- NA

# Default settings: prone to majority-class collapse on imbalanced K = 3
result <- impute(df_obs, tree300, verbose = FALSE)
table(result$completed$Migration[hide], df$Migration[hide])

# Recommended for K = 3 ordinal: more draws + mode pooling
result_mode <- impute(df_obs, tree300, n_imputations = 20L,
                      pool_method = "mode", verbose = FALSE)
table(result_mode$completed$Migration[hide], df$Migration[hide])

See also. ?impute (“Imbalanced K-class traits”), Phase H memo, issue #68.

3. “The gate stays closed and the GNN seems to do nothing”

This section applies when you fit with gnn = TRUE. With the default gnn = FALSE no GNN is trained, so r_cal_gnn is 0 by construction and predictions come from the calibrated baseline.

Symptom. You expected the GNN to dominate, but inspecting the fitted model shows the calibrated gate is fully or near-fully closed (r_cal_gnn ≈ 0) — predictions equal the BM baseline.

Why this happens. This is the gate’s design behaviour, not a bug (with the default safety_floor = FALSE the mean term is absent and r_cal_mean is 0; the next sentence describes safety_floor = TRUE). After training, pigauto picks the per-latent-column gate that minimises validation loss across the simplex \(r_\text{BM} \cdot \text{BM} + r_\text{GNN} \cdot \text{GNN} + r_\text{MEAN} \cdot \text{MEAN}\). When the GNN cannot beat BM on the held-out validation set, the optimum can be r_cal_gnn = 0. In that case the calibrated prediction stays on the validation-supported baseline or mean corner instead of forcing a GNN contribution. This is what the package was designed to do on high-phylogenetic-signal traits where BM is already hard to beat.

Diagnose.

library(pigauto)
data(avonet300, tree300)
df <- avonet300; rownames(df) <- df$Species_Key; df$Species_Key <- NULL
fit <- impute(df, tree300, verbose = FALSE)$fit

# Per-latent-column calibrated gates (since v0.9.1.9002):
fit$r_cal_bm        # r assigned to the BM baseline
fit$r_cal_gnn       # r assigned to the GNN delta
fit$r_cal_mean      # r assigned to the grand mean

A row where r_cal_gnn is small (< 0.1) means the gate has effectively closed for that latent column.

Fix. Often there is nothing to fix — the closed gate is evidence of high phylogenetic signal, not a problem. If you suspect the GNN should be helping (e.g. you’ve added covariates, or the trait has known cross-trait structure) but the gate is closed:

• Check the validation set is not pathologically small (the “small validation set” warning during fitting is a red flag — see ?fit_pigauto “Calibration at small n”).
• Verify covariates are not all-NA or constant after preprocessing.
• For ordinal / categorical traits, see Pitfall 2 — the gate may be closing onto a majority-class corner that mode pooling resolves.

See also. ?fit_pigauto (phylo_signal_gate, “Safety floor”), design spec.

4. “How do I know if my dataset has enough phylogenetic signal?”

Symptom. You aren’t sure whether phylogenetic information is useful for your dataset.

Why this matters. A lambda estimate describes fitted phylogenetic structure; it does not determine predictive performance on a new dataset. Use held-out evaluation rather than a generic threshold to compare a phylogenetic workflow with an appropriate alternative.

Diagnose. The fitted object stores the per-trait λ values used by phylo_signal_gate (they are computed only when the gate is switched on, which is no longer the default):

library(pigauto)
data(avonet300, tree300)
df <- avonet300; rownames(df) <- df$Species_Key; df$Species_Key <- NULL

fit <- impute(df, tree300, phylo_signal_gate = TRUE)$fit
fit$phylo_signal_per_trait

The output reports λ on the observed cells where it can be estimated. Discrete traits use the package’s internal continuous proxy on the latent/liability scale.

Fix. Use the lambda estimate as a diagnostic, then run cross_validate() on the input and compare the candidate workflows under the same held-out design. If phylogenetic structure is weak, consider whether a non-phylogenetic analysis is more appropriate for your scientific question.

See also. ?fit_pigauto (phylo_signal_gate), Phase 8 signal sweep memo.

5. “Predictions are way bigger than anything I observed”

Symptom. A masked log-transformed continuous trait (body mass, seed mass, fish weight) predicts a value 50–100× larger than anything observed. On AVONET, the canonical case is the cassowary: truth ≈ 35 kg, predicted up to ~540 kg.

Why this happens. For log-transformed traits, the GNN’s MC-dropout draws are on the log scale. A latent ~+3-4 σ above the training distribution survives as a ~50-100× value error after expm1() back-transformation. With n_imputations = 1, a single unlucky dropout pattern can produce this; with pool_method = "median" (default) the median of M draws is robust to one bad draw, but a small M (≤ 5) on the long tail of the latent distribution can still mis-pool.

Diagnose.

# After running impute(), check whether any imputation exceeds the
# observed maximum by an unrealistic factor:
predicted_mass <- result$completed$Mass[result$imputed_mask[, "Mass"]]
obs_max <- max(df$Mass, na.rm = TRUE)
sum(predicted_mass > 5 * obs_max)

A non-zero count is a signal of tail extrapolation.

Fix. Phase G clamp_outliers = TRUE caps post-back-transform predictions for log-transformed continuous, count, and zi_count magnitude traits at obs_max * clamp_factor (default 5). This is opt-in because for legitimate growth-curve datasets where 5× the observed maximum is plausible, you don’t want it.

result <- impute(df_obs, tree300,
                 clamp_outliers = TRUE,
                 clamp_factor   = 5,    # Tukey-style outlier cap
                 verbose = FALSE)

See also. ?impute (clamp_outliers, clamp_factor arguments), AVONET Mass diagnosis memo, Phase G results.