Super Learner: Cross-Validation Based Ensemble Learning

Super learning with functional programming!

Usage

super_learner(
  data,
  learners,
  formulas,
  y_variable = NULL,
  n_folds = 5,
  determine_super_learner_weights = NULL,
  ensemble_or_discrete = c("ensemble", "discrete"),
  cv_schema,
  outcome_type = c("continuous", "binary", "density", "multiclass"),
  extra_learner_args = NULL,
  cluster_ids = NULL,
  strata_ids = NULL,
  weights = NULL,
  rowids = NULL,
  use_complete_cases = FALSE,
  train_on_whole_dataset = TRUE
)

Arguments

data: Data to use in training a super_learner.
learners: A list of predictor/closure-returning-functions. See Details.
formulas: Either a single regression formula or a vector of regression formulas.
y_variable: Typically y_variable can be inferred automatically from the formulas, but if needed, the y_variable can be specified explicitly.
n_folds: The number of cross-validation folds to use in constructing the super_learner. Must be a scalar integer of value >= 2.
determine_super_learner_weights: A function/method to determine the weights for each of the candidate learners. The default is to use determine_super_learner_weights_nnls.
ensemble_or_discrete: Defaults to 'ensemble', but can be set to 'discrete'. Discrete super_learner() chooses only one of the candidate learners to have weight 1 in the resulting prediction algorithm, while ensemble super_learner() combines predictions from 1 or more candidate learners, with respective weights adding up to 1.
cv_schema: A function that takes data, n_folds and returns a list containing training_data and validation_data, each of which are lists of n_folds data frames.
outcome_type: One of 'continuous', 'binary', 'multiclass', or 'density'. outcome_type is used to infer the correct determine_super_learner_weights function if it is not explicitly passed.
extra_learner_args: A list of equal length to the learners with additional arguments to pass to each of the specified learners.
cluster_ids: (default: null) If specified, clusters will either be entirely assigned to training or validation (not both) in each cross-validation split.
strata_ids: (default: null) If specified, strata are balanced across training and validation splits so that strata appear in both the training and validation splits.
weights: If specified, (per observation) weights are used to indicate that risk minimization across models (i.e., the meta-learning step) should be targeted to higher weight observations.
rowids: (default: null) If specified, rowids are stored for use in out-of-fold prediction ($oof_predict()) and can also be used to validate/observe the behavior of the cross-validation folds construction.
use_complete_cases: (default: FALSE) If the data passed have any NA or NaN missing data, restrict the data to data[complete.cases(data),].
train_on_whole_dataset: (default: TRUE) If TRUE, after the cross-validation stages complete, each candidate learner is additionally fit on the whole dataset; these whole-dataset fits are combined with the meta-learned weights to form the $predict() function returned. If FALSE, the whole-dataset fits are skipped entirely, saving roughly 1/n_folds of the total learner-fitting compute. The resulting model retains all out-of-fold prediction interfaces ($oof_predictions, $oof_predict(), $oof_predict_modified(), $oof_predict_fold()) as well as fitted(), residuals(), coef(), summary(), plot(), and compare_learners(), since these depend only on the per-fold fits and held-out predictions – but $predict() / predict.nadir_sl_model() error informatively and $fit_learners is NULL. This is intended for workflows in which only out-of-fold predictions, or counterfactual modifications thereof, are needed – e.g. in nuisance estimation for causal inference estimators or simulation studies.

Value

An object of class inheriting from nadir_sl_model. This is an S3 object, with elements including a $predict(newdata) method, and some information about the fit model including y_variable, outcome_type, learner_weights, holdout_predictions and optionally information about any errors or warnings thrown by the learner fitting process. holdout_predictions rows are in cross-validation fold order, while the .sl_rowid column maps each row to its row index in the input data (after any complete-case filtering is applied if use_complete_cases is enabled). If any learners signaled warnings, they are captured (not printed) and returned in $warnings_from_training_cv_stage1, $warnings_from_predicting_cv_stage2, and $warnings_from_training_on_entire_data, each a list of warning conditions named by learner with user-legible $calls; $warning_learners lists the learners that warned. If train_on_whole_dataset = FALSE was used, $fit_learners is NULL and $predict() errors informatively; see that parameter's documentation.

Details

The goal of any super learner is to use cross-validation and a set of candidate learners to 1) evaluate how the learners perform on held out data and 2) to use that evaluation to produce a weighted average (for continuous super learner) or to pick a best learner (for discrete super learner) of the specified candidate learners.

Super learner and its statistically desirable properties have been written about at length, including at least the following references:

nadir::super_learner adopts several user-interface design-perspectives that will be useful to know in understanding what it does and how it works:

The specification of learners should be very flexible, really only constrained by the fact that candidate learners should be designed for the same prediction problem but their details can wildly vary from learner to learner.
It should be easy to specify a customized or new learner.

nadir::super_learner at its core accepts data, a formula (a single one passed to formulas is fine), and a list of learners.

learners are taken to be lists of functions of the following specification:

a learner must accept a data and formula argument,
a learner may accept more arguments, and
a learner must return a prediction function that accepts newdata and produces a vector of prediction values given newdata.

In essence, a learner is specified to be a function taking (data, formula, ...) and returning a closure (see http://adv-r.had.co.nz/Functional-programming.html#closures for an introduction to closures) which is a function accepting newdata returning predictions.

Since many candidate learners will have hyperparameters that should be tuned, like depth of trees in random forests, or the lambda parameter for glmnet, extra arguments can be passed to each learner via the extra_learner_args argument. extra_learner_args should be a list of lists, one list of extra arguments for each learner. If no additional arguments are needed for some learners, but some learners you're using do require additional arguments, you can just put a NULL value into the extra_learner_args. See the examples.

In order to seamlessly support using features implemented by extensions to the formula syntax (like random effects formatted like random intercepts or slopes that use the (age | strata) syntax in lme4 or splines like s(age | strata) in mgcv), we allow for the formulas argument to either be one fixed formula that super_learner will use for all the models, or a vector of formulas, one for each learner specified. Most learners receive the formula directly, but some learners specifically require matrix input. nadir automatically does the conversion of formula and data to matrix input for those learners like lnr_glmnet, lnr_hal, lnr_xgboost and others.

Note that in the examples a mean-squared-error (mse) is calculated on the same training/test set, and this is only useful as a crude diagnostic to see that super_learner is working. A more rigorous performance metric to evaluate super_learner on is the cv-rmse produced by cv_super_learner.

Examples

# most basic usage:
sl <- super_learner(mtcars, list(lm = lnr_lm, mean = lnr_mean),
                    mpg ~ hp + wt, n_folds = 2)
coef(sl)        # ensemble weights
#>         lm       mean 
#> 0.92388493 0.07611507 
head(fitted(sl))
#> [1] 23.77383 22.64693 23.97326 21.14440 17.71882 20.18613
head(residuals(sl))
#> [1] -2.7738289 -1.6469301 -1.1732642  0.2556017  0.9811792 -2.0861350
formula(sl)
#> mpg ~ hp + wt
#> <environment: 0x3232f6860>
nobs(sl)
#> [1] 32
summary(sl)
#> Summary of Super Learner fit
#>   outcome: mpg (continuous);  n = 32;  folds = 2
#> 
#>  learner weight cv_holdout_loss
#>       lm 0.9239           8.113
#>     mean 0.0761          39.580
#> 
#> Loss is evaluated on held-out (cross-validation) data; lower is better.

# here's a slightly more in depth example with fancy formula features.
# note all these packages (glmnet, lme4, randomForest) are highly
# Suggested (among others) with {nadir}.

if (requireNamespace("glmnet", quietly = TRUE) &&
    requireNamespace("lme4", quietly = TRUE) &&
    requireNamespace("randomForest", quietly = TRUE)) {

learners <- list(
  glm = lnr_glm,
  rf = lnr_rf,
  glmnet = lnr_glmnet,
  lmer = lnr_lmer
)

# mtcars example ---
formulas <- c(
  .default = mpg ~ cyl + hp, # first three models use same formula
  lmer = mpg ~ (1 | cyl) + hp # lme4 uses different language features
)

# fit a super_learner
sl_model <- super_learner(
  data = mtcars,
  formula = formulas,
  learners = learners
)

# We recommend taking a look at this object to see what's contained inside it:
sl_model

compare_learners(sl_model)

# iris example ---
sl_model <- super_learner(
  data = iris,
  formula = list(
    .default = Sepal.Length ~ Sepal.Width + Petal.Length + Petal.Width,
    lmer = Sepal.Length ~ (Sepal.Width | Species) + Petal.Length
  ),
  learners = learners
)

# produce super_learner predictions and compare against the individual learners
compare_learners(sl_model)
}
#> Inferring the loss metric for learner comparison based on the outcome type: 
#> outcome_type=continuous -> using mean squared error
#> boundary (singular) fit: see help('isSingular')
#> boundary (singular) fit: see help('isSingular')
#> Inferring the loss metric for learner comparison based on the outcome type: 
#> outcome_type=continuous -> using mean squared error
#> # A tibble: 1 × 4
#>     glm    rf glmnet   lmer
#>   <dbl> <dbl>  <dbl>  <dbl>
#> 1 0.101 0.129  0.207 0.1000