Cross-Validating a super_learner — cv_super

Produce cv-rmse for a super_learner specified by a closure that accepts data and returns a super_learner prediction function.

Usage

cv_super_learner(
  data,
  learners,
  formulas,
  y_variable = NULL,
  n_folds = 5,
  determine_super_learner_weights = NULL,
  ensemble_or_discrete = c("ensemble", "discrete"),
  cv_schema = NULL,
  outcome_type = c("continuous", "binary", "density", "multiclass"),
  extra_learner_args = NULL,
  cluster_ids = NULL,
  strata_ids = NULL,
  weights = NULL,
  rowids = NULL,
  loss_metric = NULL,
  use_complete_cases = FALSE,
  inner_n_folds = NULL,
  inner_cv_schema = NULL
)

Arguments

data: Data to use in training a super_learner.
learners: A list of predictor/closure-returning-functions. See Details.
formulas: Either a single regression formula or a vector of regression formulas.
y_variable: Typically y_variable can be inferred automatically from the formulas, but if needed, the y_variable can be specified explicitly.
n_folds: The number of cross-validation folds to use in constructing the super_learner. Must be a scalar integer of value >= 2.
determine_super_learner_weights: A function/method to determine the weights for each of the candidate learners. The default is to use determine_super_learner_weights_nnls.
ensemble_or_discrete: Defaults to 'ensemble', but can be set to 'discrete'. Discrete super_learner() chooses only one of the candidate learners to have weight 1 in the resulting prediction algorithm, while ensemble super_learner() combines predictions from 1 or more candidate learners, with respective weights adding up to 1.
cv_schema: A function that takes data, n_folds and returns a list containing training_data and validation_data, each of which are lists of n_folds data frames.
outcome_type: One of 'continuous', 'binary', 'multiclass', or 'density'. outcome_type is used to infer the correct determine_super_learner_weights function if it is not explicitly passed.
extra_learner_args: A list of equal length to the learners with additional arguments to pass to each of the specified learners.
cluster_ids: (default: null) If specified, clusters will either be entirely assigned to training or validation (not both) in each cross-validation split.
strata_ids: (default: null) If specified, strata are balanced across training and validation splits so that strata appear in both the training and validation splits.
weights: If specified, (per observation) weights are used to indicate that risk minimization across models (i.e., the meta-learning step) should be targeted to higher weight observations.
rowids: (default: null) If specified, rowids are stored for use in out-of-fold prediction ($oof_predict()) and can also be used to validate/observe the behavior of the cross-validation folds construction.
loss_metric: A loss metric function, like the mean-squared-error or negative-log-loss to be used in evaluating the learners on held-out data and minimized through convex optimization. A loss metric should take two (vector) arguments: predictions, and true outcomes, and produce a single statistic summarizing the performance of each learner. Defaults to nadir's internal mean-squared-error function.
use_complete_cases: (default: FALSE) If the data passed have any NA or NaN missing data, restrict the data to data[complete.cases(data),].
inner_n_folds: Number of folds used by the inner super_learner() on each outer training split to estimate ensemble weights. Defaults to n_folds, matching the historical behavior in which one fold count governed both.
inner_cv_schema: Optional cv_schema for the inner super_learner() calls; defaults to cv_schema when one is supplied, and otherwise to super_learner()'s own defaults.

Value

A list containing $trained_learners and $cv_loss which respectively include 1) the trained super learner models on each fold of the data, their holdout predictions and, 2) the cross-validated estimate of the risk (expected loss) on held-out data.

Details

The idea is that cv_super_learner splits the data into training/validation splits, trains super_learner on each training split, and then evaluates their predictions on the held-out validation data, calculating a root-mean-squared-error on those held-out data.

This function prints a message if the loss_function argument is not set explicitly, letting the user know that the mean-squared-error will be used by default. Pass in a loss function explicitly to super_learner() if you'd like to suppress this message, or use a similar approach for the appropriate loss function depending on context.

Examples


cv_sl <- cv_super_learner(
  data = mtcars,
  formula = mpg ~ cyl + hp,
  learners = list(lnr_mean, lnr_lm)
)
#> The loss_metric is being inferred based on the outcome_type=continuous -> using CV-MSE
print(cv_sl)
#> Cross-validated Super Learner (nadir_cv_sl)
#>   outcome:      mpg (continuous)
#>   outer folds:  5   inner CV folds: 5
#>   cross-validated loss on held-out data: 12.091
#> Methods: $cv_trained_learners, $cv_loss, $crossfit