Compare Learners

Compare learners using the specified loss_metric

Usage

compare_learners(sl_output, y_variable = NULL, loss_metric)

Arguments

sl_output

Output from running super_learner() with verbose_output = TRUE.

y_variable

A character vector indicating the outcome variable. y_variable will be automatically inferred if it is missing and can be inferred from the sl_output.

loss_metric

A loss metric, like the mean-squared-error or negative-log-loss to be used in comparing the learners. A loss metric should take two (vector) arguments: predictions, and true outcomes, and produce a single statistic summarizing the performance of each learner.

Value

A data.frame with the loss-metric on the held-out data for each learner.

Examples

sl_model <- super_learner(
  data = mtcars,
  learners = list(lm = lnr_lm, rf = lnr_rf, mean = lnr_mean),
  formula = mpg ~ .
)

compare_learners(sl_model)
#> Inferring the loss metric for learner comparison based on the outcome type: 
#> outcome_type=continuous -> using mean squared error
#> # A tibble: 1 × 3
#>      lm    rf  mean
#>   <dbl> <dbl> <dbl>
#> 1  17.6  6.47  38.3

sl_model <- super_learner(
  data = mtcars,
  learners = list(lnr_logistic, lnr_rf_binary, mean = lnr_mean),
  formula = am ~ mpg,
  outcome_type = "binary"
)
compare_learners(sl_model)
#> Inferring the loss metric for learner comparison based on the outcome type: 
#> outcome_type=binary -> using negative log loss
#> # A tibble: 1 × 3
#>   logistic rf_binary  mean
#>      <dbl>     <dbl> <dbl>
#> 1     20.4       Inf  23.7