Correlation Threshold Based Screening
Arguments
- data
A dataframe intended to be used with
super_learner()- formula
The formula specifying the regression to be done
- threshold
The correlation coefficient cutoff, below which variables are screened out from the dataset and regression formula.
- cor...
An optional list of extra arguments to pass to
cor. Usemethod = 'spearman'for the Spearman rank based correlation coefficient.
Value
A list of $data with columns screened out,
$formula with variables screened out, and $failed_to_correlate_names
the names of variables that failed to correlate with the outcome at least at the threshold
level.
Details
If a variable used has little correlation with the outcome being predicted, we might want to screen that variable out from the predictors.
In large datasets, this is quite important, as having a huge number of
columns could be computationally intractable or frustratingly time-consuming
to run super_learner() with.
Examples
screener_cor(
data = mtcars,
formula = mpg ~ .,
threshold = 0.5
)
#> $data
#> mpg cyl disp hp drat wt vs am carb
#> Mazda RX4 21.0 6 160.0 110 3.90 2.620 0 1 4
#> Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 0 1 4
#> Datsun 710 22.8 4 108.0 93 3.85 2.320 1 1 1
#> Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 1 0 1
#> Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 0 0 2
#> Valiant 18.1 6 225.0 105 2.76 3.460 1 0 1
#> Duster 360 14.3 8 360.0 245 3.21 3.570 0 0 4
#> Merc 240D 24.4 4 146.7 62 3.69 3.190 1 0 2
#> Merc 230 22.8 4 140.8 95 3.92 3.150 1 0 2
#> Merc 280 19.2 6 167.6 123 3.92 3.440 1 0 4
#> Merc 280C 17.8 6 167.6 123 3.92 3.440 1 0 4
#> Merc 450SE 16.4 8 275.8 180 3.07 4.070 0 0 3
#> Merc 450SL 17.3 8 275.8 180 3.07 3.730 0 0 3
#> Merc 450SLC 15.2 8 275.8 180 3.07 3.780 0 0 3
#> Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 0 0 4
#> Lincoln Continental 10.4 8 460.0 215 3.00 5.424 0 0 4
#> Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 0 0 4
#> Fiat 128 32.4 4 78.7 66 4.08 2.200 1 1 1
#> Honda Civic 30.4 4 75.7 52 4.93 1.615 1 1 2
#> Toyota Corolla 33.9 4 71.1 65 4.22 1.835 1 1 1
#> Toyota Corona 21.5 4 120.1 97 3.70 2.465 1 0 1
#> Dodge Challenger 15.5 8 318.0 150 2.76 3.520 0 0 2
#> AMC Javelin 15.2 8 304.0 150 3.15 3.435 0 0 2
#> Camaro Z28 13.3 8 350.0 245 3.73 3.840 0 0 4
#> Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 0 0 2
#> Fiat X1-9 27.3 4 79.0 66 4.08 1.935 1 1 1
#> Porsche 914-2 26.0 4 120.3 91 4.43 2.140 0 1 2
#> Lotus Europa 30.4 4 95.1 113 3.77 1.513 1 1 2
#> Ford Pantera L 15.8 8 351.0 264 4.22 3.170 0 1 4
#> Ferrari Dino 19.7 6 145.0 175 3.62 2.770 0 1 6
#> Maserati Bora 15.0 8 301.0 335 3.54 3.570 0 1 8
#> Volvo 142E 21.4 4 121.0 109 4.11 2.780 1 1 2
#>
#> $formula
#> mpg ~ cyl + disp + hp + drat + wt + vs + am + carb
#> <environment: 0x329456c38>
#>
#> $failed_to_correlate_names
#> [1] "qsec" "gear"
#>
# We're also showing how to specify that you want the Spearman rank-based
# correlation coefficient, to get away from the assumption of linearity.
screener_cor(
data = mtcars,
formula = mpg ~ .,
threshold = 0.5,
cor... = list(method = "spearman")
)
#> $data
#> mpg cyl disp hp drat wt vs am gear carb
#> Mazda RX4 21.0 6 160.0 110 3.90 2.620 0 1 4 4
#> Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 0 1 4 4
#> Datsun 710 22.8 4 108.0 93 3.85 2.320 1 1 4 1
#> Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 1 0 3 1
#> Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 0 0 3 2
#> Valiant 18.1 6 225.0 105 2.76 3.460 1 0 3 1
#> Duster 360 14.3 8 360.0 245 3.21 3.570 0 0 3 4
#> Merc 240D 24.4 4 146.7 62 3.69 3.190 1 0 4 2
#> Merc 230 22.8 4 140.8 95 3.92 3.150 1 0 4 2
#> Merc 280 19.2 6 167.6 123 3.92 3.440 1 0 4 4
#> Merc 280C 17.8 6 167.6 123 3.92 3.440 1 0 4 4
#> Merc 450SE 16.4 8 275.8 180 3.07 4.070 0 0 3 3
#> Merc 450SL 17.3 8 275.8 180 3.07 3.730 0 0 3 3
#> Merc 450SLC 15.2 8 275.8 180 3.07 3.780 0 0 3 3
#> Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 0 0 3 4
#> Lincoln Continental 10.4 8 460.0 215 3.00 5.424 0 0 3 4
#> Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 0 0 3 4
#> Fiat 128 32.4 4 78.7 66 4.08 2.200 1 1 4 1
#> Honda Civic 30.4 4 75.7 52 4.93 1.615 1 1 4 2
#> Toyota Corolla 33.9 4 71.1 65 4.22 1.835 1 1 4 1
#> Toyota Corona 21.5 4 120.1 97 3.70 2.465 1 0 3 1
#> Dodge Challenger 15.5 8 318.0 150 2.76 3.520 0 0 3 2
#> AMC Javelin 15.2 8 304.0 150 3.15 3.435 0 0 3 2
#> Camaro Z28 13.3 8 350.0 245 3.73 3.840 0 0 3 4
#> Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 0 0 3 2
#> Fiat X1-9 27.3 4 79.0 66 4.08 1.935 1 1 4 1
#> Porsche 914-2 26.0 4 120.3 91 4.43 2.140 0 1 5 2
#> Lotus Europa 30.4 4 95.1 113 3.77 1.513 1 1 5 2
#> Ford Pantera L 15.8 8 351.0 264 4.22 3.170 0 1 5 4
#> Ferrari Dino 19.7 6 145.0 175 3.62 2.770 0 1 5 6
#> Maserati Bora 15.0 8 301.0 335 3.54 3.570 0 1 5 8
#> Volvo 142E 21.4 4 121.0 109 4.11 2.780 1 1 4 2
#>
#> $formula
#> mpg ~ cyl + disp + hp + drat + wt + vs + am + gear + carb
#> <environment: 0x313665028>
#>
#> $failed_to_correlate_names
#> [1] "qsec"
#>