Correlation Threshold Based Screening
Arguments
- data
A dataframe intended to be used with
super_learner()- formula
The formula specifying the regression to be done
- keep_n_terms
Set to an integer value >=1, this indicates that the top n terms in the model frame with greatest absolute correlation with the outcome will be kept.
- cor...
An optional list of extra arguments to pass to
cor. Usemethod = 'spearman'for the Spearman rank based correlation coefficient.
Value
A list of $data with columns screened out,
$formula with variables screened out, and $failed_to_correlate_names
the names of variables that failed to correlate with the outcome at least at the threshold
level.
Details
If a variable used has little correlation with the outcome being predicted, we might want to screen that variable out from the predictors.
In large datasets, this is quite important, as having a huge number of
columns could be computationally intractable or frustratingly time-consuming
to run super_learner() with.
Examples
screener_cor_top_n(
data = mtcars,
formula = mpg ~ .,
keep_n_terms = 5
)
#> $data
#> mpg cyl disp hp drat wt
#> Mazda RX4 21.0 6 160.0 110 3.90 2.620
#> Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875
#> Datsun 710 22.8 4 108.0 93 3.85 2.320
#> Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215
#> Hornet Sportabout 18.7 8 360.0 175 3.15 3.440
#> Valiant 18.1 6 225.0 105 2.76 3.460
#> Duster 360 14.3 8 360.0 245 3.21 3.570
#> Merc 240D 24.4 4 146.7 62 3.69 3.190
#> Merc 230 22.8 4 140.8 95 3.92 3.150
#> Merc 280 19.2 6 167.6 123 3.92 3.440
#> Merc 280C 17.8 6 167.6 123 3.92 3.440
#> Merc 450SE 16.4 8 275.8 180 3.07 4.070
#> Merc 450SL 17.3 8 275.8 180 3.07 3.730
#> Merc 450SLC 15.2 8 275.8 180 3.07 3.780
#> Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250
#> Lincoln Continental 10.4 8 460.0 215 3.00 5.424
#> Chrysler Imperial 14.7 8 440.0 230 3.23 5.345
#> Fiat 128 32.4 4 78.7 66 4.08 2.200
#> Honda Civic 30.4 4 75.7 52 4.93 1.615
#> Toyota Corolla 33.9 4 71.1 65 4.22 1.835
#> Toyota Corona 21.5 4 120.1 97 3.70 2.465
#> Dodge Challenger 15.5 8 318.0 150 2.76 3.520
#> AMC Javelin 15.2 8 304.0 150 3.15 3.435
#> Camaro Z28 13.3 8 350.0 245 3.73 3.840
#> Pontiac Firebird 19.2 8 400.0 175 3.08 3.845
#> Fiat X1-9 27.3 4 79.0 66 4.08 1.935
#> Porsche 914-2 26.0 4 120.3 91 4.43 2.140
#> Lotus Europa 30.4 4 95.1 113 3.77 1.513
#> Ford Pantera L 15.8 8 351.0 264 4.22 3.170
#> Ferrari Dino 19.7 6 145.0 175 3.62 2.770
#> Maserati Bora 15.0 8 301.0 335 3.54 3.570
#> Volvo 142E 21.4 4 121.0 109 4.11 2.780
#>
#> $formula
#> mpg ~ cyl + disp + hp + drat + wt
#> <environment: 0x17e12f278>
#>
#> $failed_to_correlate_names
#> [1] "qsec" "vs" "am" "gear" "carb"
#>
# We're also showing how to specify that you want the Spearman rank-based
# correlation coefficient, to get away from the assumption of linearity.
screener_cor_top_n(
data = mtcars,
formula = mpg ~ .,
keep_n_terms = 5,
cor... = list(method = "spearman")
)
#> $data
#> mpg cyl disp hp wt vs
#> Mazda RX4 21.0 6 160.0 110 2.620 0
#> Mazda RX4 Wag 21.0 6 160.0 110 2.875 0
#> Datsun 710 22.8 4 108.0 93 2.320 1
#> Hornet 4 Drive 21.4 6 258.0 110 3.215 1
#> Hornet Sportabout 18.7 8 360.0 175 3.440 0
#> Valiant 18.1 6 225.0 105 3.460 1
#> Duster 360 14.3 8 360.0 245 3.570 0
#> Merc 240D 24.4 4 146.7 62 3.190 1
#> Merc 230 22.8 4 140.8 95 3.150 1
#> Merc 280 19.2 6 167.6 123 3.440 1
#> Merc 280C 17.8 6 167.6 123 3.440 1
#> Merc 450SE 16.4 8 275.8 180 4.070 0
#> Merc 450SL 17.3 8 275.8 180 3.730 0
#> Merc 450SLC 15.2 8 275.8 180 3.780 0
#> Cadillac Fleetwood 10.4 8 472.0 205 5.250 0
#> Lincoln Continental 10.4 8 460.0 215 5.424 0
#> Chrysler Imperial 14.7 8 440.0 230 5.345 0
#> Fiat 128 32.4 4 78.7 66 2.200 1
#> Honda Civic 30.4 4 75.7 52 1.615 1
#> Toyota Corolla 33.9 4 71.1 65 1.835 1
#> Toyota Corona 21.5 4 120.1 97 2.465 1
#> Dodge Challenger 15.5 8 318.0 150 3.520 0
#> AMC Javelin 15.2 8 304.0 150 3.435 0
#> Camaro Z28 13.3 8 350.0 245 3.840 0
#> Pontiac Firebird 19.2 8 400.0 175 3.845 0
#> Fiat X1-9 27.3 4 79.0 66 1.935 1
#> Porsche 914-2 26.0 4 120.3 91 2.140 0
#> Lotus Europa 30.4 4 95.1 113 1.513 1
#> Ford Pantera L 15.8 8 351.0 264 3.170 0
#> Ferrari Dino 19.7 6 145.0 175 2.770 0
#> Maserati Bora 15.0 8 301.0 335 3.570 0
#> Volvo 142E 21.4 4 121.0 109 2.780 1
#>
#> $formula
#> mpg ~ cyl + disp + hp + wt + vs
#> <environment: 0x17d08a050>
#>
#> $failed_to_correlate_names
#> [1] "drat" "qsec" "am" "gear" "carb"
#>