You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.Rmd
+3-5Lines changed: 3 additions & 5 deletions
Original file line number
Diff line number
Diff line change
@@ -10,10 +10,8 @@ Nested cross-validation has become a recommended technique for situations in whi
10
10
The primary issue with this technique is that it is computationally very expensive with potentially tens of 1000s of models being trained during the process. While researching this technique, I found two slightly different methods of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html).
11
11
I'll be examining two aspects of nested cross-validation:
12
12
13
-
1. Duration: Which packages and functions give us the fastest implementation of each method?
14
-
2. Performance: First, develop a testing framework. Then, using a generated dataset, find how many repeats, given the number of samples, should we expect to need in order to obtain a reasonably accurate out-of-sample error estimate.
15
-
16
-
With regards to the question of speed, I'll will be testing implementations of both methods from various packages which include {tune}, {mlr3}, {h2o}, and {sklearn}.
13
+
1. Duration: Find out which packages and combinations of model functions give us the fastest implementation of each method.
14
+
2. Performance: First, develop a testing framework. Then, using a generated dataset, calculate how many repeats, given the sample size, should we expect to need in order to obtain a reasonably accurate out-of-sample error estimate.
17
15
18
16
19
17
## Duration Experiment
@@ -81,7 +79,7 @@ kj <- runs %>%
81
79
geom_bar(aes(color = after_scale(prismatic::clr_darken(rep("#BD9865",5), 0.3))), stat = "identity", width = 0.50, fill = "#BD9865") +
82
80
coord_flip() +
83
81
scale_x_reordered() +
84
-
geom_text(hjust = 1.3, size = 3.5, color = "white") +
82
+
geom_text(hjust = 1.3, size = 3.5, color = "black") +
0 commit comments