Skip to content

Commit 90436a4

Browse files
author
ercbk
committed
readme edits
1 parent 2d898b4 commit 90436a4

File tree

2 files changed

+8
-10
lines changed

2 files changed

+8
-10
lines changed

README.Rmd

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,12 @@ output: github_document
66

77
Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to handle both hyperparameter tuning and algorithm comparison. Using standard methods such as k-fold cross-validation in such situations results in significant increases in optimization bias. Nested cross-validation has been shown to produce low bias in out-of-sample error estimates even using datasets with only a few hundred rows.
88

9-
The primary issue with this technique is that it is computationally very expensive with potentially tens of 1000s of models being trained during the process. This experiment seeks to answer two questions:
9+
The primary issue with this technique is that it is computationally very expensive with potentially tens of 1000s of models being trained during the process. While researching this technique, I found two methods of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html).
10+
This experiment seeks to answer two questions:
1011

1112
1. What's the fastest implementation of each method?
1213
2. How many *repeats*, given the size of the training set, should we expect to need to obtain a reasonably accurate out-of-sample error estimate?
1314

14-
While researching this technique, I found two *methods* of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html).
15-
1615
With regards to the question of speed, I'll will be testing implementations of both methods from various packages which include {tune}, {mlr3}, {h2o}, and {sklearn}.
1716

1817
Duration experiment details:

README.md

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -11,19 +11,18 @@ using datasets with only a few hundred rows.
1111

1212
The primary issue with this technique is that it is computationally very
1313
expensive with potentially tens of 1000s of models being trained during
14-
the process. This experiment seeks to answer two questions:
14+
the process. While researching this technique, I found two methods of
15+
performing nested cross-validation — one authored by [Sabastian
16+
Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb)
17+
and the other by [Max Kuhn and Kjell
18+
Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html).
19+
This experiment seeks to answer two questions:
1520

1621
1. What’s the fastest implementation of each method?
1722
2. How many *repeats*, given the size of the training set, should we
1823
expect to need to obtain a reasonably accurate out-of-sample error
1924
estimate?
2025

21-
While researching this technique, I found two *methods* of performing
22-
nested cross-validation — one authored by [Sabastian
23-
Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb)
24-
and the other by [Max Kuhn and Kjell
25-
Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html).
26-
2726
With regards to the question of speed, I’ll will be testing
2827
implementations of both methods from various packages which include
2928
{tune}, {mlr3}, {h2o}, and {sklearn}.

0 commit comments

Comments
 (0)