added ncv image

Author: ercbk

README.Rmd

@@ -5,6 +5,8 @@ output: github_document
 # Nested Cross-Validation: Comparing Methods and Implementations  
 ### (In-progress)
 
+![](images/ncv.png)
+
 Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Examples of such situations include: proof of concept, start-ups, medical studies, time series, etc. Using standard methods such as k-fold cross-validation in these cases may result in  significant increases in optimization bias. Nested cross-validation has been shown to produce low bias, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better judgement of generalization performance.  
 
 The primary issue with this technique is that it is computationally very expensive with potentially tens of 1000s of models being trained during the process. While researching this technique, I found two slightly different methods of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html).  
@@ -109,7 +111,7 @@ durations
 Experiment details:  
 
   * The fastest implementation of each method will be used in running a nested cross-validation with different sizes of data ranging from 100 to 5000 observations and different numbers of repeats of the outer-loop cv strategy.  
-      * The {mlr3} implementation was the fastest for Raschka's method, but the Ranger-Kuhn-Johnson implementation was close. To simplify, I'll be using Ranger-Kuhn-Johnson for both methods.  
+      * The {mlr3} implementation was the fastest for Raschka's method, but the Ranger-Kuhn-Johnson implementation was close. To simplify, I'll be using [Ranger-Kuhn-Johnson](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/kuhn-johnson/nested-cv-ranger-kj.R) for both methods.  
   * The chosen algorithm and hyperparameters will be used to predict on a 100K row simulated dataset.  
   * The percent error between the the average mean absolute error (MAE) across the outer-loop folds and the MAE of the predictions on this 100K dataset will be calculated for each combination of repeat, data size, and method.  
   * To make this experiment manageable in terms of runtimes, I'm using AWS instances: a r5.2xlarge for the Elastic Net and a r5.24xlarge for Random Forest.  

README.md

@@ -3,6 +3,8 @@
 
 ### (In-progress)
 
+![](images/ncv.png)
+
 Nested cross-validation has become a recommended technique for
 situations in which the size of our dataset is insufficient to
 simultaneously handle hyperparameter tuning and algorithm comparison.
@@ -88,7 +90,9 @@ Experiment details:
     outer-loop cv strategy.
       - The {mlr3} implementation was the fastest for Raschka’s method,
         but the Ranger-Kuhn-Johnson implementation was close. To
-        simplify, I’ll be using Ranger-Kuhn-Johnson for both methods.  
+        simplify, I’ll be using
+        [Ranger-Kuhn-Johnson](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/kuhn-johnson/nested-cv-ranger-kj.R)
+        for both methods.  
   - The chosen algorithm and hyperparameters will be used to predict on
     a 100K row simulated dataset.  
   - The percent error between the the average mean absolute error (MAE)