Graph updates

.github/workflows/bookdown.yaml

@@ -42,6 +42,12 @@ jobs:
       - name: Build site
         run: Rscript -e 'bookdown::render_book("index.Rmd", quiet = TRUE)'
 
+      - name: Upload results
+        uses: actions/upload-artifact@main
+        with:
+          name: svm_bo
+          path: RData/svm_bo.RData
+
       - name: Deploy to Netlify
         if: contains(env.isExtPR, 'false')
         id: netlify-deploy

01-software-modeling.Rmd

@@ -80,7 +80,7 @@ plm_plot <-
     axis.ticks = element_blank()
   ) + 
   labs(x = "", y = "") + 
-  scale_fill_manual(values = c("#377EB8", "#E41A1C")) + 
+  scale_fill_manual(values = c("white", "midnightblue")) + 
   coord_equal() + 
   ggtitle("(a) Evaluating the quality of two microarray chips using a model.") + 
   theme(plot.title = element_text(hjust = 0.5))

13-grid-search.Rmd

@@ -728,6 +728,7 @@ remaining <-
 ```
 
 
+
 As an example, in the multilayer perceptron tuning process with a regular grid explored in this chapter, what would the results look like after only the first three folds? Using techniques similar to those shown in Chapter \@ref(compare), we can fit a model where the outcome is the resampled area under the ROC curve and the predictor is an indicator for the parameter combination. The model takes the resample-to-resample effect into account and produces point and interval estimates for each parameter setting. The results of the model are one-sided 95% confidence intervals that measure the loss of the ROC value relative to the currently best performing parameters, as shown in Figure \@ref(fig:racing-process).
 
 ```{r racing-process}
@@ -737,10 +738,9 @@ As an example, in the multilayer perceptron tuning process with a regular grid e
 #| fig.height = 5, 
 #| out.width = "80%",
 #| fig.cap = "The racing process for 20 tuning parameters and 10 resamples.",
-#| fig.alt = "An illustration of the racing process for 20 tuning parameters and 10 resamples. The analysis is conducted at the first, third, and last resample. As the number of resamples increases, the confidence intervals show some model configurations that do not have confidence intervals that overlap with zero. These are excluded from subsequent resamples."
+#| fig.alt = "The racing process for 20 tuning parameters and 10 resamples. The analysis is conducted at the first, third, and last resample. As the number of resamples increases, the confidence intervals show some model configurations that do not have confidence intervals that overlap with zero. These are excluded from subsequent resamples."
 
 full_att <- attributes(mlp_sfd_race)
-
 race_details <- NULL
 for(iter in 1:10) {
 
@@ -758,7 +758,6 @@ for(iter in 1:10) {
       race_details,
       finetune:::test_parameters_gls(tmp) %>% mutate(iter = iter))
 }
-
 race_details <-
   race_details %>%
   mutate(
@@ -769,73 +768,35 @@ race_details <-
     decision = ifelse(pass & estimate == 0, "best", decision)
   )  %>%
   mutate(
+    .config = factor(.config),
+    .config = format(as.integer(.config)),
+    .config = paste("config", .config),
     .config = factor(.config),
     .config = reorder(.config, estimate),
     decision = factor(decision, levels = c("best", "retain", "discard"))
   ) 
 race_cols <- c(best = "blue", retain = "black", discard = "grey")
-
 iter_three <- race_details %>% dplyr::filter(iter == 3) 
-
-iter_three %>% 
+race_details %>% 
+  filter(iter %in% c(1, 3, 10)) %>% 
+  mutate(iter = paste("resamples:", format(iter))) %>% 
   ggplot(aes(x = -estimate, y = .config)) + 
-  geom_vline(xintercept = 0, lty = 2, color = "green") +
-  geom_point(size = 2, aes(color = decision)) +
-  geom_errorbarh(aes(xmin = -estimate, xmax = -upper, color = decision), height = .3, show.legend = FALSE) + 
+  geom_vline(xintercept = 0, lty = 2, col = "green") +
+  geom_point(size = 2, aes(col = decision, pch = decision)) + 
+  geom_errorbarh(aes(xmin = -estimate, xmax = -upper, col = decision), height = .3, show.legend = FALSE) + 
   labs(x = "Loss of ROC AUC", y = NULL) + 
-  scale_colour_manual(values = race_cols)
+  scale_colour_manual(values = race_cols) + 
+  facet_wrap(~iter) +
+  theme(legend.position = "top")
 ```
 
-Any parameter set whose confidence interval includes zero would lack evidence that its performance is not statistically different from the best results. We retain `r sum(iter_three$upper < 0)` settings; these are resampled more. The remaining `r sum(iter_three$upper >= 0)` submodels are no longer considered. 
+Figure \@ref(fig:racing-process) shows the results at several iterations in the process. The points shown in the panel with the first iteration show single ROC AUC values. As iterations progress, the points are averages of the resampled ROC statistics.
 
-```{r grid-mlp-racing-anim, include = FALSE, dev = 'png'}
-race_ci_plots <- function(x, iters = max(x$iter)) {
-
-  x_rng <- extendrange(c(-x$estimate, -x$upper))
-
-  for (i in 1:iters) {
-    if (i < 3) {
-      ttl <- paste0("Iteration ", i, ": burn-in")
-    } else {
-      ttl <- paste0("Iteration ", i, ": testing")
-    }
-    p <-
-      x %>% 
-      dplyr::filter(iter == i) %>% 
-      ggplot(aes(x = -estimate, y = .config, color = decision)) +
-      geom_vline(xintercept = 0, color = "green", lty = 2) +
-      geom_point(size = 2) +
-      labs(title = ttl, y = "", x = "Loss of ROC AUC") +
-      scale_color_manual(values = c(best = "blue", retain = "black", discard = "grey"), 
-                         drop = FALSE) +
-      scale_y_discrete(drop = FALSE) +
-      xlim(x_rng) + 
-      theme_bw() +
-      theme(legend.position = "top")
-
-    if (i >= 3) {
-      p <- p  + geom_errorbar(aes(xmin = -estimate, xmax = -upper), width = .3)
-    }
-
-    print(p)
-  }
-  invisible(NULL)
-}
-av_capture_graphics(
-  race_ci_plots(race_details),
-  output = "race_results.mp4",
-  width = 720,
-  height = 720,
-  res = 120,
-  framerate = 1/3
-)
-```
+On the third iteration, the leading model configuration has changed and the algorithm computes one-sided confidence intervals. Any parameter set whose confidence interval includes zero would lack evidence that its performance is not statistically different from the best results. We retain `r sum(iter_three$upper < 0)` settings; these are resampled more. The remaining `r sum(iter_three$upper >= 0)` submodels are no longer considered. 
 
-<video width="720" height="720" controls>
-  <source src="race_results.mp4" type="video/mp4">
-</video>
+The process continues to resample configurations that remain and the statistical analysis repeats with the current results. More submodels may be removed from consideration. Prior to the final resample, almost all submodels are eliminated and, after the last iteration, only one remains. 
 
-The process continues for each resample; after the next set of performance metrics, a new model is fit to these statistics, and more  submodels are potentially discarded. See @kuhn2014futility for more details on the computational aspects of this approach. 
+See @kuhn2014futility for more details on the computational aspects of this approach. 
 
 :::rmdwarning
 Racing methods can be more efficient than basic grid search as long as the interim analysis is fast and some parameter settings have poor performance. It also is most helpful when the model does _not_ have the ability to exploit submodel predictions.