Merge pull request #32 from DoubleML/sk-optuna

Add tuning simulations

Author: SvenKlaassen

.github/workflows/apo_sim.yml

@@ -52,7 +52,7 @@ jobs:
       uses: astral-sh/setup-uv@v5
       with:
         version: "0.7.8"
-  
+
     - name: Set up Python
       uses: actions/setup-python@v5
       with:

.github/workflows/pliv_sim.yml

@@ -62,7 +62,7 @@ jobs:
         cd monte-cover
         uv venv
         uv sync
-  
+
     - name: Install DoubleML from correct branch
       run: |
         source monte-cover/.venv/bin/activate

doc/did/did_cs.qmd

@@ -22,9 +22,9 @@ from utils.style_tables import generate_and_show_styled_table
 init_notebook_mode(all_interactive=True)
 ```
 
-## ATTE Coverage
+## Coverage
 
-The simulations are based on the  the [make_did_SZ2020](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_did_SZ2020.html)-DGP with $1000$ observations. Learners are only set to boosting, due to time constraints (and the nonlinearity of some of the DGPs).
+The simulations are based on the  the [make_did_SZ2020](https://docs.doubleml.org/stable/api/generated/doubleml.did.datasets.make_did_SZ2020.html)-DGP with $1000$ observations. Learners are only set to boosting, due to time constraints (and the nonlinearity of some of the DGPs).
 
 ::: {.callout-note title="Metadata"  collapse="true"}
 

doc/did/did_cs_multi.qmd

@@ -22,9 +22,9 @@ from utils.style_tables import generate_and_show_styled_table
 init_notebook_mode(all_interactive=True)
 ```
 
-## ATTE Coverage
+## Coverage
 
-The simulations are based on the [make_did_cs_CS2021](https://docs.doubleml.org/dev/api/generated/doubleml.did.datasets.make_did_cs_CS2021.html)-DGP with $2000$ observations. Learners are both set to either boosting or a linear (logistic) model. Due to time constraints we only consider the following DGPs:
+The simulations are based on the [make_did_cs_CS2021](https://docs.doubleml.org/stable/api/generated/doubleml.did.datasets.make_did_cs_CS2021.html)-DGP with $1000$ observations. Learners are both set to either boosting or a linear (logistic) model. Due to time constraints we only consider the following DGPs:
 
  - Type 1: Linear outcome model and treatment assignment
  - Type 4: Nonlinear outcome model and treatment assignment
@@ -52,7 +52,7 @@ df = pd.read_csv("../../results/did/did_cs_multi_detailed.csv", index_col=None)
 assert df["repetition"].nunique() == 1
 n_rep = df["repetition"].unique()[0]
 
-display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
+display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_d0_t0", "Loss g_d0_t1", "Loss g_d1_t0", "Loss g_d1_t1", "Loss m"]
 ```
 
 ### Observational Score
@@ -112,7 +112,7 @@ generate_and_show_styled_table(
 
 ## Aggregated Effects
 
-These simulations test different types of aggregation, as described in [DiD User Guide](https://docs.doubleml.org/dev/guide/models.html#difference-in-differences-models-did).
+These simulations test different types of aggregation, as described in [DiD User Guide](https://docs.doubleml.org/stable/guide/models.html#difference-in-differences-models-did).
 
 The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidence intervals).
 
@@ -127,7 +127,7 @@ df_group = pd.read_csv("../../results/did/did_cs_multi_group.csv", index_col=Non
 assert df_group["repetition"].nunique() == 1
 n_rep_group = df_group["repetition"].unique()[0]
 
-display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
+display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_d0_t0", "Loss g_d0_t1", "Loss g_d1_t0", "Loss g_d1_t1", "Loss m"]
 ```
 
 #### Observational Score
@@ -195,7 +195,7 @@ df_time = pd.read_csv("../../results/did/did_cs_multi_time.csv", index_col=None)
 assert df_time["repetition"].nunique() == 1
 n_rep_time = df_time["repetition"].unique()[0]
 
-display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
+display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_d0_t0", "Loss g_d0_t1", "Loss g_d1_t0", "Loss g_d1_t1", "Loss m"]
 ```
 
 #### Observational Score
@@ -263,7 +263,7 @@ df_es = pd.read_csv("../../results/did/did_cs_multi_eventstudy.csv", index_col=N
 assert df_es["repetition"].nunique() == 1
 n_rep_es = df_es["repetition"].unique()[0]
 
-display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
+display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_d0_t0", "Loss g_d0_t1", "Loss g_d1_t0", "Loss g_d1_t1", "Loss m"]
 ```
 
 #### Observational Score

doc/did/did_pa.qmd

@@ -22,9 +22,9 @@ from utils.style_tables import generate_and_show_styled_table
 init_notebook_mode(all_interactive=True)
 ```
 
-## ATTE Coverage
+## Coverage
 
-The simulations are based on the  the [make_did_SZ2020](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_did_SZ2020.html)-DGP with $1000$ observations. Learners are only set to boosting, due to time constraints (and the nonlinearity of some of the DGPs).
+The simulations are based on the  the [make_did_SZ2020](https://docs.doubleml.org/stable/api/generated/doubleml.did.datasets.make_did_SZ2020.html)-DGP with $1000$ observations. Learners are only set to boosting, due to time constraints (and the nonlinearity of some of the DGPs).
 
 ::: {.callout-note title="Metadata"  collapse="true"}
 

doc/did/did_pa_multi.qmd

@@ -22,9 +22,9 @@ from utils.style_tables import generate_and_show_styled_table
 init_notebook_mode(all_interactive=True)
 ```
 
-## ATTE Coverage
+## Coverage
 
-The simulations are based on the  the [make_did_CS2021](https://docs.doubleml.org/dev/api/generated/doubleml.did.datasets.make_did_CS2021.html)-DGP with $2000$ observations. Learners are both set to either boosting or a linear (logistic) model. Due to time constraints we only consider the following DGPs:
+The simulations are based on the  the [make_did_CS2021](https://docs.doubleml.org/stable/api/generated/doubleml.did.datasets.make_did_CS2021.html)-DGP with $1000$ observations. Learners are both set to either boosting or a linear (logistic) model. Due to time constraints we only consider the following DGPs:
 
  - Type 1: Linear outcome model and treatment assignment
  - Type 4: Nonlinear outcome model and treatment assignment
@@ -52,7 +52,7 @@ df = pd.read_csv("../../results/did/did_pa_multi_detailed.csv", index_col=None)
 assert df["repetition"].nunique() == 1
 n_rep = df["repetition"].unique()[0]
 
-display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
+display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
 ```
 
 ### Observational Score
@@ -112,7 +112,7 @@ generate_and_show_styled_table(
 
 ## Aggregated Effects
 
-These simulations test different types of aggregation, as described in [DiD User Guide](https://docs.doubleml.org/dev/guide/models.html#difference-in-differences-models-did).
+These simulations test different types of aggregation, as described in [DiD User Guide](https://docs.doubleml.org/stable/guide/models.html#difference-in-differences-models-did).
 
 The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidende intervals).
 
@@ -127,7 +127,7 @@ df_group = pd.read_csv("../../results/did/did_pa_multi_group.csv", index_col=Non
 assert df_group["repetition"].nunique() == 1
 n_rep_group = df_group["repetition"].unique()[0]
 
-display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
+display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
 ```
 
 #### Observational Score
@@ -195,7 +195,7 @@ df_time = pd.read_csv("../../results/did/did_pa_multi_time.csv", index_col=None)
 assert df_time["repetition"].nunique() == 1
 n_rep_time = df_time["repetition"].unique()[0]
 
-display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
+display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
 ```
 
 #### Observational Score
@@ -263,7 +263,7 @@ df_es = pd.read_csv("../../results/did/did_pa_multi_eventstudy.csv", index_col=N
 assert df_es["repetition"].nunique() == 1
 n_rep_es = df_es["repetition"].unique()[0]
 
-display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
+display_columns = ["Learner g", "Learner m", "DGP", "In-sample-norm.", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
 ```
 
 #### Observational Score
@@ -320,3 +320,194 @@ generate_and_show_styled_table(
     coverage_highlight_cols=["Coverage", "Uniform Coverage"]
 )
 ```
+
+
+## Tuning
+
+The simulations are based on the  the [make_did_CS2021](https://docs.doubleml.org/stable/api/generated/doubleml.did.datasets.make_did_CS2021.html)-DGP with $1000$ observations. Due to time constraints we only consider one learner, use in-sample normalization and the following DGPs:
+
+ - Type 1: Linear outcome model and treatment assignment
+ - Type 4: Nonlinear outcome model and treatment assignment
+
+The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidende intervals). This is only an example as the untuned version just relies on the default configuration.
+
+::: {.callout-note title="Metadata"  collapse="true"}
+
+```{python}
+#| echo: false
+metadata_file = '../../results/did/did_pa_multi_tune_metadata.csv'
+metadata_df = pd.read_csv(metadata_file)
+print(metadata_df.T.to_string(header=False))
+```
+
+:::
+
+```{python}
+#| echo: false
+
+# set up data
+df = pd.read_csv("../../results/did/did_pa_multi_tune_detailed.csv", index_col=None)
+
+assert df["repetition"].nunique() == 1
+n_rep = df["repetition"].unique()[0]
+
+display_columns = ["Learner g", "Learner m", "DGP", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
+```
+
+### Observational Score
+
+```{python}
+#| echo: false
+generate_and_show_styled_table(
+    main_df=df,
+    filters={"level": 0.95, "Score": "observational"},
+    display_cols=display_columns,
+    n_rep=n_rep,
+    level_col="level",
+    coverage_highlight_cols=["Coverage", "Uniform Coverage"]
+)
+```
+
+```{python}
+#| echo: false
+generate_and_show_styled_table(
+    main_df=df,
+    filters={"level": 0.9, "Score": "observational"},
+    display_cols=display_columns,
+    n_rep=n_rep,
+    level_col="level",
+    coverage_highlight_cols=["Coverage", "Uniform Coverage"]
+)
+```
+
+## Tuning Aggregated Effects
+
+These simulations test different types of aggregation, as described in [DiD User Guide](https://docs.doubleml.org/stable/guide/models.html#difference-in-differences-models-did).
+
+As before, we only consider one learner, use in-sample normalization and the following DGPs:
+
+ - Type 1: Linear outcome model and treatment assignment
+ - Type 4: Nonlinear outcome model and treatment assignment
+
+The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidende intervals). This is only an example as the untuned version just relies on the default configuration.
+
+### Group Effects
+
+```{python}
+#| echo: false
+
+# set up data
+df_group_tune = pd.read_csv("../../results/did/did_pa_multi_tune_group.csv", index_col=None)
+
+assert df_group_tune["repetition"].nunique() == 1
+n_rep_group_tune = df_group_tune["repetition"].unique()[0]
+
+display_columns_tune = ["Learner g", "Learner m", "DGP", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
+```
+
+#### Observational Score
+
+```{python}
+#| echo: false
+generate_and_show_styled_table(
+    main_df=df_group_tune,
+    filters={"level": 0.95, "Score": "observational"},
+    display_cols=display_columns_tune,
+    n_rep=n_rep_group_tune,
+    level_col="level",
+    coverage_highlight_cols=["Coverage", "Uniform Coverage"]
+)
+```
+
+```{python}
+#| echo: false
+generate_and_show_styled_table(
+    main_df=df_group_tune,
+    filters={"level": 0.9, "Score": "observational"},
+    display_cols=display_columns_tune,
+    n_rep=n_rep_group_tune,
+    level_col="level",
+    coverage_highlight_cols=["Coverage", "Uniform Coverage"]
+)
+```
+
+
+### Time Effects
+
+```{python}
+#| echo: false
+
+# set up data
+df_time_tune = pd.read_csv("../../results/did/did_pa_multi_tune_time.csv", index_col=None)
+
+assert df_time_tune["repetition"].nunique() == 1
+n_rep_time_tune = df_time_tune["repetition"].unique()[0]
+
+display_columns_tune = ["Learner g", "Learner m", "DGP", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
+```
+
+#### Observational Score
+
+```{python}
+#| echo: false
+generate_and_show_styled_table(
+    main_df=df_time_tune,
+    filters={"level": 0.95, "Score": "observational"},
+    display_cols=display_columns_tune,
+    n_rep=n_rep_time_tune,
+    level_col="level",
+    coverage_highlight_cols=["Coverage", "Uniform Coverage"]
+)
+```
+
+```{python}
+#| echo: false
+generate_and_show_styled_table(
+    main_df=df_time_tune,
+    filters={"level": 0.9, "Score": "observational"},
+    display_cols=display_columns_tune,
+    n_rep=n_rep_time_tune,
+    level_col="level",
+    coverage_highlight_cols=["Coverage", "Uniform Coverage"]
+)
+```
+
+### Event Study Aggregation
+
+```{python}
+#| echo: false
+
+# set up data
+df_es_tune = pd.read_csv("../../results/did/did_pa_multi_tune_eventstudy.csv", index_col=None)
+
+assert df_es_tune["repetition"].nunique() == 1
+n_rep_es_tune = df_es_tune["repetition"].unique()[0]
+
+display_columns_tune = ["Learner g", "Learner m", "DGP", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
+```
+
+#### Observational Score
+
+```{python}
+#| echo: false
+generate_and_show_styled_table(
+    main_df=df_es_tune,
+    filters={"level": 0.95, "Score": "observational"},
+    display_cols=display_columns_tune,
+    n_rep=n_rep_es_tune,
+    level_col="level",
+    coverage_highlight_cols=["Coverage", "Uniform Coverage"]
+)
+```
+
+```{python}
+#| echo: false
+generate_and_show_styled_table(
+    main_df=df_es_tune,
+    filters={"level": 0.9, "Score": "observational"},
+    display_cols=display_columns_tune,
+    n_rep=n_rep_es_tune,
+    level_col="level",
+    coverage_highlight_cols=["Coverage", "Uniform Coverage"]
+)
+```