Merge remote-tracking branch 'rael/wecc' into plots

Author: staadecker

README.md

@@ -3,21 +3,43 @@
 Welcome! This repository contains the SWITCH electricity planning model adapted for the
 REAM research lab.
 
-For **an overview** of what SWITCH is and how it works, read [`docs/Overview.md`](./docs/Overiew.md).
+## Available documentation
 
-To discover **how to install, run or debug** the different components of Switch, read [`docs/Usage.md`](./docs/Usage.md)
+In `docs/`:
 
-To **see examples** of smaller SWITCH models, see the `examples/` folder.
+- [`Overview.md`](./docs/Overiew.md): An overview of what SWITCH is and how it works, read
 
-To discover **how to contribute to the model**, read [`docs/Contribute.md`](/docs/Contribute.md)
+- [`Usage.md`](./docs/Usage.md): How to install, run or debug the different components of Switch
 
-To **learn about our database** (e.g. how to connect to it and modify it), read [`docs/Database.md`](/docs/Database.md).
+- [`Developing Modules.md`](./docs/Developing%20Modules.md): How to create SWITCH modules from scratch
 
-To **generate documentation**, run `pydoc -w switch_model` after having installed
+- [`Contribute.md`](/docs/Contribute.md): How to contribute code to the project.
+
+- [`Graphs`](/docs/Graphs.md): How to create new graphs to analyze your results.
+
+- [`Database.md`](/docs/Database.md): All about the REAM database (e.g. how to connect to it and modify it)
+
+- [`Numerical Solvers.md`](/docs/Numerical%20Solvers.md): Information about numerical solvers, specifically Gurobi.
+
+- [`Numerical Issues.md`](/docs/Numerical%20Issues.md): Information about detecting and resolving numerical issues.
+
+- [`Pandas.md`](/docs/Pandas.md): Crash course on the Pandas data manipulation library.
+
+Finally, you can generate documentation for the SWITCH modules by running `pydoc -w switch_model` after having installed
 SWITCH. This will build HTML documentation files from python doc strings which
 will include descriptions of each module, their intentions, model
 components they define, and what input files they expect.
 
-To learn about **numerical solvers** read [`docs/Numerical Solvers.md`](/docs/Numerical%20Solvers.md)
+## Key folders
+
+- [`/database`](/database) Folder containing SQL scripts and files that keep track of updates to our PostgreSQL database.
+
+- [`/examples`](/examples) Folder with numerous examples of SWITCH projects often used for testing.
+
+- [`/switch_model`](/switch_model) Folder containing all the source code for the SWITCH modules.
+
+- [`/switch_model/wecc`](/switch_model/wecc) Folder containing modules specific to the REAM team.
+
+- [`/switch_model/wecc/get_inputs`](/switch_model/wecc/get_inputs) Scripts that fetch the input data from the PostgreSQL database.
 
-To learn about **numerical issues** read [`docs/Numerical Issues.md`](/docs/Numerical%20Issues.md)
+- [`/tests`](/tests) Folder containing tests for SWITCH. Can be run via `python run_tests.py`.

REAM Model Changelog.md

@@ -16,4 +16,5 @@ Changes are listed from oldest (first line) to newest (last line of table).
 | #56 | June 2021 | Convert 2020 predetermined build years to 2019 in `get_inputs.py` to avoid conflicts with 2020 period. |
 | #57 | June 2021 | Specify predetermined storage energy capacity in inputs (previously left unspecified). |
 | #68 | June 2021 | Change financial params to 2018 dollars & 5% interest rate. Start using terrain multipliers (which now include the economic multiplier). |
-| #72 | July 2021 | Drop build and O&M costs of existing transmission lines. |
+| #72 | July 2021 | Drop build and O&M costs of existing transmission lines. |
+| #89 | August 2021 | Change hydro module average flow constraint to a monthly constraint rather than per timeseries and change it to a <= rather than ==. |

database/2021-07-29_create_low_hydro_scenario.py

@@ -0,0 +1,128 @@
+"""
+####################
+Create a low hydro scenario
+
+Date applied: 2021-07-29
+Description:
+This script adds a scenario to the database for low hydro power.
+The worst year for hydro is 2015. As such we use those values for every year unless a plant is missing
+in 2015 in which case we use the lowest value in the other years for that plant.
+#################
+"""
+import time
+
+from switch_model.utilities import query_yes_no, format_seconds
+from switch_model.wecc.utilities import connect
+import pandas as pd
+
+raw_data_scenario = 21
+all_plants_scenario = 23
+worst_year = 2015
+
+new_start_year = 2020
+new_end_year = 2050
+new_scenario_id = 24
+new_scenario_name = "Lowest year (2015) repeated. Using EIA and AMPL Canada and Mex data."
+new_scenario_description = "Lowest year (2015) repeated from 2020 to 2050, based on data from id 21 (EIA + AMPL Canada & Mex)."
+
+
+def main():
+    db_conn = connect()
+    db_cursor = db_conn.cursor()
+
+    # 1. Get all the hydro plants
+    db_cursor.execute(
+        f"""
+        SELECT DISTINCT generation_plant_id FROM hydro_historical_monthly_capacity_factors
+        WHERE hydro_simple_scenario_id={all_plants_scenario};
+    """)
+    hydro_plants = pd.DataFrame(db_cursor.fetchall(), columns=["generation_plant_id"])["generation_plant_id"]
+
+    # 2. Get all the hydro flow data for the worst year
+    db_cursor.execute(
+        f"""
+        SELECT generation_plant_id, month, hydro_min_flow_mw, hydro_avg_flow_mw FROM hydro_historical_monthly_capacity_factors
+        WHERE hydro_simple_scenario_id={raw_data_scenario} and year={worst_year};
+    """)
+    worst_year_data = pd.DataFrame(db_cursor.fetchall(),
+                                   columns=["generation_plant_id", "month", "hydro_min_flow_mw", "hydro_avg_flow_mw"])
+
+    # 3. Identify plants where data is missing
+    missing_hydro_plants = hydro_plants[~hydro_plants.isin(worst_year_data["generation_plant_id"])].values
+
+    # 4. For each missing plant get the data for all the years
+    db_cursor.execute(
+        f"""
+        SELECT generation_plant_id, year, month, hydro_min_flow_mw, hydro_avg_flow_mw FROM hydro_historical_monthly_capacity_factors
+        WHERE hydro_simple_scenario_id={raw_data_scenario} and generation_plant_id in ({",".join(missing_hydro_plants.astype(str))});
+    """)
+    missing_plants_data = pd.DataFrame(db_cursor.fetchall(),
+                                       columns=["generation_plant_id", "year", "month", "hydro_min_flow_mw",
+                                                "hydro_avg_flow_mw"])
+
+    # 5. Pick the year with the least flow
+    # Aggregate by year
+    missing_data_by_year = missing_plants_data.groupby(["generation_plant_id", "year"], as_index=False)[
+        "hydro_avg_flow_mw"].mean()
+    # Select years where the flow is at its lowest
+    year_to_use = \
+    missing_data_by_year.loc[missing_data_by_year.groupby("generation_plant_id")["hydro_avg_flow_mw"].idxmin()][
+        ["generation_plant_id", "year"]]
+    # Essentially filter missing_plants_data to only include keys from the right table, aka plants and years that are lowest
+    missing_plants_data = missing_plants_data.merge(
+        year_to_use,
+        on=["generation_plant_id", "year"],
+        how="right"
+    ).drop("year", axis=1)
+
+    # 6. Add the missing data to our worst year data and verify we have data for all the plants
+    worst_year_data = pd.concat([worst_year_data, missing_plants_data])
+    assert all(hydro_plants.isin(worst_year_data["generation_plant_id"]))
+
+    # 7. Cross join the series with all the years from 2020 to 2050
+    years = pd.Series(range(new_start_year, new_end_year + 1), name="year")
+    worst_year_data = worst_year_data.merge(
+        years,
+        how="cross"
+    )
+    worst_year_data["hydro_simple_scenario_id"] = new_scenario_id
+
+    # 8. Complete some data checks
+    assert len(worst_year_data) == 12 * (new_end_year - new_start_year + 1) * len(hydro_plants)
+
+    # 9. Add data to database
+    print(f"hydro_simple_scenario: {new_scenario_id}")
+    print(f"name: {new_scenario_name}")
+    print(f"description: {new_scenario_description}")
+    print(f"Num hydro plants: {worst_year_data.generation_plant_id.nunique()}")
+    print(f"From year: {new_start_year}")
+    print(f"To year: {new_end_year}")
+    print(f"Example data:\n{worst_year_data.head()}")
+
+    if not query_yes_no("\nAre you sure you want to add this data to the database?", default="no"):
+        raise SystemExit
+
+    db_cursor.execute(
+        "INSERT INTO hydro_simple_scenario(hydro_simple_scenario_id, name, description) "
+        f"VALUES ('{new_scenario_id}','{new_scenario_name}','{new_scenario_description}')"
+    )
+
+    n = len(worst_year_data)
+    start_time = time.time()
+    for i, r in enumerate(worst_year_data.itertuples(index=False)):
+        if i !=0 and i % 1000 == 0:
+            print(
+                f"{i}/{n} inserts completed. Estimated time remaining {format_seconds((n - i) * (time.time() - start_time) / i)}")
+        db_cursor.execute(
+            f"INSERT INTO hydro_historical_monthly_capacity_factors(hydro_simple_scenario_id, generation_plant_id, year, month, hydro_min_flow_mw, hydro_avg_flow_mw) "
+            f"VALUES ({r.hydro_simple_scenario_id},{r.generation_plant_id},{r.year},{r.month},{r.hydro_min_flow_mw},{r.hydro_avg_flow_mw})"
+        )
+
+    db_conn.commit()
+    db_cursor.close()
+    db_conn.close()
+    print("Done.")
+
+
+if __name__ == "__main__":
+    main()

database/2021-07-30_create_no_hydro_scenario.py

@@ -0,0 +1,75 @@
+"""
+####################
+Create a no hydro scenario
+
+Date applied: 2021-07-30
+Description:
+This script adds a scenario to the database that effectively removes all hydro generation from the model.
+#################
+"""
+import time
+
+from switch_model.utilities import query_yes_no, format_seconds
+from switch_model.wecc.utilities import connect
+import pandas as pd
+
+all_plants_scenario = 23
+
+new_scenario_id = 25
+new_scenario_name = "No Hydro"
+new_scenario_description = "All average flows are zero effectively removing all hydro generation from the model." \
+                           " Represents as an extreme edge case of no hydro generation."
+
+
+def main():
+    db_conn = connect()
+    db_cursor = db_conn.cursor()
+
+    # 1. Get all the hydro plants
+    db_cursor.execute(
+        f"""
+        SELECT DISTINCT generation_plant_id, year, month, hydro_min_flow_mw, hydro_avg_flow_mw FROM hydro_historical_monthly_capacity_factors
+        WHERE hydro_simple_scenario_id={all_plants_scenario};
+    """)
+    df = pd.DataFrame(db_cursor.fetchall(),
+                      columns=["generation_plant_id", "year", "month", "hydro_min_flow_mw", "hydro_avg_flow_mw"])
+
+    # 2. Set all the flows to zero and set the scenario id
+    df["hydro_min_flow_mw"] = 0
+    df["hydro_avg_flow_mw"] = 0
+    df["hydro_simple_scenario_id"] = new_scenario_id
+
+    # 3. Add data to database
+    print(f"hydro_simple_scenario: {new_scenario_id}")
+    print(f"name: {new_scenario_name}")
+    print(f"description: {new_scenario_description}")
+    print(f"Num hydro plants: {df.generation_plant_id.nunique()}")
+    print(f"Example data:\n{df.head()}")
+
+    if not query_yes_no("\nAre you sure you want to add this data to the database?", default="no"):
+        raise SystemExit
+
+    db_cursor.execute(
+        "INSERT INTO hydro_simple_scenario(hydro_simple_scenario_id, name, description) "
+        f"VALUES ('{new_scenario_id}','{new_scenario_name}','{new_scenario_description}')"
+    )
+
+    n = len(df)
+    start_time = time.time()
+    for i, r in enumerate(df.itertuples(index=False)):
+        if i != 0 and i % 1000 == 0:
+            print(
+                f"{i}/{n} inserts completed. Estimated time remaining {format_seconds((n - i) * (time.time() - start_time) / i)}")
+        db_cursor.execute(
+            f"INSERT INTO hydro_historical_monthly_capacity_factors(hydro_simple_scenario_id, generation_plant_id, year, month, hydro_min_flow_mw, hydro_avg_flow_mw) "
+            f"VALUES ({r.hydro_simple_scenario_id},{r.generation_plant_id},{r.year},{r.month},{r.hydro_min_flow_mw},{r.hydro_avg_flow_mw})"
+        )
+
+    db_conn.commit()
+    db_cursor.close()
+    db_conn.close()
+    print("Done.")
+
+
+if __name__ == "__main__":
+    main()

database/2021-08-02_create_half_hydro_scenario.py

@@ -0,0 +1,74 @@
+"""
+####################
+Create a hydro scenario with only half the flow of the baseline
+
+Date applied: 2021-08-02
+Description:
+This script adds a scenario to the database that makes all hydro flows half of the expected hydro flow.
+#################
+"""
+import time
+
+from switch_model.utilities import query_yes_no, format_seconds
+from switch_model.wecc.utilities import connect
+import pandas as pd
+
+all_plants_scenario = 23
+
+new_scenario_id = 26
+new_scenario_name = "50% of scenario 23"
+new_scenario_description = "All average flows are halved to represent a scenario where hydro generation is low."
+
+
+def main():
+    db_conn = connect()
+    db_cursor = db_conn.cursor()
+
+    # 1. Get all the hydro plants
+    db_cursor.execute(
+        f"""
+        SELECT DISTINCT generation_plant_id, year, month, hydro_min_flow_mw, hydro_avg_flow_mw FROM hydro_historical_monthly_capacity_factors
+        WHERE hydro_simple_scenario_id={all_plants_scenario};
+    """)
+    df = pd.DataFrame(db_cursor.fetchall(),
+                      columns=["generation_plant_id", "year", "month", "hydro_min_flow_mw", "hydro_avg_flow_mw"])
+
+    # 2. Set all the flows to zero and set the scenario id
+    df["hydro_avg_flow_mw"] /= 2
+    df["hydro_min_flow_mw"] = df[["hydro_min_flow_mw", "hydro_avg_flow_mw"]].min(axis=1)
+    df["hydro_simple_scenario_id"] = new_scenario_id
+
+    # 3. Add data to database
+    print(f"hydro_simple_scenario: {new_scenario_id}")
+    print(f"name: {new_scenario_name}")
+    print(f"description: {new_scenario_description}")
+    print(f"Num hydro plants: {df.generation_plant_id.nunique()}")
+    print(f"Example data:\n{df.head()}")
+
+    if not query_yes_no("\nAre you sure you want to add this data to the database?", default="no"):
+        raise SystemExit
+
+    db_cursor.execute(
+        "INSERT INTO hydro_simple_scenario(hydro_simple_scenario_id, name, description) "
+        f"VALUES ('{new_scenario_id}','{new_scenario_name}','{new_scenario_description}')"
+    )
+
+    n = len(df)
+    start_time = time.time()
+    for i, r in enumerate(df.itertuples(index=False)):
+        if i != 0 and i % 1000 == 0:
+            print(
+                f"{i}/{n} inserts completed. Estimated time remaining {format_seconds((n - i) * (time.time() - start_time) / i)}")
+        db_cursor.execute(
+            f"INSERT INTO hydro_historical_monthly_capacity_factors(hydro_simple_scenario_id, generation_plant_id, year, month, hydro_min_flow_mw, hydro_avg_flow_mw) "
+            f"VALUES ({r.hydro_simple_scenario_id},{r.generation_plant_id},{r.year},{r.month},{r.hydro_min_flow_mw},{r.hydro_avg_flow_mw})"
+        )
+
+    db_conn.commit()
+    db_cursor.close()
+    db_conn.close()
+    print("Done.")
+
+
+if __name__ == "__main__":
+    main()

docs/Contribute.md

@@ -1,6 +1,7 @@
 # Contributing Code
 
-This document describes the best practices for contributing code.
+You've made changes to the codebase and now you want to share them
+with the rest of the team! Here are the best practices for the process.
 
 ## The process
 
@@ -39,43 +40,6 @@ supposed to alter the results of the examples, you'll need
 to follow the instructions that appear on screen to suppress the errors
 produced by `python run_tests.py`.
 
-## Contributing graphs
+## Important notes
 
-Read [`docs/Graphs.md`](./Graphs.md) to see learn to add graphs.
-
-## Modifying the database
-
-Read [`docs/Database.md`](./Database.md) to learn about the database.
-
-## Outputting results
-
-Once the model is solved, the `post_solve()` function in each module is called.
-Within the `post_solve()` function you may 
-
-- Call `write_table()` to create
-a .csv file with data from the solution (see existing modules for examples).
-  
-- Call `add_info()` (from `utilities/result_info.py`) to add a line 
-of information to the `outputs/info.txt` file. `add_info()` can also be added to `graph()`.
-
-### Example
-
-```python
-from switch_model.utilities.results_info import add_info
-from switch_model.reporting import write_table
-import os
-...
-def post_solve(instance, outdir):
-    ...
-    # This will add the a line to info.txt in the outputs folder
-    add_info("Some important value", instance.important_value)
-    ...
-    # This will create my_table.csv
-    write_table(
-        instance, 
-        instance.TIMEPOINTS, # Set that the values function will iterate over
-        output_file=os.path.join(outdir, "my_table.csv"),
-        headings=("timepoint", "some value"),
-        values=lambda m, t: (t, m.some_value[t])
-    )
-```
+- If your change is modifying the database, make sure you've read [`Database.md`](./Database.md).