Merge branch 'improve_load_aug' into improv_hydro

# Conflicts:
#	REAM Model Changelog.md
#	switch_model/solve.py

Author: staadecker

.gitignore

@@ -2,9 +2,6 @@
 # Edit at https://www.toptal.com/developers/gitignore?templates=macos,python,venv,data,visualstudiocode,pycharm,jupyternotebooks,dotenv,virtualenv
 
 ### Data ###
-*.csv
-# Don't ignore data in switch_model.csv
-!switch_model/**/*.csv
 *.dat
 *.efx
 *.gbr
@@ -18,8 +15,11 @@
 *.vcf
 *.xml
 *.pickle
+examples/**/temp/**
+examples/**/outputs/*.csv
+examples/**/outputs/*.txt
+examples/**/graphs/*.png
 examples/**/info.txt
-examples/**/*.png
 
 ### dotenv ###
 .env

database/2021-06-24_add_transmission_options.sql

@@ -0,0 +1,19 @@
+/*
+####################
+Add transmission options
+
+Date applied: 2021-06-23
+Description:
+Adds two rows to table transmission_base_capital_cost_scenario_id
+1. A scenario where transmission costs are zero.
+2. A scenario where transmission costs are infinity (building not allowed).
+#################
+*/
+
+INSERT INTO switch.transmission_base_capital_cost (transmission_base_capital_cost_scenario_id,
+                                                   trans_capital_cost_per_mw_km, description)
+VALUES (3, 'Infinity', 'For scenarios where building transmission is forbidden.');
+
+INSERT INTO switch.transmission_base_capital_cost (transmission_base_capital_cost_scenario_id,
+                                                   trans_capital_cost_per_mw_km, description)
+VALUES (4, 0, 'For scenarios where transmission is unlimited.');

database/2021-06-29_add_transmission_options_v2.sql

@@ -0,0 +1,13 @@
+/*
+####################
+Add transmission options
+
+Date applied: 2021-06-29
+Description:
+Adds an extra scenario to the database for a 10x increase in transmission costs.
+#################
+*/
+
+INSERT INTO switch.transmission_base_capital_cost (transmission_base_capital_cost_scenario_id,
+                                                   trans_capital_cost_per_mw_km, description)
+VALUES (5, 9600, '10x the costs of scenario #2. Approximates the no TX case.');

database/2021-07-07_add_wind_solar_ratio_column.sql

@@ -0,0 +1,12 @@
+/*
+####################
+Add wind_to_solar_ratio column
+
+Date applied: 2021-07-07
+Description:
+Adds a column called wind_to_solar_ratio to the database which is used by
+switch_model.policies.wind_to_solar_ratio
+#################
+*/
+
+ALTER TABLE switch.scenario ADD COLUMN wind_to_solar_ratio real;

docs/Performance.md

@@ -0,0 +1,74 @@
+# Performance
+
+Memory use and solve time are two important factors that we try to keep to a minimum in our models. There are multiple
+things one can do to improve performance.
+
+## Solving methods
+
+By far the biggest factor that impacts performance is the method used by Gurobi. The fastest method is barrier solve
+without crossover (use `--recommended-fast`)
+however this method often returns a suboptimal solution. The next fastest is barrier solve followed by crossover and
+simplex (use `--recommended`) which almost always works. In some cases barrier solve encounters numerical issues (
+see [`Numerical Issues.md`](./Numerical%20Issues.md))
+in which case the slower Simplex method must be used (`--solver-options-string method=1`).
+
+## Solver interface
+
+Solver interfaces are how Pyomo communicates with Gurobi (or any solver).
+
+There are two solver interfaces that you should know about: `gurobi` and `gurobi_direct`.
+
+- When using `gurobi`, Pyomo will write the entire model to a temporary text file and then start a *separate Gurobi
+  process* that will read the file, solve the model and write the results to another temporary text file. Once Gurobi
+  finishes writing the results Pyomo will read the results text file and load the results back into the Python program
+  before running post_solve (e.g. generate csv files, create graphs, etc). Note that these temporary text files are
+  stored in `/tmp` but if you use `--recommended-debug` Pyomo and Gurobi will instead use a `temp` folder in your model.
+
+- `gurobi_direct` uses Gurobi's Python library to create and solve the model directly in Python without the use of
+  intermediate text files.
+
+In theory `gurobi_direct` should be faster and more efficient however in practice we find that that's not the case. As
+such we recommend using `gurobi` and all our defaults do so. If someone has the time they could profile `gurobi_direct`
+to improve performance at which point we could make `gurobi_direct` the default (and enable `--save-warm-start` by default, see below).
+
+The `gurobi` interface has the added advantage of separating Gurobi and Pyomo into separate threads. This means that
+while Gurobi is solving and Pyomo is idle, the operating system can automatically move Pyomo's memory usage
+to [virtual memory](https://serverfault.com/questions/48486/what-is-swap-memory)
+which will free up more memory for Gurobi.
+
+## Warm starting
+
+Warm starting is the act of using a solution from a previous similar model to start the solver closer to your expected
+solution. Theoretically this can help performance however in practice there are several limitations. For this section, *
+previous solution* refers to the results from an already solved model that you are using to warm start the solver. *
+Current solution* refers to the solution you are trying to find while using the warm start feature.
+
+- To warm start a model use `switch solve --warm-start <path_to_previous_solution>`.
+
+- Warm starting only works if the previous solution does not break any constraints of the current solution. This usually
+  only happens if a) the model has the exact same set of variables b)
+  the previous solution was "harder" (e.g. it had more constraints to satisfy).
+
+- Warm starting always uses the slower Simplex method. This means unless you expect the previous solution and current
+  solution to be very similar, it may be faster to solve without warm start using the barrier method.
+
+- If your previous solution didn't use crossover (e.g. you used `--recommended-fast`) then warm starting will be even
+  slower since the solver will need to first run crossover before warm starting.
+
+- Our implementation of warm starting only works if your previous solution has an `outputs/warm_start.pickle`
+  file. This file is only generated when you use `--save-warm-start`.
+
+- `--save-warm-start` and `--warm-start` both use an extension of the `gurobi_direct` solver interface which is
+  generally slower than the `gurobi` solver interface (see section above).
+  
+## Tools for improving performance
+
+- [Memory profiler](https://pypi.org/project/memory-profiler/) for generating plots of the memory
+use over time. Use `mprof run --interval 60 --multiprocess switch solve ...` and once solving is done
+  run `mprof plot -o profile.png` to make the plot.
+  
+- [Fil Profiler](https://pypi.org/project/filprofiler/) is an amazing tool for seeing which parts of the code are
+using up memory during peak memory usage.
+  
+- Using `switch_model.utilities.StepTimer` to measure how long certain code blocks take to run. See examples
+throughout the code.

examples/3zone_toy/outputs/total_cost.txt

@@ -1 +1 @@
-128823582.05
+126714451.7

examples/3zone_toy_lp/outputs/total_cost.txt

@@ -1 +1 @@
-128823582.05
+126714451.7

examples/ca_policies/outputs/total_cost.txt

@@ -1 +1 @@
-136941837.16
+134832706.81

examples/carbon_cap/outputs/total_cost.txt

@@ -1 +1 @@
-136638207.50592738
+134529077.16

examples/ccs/outputs/total_cost.txt

@@ -1 +1 @@
-19218883.587348036
+19218883.59