Merge pull request #7 from NSAPH-Data-Processing/TinasheMTapera/issue6

Tinashe m tapera/issue6

Author: TinasheMTapera

README.md

@@ -1 +1,3 @@
 This file will be overwritten by `index.ipynb`
+
+In the meantime, see `notes/index.ipynb` for the notes..

_proc/00_core.ipynb

@@ -24,7 +24,39 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This file will become your README and also the index of your documentation."
+    "Here we are developing functions and code for the Madagascar ERA5 dataset project. The goal is for exposure data to be made available at the daily resolution when possible. Finer resolutions shouldn’t ever be needed for our purposes, and it should then be relatively easy to aggregate at coarser resolutions, such as weekly or monthly.\n",
+    "\n",
+    "Variables should generally be made available from 2010 onward, as that’s where our clinic data starts.\n",
+    "\n",
+    "All data are ideally made available at the “healthshed” geographical level. Healthsheds are defined as geographical areas where people who live all go to the same clinic. There are a total of ~2700 public clinics in Madagascar, hence ~2700 healthsheds, with each healthshed containing ~10000 people on average.\n",
+    "\n",
+    "Preliminary list of environmental variables\n",
+    "\n",
+    "- [ ] 2-m air temperature from ERA5: daily min, max, mean\n",
+    " \n",
+    "- [ ] 2-m air dew point temperature from ERA5: daily min, max, mean\n",
+    "\n",
+    "- [ ] Precipitation: daily total (ERA5)\n",
+    "\n",
+    "- [ ] Sea surface temperature: daily average and maximum in the nearest neighbor for each healthshed.\n",
+    "\n",
+    "- [ ] Precipitation: daily total (CHIRPS)\n",
+    "\n",
+    "- [ ] Chlorophyll-A (Giacomo)\n",
+    "\n",
+    "- [ ] Wealth index: Available from Giacomo \n",
+    "\n",
+    "- [ ] NDVI\n",
+    "\n",
+    "- [ ] Tropical storm\n",
+    "\n",
+    "- [ ] Flooding\n",
+    "\n",
+    "- [ ] Deforestation\n",
+    "\n",
+    "- [ ] Linking/segmenting healthsheds into climate zones and other \n",
+    "\n",
+    "- [ ] Relative humidity: daily average (lower priority)"
    ]
   },
   {
@@ -56,12 +88,14 @@
     "# make sure era5_sandbox package is installed in development mode\n",
     "$ pip install -e .\n",
     "\n",
-    "# make changes under nbs/ directory\n",
-    "# ...\n",
+    "# To make changes, go to the \"notes\" directory and edit the notebooks as necessary.\n",
+    "# Each notebook refers to a module in the era5_sandbox package. Cells are exported to the module\n",
+    "# when the notebook is saved and you run the following command:\n",
     "\n",
-    "# compile to have changes apply to era5_sandbox\n",
-    "$ nbdev_prepare\n",
-    "```"
+    "$ nbdev_export\n",
+    "```\n",
+    "\n",
+    "For e.g., to change functionality of the [`testAPI()`](https://TinasheMTapera.github.io/era5_sandbox/core.html#testapi) function in the testAPI Hydra rule, you would edit the [`testAPI`](https://TinasheMTapera.github.io/era5_sandbox/core.html#testapi) notebook in the `notes` directory `notes/testAPI.ipynb`, and then save that notebook and run `nbdev_export` to update the `core` module in the package."
    ]
   },
   {
@@ -85,27 +119,18 @@
     "Install latest from the GitHub [repository][repo]:\n",
     "\n",
     "```sh\n",
-    "$ pip install git+https://github.com/TinasheMTapera/era5_sandbox.git\n",
-    "```\n",
-    "\n",
-    "or from [conda][conda]\n",
-    "\n",
-    "```sh\n",
-    "$ conda install -c TinasheMTapera era5_sandbox\n",
+    "$ pip install git+https://github.com/NSAPH-Data-Processing/era5_sandbox\n",
     "```\n",
     "\n",
-    "or from [pypi][pypi]\n",
-    "\n",
+    "or clone and install in development mode:\n",
     "\n",
     "```sh\n",
-    "$ pip install era5_sandbox\n",
+    "$ git clone https://github.com/NSAPH-Data-Processing/era5_sandbox\n",
+    "$ pip install -e .\n",
     "```\n",
     "\n",
     "\n",
-    "[repo]: https://github.com/TinasheMTapera/era5_sandbox\n",
-    "[docs]: https://TinasheMTapera.github.io/era5_sandbox/\n",
-    "[pypi]: https://pypi.org/project/era5_sandbox/\n",
-    "[conda]: https://anaconda.org/TinasheMTapera/era5_sandbox"
+    "[repo]: https://github.com/NSAPH-Data-Processing/era5_sandbox"
    ]
   },
   {
@@ -119,12 +144,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Documentation can be found hosted on this GitHub [repository][repo]'s [pages][docs]. Additionally you can find package manager specific guidelines on [conda][conda] and [pypi][pypi] respectively.\n",
-    "\n",
-    "[repo]: https://github.com/TinasheMTapera/era5_sandbox\n",
-    "[docs]: https://TinasheMTapera.github.io/era5_sandbox/\n",
-    "[pypi]: https://pypi.org/project/era5_sandbox/\n",
-    "[conda]: https://anaconda.org/TinasheMTapera/era5_sandbox"
+    "🚧Documentation is in development 🚧"
    ]
   },
   {
@@ -138,7 +158,43 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Fill me in please! Don't forget code examples:"
+    "The pipeline currently downloads ERA5 temperature and dew point temperature data for a given date range and geographical bounding box. You can learn each of these steps by following the notebooks in `notes` in numerical order.\n",
+    "\n",
+    "To run the pipeline, the config at `config/config.yaml` should be updated with the desired date range and geographical bounding box. The pipeline can then be run with the following command:\n",
+    "\n",
+    "```sh\n",
+    "sbatch snakemake.sbatch\n",
+    "```\n",
+    "\n",
+    "You can investigate the downloaded raw data with python, eg.:\n",
+    "\n",
+    "```python\n",
+    "import xarray as xr\n",
+    "import matplotlib.pyplot as plt\n",
+    "import cartopy.crs as ccrs\n",
+    "import cartopy.feature as cfeature\n",
+    "\n",
+    "### the path to any of the downloaded files\n",
+    "file_path = \"/n/dominici_lab/lab/data_processing/csph-era5_sandbox/data/input/2010_01.nc\"\n",
+    "data = xr.open_dataset(file_path)\n",
+    "\n",
+    "\n",
+    "temperature = data[\"t2m\"]\n",
+    "\n",
+    "\n",
+    "\n",
+    "# Select a specific time step\n",
+    "temperature_at_time = temperature.isel(valid_time=0)\n",
+    "\n",
+    "# Plot the data on a map\n",
+    "plt.figure(figsize=(12, 8))\n",
+    "ax = plt.axes(projection=ccrs.PlateCarree())\n",
+    "temperature_at_time.plot(ax=ax, cmap=\"coolwarm\", transform=ccrs.PlateCarree(), cbar_kwargs={\"label\": \"Temperature (K)\"})\n",
+    "ax.coastlines()\n",
+    "ax.add_feature(cfeature.BORDERS, linestyle=\":\")\n",
+    "ax.set_title(\"Temperature at Time Step 0\")\n",
+    "plt.show()\n",
+    "```"
    ]
   },
   {
@@ -175,28 +231,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "python3",
    "language": "python",
    "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.8"
-  },
-  "widgets": {
-   "application/vnd.jupyter.widget-state+json": {
-    "state": {},
-    "version_major": 2,
-    "version_minor": 0
-   }
   }
  },
  "nbformat": 4,

_proc/index.ipynb

@@ -0,0 +1,6 @@
+website:
+  sidebar:
+    contents:
+      - index.ipynb
+      - 00_core.ipynb
+      - 01_download_raw_data.ipynb

_proc/sidebar.yml

@@ -0,0 +1,9 @@
+daily: 
+  function: "numpy.mean"
+  string: mean
+
+monthly: 
+  function: "numpy.mean"
+  string: mean
+
+variable: ['t2m', 'd2m']

conf/aggregation/aggregation.yaml

@@ -1,20 +1,27 @@
 defaults:
   - _self_
   - datapaths: datapaths
+  - aggregation: aggregation
 
 development_mode: false
 
 CDS_API_KEY: 
   path: "$HOME/.cdsapirc"
 
-gadm_file: "https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_MDG.gpkg"
+GOOGLE_DRIVE_AUTH_JSON:
+  path: "sandbox/harvard-csph-driveauth-f5f9a2682ecf.json"
+  healthsheds_id: "healthsheds2022.zip"
+
+mdg_shapefile: "https://data.humdata.org/dataset/26fa506b-0727-4d9d-a590-d2abee21ee22/resource/ed94d52e-349e-41be-80cb-62dc0435bd34/download/mdg_adm_bngrc_ocha_20181031_shp.zip"
+
+dataset: "reanalysis-era5-single-levels"
 
 query: 
   product_type: reanalysis
   # check precipitation
   # variable: ["2m_dewpoint_temperature", "2m_temperature", "skin_temperature", "total_precipitation"]
   variable: ["2m_dewpoint_temperature", "2m_temperature"]
-  year: [2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024]
+  year: [2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024]
   month: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
   day: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
   time: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]