From 5e2e339e0c3e573feab57c0f3d8157a340f1d002 Mon Sep 17 00:00:00 2001
From: Steven Kolawole <45284829+SteveKola@users.noreply.github.com>
Date: Wed, 22 Apr 2020 12:20:32 +0100
Subject: [PATCH 1/9] Create empty.gitkeep

---
 Kolawole_Steven/empty.gitkeep | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 Kolawole_Steven/empty.gitkeep

diff --git a/Kolawole_Steven/empty.gitkeep b/Kolawole_Steven/empty.gitkeep
new file mode 100644
index 0000000..8b13789
--- /dev/null
+++ b/Kolawole_Steven/empty.gitkeep
@@ -0,0 +1 @@
+

From 27653cda4265ff290e20faacca4e6d3a7354b9a2 Mon Sep 17 00:00:00 2001
From: Steven Kolawole <45284829+SteveKola@users.noreply.github.com>
Date: Wed, 22 Apr 2020 22:21:04 +0100
Subject: [PATCH 2/9] Add files via upload
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

A Customized Logistic Regression with Nesterov Accelerated Gradient with Early Stopping option:


Nesterov Accelerated Gradient is theorized to converge by at least, a 10 times faster rate than Stochastic Gradient Descent, and over 25 times faster rate than the naive batch gradient descent.

Nesterov Gradient combines the properties of Stochastic Gradient Descent -which supposedly have the properties that allows it to “jump” out of shallow local minima giving it a better chance of finding a true global minimum - with a 'smarter' momentum, that has a somewhat prescient notion of the global minimum, and knows to slow down before the hill slopes up again.

But there is a catch;
Converging too fast makes it easier for the model to overfit, causing the well-known bias-variance tradeoff.
A way to avoid that is to introduce Early Stopping, which works by simply waiting for a certain number of epochs with no improvement in validation loss, and then stopping the iterations.
---
 .../Logistic Regression from Scratch.ipynb    | 322 ++++++++++++++++++
 1 file changed, 322 insertions(+)
 create mode 100644 Kolawole_Steven/Logistic Regression from Scratch.ipynb

diff --git a/Kolawole_Steven/Logistic Regression from Scratch.ipynb b/Kolawole_Steven/Logistic Regression from Scratch.ipynb
new file mode 100644
index 0000000..6031d92
--- /dev/null
+++ b/Kolawole_Steven/Logistic Regression from Scratch.ipynb	
@@ -0,0 +1,322 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Logistic Regression from Scratch\n",
+    "## - Steven Kolawole                                "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Intro\n",
+    "\n",
+    "Logistic Regression is simply a Linear Regression with a Sigmoid function at its end.\n",
+    "\n",
+    "The Sigmoid function generates probability (i.e. outputs between 0 and 1) for all values of X."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "sns.set()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEMCAYAAADQ553CAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZhcZZn38W9V9Z49nc6eANlukkAIISzK4hZ0BLcRRMVhZBRxw1dHR8cRZkYdGX11RhnXcYy8ohgXcNyGgIqJgywhhCQEstxZCFk76SVrd3qtqvePczpUikq6Oqnqqq7+fa6rr65zznOqfv101V2nnnPqnEgymUREREpHtNABREQkt1TYRURKjAq7iEiJUWEXESkxKuwiIiVGhV1EpMSUFTqAnJqZ/QB49yma7HD3s83sBWCxu38hx4//18BtwFwgAawDvu7uP0tpkwRucvd7c/nYfch4M8HfnvH5bGaTgV3Aq9z9TxmWvxJYfpK7b3b3MblJempm9jIg6u6PhdN571cziwC3AO8B5gDdwFrgi+7+cL4eV/JLW+zF76PAhPDnknDem1PmXZyvBzazW4FvAt8GLgAuBR4AfmJmqW82E4D785UjCz8DJuXgfhbwYr/2/MzJwf1m6xFgZsp0XvvVzKLAr4A7gXsInl9XAU8DvzOzG/P12JJf2mIvcu5+GDgMYGZV4ewD7r6vHx7+/cD33P0HKfM2mJkRvOHcE2bsjywn5e5tQFsO7qqxwH9LJHWiH7LcBlwDXOTu61Lmf8rMhgJfN7Nfu3trnnNIjqmwl5ZJZvYbYBHBm8E33P1fexaa2VuAzwEGvAAsBr7q7omT3F8cuNzMRoRvMD3+DhiScr8nDBmY2SeBjwBjgN8BO4EL3P2V4bDHQ8BbgK8DU4DHgJuBTwM3Ae3A19z9SymP8R7g48B0YA9wl7t/M1x2MylDMWZ2FsGnjFcADcDxPjhdmYZ7MjxuEnhv+LdcDOwg6N//SlnnJuBTwIxw+Rfd/Z5wKC0G/D8zuznsq/R+7a0PPg18FfgMUAs8BXzY3Tee5M96P/DrtKLe43PADwj+FxmHhVLnhUOG1cBYgk8+nyb4H0xz9+0p66wHfunud5jZFOBrwGsJ3piXAx93970nyStZ0lBMaXkvsJRgPPw/gDvN7CoAM7sG+HE4fy5Bcfko8I+nuL+vEHw832tmvzGzvzOz+e7e6O4vZFrBzD4C/BNBcbmQ4A3kI2nNKoDPAzcCryYoBOuAFoKC+F3gi2Y2J7zPjxMMCd0FzAtzfcXMPpHh8csJ3jhqgMvDPvn0Kf7GXPu/BFkvBP4MfCd8o8HM3g7cTfCGej7wb8BiM3stwd8dBz4GvDX9TrPsg2nAu4DrCIrlWcA3MoUMP/3NAVZkWu7u+919pbvH+/C33wD8kmDI7lcEb1zvSHnM+eFj/tDMhgB/IijoLwdeR/C8WGZmFX14TMlAW+yl5T53/8/w9pfM7NPAQoKx288A33b3u8Pl28xsGPA9M/uXTFvt7n6fme0hKDavBd4IYGZrCLbU1mfI8AmCre2eLbu/NbMr0tpEgNvdfVV4f38kKGz/4O5JM/siwRvOXDPbSPAmdJe7Lw7X32Jm04C/N7Ovpt33IoJPJK9z953h/f8fgn0DvfFwKzTVPHd/Pot1e9zt7j8PH/eTBDsmLyEoch8Dfuzu/xG23RoOeUTdvTEY4eKwux9IvcNwB2c2fVAOfKBnC93MvkEwfp7JqPD3oT78bb3Z5+5fT8n9I+CdwBfDWe8CVrj7ZjO7heBT3809bx5m9k6gieCN6Sc5zDXoqLCXls1p04cIPh5DsAV5sZl9MGV5NFx+NpCxeLn748DjZhYDLiIo7h8BHjSzGe7e2dPWzGoJthKfSLubx4D5afO2ptxuBZ5392T4mG1hkasE6oBx4X2keoSg2I1Nm38e0NRT1EMZt0ozeB2QPq69K8t1exz/H7j7ofDv6NkCPR/4UWpjd78ri/vMtg+SwJaU5YdSHjtdc9h+dBaPn63059A9wB1mNhfYSLD13vNGcyHB33U47KMeNcDsHGYalFTYS0umj809O+Q6gS8TDMek250+Ixz//Afg8+6+L9yqWgmsNLM/E4ydzwNWpazWHf7OZoivK236ZOP87SeZHzvJ/SRJ2wlJ8Ldn4wV3f0lfnEKm109Hhnk9edKzZivbPki4e3dam/S+AMDdO81sNXBZpuVmNhP4FvC3mT6ZmVmmv/2EHdjuvtXMHifYan+YoJD3HCbbCawnw7ATuf0UMShpjH3wWA/MdPetPT8EW5B3kvnF30YwjJDpkLdDBAW0IXVmuIN1J8EYa6r06ay5+xGCN5704ZwrCLauD6bNXwuMCQtTj4Wn+/gpOoGYmdWkzJt5ssYnsTE9i5n90Mx6hi8ynkP7NPogW98H3mhm8zIs+yTBENIL4XQXMDxlebZ/+z0EQys3AL91956s64FzCL4n0PN8bCDY+Xt+X/4IeSltsQ8eXwAeMLPngF8Aswh2Ui5195dsZbp7k5l9mWAn5vBwnTaCF90XgHvShjt69KyziWBr/maCrcI/nWH2r5nZtvB+XkUwHPRP4Zh8atvlBMdh32tmHyYYivg6Z24FQeH9nJl9i+DN6uY+3seXgZ+b2UrgDwQ7jt8JvD5cfhSYY2Zj3b0hbd2+9EG2/ovg6KQ/mtlnCPpuOHArwU7nd6Yc6vgEcKuZPUbwSeFrZP50ku5nBDvs/4pgjL3Hj4HbCfrjHwg+lXyJ4M0k074b6QNtsQ8S7v4QwaGENwLPERT1HxIc8nayde4Il18NPErwguv5MsutJ1nt28C/hz/PELyB/Irsh0My5fguwc7ffwgzfJzgsLivZGgbJzg2eydBobqPoAidkXAH6geBtwGbgPcRbNX25T5+BXyYIP96gp2pN6V8w/NLwIcIhrnS1826D/qQJw68geAN5zZgDfB7gqNrXt2zEzj0QYJDaJ8k+NLUf5FhCC/DYxwGfk1QuB9Mmd9G8Lw6Biwj2H9QFj5u+pua9FFEV1CSXDKzvwCedfc9KfMeAva4+3sLl0xk8NBQjOTau4FzwmGQZoItwkUEh0uKSD9QYZdcu43gSzQPAsMIhi3e6e7LCppKZBDRUIyISInRzlMRkRJT6KGYSoKvkteT+cs1IiLyUjGC0zo/RYbDTgtd2C8mOFGSiIj03ZUEhyKfoNCFvR7g4MFWEom+j/XX1g6lubkl56FyoVizFWsuKN5sxZoLijebcvVdX7JFoxFGjRoCYQ1NV+jCHgdIJJKnVdh71i1WxZqtWHNB8WYr1lxQvNmUq+9OI1vGIWztPBURKTEq7CIiJUaFXUSkxGQ9xh6e4e9x4A3pl0ULL3m1mODMcI8QXMUl/bzQIiLSD7LaYjezSwkOqZl1kib3Are5+yyCc3u/LzfxRESkr7IdinkfwelGX3L18PBCvdXu3nP5sR8QnNpURGTQSyaTx38SaT/5ktVQjLvfAnCSk/lP5MRjKeuByWecTERKRld3gmPtXRzr6Ka9M057RzftXXE6OuN0dMXp7ErQ2R2nqzsR/MQTdIe/4/Ek3fEE3fEk8USSeCKYF08Gh0nHw8OlE8me35AMp4NiSngbopEI3fEEPefICo4uDJaFN4MiDCR7Zhy/zfHbyZ6LXaXO72OfRCLwwTefx8Jz0y/be+ZycRx7lBP/pggnv35lRrW1Q0/7wevqhp32uvlWrNmKNRcUb7ZizQWFy9bVnaD5cBuNh9poPNjGgSPtHDzazqEjHRxq6eBIaydHj3VytLWTzu7sS0JZLEpleZTyshhlZVHKy6KUxaKUxSLh7yhl5VGqYhFi0SjRaIRYNEI0GiEaSf3N8duRSIRIJCjshL8jEYL5ABGIcOK8SHjByEh4I5JyAcnj64XrvngzktImdX6KcEE0CpfMm0jtiOrji3L1v8xFYd9NcM6CHuPJMGRzKs3NLaf1pYG6umE0Nh7t83r9oVizFWsuKN5sxZoL+idbS1sXexpb2NPUyp6mVvYfOEbDwTaaj7STPppQWRFjxJAKRg+vYuSQCibXDWFoVTk1VWXUVJVRXVlGdUUZVRUxKitiwe/yGBXlMcrDIh6NZLz+dk4U2/8y0dl9PE9fskWjkVNuEJ9xYXf3HWbWbmaXu/tjBJdfe7C39USk+MQTCXbub8F3HmJ7/RG21x+h6XD78eXVlWWMH13DjEkjeNnc8YwZWcXo4VWMHlbJqGGVVFUEJaXYCuhgc9qF3cyWElxIdxXBRWq/Fx4SuZrcXDxYRPrBgSPtPLO1iXXbmtm8+xBtHcG31GuHV3HOhGG86sJJTBk7lEl1Qxk5tOL40IQUrz4Vdnc/O+X2NSm3nyG4uriIDAAHj3bw5Ib9rNiwj537gxNPjR1ZzaVzxnPu1JHMmjKSkUMrC5xSTlehTwImIv0kkUjyzNYmlq/Zw/oXDpBMwjkThvO2V07nghljmFBbo63xEqHCLlLiOrriPLqunj88tYuGQ22MHl7JtS87m5efN57xo2sKHU/yQIVdpER1xxP8eV09v3lsO4dbOpk+cThvfcU0LrI6YlGdJqqUqbCLlKC1W5r46bItNBxsY8bkEXzgTXOxqaMKHUv6iQq7SAk51NLBkj9sZpU3MmnMED56/TzmTa/V2Pkgo8IuUiKeeG4f9/5hM13dCd561TT+4tKplMU05DIYqbCLDHCdXXGWPLyZR56pZ+bkEfzNNbO1U3SQU2EXGcD2NrVw54+eZldDC9e+7CzecuU52jEqKuwiA9Xze4/w9V+sIx5P8LG3zWPe9DGFjiRFQoVdZABat62Zb//qWUYPr+Kj189j3CgNvciLVNhFBpgn1u/j7gc2MqluCF/4wOV0d3QVOpIUGQ3GiQwgT3sDi/9nAzMnj+Dvb1zAqOFVhY4kRUhb7CIDxHPbm/nPX69n+sQRfPT6C6isiBU6khQpbbGLDABbdx/mm//9LBPHDOFjb5unoi6npMIuUuSaDrXx9V+sY9TQSj7+9vnUVJUXOpIUORV2kSLW0RXnm//9LPFEko+97QJGDKkodCQZAFTYRYpUMpnkngc3sauhhfe/aQ7j9G1SyZIKu0iR+sNTu1ixYT9vuWqavnwkfaLCLlKEduw7yn1/2saCWXW84WVnFTqODDAq7CJFpqs7zuL/2cDQmnJufv25OuWu9JkKu0iR+eUj29nT1Mp7rpnN0GodASN9p8IuUkR850F+t3Inr7xwEudPqy10HBmgVNhFikRnV5y7l26kbmQ1N7xqeqHjyACmwi5SJJau2EHjoXbe/fpzqarQ2T7k9KmwixSBhkNtLF2xk0tmj2X2WbrotJwZFXaRIvDTh7cQi0Z4+6tnFjqKlAAVdpECW7etibVbm3jT5WczalhloeNICVBhFymg7niCJQ9vYfzoGq6+eEqh40iJUGEXKaA/r6un4WAb73jNDMpiejlKbuiZJFIgnV1xfvvYdmZMHqFj1iWnVNhFCmTZ6j0caunkuqum6bQBklNZHSxrZjcCdwDlwF3u/q205QuA7wIVwC7gr9z9UI6zipSMto5ulq7YwdxzRmNTdXij5FavW+xmNgm4E7gCmA/camZz0pr9B/BP7n4B4MDf5TqoSCn5w1O7aGnr4q1XTSt0FClB2QzFLAKWufsBd28F7geuT2sTA4aHt2uAttxFFCktre1d/O6pnSyYVcc5E4b3voJIH2UzFDMRqE+ZrgcuSWvzceD3ZnYX0Apc2pcQtbVD+9L8BHV1w0573Xwr1mzFmguKN1sucy1/eDNtHXFufuPcnNzvYOizXCrWXJC7bNkU9iiQTJmOAImeCTOrBr4PLHL3lWb2ceCHwLXZhmhubiGRSPbeME1d3TAaG4/2eb3+UKzZijUXFG+2XObq6o7z6//dynnTRjO0PHrG9zsY+iyXijUX9C1bNBo55QZxNkMxu4EJKdPjgb0p0+cBbe6+Mpz+LvDKrNKJDDKPPbuPI8e6eP2luiqS5E82hf1h4DVmVmdmNcB1wEMpy7cCU8zMwuk3A0/lNqbIwJdIJHlo5U7OHj+Mc6eOLHQcKWG9FnZ33wPcDiwH1gJLwiGXpWa20N0PAjcDPzezdcB7gL/JY2aRAWn15kYaDrZxzWVn6bh1yausjmN39yXAkrR516TcfhB4MLfRREpHMpnkwSd3MHZUNQtm1RU6jpQ4ffNUpB9s2X2Y7fVHed0lU4lGtbUu+aXCLtIPlq/ZQ3VlGS+fO77QUWQQUGEXybPDrZ2s2tTAFedPoLIiVug4MgiosIvk2SPP7CWeSPLKCycWOooMEirsInkUTyT437V7mHP2KCbUDil0HBkkVNhF8mjd1mYOHOngVRdOLnQUGURU2EXyaNmaPYwaVsn8mbqQhvQfFXaRPNl/8Bjrtx/glfMnEovqpSb9R882kTx5dF09kQhcMU87TaV/qbCL5EEikeTx5/Zx/rRaRg2rLHQcGWRU2EXyYP0LBzh4tIMrzp/Qe2ORHFNhF8mDR9fVM6SqjAtmjCl0FBmEVNhFcqy1vYs1Wxq5bO54ysv0EpP+p2edSI49uWE/3fGkhmGkYFTYRXLs0XX1TBk7lLPGF++1NaW0qbCL5NDuxhZe2HeUy7W1LgWkwi6SQ0+s30c0EuGyOeMKHUUGMRV2kRxJJJOs3NDA3HNGM3xIRaHjyCCmwi6SI9v2HKb5SLu21qXgVNhFcmTFhv1UlEWZP1PHrkthqbCL5EB3PMFTGxuYP3MM1ZVZXSNeJG9U2EVyYOOOg7S0dXHpbA3DSOGpsIvkwIr1+6mpLOO8aTrvuhSeCrvIGeroirN6SyMLz63TKQSkKOhZKHKG1m1rpqMzzqVzxhc6igigwi5yxp7auJ/hQyqwKSMLHUUEUGEXOSMdnXHWbWvmIqsjGo0UOo4IoMIuckbWPd9MZ3eCi21soaOIHKfCLnIGVm1qYHhNObM0DCNFRIVd5DR1dMV5ZlsTC2yshmGkqGT1FTkzuxG4AygH7nL3b6UtN+C7wChgH/AOdz+Y46wiReXZbc10diW42OoKHUXkBL1usZvZJOBO4ApgPnCrmc1JWR4BfgN8yd0vANYAn85PXJHiscobGFZTzqypGoaR4pLNUMwiYJm7H3D3VuB+4PqU5QuAVnd/KJz+V+BbiJSwzq44z2xtZsGsOmJRjWhKcclmKGYiUJ8yXQ9ckjI9A9hnZt8HLgQ2Ah/JWUKRIvTs8wfo6Iqz8FwdDSPFJ5vCHgWSKdMRIJF2H68ErnL3VWb2L8BXgZuzDVFbOzTbpi9RV1e815Us1mzFmguKN1t6rvW/38ywmgquXDCFWKywW+wDpc+KRbHmgtxly6aw7wauTJkeD+xNmd4HbHH3VeH0TwiGa7LW3NxCIpHsvWGaurphNDYe7fN6/aFYsxVrLijebOm5uuMJnly/j4tm1XHgQGsBkw2cPisWxZoL+pYtGo2ccoM4m02Nh4HXmFmdmdUA1wEPpSx/HKgzswvC6TcCT2eVTmQA2rTjIG0d3SzQ0TBSpHot7O6+B7gdWA6sBZa4+0ozW2pmC929DfhL4Htmth54NfCJfIYWKaTVmxuprIgx9+xRhY4iklFWx7G7+xJgSdq8a1JuP8mJO1RFSlIikWT1libmTaulvCxW6DgiGek4LZE+2LrnMEdaO1kwS8MwUrxU2EX6YPXmRspiEeZN15WSpHipsItkKZlMsnpzI3POHq0LVktRU2EXydKuhhaaDrdrGEaKngq7SJZWb24kEoH5M8cUOorIKamwi2Rp9eYmZk4eyfCaikJHETklFXaRLDQcamN3YwsLtLUuA4AKu0gW1m5uBGC+xtdlAFBhF8nC6i1NTK4bytiR1YWOItIrFXaRXhxu6WDL7kMsmKVhGBkYVNhFevHUhn0kk3DhTA3DyMCgwi7SixXP7aN2eCVTx53+dQNE+pMKu8gpdHTFWbO5kQtn1hGJRAodRyQrKuwip7B++wE6u+JcqKNhZABRYRc5hTWbGxlaXc6sKSMKHUUkayrsIicRTyRYu7WJi+eMIxbVS0UGDj1bRU5iy67DtLZ3c9l5EwodRaRPVNhFTmL1lkbKy6IssLGFjiLSJyrsIhkkk0nWbG5i7tmjqdK512WAUWEXyWBXQwvNR9q5UCf9kgFIhV0kgzVbmohE4AIVdhmAVNhFMlizuZEZk0bo3OsyIKmwi6RpOtTGzoYWnRtGBiwVdpE0a7Y0AXChzuYoA5QKu0ia1ZsbmTRmCONG1RQ6ishpUWEXSXHkWCebdx9igc4NIwOYCrtIirVbmkgmUWGXAU2FXSTF6s2NjBlRpXOvy4Cmwi4SauvoZsMLB1gwS+del4FNhV0k9OzzzXTHkxqGkQFPhV0k9LQ3MrymnBmTdO51GdiyKuxmdqOZbTCzLWb24VO0u9bMtucunkj/6OqOs+75Zi6cVUc0qmEYGdh6LexmNgm4E7gCmA/camZzMrQbB/wboFeFDDjrXzhIR2dcwzBSErLZYl8ELHP3A+7eCtwPXJ+h3WLgc7kMJ9JfVnsj1ZUxZp81qtBRRM5YNieangjUp0zXA5ekNjCz/wOsBlacToja2tM/tKyubthpr5tvxZqtWHNBYbJ1xxM8s62JS+ZMYML4zOPr6rO+U66+y1W2bAp7FEimTEeARM+EmZ0HXAe8Bph8OiGam1tIJJK9N0xTVzeMxsajp/OQeVes2Yo1FxQu23Pbmzl6rIvzzxmV8fHVZ32nXH3Xl2zRaOSUG8TZDMXsBlIv+jge2Jsy/bZw+SpgKTDRzP6cVTqRIrBqUwOVFTHOO2d0oaOI5EQ2W+wPA581szqglWDr/Naehe7+z8A/A5jZ2cCf3P3K3EcVyb3ueILVm5uYP2MMFeWxQscRyYlet9jdfQ9wO7AcWAsscfeVZrbUzBbmO6BIPvmuQ7S0dbHQdDSMlI6srtLr7kuAJWnzrsnQ7gXg7FwEE+kPqzY1UFke4/xptYWOIpIz+uapDFrxRIKnvZELZtRqGEZKigq7DFq+s2cYZmyho4jklAq7DFqrvJGK8ijnT9cwjJQWFXYZlIJhmAbmTR9DpYZhpMSosMugtPGFgxw91sVlc8YVOopIzqmwy6C0YsN+qivLdDSMlCQVdhl0OrviPL25kYVWR3mZXgJSevSslkHnmW3NdHTGNQwjJUuFXQadJzfsZ8TQCmyqTtErpUmFXQaVY+1drNvWxKWzx+lKSVKyVNhlUHnaG+mOJ7lUwzBSwlTYZVBZsWE/Y0dVc/b44r3YgsiZUmGXQePAkXY27TjIZXPGEYloGEZKlwq7DBqPPbePJHD5+RN6bSsykKmwy6CQTCZ57Nl6zp06krqR1YWOI5JXKuwyKGzZfZiGg23aWpdBQYVdBoVH19VTVRHTKXplUFBhl5LX3tnNU5sauPjcsVRW6EyOUvpU2KXkrdrUSEdXnCvmaRhGBgcVdil5jz5bz7jRNcyYNKLQUUT6hQq7lLS9Ta1s3nWIK84fr2PXZdBQYZeS9qc1e4hFI1w5b2Kho4j0GxV2KVkdnXEee66ei88dy/AhFYWOI9JvVNilZK3YsI+2jjivWjCp0FFE+pUKu5SkZDLJstV7mFw3VDtNZdBRYZeStG3PEXY1tPDqiyZpp6kMOirsUpKWrdlNdWVMl7+TQUmFXUrO4ZYOVm1q4OVzJ1BVUVboOCL9ToVdSs7DT+8mHk+y6OLJhY4iUhAq7FJS2jq6WbZ6DxdZHeNG1RQ6jkhBZPU51cxuBO4AyoG73P1bacvfDHwOiADbgb9x94M5zirSq/9du5e2jm5ef9lZhY4iUjC9brGb2STgTuAKYD5wq5nNSVk+HPgOcK27XwCsAz6bl7Qip9AdT/CHVbs4d+pIzpkwvNBxRAomm6GYRcAydz/g7q3A/cD1KcvLgQ+7+55weh0wNbcxRXq3Yv1+Dh7t0Na6DHrZDMVMBOpTpuuBS3om3L0Z+CWAmVUDnwa+kcOMIr1KJJM8tHInk+uGct45owsdR6SgsinsUSCZMh0BEumNzGwEQYF/xt3v6UuI2tqhfWl+grq6Yae9br4Va7ZizQWnn+2xZ/ayt6mVT9y4gLFjcz8MU4p9lm/K1Xe5ypZNYd8NXJkyPR7Ym9rAzCYAvwOWAX/b1xDNzS0kEsneG6apqxtGY+PRPq/XH4o1W7HmgtPPlkgkueeB9UyorWH25BE5//tKsc/yTbn6ri/ZotHIKTeIsynsDwOfNbM6oBW4Dri1Z6GZxYDfAj939y9klUokh55Yv4/65mN86C3nEY3q9AEivRZ2d99jZrcDy4EKYLG7rzSzpcA/AVOABUCZmfXsVF3l7rfkK7RIj+54gl8/up2zxg3jIqsrdByRopDVcezuvgRYkjbvmvDmKvRFJymQR57ZS9Phdm56nelkXyIhFWQZsDq64vz28ReYOXmEjoQRSaHCLgPWgyt2cLilk+teMV1b6yIpVNhlQGo41MbSFTu5ZPZYZk0ZWeg4IkVFhV0GpJ8+vIVYNMLbXz2z0FFEio4Kuww467Y1sXZrE2+6/GxGDassdByRoqPCLgNKV3eCJQ9vYfzoGq6+eEqh44gUJRV2GVB+/eh2Gg62cePVMymL6ekrkoleGTJgbN19mAef3MGV8yZw3jm1hY4jUrRU2GVAaO/sZvH/bKB2eBXveI12mIqcigq7DAj3Ld9G46E23nvtbKordYFqkVNRYZeit3ZrE8vX7OHqi6dgU0cVOo5I0VNhl6K2/8AxvvfbDUwdO5TrXjGt0HFEBgQVdilabR3dfOO/nyUWjXDbW8+nvCxW6EgiA4IKuxSlRDLJ9x/YyL7mY3zwzXMZM7K60JFEBgwVdilKv3zkeVZvbuSGV01n9tk6c6NIX6iwS9F5cMUOHnhiB6+YP1HfLhU5DSrsUlT+tGYP9/1pG5fMHstNr9XFM0ROhwq7FI0/PrWTH/3OmTe9llveMEfXLxU5TfqmhxRcMpnkoZU7uW/5NmafNYoPveU8nQdG5AyosEtBJZJJfr5sK79/ahdXzp/EXy2aSXmZirrImVBhl4I51t7N3Us3snpzI4sumsxH3rGA5uaWQscSGfBU2KUgduw7yrd/9SwHjnTwjlfP4OqLp2hMXSRHVNilXyUSSZat3s3Pl342r5gAAAtySURBVG9jWE05f3/jAmZMHlHoWCIlRYVd+s3uxhbueXAT2/YeYd70Wt577WyG1VQUOpZIyVFhl7w71t7FAyt28PuVu6iuLON9b5jDZXPH6Rh1kTxRYZe86eiKs+zp3SxdsYPW9m4uP288N7x6hrbSRfJMhV1y7nBrJ8tX72b5mj0cPdbFvOm1vPWqaUwdN6zQ0UQGBRV2yYlEMonvOMijz+7jqU376Y4nuWB6La+/7CxmTRlZ6Hgig4oKu5y2RCLJ83uPsGZLI09u3M+BIx1UV8a48oKJXL1wCuNH1xQ6osigpMIufdJ0uA3feYiNOw7y7PPNHD3WRSwaYc7Zo7nhVTOYP2MMFeW6IIZIIamwy0kdPdbJ7sZWXth3hO31R9m+9wjNR9oBGFJVxnnTapk/YwznTxtNTVV5gdOKSI+sCruZ3QjcAZQDd7n7t9KWzwcWA8OBR4APuHt3jrNKjiWTSVraujhwpIPmI+00HGyj4eAx9h9sY09TK0daO4+3rR1exTkThvHaS6Zw7tRRTKobQlSHK4oUpV4Lu5lNAu4ELgI6gMfNbLm7b0hpdi9wi7uvMLPvA+8DvpOPwPJSyWSS7niC9s44bZ1x2ju6ae+M09rexbH2blrbu2lp66KlrYuO7gSNB45xuLWDwy2ddHYnTrivIVVljB1Vw7xptUwcM4TJdUOYOn4Yw3WIosiAkc0W+yJgmbsfADCz+4Hrgc+H02cB1e6+Imz/A+Bz5LmwJ5JJVm3cz/7Go6dsl0ym3OaEiRd/ZWhzwnrJcH7yxabJZDKcHzRIJE+cXzOkkqNH20kSTCcSSRLJE28nksngdgLiiQSJRJLuRJJ4PEk8kSAe3u6KJ+juTtAdT9AVT9DVnaCzK0FXd5yO7gSdXfET8mYSicCw6nJGDKtiaFUZ0yeOYPiQCmqHVzF6eCWjh1cxdlQ1QzSkIjLgZVPYJwL1KdP1wCW9LJ/clxC1tUP70hyArbsO8bnFK3pvWKSi0QjRSIRoNEJZLEIsGiEWjRKLRYjFopRFI5SVRSmLRSmPRSkrj1FdXU55WZSKshgV5VEqymNUVsSoqiijKvxdU1VGdWXwe2h1BUOqyxlaU86QqvIBcZKturriPNa9WHNB8WZTrr7LVbZsCnuUE7ZpiQCJPizvVXNzC4lEL5ucaUZUxVh8+9XU7z9ywgNncrKh4J6vtEfSVo682OD4dE+bCJHj9xeJBLcjPfcV3o5GI9SNGUZzcwvRnjaRCNFo+Ls/x6aTCdpbO2hv7QCCJ05jL59yCqVYsxVrLijebMrVd33JFo1GTrlBnE1h3w1cmTI9HtibtnzCKZbnzbjRNUTj8f54qD4bUl3OsUoddCQi/S+bS9U8DLzGzOrMrAa4DnioZ6G77wDazezycNZNwIM5TyoiIlnptbC7+x7gdmA5sBZY4u4rzWypmS0Mm70L+JqZbQKGAl/PV2ARETm1rMYK3H0JsCRt3jUpt5/hxB2qIiJSILpqsIhIiVFhFxEpMSrsIiIlptDH48WAM/riTDF/6aZYsxVrLijebMWaC4o3m3L1XbbZUtplPJVqJNnbd9Hz6wrgz4UMICIygF0JPJo+s9CFvRK4mOA0BMX5TSMRkeITI/hi6FMEJ2c8QaELu4iI5Jh2noqIlBgVdhGREqPCLiJSYlTYRURKjAq7iEiJUWEXESkxKuwiIiWm0KcUyJqZ/QsQd/fPhtMjgR8D04BG4AZ335e2TgT4CvAGgsv1vc/dH8tDtrHA71NmjQDq3H1oWruzgOeAbeGs/e7+ulznSXvMdwNfAvaHsx5w99vT2vTal3nKdjnwNaACaAbeE164JbVNv/WZmd0I3AGUA3e5+7fSls8HFgPDgUeAD7h7dz6ypD3uPwM3hJMPuPunMix/D3AwnPW99Ox5zLYcGAt0hbPe7+5PpixfBHwVqAZ+5u539EOmW4DbUmadA/zI3W9LadOvfWZmw4HHgTe4+wvZ9IuZTQXuJehfB97l7i3ZPF7RF3YzG0HQAe8Evpyy6AvAn939WjO7CfgP4O1pq18HzAbmADOAB8xsdq5fjO7eAMwP80aBPxJcnCTdQoILlbw/l4/fi4XAx939J6dok01f5sOPgTe5+zozew/BBVrenNamX/rMzCYBdwIXEXyT73EzW+7uG1Ka3Qvc4u4rzOz7wPuA7+Q51yLgtcCFBNcWfsjM/tLdf5nSbCHwDnd/Ip9ZMmSLALOAszK9psysGrgbeAWwi+D193p3z+sV1tx9McEbMGY2F/gV8Nm0Zv3WZ2Z2KfA9gr7qS798G/i2u//UzP4R+Efg77N5zIEwFPNmYAvw72nzryUoDAA/AV5vZuUZ2vzU3RPuvhnYCbw8n2GBvwGOhRcnSXcxcJ6ZrTWzZWZ2fp6z9Dzmu83sWTO718xGZWiTTV/mlJlVAne4+7pw1jpgaoam/dVni4Bl7n7A3VuB+4HrU/KeBVS7+4pw1g+At+UpS6p64BPu3unuXcBGXtpPC4HPmNk6M/ummVX1Qy4AC3//3syeMbPb0pZfAmxx9+1h4b+X/umzVN8BPuPuTWnz+7PP3gd8mBevBd1rv4Svv6sInofQx+db0Rd2d/+hu3+Jl55LZiLBk56wc44AdSdrE6oHJucpKmYWI9hS//RJmrQT/BMXAP8G/MrMKvKVJ1QP/Aswj2Dr4JsZ2mTTlznl7h3ufi8c/5TzWYItq3T91We9PVf69bnUw93X97yZmNlMgiGZpT3LzWwosAb4JEEfjSTYsusPowg+nf4l8BrgA2Z2dcrygvRZj/DTTrW735c2v1/7zN1vcffUkx1m0y9jgCMpn4T61HdFMxRjZm8jGG9NtcndF51klfTzW0YIxtFTRQk+vp6qTZ/0kvMvCN6Jn820bs/+gdBSM/siwVDRM2eSKYtcPW2+zItj1amy6cu8ZAuL9D0Ez8V/TV83n32WprfnSs6fS30RDik8AHzS3bf0zA/HXK9JaffvBB/zMw0F5lQ4jHF8KCMcnroG+EM4q6B9BryfYBj3BIXss1A2/ZLehgxtTqpoCnv4rnpfrw1ftAcYD+w2szJgGMEOuFS7Cc6A1mM8L34cykfOtwA/Pdm6ZvYRgvHinpwRXtzpdEYy5TKzEWb2t+7eU1QjQKb9C9n0ZU6zhfmGAr8JH+vN4VBDepu89Vma3QSnQO2R/lzJ+XMpW+FO5l8AH3P3n6Ytmwoscve7w1n56p9Mua4AKt39jyd57EL2WQXBGPbNGZYVrM9C2fRLAzDCzGLuHg/bZ913RT8UcwpLgb8Ob7+dYOdf+j9nKfAuM4uZ2QyCnRdP5THTyzj1+eVfAbwXwMxeQXDqzU15zNMCfCrceQPBkQK/zNAum77Mh3uBrcDb3f0lpx4N9VefPQy8xszqzKyGYMf7Qz0Lw6N12sMiC3ATkNedgABmNoVgiOrG9KIeagO+bGbnhDszP0zm/3E+jAS+YmZVZjYMeHfaYz8JmJnNCIcpb6Qf+iw0D9gc7i9JV8g+gyz6JXz9/ZkXD2L46/Q2pzKQC/s/ApeZ2XrgQwT/HMzsTWa2OGxzP7CeYMfcr4H3untbHjNNI3g3Ps7MPmBmnw8nPwpcbWbPEYwXv9Pd8/bRNHynvwH4jpltJDji41Nhrs+b2QfCphn7Mp/M7EKCHeOXA6vDnaNLw2X93mfuvofgo/hyYC3Bp4SVZrbUzBaGzd4FfM3MNgFDCY7iybe/A6qAr4Z9tDbsn6VmttDdGwmGHH5LcEhchJceaJAX7v4/BMNDa4Cngbvd/Ykw40R3byfYYv4FsIHgDfn+k91fjmV6LRa8zwBO1S9mttjM3hQ2/RBwq5ltIPg0mfWhojofu4hIiRnIW+wiIpKBCruISIlRYRcRKTEq7CIiJUaFXUSkxKiwi4iUGBV2EZESo8IuIlJi/j/mMU566a77JQAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "<Figure size 432x288 with 1 Axes>"
+      ]
+     },
+     "metadata": {
+      "needs_background": "light"
+     },
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def sigmoid_fxn(x):\n",
+    "    yhat = list(map(lambda i: 1 / (1 + np.exp(-i)), x))\n",
+    "    return yhat\n",
+    "\n",
+    "x = np.arange(-10., 10., 0.2)\n",
+    "logit = sigmoid_fxn(x)\n",
+    "plt.title(\"The Sigmoid Function Curve\", fontsize=15)\n",
+    "\n",
+    "plt.plot(x, logit)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Building my Customized Logistic Regression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class LogisticClassifier():\n",
+    "    def __init__(self, \n",
+    "            learning_rate=0.1, \n",
+    "            tolerance=1e-4, \n",
+    "            max_iter=1000, \n",
+    "            batch_size=32, \n",
+    "            momentum_decay=0.9, \n",
+    "            early_stopping=3, \n",
+    "            validation_set=(None,None)):\n",
+    "        \n",
+    "        # Gradient descent parameters\n",
+    "        self.learning_rate = float(learning_rate)\n",
+    "        self.tolerance = float(tolerance)\n",
+    "        self.max_iter = int(max_iter)\n",
+    "        self.batch_size=32\n",
+    "        self.momentum_decay = float(momentum_decay)\n",
+    "        self.early_stopping = int(early_stopping)\n",
+    "        self.X_validation, self.y_validation = validation_set\n",
+    "        \n",
+    "        # to construct the design matrix\n",
+    "        self.add_intercept = True\n",
+    "        self.center = True \n",
+    "        self.scale = True\n",
+    "        \n",
+    "        self.training_loss_history = []\n",
+    "        \n",
+    "    def __sigmoid(self, X):\n",
+    "        return 1 / (1 + np.exp(-X))\n",
+    "    \n",
+    "    # z-score normalization and intercept addition\n",
+    "    def __design_matrix(self, X):\n",
+    "        if self.center:\n",
+    "            X = X - self.means\n",
+    "        if self.scale:\n",
+    "            X = X / self.standard_error\n",
+    "        if self.add_intercept:\n",
+    "            intercept = np.ones((X.shape[0], 1))\n",
+    "            X = np.hstack([intercept, X])\n",
+    "            \n",
+    "        return X\n",
+    "    \n",
+    "    def __fit_center_scale(self, X):\n",
+    "        self.means = X.mean(axis=0)\n",
+    "        self.standard_error = np.std(X, axis=0)\n",
+    "        \n",
+    "    def fit(self, X, y):\n",
+    "        self.__fit_center_scale(X)\n",
+    "\n",
+    "        n, k = X.shape\n",
+    "        \n",
+    "        # add intercept column to the design matrix\n",
+    "        X = self.__design_matrix(X)\n",
+    "\n",
+    "        # used for the convergence check\n",
+    "        previous_loss = -float('inf')\n",
+    "        self.converged = False\n",
+    "        self.stopped_early = False\n",
+    "        \n",
+    "        # initialize parameters\n",
+    "        self.beta = np.zeros(k + (1 if self.add_intercept else 0))\n",
+    "        momentum = self.beta * 0  # to get the same shape and dtype as beta\n",
+    "        \n",
+    "        for i in range(self.max_iter):\n",
+    "            shuffle = np.random.permutation(len(y))\n",
+    "            X = X[shuffle, :]\n",
+    "            y = y[shuffle]\n",
+    "            \n",
+    "            # we'll add one more batch, incase the batch size doesn't divide n evenly\n",
+    "            extra = (1 if n % self.batch_size else 0)\n",
+    "\n",
+    "            for batch_index in range(n // self.batch_size + extra):\n",
+    "                batch_slice = slice(\n",
+    "                    self.batch_size * batch_index, \n",
+    "                    self.batch_size * (batch_index + 1) )\n",
+    "                X_batch = X[batch_slice, :]\n",
+    "                y_batch = y[batch_slice]\n",
+    "            \n",
+    "                beta_ahead = self.beta + self.momentum_decay * momentum\n",
+    "                y_hat = self.__sigmoid(np.dot(X_batch, self.beta))\n",
+    "                \n",
+    "                # gradient descent\n",
+    "                residuals = (y_hat - y_batch).reshape( (X_batch.shape[0], 1) )\n",
+    "                gradient = (X_batch * residuals).mean(axis=0)\n",
+    "                momentum = self.momentum_decay * momentum - self.learning_rate * gradient\n",
+    "                self.beta += momentum\n",
+    "\n",
+    "            # with minibatch, we only check convergence at the end of every epoch.    \n",
+    "            y_hat = self.__sigmoid(np.dot(X, self.beta))\n",
+    "            self.loss = np.mean(-y * np.log(y_hat) - (1-y) * np.log(1-y_hat))\n",
+    "            self.training_loss_history.append(self.loss)\n",
+    "            \n",
+    "            # early stopping\n",
+    "            if self.check_validation_loss():\n",
+    "                self.stopped_early = True\n",
+    "                break \n",
+    "                \n",
+    "            if abs(previous_loss - self.loss) < self.tolerance:\n",
+    "                self.converged = True\n",
+    "                break\n",
+    "            else:\n",
+    "                previous_loss = self.loss\n",
+    "        \n",
+    "        self.iterations = i+1\n",
+    "        \n",
+    "    def predict_proba(self, X):\n",
+    "        # add intercept column to the design matrix\n",
+    "        X = self.__design_matrix(X)\n",
+    "        return self.__sigmoid(np.dot(X, self.beta))\n",
+    "\n",
+    "    def predict(self, X):\n",
+    "        predictions = self.predict_proba(X).round()\n",
+    "        return predictions\n",
+    "\n",
+    "    def check_validation_loss(self):\n",
+    "        # validation set loss\n",
+    "        if not hasattr(self, 'validation_loss_history'):\n",
+    "            self.validation_loss_history = []\n",
+    "        p_hat = self.predict_proba(self.X_validation)\n",
+    "        loss = np.mean(-self.y_validation * np.log(p_hat) - \\\n",
+    "               (1-self.y_validation) * np.log(1-p_hat))\n",
+    "        self.validation_loss_history.append(loss)\n",
+    "        \n",
+    "        t = self.early_stopping\n",
+    "        if t and len(self.validation_loss_history) > t * 2:\n",
+    "            recent_best = min(self.validation_loss_history[-t:])\n",
+    "            previous_best = min(self.validation_loss_history[:-t])\n",
+    "            if recent_best > previous_best:\n",
+    "                return True\n",
+    "        return False"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Testing and Evaluating the Customized Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.datasets import load_breast_cancer\n",
+    "from sklearn.model_selection import train_test_split as tts\n",
+    "\n",
+    "dummy_data = load_breast_cancer()\n",
+    "X, y = dummy_data.data, dummy_data.target\n",
+    "\n",
+    "X_train, X_test, y_train, y_test = tts(X, y, test_size=0.2, random_state=103)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = LogisticClassifier(learning_rate = 0.01,\n",
+    "                           tolerance=1e-5,\n",
+    "                           max_iter=2000,\n",
+    "                           early_stopping=3,\n",
+    "                           validation_set=(X_test, y_test))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.fit(X_train, y_train)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "y_pred = model.predict(X_test)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# defining the metrics function\n",
+    "def accuracy(predictions, actual):\n",
+    "    return sum(predictions == actual) / len(actual)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.9736842105263158"
+      ]
+     },
+     "execution_count": 23,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "accuracy(y_pred, y_test)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

From 636d0024ef34e3221ba9643f06f9846c5d173d91 Mon Sep 17 00:00:00 2001
From: Steven Kolawole <45284829+SteveKola@users.noreply.github.com>
Date: Wed, 22 Apr 2020 22:21:52 +0100
Subject: [PATCH 3/9] Delete empty.gitkeep

---
 Kolawole_Steven/empty.gitkeep | 1 -
 1 file changed, 1 deletion(-)
 delete mode 100644 Kolawole_Steven/empty.gitkeep

diff --git a/Kolawole_Steven/empty.gitkeep b/Kolawole_Steven/empty.gitkeep
deleted file mode 100644
index 8b13789..0000000
--- a/Kolawole_Steven/empty.gitkeep
+++ /dev/null
@@ -1 +0,0 @@
-

From 9fddf40e8b8d3c692f7afb2d9b8f1a6430b391f0 Mon Sep 17 00:00:00 2001
From: Steven Kolawole <45284829+SteveKola@users.noreply.github.com>
Date: Wed, 22 Apr 2020 22:33:55 +0100
Subject: [PATCH 4/9] Delete Logistic Regression from Scratch.ipynb

---
 .../Logistic Regression from Scratch.ipynb    | 322 ------------------
 1 file changed, 322 deletions(-)
 delete mode 100644 Kolawole_Steven/Logistic Regression from Scratch.ipynb

diff --git a/Kolawole_Steven/Logistic Regression from Scratch.ipynb b/Kolawole_Steven/Logistic Regression from Scratch.ipynb
deleted file mode 100644
index 6031d92..0000000
--- a/Kolawole_Steven/Logistic Regression from Scratch.ipynb	
+++ /dev/null
@@ -1,322 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Logistic Regression from Scratch\n",
-    "## - Steven Kolawole                                "
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Intro\n",
-    "\n",
-    "Logistic Regression is simply a Linear Regression with a Sigmoid function at its end.\n",
-    "\n",
-    "The Sigmoid function generates probability (i.e. outputs between 0 and 1) for all values of X."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "import seaborn as sns\n",
-    "sns.set()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEMCAYAAADQ553CAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZhcZZn38W9V9Z49nc6eANlukkAIISzK4hZ0BLcRRMVhZBRxw1dHR8cRZkYdGX11RhnXcYy8ohgXcNyGgIqJgywhhCQEstxZCFk76SVrd3qtqvePczpUikq6Oqnqqq7+fa6rr65zznOqfv101V2nnnPqnEgymUREREpHtNABREQkt1TYRURKjAq7iEiJUWEXESkxKuwiIiVGhV1EpMSUFTqAnJqZ/QB49yma7HD3s83sBWCxu38hx4//18BtwFwgAawDvu7uP0tpkwRucvd7c/nYfch4M8HfnvH5bGaTgV3Aq9z9TxmWvxJYfpK7b3b3MblJempm9jIg6u6PhdN571cziwC3AO8B5gDdwFrgi+7+cL4eV/JLW+zF76PAhPDnknDem1PmXZyvBzazW4FvAt8GLgAuBR4AfmJmqW82E4D785UjCz8DJuXgfhbwYr/2/MzJwf1m6xFgZsp0XvvVzKLAr4A7gXsInl9XAU8DvzOzG/P12JJf2mIvcu5+GDgMYGZV4ewD7r6vHx7+/cD33P0HKfM2mJkRvOHcE2bsjywn5e5tQFsO7qqxwH9LJHWiH7LcBlwDXOTu61Lmf8rMhgJfN7Nfu3trnnNIjqmwl5ZJZvYbYBHBm8E33P1fexaa2VuAzwEGvAAsBr7q7omT3F8cuNzMRoRvMD3+DhiScr8nDBmY2SeBjwBjgN8BO4EL3P2V4bDHQ8BbgK8DU4DHgJuBTwM3Ae3A19z9SymP8R7g48B0YA9wl7t/M1x2MylDMWZ2FsGnjFcADcDxPjhdmYZ7MjxuEnhv+LdcDOwg6N//SlnnJuBTwIxw+Rfd/Z5wKC0G/D8zuznsq/R+7a0PPg18FfgMUAs8BXzY3Tee5M96P/DrtKLe43PADwj+FxmHhVLnhUOG1cBYgk8+nyb4H0xz9+0p66wHfunud5jZFOBrwGsJ3piXAx93970nyStZ0lBMaXkvsJRgPPw/gDvN7CoAM7sG+HE4fy5Bcfko8I+nuL+vEHw832tmvzGzvzOz+e7e6O4vZFrBzD4C/BNBcbmQ4A3kI2nNKoDPAzcCryYoBOuAFoKC+F3gi2Y2J7zPjxMMCd0FzAtzfcXMPpHh8csJ3jhqgMvDPvn0Kf7GXPu/BFkvBP4MfCd8o8HM3g7cTfCGej7wb8BiM3stwd8dBz4GvDX9TrPsg2nAu4DrCIrlWcA3MoUMP/3NAVZkWu7u+919pbvH+/C33wD8kmDI7lcEb1zvSHnM+eFj/tDMhgB/IijoLwdeR/C8WGZmFX14TMlAW+yl5T53/8/w9pfM7NPAQoKx288A33b3u8Pl28xsGPA9M/uXTFvt7n6fme0hKDavBd4IYGZrCLbU1mfI8AmCre2eLbu/NbMr0tpEgNvdfVV4f38kKGz/4O5JM/siwRvOXDPbSPAmdJe7Lw7X32Jm04C/N7Ovpt33IoJPJK9z953h/f8fgn0DvfFwKzTVPHd/Pot1e9zt7j8PH/eTBDsmLyEoch8Dfuzu/xG23RoOeUTdvTEY4eKwux9IvcNwB2c2fVAOfKBnC93MvkEwfp7JqPD3oT78bb3Z5+5fT8n9I+CdwBfDWe8CVrj7ZjO7heBT3809bx5m9k6gieCN6Sc5zDXoqLCXls1p04cIPh5DsAV5sZl9MGV5NFx+NpCxeLn748DjZhYDLiIo7h8BHjSzGe7e2dPWzGoJthKfSLubx4D5afO2ptxuBZ5392T4mG1hkasE6oBx4X2keoSg2I1Nm38e0NRT1EMZt0ozeB2QPq69K8t1exz/H7j7ofDv6NkCPR/4UWpjd78ri/vMtg+SwJaU5YdSHjtdc9h+dBaPn63059A9wB1mNhfYSLD13vNGcyHB33U47KMeNcDsHGYalFTYS0umj809O+Q6gS8TDMek250+Ixz//Afg8+6+L9yqWgmsNLM/E4ydzwNWpazWHf7OZoivK236ZOP87SeZHzvJ/SRJ2wlJ8Ldn4wV3f0lfnEKm109Hhnk9edKzZivbPki4e3dam/S+AMDdO81sNXBZpuVmNhP4FvC3mT6ZmVmmv/2EHdjuvtXMHifYan+YoJD3HCbbCawnw7ATuf0UMShpjH3wWA/MdPetPT8EW5B3kvnF30YwjJDpkLdDBAW0IXVmuIN1J8EYa6r06ay5+xGCN5704ZwrCLauD6bNXwuMCQtTj4Wn+/gpOoGYmdWkzJt5ssYnsTE9i5n90Mx6hi8ynkP7NPogW98H3mhm8zIs+yTBENIL4XQXMDxlebZ/+z0EQys3AL91956s64FzCL4n0PN8bCDY+Xt+X/4IeSltsQ8eXwAeMLPngF8Aswh2Ui5195dsZbp7k5l9mWAn5vBwnTaCF90XgHvShjt69KyziWBr/maCrcI/nWH2r5nZtvB+XkUwHPRP4Zh8atvlBMdh32tmHyYYivg6Z24FQeH9nJl9i+DN6uY+3seXgZ+b2UrgDwQ7jt8JvD5cfhSYY2Zj3b0hbd2+9EG2/ovg6KQ/mtlnCPpuOHArwU7nd6Yc6vgEcKuZPUbwSeFrZP50ku5nBDvs/4pgjL3Hj4HbCfrjHwg+lXyJ4M0k074b6QNtsQ8S7v4QwaGENwLPERT1HxIc8nayde4Il18NPErwguv5MsutJ1nt28C/hz/PELyB/Irsh0My5fguwc7ffwgzfJzgsLivZGgbJzg2eydBobqPoAidkXAH6geBtwGbgPcRbNX25T5+BXyYIP96gp2pN6V8w/NLwIcIhrnS1826D/qQJw68geAN5zZgDfB7gqNrXt2zEzj0QYJDaJ8k+NLUf5FhCC/DYxwGfk1QuB9Mmd9G8Lw6Biwj2H9QFj5u+pua9FFEV1CSXDKzvwCedfc9KfMeAva4+3sLl0xk8NBQjOTau4FzwmGQZoItwkUEh0uKSD9QYZdcu43gSzQPAsMIhi3e6e7LCppKZBDRUIyISInRzlMRkRJT6KGYSoKvkteT+cs1IiLyUjGC0zo/RYbDTgtd2C8mOFGSiIj03ZUEhyKfoNCFvR7g4MFWEom+j/XX1g6lubkl56FyoVizFWsuKN5sxZoLijebcvVdX7JFoxFGjRoCYQ1NV+jCHgdIJJKnVdh71i1WxZqtWHNB8WYr1lxQvNmUq+9OI1vGIWztPBURKTEq7CIiJUaFXUSkxGQ9xh6e4e9x4A3pl0ULL3m1mODMcI8QXMUl/bzQIiLSD7LaYjezSwkOqZl1kib3Are5+yyCc3u/LzfxRESkr7IdinkfwelGX3L18PBCvdXu3nP5sR8QnNpURGTQSyaTx38SaT/5ktVQjLvfAnCSk/lP5MRjKeuByWecTERKRld3gmPtXRzr6Ka9M057RzftXXE6OuN0dMXp7ErQ2R2nqzsR/MQTdIe/4/Ek3fEE3fEk8USSeCKYF08Gh0nHw8OlE8me35AMp4NiSngbopEI3fEEPefICo4uDJaFN4MiDCR7Zhy/zfHbyZ6LXaXO72OfRCLwwTefx8Jz0y/be+ZycRx7lBP/pggnv35lRrW1Q0/7wevqhp32uvlWrNmKNRcUb7ZizQWFy9bVnaD5cBuNh9poPNjGgSPtHDzazqEjHRxq6eBIaydHj3VytLWTzu7sS0JZLEpleZTyshhlZVHKy6KUxaKUxSLh7yhl5VGqYhFi0SjRaIRYNEI0GiEaSf3N8duRSIRIJCjshL8jEYL5ABGIcOK8SHjByEh4I5JyAcnj64XrvngzktImdX6KcEE0CpfMm0jtiOrji3L1v8xFYd9NcM6CHuPJMGRzKs3NLaf1pYG6umE0Nh7t83r9oVizFWsuKN5sxZoL+idbS1sXexpb2NPUyp6mVvYfOEbDwTaaj7STPppQWRFjxJAKRg+vYuSQCibXDWFoVTk1VWXUVJVRXVlGdUUZVRUxKitiwe/yGBXlMcrDIh6NZLz+dk4U2/8y0dl9PE9fskWjkVNuEJ9xYXf3HWbWbmaXu/tjBJdfe7C39USk+MQTCXbub8F3HmJ7/RG21x+h6XD78eXVlWWMH13DjEkjeNnc8YwZWcXo4VWMHlbJqGGVVFUEJaXYCuhgc9qF3cyWElxIdxXBRWq/Fx4SuZrcXDxYRPrBgSPtPLO1iXXbmtm8+xBtHcG31GuHV3HOhGG86sJJTBk7lEl1Qxk5tOL40IQUrz4Vdnc/O+X2NSm3nyG4uriIDAAHj3bw5Ib9rNiwj537gxNPjR1ZzaVzxnPu1JHMmjKSkUMrC5xSTlehTwImIv0kkUjyzNYmlq/Zw/oXDpBMwjkThvO2V07nghljmFBbo63xEqHCLlLiOrriPLqunj88tYuGQ22MHl7JtS87m5efN57xo2sKHU/yQIVdpER1xxP8eV09v3lsO4dbOpk+cThvfcU0LrI6YlGdJqqUqbCLlKC1W5r46bItNBxsY8bkEXzgTXOxqaMKHUv6iQq7SAk51NLBkj9sZpU3MmnMED56/TzmTa/V2Pkgo8IuUiKeeG4f9/5hM13dCd561TT+4tKplMU05DIYqbCLDHCdXXGWPLyZR56pZ+bkEfzNNbO1U3SQU2EXGcD2NrVw54+eZldDC9e+7CzecuU52jEqKuwiA9Xze4/w9V+sIx5P8LG3zWPe9DGFjiRFQoVdZABat62Zb//qWUYPr+Kj189j3CgNvciLVNhFBpgn1u/j7gc2MqluCF/4wOV0d3QVOpIUGQ3GiQwgT3sDi/9nAzMnj+Dvb1zAqOFVhY4kRUhb7CIDxHPbm/nPX69n+sQRfPT6C6isiBU6khQpbbGLDABbdx/mm//9LBPHDOFjb5unoi6npMIuUuSaDrXx9V+sY9TQSj7+9vnUVJUXOpIUORV2kSLW0RXnm//9LPFEko+97QJGDKkodCQZAFTYRYpUMpnkngc3sauhhfe/aQ7j9G1SyZIKu0iR+sNTu1ixYT9vuWqavnwkfaLCLlKEduw7yn1/2saCWXW84WVnFTqODDAq7CJFpqs7zuL/2cDQmnJufv25OuWu9JkKu0iR+eUj29nT1Mp7rpnN0GodASN9p8IuUkR850F+t3Inr7xwEudPqy10HBmgVNhFikRnV5y7l26kbmQ1N7xqeqHjyACmwi5SJJau2EHjoXbe/fpzqarQ2T7k9KmwixSBhkNtLF2xk0tmj2X2WbrotJwZFXaRIvDTh7cQi0Z4+6tnFjqKlAAVdpECW7etibVbm3jT5WczalhloeNICVBhFymg7niCJQ9vYfzoGq6+eEqh40iJUGEXKaA/r6un4WAb73jNDMpiejlKbuiZJFIgnV1xfvvYdmZMHqFj1iWnVNhFCmTZ6j0caunkuqum6bQBklNZHSxrZjcCdwDlwF3u/q205QuA7wIVwC7gr9z9UI6zipSMto5ulq7YwdxzRmNTdXij5FavW+xmNgm4E7gCmA/camZz0pr9B/BP7n4B4MDf5TqoSCn5w1O7aGnr4q1XTSt0FClB2QzFLAKWufsBd28F7geuT2sTA4aHt2uAttxFFCktre1d/O6pnSyYVcc5E4b3voJIH2UzFDMRqE+ZrgcuSWvzceD3ZnYX0Apc2pcQtbVD+9L8BHV1w0573Xwr1mzFmguKN1sucy1/eDNtHXFufuPcnNzvYOizXCrWXJC7bNkU9iiQTJmOAImeCTOrBr4PLHL3lWb2ceCHwLXZhmhubiGRSPbeME1d3TAaG4/2eb3+UKzZijUXFG+2XObq6o7z6//dynnTRjO0PHrG9zsY+iyXijUX9C1bNBo55QZxNkMxu4EJKdPjgb0p0+cBbe6+Mpz+LvDKrNKJDDKPPbuPI8e6eP2luiqS5E82hf1h4DVmVmdmNcB1wEMpy7cCU8zMwuk3A0/lNqbIwJdIJHlo5U7OHj+Mc6eOLHQcKWG9FnZ33wPcDiwH1gJLwiGXpWa20N0PAjcDPzezdcB7gL/JY2aRAWn15kYaDrZxzWVn6bh1yausjmN39yXAkrR516TcfhB4MLfRREpHMpnkwSd3MHZUNQtm1RU6jpQ4ffNUpB9s2X2Y7fVHed0lU4lGtbUu+aXCLtIPlq/ZQ3VlGS+fO77QUWQQUGEXybPDrZ2s2tTAFedPoLIiVug4MgiosIvk2SPP7CWeSPLKCycWOooMEirsInkUTyT437V7mHP2KCbUDil0HBkkVNhF8mjd1mYOHOngVRdOLnQUGURU2EXyaNmaPYwaVsn8mbqQhvQfFXaRPNl/8Bjrtx/glfMnEovqpSb9R882kTx5dF09kQhcMU87TaV/qbCL5EEikeTx5/Zx/rRaRg2rLHQcGWRU2EXyYP0LBzh4tIMrzp/Qe2ORHFNhF8mDR9fVM6SqjAtmjCl0FBmEVNhFcqy1vYs1Wxq5bO54ysv0EpP+p2edSI49uWE/3fGkhmGkYFTYRXLs0XX1TBk7lLPGF++1NaW0qbCL5NDuxhZe2HeUy7W1LgWkwi6SQ0+s30c0EuGyOeMKHUUGMRV2kRxJJJOs3NDA3HNGM3xIRaHjyCCmwi6SI9v2HKb5SLu21qXgVNhFcmTFhv1UlEWZP1PHrkthqbCL5EB3PMFTGxuYP3MM1ZVZXSNeJG9U2EVyYOOOg7S0dXHpbA3DSOGpsIvkwIr1+6mpLOO8aTrvuhSeCrvIGeroirN6SyMLz63TKQSkKOhZKHKG1m1rpqMzzqVzxhc6igigwi5yxp7auJ/hQyqwKSMLHUUEUGEXOSMdnXHWbWvmIqsjGo0UOo4IoMIuckbWPd9MZ3eCi21soaOIHKfCLnIGVm1qYHhNObM0DCNFRIVd5DR1dMV5ZlsTC2yshmGkqGT1FTkzuxG4AygH7nL3b6UtN+C7wChgH/AOdz+Y46wiReXZbc10diW42OoKHUXkBL1usZvZJOBO4ApgPnCrmc1JWR4BfgN8yd0vANYAn85PXJHiscobGFZTzqypGoaR4pLNUMwiYJm7H3D3VuB+4PqU5QuAVnd/KJz+V+BbiJSwzq44z2xtZsGsOmJRjWhKcclmKGYiUJ8yXQ9ckjI9A9hnZt8HLgQ2Ah/JWUKRIvTs8wfo6Iqz8FwdDSPFJ5vCHgWSKdMRIJF2H68ErnL3VWb2L8BXgZuzDVFbOzTbpi9RV1e815Us1mzFmguKN1t6rvW/38ywmgquXDCFWKywW+wDpc+KRbHmgtxly6aw7wauTJkeD+xNmd4HbHH3VeH0TwiGa7LW3NxCIpHsvWGaurphNDYe7fN6/aFYsxVrLijebOm5uuMJnly/j4tm1XHgQGsBkw2cPisWxZoL+pYtGo2ccoM4m02Nh4HXmFmdmdUA1wEPpSx/HKgzswvC6TcCT2eVTmQA2rTjIG0d3SzQ0TBSpHot7O6+B7gdWA6sBZa4+0ozW2pmC929DfhL4Htmth54NfCJfIYWKaTVmxuprIgx9+xRhY4iklFWx7G7+xJgSdq8a1JuP8mJO1RFSlIikWT1libmTaulvCxW6DgiGek4LZE+2LrnMEdaO1kwS8MwUrxU2EX6YPXmRspiEeZN15WSpHipsItkKZlMsnpzI3POHq0LVktRU2EXydKuhhaaDrdrGEaKngq7SJZWb24kEoH5M8cUOorIKamwi2Rp9eYmZk4eyfCaikJHETklFXaRLDQcamN3YwsLtLUuA4AKu0gW1m5uBGC+xtdlAFBhF8nC6i1NTK4bytiR1YWOItIrFXaRXhxu6WDL7kMsmKVhGBkYVNhFevHUhn0kk3DhTA3DyMCgwi7SixXP7aN2eCVTx53+dQNE+pMKu8gpdHTFWbO5kQtn1hGJRAodRyQrKuwip7B++wE6u+JcqKNhZABRYRc5hTWbGxlaXc6sKSMKHUUkayrsIicRTyRYu7WJi+eMIxbVS0UGDj1bRU5iy67DtLZ3c9l5EwodRaRPVNhFTmL1lkbKy6IssLGFjiLSJyrsIhkkk0nWbG5i7tmjqdK512WAUWEXyWBXQwvNR9q5UCf9kgFIhV0kgzVbmohE4AIVdhmAVNhFMlizuZEZk0bo3OsyIKmwi6RpOtTGzoYWnRtGBiwVdpE0a7Y0AXChzuYoA5QKu0ia1ZsbmTRmCONG1RQ6ishpUWEXSXHkWCebdx9igc4NIwOYCrtIirVbmkgmUWGXAU2FXSTF6s2NjBlRpXOvy4Cmwi4SauvoZsMLB1gwS+del4FNhV0k9OzzzXTHkxqGkQFPhV0k9LQ3MrymnBmTdO51GdiyKuxmdqOZbTCzLWb24VO0u9bMtucunkj/6OqOs+75Zi6cVUc0qmEYGdh6LexmNgm4E7gCmA/camZzMrQbB/wboFeFDDjrXzhIR2dcwzBSErLZYl8ELHP3A+7eCtwPXJ+h3WLgc7kMJ9JfVnsj1ZUxZp81qtBRRM5YNieangjUp0zXA5ekNjCz/wOsBlacToja2tM/tKyubthpr5tvxZqtWHNBYbJ1xxM8s62JS+ZMYML4zOPr6rO+U66+y1W2bAp7FEimTEeARM+EmZ0HXAe8Bph8OiGam1tIJJK9N0xTVzeMxsajp/OQeVes2Yo1FxQu23Pbmzl6rIvzzxmV8fHVZ32nXH3Xl2zRaOSUG8TZDMXsBlIv+jge2Jsy/bZw+SpgKTDRzP6cVTqRIrBqUwOVFTHOO2d0oaOI5EQ2W+wPA581szqglWDr/Naehe7+z8A/A5jZ2cCf3P3K3EcVyb3ueILVm5uYP2MMFeWxQscRyYlet9jdfQ9wO7AcWAsscfeVZrbUzBbmO6BIPvmuQ7S0dbHQdDSMlI6srtLr7kuAJWnzrsnQ7gXg7FwEE+kPqzY1UFke4/xptYWOIpIz+uapDFrxRIKnvZELZtRqGEZKigq7DFq+s2cYZmyho4jklAq7DFqrvJGK8ijnT9cwjJQWFXYZlIJhmAbmTR9DpYZhpMSosMugtPGFgxw91sVlc8YVOopIzqmwy6C0YsN+qivLdDSMlCQVdhl0OrviPL25kYVWR3mZXgJSevSslkHnmW3NdHTGNQwjJUuFXQadJzfsZ8TQCmyqTtErpUmFXQaVY+1drNvWxKWzx+lKSVKyVNhlUHnaG+mOJ7lUwzBSwlTYZVBZsWE/Y0dVc/b44r3YgsiZUmGXQePAkXY27TjIZXPGEYloGEZKlwq7DBqPPbePJHD5+RN6bSsykKmwy6CQTCZ57Nl6zp06krqR1YWOI5JXKuwyKGzZfZiGg23aWpdBQYVdBoVH19VTVRHTKXplUFBhl5LX3tnNU5sauPjcsVRW6EyOUvpU2KXkrdrUSEdXnCvmaRhGBgcVdil5jz5bz7jRNcyYNKLQUUT6hQq7lLS9Ta1s3nWIK84fr2PXZdBQYZeS9qc1e4hFI1w5b2Kho4j0GxV2KVkdnXEee66ei88dy/AhFYWOI9JvVNilZK3YsI+2jjivWjCp0FFE+pUKu5SkZDLJstV7mFw3VDtNZdBRYZeStG3PEXY1tPDqiyZpp6kMOirsUpKWrdlNdWVMl7+TQUmFXUrO4ZYOVm1q4OVzJ1BVUVboOCL9ToVdSs7DT+8mHk+y6OLJhY4iUhAq7FJS2jq6WbZ6DxdZHeNG1RQ6jkhBZPU51cxuBO4AyoG73P1bacvfDHwOiADbgb9x94M5zirSq/9du5e2jm5ef9lZhY4iUjC9brGb2STgTuAKYD5wq5nNSVk+HPgOcK27XwCsAz6bl7Qip9AdT/CHVbs4d+pIzpkwvNBxRAomm6GYRcAydz/g7q3A/cD1KcvLgQ+7+55weh0wNbcxRXq3Yv1+Dh7t0Na6DHrZDMVMBOpTpuuBS3om3L0Z+CWAmVUDnwa+kcOMIr1KJJM8tHInk+uGct45owsdR6SgsinsUSCZMh0BEumNzGwEQYF/xt3v6UuI2tqhfWl+grq6Yae9br4Va7ZizQWnn+2xZ/ayt6mVT9y4gLFjcz8MU4p9lm/K1Xe5ypZNYd8NXJkyPR7Ym9rAzCYAvwOWAX/b1xDNzS0kEsneG6apqxtGY+PRPq/XH4o1W7HmgtPPlkgkueeB9UyorWH25BE5//tKsc/yTbn6ri/ZotHIKTeIsynsDwOfNbM6oBW4Dri1Z6GZxYDfAj939y9klUokh55Yv4/65mN86C3nEY3q9AEivRZ2d99jZrcDy4EKYLG7rzSzpcA/AVOABUCZmfXsVF3l7rfkK7RIj+54gl8/up2zxg3jIqsrdByRopDVcezuvgRYkjbvmvDmKvRFJymQR57ZS9Phdm56nelkXyIhFWQZsDq64vz28ReYOXmEjoQRSaHCLgPWgyt2cLilk+teMV1b6yIpVNhlQGo41MbSFTu5ZPZYZk0ZWeg4IkVFhV0GpJ8+vIVYNMLbXz2z0FFEio4Kuww467Y1sXZrE2+6/GxGDassdByRoqPCLgNKV3eCJQ9vYfzoGq6+eEqh44gUJRV2GVB+/eh2Gg62cePVMymL6ekrkoleGTJgbN19mAef3MGV8yZw3jm1hY4jUrRU2GVAaO/sZvH/bKB2eBXveI12mIqcigq7DAj3Ld9G46E23nvtbKordYFqkVNRYZeit3ZrE8vX7OHqi6dgU0cVOo5I0VNhl6K2/8AxvvfbDUwdO5TrXjGt0HFEBgQVdilabR3dfOO/nyUWjXDbW8+nvCxW6EgiA4IKuxSlRDLJ9x/YyL7mY3zwzXMZM7K60JFEBgwVdilKv3zkeVZvbuSGV01n9tk6c6NIX6iwS9F5cMUOHnhiB6+YP1HfLhU5DSrsUlT+tGYP9/1pG5fMHstNr9XFM0ROhwq7FI0/PrWTH/3OmTe9llveMEfXLxU5TfqmhxRcMpnkoZU7uW/5NmafNYoPveU8nQdG5AyosEtBJZJJfr5sK79/ahdXzp/EXy2aSXmZirrImVBhl4I51t7N3Us3snpzI4sumsxH3rGA5uaWQscSGfBU2KUgduw7yrd/9SwHjnTwjlfP4OqLp2hMXSRHVNilXyUSSZat3s3Pl342r5gAAAtySURBVG9jWE05f3/jAmZMHlHoWCIlRYVd+s3uxhbueXAT2/YeYd70Wt577WyG1VQUOpZIyVFhl7w71t7FAyt28PuVu6iuLON9b5jDZXPH6Rh1kTxRYZe86eiKs+zp3SxdsYPW9m4uP288N7x6hrbSRfJMhV1y7nBrJ8tX72b5mj0cPdbFvOm1vPWqaUwdN6zQ0UQGBRV2yYlEMonvOMijz+7jqU376Y4nuWB6La+/7CxmTRlZ6Hgig4oKu5y2RCLJ83uPsGZLI09u3M+BIx1UV8a48oKJXL1wCuNH1xQ6osigpMIufdJ0uA3feYiNOw7y7PPNHD3WRSwaYc7Zo7nhVTOYP2MMFeW6IIZIIamwy0kdPdbJ7sZWXth3hO31R9m+9wjNR9oBGFJVxnnTapk/YwznTxtNTVV5gdOKSI+sCruZ3QjcAZQDd7n7t9KWzwcWA8OBR4APuHt3jrNKjiWTSVraujhwpIPmI+00HGyj4eAx9h9sY09TK0daO4+3rR1exTkThvHaS6Zw7tRRTKobQlSHK4oUpV4Lu5lNAu4ELgI6gMfNbLm7b0hpdi9wi7uvMLPvA+8DvpOPwPJSyWSS7niC9s44bZ1x2ju6ae+M09rexbH2blrbu2lp66KlrYuO7gSNB45xuLWDwy2ddHYnTrivIVVljB1Vw7xptUwcM4TJdUOYOn4Yw3WIosiAkc0W+yJgmbsfADCz+4Hrgc+H02cB1e6+Imz/A+Bz5LmwJ5JJVm3cz/7Go6dsl0ym3OaEiRd/ZWhzwnrJcH7yxabJZDKcHzRIJE+cXzOkkqNH20kSTCcSSRLJE28nksngdgLiiQSJRJLuRJJ4PEk8kSAe3u6KJ+juTtAdT9AVT9DVnaCzK0FXd5yO7gSdXfET8mYSicCw6nJGDKtiaFUZ0yeOYPiQCmqHVzF6eCWjh1cxdlQ1QzSkIjLgZVPYJwL1KdP1wCW9LJ/clxC1tUP70hyArbsO8bnFK3pvWKSi0QjRSIRoNEJZLEIsGiEWjRKLRYjFopRFI5SVRSmLRSmPRSkrj1FdXU55WZSKshgV5VEqymNUVsSoqiijKvxdU1VGdWXwe2h1BUOqyxlaU86QqvIBcZKturriPNa9WHNB8WZTrr7LVbZsCnuUE7ZpiQCJPizvVXNzC4lEL5ucaUZUxVh8+9XU7z9ywgNncrKh4J6vtEfSVo682OD4dE+bCJHj9xeJBLcjPfcV3o5GI9SNGUZzcwvRnjaRCNFo+Ls/x6aTCdpbO2hv7QCCJ05jL59yCqVYsxVrLijebMrVd33JFo1GTrlBnE1h3w1cmTI9HtibtnzCKZbnzbjRNUTj8f54qD4bUl3OsUoddCQi/S+bS9U8DLzGzOrMrAa4DnioZ6G77wDazezycNZNwIM5TyoiIlnptbC7+x7gdmA5sBZY4u4rzWypmS0Mm70L+JqZbQKGAl/PV2ARETm1rMYK3H0JsCRt3jUpt5/hxB2qIiJSILpqsIhIiVFhFxEpMSrsIiIlptDH48WAM/riTDF/6aZYsxVrLijebMWaC4o3m3L1XbbZUtplPJVqJNnbd9Hz6wrgz4UMICIygF0JPJo+s9CFvRK4mOA0BMX5TSMRkeITI/hi6FMEJ2c8QaELu4iI5Jh2noqIlBgVdhGREqPCLiJSYlTYRURKjAq7iEiJUWEXESkxKuwiIiWm0KcUyJqZ/QsQd/fPhtMjgR8D04BG4AZ335e2TgT4CvAGgsv1vc/dH8tDtrHA71NmjQDq3H1oWruzgOeAbeGs/e7+ulznSXvMdwNfAvaHsx5w99vT2vTal3nKdjnwNaACaAbeE164JbVNv/WZmd0I3AGUA3e5+7fSls8HFgPDgUeAD7h7dz6ypD3uPwM3hJMPuPunMix/D3AwnPW99Ox5zLYcGAt0hbPe7+5PpixfBHwVqAZ+5u539EOmW4DbUmadA/zI3W9LadOvfWZmw4HHgTe4+wvZ9IuZTQXuJehfB97l7i3ZPF7RF3YzG0HQAe8Evpyy6AvAn939WjO7CfgP4O1pq18HzAbmADOAB8xsdq5fjO7eAMwP80aBPxJcnCTdQoILlbw/l4/fi4XAx939J6dok01f5sOPgTe5+zozew/BBVrenNamX/rMzCYBdwIXEXyT73EzW+7uG1Ka3Qvc4u4rzOz7wPuA7+Q51yLgtcCFBNcWfsjM/tLdf5nSbCHwDnd/Ip9ZMmSLALOAszK9psysGrgbeAWwi+D193p3z+sV1tx9McEbMGY2F/gV8Nm0Zv3WZ2Z2KfA9gr7qS798G/i2u//UzP4R+Efg77N5zIEwFPNmYAvw72nzryUoDAA/AV5vZuUZ2vzU3RPuvhnYCbw8n2GBvwGOhRcnSXcxcJ6ZrTWzZWZ2fp6z9Dzmu83sWTO718xGZWiTTV/mlJlVAne4+7pw1jpgaoam/dVni4Bl7n7A3VuB+4HrU/KeBVS7+4pw1g+At+UpS6p64BPu3unuXcBGXtpPC4HPmNk6M/ummVX1Qy4AC3//3syeMbPb0pZfAmxx9+1h4b+X/umzVN8BPuPuTWnz+7PP3gd8mBevBd1rv4Svv6sInofQx+db0Rd2d/+hu3+Jl55LZiLBk56wc44AdSdrE6oHJucpKmYWI9hS//RJmrQT/BMXAP8G/MrMKvKVJ1QP/Aswj2Dr4JsZ2mTTlznl7h3ufi8c/5TzWYItq3T91We9PVf69bnUw93X97yZmNlMgiGZpT3LzWwosAb4JEEfjSTYsusPowg+nf4l8BrgA2Z2dcrygvRZj/DTTrW735c2v1/7zN1vcffUkx1m0y9jgCMpn4T61HdFMxRjZm8jGG9NtcndF51klfTzW0YIxtFTRQk+vp6qTZ/0kvMvCN6Jn820bs/+gdBSM/siwVDRM2eSKYtcPW2+zItj1amy6cu8ZAuL9D0Ez8V/TV83n32WprfnSs6fS30RDik8AHzS3bf0zA/HXK9JaffvBB/zMw0F5lQ4jHF8KCMcnroG+EM4q6B9BryfYBj3BIXss1A2/ZLehgxtTqpoCnv4rnpfrw1ftAcYD+w2szJgGMEOuFS7Cc6A1mM8L34cykfOtwA/Pdm6ZvYRgvHinpwRXtzpdEYy5TKzEWb2t+7eU1QjQKb9C9n0ZU6zhfmGAr8JH+vN4VBDepu89Vma3QSnQO2R/lzJ+XMpW+FO5l8AH3P3n6Ytmwoscve7w1n56p9Mua4AKt39jyd57EL2WQXBGPbNGZYVrM9C2fRLAzDCzGLuHg/bZ913RT8UcwpLgb8Ob7+dYOdf+j9nKfAuM4uZ2QyCnRdP5THTyzj1+eVfAbwXwMxeQXDqzU15zNMCfCrceQPBkQK/zNAum77Mh3uBrcDb3f0lpx4N9VefPQy8xszqzKyGYMf7Qz0Lw6N12sMiC3ATkNedgABmNoVgiOrG9KIeagO+bGbnhDszP0zm/3E+jAS+YmZVZjYMeHfaYz8JmJnNCIcpb6Qf+iw0D9gc7i9JV8g+gyz6JXz9/ZkXD2L46/Q2pzKQC/s/ApeZ2XrgQwT/HMzsTWa2OGxzP7CeYMfcr4H3untbHjNNI3g3Ps7MPmBmnw8nPwpcbWbPEYwXv9Pd8/bRNHynvwH4jpltJDji41Nhrs+b2QfCphn7Mp/M7EKCHeOXA6vDnaNLw2X93mfuvofgo/hyYC3Bp4SVZrbUzBaGzd4FfM3MNgFDCY7iybe/A6qAr4Z9tDbsn6VmttDdGwmGHH5LcEhchJceaJAX7v4/BMNDa4Cngbvd/Ykw40R3byfYYv4FsIHgDfn+k91fjmV6LRa8zwBO1S9mttjM3hQ2/RBwq5ltIPg0mfWhojofu4hIiRnIW+wiIpKBCruISIlRYRcRKTEq7CIiJUaFXUSkxKiwi4iUGBV2EZESo8IuIlJi/j/mMU566a77JQAAAABJRU5ErkJggg==\n",
-      "text/plain": [
-       "<Figure size 432x288 with 1 Axes>"
-      ]
-     },
-     "metadata": {
-      "needs_background": "light"
-     },
-     "output_type": "display_data"
-    }
-   ],
-   "source": [
-    "def sigmoid_fxn(x):\n",
-    "    yhat = list(map(lambda i: 1 / (1 + np.exp(-i)), x))\n",
-    "    return yhat\n",
-    "\n",
-    "x = np.arange(-10., 10., 0.2)\n",
-    "logit = sigmoid_fxn(x)\n",
-    "plt.title(\"The Sigmoid Function Curve\", fontsize=15)\n",
-    "\n",
-    "plt.plot(x, logit)\n",
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Building my Customized Logistic Regression"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class LogisticClassifier():\n",
-    "    def __init__(self, \n",
-    "            learning_rate=0.1, \n",
-    "            tolerance=1e-4, \n",
-    "            max_iter=1000, \n",
-    "            batch_size=32, \n",
-    "            momentum_decay=0.9, \n",
-    "            early_stopping=3, \n",
-    "            validation_set=(None,None)):\n",
-    "        \n",
-    "        # Gradient descent parameters\n",
-    "        self.learning_rate = float(learning_rate)\n",
-    "        self.tolerance = float(tolerance)\n",
-    "        self.max_iter = int(max_iter)\n",
-    "        self.batch_size=32\n",
-    "        self.momentum_decay = float(momentum_decay)\n",
-    "        self.early_stopping = int(early_stopping)\n",
-    "        self.X_validation, self.y_validation = validation_set\n",
-    "        \n",
-    "        # to construct the design matrix\n",
-    "        self.add_intercept = True\n",
-    "        self.center = True \n",
-    "        self.scale = True\n",
-    "        \n",
-    "        self.training_loss_history = []\n",
-    "        \n",
-    "    def __sigmoid(self, X):\n",
-    "        return 1 / (1 + np.exp(-X))\n",
-    "    \n",
-    "    # z-score normalization and intercept addition\n",
-    "    def __design_matrix(self, X):\n",
-    "        if self.center:\n",
-    "            X = X - self.means\n",
-    "        if self.scale:\n",
-    "            X = X / self.standard_error\n",
-    "        if self.add_intercept:\n",
-    "            intercept = np.ones((X.shape[0], 1))\n",
-    "            X = np.hstack([intercept, X])\n",
-    "            \n",
-    "        return X\n",
-    "    \n",
-    "    def __fit_center_scale(self, X):\n",
-    "        self.means = X.mean(axis=0)\n",
-    "        self.standard_error = np.std(X, axis=0)\n",
-    "        \n",
-    "    def fit(self, X, y):\n",
-    "        self.__fit_center_scale(X)\n",
-    "\n",
-    "        n, k = X.shape\n",
-    "        \n",
-    "        # add intercept column to the design matrix\n",
-    "        X = self.__design_matrix(X)\n",
-    "\n",
-    "        # used for the convergence check\n",
-    "        previous_loss = -float('inf')\n",
-    "        self.converged = False\n",
-    "        self.stopped_early = False\n",
-    "        \n",
-    "        # initialize parameters\n",
-    "        self.beta = np.zeros(k + (1 if self.add_intercept else 0))\n",
-    "        momentum = self.beta * 0  # to get the same shape and dtype as beta\n",
-    "        \n",
-    "        for i in range(self.max_iter):\n",
-    "            shuffle = np.random.permutation(len(y))\n",
-    "            X = X[shuffle, :]\n",
-    "            y = y[shuffle]\n",
-    "            \n",
-    "            # we'll add one more batch, incase the batch size doesn't divide n evenly\n",
-    "            extra = (1 if n % self.batch_size else 0)\n",
-    "\n",
-    "            for batch_index in range(n // self.batch_size + extra):\n",
-    "                batch_slice = slice(\n",
-    "                    self.batch_size * batch_index, \n",
-    "                    self.batch_size * (batch_index + 1) )\n",
-    "                X_batch = X[batch_slice, :]\n",
-    "                y_batch = y[batch_slice]\n",
-    "            \n",
-    "                beta_ahead = self.beta + self.momentum_decay * momentum\n",
-    "                y_hat = self.__sigmoid(np.dot(X_batch, self.beta))\n",
-    "                \n",
-    "                # gradient descent\n",
-    "                residuals = (y_hat - y_batch).reshape( (X_batch.shape[0], 1) )\n",
-    "                gradient = (X_batch * residuals).mean(axis=0)\n",
-    "                momentum = self.momentum_decay * momentum - self.learning_rate * gradient\n",
-    "                self.beta += momentum\n",
-    "\n",
-    "            # with minibatch, we only check convergence at the end of every epoch.    \n",
-    "            y_hat = self.__sigmoid(np.dot(X, self.beta))\n",
-    "            self.loss = np.mean(-y * np.log(y_hat) - (1-y) * np.log(1-y_hat))\n",
-    "            self.training_loss_history.append(self.loss)\n",
-    "            \n",
-    "            # early stopping\n",
-    "            if self.check_validation_loss():\n",
-    "                self.stopped_early = True\n",
-    "                break \n",
-    "                \n",
-    "            if abs(previous_loss - self.loss) < self.tolerance:\n",
-    "                self.converged = True\n",
-    "                break\n",
-    "            else:\n",
-    "                previous_loss = self.loss\n",
-    "        \n",
-    "        self.iterations = i+1\n",
-    "        \n",
-    "    def predict_proba(self, X):\n",
-    "        # add intercept column to the design matrix\n",
-    "        X = self.__design_matrix(X)\n",
-    "        return self.__sigmoid(np.dot(X, self.beta))\n",
-    "\n",
-    "    def predict(self, X):\n",
-    "        predictions = self.predict_proba(X).round()\n",
-    "        return predictions\n",
-    "\n",
-    "    def check_validation_loss(self):\n",
-    "        # validation set loss\n",
-    "        if not hasattr(self, 'validation_loss_history'):\n",
-    "            self.validation_loss_history = []\n",
-    "        p_hat = self.predict_proba(self.X_validation)\n",
-    "        loss = np.mean(-self.y_validation * np.log(p_hat) - \\\n",
-    "               (1-self.y_validation) * np.log(1-p_hat))\n",
-    "        self.validation_loss_history.append(loss)\n",
-    "        \n",
-    "        t = self.early_stopping\n",
-    "        if t and len(self.validation_loss_history) > t * 2:\n",
-    "            recent_best = min(self.validation_loss_history[-t:])\n",
-    "            previous_best = min(self.validation_loss_history[:-t])\n",
-    "            if recent_best > previous_best:\n",
-    "                return True\n",
-    "        return False"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Testing and Evaluating the Customized Model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from sklearn.datasets import load_breast_cancer\n",
-    "from sklearn.model_selection import train_test_split as tts\n",
-    "\n",
-    "dummy_data = load_breast_cancer()\n",
-    "X, y = dummy_data.data, dummy_data.target\n",
-    "\n",
-    "X_train, X_test, y_train, y_test = tts(X, y, test_size=0.2, random_state=103)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = LogisticClassifier(learning_rate = 0.01,\n",
-    "                           tolerance=1e-5,\n",
-    "                           max_iter=2000,\n",
-    "                           early_stopping=3,\n",
-    "                           validation_set=(X_test, y_test))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model.fit(X_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "y_pred = model.predict(X_test)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# defining the metrics function\n",
-    "def accuracy(predictions, actual):\n",
-    "    return sum(predictions == actual) / len(actual)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 23,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "0.9736842105263158"
-      ]
-     },
-     "execution_count": 23,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "accuracy(y_pred, y_test)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

From 4a6e86bbad037966916a8452ad2a9abd8058c84d Mon Sep 17 00:00:00 2001
From: Steven Kolawole <45284829+SteveKola@users.noreply.github.com>
Date: Wed, 22 Apr 2020 22:35:34 +0100
Subject: [PATCH 5/9] Add files via upload
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

A Customized Logistic Regression with Nesterov Accelerated Gradient with Early Stopping option:


Nesterov Accelerated Gradient is theorized to converge by at least, a 10 times faster rate than Stochastic Gradient Descent, and over 25 times faster rate than the naive batch gradient descent.

Nesterov Gradient combines the properties of Stochastic Gradient Descent -which supposedly have the properties that allows it to “jump” out of shallow local minima giving it a better chance of finding a true global minimum- with a 'smarter' momentum, that has a somewhat prescient notion of the global minimum, and knows to slow down before the hill slopes up again.

But there is a catch;
Converging too fast makes it easier for the model to overfit, causing the well-known bias-variance tradeoff.
My way of avoiding that is to introduce Early Stopping, which works by simply waiting for a certain number of epochs with no improvement in validation loss.
---
 Logistic Regression from Scratch.ipynb | 322 +++++++++++++++++++++++++
 1 file changed, 322 insertions(+)
 create mode 100644 Logistic Regression from Scratch.ipynb

diff --git a/Logistic Regression from Scratch.ipynb b/Logistic Regression from Scratch.ipynb
new file mode 100644
index 0000000..6031d92
--- /dev/null
+++ b/Logistic Regression from Scratch.ipynb	
@@ -0,0 +1,322 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Logistic Regression from Scratch\n",
+    "## - Steven Kolawole                                "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Intro\n",
+    "\n",
+    "Logistic Regression is simply a Linear Regression with a Sigmoid function at its end.\n",
+    "\n",
+    "The Sigmoid function generates probability (i.e. outputs between 0 and 1) for all values of X."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "sns.set()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEMCAYAAADQ553CAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZhcZZn38W9V9Z49nc6eANlukkAIISzK4hZ0BLcRRMVhZBRxw1dHR8cRZkYdGX11RhnXcYy8ohgXcNyGgIqJgywhhCQEstxZCFk76SVrd3qtqvePczpUikq6Oqnqqq7+fa6rr65zznOqfv101V2nnnPqnEgymUREREpHtNABREQkt1TYRURKjAq7iEiJUWEXESkxKuwiIiVGhV1EpMSUFTqAnJqZ/QB49yma7HD3s83sBWCxu38hx4//18BtwFwgAawDvu7uP0tpkwRucvd7c/nYfch4M8HfnvH5bGaTgV3Aq9z9TxmWvxJYfpK7b3b3MblJempm9jIg6u6PhdN571cziwC3AO8B5gDdwFrgi+7+cL4eV/JLW+zF76PAhPDnknDem1PmXZyvBzazW4FvAt8GLgAuBR4AfmJmqW82E4D785UjCz8DJuXgfhbwYr/2/MzJwf1m6xFgZsp0XvvVzKLAr4A7gXsInl9XAU8DvzOzG/P12JJf2mIvcu5+GDgMYGZV4ewD7r6vHx7+/cD33P0HKfM2mJkRvOHcE2bsjywn5e5tQFsO7qqxwH9LJHWiH7LcBlwDXOTu61Lmf8rMhgJfN7Nfu3trnnNIjqmwl5ZJZvYbYBHBm8E33P1fexaa2VuAzwEGvAAsBr7q7omT3F8cuNzMRoRvMD3+DhiScr8nDBmY2SeBjwBjgN8BO4EL3P2V4bDHQ8BbgK8DU4DHgJuBTwM3Ae3A19z9SymP8R7g48B0YA9wl7t/M1x2MylDMWZ2FsGnjFcADcDxPjhdmYZ7MjxuEnhv+LdcDOwg6N//SlnnJuBTwIxw+Rfd/Z5wKC0G/D8zuznsq/R+7a0PPg18FfgMUAs8BXzY3Tee5M96P/DrtKLe43PADwj+FxmHhVLnhUOG1cBYgk8+nyb4H0xz9+0p66wHfunud5jZFOBrwGsJ3piXAx93970nyStZ0lBMaXkvsJRgPPw/gDvN7CoAM7sG+HE4fy5Bcfko8I+nuL+vEHw832tmvzGzvzOz+e7e6O4vZFrBzD4C/BNBcbmQ4A3kI2nNKoDPAzcCryYoBOuAFoKC+F3gi2Y2J7zPjxMMCd0FzAtzfcXMPpHh8csJ3jhqgMvDPvn0Kf7GXPu/BFkvBP4MfCd8o8HM3g7cTfCGej7wb8BiM3stwd8dBz4GvDX9TrPsg2nAu4DrCIrlWcA3MoUMP/3NAVZkWu7u+919pbvH+/C33wD8kmDI7lcEb1zvSHnM+eFj/tDMhgB/IijoLwdeR/C8WGZmFX14TMlAW+yl5T53/8/w9pfM7NPAQoKx288A33b3u8Pl28xsGPA9M/uXTFvt7n6fme0hKDavBd4IYGZrCLbU1mfI8AmCre2eLbu/NbMr0tpEgNvdfVV4f38kKGz/4O5JM/siwRvOXDPbSPAmdJe7Lw7X32Jm04C/N7Ovpt33IoJPJK9z953h/f8fgn0DvfFwKzTVPHd/Pot1e9zt7j8PH/eTBDsmLyEoch8Dfuzu/xG23RoOeUTdvTEY4eKwux9IvcNwB2c2fVAOfKBnC93MvkEwfp7JqPD3oT78bb3Z5+5fT8n9I+CdwBfDWe8CVrj7ZjO7heBT3809bx5m9k6gieCN6Sc5zDXoqLCXls1p04cIPh5DsAV5sZl9MGV5NFx+NpCxeLn748DjZhYDLiIo7h8BHjSzGe7e2dPWzGoJthKfSLubx4D5afO2ptxuBZ5392T4mG1hkasE6oBx4X2keoSg2I1Nm38e0NRT1EMZt0ozeB2QPq69K8t1exz/H7j7ofDv6NkCPR/4UWpjd78ri/vMtg+SwJaU5YdSHjtdc9h+dBaPn63059A9wB1mNhfYSLD13vNGcyHB33U47KMeNcDsHGYalFTYS0umj809O+Q6gS8TDMek250+Ixz//Afg8+6+L9yqWgmsNLM/E4ydzwNWpazWHf7OZoivK236ZOP87SeZHzvJ/SRJ2wlJ8Ldn4wV3f0lfnEKm109Hhnk9edKzZivbPki4e3dam/S+AMDdO81sNXBZpuVmNhP4FvC3mT6ZmVmmv/2EHdjuvtXMHifYan+YoJD3HCbbCawnw7ATuf0UMShpjH3wWA/MdPetPT8EW5B3kvnF30YwjJDpkLdDBAW0IXVmuIN1J8EYa6r06ay5+xGCN5704ZwrCLauD6bNXwuMCQtTj4Wn+/gpOoGYmdWkzJt5ssYnsTE9i5n90Mx6hi8ynkP7NPogW98H3mhm8zIs+yTBENIL4XQXMDxlebZ/+z0EQys3AL91956s64FzCL4n0PN8bCDY+Xt+X/4IeSltsQ8eXwAeMLPngF8Aswh2Ui5195dsZbp7k5l9mWAn5vBwnTaCF90XgHvShjt69KyziWBr/maCrcI/nWH2r5nZtvB+XkUwHPRP4Zh8atvlBMdh32tmHyYYivg6Z24FQeH9nJl9i+DN6uY+3seXgZ+b2UrgDwQ7jt8JvD5cfhSYY2Zj3b0hbd2+9EG2/ovg6KQ/mtlnCPpuOHArwU7nd6Yc6vgEcKuZPUbwSeFrZP50ku5nBDvs/4pgjL3Hj4HbCfrjHwg+lXyJ4M0k074b6QNtsQ8S7v4QwaGENwLPERT1HxIc8nayde4Il18NPErwguv5MsutJ1nt28C/hz/PELyB/Irsh0My5fguwc7ffwgzfJzgsLivZGgbJzg2eydBobqPoAidkXAH6geBtwGbgPcRbNX25T5+BXyYIP96gp2pN6V8w/NLwIcIhrnS1826D/qQJw68geAN5zZgDfB7gqNrXt2zEzj0QYJDaJ8k+NLUf5FhCC/DYxwGfk1QuB9Mmd9G8Lw6Biwj2H9QFj5u+pua9FFEV1CSXDKzvwCedfc9KfMeAva4+3sLl0xk8NBQjOTau4FzwmGQZoItwkUEh0uKSD9QYZdcu43gSzQPAsMIhi3e6e7LCppKZBDRUIyISInRzlMRkRJT6KGYSoKvkteT+cs1IiLyUjGC0zo/RYbDTgtd2C8mOFGSiIj03ZUEhyKfoNCFvR7g4MFWEom+j/XX1g6lubkl56FyoVizFWsuKN5sxZoLijebcvVdX7JFoxFGjRoCYQ1NV+jCHgdIJJKnVdh71i1WxZqtWHNB8WYr1lxQvNmUq+9OI1vGIWztPBURKTEq7CIiJUaFXUSkxGQ9xh6e4e9x4A3pl0ULL3m1mODMcI8QXMUl/bzQIiLSD7LaYjezSwkOqZl1kib3Are5+yyCc3u/LzfxRESkr7IdinkfwelGX3L18PBCvdXu3nP5sR8QnNpURGTQSyaTx38SaT/5ktVQjLvfAnCSk/lP5MRjKeuByWecTERKRld3gmPtXRzr6Ka9M057RzftXXE6OuN0dMXp7ErQ2R2nqzsR/MQTdIe/4/Ek3fEE3fEk8USSeCKYF08Gh0nHw8OlE8me35AMp4NiSngbopEI3fEEPefICo4uDJaFN4MiDCR7Zhy/zfHbyZ6LXaXO72OfRCLwwTefx8Jz0y/be+ZycRx7lBP/pggnv35lRrW1Q0/7wevqhp32uvlWrNmKNRcUb7ZizQWFy9bVnaD5cBuNh9poPNjGgSPtHDzazqEjHRxq6eBIaydHj3VytLWTzu7sS0JZLEpleZTyshhlZVHKy6KUxaKUxSLh7yhl5VGqYhFi0SjRaIRYNEI0GiEaSf3N8duRSIRIJCjshL8jEYL5ABGIcOK8SHjByEh4I5JyAcnj64XrvngzktImdX6KcEE0CpfMm0jtiOrji3L1v8xFYd9NcM6CHuPJMGRzKs3NLaf1pYG6umE0Nh7t83r9oVizFWsuKN5sxZoL+idbS1sXexpb2NPUyp6mVvYfOEbDwTaaj7STPppQWRFjxJAKRg+vYuSQCibXDWFoVTk1VWXUVJVRXVlGdUUZVRUxKitiwe/yGBXlMcrDIh6NZLz+dk4U2/8y0dl9PE9fskWjkVNuEJ9xYXf3HWbWbmaXu/tjBJdfe7C39USk+MQTCXbub8F3HmJ7/RG21x+h6XD78eXVlWWMH13DjEkjeNnc8YwZWcXo4VWMHlbJqGGVVFUEJaXYCuhgc9qF3cyWElxIdxXBRWq/Fx4SuZrcXDxYRPrBgSPtPLO1iXXbmtm8+xBtHcG31GuHV3HOhGG86sJJTBk7lEl1Qxk5tOL40IQUrz4Vdnc/O+X2NSm3nyG4uriIDAAHj3bw5Ib9rNiwj537gxNPjR1ZzaVzxnPu1JHMmjKSkUMrC5xSTlehTwImIv0kkUjyzNYmlq/Zw/oXDpBMwjkThvO2V07nghljmFBbo63xEqHCLlLiOrriPLqunj88tYuGQ22MHl7JtS87m5efN57xo2sKHU/yQIVdpER1xxP8eV09v3lsO4dbOpk+cThvfcU0LrI6YlGdJqqUqbCLlKC1W5r46bItNBxsY8bkEXzgTXOxqaMKHUv6iQq7SAk51NLBkj9sZpU3MmnMED56/TzmTa/V2Pkgo8IuUiKeeG4f9/5hM13dCd561TT+4tKplMU05DIYqbCLDHCdXXGWPLyZR56pZ+bkEfzNNbO1U3SQU2EXGcD2NrVw54+eZldDC9e+7CzecuU52jEqKuwiA9Xze4/w9V+sIx5P8LG3zWPe9DGFjiRFQoVdZABat62Zb//qWUYPr+Kj189j3CgNvciLVNhFBpgn1u/j7gc2MqluCF/4wOV0d3QVOpIUGQ3GiQwgT3sDi/9nAzMnj+Dvb1zAqOFVhY4kRUhb7CIDxHPbm/nPX69n+sQRfPT6C6isiBU6khQpbbGLDABbdx/mm//9LBPHDOFjb5unoi6npMIuUuSaDrXx9V+sY9TQSj7+9vnUVJUXOpIUORV2kSLW0RXnm//9LPFEko+97QJGDKkodCQZAFTYRYpUMpnkngc3sauhhfe/aQ7j9G1SyZIKu0iR+sNTu1ixYT9vuWqavnwkfaLCLlKEduw7yn1/2saCWXW84WVnFTqODDAq7CJFpqs7zuL/2cDQmnJufv25OuWu9JkKu0iR+eUj29nT1Mp7rpnN0GodASN9p8IuUkR850F+t3Inr7xwEudPqy10HBmgVNhFikRnV5y7l26kbmQ1N7xqeqHjyACmwi5SJJau2EHjoXbe/fpzqarQ2T7k9KmwixSBhkNtLF2xk0tmj2X2WbrotJwZFXaRIvDTh7cQi0Z4+6tnFjqKlAAVdpECW7etibVbm3jT5WczalhloeNICVBhFymg7niCJQ9vYfzoGq6+eEqh40iJUGEXKaA/r6un4WAb73jNDMpiejlKbuiZJFIgnV1xfvvYdmZMHqFj1iWnVNhFCmTZ6j0caunkuqum6bQBklNZHSxrZjcCdwDlwF3u/q205QuA7wIVwC7gr9z9UI6zipSMto5ulq7YwdxzRmNTdXij5FavW+xmNgm4E7gCmA/camZz0pr9B/BP7n4B4MDf5TqoSCn5w1O7aGnr4q1XTSt0FClB2QzFLAKWufsBd28F7geuT2sTA4aHt2uAttxFFCktre1d/O6pnSyYVcc5E4b3voJIH2UzFDMRqE+ZrgcuSWvzceD3ZnYX0Apc2pcQtbVD+9L8BHV1w0573Xwr1mzFmguKN1sucy1/eDNtHXFufuPcnNzvYOizXCrWXJC7bNkU9iiQTJmOAImeCTOrBr4PLHL3lWb2ceCHwLXZhmhubiGRSPbeME1d3TAaG4/2eb3+UKzZijUXFG+2XObq6o7z6//dynnTRjO0PHrG9zsY+iyXijUX9C1bNBo55QZxNkMxu4EJKdPjgb0p0+cBbe6+Mpz+LvDKrNKJDDKPPbuPI8e6eP2luiqS5E82hf1h4DVmVmdmNcB1wEMpy7cCU8zMwuk3A0/lNqbIwJdIJHlo5U7OHj+Mc6eOLHQcKWG9FnZ33wPcDiwH1gJLwiGXpWa20N0PAjcDPzezdcB7gL/JY2aRAWn15kYaDrZxzWVn6bh1yausjmN39yXAkrR516TcfhB4MLfRREpHMpnkwSd3MHZUNQtm1RU6jpQ4ffNUpB9s2X2Y7fVHed0lU4lGtbUu+aXCLtIPlq/ZQ3VlGS+fO77QUWQQUGEXybPDrZ2s2tTAFedPoLIiVug4MgiosIvk2SPP7CWeSPLKCycWOooMEirsInkUTyT437V7mHP2KCbUDil0HBkkVNhF8mjd1mYOHOngVRdOLnQUGURU2EXyaNmaPYwaVsn8mbqQhvQfFXaRPNl/8Bjrtx/glfMnEovqpSb9R882kTx5dF09kQhcMU87TaV/qbCL5EEikeTx5/Zx/rRaRg2rLHQcGWRU2EXyYP0LBzh4tIMrzp/Qe2ORHFNhF8mDR9fVM6SqjAtmjCl0FBmEVNhFcqy1vYs1Wxq5bO54ysv0EpP+p2edSI49uWE/3fGkhmGkYFTYRXLs0XX1TBk7lLPGF++1NaW0qbCL5NDuxhZe2HeUy7W1LgWkwi6SQ0+s30c0EuGyOeMKHUUGMRV2kRxJJJOs3NDA3HNGM3xIRaHjyCCmwi6SI9v2HKb5SLu21qXgVNhFcmTFhv1UlEWZP1PHrkthqbCL5EB3PMFTGxuYP3MM1ZVZXSNeJG9U2EVyYOOOg7S0dXHpbA3DSOGpsIvkwIr1+6mpLOO8aTrvuhSeCrvIGeroirN6SyMLz63TKQSkKOhZKHKG1m1rpqMzzqVzxhc6igigwi5yxp7auJ/hQyqwKSMLHUUEUGEXOSMdnXHWbWvmIqsjGo0UOo4IoMIuckbWPd9MZ3eCi21soaOIHKfCLnIGVm1qYHhNObM0DCNFRIVd5DR1dMV5ZlsTC2yshmGkqGT1FTkzuxG4AygH7nL3b6UtN+C7wChgH/AOdz+Y46wiReXZbc10diW42OoKHUXkBL1usZvZJOBO4ApgPnCrmc1JWR4BfgN8yd0vANYAn85PXJHiscobGFZTzqypGoaR4pLNUMwiYJm7H3D3VuB+4PqU5QuAVnd/KJz+V+BbiJSwzq44z2xtZsGsOmJRjWhKcclmKGYiUJ8yXQ9ckjI9A9hnZt8HLgQ2Ah/JWUKRIvTs8wfo6Iqz8FwdDSPFJ5vCHgWSKdMRIJF2H68ErnL3VWb2L8BXgZuzDVFbOzTbpi9RV1e815Us1mzFmguKN1t6rvW/38ywmgquXDCFWKywW+wDpc+KRbHmgtxly6aw7wauTJkeD+xNmd4HbHH3VeH0TwiGa7LW3NxCIpHsvWGaurphNDYe7fN6/aFYsxVrLijebOm5uuMJnly/j4tm1XHgQGsBkw2cPisWxZoL+pYtGo2ccoM4m02Nh4HXmFmdmdUA1wEPpSx/HKgzswvC6TcCT2eVTmQA2rTjIG0d3SzQ0TBSpHot7O6+B7gdWA6sBZa4+0ozW2pmC929DfhL4Htmth54NfCJfIYWKaTVmxuprIgx9+xRhY4iklFWx7G7+xJgSdq8a1JuP8mJO1RFSlIikWT1libmTaulvCxW6DgiGek4LZE+2LrnMEdaO1kwS8MwUrxU2EX6YPXmRspiEeZN15WSpHipsItkKZlMsnpzI3POHq0LVktRU2EXydKuhhaaDrdrGEaKngq7SJZWb24kEoH5M8cUOorIKamwi2Rp9eYmZk4eyfCaikJHETklFXaRLDQcamN3YwsLtLUuA4AKu0gW1m5uBGC+xtdlAFBhF8nC6i1NTK4bytiR1YWOItIrFXaRXhxu6WDL7kMsmKVhGBkYVNhFevHUhn0kk3DhTA3DyMCgwi7SixXP7aN2eCVTx53+dQNE+pMKu8gpdHTFWbO5kQtn1hGJRAodRyQrKuwip7B++wE6u+JcqKNhZABRYRc5hTWbGxlaXc6sKSMKHUUkayrsIicRTyRYu7WJi+eMIxbVS0UGDj1bRU5iy67DtLZ3c9l5EwodRaRPVNhFTmL1lkbKy6IssLGFjiLSJyrsIhkkk0nWbG5i7tmjqdK512WAUWEXyWBXQwvNR9q5UCf9kgFIhV0kgzVbmohE4AIVdhmAVNhFMlizuZEZk0bo3OsyIKmwi6RpOtTGzoYWnRtGBiwVdpE0a7Y0AXChzuYoA5QKu0ia1ZsbmTRmCONG1RQ6ishpUWEXSXHkWCebdx9igc4NIwOYCrtIirVbmkgmUWGXAU2FXSTF6s2NjBlRpXOvy4Cmwi4SauvoZsMLB1gwS+del4FNhV0k9OzzzXTHkxqGkQFPhV0k9LQ3MrymnBmTdO51GdiyKuxmdqOZbTCzLWb24VO0u9bMtucunkj/6OqOs+75Zi6cVUc0qmEYGdh6LexmNgm4E7gCmA/camZzMrQbB/wboFeFDDjrXzhIR2dcwzBSErLZYl8ELHP3A+7eCtwPXJ+h3WLgc7kMJ9JfVnsj1ZUxZp81qtBRRM5YNieangjUp0zXA5ekNjCz/wOsBlacToja2tM/tKyubthpr5tvxZqtWHNBYbJ1xxM8s62JS+ZMYML4zOPr6rO+U66+y1W2bAp7FEimTEeARM+EmZ0HXAe8Bph8OiGam1tIJJK9N0xTVzeMxsajp/OQeVes2Yo1FxQu23Pbmzl6rIvzzxmV8fHVZ32nXH3Xl2zRaOSUG8TZDMXsBlIv+jge2Jsy/bZw+SpgKTDRzP6cVTqRIrBqUwOVFTHOO2d0oaOI5EQ2W+wPA581szqglWDr/Naehe7+z8A/A5jZ2cCf3P3K3EcVyb3ueILVm5uYP2MMFeWxQscRyYlet9jdfQ9wO7AcWAsscfeVZrbUzBbmO6BIPvmuQ7S0dbHQdDSMlI6srtLr7kuAJWnzrsnQ7gXg7FwEE+kPqzY1UFke4/xptYWOIpIz+uapDFrxRIKnvZELZtRqGEZKigq7DFq+s2cYZmyho4jklAq7DFqrvJGK8ijnT9cwjJQWFXYZlIJhmAbmTR9DpYZhpMSosMugtPGFgxw91sVlc8YVOopIzqmwy6C0YsN+qivLdDSMlCQVdhl0OrviPL25kYVWR3mZXgJSevSslkHnmW3NdHTGNQwjJUuFXQadJzfsZ8TQCmyqTtErpUmFXQaVY+1drNvWxKWzx+lKSVKyVNhlUHnaG+mOJ7lUwzBSwlTYZVBZsWE/Y0dVc/b44r3YgsiZUmGXQePAkXY27TjIZXPGEYloGEZKlwq7DBqPPbePJHD5+RN6bSsykKmwy6CQTCZ57Nl6zp06krqR1YWOI5JXKuwyKGzZfZiGg23aWpdBQYVdBoVH19VTVRHTKXplUFBhl5LX3tnNU5sauPjcsVRW6EyOUvpU2KXkrdrUSEdXnCvmaRhGBgcVdil5jz5bz7jRNcyYNKLQUUT6hQq7lLS9Ta1s3nWIK84fr2PXZdBQYZeS9qc1e4hFI1w5b2Kho4j0GxV2KVkdnXEee66ei88dy/AhFYWOI9JvVNilZK3YsI+2jjivWjCp0FFE+pUKu5SkZDLJstV7mFw3VDtNZdBRYZeStG3PEXY1tPDqiyZpp6kMOirsUpKWrdlNdWVMl7+TQUmFXUrO4ZYOVm1q4OVzJ1BVUVboOCL9ToVdSs7DT+8mHk+y6OLJhY4iUhAq7FJS2jq6WbZ6DxdZHeNG1RQ6jkhBZPU51cxuBO4AyoG73P1bacvfDHwOiADbgb9x94M5zirSq/9du5e2jm5ef9lZhY4iUjC9brGb2STgTuAKYD5wq5nNSVk+HPgOcK27XwCsAz6bl7Qip9AdT/CHVbs4d+pIzpkwvNBxRAomm6GYRcAydz/g7q3A/cD1KcvLgQ+7+55weh0wNbcxRXq3Yv1+Dh7t0Na6DHrZDMVMBOpTpuuBS3om3L0Z+CWAmVUDnwa+kcOMIr1KJJM8tHInk+uGct45owsdR6SgsinsUSCZMh0BEumNzGwEQYF/xt3v6UuI2tqhfWl+grq6Yae9br4Va7ZizQWnn+2xZ/ayt6mVT9y4gLFjcz8MU4p9lm/K1Xe5ypZNYd8NXJkyPR7Ym9rAzCYAvwOWAX/b1xDNzS0kEsneG6apqxtGY+PRPq/XH4o1W7HmgtPPlkgkueeB9UyorWH25BE5//tKsc/yTbn6ri/ZotHIKTeIsynsDwOfNbM6oBW4Dri1Z6GZxYDfAj939y9klUokh55Yv4/65mN86C3nEY3q9AEivRZ2d99jZrcDy4EKYLG7rzSzpcA/AVOABUCZmfXsVF3l7rfkK7RIj+54gl8/up2zxg3jIqsrdByRopDVcezuvgRYkjbvmvDmKvRFJymQR57ZS9Phdm56nelkXyIhFWQZsDq64vz28ReYOXmEjoQRSaHCLgPWgyt2cLilk+teMV1b6yIpVNhlQGo41MbSFTu5ZPZYZk0ZWeg4IkVFhV0GpJ8+vIVYNMLbXz2z0FFEio4Kuww467Y1sXZrE2+6/GxGDassdByRoqPCLgNKV3eCJQ9vYfzoGq6+eEqh44gUJRV2GVB+/eh2Gg62cePVMymL6ekrkoleGTJgbN19mAef3MGV8yZw3jm1hY4jUrRU2GVAaO/sZvH/bKB2eBXveI12mIqcigq7DAj3Ld9G46E23nvtbKordYFqkVNRYZeit3ZrE8vX7OHqi6dgU0cVOo5I0VNhl6K2/8AxvvfbDUwdO5TrXjGt0HFEBgQVdilabR3dfOO/nyUWjXDbW8+nvCxW6EgiA4IKuxSlRDLJ9x/YyL7mY3zwzXMZM7K60JFEBgwVdilKv3zkeVZvbuSGV01n9tk6c6NIX6iwS9F5cMUOHnhiB6+YP1HfLhU5DSrsUlT+tGYP9/1pG5fMHstNr9XFM0ROhwq7FI0/PrWTH/3OmTe9llveMEfXLxU5TfqmhxRcMpnkoZU7uW/5NmafNYoPveU8nQdG5AyosEtBJZJJfr5sK79/ahdXzp/EXy2aSXmZirrImVBhl4I51t7N3Us3snpzI4sumsxH3rGA5uaWQscSGfBU2KUgduw7yrd/9SwHjnTwjlfP4OqLp2hMXSRHVNilXyUSSZat3s3Pl342r5gAAAtySURBVG9jWE05f3/jAmZMHlHoWCIlRYVd+s3uxhbueXAT2/YeYd70Wt577WyG1VQUOpZIyVFhl7w71t7FAyt28PuVu6iuLON9b5jDZXPH6Rh1kTxRYZe86eiKs+zp3SxdsYPW9m4uP288N7x6hrbSRfJMhV1y7nBrJ8tX72b5mj0cPdbFvOm1vPWqaUwdN6zQ0UQGBRV2yYlEMonvOMijz+7jqU376Y4nuWB6La+/7CxmTRlZ6Hgig4oKu5y2RCLJ83uPsGZLI09u3M+BIx1UV8a48oKJXL1wCuNH1xQ6osigpMIufdJ0uA3feYiNOw7y7PPNHD3WRSwaYc7Zo7nhVTOYP2MMFeW6IIZIIamwy0kdPdbJ7sZWXth3hO31R9m+9wjNR9oBGFJVxnnTapk/YwznTxtNTVV5gdOKSI+sCruZ3QjcAZQDd7n7t9KWzwcWA8OBR4APuHt3jrNKjiWTSVraujhwpIPmI+00HGyj4eAx9h9sY09TK0daO4+3rR1exTkThvHaS6Zw7tRRTKobQlSHK4oUpV4Lu5lNAu4ELgI6gMfNbLm7b0hpdi9wi7uvMLPvA+8DvpOPwPJSyWSS7niC9s44bZ1x2ju6ae+M09rexbH2blrbu2lp66KlrYuO7gSNB45xuLWDwy2ddHYnTrivIVVljB1Vw7xptUwcM4TJdUOYOn4Yw3WIosiAkc0W+yJgmbsfADCz+4Hrgc+H02cB1e6+Imz/A+Bz5LmwJ5JJVm3cz/7Go6dsl0ym3OaEiRd/ZWhzwnrJcH7yxabJZDKcHzRIJE+cXzOkkqNH20kSTCcSSRLJE28nksngdgLiiQSJRJLuRJJ4PEk8kSAe3u6KJ+juTtAdT9AVT9DVnaCzK0FXd5yO7gSdXfET8mYSicCw6nJGDKtiaFUZ0yeOYPiQCmqHVzF6eCWjh1cxdlQ1QzSkIjLgZVPYJwL1KdP1wCW9LJ/clxC1tUP70hyArbsO8bnFK3pvWKSi0QjRSIRoNEJZLEIsGiEWjRKLRYjFopRFI5SVRSmLRSmPRSkrj1FdXU55WZSKshgV5VEqymNUVsSoqiijKvxdU1VGdWXwe2h1BUOqyxlaU86QqvIBcZKturriPNa9WHNB8WZTrr7LVbZsCnuUE7ZpiQCJPizvVXNzC4lEL5ucaUZUxVh8+9XU7z9ywgNncrKh4J6vtEfSVo682OD4dE+bCJHj9xeJBLcjPfcV3o5GI9SNGUZzcwvRnjaRCNFo+Ls/x6aTCdpbO2hv7QCCJ05jL59yCqVYsxVrLijebMrVd33JFo1GTrlBnE1h3w1cmTI9HtibtnzCKZbnzbjRNUTj8f54qD4bUl3OsUoddCQi/S+bS9U8DLzGzOrMrAa4DnioZ6G77wDazezycNZNwIM5TyoiIlnptbC7+x7gdmA5sBZY4u4rzWypmS0Mm70L+JqZbQKGAl/PV2ARETm1rMYK3H0JsCRt3jUpt5/hxB2qIiJSILpqsIhIiVFhFxEpMSrsIiIlptDH48WAM/riTDF/6aZYsxVrLijebMWaC4o3m3L1XbbZUtplPJVqJNnbd9Hz6wrgz4UMICIygF0JPJo+s9CFvRK4mOA0BMX5TSMRkeITI/hi6FMEJ2c8QaELu4iI5Jh2noqIlBgVdhGREqPCLiJSYlTYRURKjAq7iEiJUWEXESkxKuwiIiWm0KcUyJqZ/QsQd/fPhtMjgR8D04BG4AZ335e2TgT4CvAGgsv1vc/dH8tDtrHA71NmjQDq3H1oWruzgOeAbeGs/e7+ulznSXvMdwNfAvaHsx5w99vT2vTal3nKdjnwNaACaAbeE164JbVNv/WZmd0I3AGUA3e5+7fSls8HFgPDgUeAD7h7dz6ypD3uPwM3hJMPuPunMix/D3AwnPW99Ox5zLYcGAt0hbPe7+5PpixfBHwVqAZ+5u539EOmW4DbUmadA/zI3W9LadOvfWZmw4HHgTe4+wvZ9IuZTQXuJehfB97l7i3ZPF7RF3YzG0HQAe8Evpyy6AvAn939WjO7CfgP4O1pq18HzAbmADOAB8xsdq5fjO7eAMwP80aBPxJcnCTdQoILlbw/l4/fi4XAx939J6dok01f5sOPgTe5+zozew/BBVrenNamX/rMzCYBdwIXEXyT73EzW+7uG1Ka3Qvc4u4rzOz7wPuA7+Q51yLgtcCFBNcWfsjM/tLdf5nSbCHwDnd/Ip9ZMmSLALOAszK9psysGrgbeAWwi+D193p3z+sV1tx9McEbMGY2F/gV8Nm0Zv3WZ2Z2KfA9gr7qS798G/i2u//UzP4R+Efg77N5zIEwFPNmYAvw72nzryUoDAA/AV5vZuUZ2vzU3RPuvhnYCbw8n2GBvwGOhRcnSXcxcJ6ZrTWzZWZ2fp6z9Dzmu83sWTO718xGZWiTTV/mlJlVAne4+7pw1jpgaoam/dVni4Bl7n7A3VuB+4HrU/KeBVS7+4pw1g+At+UpS6p64BPu3unuXcBGXtpPC4HPmNk6M/ummVX1Qy4AC3//3syeMbPb0pZfAmxx9+1h4b+X/umzVN8BPuPuTWnz+7PP3gd8mBevBd1rv4Svv6sInofQx+db0Rd2d/+hu3+Jl55LZiLBk56wc44AdSdrE6oHJucpKmYWI9hS//RJmrQT/BMXAP8G/MrMKvKVJ1QP/Aswj2Dr4JsZ2mTTlznl7h3ufi8c/5TzWYItq3T91We9PVf69bnUw93X97yZmNlMgiGZpT3LzWwosAb4JEEfjSTYsusPowg+nf4l8BrgA2Z2dcrygvRZj/DTTrW735c2v1/7zN1vcffUkx1m0y9jgCMpn4T61HdFMxRjZm8jGG9NtcndF51klfTzW0YIxtFTRQk+vp6qTZ/0kvMvCN6Jn820bs/+gdBSM/siwVDRM2eSKYtcPW2+zItj1amy6cu8ZAuL9D0Ez8V/TV83n32WprfnSs6fS30RDik8AHzS3bf0zA/HXK9JaffvBB/zMw0F5lQ4jHF8KCMcnroG+EM4q6B9BryfYBj3BIXss1A2/ZLehgxtTqpoCnv4rnpfrw1ftAcYD+w2szJgGMEOuFS7Cc6A1mM8L34cykfOtwA/Pdm6ZvYRgvHinpwRXtzpdEYy5TKzEWb2t+7eU1QjQKb9C9n0ZU6zhfmGAr8JH+vN4VBDepu89Vma3QSnQO2R/lzJ+XMpW+FO5l8AH3P3n6Ytmwoscve7w1n56p9Mua4AKt39jyd57EL2WQXBGPbNGZYVrM9C2fRLAzDCzGLuHg/bZ913RT8UcwpLgb8Ob7+dYOdf+j9nKfAuM4uZ2QyCnRdP5THTyzj1+eVfAbwXwMxeQXDqzU15zNMCfCrceQPBkQK/zNAum77Mh3uBrcDb3f0lpx4N9VefPQy8xszqzKyGYMf7Qz0Lw6N12sMiC3ATkNedgABmNoVgiOrG9KIeagO+bGbnhDszP0zm/3E+jAS+YmZVZjYMeHfaYz8JmJnNCIcpb6Qf+iw0D9gc7i9JV8g+gyz6JXz9/ZkXD2L46/Q2pzKQC/s/ApeZ2XrgQwT/HMzsTWa2OGxzP7CeYMfcr4H3untbHjNNI3g3Ps7MPmBmnw8nPwpcbWbPEYwXv9Pd8/bRNHynvwH4jpltJDji41Nhrs+b2QfCphn7Mp/M7EKCHeOXA6vDnaNLw2X93mfuvofgo/hyYC3Bp4SVZrbUzBaGzd4FfM3MNgFDCY7iybe/A6qAr4Z9tDbsn6VmttDdGwmGHH5LcEhchJceaJAX7v4/BMNDa4Cngbvd/Ykw40R3byfYYv4FsIHgDfn+k91fjmV6LRa8zwBO1S9mttjM3hQ2/RBwq5ltIPg0mfWhojofu4hIiRnIW+wiIpKBCruISIlRYRcRKTEq7CIiJUaFXUSkxKiwi4iUGBV2EZESo8IuIlJi/j/mMU566a77JQAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "<Figure size 432x288 with 1 Axes>"
+      ]
+     },
+     "metadata": {
+      "needs_background": "light"
+     },
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def sigmoid_fxn(x):\n",
+    "    yhat = list(map(lambda i: 1 / (1 + np.exp(-i)), x))\n",
+    "    return yhat\n",
+    "\n",
+    "x = np.arange(-10., 10., 0.2)\n",
+    "logit = sigmoid_fxn(x)\n",
+    "plt.title(\"The Sigmoid Function Curve\", fontsize=15)\n",
+    "\n",
+    "plt.plot(x, logit)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Building my Customized Logistic Regression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class LogisticClassifier():\n",
+    "    def __init__(self, \n",
+    "            learning_rate=0.1, \n",
+    "            tolerance=1e-4, \n",
+    "            max_iter=1000, \n",
+    "            batch_size=32, \n",
+    "            momentum_decay=0.9, \n",
+    "            early_stopping=3, \n",
+    "            validation_set=(None,None)):\n",
+    "        \n",
+    "        # Gradient descent parameters\n",
+    "        self.learning_rate = float(learning_rate)\n",
+    "        self.tolerance = float(tolerance)\n",
+    "        self.max_iter = int(max_iter)\n",
+    "        self.batch_size=32\n",
+    "        self.momentum_decay = float(momentum_decay)\n",
+    "        self.early_stopping = int(early_stopping)\n",
+    "        self.X_validation, self.y_validation = validation_set\n",
+    "        \n",
+    "        # to construct the design matrix\n",
+    "        self.add_intercept = True\n",
+    "        self.center = True \n",
+    "        self.scale = True\n",
+    "        \n",
+    "        self.training_loss_history = []\n",
+    "        \n",
+    "    def __sigmoid(self, X):\n",
+    "        return 1 / (1 + np.exp(-X))\n",
+    "    \n",
+    "    # z-score normalization and intercept addition\n",
+    "    def __design_matrix(self, X):\n",
+    "        if self.center:\n",
+    "            X = X - self.means\n",
+    "        if self.scale:\n",
+    "            X = X / self.standard_error\n",
+    "        if self.add_intercept:\n",
+    "            intercept = np.ones((X.shape[0], 1))\n",
+    "            X = np.hstack([intercept, X])\n",
+    "            \n",
+    "        return X\n",
+    "    \n",
+    "    def __fit_center_scale(self, X):\n",
+    "        self.means = X.mean(axis=0)\n",
+    "        self.standard_error = np.std(X, axis=0)\n",
+    "        \n",
+    "    def fit(self, X, y):\n",
+    "        self.__fit_center_scale(X)\n",
+    "\n",
+    "        n, k = X.shape\n",
+    "        \n",
+    "        # add intercept column to the design matrix\n",
+    "        X = self.__design_matrix(X)\n",
+    "\n",
+    "        # used for the convergence check\n",
+    "        previous_loss = -float('inf')\n",
+    "        self.converged = False\n",
+    "        self.stopped_early = False\n",
+    "        \n",
+    "        # initialize parameters\n",
+    "        self.beta = np.zeros(k + (1 if self.add_intercept else 0))\n",
+    "        momentum = self.beta * 0  # to get the same shape and dtype as beta\n",
+    "        \n",
+    "        for i in range(self.max_iter):\n",
+    "            shuffle = np.random.permutation(len(y))\n",
+    "            X = X[shuffle, :]\n",
+    "            y = y[shuffle]\n",
+    "            \n",
+    "            # we'll add one more batch, incase the batch size doesn't divide n evenly\n",
+    "            extra = (1 if n % self.batch_size else 0)\n",
+    "\n",
+    "            for batch_index in range(n // self.batch_size + extra):\n",
+    "                batch_slice = slice(\n",
+    "                    self.batch_size * batch_index, \n",
+    "                    self.batch_size * (batch_index + 1) )\n",
+    "                X_batch = X[batch_slice, :]\n",
+    "                y_batch = y[batch_slice]\n",
+    "            \n",
+    "                beta_ahead = self.beta + self.momentum_decay * momentum\n",
+    "                y_hat = self.__sigmoid(np.dot(X_batch, self.beta))\n",
+    "                \n",
+    "                # gradient descent\n",
+    "                residuals = (y_hat - y_batch).reshape( (X_batch.shape[0], 1) )\n",
+    "                gradient = (X_batch * residuals).mean(axis=0)\n",
+    "                momentum = self.momentum_decay * momentum - self.learning_rate * gradient\n",
+    "                self.beta += momentum\n",
+    "\n",
+    "            # with minibatch, we only check convergence at the end of every epoch.    \n",
+    "            y_hat = self.__sigmoid(np.dot(X, self.beta))\n",
+    "            self.loss = np.mean(-y * np.log(y_hat) - (1-y) * np.log(1-y_hat))\n",
+    "            self.training_loss_history.append(self.loss)\n",
+    "            \n",
+    "            # early stopping\n",
+    "            if self.check_validation_loss():\n",
+    "                self.stopped_early = True\n",
+    "                break \n",
+    "                \n",
+    "            if abs(previous_loss - self.loss) < self.tolerance:\n",
+    "                self.converged = True\n",
+    "                break\n",
+    "            else:\n",
+    "                previous_loss = self.loss\n",
+    "        \n",
+    "        self.iterations = i+1\n",
+    "        \n",
+    "    def predict_proba(self, X):\n",
+    "        # add intercept column to the design matrix\n",
+    "        X = self.__design_matrix(X)\n",
+    "        return self.__sigmoid(np.dot(X, self.beta))\n",
+    "\n",
+    "    def predict(self, X):\n",
+    "        predictions = self.predict_proba(X).round()\n",
+    "        return predictions\n",
+    "\n",
+    "    def check_validation_loss(self):\n",
+    "        # validation set loss\n",
+    "        if not hasattr(self, 'validation_loss_history'):\n",
+    "            self.validation_loss_history = []\n",
+    "        p_hat = self.predict_proba(self.X_validation)\n",
+    "        loss = np.mean(-self.y_validation * np.log(p_hat) - \\\n",
+    "               (1-self.y_validation) * np.log(1-p_hat))\n",
+    "        self.validation_loss_history.append(loss)\n",
+    "        \n",
+    "        t = self.early_stopping\n",
+    "        if t and len(self.validation_loss_history) > t * 2:\n",
+    "            recent_best = min(self.validation_loss_history[-t:])\n",
+    "            previous_best = min(self.validation_loss_history[:-t])\n",
+    "            if recent_best > previous_best:\n",
+    "                return True\n",
+    "        return False"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Testing and Evaluating the Customized Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.datasets import load_breast_cancer\n",
+    "from sklearn.model_selection import train_test_split as tts\n",
+    "\n",
+    "dummy_data = load_breast_cancer()\n",
+    "X, y = dummy_data.data, dummy_data.target\n",
+    "\n",
+    "X_train, X_test, y_train, y_test = tts(X, y, test_size=0.2, random_state=103)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = LogisticClassifier(learning_rate = 0.01,\n",
+    "                           tolerance=1e-5,\n",
+    "                           max_iter=2000,\n",
+    "                           early_stopping=3,\n",
+    "                           validation_set=(X_test, y_test))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.fit(X_train, y_train)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "y_pred = model.predict(X_test)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# defining the metrics function\n",
+    "def accuracy(predictions, actual):\n",
+    "    return sum(predictions == actual) / len(actual)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.9736842105263158"
+      ]
+     },
+     "execution_count": 23,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "accuracy(y_pred, y_test)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

From f9b9081bc03172e6f6e11a5f585a81c64e367dd9 Mon Sep 17 00:00:00 2001
From: Steven Kolawole <45284829+SteveKola@users.noreply.github.com>
Date: Wed, 22 Apr 2020 22:36:31 +0100
Subject: [PATCH 6/9] Delete Logistic Regression from Scratch.ipynb

---
 Logistic Regression from Scratch.ipynb | 322 -------------------------
 1 file changed, 322 deletions(-)
 delete mode 100644 Logistic Regression from Scratch.ipynb

diff --git a/Logistic Regression from Scratch.ipynb b/Logistic Regression from Scratch.ipynb
deleted file mode 100644
index 6031d92..0000000
--- a/Logistic Regression from Scratch.ipynb	
+++ /dev/null
@@ -1,322 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Logistic Regression from Scratch\n",
-    "## - Steven Kolawole                                "
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Intro\n",
-    "\n",
-    "Logistic Regression is simply a Linear Regression with a Sigmoid function at its end.\n",
-    "\n",
-    "The Sigmoid function generates probability (i.e. outputs between 0 and 1) for all values of X."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "import seaborn as sns\n",
-    "sns.set()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEMCAYAAADQ553CAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZhcZZn38W9V9Z49nc6eANlukkAIISzK4hZ0BLcRRMVhZBRxw1dHR8cRZkYdGX11RhnXcYy8ohgXcNyGgIqJgywhhCQEstxZCFk76SVrd3qtqvePczpUikq6Oqnqqq7+fa6rr65zznOqfv101V2nnnPqnEgymUREREpHtNABREQkt1TYRURKjAq7iEiJUWEXESkxKuwiIiVGhV1EpMSUFTqAnJqZ/QB49yma7HD3s83sBWCxu38hx4//18BtwFwgAawDvu7uP0tpkwRucvd7c/nYfch4M8HfnvH5bGaTgV3Aq9z9TxmWvxJYfpK7b3b3MblJempm9jIg6u6PhdN571cziwC3AO8B5gDdwFrgi+7+cL4eV/JLW+zF76PAhPDnknDem1PmXZyvBzazW4FvAt8GLgAuBR4AfmJmqW82E4D785UjCz8DJuXgfhbwYr/2/MzJwf1m6xFgZsp0XvvVzKLAr4A7gXsInl9XAU8DvzOzG/P12JJf2mIvcu5+GDgMYGZV4ewD7r6vHx7+/cD33P0HKfM2mJkRvOHcE2bsjywn5e5tQFsO7qqxwH9LJHWiH7LcBlwDXOTu61Lmf8rMhgJfN7Nfu3trnnNIjqmwl5ZJZvYbYBHBm8E33P1fexaa2VuAzwEGvAAsBr7q7omT3F8cuNzMRoRvMD3+DhiScr8nDBmY2SeBjwBjgN8BO4EL3P2V4bDHQ8BbgK8DU4DHgJuBTwM3Ae3A19z9SymP8R7g48B0YA9wl7t/M1x2MylDMWZ2FsGnjFcADcDxPjhdmYZ7MjxuEnhv+LdcDOwg6N//SlnnJuBTwIxw+Rfd/Z5wKC0G/D8zuznsq/R+7a0PPg18FfgMUAs8BXzY3Tee5M96P/DrtKLe43PADwj+FxmHhVLnhUOG1cBYgk8+nyb4H0xz9+0p66wHfunud5jZFOBrwGsJ3piXAx93970nyStZ0lBMaXkvsJRgPPw/gDvN7CoAM7sG+HE4fy5Bcfko8I+nuL+vEHw832tmvzGzvzOz+e7e6O4vZFrBzD4C/BNBcbmQ4A3kI2nNKoDPAzcCryYoBOuAFoKC+F3gi2Y2J7zPjxMMCd0FzAtzfcXMPpHh8csJ3jhqgMvDPvn0Kf7GXPu/BFkvBP4MfCd8o8HM3g7cTfCGej7wb8BiM3stwd8dBz4GvDX9TrPsg2nAu4DrCIrlWcA3MoUMP/3NAVZkWu7u+919pbvH+/C33wD8kmDI7lcEb1zvSHnM+eFj/tDMhgB/IijoLwdeR/C8WGZmFX14TMlAW+yl5T53/8/w9pfM7NPAQoKx288A33b3u8Pl28xsGPA9M/uXTFvt7n6fme0hKDavBd4IYGZrCLbU1mfI8AmCre2eLbu/NbMr0tpEgNvdfVV4f38kKGz/4O5JM/siwRvOXDPbSPAmdJe7Lw7X32Jm04C/N7Ovpt33IoJPJK9z953h/f8fgn0DvfFwKzTVPHd/Pot1e9zt7j8PH/eTBDsmLyEoch8Dfuzu/xG23RoOeUTdvTEY4eKwux9IvcNwB2c2fVAOfKBnC93MvkEwfp7JqPD3oT78bb3Z5+5fT8n9I+CdwBfDWe8CVrj7ZjO7heBT3809bx5m9k6gieCN6Sc5zDXoqLCXls1p04cIPh5DsAV5sZl9MGV5NFx+NpCxeLn748DjZhYDLiIo7h8BHjSzGe7e2dPWzGoJthKfSLubx4D5afO2ptxuBZ5392T4mG1hkasE6oBx4X2keoSg2I1Nm38e0NRT1EMZt0ozeB2QPq69K8t1exz/H7j7ofDv6NkCPR/4UWpjd78ri/vMtg+SwJaU5YdSHjtdc9h+dBaPn63059A9wB1mNhfYSLD13vNGcyHB33U47KMeNcDsHGYalFTYS0umj809O+Q6gS8TDMek250+Ixz//Afg8+6+L9yqWgmsNLM/E4ydzwNWpazWHf7OZoivK236ZOP87SeZHzvJ/SRJ2wlJ8Ldn4wV3f0lfnEKm109Hhnk9edKzZivbPki4e3dam/S+AMDdO81sNXBZpuVmNhP4FvC3mT6ZmVmmv/2EHdjuvtXMHifYan+YoJD3HCbbCawnw7ATuf0UMShpjH3wWA/MdPetPT8EW5B3kvnF30YwjJDpkLdDBAW0IXVmuIN1J8EYa6r06ay5+xGCN5704ZwrCLauD6bNXwuMCQtTj4Wn+/gpOoGYmdWkzJt5ssYnsTE9i5n90Mx6hi8ynkP7NPogW98H3mhm8zIs+yTBENIL4XQXMDxlebZ/+z0EQys3AL91956s64FzCL4n0PN8bCDY+Xt+X/4IeSltsQ8eXwAeMLPngF8Aswh2Ui5195dsZbp7k5l9mWAn5vBwnTaCF90XgHvShjt69KyziWBr/maCrcI/nWH2r5nZtvB+XkUwHPRP4Zh8atvlBMdh32tmHyYYivg6Z24FQeH9nJl9i+DN6uY+3seXgZ+b2UrgDwQ7jt8JvD5cfhSYY2Zj3b0hbd2+9EG2/ovg6KQ/mtlnCPpuOHArwU7nd6Yc6vgEcKuZPUbwSeFrZP50ku5nBDvs/4pgjL3Hj4HbCfrjHwg+lXyJ4M0k074b6QNtsQ8S7v4QwaGENwLPERT1HxIc8nayde4Il18NPErwguv5MsutJ1nt28C/hz/PELyB/Irsh0My5fguwc7ffwgzfJzgsLivZGgbJzg2eydBobqPoAidkXAH6geBtwGbgPcRbNX25T5+BXyYIP96gp2pN6V8w/NLwIcIhrnS1826D/qQJw68geAN5zZgDfB7gqNrXt2zEzj0QYJDaJ8k+NLUf5FhCC/DYxwGfk1QuB9Mmd9G8Lw6Biwj2H9QFj5u+pua9FFEV1CSXDKzvwCedfc9KfMeAva4+3sLl0xk8NBQjOTau4FzwmGQZoItwkUEh0uKSD9QYZdcu43gSzQPAsMIhi3e6e7LCppKZBDRUIyISInRzlMRkRJT6KGYSoKvkteT+cs1IiLyUjGC0zo/RYbDTgtd2C8mOFGSiIj03ZUEhyKfoNCFvR7g4MFWEom+j/XX1g6lubkl56FyoVizFWsuKN5sxZoLijebcvVdX7JFoxFGjRoCYQ1NV+jCHgdIJJKnVdh71i1WxZqtWHNB8WYr1lxQvNmUq+9OI1vGIWztPBURKTEq7CIiJUaFXUSkxGQ9xh6e4e9x4A3pl0ULL3m1mODMcI8QXMUl/bzQIiLSD7LaYjezSwkOqZl1kib3Are5+yyCc3u/LzfxRESkr7IdinkfwelGX3L18PBCvdXu3nP5sR8QnNpURGTQSyaTx38SaT/5ktVQjLvfAnCSk/lP5MRjKeuByWecTERKRld3gmPtXRzr6Ka9M057RzftXXE6OuN0dMXp7ErQ2R2nqzsR/MQTdIe/4/Ek3fEE3fEk8USSeCKYF08Gh0nHw8OlE8me35AMp4NiSngbopEI3fEEPefICo4uDJaFN4MiDCR7Zhy/zfHbyZ6LXaXO72OfRCLwwTefx8Jz0y/be+ZycRx7lBP/pggnv35lRrW1Q0/7wevqhp32uvlWrNmKNRcUb7ZizQWFy9bVnaD5cBuNh9poPNjGgSPtHDzazqEjHRxq6eBIaydHj3VytLWTzu7sS0JZLEpleZTyshhlZVHKy6KUxaKUxSLh7yhl5VGqYhFi0SjRaIRYNEI0GiEaSf3N8duRSIRIJCjshL8jEYL5ABGIcOK8SHjByEh4I5JyAcnj64XrvngzktImdX6KcEE0CpfMm0jtiOrji3L1v8xFYd9NcM6CHuPJMGRzKs3NLaf1pYG6umE0Nh7t83r9oVizFWsuKN5sxZoL+idbS1sXexpb2NPUyp6mVvYfOEbDwTaaj7STPppQWRFjxJAKRg+vYuSQCibXDWFoVTk1VWXUVJVRXVlGdUUZVRUxKitiwe/yGBXlMcrDIh6NZLz+dk4U2/8y0dl9PE9fskWjkVNuEJ9xYXf3HWbWbmaXu/tjBJdfe7C39USk+MQTCXbub8F3HmJ7/RG21x+h6XD78eXVlWWMH13DjEkjeNnc8YwZWcXo4VWMHlbJqGGVVFUEJaXYCuhgc9qF3cyWElxIdxXBRWq/Fx4SuZrcXDxYRPrBgSPtPLO1iXXbmtm8+xBtHcG31GuHV3HOhGG86sJJTBk7lEl1Qxk5tOL40IQUrz4Vdnc/O+X2NSm3nyG4uriIDAAHj3bw5Ib9rNiwj537gxNPjR1ZzaVzxnPu1JHMmjKSkUMrC5xSTlehTwImIv0kkUjyzNYmlq/Zw/oXDpBMwjkThvO2V07nghljmFBbo63xEqHCLlLiOrriPLqunj88tYuGQ22MHl7JtS87m5efN57xo2sKHU/yQIVdpER1xxP8eV09v3lsO4dbOpk+cThvfcU0LrI6YlGdJqqUqbCLlKC1W5r46bItNBxsY8bkEXzgTXOxqaMKHUv6iQq7SAk51NLBkj9sZpU3MmnMED56/TzmTa/V2Pkgo8IuUiKeeG4f9/5hM13dCd561TT+4tKplMU05DIYqbCLDHCdXXGWPLyZR56pZ+bkEfzNNbO1U3SQU2EXGcD2NrVw54+eZldDC9e+7CzecuU52jEqKuwiA9Xze4/w9V+sIx5P8LG3zWPe9DGFjiRFQoVdZABat62Zb//qWUYPr+Kj189j3CgNvciLVNhFBpgn1u/j7gc2MqluCF/4wOV0d3QVOpIUGQ3GiQwgT3sDi/9nAzMnj+Dvb1zAqOFVhY4kRUhb7CIDxHPbm/nPX69n+sQRfPT6C6isiBU6khQpbbGLDABbdx/mm//9LBPHDOFjb5unoi6npMIuUuSaDrXx9V+sY9TQSj7+9vnUVJUXOpIUORV2kSLW0RXnm//9LPFEko+97QJGDKkodCQZAFTYRYpUMpnkngc3sauhhfe/aQ7j9G1SyZIKu0iR+sNTu1ixYT9vuWqavnwkfaLCLlKEduw7yn1/2saCWXW84WVnFTqODDAq7CJFpqs7zuL/2cDQmnJufv25OuWu9JkKu0iR+eUj29nT1Mp7rpnN0GodASN9p8IuUkR850F+t3Inr7xwEudPqy10HBmgVNhFikRnV5y7l26kbmQ1N7xqeqHjyACmwi5SJJau2EHjoXbe/fpzqarQ2T7k9KmwixSBhkNtLF2xk0tmj2X2WbrotJwZFXaRIvDTh7cQi0Z4+6tnFjqKlAAVdpECW7etibVbm3jT5WczalhloeNICVBhFymg7niCJQ9vYfzoGq6+eEqh40iJUGEXKaA/r6un4WAb73jNDMpiejlKbuiZJFIgnV1xfvvYdmZMHqFj1iWnVNhFCmTZ6j0caunkuqum6bQBklNZHSxrZjcCdwDlwF3u/q205QuA7wIVwC7gr9z9UI6zipSMto5ulq7YwdxzRmNTdXij5FavW+xmNgm4E7gCmA/camZz0pr9B/BP7n4B4MDf5TqoSCn5w1O7aGnr4q1XTSt0FClB2QzFLAKWufsBd28F7geuT2sTA4aHt2uAttxFFCktre1d/O6pnSyYVcc5E4b3voJIH2UzFDMRqE+ZrgcuSWvzceD3ZnYX0Apc2pcQtbVD+9L8BHV1w0573Xwr1mzFmguKN1sucy1/eDNtHXFufuPcnNzvYOizXCrWXJC7bNkU9iiQTJmOAImeCTOrBr4PLHL3lWb2ceCHwLXZhmhubiGRSPbeME1d3TAaG4/2eb3+UKzZijUXFG+2XObq6o7z6//dynnTRjO0PHrG9zsY+iyXijUX9C1bNBo55QZxNkMxu4EJKdPjgb0p0+cBbe6+Mpz+LvDKrNKJDDKPPbuPI8e6eP2luiqS5E82hf1h4DVmVmdmNcB1wEMpy7cCU8zMwuk3A0/lNqbIwJdIJHlo5U7OHj+Mc6eOLHQcKWG9FnZ33wPcDiwH1gJLwiGXpWa20N0PAjcDPzezdcB7gL/JY2aRAWn15kYaDrZxzWVn6bh1yausjmN39yXAkrR516TcfhB4MLfRREpHMpnkwSd3MHZUNQtm1RU6jpQ4ffNUpB9s2X2Y7fVHed0lU4lGtbUu+aXCLtIPlq/ZQ3VlGS+fO77QUWQQUGEXybPDrZ2s2tTAFedPoLIiVug4MgiosIvk2SPP7CWeSPLKCycWOooMEirsInkUTyT437V7mHP2KCbUDil0HBkkVNhF8mjd1mYOHOngVRdOLnQUGURU2EXyaNmaPYwaVsn8mbqQhvQfFXaRPNl/8Bjrtx/glfMnEovqpSb9R882kTx5dF09kQhcMU87TaV/qbCL5EEikeTx5/Zx/rRaRg2rLHQcGWRU2EXyYP0LBzh4tIMrzp/Qe2ORHFNhF8mDR9fVM6SqjAtmjCl0FBmEVNhFcqy1vYs1Wxq5bO54ysv0EpP+p2edSI49uWE/3fGkhmGkYFTYRXLs0XX1TBk7lLPGF++1NaW0qbCL5NDuxhZe2HeUy7W1LgWkwi6SQ0+s30c0EuGyOeMKHUUGMRV2kRxJJJOs3NDA3HNGM3xIRaHjyCCmwi6SI9v2HKb5SLu21qXgVNhFcmTFhv1UlEWZP1PHrkthqbCL5EB3PMFTGxuYP3MM1ZVZXSNeJG9U2EVyYOOOg7S0dXHpbA3DSOGpsIvkwIr1+6mpLOO8aTrvuhSeCrvIGeroirN6SyMLz63TKQSkKOhZKHKG1m1rpqMzzqVzxhc6igigwi5yxp7auJ/hQyqwKSMLHUUEUGEXOSMdnXHWbWvmIqsjGo0UOo4IoMIuckbWPd9MZ3eCi21soaOIHKfCLnIGVm1qYHhNObM0DCNFRIVd5DR1dMV5ZlsTC2yshmGkqGT1FTkzuxG4AygH7nL3b6UtN+C7wChgH/AOdz+Y46wiReXZbc10diW42OoKHUXkBL1usZvZJOBO4ApgPnCrmc1JWR4BfgN8yd0vANYAn85PXJHiscobGFZTzqypGoaR4pLNUMwiYJm7H3D3VuB+4PqU5QuAVnd/KJz+V+BbiJSwzq44z2xtZsGsOmJRjWhKcclmKGYiUJ8yXQ9ckjI9A9hnZt8HLgQ2Ah/JWUKRIvTs8wfo6Iqz8FwdDSPFJ5vCHgWSKdMRIJF2H68ErnL3VWb2L8BXgZuzDVFbOzTbpi9RV1e815Us1mzFmguKN1t6rvW/38ywmgquXDCFWKywW+wDpc+KRbHmgtxly6aw7wauTJkeD+xNmd4HbHH3VeH0TwiGa7LW3NxCIpHsvWGaurphNDYe7fN6/aFYsxVrLijebOm5uuMJnly/j4tm1XHgQGsBkw2cPisWxZoL+pYtGo2ccoM4m02Nh4HXmFmdmdUA1wEPpSx/HKgzswvC6TcCT2eVTmQA2rTjIG0d3SzQ0TBSpHot7O6+B7gdWA6sBZa4+0ozW2pmC929DfhL4Htmth54NfCJfIYWKaTVmxuprIgx9+xRhY4iklFWx7G7+xJgSdq8a1JuP8mJO1RFSlIikWT1libmTaulvCxW6DgiGek4LZE+2LrnMEdaO1kwS8MwUrxU2EX6YPXmRspiEeZN15WSpHipsItkKZlMsnpzI3POHq0LVktRU2EXydKuhhaaDrdrGEaKngq7SJZWb24kEoH5M8cUOorIKamwi2Rp9eYmZk4eyfCaikJHETklFXaRLDQcamN3YwsLtLUuA4AKu0gW1m5uBGC+xtdlAFBhF8nC6i1NTK4bytiR1YWOItIrFXaRXhxu6WDL7kMsmKVhGBkYVNhFevHUhn0kk3DhTA3DyMCgwi7SixXP7aN2eCVTx53+dQNE+pMKu8gpdHTFWbO5kQtn1hGJRAodRyQrKuwip7B++wE6u+JcqKNhZABRYRc5hTWbGxlaXc6sKSMKHUUkayrsIicRTyRYu7WJi+eMIxbVS0UGDj1bRU5iy67DtLZ3c9l5EwodRaRPVNhFTmL1lkbKy6IssLGFjiLSJyrsIhkkk0nWbG5i7tmjqdK512WAUWEXyWBXQwvNR9q5UCf9kgFIhV0kgzVbmohE4AIVdhmAVNhFMlizuZEZk0bo3OsyIKmwi6RpOtTGzoYWnRtGBiwVdpE0a7Y0AXChzuYoA5QKu0ia1ZsbmTRmCONG1RQ6ishpUWEXSXHkWCebdx9igc4NIwOYCrtIirVbmkgmUWGXAU2FXSTF6s2NjBlRpXOvy4Cmwi4SauvoZsMLB1gwS+del4FNhV0k9OzzzXTHkxqGkQFPhV0k9LQ3MrymnBmTdO51GdiyKuxmdqOZbTCzLWb24VO0u9bMtucunkj/6OqOs+75Zi6cVUc0qmEYGdh6LexmNgm4E7gCmA/camZzMrQbB/wboFeFDDjrXzhIR2dcwzBSErLZYl8ELHP3A+7eCtwPXJ+h3WLgc7kMJ9JfVnsj1ZUxZp81qtBRRM5YNieangjUp0zXA5ekNjCz/wOsBlacToja2tM/tKyubthpr5tvxZqtWHNBYbJ1xxM8s62JS+ZMYML4zOPr6rO+U66+y1W2bAp7FEimTEeARM+EmZ0HXAe8Bph8OiGam1tIJJK9N0xTVzeMxsajp/OQeVes2Yo1FxQu23Pbmzl6rIvzzxmV8fHVZ32nXH3Xl2zRaOSUG8TZDMXsBlIv+jge2Jsy/bZw+SpgKTDRzP6cVTqRIrBqUwOVFTHOO2d0oaOI5EQ2W+wPA581szqglWDr/Naehe7+z8A/A5jZ2cCf3P3K3EcVyb3ueILVm5uYP2MMFeWxQscRyYlet9jdfQ9wO7AcWAsscfeVZrbUzBbmO6BIPvmuQ7S0dbHQdDSMlI6srtLr7kuAJWnzrsnQ7gXg7FwEE+kPqzY1UFke4/xptYWOIpIz+uapDFrxRIKnvZELZtRqGEZKigq7DFq+s2cYZmyho4jklAq7DFqrvJGK8ijnT9cwjJQWFXYZlIJhmAbmTR9DpYZhpMSosMugtPGFgxw91sVlc8YVOopIzqmwy6C0YsN+qivLdDSMlCQVdhl0OrviPL25kYVWR3mZXgJSevSslkHnmW3NdHTGNQwjJUuFXQadJzfsZ8TQCmyqTtErpUmFXQaVY+1drNvWxKWzx+lKSVKyVNhlUHnaG+mOJ7lUwzBSwlTYZVBZsWE/Y0dVc/b44r3YgsiZUmGXQePAkXY27TjIZXPGEYloGEZKlwq7DBqPPbePJHD5+RN6bSsykKmwy6CQTCZ57Nl6zp06krqR1YWOI5JXKuwyKGzZfZiGg23aWpdBQYVdBoVH19VTVRHTKXplUFBhl5LX3tnNU5sauPjcsVRW6EyOUvpU2KXkrdrUSEdXnCvmaRhGBgcVdil5jz5bz7jRNcyYNKLQUUT6hQq7lLS9Ta1s3nWIK84fr2PXZdBQYZeS9qc1e4hFI1w5b2Kho4j0GxV2KVkdnXEee66ei88dy/AhFYWOI9JvVNilZK3YsI+2jjivWjCp0FFE+pUKu5SkZDLJstV7mFw3VDtNZdBRYZeStG3PEXY1tPDqiyZpp6kMOirsUpKWrdlNdWVMl7+TQUmFXUrO4ZYOVm1q4OVzJ1BVUVboOCL9ToVdSs7DT+8mHk+y6OLJhY4iUhAq7FJS2jq6WbZ6DxdZHeNG1RQ6jkhBZPU51cxuBO4AyoG73P1bacvfDHwOiADbgb9x94M5zirSq/9du5e2jm5ef9lZhY4iUjC9brGb2STgTuAKYD5wq5nNSVk+HPgOcK27XwCsAz6bl7Qip9AdT/CHVbs4d+pIzpkwvNBxRAomm6GYRcAydz/g7q3A/cD1KcvLgQ+7+55weh0wNbcxRXq3Yv1+Dh7t0Na6DHrZDMVMBOpTpuuBS3om3L0Z+CWAmVUDnwa+kcOMIr1KJJM8tHInk+uGct45owsdR6SgsinsUSCZMh0BEumNzGwEQYF/xt3v6UuI2tqhfWl+grq6Yae9br4Va7ZizQWnn+2xZ/ayt6mVT9y4gLFjcz8MU4p9lm/K1Xe5ypZNYd8NXJkyPR7Ym9rAzCYAvwOWAX/b1xDNzS0kEsneG6apqxtGY+PRPq/XH4o1W7HmgtPPlkgkueeB9UyorWH25BE5//tKsc/yTbn6ri/ZotHIKTeIsynsDwOfNbM6oBW4Dri1Z6GZxYDfAj939y9klUokh55Yv4/65mN86C3nEY3q9AEivRZ2d99jZrcDy4EKYLG7rzSzpcA/AVOABUCZmfXsVF3l7rfkK7RIj+54gl8/up2zxg3jIqsrdByRopDVcezuvgRYkjbvmvDmKvRFJymQR57ZS9Phdm56nelkXyIhFWQZsDq64vz28ReYOXmEjoQRSaHCLgPWgyt2cLilk+teMV1b6yIpVNhlQGo41MbSFTu5ZPZYZk0ZWeg4IkVFhV0GpJ8+vIVYNMLbXz2z0FFEio4Kuww467Y1sXZrE2+6/GxGDassdByRoqPCLgNKV3eCJQ9vYfzoGq6+eEqh44gUJRV2GVB+/eh2Gg62cePVMymL6ekrkoleGTJgbN19mAef3MGV8yZw3jm1hY4jUrRU2GVAaO/sZvH/bKB2eBXveI12mIqcigq7DAj3Ld9G46E23nvtbKordYFqkVNRYZeit3ZrE8vX7OHqi6dgU0cVOo5I0VNhl6K2/8AxvvfbDUwdO5TrXjGt0HFEBgQVdilabR3dfOO/nyUWjXDbW8+nvCxW6EgiA4IKuxSlRDLJ9x/YyL7mY3zwzXMZM7K60JFEBgwVdilKv3zkeVZvbuSGV01n9tk6c6NIX6iwS9F5cMUOHnhiB6+YP1HfLhU5DSrsUlT+tGYP9/1pG5fMHstNr9XFM0ROhwq7FI0/PrWTH/3OmTe9llveMEfXLxU5TfqmhxRcMpnkoZU7uW/5NmafNYoPveU8nQdG5AyosEtBJZJJfr5sK79/ahdXzp/EXy2aSXmZirrImVBhl4I51t7N3Us3snpzI4sumsxH3rGA5uaWQscSGfBU2KUgduw7yrd/9SwHjnTwjlfP4OqLp2hMXSRHVNilXyUSSZat3s3Pl342r5gAAAtySURBVG9jWE05f3/jAmZMHlHoWCIlRYVd+s3uxhbueXAT2/YeYd70Wt577WyG1VQUOpZIyVFhl7w71t7FAyt28PuVu6iuLON9b5jDZXPH6Rh1kTxRYZe86eiKs+zp3SxdsYPW9m4uP288N7x6hrbSRfJMhV1y7nBrJ8tX72b5mj0cPdbFvOm1vPWqaUwdN6zQ0UQGBRV2yYlEMonvOMijz+7jqU376Y4nuWB6La+/7CxmTRlZ6Hgig4oKu5y2RCLJ83uPsGZLI09u3M+BIx1UV8a48oKJXL1wCuNH1xQ6osigpMIufdJ0uA3feYiNOw7y7PPNHD3WRSwaYc7Zo7nhVTOYP2MMFeW6IIZIIamwy0kdPdbJ7sZWXth3hO31R9m+9wjNR9oBGFJVxnnTapk/YwznTxtNTVV5gdOKSI+sCruZ3QjcAZQDd7n7t9KWzwcWA8OBR4APuHt3jrNKjiWTSVraujhwpIPmI+00HGyj4eAx9h9sY09TK0daO4+3rR1exTkThvHaS6Zw7tRRTKobQlSHK4oUpV4Lu5lNAu4ELgI6gMfNbLm7b0hpdi9wi7uvMLPvA+8DvpOPwPJSyWSS7niC9s44bZ1x2ju6ae+M09rexbH2blrbu2lp66KlrYuO7gSNB45xuLWDwy2ddHYnTrivIVVljB1Vw7xptUwcM4TJdUOYOn4Yw3WIosiAkc0W+yJgmbsfADCz+4Hrgc+H02cB1e6+Imz/A+Bz5LmwJ5JJVm3cz/7Go6dsl0ym3OaEiRd/ZWhzwnrJcH7yxabJZDKcHzRIJE+cXzOkkqNH20kSTCcSSRLJE28nksngdgLiiQSJRJLuRJJ4PEk8kSAe3u6KJ+juTtAdT9AVT9DVnaCzK0FXd5yO7gSdXfET8mYSicCw6nJGDKtiaFUZ0yeOYPiQCmqHVzF6eCWjh1cxdlQ1QzSkIjLgZVPYJwL1KdP1wCW9LJ/clxC1tUP70hyArbsO8bnFK3pvWKSi0QjRSIRoNEJZLEIsGiEWjRKLRYjFopRFI5SVRSmLRSmPRSkrj1FdXU55WZSKshgV5VEqymNUVsSoqiijKvxdU1VGdWXwe2h1BUOqyxlaU86QqvIBcZKturriPNa9WHNB8WZTrr7LVbZsCnuUE7ZpiQCJPizvVXNzC4lEL5ucaUZUxVh8+9XU7z9ywgNncrKh4J6vtEfSVo682OD4dE+bCJHj9xeJBLcjPfcV3o5GI9SNGUZzcwvRnjaRCNFo+Ls/x6aTCdpbO2hv7QCCJ05jL59yCqVYsxVrLijebMrVd33JFo1GTrlBnE1h3w1cmTI9HtibtnzCKZbnzbjRNUTj8f54qD4bUl3OsUoddCQi/S+bS9U8DLzGzOrMrAa4DnioZ6G77wDazezycNZNwIM5TyoiIlnptbC7+x7gdmA5sBZY4u4rzWypmS0Mm70L+JqZbQKGAl/PV2ARETm1rMYK3H0JsCRt3jUpt5/hxB2qIiJSILpqsIhIiVFhFxEpMSrsIiIlptDH48WAM/riTDF/6aZYsxVrLijebMWaC4o3m3L1XbbZUtplPJVqJNnbd9Hz6wrgz4UMICIygF0JPJo+s9CFvRK4mOA0BMX5TSMRkeITI/hi6FMEJ2c8QaELu4iI5Jh2noqIlBgVdhGREqPCLiJSYlTYRURKjAq7iEiJUWEXESkxKuwiIiWm0KcUyJqZ/QsQd/fPhtMjgR8D04BG4AZ335e2TgT4CvAGgsv1vc/dH8tDtrHA71NmjQDq3H1oWruzgOeAbeGs/e7+ulznSXvMdwNfAvaHsx5w99vT2vTal3nKdjnwNaACaAbeE164JbVNv/WZmd0I3AGUA3e5+7fSls8HFgPDgUeAD7h7dz6ypD3uPwM3hJMPuPunMix/D3AwnPW99Ox5zLYcGAt0hbPe7+5PpixfBHwVqAZ+5u539EOmW4DbUmadA/zI3W9LadOvfWZmw4HHgTe4+wvZ9IuZTQXuJehfB97l7i3ZPF7RF3YzG0HQAe8Evpyy6AvAn939WjO7CfgP4O1pq18HzAbmADOAB8xsdq5fjO7eAMwP80aBPxJcnCTdQoILlbw/l4/fi4XAx939J6dok01f5sOPgTe5+zozew/BBVrenNamX/rMzCYBdwIXEXyT73EzW+7uG1Ka3Qvc4u4rzOz7wPuA7+Q51yLgtcCFBNcWfsjM/tLdf5nSbCHwDnd/Ip9ZMmSLALOAszK9psysGrgbeAWwi+D193p3z+sV1tx9McEbMGY2F/gV8Nm0Zv3WZ2Z2KfA9gr7qS798G/i2u//UzP4R+Efg77N5zIEwFPNmYAvw72nzryUoDAA/AV5vZuUZ2vzU3RPuvhnYCbw8n2GBvwGOhRcnSXcxcJ6ZrTWzZWZ2fp6z9Dzmu83sWTO718xGZWiTTV/mlJlVAne4+7pw1jpgaoam/dVni4Bl7n7A3VuB+4HrU/KeBVS7+4pw1g+At+UpS6p64BPu3unuXcBGXtpPC4HPmNk6M/ummVX1Qy4AC3//3syeMbPb0pZfAmxx9+1h4b+X/umzVN8BPuPuTWnz+7PP3gd8mBevBd1rv4Svv6sInofQx+db0Rd2d/+hu3+Jl55LZiLBk56wc44AdSdrE6oHJucpKmYWI9hS//RJmrQT/BMXAP8G/MrMKvKVJ1QP/Aswj2Dr4JsZ2mTTlznl7h3ufi8c/5TzWYItq3T91We9PVf69bnUw93X97yZmNlMgiGZpT3LzWwosAb4JEEfjSTYsusPowg+nf4l8BrgA2Z2dcrygvRZj/DTTrW735c2v1/7zN1vcffUkx1m0y9jgCMpn4T61HdFMxRjZm8jGG9NtcndF51klfTzW0YIxtFTRQk+vp6qTZ/0kvMvCN6Jn820bs/+gdBSM/siwVDRM2eSKYtcPW2+zItj1amy6cu8ZAuL9D0Ez8V/TV83n32WprfnSs6fS30RDik8AHzS3bf0zA/HXK9JaffvBB/zMw0F5lQ4jHF8KCMcnroG+EM4q6B9BryfYBj3BIXss1A2/ZLehgxtTqpoCnv4rnpfrw1ftAcYD+w2szJgGMEOuFS7Cc6A1mM8L34cykfOtwA/Pdm6ZvYRgvHinpwRXtzpdEYy5TKzEWb2t+7eU1QjQKb9C9n0ZU6zhfmGAr8JH+vN4VBDepu89Vma3QSnQO2R/lzJ+XMpW+FO5l8AH3P3n6Ytmwoscve7w1n56p9Mua4AKt39jyd57EL2WQXBGPbNGZYVrM9C2fRLAzDCzGLuHg/bZ913RT8UcwpLgb8Ob7+dYOdf+j9nKfAuM4uZ2QyCnRdP5THTyzj1+eVfAbwXwMxeQXDqzU15zNMCfCrceQPBkQK/zNAum77Mh3uBrcDb3f0lpx4N9VefPQy8xszqzKyGYMf7Qz0Lw6N12sMiC3ATkNedgABmNoVgiOrG9KIeagO+bGbnhDszP0zm/3E+jAS+YmZVZjYMeHfaYz8JmJnNCIcpb6Qf+iw0D9gc7i9JV8g+gyz6JXz9/ZkXD2L46/Q2pzKQC/s/ApeZ2XrgQwT/HMzsTWa2OGxzP7CeYMfcr4H3untbHjNNI3g3Ps7MPmBmnw8nPwpcbWbPEYwXv9Pd8/bRNHynvwH4jpltJDji41Nhrs+b2QfCphn7Mp/M7EKCHeOXA6vDnaNLw2X93mfuvofgo/hyYC3Bp4SVZrbUzBaGzd4FfM3MNgFDCY7iybe/A6qAr4Z9tDbsn6VmttDdGwmGHH5LcEhchJceaJAX7v4/BMNDa4Cngbvd/Ykw40R3byfYYv4FsIHgDfn+k91fjmV6LRa8zwBO1S9mttjM3hQ2/RBwq5ltIPg0mfWhojofu4hIiRnIW+wiIpKBCruISIlRYRcRKTEq7CIiJUaFXUSkxKiwi4iUGBV2EZESo8IuIlJi/j/mMU566a77JQAAAABJRU5ErkJggg==\n",
-      "text/plain": [
-       "<Figure size 432x288 with 1 Axes>"
-      ]
-     },
-     "metadata": {
-      "needs_background": "light"
-     },
-     "output_type": "display_data"
-    }
-   ],
-   "source": [
-    "def sigmoid_fxn(x):\n",
-    "    yhat = list(map(lambda i: 1 / (1 + np.exp(-i)), x))\n",
-    "    return yhat\n",
-    "\n",
-    "x = np.arange(-10., 10., 0.2)\n",
-    "logit = sigmoid_fxn(x)\n",
-    "plt.title(\"The Sigmoid Function Curve\", fontsize=15)\n",
-    "\n",
-    "plt.plot(x, logit)\n",
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Building my Customized Logistic Regression"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class LogisticClassifier():\n",
-    "    def __init__(self, \n",
-    "            learning_rate=0.1, \n",
-    "            tolerance=1e-4, \n",
-    "            max_iter=1000, \n",
-    "            batch_size=32, \n",
-    "            momentum_decay=0.9, \n",
-    "            early_stopping=3, \n",
-    "            validation_set=(None,None)):\n",
-    "        \n",
-    "        # Gradient descent parameters\n",
-    "        self.learning_rate = float(learning_rate)\n",
-    "        self.tolerance = float(tolerance)\n",
-    "        self.max_iter = int(max_iter)\n",
-    "        self.batch_size=32\n",
-    "        self.momentum_decay = float(momentum_decay)\n",
-    "        self.early_stopping = int(early_stopping)\n",
-    "        self.X_validation, self.y_validation = validation_set\n",
-    "        \n",
-    "        # to construct the design matrix\n",
-    "        self.add_intercept = True\n",
-    "        self.center = True \n",
-    "        self.scale = True\n",
-    "        \n",
-    "        self.training_loss_history = []\n",
-    "        \n",
-    "    def __sigmoid(self, X):\n",
-    "        return 1 / (1 + np.exp(-X))\n",
-    "    \n",
-    "    # z-score normalization and intercept addition\n",
-    "    def __design_matrix(self, X):\n",
-    "        if self.center:\n",
-    "            X = X - self.means\n",
-    "        if self.scale:\n",
-    "            X = X / self.standard_error\n",
-    "        if self.add_intercept:\n",
-    "            intercept = np.ones((X.shape[0], 1))\n",
-    "            X = np.hstack([intercept, X])\n",
-    "            \n",
-    "        return X\n",
-    "    \n",
-    "    def __fit_center_scale(self, X):\n",
-    "        self.means = X.mean(axis=0)\n",
-    "        self.standard_error = np.std(X, axis=0)\n",
-    "        \n",
-    "    def fit(self, X, y):\n",
-    "        self.__fit_center_scale(X)\n",
-    "\n",
-    "        n, k = X.shape\n",
-    "        \n",
-    "        # add intercept column to the design matrix\n",
-    "        X = self.__design_matrix(X)\n",
-    "\n",
-    "        # used for the convergence check\n",
-    "        previous_loss = -float('inf')\n",
-    "        self.converged = False\n",
-    "        self.stopped_early = False\n",
-    "        \n",
-    "        # initialize parameters\n",
-    "        self.beta = np.zeros(k + (1 if self.add_intercept else 0))\n",
-    "        momentum = self.beta * 0  # to get the same shape and dtype as beta\n",
-    "        \n",
-    "        for i in range(self.max_iter):\n",
-    "            shuffle = np.random.permutation(len(y))\n",
-    "            X = X[shuffle, :]\n",
-    "            y = y[shuffle]\n",
-    "            \n",
-    "            # we'll add one more batch, incase the batch size doesn't divide n evenly\n",
-    "            extra = (1 if n % self.batch_size else 0)\n",
-    "\n",
-    "            for batch_index in range(n // self.batch_size + extra):\n",
-    "                batch_slice = slice(\n",
-    "                    self.batch_size * batch_index, \n",
-    "                    self.batch_size * (batch_index + 1) )\n",
-    "                X_batch = X[batch_slice, :]\n",
-    "                y_batch = y[batch_slice]\n",
-    "            \n",
-    "                beta_ahead = self.beta + self.momentum_decay * momentum\n",
-    "                y_hat = self.__sigmoid(np.dot(X_batch, self.beta))\n",
-    "                \n",
-    "                # gradient descent\n",
-    "                residuals = (y_hat - y_batch).reshape( (X_batch.shape[0], 1) )\n",
-    "                gradient = (X_batch * residuals).mean(axis=0)\n",
-    "                momentum = self.momentum_decay * momentum - self.learning_rate * gradient\n",
-    "                self.beta += momentum\n",
-    "\n",
-    "            # with minibatch, we only check convergence at the end of every epoch.    \n",
-    "            y_hat = self.__sigmoid(np.dot(X, self.beta))\n",
-    "            self.loss = np.mean(-y * np.log(y_hat) - (1-y) * np.log(1-y_hat))\n",
-    "            self.training_loss_history.append(self.loss)\n",
-    "            \n",
-    "            # early stopping\n",
-    "            if self.check_validation_loss():\n",
-    "                self.stopped_early = True\n",
-    "                break \n",
-    "                \n",
-    "            if abs(previous_loss - self.loss) < self.tolerance:\n",
-    "                self.converged = True\n",
-    "                break\n",
-    "            else:\n",
-    "                previous_loss = self.loss\n",
-    "        \n",
-    "        self.iterations = i+1\n",
-    "        \n",
-    "    def predict_proba(self, X):\n",
-    "        # add intercept column to the design matrix\n",
-    "        X = self.__design_matrix(X)\n",
-    "        return self.__sigmoid(np.dot(X, self.beta))\n",
-    "\n",
-    "    def predict(self, X):\n",
-    "        predictions = self.predict_proba(X).round()\n",
-    "        return predictions\n",
-    "\n",
-    "    def check_validation_loss(self):\n",
-    "        # validation set loss\n",
-    "        if not hasattr(self, 'validation_loss_history'):\n",
-    "            self.validation_loss_history = []\n",
-    "        p_hat = self.predict_proba(self.X_validation)\n",
-    "        loss = np.mean(-self.y_validation * np.log(p_hat) - \\\n",
-    "               (1-self.y_validation) * np.log(1-p_hat))\n",
-    "        self.validation_loss_history.append(loss)\n",
-    "        \n",
-    "        t = self.early_stopping\n",
-    "        if t and len(self.validation_loss_history) > t * 2:\n",
-    "            recent_best = min(self.validation_loss_history[-t:])\n",
-    "            previous_best = min(self.validation_loss_history[:-t])\n",
-    "            if recent_best > previous_best:\n",
-    "                return True\n",
-    "        return False"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Testing and Evaluating the Customized Model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from sklearn.datasets import load_breast_cancer\n",
-    "from sklearn.model_selection import train_test_split as tts\n",
-    "\n",
-    "dummy_data = load_breast_cancer()\n",
-    "X, y = dummy_data.data, dummy_data.target\n",
-    "\n",
-    "X_train, X_test, y_train, y_test = tts(X, y, test_size=0.2, random_state=103)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = LogisticClassifier(learning_rate = 0.01,\n",
-    "                           tolerance=1e-5,\n",
-    "                           max_iter=2000,\n",
-    "                           early_stopping=3,\n",
-    "                           validation_set=(X_test, y_test))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model.fit(X_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "y_pred = model.predict(X_test)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# defining the metrics function\n",
-    "def accuracy(predictions, actual):\n",
-    "    return sum(predictions == actual) / len(actual)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 23,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "0.9736842105263158"
-      ]
-     },
-     "execution_count": 23,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "accuracy(y_pred, y_test)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

From 8626fe6a9801840f33c1f03bd182fd245d728ac8 Mon Sep 17 00:00:00 2001
From: Steven Kolawole <45284829+SteveKola@users.noreply.github.com>
Date: Wed, 22 Apr 2020 22:37:43 +0100
Subject: [PATCH 7/9] Create readme.md

---
 Kolawole_Steven/readme.md | 10 ++++++++++
 1 file changed, 10 insertions(+)
 create mode 100644 Kolawole_Steven/readme.md

diff --git a/Kolawole_Steven/readme.md b/Kolawole_Steven/readme.md
new file mode 100644
index 0000000..cd782a5
--- /dev/null
+++ b/Kolawole_Steven/readme.md
@@ -0,0 +1,10 @@
+A Customized Logistic Regression with Nesterov Accelerated Gradient with Early Stopping option:
+
+
+Nesterov Accelerated Gradient is theorized to converge by at least, a 10 times faster rate than Stochastic Gradient Descent, and over 25 times faster rate than the naive batch gradient descent.
+
+Nesterov Gradient combines the properties of Stochastic Gradient Descent -which supposedly have the properties that allows it to “jump” out of shallow local minima giving it a better chance of finding a true global minimum- with a 'smarter' momentum, that has a somewhat prescient notion of the global minimum, and knows to slow down before the hill slopes up again.
+
+But there is a catch; 
+Converging too fast makes it easier for the model to overfit, causing the well-known bias-variance tradeoff.
+My way of avoiding that is to introduce Early Stopping, which works by simply waiting for a certain number of epochs with no improvement in validation loss.

From ab9a6da8a2d028cce57906fb34efb6c7ae734340 Mon Sep 17 00:00:00 2001
From: Steven Kolawole <45284829+SteveKola@users.noreply.github.com>
Date: Wed, 22 Apr 2020 22:38:33 +0100
Subject: [PATCH 8/9] Add notebook via upload

---
 .../Logistic Regression from Scratch.ipynb    | 322 ++++++++++++++++++
 1 file changed, 322 insertions(+)
 create mode 100644 Kolawole_Steven/Logistic Regression from Scratch.ipynb

diff --git a/Kolawole_Steven/Logistic Regression from Scratch.ipynb b/Kolawole_Steven/Logistic Regression from Scratch.ipynb
new file mode 100644
index 0000000..6031d92
--- /dev/null
+++ b/Kolawole_Steven/Logistic Regression from Scratch.ipynb	
@@ -0,0 +1,322 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Logistic Regression from Scratch\n",
+    "## - Steven Kolawole                                "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Intro\n",
+    "\n",
+    "Logistic Regression is simply a Linear Regression with a Sigmoid function at its end.\n",
+    "\n",
+    "The Sigmoid function generates probability (i.e. outputs between 0 and 1) for all values of X."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "sns.set()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEMCAYAAADQ553CAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZhcZZn38W9V9Z49nc6eANlukkAIISzK4hZ0BLcRRMVhZBRxw1dHR8cRZkYdGX11RhnXcYy8ohgXcNyGgIqJgywhhCQEstxZCFk76SVrd3qtqvePczpUikq6Oqnqqq7+fa6rr65zznOqfv101V2nnnPqnEgymUREREpHtNABREQkt1TYRURKjAq7iEiJUWEXESkxKuwiIiVGhV1EpMSUFTqAnJqZ/QB49yma7HD3s83sBWCxu38hx4//18BtwFwgAawDvu7uP0tpkwRucvd7c/nYfch4M8HfnvH5bGaTgV3Aq9z9TxmWvxJYfpK7b3b3MblJempm9jIg6u6PhdN571cziwC3AO8B5gDdwFrgi+7+cL4eV/JLW+zF76PAhPDnknDem1PmXZyvBzazW4FvAt8GLgAuBR4AfmJmqW82E4D785UjCz8DJuXgfhbwYr/2/MzJwf1m6xFgZsp0XvvVzKLAr4A7gXsInl9XAU8DvzOzG/P12JJf2mIvcu5+GDgMYGZV4ewD7r6vHx7+/cD33P0HKfM2mJkRvOHcE2bsjywn5e5tQFsO7qqxwH9LJHWiH7LcBlwDXOTu61Lmf8rMhgJfN7Nfu3trnnNIjqmwl5ZJZvYbYBHBm8E33P1fexaa2VuAzwEGvAAsBr7q7omT3F8cuNzMRoRvMD3+DhiScr8nDBmY2SeBjwBjgN8BO4EL3P2V4bDHQ8BbgK8DU4DHgJuBTwM3Ae3A19z9SymP8R7g48B0YA9wl7t/M1x2MylDMWZ2FsGnjFcADcDxPjhdmYZ7MjxuEnhv+LdcDOwg6N//SlnnJuBTwIxw+Rfd/Z5wKC0G/D8zuznsq/R+7a0PPg18FfgMUAs8BXzY3Tee5M96P/DrtKLe43PADwj+FxmHhVLnhUOG1cBYgk8+nyb4H0xz9+0p66wHfunud5jZFOBrwGsJ3piXAx93970nyStZ0lBMaXkvsJRgPPw/gDvN7CoAM7sG+HE4fy5Bcfko8I+nuL+vEHw832tmvzGzvzOz+e7e6O4vZFrBzD4C/BNBcbmQ4A3kI2nNKoDPAzcCryYoBOuAFoKC+F3gi2Y2J7zPjxMMCd0FzAtzfcXMPpHh8csJ3jhqgMvDPvn0Kf7GXPu/BFkvBP4MfCd8o8HM3g7cTfCGej7wb8BiM3stwd8dBz4GvDX9TrPsg2nAu4DrCIrlWcA3MoUMP/3NAVZkWu7u+919pbvH+/C33wD8kmDI7lcEb1zvSHnM+eFj/tDMhgB/IijoLwdeR/C8WGZmFX14TMlAW+yl5T53/8/w9pfM7NPAQoKx288A33b3u8Pl28xsGPA9M/uXTFvt7n6fme0hKDavBd4IYGZrCLbU1mfI8AmCre2eLbu/NbMr0tpEgNvdfVV4f38kKGz/4O5JM/siwRvOXDPbSPAmdJe7Lw7X32Jm04C/N7Ovpt33IoJPJK9z953h/f8fgn0DvfFwKzTVPHd/Pot1e9zt7j8PH/eTBDsmLyEoch8Dfuzu/xG23RoOeUTdvTEY4eKwux9IvcNwB2c2fVAOfKBnC93MvkEwfp7JqPD3oT78bb3Z5+5fT8n9I+CdwBfDWe8CVrj7ZjO7heBT3809bx5m9k6gieCN6Sc5zDXoqLCXls1p04cIPh5DsAV5sZl9MGV5NFx+NpCxeLn748DjZhYDLiIo7h8BHjSzGe7e2dPWzGoJthKfSLubx4D5afO2ptxuBZ5392T4mG1hkasE6oBx4X2keoSg2I1Nm38e0NRT1EMZt0ozeB2QPq69K8t1exz/H7j7ofDv6NkCPR/4UWpjd78ri/vMtg+SwJaU5YdSHjtdc9h+dBaPn63059A9wB1mNhfYSLD13vNGcyHB33U47KMeNcDsHGYalFTYS0umj809O+Q6gS8TDMek250+Ixz//Afg8+6+L9yqWgmsNLM/E4ydzwNWpazWHf7OZoivK236ZOP87SeZHzvJ/SRJ2wlJ8Ldn4wV3f0lfnEKm109Hhnk9edKzZivbPki4e3dam/S+AMDdO81sNXBZpuVmNhP4FvC3mT6ZmVmmv/2EHdjuvtXMHifYan+YoJD3HCbbCawnw7ATuf0UMShpjH3wWA/MdPetPT8EW5B3kvnF30YwjJDpkLdDBAW0IXVmuIN1J8EYa6r06ay5+xGCN5704ZwrCLauD6bNXwuMCQtTj4Wn+/gpOoGYmdWkzJt5ssYnsTE9i5n90Mx6hi8ynkP7NPogW98H3mhm8zIs+yTBENIL4XQXMDxlebZ/+z0EQys3AL91956s64FzCL4n0PN8bCDY+Xt+X/4IeSltsQ8eXwAeMLPngF8Aswh2Ui5195dsZbp7k5l9mWAn5vBwnTaCF90XgHvShjt69KyziWBr/maCrcI/nWH2r5nZtvB+XkUwHPRP4Zh8atvlBMdh32tmHyYYivg6Z24FQeH9nJl9i+DN6uY+3seXgZ+b2UrgDwQ7jt8JvD5cfhSYY2Zj3b0hbd2+9EG2/ovg6KQ/mtlnCPpuOHArwU7nd6Yc6vgEcKuZPUbwSeFrZP50ku5nBDvs/4pgjL3Hj4HbCfrjHwg+lXyJ4M0k074b6QNtsQ8S7v4QwaGENwLPERT1HxIc8nayde4Il18NPErwguv5MsutJ1nt28C/hz/PELyB/Irsh0My5fguwc7ffwgzfJzgsLivZGgbJzg2eydBobqPoAidkXAH6geBtwGbgPcRbNX25T5+BXyYIP96gp2pN6V8w/NLwIcIhrnS1826D/qQJw68geAN5zZgDfB7gqNrXt2zEzj0QYJDaJ8k+NLUf5FhCC/DYxwGfk1QuB9Mmd9G8Lw6Biwj2H9QFj5u+pua9FFEV1CSXDKzvwCedfc9KfMeAva4+3sLl0xk8NBQjOTau4FzwmGQZoItwkUEh0uKSD9QYZdcu43gSzQPAsMIhi3e6e7LCppKZBDRUIyISInRzlMRkRJT6KGYSoKvkteT+cs1IiLyUjGC0zo/RYbDTgtd2C8mOFGSiIj03ZUEhyKfoNCFvR7g4MFWEom+j/XX1g6lubkl56FyoVizFWsuKN5sxZoLijebcvVdX7JFoxFGjRoCYQ1NV+jCHgdIJJKnVdh71i1WxZqtWHNB8WYr1lxQvNmUq+9OI1vGIWztPBURKTEq7CIiJUaFXUSkxGQ9xh6e4e9x4A3pl0ULL3m1mODMcI8QXMUl/bzQIiLSD7LaYjezSwkOqZl1kib3Are5+yyCc3u/LzfxRESkr7IdinkfwelGX3L18PBCvdXu3nP5sR8QnNpURGTQSyaTx38SaT/5ktVQjLvfAnCSk/lP5MRjKeuByWecTERKRld3gmPtXRzr6Ka9M057RzftXXE6OuN0dMXp7ErQ2R2nqzsR/MQTdIe/4/Ek3fEE3fEk8USSeCKYF08Gh0nHw8OlE8me35AMp4NiSngbopEI3fEEPefICo4uDJaFN4MiDCR7Zhy/zfHbyZ6LXaXO72OfRCLwwTefx8Jz0y/be+ZycRx7lBP/pggnv35lRrW1Q0/7wevqhp32uvlWrNmKNRcUb7ZizQWFy9bVnaD5cBuNh9poPNjGgSPtHDzazqEjHRxq6eBIaydHj3VytLWTzu7sS0JZLEpleZTyshhlZVHKy6KUxaKUxSLh7yhl5VGqYhFi0SjRaIRYNEI0GiEaSf3N8duRSIRIJCjshL8jEYL5ABGIcOK8SHjByEh4I5JyAcnj64XrvngzktImdX6KcEE0CpfMm0jtiOrji3L1v8xFYd9NcM6CHuPJMGRzKs3NLaf1pYG6umE0Nh7t83r9oVizFWsuKN5sxZoL+idbS1sXexpb2NPUyp6mVvYfOEbDwTaaj7STPppQWRFjxJAKRg+vYuSQCibXDWFoVTk1VWXUVJVRXVlGdUUZVRUxKitiwe/yGBXlMcrDIh6NZLz+dk4U2/8y0dl9PE9fskWjkVNuEJ9xYXf3HWbWbmaXu/tjBJdfe7C39USk+MQTCXbub8F3HmJ7/RG21x+h6XD78eXVlWWMH13DjEkjeNnc8YwZWcXo4VWMHlbJqGGVVFUEJaXYCuhgc9qF3cyWElxIdxXBRWq/Fx4SuZrcXDxYRPrBgSPtPLO1iXXbmtm8+xBtHcG31GuHV3HOhGG86sJJTBk7lEl1Qxk5tOL40IQUrz4Vdnc/O+X2NSm3nyG4uriIDAAHj3bw5Ib9rNiwj537gxNPjR1ZzaVzxnPu1JHMmjKSkUMrC5xSTlehTwImIv0kkUjyzNYmlq/Zw/oXDpBMwjkThvO2V07nghljmFBbo63xEqHCLlLiOrriPLqunj88tYuGQ22MHl7JtS87m5efN57xo2sKHU/yQIVdpER1xxP8eV09v3lsO4dbOpk+cThvfcU0LrI6YlGdJqqUqbCLlKC1W5r46bItNBxsY8bkEXzgTXOxqaMKHUv6iQq7SAk51NLBkj9sZpU3MmnMED56/TzmTa/V2Pkgo8IuUiKeeG4f9/5hM13dCd561TT+4tKplMU05DIYqbCLDHCdXXGWPLyZR56pZ+bkEfzNNbO1U3SQU2EXGcD2NrVw54+eZldDC9e+7CzecuU52jEqKuwiA9Xze4/w9V+sIx5P8LG3zWPe9DGFjiRFQoVdZABat62Zb//qWUYPr+Kj189j3CgNvciLVNhFBpgn1u/j7gc2MqluCF/4wOV0d3QVOpIUGQ3GiQwgT3sDi/9nAzMnj+Dvb1zAqOFVhY4kRUhb7CIDxHPbm/nPX69n+sQRfPT6C6isiBU6khQpbbGLDABbdx/mm//9LBPHDOFjb5unoi6npMIuUuSaDrXx9V+sY9TQSj7+9vnUVJUXOpIUORV2kSLW0RXnm//9LPFEko+97QJGDKkodCQZAFTYRYpUMpnkngc3sauhhfe/aQ7j9G1SyZIKu0iR+sNTu1ixYT9vuWqavnwkfaLCLlKEduw7yn1/2saCWXW84WVnFTqODDAq7CJFpqs7zuL/2cDQmnJufv25OuWu9JkKu0iR+eUj29nT1Mp7rpnN0GodASN9p8IuUkR850F+t3Inr7xwEudPqy10HBmgVNhFikRnV5y7l26kbmQ1N7xqeqHjyACmwi5SJJau2EHjoXbe/fpzqarQ2T7k9KmwixSBhkNtLF2xk0tmj2X2WbrotJwZFXaRIvDTh7cQi0Z4+6tnFjqKlAAVdpECW7etibVbm3jT5WczalhloeNICVBhFymg7niCJQ9vYfzoGq6+eEqh40iJUGEXKaA/r6un4WAb73jNDMpiejlKbuiZJFIgnV1xfvvYdmZMHqFj1iWnVNhFCmTZ6j0caunkuqum6bQBklNZHSxrZjcCdwDlwF3u/q205QuA7wIVwC7gr9z9UI6zipSMto5ulq7YwdxzRmNTdXij5FavW+xmNgm4E7gCmA/camZz0pr9B/BP7n4B4MDf5TqoSCn5w1O7aGnr4q1XTSt0FClB2QzFLAKWufsBd28F7geuT2sTA4aHt2uAttxFFCktre1d/O6pnSyYVcc5E4b3voJIH2UzFDMRqE+ZrgcuSWvzceD3ZnYX0Apc2pcQtbVD+9L8BHV1w0573Xwr1mzFmguKN1sucy1/eDNtHXFufuPcnNzvYOizXCrWXJC7bNkU9iiQTJmOAImeCTOrBr4PLHL3lWb2ceCHwLXZhmhubiGRSPbeME1d3TAaG4/2eb3+UKzZijUXFG+2XObq6o7z6//dynnTRjO0PHrG9zsY+iyXijUX9C1bNBo55QZxNkMxu4EJKdPjgb0p0+cBbe6+Mpz+LvDKrNKJDDKPPbuPI8e6eP2luiqS5E82hf1h4DVmVmdmNcB1wEMpy7cCU8zMwuk3A0/lNqbIwJdIJHlo5U7OHj+Mc6eOLHQcKWG9FnZ33wPcDiwH1gJLwiGXpWa20N0PAjcDPzezdcB7gL/JY2aRAWn15kYaDrZxzWVn6bh1yausjmN39yXAkrR516TcfhB4MLfRREpHMpnkwSd3MHZUNQtm1RU6jpQ4ffNUpB9s2X2Y7fVHed0lU4lGtbUu+aXCLtIPlq/ZQ3VlGS+fO77QUWQQUGEXybPDrZ2s2tTAFedPoLIiVug4MgiosIvk2SPP7CWeSPLKCycWOooMEirsInkUTyT437V7mHP2KCbUDil0HBkkVNhF8mjd1mYOHOngVRdOLnQUGURU2EXyaNmaPYwaVsn8mbqQhvQfFXaRPNl/8Bjrtx/glfMnEovqpSb9R882kTx5dF09kQhcMU87TaV/qbCL5EEikeTx5/Zx/rRaRg2rLHQcGWRU2EXyYP0LBzh4tIMrzp/Qe2ORHFNhF8mDR9fVM6SqjAtmjCl0FBmEVNhFcqy1vYs1Wxq5bO54ysv0EpP+p2edSI49uWE/3fGkhmGkYFTYRXLs0XX1TBk7lLPGF++1NaW0qbCL5NDuxhZe2HeUy7W1LgWkwi6SQ0+s30c0EuGyOeMKHUUGMRV2kRxJJJOs3NDA3HNGM3xIRaHjyCCmwi6SI9v2HKb5SLu21qXgVNhFcmTFhv1UlEWZP1PHrkthqbCL5EB3PMFTGxuYP3MM1ZVZXSNeJG9U2EVyYOOOg7S0dXHpbA3DSOGpsIvkwIr1+6mpLOO8aTrvuhSeCrvIGeroirN6SyMLz63TKQSkKOhZKHKG1m1rpqMzzqVzxhc6igigwi5yxp7auJ/hQyqwKSMLHUUEUGEXOSMdnXHWbWvmIqsjGo0UOo4IoMIuckbWPd9MZ3eCi21soaOIHKfCLnIGVm1qYHhNObM0DCNFRIVd5DR1dMV5ZlsTC2yshmGkqGT1FTkzuxG4AygH7nL3b6UtN+C7wChgH/AOdz+Y46wiReXZbc10diW42OoKHUXkBL1usZvZJOBO4ApgPnCrmc1JWR4BfgN8yd0vANYAn85PXJHiscobGFZTzqypGoaR4pLNUMwiYJm7H3D3VuB+4PqU5QuAVnd/KJz+V+BbiJSwzq44z2xtZsGsOmJRjWhKcclmKGYiUJ8yXQ9ckjI9A9hnZt8HLgQ2Ah/JWUKRIvTs8wfo6Iqz8FwdDSPFJ5vCHgWSKdMRIJF2H68ErnL3VWb2L8BXgZuzDVFbOzTbpi9RV1e815Us1mzFmguKN1t6rvW/38ywmgquXDCFWKywW+wDpc+KRbHmgtxly6aw7wauTJkeD+xNmd4HbHH3VeH0TwiGa7LW3NxCIpHsvWGaurphNDYe7fN6/aFYsxVrLijebOm5uuMJnly/j4tm1XHgQGsBkw2cPisWxZoL+pYtGo2ccoM4m02Nh4HXmFmdmdUA1wEPpSx/HKgzswvC6TcCT2eVTmQA2rTjIG0d3SzQ0TBSpHot7O6+B7gdWA6sBZa4+0ozW2pmC929DfhL4Htmth54NfCJfIYWKaTVmxuprIgx9+xRhY4iklFWx7G7+xJgSdq8a1JuP8mJO1RFSlIikWT1libmTaulvCxW6DgiGek4LZE+2LrnMEdaO1kwS8MwUrxU2EX6YPXmRspiEeZN15WSpHipsItkKZlMsnpzI3POHq0LVktRU2EXydKuhhaaDrdrGEaKngq7SJZWb24kEoH5M8cUOorIKamwi2Rp9eYmZk4eyfCaikJHETklFXaRLDQcamN3YwsLtLUuA4AKu0gW1m5uBGC+xtdlAFBhF8nC6i1NTK4bytiR1YWOItIrFXaRXhxu6WDL7kMsmKVhGBkYVNhFevHUhn0kk3DhTA3DyMCgwi7SixXP7aN2eCVTx53+dQNE+pMKu8gpdHTFWbO5kQtn1hGJRAodRyQrKuwip7B++wE6u+JcqKNhZABRYRc5hTWbGxlaXc6sKSMKHUUkayrsIicRTyRYu7WJi+eMIxbVS0UGDj1bRU5iy67DtLZ3c9l5EwodRaRPVNhFTmL1lkbKy6IssLGFjiLSJyrsIhkkk0nWbG5i7tmjqdK512WAUWEXyWBXQwvNR9q5UCf9kgFIhV0kgzVbmohE4AIVdhmAVNhFMlizuZEZk0bo3OsyIKmwi6RpOtTGzoYWnRtGBiwVdpE0a7Y0AXChzuYoA5QKu0ia1ZsbmTRmCONG1RQ6ishpUWEXSXHkWCebdx9igc4NIwOYCrtIirVbmkgmUWGXAU2FXSTF6s2NjBlRpXOvy4Cmwi4SauvoZsMLB1gwS+del4FNhV0k9OzzzXTHkxqGkQFPhV0k9LQ3MrymnBmTdO51GdiyKuxmdqOZbTCzLWb24VO0u9bMtucunkj/6OqOs+75Zi6cVUc0qmEYGdh6LexmNgm4E7gCmA/camZzMrQbB/wboFeFDDjrXzhIR2dcwzBSErLZYl8ELHP3A+7eCtwPXJ+h3WLgc7kMJ9JfVnsj1ZUxZp81qtBRRM5YNieangjUp0zXA5ekNjCz/wOsBlacToja2tM/tKyubthpr5tvxZqtWHNBYbJ1xxM8s62JS+ZMYML4zOPr6rO+U66+y1W2bAp7FEimTEeARM+EmZ0HXAe8Bph8OiGam1tIJJK9N0xTVzeMxsajp/OQeVes2Yo1FxQu23Pbmzl6rIvzzxmV8fHVZ32nXH3Xl2zRaOSUG8TZDMXsBlIv+jge2Jsy/bZw+SpgKTDRzP6cVTqRIrBqUwOVFTHOO2d0oaOI5EQ2W+wPA581szqglWDr/Naehe7+z8A/A5jZ2cCf3P3K3EcVyb3ueILVm5uYP2MMFeWxQscRyYlet9jdfQ9wO7AcWAsscfeVZrbUzBbmO6BIPvmuQ7S0dbHQdDSMlI6srtLr7kuAJWnzrsnQ7gXg7FwEE+kPqzY1UFke4/xptYWOIpIz+uapDFrxRIKnvZELZtRqGEZKigq7DFq+s2cYZmyho4jklAq7DFqrvJGK8ijnT9cwjJQWFXYZlIJhmAbmTR9DpYZhpMSosMugtPGFgxw91sVlc8YVOopIzqmwy6C0YsN+qivLdDSMlCQVdhl0OrviPL25kYVWR3mZXgJSevSslkHnmW3NdHTGNQwjJUuFXQadJzfsZ8TQCmyqTtErpUmFXQaVY+1drNvWxKWzx+lKSVKyVNhlUHnaG+mOJ7lUwzBSwlTYZVBZsWE/Y0dVc/b44r3YgsiZUmGXQePAkXY27TjIZXPGEYloGEZKlwq7DBqPPbePJHD5+RN6bSsykKmwy6CQTCZ57Nl6zp06krqR1YWOI5JXKuwyKGzZfZiGg23aWpdBQYVdBoVH19VTVRHTKXplUFBhl5LX3tnNU5sauPjcsVRW6EyOUvpU2KXkrdrUSEdXnCvmaRhGBgcVdil5jz5bz7jRNcyYNKLQUUT6hQq7lLS9Ta1s3nWIK84fr2PXZdBQYZeS9qc1e4hFI1w5b2Kho4j0GxV2KVkdnXEee66ei88dy/AhFYWOI9JvVNilZK3YsI+2jjivWjCp0FFE+pUKu5SkZDLJstV7mFw3VDtNZdBRYZeStG3PEXY1tPDqiyZpp6kMOirsUpKWrdlNdWVMl7+TQUmFXUrO4ZYOVm1q4OVzJ1BVUVboOCL9ToVdSs7DT+8mHk+y6OLJhY4iUhAq7FJS2jq6WbZ6DxdZHeNG1RQ6jkhBZPU51cxuBO4AyoG73P1bacvfDHwOiADbgb9x94M5zirSq/9du5e2jm5ef9lZhY4iUjC9brGb2STgTuAKYD5wq5nNSVk+HPgOcK27XwCsAz6bl7Qip9AdT/CHVbs4d+pIzpkwvNBxRAomm6GYRcAydz/g7q3A/cD1KcvLgQ+7+55weh0wNbcxRXq3Yv1+Dh7t0Na6DHrZDMVMBOpTpuuBS3om3L0Z+CWAmVUDnwa+kcOMIr1KJJM8tHInk+uGct45owsdR6SgsinsUSCZMh0BEumNzGwEQYF/xt3v6UuI2tqhfWl+grq6Yae9br4Va7ZizQWnn+2xZ/ayt6mVT9y4gLFjcz8MU4p9lm/K1Xe5ypZNYd8NXJkyPR7Ym9rAzCYAvwOWAX/b1xDNzS0kEsneG6apqxtGY+PRPq/XH4o1W7HmgtPPlkgkueeB9UyorWH25BE5//tKsc/yTbn6ri/ZotHIKTeIsynsDwOfNbM6oBW4Dri1Z6GZxYDfAj939y9klUokh55Yv4/65mN86C3nEY3q9AEivRZ2d99jZrcDy4EKYLG7rzSzpcA/AVOABUCZmfXsVF3l7rfkK7RIj+54gl8/up2zxg3jIqsrdByRopDVcezuvgRYkjbvmvDmKvRFJymQR57ZS9Phdm56nelkXyIhFWQZsDq64vz28ReYOXmEjoQRSaHCLgPWgyt2cLilk+teMV1b6yIpVNhlQGo41MbSFTu5ZPZYZk0ZWeg4IkVFhV0GpJ8+vIVYNMLbXz2z0FFEio4Kuww467Y1sXZrE2+6/GxGDassdByRoqPCLgNKV3eCJQ9vYfzoGq6+eEqh44gUJRV2GVB+/eh2Gg62cePVMymL6ekrkoleGTJgbN19mAef3MGV8yZw3jm1hY4jUrRU2GVAaO/sZvH/bKB2eBXveI12mIqcigq7DAj3Ld9G46E23nvtbKordYFqkVNRYZeit3ZrE8vX7OHqi6dgU0cVOo5I0VNhl6K2/8AxvvfbDUwdO5TrXjGt0HFEBgQVdilabR3dfOO/nyUWjXDbW8+nvCxW6EgiA4IKuxSlRDLJ9x/YyL7mY3zwzXMZM7K60JFEBgwVdilKv3zkeVZvbuSGV01n9tk6c6NIX6iwS9F5cMUOHnhiB6+YP1HfLhU5DSrsUlT+tGYP9/1pG5fMHstNr9XFM0ROhwq7FI0/PrWTH/3OmTe9llveMEfXLxU5TfqmhxRcMpnkoZU7uW/5NmafNYoPveU8nQdG5AyosEtBJZJJfr5sK79/ahdXzp/EXy2aSXmZirrImVBhl4I51t7N3Us3snpzI4sumsxH3rGA5uaWQscSGfBU2KUgduw7yrd/9SwHjnTwjlfP4OqLp2hMXSRHVNilXyUSSZat3s3Pl342r5gAAAtySURBVG9jWE05f3/jAmZMHlHoWCIlRYVd+s3uxhbueXAT2/YeYd70Wt577WyG1VQUOpZIyVFhl7w71t7FAyt28PuVu6iuLON9b5jDZXPH6Rh1kTxRYZe86eiKs+zp3SxdsYPW9m4uP288N7x6hrbSRfJMhV1y7nBrJ8tX72b5mj0cPdbFvOm1vPWqaUwdN6zQ0UQGBRV2yYlEMonvOMijz+7jqU376Y4nuWB6La+/7CxmTRlZ6Hgig4oKu5y2RCLJ83uPsGZLI09u3M+BIx1UV8a48oKJXL1wCuNH1xQ6osigpMIufdJ0uA3feYiNOw7y7PPNHD3WRSwaYc7Zo7nhVTOYP2MMFeW6IIZIIamwy0kdPdbJ7sZWXth3hO31R9m+9wjNR9oBGFJVxnnTapk/YwznTxtNTVV5gdOKSI+sCruZ3QjcAZQDd7n7t9KWzwcWA8OBR4APuHt3jrNKjiWTSVraujhwpIPmI+00HGyj4eAx9h9sY09TK0daO4+3rR1exTkThvHaS6Zw7tRRTKobQlSHK4oUpV4Lu5lNAu4ELgI6gMfNbLm7b0hpdi9wi7uvMLPvA+8DvpOPwPJSyWSS7niC9s44bZ1x2ju6ae+M09rexbH2blrbu2lp66KlrYuO7gSNB45xuLWDwy2ddHYnTrivIVVljB1Vw7xptUwcM4TJdUOYOn4Yw3WIosiAkc0W+yJgmbsfADCz+4Hrgc+H02cB1e6+Imz/A+Bz5LmwJ5JJVm3cz/7Go6dsl0ym3OaEiRd/ZWhzwnrJcH7yxabJZDKcHzRIJE+cXzOkkqNH20kSTCcSSRLJE28nksngdgLiiQSJRJLuRJJ4PEk8kSAe3u6KJ+juTtAdT9AVT9DVnaCzK0FXd5yO7gSdXfET8mYSicCw6nJGDKtiaFUZ0yeOYPiQCmqHVzF6eCWjh1cxdlQ1QzSkIjLgZVPYJwL1KdP1wCW9LJ/clxC1tUP70hyArbsO8bnFK3pvWKSi0QjRSIRoNEJZLEIsGiEWjRKLRYjFopRFI5SVRSmLRSmPRSkrj1FdXU55WZSKshgV5VEqymNUVsSoqiijKvxdU1VGdWXwe2h1BUOqyxlaU86QqvIBcZKturriPNa9WHNB8WZTrr7LVbZsCnuUE7ZpiQCJPizvVXNzC4lEL5ucaUZUxVh8+9XU7z9ywgNncrKh4J6vtEfSVo682OD4dE+bCJHj9xeJBLcjPfcV3o5GI9SNGUZzcwvRnjaRCNFo+Ls/x6aTCdpbO2hv7QCCJ05jL59yCqVYsxVrLijebMrVd33JFo1GTrlBnE1h3w1cmTI9HtibtnzCKZbnzbjRNUTj8f54qD4bUl3OsUoddCQi/S+bS9U8DLzGzOrMrAa4DnioZ6G77wDazezycNZNwIM5TyoiIlnptbC7+x7gdmA5sBZY4u4rzWypmS0Mm70L+JqZbQKGAl/PV2ARETm1rMYK3H0JsCRt3jUpt5/hxB2qIiJSILpqsIhIiVFhFxEpMSrsIiIlptDH48WAM/riTDF/6aZYsxVrLijebMWaC4o3m3L1XbbZUtplPJVqJNnbd9Hz6wrgz4UMICIygF0JPJo+s9CFvRK4mOA0BMX5TSMRkeITI/hi6FMEJ2c8QaELu4iI5Jh2noqIlBgVdhGREqPCLiJSYlTYRURKjAq7iEiJUWEXESkxKuwiIiWm0KcUyJqZ/QsQd/fPhtMjgR8D04BG4AZ335e2TgT4CvAGgsv1vc/dH8tDtrHA71NmjQDq3H1oWruzgOeAbeGs/e7+ulznSXvMdwNfAvaHsx5w99vT2vTal3nKdjnwNaACaAbeE164JbVNv/WZmd0I3AGUA3e5+7fSls8HFgPDgUeAD7h7dz6ypD3uPwM3hJMPuPunMix/D3AwnPW99Ox5zLYcGAt0hbPe7+5PpixfBHwVqAZ+5u539EOmW4DbUmadA/zI3W9LadOvfWZmw4HHgTe4+wvZ9IuZTQXuJehfB97l7i3ZPF7RF3YzG0HQAe8Evpyy6AvAn939WjO7CfgP4O1pq18HzAbmADOAB8xsdq5fjO7eAMwP80aBPxJcnCTdQoILlbw/l4/fi4XAx939J6dok01f5sOPgTe5+zozew/BBVrenNamX/rMzCYBdwIXEXyT73EzW+7uG1Ka3Qvc4u4rzOz7wPuA7+Q51yLgtcCFBNcWfsjM/tLdf5nSbCHwDnd/Ip9ZMmSLALOAszK9psysGrgbeAWwi+D193p3z+sV1tx9McEbMGY2F/gV8Nm0Zv3WZ2Z2KfA9gr7qS798G/i2u//UzP4R+Efg77N5zIEwFPNmYAvw72nzryUoDAA/AV5vZuUZ2vzU3RPuvhnYCbw8n2GBvwGOhRcnSXcxcJ6ZrTWzZWZ2fp6z9Dzmu83sWTO718xGZWiTTV/mlJlVAne4+7pw1jpgaoam/dVni4Bl7n7A3VuB+4HrU/KeBVS7+4pw1g+At+UpS6p64BPu3unuXcBGXtpPC4HPmNk6M/ummVX1Qy4AC3//3syeMbPb0pZfAmxx9+1h4b+X/umzVN8BPuPuTWnz+7PP3gd8mBevBd1rv4Svv6sInofQx+db0Rd2d/+hu3+Jl55LZiLBk56wc44AdSdrE6oHJucpKmYWI9hS//RJmrQT/BMXAP8G/MrMKvKVJ1QP/Aswj2Dr4JsZ2mTTlznl7h3ufi8c/5TzWYItq3T91We9PVf69bnUw93X97yZmNlMgiGZpT3LzWwosAb4JEEfjSTYsusPowg+nf4l8BrgA2Z2dcrygvRZj/DTTrW735c2v1/7zN1vcffUkx1m0y9jgCMpn4T61HdFMxRjZm8jGG9NtcndF51klfTzW0YIxtFTRQk+vp6qTZ/0kvMvCN6Jn820bs/+gdBSM/siwVDRM2eSKYtcPW2+zItj1amy6cu8ZAuL9D0Ez8V/TV83n32WprfnSs6fS30RDik8AHzS3bf0zA/HXK9JaffvBB/zMw0F5lQ4jHF8KCMcnroG+EM4q6B9BryfYBj3BIXss1A2/ZLehgxtTqpoCnv4rnpfrw1ftAcYD+w2szJgGMEOuFS7Cc6A1mM8L34cykfOtwA/Pdm6ZvYRgvHinpwRXtzpdEYy5TKzEWb2t+7eU1QjQKb9C9n0ZU6zhfmGAr8JH+vN4VBDepu89Vma3QSnQO2R/lzJ+XMpW+FO5l8AH3P3n6Ytmwoscve7w1n56p9Mua4AKt39jyd57EL2WQXBGPbNGZYVrM9C2fRLAzDCzGLuHg/bZ913RT8UcwpLgb8Ob7+dYOdf+j9nKfAuM4uZ2QyCnRdP5THTyzj1+eVfAbwXwMxeQXDqzU15zNMCfCrceQPBkQK/zNAum77Mh3uBrcDb3f0lpx4N9VefPQy8xszqzKyGYMf7Qz0Lw6N12sMiC3ATkNedgABmNoVgiOrG9KIeagO+bGbnhDszP0zm/3E+jAS+YmZVZjYMeHfaYz8JmJnNCIcpb6Qf+iw0D9gc7i9JV8g+gyz6JXz9/ZkXD2L46/Q2pzKQC/s/ApeZ2XrgQwT/HMzsTWa2OGxzP7CeYMfcr4H3untbHjNNI3g3Ps7MPmBmnw8nPwpcbWbPEYwXv9Pd8/bRNHynvwH4jpltJDji41Nhrs+b2QfCphn7Mp/M7EKCHeOXA6vDnaNLw2X93mfuvofgo/hyYC3Bp4SVZrbUzBaGzd4FfM3MNgFDCY7iybe/A6qAr4Z9tDbsn6VmttDdGwmGHH5LcEhchJceaJAX7v4/BMNDa4Cngbvd/Ykw40R3byfYYv4FsIHgDfn+k91fjmV6LRa8zwBO1S9mttjM3hQ2/RBwq5ltIPg0mfWhojofu4hIiRnIW+wiIpKBCruISIlRYRcRKTEq7CIiJUaFXUSkxKiwi4iUGBV2EZESo8IuIlJi/j/mMU566a77JQAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "<Figure size 432x288 with 1 Axes>"
+      ]
+     },
+     "metadata": {
+      "needs_background": "light"
+     },
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "def sigmoid_fxn(x):\n",
+    "    yhat = list(map(lambda i: 1 / (1 + np.exp(-i)), x))\n",
+    "    return yhat\n",
+    "\n",
+    "x = np.arange(-10., 10., 0.2)\n",
+    "logit = sigmoid_fxn(x)\n",
+    "plt.title(\"The Sigmoid Function Curve\", fontsize=15)\n",
+    "\n",
+    "plt.plot(x, logit)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Building my Customized Logistic Regression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class LogisticClassifier():\n",
+    "    def __init__(self, \n",
+    "            learning_rate=0.1, \n",
+    "            tolerance=1e-4, \n",
+    "            max_iter=1000, \n",
+    "            batch_size=32, \n",
+    "            momentum_decay=0.9, \n",
+    "            early_stopping=3, \n",
+    "            validation_set=(None,None)):\n",
+    "        \n",
+    "        # Gradient descent parameters\n",
+    "        self.learning_rate = float(learning_rate)\n",
+    "        self.tolerance = float(tolerance)\n",
+    "        self.max_iter = int(max_iter)\n",
+    "        self.batch_size=32\n",
+    "        self.momentum_decay = float(momentum_decay)\n",
+    "        self.early_stopping = int(early_stopping)\n",
+    "        self.X_validation, self.y_validation = validation_set\n",
+    "        \n",
+    "        # to construct the design matrix\n",
+    "        self.add_intercept = True\n",
+    "        self.center = True \n",
+    "        self.scale = True\n",
+    "        \n",
+    "        self.training_loss_history = []\n",
+    "        \n",
+    "    def __sigmoid(self, X):\n",
+    "        return 1 / (1 + np.exp(-X))\n",
+    "    \n",
+    "    # z-score normalization and intercept addition\n",
+    "    def __design_matrix(self, X):\n",
+    "        if self.center:\n",
+    "            X = X - self.means\n",
+    "        if self.scale:\n",
+    "            X = X / self.standard_error\n",
+    "        if self.add_intercept:\n",
+    "            intercept = np.ones((X.shape[0], 1))\n",
+    "            X = np.hstack([intercept, X])\n",
+    "            \n",
+    "        return X\n",
+    "    \n",
+    "    def __fit_center_scale(self, X):\n",
+    "        self.means = X.mean(axis=0)\n",
+    "        self.standard_error = np.std(X, axis=0)\n",
+    "        \n",
+    "    def fit(self, X, y):\n",
+    "        self.__fit_center_scale(X)\n",
+    "\n",
+    "        n, k = X.shape\n",
+    "        \n",
+    "        # add intercept column to the design matrix\n",
+    "        X = self.__design_matrix(X)\n",
+    "\n",
+    "        # used for the convergence check\n",
+    "        previous_loss = -float('inf')\n",
+    "        self.converged = False\n",
+    "        self.stopped_early = False\n",
+    "        \n",
+    "        # initialize parameters\n",
+    "        self.beta = np.zeros(k + (1 if self.add_intercept else 0))\n",
+    "        momentum = self.beta * 0  # to get the same shape and dtype as beta\n",
+    "        \n",
+    "        for i in range(self.max_iter):\n",
+    "            shuffle = np.random.permutation(len(y))\n",
+    "            X = X[shuffle, :]\n",
+    "            y = y[shuffle]\n",
+    "            \n",
+    "            # we'll add one more batch, incase the batch size doesn't divide n evenly\n",
+    "            extra = (1 if n % self.batch_size else 0)\n",
+    "\n",
+    "            for batch_index in range(n // self.batch_size + extra):\n",
+    "                batch_slice = slice(\n",
+    "                    self.batch_size * batch_index, \n",
+    "                    self.batch_size * (batch_index + 1) )\n",
+    "                X_batch = X[batch_slice, :]\n",
+    "                y_batch = y[batch_slice]\n",
+    "            \n",
+    "                beta_ahead = self.beta + self.momentum_decay * momentum\n",
+    "                y_hat = self.__sigmoid(np.dot(X_batch, self.beta))\n",
+    "                \n",
+    "                # gradient descent\n",
+    "                residuals = (y_hat - y_batch).reshape( (X_batch.shape[0], 1) )\n",
+    "                gradient = (X_batch * residuals).mean(axis=0)\n",
+    "                momentum = self.momentum_decay * momentum - self.learning_rate * gradient\n",
+    "                self.beta += momentum\n",
+    "\n",
+    "            # with minibatch, we only check convergence at the end of every epoch.    \n",
+    "            y_hat = self.__sigmoid(np.dot(X, self.beta))\n",
+    "            self.loss = np.mean(-y * np.log(y_hat) - (1-y) * np.log(1-y_hat))\n",
+    "            self.training_loss_history.append(self.loss)\n",
+    "            \n",
+    "            # early stopping\n",
+    "            if self.check_validation_loss():\n",
+    "                self.stopped_early = True\n",
+    "                break \n",
+    "                \n",
+    "            if abs(previous_loss - self.loss) < self.tolerance:\n",
+    "                self.converged = True\n",
+    "                break\n",
+    "            else:\n",
+    "                previous_loss = self.loss\n",
+    "        \n",
+    "        self.iterations = i+1\n",
+    "        \n",
+    "    def predict_proba(self, X):\n",
+    "        # add intercept column to the design matrix\n",
+    "        X = self.__design_matrix(X)\n",
+    "        return self.__sigmoid(np.dot(X, self.beta))\n",
+    "\n",
+    "    def predict(self, X):\n",
+    "        predictions = self.predict_proba(X).round()\n",
+    "        return predictions\n",
+    "\n",
+    "    def check_validation_loss(self):\n",
+    "        # validation set loss\n",
+    "        if not hasattr(self, 'validation_loss_history'):\n",
+    "            self.validation_loss_history = []\n",
+    "        p_hat = self.predict_proba(self.X_validation)\n",
+    "        loss = np.mean(-self.y_validation * np.log(p_hat) - \\\n",
+    "               (1-self.y_validation) * np.log(1-p_hat))\n",
+    "        self.validation_loss_history.append(loss)\n",
+    "        \n",
+    "        t = self.early_stopping\n",
+    "        if t and len(self.validation_loss_history) > t * 2:\n",
+    "            recent_best = min(self.validation_loss_history[-t:])\n",
+    "            previous_best = min(self.validation_loss_history[:-t])\n",
+    "            if recent_best > previous_best:\n",
+    "                return True\n",
+    "        return False"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Testing and Evaluating the Customized Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.datasets import load_breast_cancer\n",
+    "from sklearn.model_selection import train_test_split as tts\n",
+    "\n",
+    "dummy_data = load_breast_cancer()\n",
+    "X, y = dummy_data.data, dummy_data.target\n",
+    "\n",
+    "X_train, X_test, y_train, y_test = tts(X, y, test_size=0.2, random_state=103)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = LogisticClassifier(learning_rate = 0.01,\n",
+    "                           tolerance=1e-5,\n",
+    "                           max_iter=2000,\n",
+    "                           early_stopping=3,\n",
+    "                           validation_set=(X_test, y_test))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.fit(X_train, y_train)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "y_pred = model.predict(X_test)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# defining the metrics function\n",
+    "def accuracy(predictions, actual):\n",
+    "    return sum(predictions == actual) / len(actual)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.9736842105263158"
+      ]
+     },
+     "execution_count": 23,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "accuracy(y_pred, y_test)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

From 1812f1172d93fc9109ab9887a987c979d189611c Mon Sep 17 00:00:00 2001
From: Steven Kolawole <45284829+SteveKola@users.noreply.github.com>
Date: Wed, 22 Apr 2020 22:40:53 +0100
Subject: [PATCH 9/9] Update readme.md

---
 Kolawole_Steven/readme.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Kolawole_Steven/readme.md b/Kolawole_Steven/readme.md
index cd782a5..daff5c8 100644
--- a/Kolawole_Steven/readme.md
+++ b/Kolawole_Steven/readme.md
@@ -6,5 +6,7 @@ Nesterov Accelerated Gradient is theorized to converge by at least, a 10 times f
 Nesterov Gradient combines the properties of Stochastic Gradient Descent -which supposedly have the properties that allows it to “jump” out of shallow local minima giving it a better chance of finding a true global minimum- with a 'smarter' momentum, that has a somewhat prescient notion of the global minimum, and knows to slow down before the hill slopes up again.
 
 But there is a catch; 
+
 Converging too fast makes it easier for the model to overfit, causing the well-known bias-variance tradeoff.
-My way of avoiding that is to introduce Early Stopping, which works by simply waiting for a certain number of epochs with no improvement in validation loss.
+
+My way of avoiding that is to introduce Early Stopping, which works by simply terminating the iterations when there is no improvement in validation loss after a certain number of epochs.