major refactoring for v0.2, changing EVERYTHING

Author: alexbarcelo

.gitignore

@@ -6,4 +6,3 @@
 __pycache__
 build/
 dist/
-webhooks_automata.egg-info/

Dockerfile

@@ -0,0 +1,18 @@
+FROM python:3.11
+
+WORKDIR /app
+
+COPY requirements.txt /tmp
+
+RUN pip install --no-cache-dir -r /tmp/requirements.txt
+
+COPY webhooks_automata ./webhooks_automata
+
+ENV AUTOMATA_SETTINGS /config/wha_settings.yaml
+
+VOLUME [ "/config" ]
+VOLUME [ "/plugins" ]
+
+EXPOSE 8000
+
+CMD ["uvicorn", "webhooks_automata.app:app", "--host", "0.0.0.0", "--port", "8000"]

README.md

@@ -1,6 +1,5 @@
-
-Webhook receiver for automations
-================================
+Webhook server for triggering automations
+=========================================
 
 This project started as a dirty & quick hack to perform some deployment actions.
 Initially, the focus was on Git webhooks, but after a while this project was
@@ -10,57 +9,94 @@ I looked a little bit at other projects and didn't found any one that suited my
 needs, so I started this project... and then it grew a little bit and become
 something more versatile than the quick hack originally intended.
 
-This project is aimed at DevOps or sysadmin that have git repositories, typically
-with a VIP-branch, and automatic deployment. You may want to have different branches
-for stage and production or set up push permissions differently to the different 
-branches. This utility, when a webhook is received, will update the local Git repository
-and perform the commands in the settings.
+This project is aimed at DevOps or sysadmin that have git repositories in GitHub, Gitlab,
+or any other kind of webhook provider (git or otherwise).
 
-Quickstart
-----------
+This utility starts a server that listens to webhooks and when a webhook is received,
+it will perform the actions in the settings.
 
-- Create and activate a Virtual Environment:
+Quickstart demo with GitHub and ngrok
+-------------------------------------
 
-```bash
-$ virtualenv --python=/usr/bin/python3 /path/to/venv
-$ source /path/to/venv/bin/activate
-```
+Using [ngrok](https://ngrok.com/) is a quick way to obtain a publicly reachable HTTPS endpoint.
+We will be using that for this demo.
 
-- Install the package: `pip install webhooks_automata`
-- Create a `settings.yaml`
-- Set up a service (e.g. a systemd _service_ file) that does something along:
+- Create and activate a Virtual Environment in a Python 3.11 environment.
+- Install the package: `pip install webhooks-automata`
+  - If you want to use `main` branch, do
+  `pip install git+https://github.com/alexbarcelo/webhooks-automata`
+  instead.
+- Create a `settings.yaml`, like the one shown in [examples](examples/github_sample.yaml).
+- Start the server:
 
 ```bash
-$ /path/to/venv/bin/wh-automatad /path/to/settings.yaml
+$ AUTOMATA_SETTINGS=examples/github_sample.yaml uvicorn webhooks_automata.app:app
+INFO:     Started server process [57971]
+INFO:     Waiting for application startup.
+Current endpoints active:
+        /my_webhook
+INFO:     Application startup complete.
+INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
 ```
 
+- Run the ngrok ingress to this service with `ngrok http 8000`
+- The webhook endpoint will be something like `https://c94e-2600-70ff-f2f3-0-13ef-f451-8b4f-cc76.ngrok-free.app/my_webhook`
+(the host on this URL is given by ngrok and the path is set in the settings YAML file).
+- Go to your GitHub repository > Settings > Webhooks > Add webhook.
+- Fill the form:
+  - **Payload URL**: The webhook endpoint.
+  - **Content type**: Does not matter, but use JSON for future proofing it.
+  - **Secret**: Put the shared secret in the settings YAML file, in this demo it is `secret_token`
+  - _Add webhook_
+- When a webhook is created, GitHub makes a "ping delivery". You can check in the Recent Deliveries
+tab if there was a problem with the webhook endpoint.
+
+Deploying the webhook server
+----------------------------
+
+Instead of using ngrok, you may want to deploy this application into a
+domain/server under your control. The steps are similar like on the previous
+demo, but you will need to include a reverse proxy (and you probably want
+to include SSL termination on it).
+
+Take a look into [nginx](https://nginx.org/), [Traefik](https://traefik.io/)
+or [HAProxy](https://www.haproxy.org/).
+
+In addition to that, you will want to automatically start the Uvicorn server,
+which may done with either systemd (or whatever is used by your OS) or
+[supervisord](http://supervisord.org/).
+
 Settings
 --------
 
 Extra CLI features
 ------------------
 
-To force a manul trigger, you can use
-the built-in `wh-automata-trigger` command. Example usage:
+The `wh-automatactl` is a CLI that can be used to aid and complement the webhook server.
 
-```bash
-$ wh-git-trigger /path/to/setings.yaml entryname
-```
-
-This will react just as if a trigger had been received. Future versions of this tool will
-include more fine-grain control --e.g. dry-run, display status information...
+You can check its documentation by calling `wh-automatactl --help`. Keep in mind that this
+CLI will be accessible from within the virtual environment; that typically means that the
+script can be found in /pat/to/venv/bin/wh-automatactl.
 
 Implementation details
 ----------------------
 
-This project contains a minimal Flask server that answers the POST webhooks. A typical source
-of those webhooks call are Git servers such as GitLab, GitHub or Gogs. The server is started
-through the Flask's `app.run` method.
-
-Not a lot of traffic is expected, but you may want to set up a reverse proxy in front
-of the Flask server, or add some fancier method like a WSGI or uWSGI or similar layer.
-
-Typical webhooks expect the a quick reply (in general, HTTP connections are intended to be short
-lived) so there is a worker/tasks approach. There is a very simple implementation base on
-`Threading` and a shared `Queue`. More complex implementations may be added in the future
-(pull requests are welcome).
+This project contains a minimal [Starlette](https://www.starlette.io/) ASGI
+application that answers the POST webhooks. My recommendation is to start the
+application through [uvicorn](http://www.uvicorn.org/).
+
+The actions are run as [Starlette Background Tasks](https://www.starlette.io/background/)
+because generally providers (GitHub, Gitlab, etc.) expect a quick webhook and
+do not care on the return. A 200 success response is given for all properly
+authorized requests and actions are run after that; a 200 success response
+**does not mean** that actions have run successfully (in fact, the actions start
+**after** the response).
+
+Most internal code assumes that things can be made asynchronously, so the
+webhook server should be a lightweight but capable process --the heavy-lifting
+will be performed by asynchronous actions. This is assured by using 
+[`asyncio-subprocess`](https://docs.python.org/3/library/asyncio-subprocess.html)
+and other async-centric tools.
+
+Feel free to use other ASGI servers, or embed the Starlette routes into a more
+complex application.

examples/github_sample.yaml

@@ -0,0 +1,9 @@
+automata:
+- endpoint:
+    suffix: my_webhook
+    provider: github
+    secret_token: secret_token
+  action:
+    type: commands
+    commands:
+    - echo "Hello World"

pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "webhooks-automata"
+dynamic = ["version"]
+description = "Webhook receiver for flexible automations"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "Apache-2.0"
+authors = [
+  { name = "Alex Barcelo", email = "alex@betarho.net" },
+]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: Apache Software License",
+    "Operating System :: OS Independent",
+]
+dependencies = [
+    "starlette",
+    "uvicorn",
+    "pydantic>=2.0",
+    "click",
+    "pyyaml>=6.0.1",
+]
+
+[project.urls]
+"Source code" = "https://github.com/alexbarcelo/webhooks-automata"
+
+[project.scripts]
+wh-automatactl = "webhooks_automata.scripts:cli"
+
+[tool.hatch.version]
+path = "webhooks_automata/__init__.py"

requirements.txt

@@ -0,0 +1,5 @@
+starlette==0.30.0
+uvicorn==0.23.1
+pydantic==2.0.3
+click==8.1.6
+PyYAML==6.0.1

setup.py

@@ -0,0 +1 @@
+__version__ = "v0.2.0"

webhooks_automata/__init__.py

@@ -0,0 +1,53 @@
+from typing import ClassVar, TYPE_CHECKING
+import yaml
+
+from starlette.applications import Starlette
+from starlette.background import BackgroundTask
+from starlette.responses import JSONResponse
+from starlette.routing import Route
+from starlette.config import Config
+
+from .settings import Settings, Automaton
+
+if TYPE_CHECKING:
+    from .union_models.actions import ActionBase
+    from .union_models.endpoints import EndpointBase
+
+config = Config(".env")
+settings_path = config("AUTOMATA_SETTINGS", default="wha_settings.yaml")
+with open(settings_path) as f:
+    settings = Settings(**yaml.load(f, Loader=yaml.Loader))
+
+
+class ActionDispatcher:
+    active_endpoints: ClassVar[list[str]] = list()
+    action: "ActionBase"
+    endpoint: "EndpointBase"
+
+    def __new__(cls, autom: Automaton):
+        cls.active_endpoints.append(autom.endpoint.suffix)
+        return super().__new__(cls)
+
+    def __init__(self, autom: Automaton):
+        self.action = autom.action
+        self.endpoint = autom.endpoint
+
+    async def dispatch(self, request):
+        allowed = await self.endpoint.authorize(request)
+        if not allowed:
+            return JSONResponse({'status': 'unauthorized'}, status_code=401)
+        task = BackgroundTask(self.action.perform_action, request)
+        return JSONResponse({'status': 'triggered'}, background=task)
+
+
+def startup():
+    print("Current active endpoints:")
+    for s in ActionDispatcher.active_endpoints:
+        print(f"\t/{s}")
+
+routes = list()
+for automata in settings.automata:
+    suffix = automata.endpoint.suffix
+    routes.append(Route(f"/{suffix}", ActionDispatcher(automata).dispatch, methods=["POST"]))
+
+app = Starlette(routes=routes, on_startup=[startup])