Added multiple docker files for use-cases (prod, dev, etc) along with Makefile. Updated env vars with service specific prefix.

Author: vladd-bit

README.md

@@ -37,23 +37,41 @@ Full list found in [requirements.txt](./requirements.txt).
 
 ## Security
 
-Certificates used are located in the `./security/` folder, taken from the [Cogstack-NiFi](https://github.com/CogStack-NiFi) security folder, [root-ca.key](https://raw.githubusercontent.com/CogStack/CogStack-NiFi/refs/heads/main/security/root-ca.key) and [root-ca.pem](https://raw.githubusercontent.com/CogStack/CogStack-NiFi/refs/heads/main/security/root-capem), read the [security section](https://cogstack-nifi.readthedocs.io/en/latest/security.html) for more info on how to generate them from the main NiFi repository.
+Certificates used are located in the `./security/` folder, taken from the [Cogstack-NiFi](https://github.com/CogStack-NiFi) security folder, [nifi.key](https://raw.githubusercontent.com/CogStack/CogStack-NiFi/refs/heads/main/security/certificates/nifi/nifi.key) and [nifi.pem](https://raw.githubusercontent.com/CogStack/CogStack-NiFi/refs/heads/main/security/certificates/nifi/nifi.pem), read the [security section](https://cogstack-nifi.readthedocs.io/en/latest/security.html) for more info on how to generate them from the main NiFi repository.
 
 ## Setting up your own hub
 
-There are two docker compose files:
-`docker-compose-dev.yml` - this will build the build the hub image from scratch, it will not build the singleuser one however.
-`docker-compose.yml` - default, for production.
+This folder contains a modular Docker Compose setup for running the CogStack Jupyter Hub across multiple environments.
+
+    ```bash
+        docker/
+        ├── docker-compose.base.yml        # Canonical base definition
+        ├── docker-compose.yml             # Extends the base (always loaded)
+        ├── docker-compose.override.yml    # Local overrides (auto-loaded)
+        ├── docker-compose.dev.yml         # Dev overrides
+        ├── docker-compose.prod.yml        # Prod overrides
+        ├── Makefile                       # Helper commands (up, down, ps, show-env)
+        └── export_env_vars.sh             # Loads ../env/*.env files
+    ```
+
+### 🧩 Usage
+
+- **Local:** `make up` → uses base + override (bridge network)
+- **Dev:** `make up-dev` → uses base + dev overrides
+- **Prod:** `make up-prod` → uses base + prod overrides
+- **Stop:** `make down`  
+- **Show envs:** `make show-env`
+
+This structure keeps the base clean while allowing environment-specific overlays.
 
 Check the [env/general.env](./env/general.env), set the `CPU_ARCHITECTURE` variable to whatever you need, the default for most Laptops/PCs is `amd64`, if you have an ARM laptop/device then set to `arm64`, that should suffice.
 
 Execute the following in the main repo directory:
 
-```bash
-bash export_env_vars.sh
-cd docker
-docker compose up -d -f docker-compose.yml cogstack-jupyter-hub
-```
+    ```bash
+        cd docker
+        make start
+    ```
 
 Updating certificates and env settings from the main repo:
     - sometimes it is necessary to grab new certificates if the old ones expired (from the main Cogstack-NiFi repo)
@@ -77,8 +95,8 @@ Pre-requisites (for Linux and Windows): - for Linux, you need to install the nvi
 
 In [env/jupyter.env](./env/jupyter.env):
 
-    - change `DOCKER_ENABLE_GPU_SUPPORT` from `false` to `true`.
-    - change `DOCKER_NOTEBOOK_IMAGE` from `cogstacksystems/jupyter-singleuser:latest` to `cogstacksystems/jupyter-singleuser-gpu:latest`.
+    - change `JUPYTERHUB_DOCKER_ENABLE_GPU_SUPPORT` from `false` to `true`.
+    - change `JUPYTERHUB_JUPYTER_HUB_SINGLEUSER_DOCKER_NOTEBOOK_IMAGE` from `cogstacksystems/jupyter-singleuser:latest` to `cogstacksystems/jupyter-singleuser-gpu:latest`.
     - in the main repo folder, execute the following command in terminal: `source env/jupyter.env`, then `docker compose up -d`.
 
 *Use any release version you want instead of `latest` as necessary .
@@ -89,12 +107,12 @@ Users can have their resources limited (currently only CPU + RAM), there is a de
 Change the coresponding variables in [env/jupyter.env](./env/jupyter.env):
 
     *   General user resource cap per container, default 2 cores, 2GB ram:
-        - `RESOURCE_ALLOCATION_USER_CPU_LIMIT`="2"
-        - `RESOURCE_ALLOCATION_USER_RAM_LIMIT`="2.0G"
+        - `JUPYTER_HUB_SINGLEUSER_RESOURCE_ALLOCATION_USER_CPU_LIMIT`="2"
+        - `JUPYTER_HUB_SINGLEUSER_RESOURCE_ALLOCATION_USER_RAM_LIMIT`="2.0G"
 
     *   Admin resource cap per container, default 2 cores, 4 GB RAM:
-        - `RESOURCE_ALLOCATION_ADMIN_CPU_LIMIT`="2"
-        - `RESOURCE_ALLOCATION_ADMIN_RAM_LIMIT`="4.0G"
+        - `JUPYTER_HUB_SINGLEUSER_RESOURCE_ALLOCATION_ADMIN_CPU_LIMIT`="2"
+        - `JUPYTER_HUB_SINGLEUSER_RESOURCE_ALLOCATION_ADMIN_RAM_LIMIT`="4.0G"
 
 ## Sharing storage between users
 
@@ -103,4 +121,4 @@ This feature is currently experiemntal, it requires admins to add users to the s
 
 ## DEVELOPING
 
-Please make sure to set DOCKER_JUPYTER_HUB_CONTAINER_NAME="cogstack-jupyter-hub-dev" in the `env/jupyter.env` otherwise singleuser containers won't be able to start.
+Please make sure to set JUPYTER_HUB_DOCKER_CONTAINER_NAME="cogstack-jupyter-hub-dev" in the `env/jupyter.env` otherwise singleuser containers won't be able to start.

config/jupyterhub_config.py

@@ -17,35 +17,36 @@ class LocalNativeAuthenticator(NativeAuthenticator, LocalAuthenticator):
     pass
 
 
-DOCKER_NOTEBOOK_IMAGE = os.getenv("DOCKER_NOTEBOOK_IMAGE", "cogstacksystems:jupyterhub/singleuser:latest")
+DOCKER_NOTEBOOK_IMAGE = os.getenv("JUPYTER_HUB_SINGLEUSER_DOCKER_NOTEBOOK_IMAGE", 
+                                  "cogstacksystems:jupyterhub/singleuser:latest")
 
 # JupyterHub requires a single-user instance of the Notebook server, so we
 # default to using the `start-singleuser.sh` script included in the
 # jupyter/docker-stacks *-notebook images as the Docker run command when
 # spawning containers.  Optionally, you can override the Docker run command
-# using the DOCKER_SPAWN_CMD environment variable.
-SPAWN_CMD = os.environ.get("DOCKER_SPAWN_CMD", "start-singleuser.py")
+# using the JUPYTER_HUB_DOCKER_SPAWN_CMD environment variable.
+SPAWN_CMD = os.environ.get("JUPYTER_HUB_DOCKER_SPAWN_CMD", "start-singleuser.py")
 
 # Connect containers to this Docker network
 # IMPORTANT, THIS MUST MATCH THE NETWORK DECLARED in "services.yml", by default: "cogstack-net"
-NETWORK_NAME = os.environ.get("DOCKER_NETWORK_NAME", "cogstack-net")
+NETWORK_NAME = os.environ.get("JUPYTER_HUB_DOCKER_NETWORK_NAME", "cogstack-net")
 
 # The IP address or hostname of the JupyterHub container in the Docker network
-HUB_CONTAINER_IP_OR_NAME = os.environ.get("DOCKER_JUPYTER_HUB_CONTAINER_NAME", "cogstack-jupyter-hub")
+HUB_CONTAINER_IP_OR_NAME = os.environ.get("JUPYTER_HUB_DOCKER_CONTAINER_NAME", "cogstack-jupyter-hub")
 
 # The timeout in seconds after which the idle notebook container will be shutdown
 NOTEBOOK_IDLE_TIMEOUT = int(os.environ.get("DOCKER_NOTEBOOK_IDLE_TIMEOUT", "7200"))
 
-SELECT_NOTEBOOK_IMAGE_ALLOWED = str(os.environ.get("DOCKER_SELECT_NOTEBOOK_IMAGE_ALLOWED", "false")).lower()
+SELECT_NOTEBOOK_IMAGE_ALLOWED = str(os.environ.get("JUPYTERHUB_DOCKER_SELECT_NOTEBOOK_IMAGE_ALLOWED", "false")).lower()
 
-RUN_IN_DEBUG_MODE = str(os.environ.get("DOCKER_NOTEBOOK_DEBUG_MODE", "false")).lower()
+RUN_IN_DEBUG_MODE = str(os.environ.get("JUPYTERHUB_DOCKER_NOTEBOOK_DEBUG_MODE", "false")).lower()
 
 # Explicitly set notebook directory because we"ll be mounting a host volume to
 # it.  Most jupyter/docker-stacks *-notebook images run the Notebook server as
 # user `jovyan`, and set the notebook directory to `/home/jovyan/work`.
 # We follow the same convention.
-NOTEBOOK_DIR = os.environ.get("DOCKER_NOTEBOOK_DIR", "/home/jovyan/work")
-SHARED_CONTENT_DIR = os.environ.get("DOCKER_SHARED_DIR", "/home/jovyan/scratch")
+NOTEBOOK_DIR = os.environ.get("JUPYTER_HUB_DOCKER_NOTEBOOK_DIR", "/home/jovyan/work")
+SHARED_CONTENT_DIR = os.environ.get("JUPYTER_HUB_DOCKER_SHARED_DIR", "/home/jovyan/scratch")
 #WORK_DIR = os.environ.get("DOCKER_JUPYTER_WORK_DIR", "/lab/workspaces/auto-b/tree/" + str(NOTEBOOK_DIR.split("/")[-1]))
 WORK_DIR = "/lab/"
 
@@ -127,7 +128,7 @@ def start(self):
                 }
 
         # this is a temporary fix, need to actually check permissions
-        self.mem_limit = resource_allocation_user_ram_limit
+        self.mem_limit = RESOURCE_ALLOCATION_USER_RAM_LIMIT
         self.post_start_cmd = "chmod -R 777 " + SHARED_CONTENT_DIR
 
         return super().start()
@@ -190,15 +191,15 @@ def pre_spawn_hook(spawner):
 
 # Resource allocation env vars
 # RAM - GB of ram, CPU - num of cores
-resource_allocation_user_cpu_limit = os.environ.get("RESOURCE_ALLOCATION_USER_CPU_LIMIT", "2")
-resource_allocation_user_ram_limit = os.environ.get("RESOURCE_ALLOCATION_USER_RAM_LIMIT", "2.0G")
-resource_allocation_admin_cpu_limit = os.environ.get("RESOURCE_ALLOCATION_ADMIN_CPU_LIMIT", "2")
-resource_allocation_admin_ram_limit = os.environ.get("RESOURCE_ALLOCATION_ADMIN_RAM_LIMIT", "4.0G")
+RESOURCE_ALLOCATION_USER_CPU_LIMIT = os.environ.get("JUPYTER_HUB_SINGLEUSER_RESOURCE_ALLOCATION_USER_CPU_LIMIT", "2")
+RESOURCE_ALLOCATION_USER_RAM_LIMIT = os.environ.get("JUPYTER_HUB_SINGLEUSER_RESOURCE_ALLOCATION_USER_RAM_LIMIT", "2.0G")
+RESOURCE_ALLOCATION_ADMIN_CPU_LIMIT = os.environ.get("JUPYTER_HUB_SINGLEUSER_RESOURCE_ALLOCATION_ADMIN_CPU_LIMIT", "2")
+RESOURCE_ALLOCATION_ADMIN_RAM_LIMIT = os.environ.get("JUPYTER_HUB_SINGLEUSER_RESOURCE_ALLOCATION_ADMIN_RAM_LIMIT", "4.0G")
 
 
 def per_user_limit(role):
-    ram_limits = {"user": (int(resource_allocation_user_cpu_limit), resource_allocation_user_ram_limit),
-                   "admin": (int(resource_allocation_admin_cpu_limit), resource_allocation_admin_ram_limit)}
+    ram_limits = {"user": (int(RESOURCE_ALLOCATION_USER_CPU_LIMIT), RESOURCE_ALLOCATION_USER_RAM_LIMIT),
+                   "admin": (int(RESOURCE_ALLOCATION_ADMIN_CPU_LIMIT), RESOURCE_ALLOCATION_ADMIN_RAM_LIMIT)}
     return ram_limits.get(role)
 
 
@@ -256,7 +257,7 @@ def per_user_limit(role):
             for member in members:
                 team_map[member].add(team)
 
-gpu_support_enabled = os.environ.get("DOCKER_ENABLE_GPU_SUPPORT", "false")
+gpu_support_enabled = os.environ.get("JUPYTERHUB_DOCKER_ENABLE_GPU_SUPPORT", "false")
 
 if gpu_support_enabled.lower() == "true":
     c.DockerSpawner.extra_host_config = {
@@ -301,11 +302,11 @@ def per_user_limit(role):
 
 # TLS config
 c.JupyterHub.port = jupyter_hub_ssl_port
-c.JupyterHub.ssl_key = os.environ.get("SSL_KEY", "/srv/jupyterhub/security/nifi.key")
-c.JupyterHub.ssl_cert = os.environ.get("SSL_CERT", "/srv/jupyterhub/security/nifi.pem")
+c.JupyterHub.ssl_key = os.environ.get("JUPYTERHUB_SSL_KEY", "/srv/jupyterhub/security/nifi.key")
+c.JupyterHub.ssl_cert = os.environ.get("JUPYTERHUB_SSL_CERT", "/srv/jupyterhub/security/nifi.pem")
 
 # Persist hub data on volume mounted inside container
-data_dir = os.environ.get("DATA_VOLUME_CONTAINER", "")
+data_dir = os.environ.get("JUPYTERHUB_DATA_VOLUME_CONTAINER", "")
 
 c.JupyterHub.cookie_secret_file = data_dir + "jupyterhub_cookie_secret"
 

docker/Makefile

@@ -0,0 +1,46 @@
+PROJECT = cogstack-jupyter-hub
+CONTAINER_NAME = $(PROJECT)
+CONTAINER_NAME_DEV= $(PROJECT)-dev
+COMPOSE_BASE = docker-compose.base.yml
+SHELL := /usr/bin/env bash
+.SHELLFLAGS := -o pipefail -c
+
+DC_START_CMD := up -d
+DC_STOP_CMD := stop
+DC_DOWN_CMD := down
+
+define WITH_ENV
+set -a && source ../export_env_vars.sh;
+endef
+
+.PHONY: up down restart build logs cert-check health-check
+
+load-env:
+	$(WITH_ENV) echo "Environment variables loaded."
+
+show-env:
+	${WITH_ENV} >/dev/null 2>&1; printenv | grep -E "^(JUPYTER)" | sort
+
+start-dev:
+	$(WITH_ENV) docker compose -f docker-compose.dev.yml $(DC_START_CMD)
+
+start-dev-build:
+	$(WITH_ENV) docker compose -f docker-compose.dev.yml ${DC_START_CMD} --build
+
+start:
+	$(WITH_ENV) docker compose -f $(COMPOSE_BASE) -f docker-compose.yml $(DC_START_CMD)
+
+start-prod:
+	$(WITH_ENV) docker compose -f $(COMPOSE_BASE) -f docker-compose.prod.yml $(DC_START_CMD)
+
+stop-all:
+	$(WITH_ENV) docker compose  -f $(COMPOSE_BASE) -f docker-compose.dev.yml -f docker-compose.local.yml -f docker-compose.prod.yml docker-compose.yml  ${DC_DOWN_CMD}
+
+logs-dev:
+	docker logs -f --tail 10000 ${CONTAINER_NAME_DEV}
+
+logs:
+	docker logs -f --tail 10000 $(CONTAINER_NAME)
+
+health-check:
+	curl -k -v https://localhost:8888/

docker/docker-compose-dev.yml renamed to docker/docker-compose.base.yml

@@ -1,9 +1,7 @@
 services:
-  cogstack-jupyter-hub-dev:
-    build:
-      context: ../
-      dockerfile: "Dockerfile_hub"
-    container_name: cogstack-jupyter-hub-dev
+  cogstack-jupyter-hub:
+    image: cogstacksystems/jupyter-hub:${JUPYTER_HUB_IMAGE_RELEASE_VERSION:-latest}
+    container_name: cogstack-jupyter-hub
     restart: always
     environment:
       - http_proxy=$HTTP_PROXY
@@ -13,8 +11,9 @@ services:
       - JUPYTERHUB_INTERNAL_PROXY_API_PORT=${JUPYTERHUB_INTERNAL_PROXY_API_PORT:-8887}
       - JUPYTERHUB_SSL_PORT=${JUPYTERHUB_SSL_PORT:-443}
     env_file:
-      - ../env/general.env
-      - ../env/jupyter.env
+      - ${DEPLOYMENT_ENV_FILE_PATH_GENERAL:-../env/general.env}
+      - ${JUPYTERHUB_ENV_FILE_PATH_CONFIG:-../env/jupyter.env}
+    user: "root"
     volumes:
       - jupyter-hub-shared-scratch:/home/jovyan/scratch
       - jupyter-hub-vol:/srv/jupyterhub
@@ -23,8 +22,8 @@ services:
       - ../config/jupyterhub_cookie_secret:/srv/jupyterhub/jupyterhub_cookie_secret:ro
       # User list and jupyter config
       - ../config/jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py:ro
-      - ../config/userlist:/srv/jupyterhub/userlist:ro
-      - ../config/teamlist:/srv/jupyterhub/teamlist:ro
+      - ../config/userlist:/srv/jupyterhub/userlist:rw
+      - ../config/teamlist:/srv/jupyterhub/teamlist:rw
       # Give access to Docker socket
       - /var/run/docker.sock:/var/run/docker.sock
     ulimits:
@@ -38,15 +37,17 @@ services:
       - "${JUPYTERHUB_INTERNAL_PORT:-8888}:${JUPYTERHUB_SSL_PORT:-443}"
     networks:
       - cognet
+    extra_hosts:
+      - ${ELASTICSEARCH_1_HOST_NAME:-test-1:0.0.0.0}
+      - ${ELASTICSEARCH_2_HOST_NAME:-test-2:0.0.0.0}
+      - ${ELASTICSEARCH_3_HOST_NAME:-test-3:0.0.0.0}
+      - ${KIBANA_HOST_NAME:-test-4:0.0.0.0}
+      - ${NIFI_HOST_NAME:-test-5:0.0.0.0}
 
 volumes:
   jupyter-hub-vol:
+    name: "${VOLUME_PREFIX:-deploy_}jupyter-hub-vol"
     driver: local
   jupyter-hub-shared-scratch:
+    name: "${VOLUME_PREFIX:-deploy_}jupyter-hub-shared-scratch"
     driver: local
-
-networks:
-  cognet:
-    name: cogstack-net
-
-

docker/docker-compose.dev.yml

@@ -0,0 +1,23 @@
+services:
+  cogstack-jupyter-hub-dev:
+    extends:
+      file: docker-compose.base.yml
+      service: cogstack-jupyter-hub
+    build:
+      context: ../
+      dockerfile: Dockerfile_hub
+    container_name: cogstack-jupyter-hub-dev
+    networks:
+      - cognet
+
+volumes:
+  jupyter-hub-vol:
+    driver: local
+  jupyter-hub-shared-scratch:
+    driver: local
+
+networks:
+  cognet:
+    name: cogstack-net
+
+

docker/docker-compose.override.yml

@@ -0,0 +1,14 @@
+services:
+  cogstack-jupyter-hub:
+    container_name: cogstack-jupyter-hub
+    networks:
+      - cognet
+
+volumes:
+  jupyter-hub-vol:
+  jupyter-hub-shared-scratch:
+
+networks:
+  cognet:
+    name: cogstack-net
+    driver: bridge

docker/docker-compose.prod.yml

@@ -0,0 +1,9 @@
+services:
+  cogstack-jupyter-hub:
+    networks:
+      - cognet
+
+networks:
+  cognet:
+    name: cogstack-net
+    external: true