diff --git a/docker/.dockerignore b/docker/.dockerignore new file mode 100644 index 000000000..591bfe816 --- /dev/null +++ b/docker/.dockerignore @@ -0,0 +1,23 @@ +# Ignore common build artifacts and large nix store paths +result +result.tar.gz +result-*.tar.gz +dist +dist-newstyle +dist/* +*.o +*.hi +.stack-work +.ghc.environment.* +.cabal-sandbox +.cabal-sandbox/* +node_modules +.git +*.lock +**/.DS_Store + +# Ignore Nix store references and temporary files +/nix +*.drv +*.nar + diff --git a/docker/Dockerfile b/docker/Dockerfile new file mode 100644 index 000000000..1be97396f --- /dev/null +++ b/docker/Dockerfile @@ -0,0 +1,29 @@ +# Lightweight Dockerfile for a static cardano-db-sync binary +# +# Usage (example): +# 1. Build a static executable with Nix: +# nix build .#cardano-db-sync-linux +# 2. Build the image from the project root: +# docker build -f docker/Dockerfile -t cardano-db-sync:lightweight . +# +# Note: Since result/ is a Nix symlink, Docker may have issues following it. +# If this fails, you can dereference the symlink first: +# mkdir -p docker/build && cp -L result/*.tar.gz docker/build/ +# docker build -f docker/Dockerfile -t cardano-db-sync:lightweight . +# +# Important: this Dockerfile assumes the binary is a fully static executable +# (statically linked). If it's not static, use a small base image (see docs). + +# Stage 1: Extract the binary from the tarball +FROM busybox:latest AS extractor +ARG TARBALL +# ADD automatically extracts tarballs +ADD ${TARBALL} /tmp/extract/ +RUN find /tmp/extract -name cardano-db-sync -type f -exec mv {} /cardano-db-sync \; && \ + chmod +x /cardano-db-sync + +# Stage 2: Minimal runtime image +FROM scratch +COPY --from=extractor /cardano-db-sync /usr/local/bin/cardano-db-sync +ENTRYPOINT ["/usr/local/bin/cardano-db-sync"] +CMD ["--help"] diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 000000000..ec832bc45 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,65 @@ +Lightweight Docker image for cardano-db-sync +========================================== + +This folder contains a minimal Docker context and `Dockerfile` intended for use with +a statically-built `cardano-db-sync` executable produced by Nix. + +Why this exists +--------------- + +Building the full NixOS-based Docker image is a reproducible approach, but can lead +to large images and bundled configuration that make customization more difficult. +This lightweight approach lets users build a small, standard Docker image and supply +configuration at runtime via volumes or create their own derived images. + +Quickstart +---------- + +1. Build the cardano-db-sync package with Nix: + + ```bash + nix build .#cardano-db-sync-linux + ``` + +2. a) Extract the binary from the tarball and copy it to this directory: + + ```bash + tar xf result/cardano-db-sync-*-linux.tar.gz -C docker/ --strip-components=1 ./bin/cardano-db-sync + ``` + +2. b) move the binary: + + ```bash + mv docker/bin/cardano-db-sync ./docker && rm -r docker/bin/ + ``` + +3. Build the image: + + ```bash + docker build -t cardano-db-sync:lightweight docker/ + ``` + +4. get the configs: +- https://book.world.dev.cardano.org/environments/preprod/db-sync-config.json + +4. Run the container and mount your configuration (recommended): + + ```bash + docker run -v $PWD/environments/preprod:/config \ + --env POSTGRES_HOST=postgres --env POSTGRES_PORT=5432 \ + cardano-db-sync:lightweight --config $PWD/config/db-sync-config.json \ + --socket-path "$HOME/workshop/cardano-node/configuration/preprod/db/node.socket" \ + --schema-dir /schema + ``` + +Notes +----- + +- The `Dockerfile` uses `FROM scratch` and therefore requires the binary to be + statically linked. If you need dynamic linking, change the base image to a small + distro such as `ubuntu:24.04` or `debian:bookworm-slim`. +- The image intentionally omits network configuration files. Use volumes or a + derived image if you want to embed configs. + +See also: the project's `doc/docker.md` for more details and guidance. + diff --git a/docker/build.sh b/docker/build.sh new file mode 100755 index 000000000..a720b4886 --- /dev/null +++ b/docker/build.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Dereference the Nix result symlink +TARBALL=$(basename "$(readlink -f result/*.tar.gz)") +cp -L result/*.tar.gz docker/ + +# Build with the exact tarball name as build arg +docker build --build-arg TARBALL="$TARBALL" -t cardano-db-sync:lightweight docker/ + +# Cleanup +rm -f docker/*.tar.gz diff --git a/docker/docker.md b/docker/docker.md new file mode 100644 index 000000000..697de7507 --- /dev/null +++ b/docker/docker.md @@ -0,0 +1,354 @@ +# Docker + +## Quickstart + +We provide an example +[docker-compose.yml](https://github.com/IntersectMBO/cardano-db-sync/blob/master/docker-compose.example.yml) +to quickly get set up with `cardano-db-sync`. Keep in mind that this is only a template, +and users are encouraged to tweak it to meet their needs. + +It is not recommended to use this for a production set up, and should only be +used for local development. + +Create a working directory: + +```bash +mkdir ~/src +cd ~/src +``` + +Download the example Docker Compose file: + +```bash +curl -o docker-compose.yml \ + https://raw.githubusercontent.com/IntersectMBO/cardano-db-sync/master/docker-compose.example.yml +``` + +Start `cardano-node`, `postgresql`, and `cardano-db-sync` services: + +```bash +docker compose up -d && docker compose logs -f +``` + +The PostgreSQL database should now be exposed on localhost port `5432` + +To connect to PostgreSQL database: + +```bash +psql -h 0.0.0.0 -p 5432 -U postgres -d cexplorer +``` + +### Connecting to a Other Networks + +To connect to different network use `NETWORK` environment variable: + +```bash +NETWORK=preprod docker compose up -d && docker compose logs -f +``` + +## Running `cardano-db-sync` + +Start `cardano-db-sync`: + +```bash +docker run \ + --env NETWORK=mainnet \ + --env POSTGRES_HOST=postgres \ + --env POSTGRES_PORT=5432 \ + --volume db-sync-data:/var/lib/cexplorer \ + --volume node-ipc \ + ghcr.io/IntersectMBO/cardano-db-sync:13.3.0.0 +``` + +### Environment Variables + +#### `NETWORK` + +Specifies a network to connect to. If specified, `cardano-db-sync` will provide the +configuration files. If not specified, will call the `cardano-db-sync` executable as the +entrypoint. Possible values are: + + * mainnet + * preprod + * preview + * private + * sanchonet + * shelley_qa + +#### `POSTGRES_HOST` (required) + +The PostgreSQL server host to connect to. + +#### `POSTGRES_PORT` (required) + +Specifies the PostgreSQL server port to connect to. + +#### `POSTGRES_USER` (required) + +The PostgreSQL server user to connect as. + +#### `POSTGRES_PASSWORD` (required) + +Specifies the PostgreSQL server password to connect as. + +#### `RESTORE_SNAPSHOT` (optional) + +Specifies a `cardano-db-sync` snapshot archive to download and restore from. If omitted, +it will sync from genesis. see [Restoring From Snapshots](#restoring-from-snapshots) + +### `DB_SYNC_CONFIG` (optional) + +Overrides the `db-sync-config.json` provided by the network configuration. See [Overriding +Network Configuration](#overriding-network-configuration). + +#### `EXTRA_DB_SYNC_ARGS` (optional) + +Extra command line arguments to pass to `cardano-db-sync`. For example: + +```bash +docker run \ + --env NETWORK=mainnet \ + --env POSTGRES_HOST=postgres \ + --env POSTGRES_PORT=5432 \ + --env EXTRA_DB_SYNC_ARGS="--skip-fix" + --volume db-sync-data:/var/lib/cexplorer \ + --volume node-ipc \ + ghcr.io/IntersectMBO/cardano-db-sync:13.3.0.0 +``` + +### Overriding Network Configuration + +Overriding the configuration file can be done by passing the `DB_SYNC_CONFIG` environment +variable: + +```bash +docker run \ + --env NETWORK=mainnet \ + --env POSTGRES_HOST=postgres \ + --env POSTGRES_PORT=5432 \ + --env DB_SYNC_CONFIG=/config/db-sync-config.json \ + --volume db-sync-data:/var/lib/cexplorer \ + --volume node-ipc:/node-ipc \ + --volume $PWD/environments/mainnet:/config \ + ghcr.io/IntersectMBO/cardano-db-sync:13.3.0.0 +``` + +### Restoring From Snapshots + +Snapshots are provided for mainnet to restore the PostgreSQL database. See the [latest +releases](https://github.com/IntersectMBO/cardano-db-sync/releases) for a recent snapshot +matched with the `cardano-db-sync` version. + +Although, you can specify a URL directly, we recommend downloading the image separately to avoid +having to download it multiple times in case of failure. First, download the latest snapshot: + +```bash +curl -O https://update-cardano-mainnet.iohk.io/cardano-db-sync/13.3/db-sync-snapshot-schema-13.3-block-10611621-x86_64.tgz +``` + +Restore the snapshot with `RESTORE_SNAPSHOT`: + +```bash +docker run \ + --env NETWORK=mainnet \ + --env POSTGRES_HOST=postgres \ + --env POSTGRES_PORT=5432 \ + --env RESTORE_SNAPSHOT=/data/db-sync-snapshot-schema-13.3-block-10611621-x86_64.tgz \ + --volume db-sync-data:/var/lib/cexplorer \ + --volume node-ipc:/node-ipc \ + --volume $PWD:/data + ghcr.io/IntersectMBO/cardano-db-sync:13.3.0.0 +``` + +### Advanced Usage + +Excluding the `NETWORK` environment variable will simply call the `cardano-db-sync` +executable as the entrypoint, so you must pass command line arguments to projvide runtime +configuration. The `--schema-dir` argument is automatically set and is not required. + +```bash +docker run \ + --volume db-sync-data:/var/lib/cexplorer \ + --volume node-ipc:/node-ipc \ + --volume $PWD/environments/mainnet:/config \ + ghcr.io/IntersectMBO/cardano-db-sync:13.3.0.0 \ + run --config /config/db-sync-config.json --socket-path /node-ipc/node.socket +``` + +## Running `cardano-smash-server` + +Start `cardano-smash-server`: + +```bash +docker run \ + --env NETWORK=mainnet \ + --env POSTGRES_HOST=postgres \ + --env POSTGRES_PORT=5432 \ + --env SMASH_USER=smash-admin \ + --env SMASH_PASSWORD=smash-password \ + --publish 3100:3100 \ + --volume node-ipc:/node-ipc \ + ghcr.io/IntersectMBO/cardano-smash-server:13.3.0.0 +``` + +### Environment Variables + +#### `NETWORK` (optional) + +Specifies a network to connect to. If specified, `cardano-db-sync` will provide the +configuration files. If not specified, will call the `cardano-db-sync` executable as the +entrypoint. Possible values are: + + * mainnet + * preprod + * preview + * private + * sanchonet + * shelley_qa + +#### `POSTGRES_HOST` (required) + +The PostgreSQL server host to connect to. + +#### `POSTGRES_PORT` (required) + +Specifies the PostgreSQL server port to connect to. + +#### `POSTGRES_USER` (required) + +The PostgreSQL server user to connect as. + +#### `POSTGRES_PASSWORD` (required) + +Specifies the PostgreSQL server password to connect as. + +#### `SMASH_USER` (optional) + +Sets the user for HTTP basic authentication + +#### `SMASH_PASSWORD` (optional) + +Sets the password for HTTP basic authentication + +### Advanced Usage + +Excluding the `NETWORK` environment variable will simply call the `cardano-smash-server` +executable as the entrypoint, so you must pass command line arguments to provide runtime +configuration. + +```bash +docker run \ + --volume node-ipc:/node-ipc \ + --volume $PWD/environments/mainnet:/config \ + ghcr.io/IntersectMBO/cardano-db-sync:13.3.0.0 \ + --config /config/db-sync-config.json --port 3100 +``` + +## Building + +Building Docker images is done with [Nix](https://nixos.org/) and requires Linux: + +```bash +nix build .#cardano-db-sync-docker +``` + +This will generate a `tar.gz` file linked to `./result` that can be loaded into docker and +run as a normal image. + +```bash +docker load < result +docker run cardano-db-sync + +## Lightweight (hybrid) image workflow + +If you prefer a smaller, standard Docker image that is easy to customise, you can use +a hybrid workflow: build a static `cardano-db-sync` binary with Nix and then build a +minimal Docker image by copying the binary into a tiny base image. + +This keeps images small (the binary is typically <100MB) and avoids shipping the NixOS +module’s network configuration inside the image. The project includes a small Docker +context in `docker/` with a minimal `Dockerfile` designed for static executables. + +Quick example (assumes a static executable): + +```bash +# Build the cardano-db-sync package with Nix +nix build .#cardano-db-sync-linux + +# Extract the binary from the tarball into the docker context +tar xf result/cardano-db-sync-*-linux.tar.gz -C docker/ --strip-components=1 bin/cardano-db-sync + +# Build the lightweight image +docker build -t cardano-db-sync:lightweight docker/ +``` + +Notes and configuration options: + +- The `docker/Dockerfile` uses a `scratch` base image and therefore requires a fully + static binary. If you need dynamic linking, use a small base image (for example + `ubuntu:24.04` or `debian:bookworm-slim`) and replace the `FROM scratch` line. +- The image intentionally does not contain any network configuration files. To supply + configuration you have two options: + 1. Mount your configuration at runtime (recommended): + + ```bash + docker run -v $PWD/environments/mainnet:/config \ + --env POSTGRES_HOST=postgres \ + --env POSTGRES_PORT=5432 \ + cardano-db-sync:lightweight run --config /config/db-sync-config.json + ``` + + 2. Bake configuration into a derived image: create a small Dockerfile that copies + your config into the image on top of the lightweight runtime image. + +- For the canonical network configuration files (db-sync-config.json and related + assets), see the Cardano Operations Book: https://book.play.dev.cardano.org/. We + intentionally follow the same workflow used by non-containerized installations. + +Benefits: + +- Much smaller images (typically ~100MB vs 1GB+). +- Easier for non-Nix users to inspect and debug. +- Explicit configuration via volumes or derived images reduces surprises and makes + customization straightforward. + +If you plan to use the GitHub release workflow, note that the project's release +action currently builds and publishes the Nix-built Docker archive as before. The +lightweight workflow is provided as an opt-in alternative for users who want smaller +images or simpler customization. +``` + +## Running Tests with Docker PostgreSQL + +Create a `pgpass-test` file with the credentials of (taken from config/secrets/postgres_* files): + +```bash +echo "localhost:5432:cexplorer:postgres:v8hlDV0yMAHHlIurYupj" > config/pgpass-test +chmod 0600 config/pgpass-test +export PGPASSFILE=$PWD/config/pgpass-test +``` + +Start PostgreSQL via Docker Compose: + +```bash +docker compose -f docker-test.yml up +``` + +Run the migrations: + +```bash +cabal run cardano-db-tool run-migrations -- --mdir schema/ --ldir . +``` + +Run the tests: + +```bash +cabal test cardano-db +``` + +When you've finished with testing, stop and remove the test containers: + +```bash +docker compose -f docker-test.yml down +``` +