Skip to content

Commit 9a2fecf

Browse files
Copilotwdconinc
Copilot
and
wdconinc
committed
Add container infrastructure documentation with docsify and GitHub Pages deployment (#74)
* Initial plan * Add comprehensive documentation with docsify and GitHub Pages deployment Co-authored-by: wdconinc <4656391+wdconinc@users.noreply.github.com> * Fix spelling of Subenvironment in spack-environment.md Co-authored-by: wdconinc <4656391+wdconinc@users.noreply.github.com> * Remove spack/spack from package definitions --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: wdconinc <4656391+wdconinc@users.noreply.github.com> Co-authored-by: Wouter Deconinck <wdconinc@gmail.com>
1 parent 6c37b69 commit 9a2fecf

File tree

9 files changed

+1110
-0
lines changed

9 files changed

+1110
-0
lines changed

‎.github/workflows/docs.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: Deploy Documentation
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
paths:
8+
- 'docs/**'
9+
workflow_dispatch:
10+
11+
# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
12+
permissions:
13+
contents: read
14+
pages: write
15+
id-token: write
16+
17+
# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
18+
concurrency:
19+
group: "pages"
20+
cancel-in-progress: true
21+
22+
jobs:
23+
deploy:
24+
environment:
25+
name: github-pages
26+
url: ${{ steps.deployment.outputs.page_url }}
27+
runs-on: ubuntu-latest
28+
steps:
29+
- name: Checkout
30+
uses: actions/checkout@v4
31+
32+
- name: Setup Pages
33+
uses: actions/configure-pages@v4
34+
35+
- name: Upload artifact
36+
uses: actions/upload-pages-artifact@v3
37+
with:
38+
path: 'docs'
39+
40+
- name: Deploy to GitHub Pages
41+
id: deployment
42+
uses: actions/deploy-pages@v4

‎docs/.nojekyll‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Docsify does not need building - this file ensures GitHub Pages serves the docs directory directly

‎docs/README.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# EIC Container Infrastructure
2+
3+
Welcome to the EIC (Electron-Ion Collider) container infrastructure documentation. This documentation provides comprehensive information about the container build system, architecture, and instructions for building containers locally.
4+
5+
## Overview
6+
7+
The EIC container infrastructure provides scientific software environments for the Electron-Ion Collider project. These containers are built using Docker/OCI standards and managed through GitHub Actions, with packages installed via the [Spack](https://spack.io/) package manager.
8+
9+
## Quick Links
10+
11+
- [Architecture Overview](architecture.md) - Understanding the container build system
12+
- [Build Pipeline](build-pipeline.md) - GitHub Actions workflow details
13+
- [Building Locally](building-locally.md) - Instructions for local builds with caching
14+
- [Spack Environment](spack-environment.md) - Spack configuration and packages
15+
16+
## Container Images
17+
18+
The infrastructure produces multi-architecture (amd64/arm64) container images:
19+
20+
| Image | Description | Registry |
21+
|-------|-------------|----------|
22+
| `debian_stable_base` | Base image with compilers and Spack | `ghcr.io/eic/debian_stable_base` |
23+
| `eic_ci` | CI environment (minimal packages) | `ghcr.io/eic/eic_ci` |
24+
| `eic_xl` | Full development environment | `ghcr.io/eic/eic_xl` |
25+
26+
## Getting Started
27+
28+
For installation instructions of `eic-shell`, see the [eic-shell repository](https://github.com/eic/eic-shell).
29+
30+
### Using the Container
31+
32+
```bash
33+
# Pull the latest eic_xl image
34+
docker pull ghcr.io/eic/eic_xl:latest
35+
36+
# Run interactively
37+
docker run -it ghcr.io/eic/eic_xl:latest
38+
39+
# Or use Singularity/Apptainer
40+
singularity pull docker://ghcr.io/eic/eic_xl:latest
41+
singularity shell eic_xl_latest.sif
42+
```
43+
44+
## Repository Structure
45+
46+
```
47+
containers/
48+
├── .github/workflows/ # GitHub Actions workflows
49+
│ └── build-push.yml # Main build pipeline
50+
├── containers/
51+
│ ├── debian/ # Base image Dockerfile
52+
│ └── eic/ # EIC image Dockerfile
53+
├── spack-environment/ # Spack environment configurations
54+
│ ├── packages.yaml # Package versions and variants
55+
│ ├── ci/ # CI environment specs
56+
│ └── xl/ # Full (XL) environment specs
57+
├── spack.sh # Spack version configuration
58+
├── spack-packages.sh # Spack-packages version and cherry-picks
59+
├── eic-spack.sh # EIC-spack repository configuration
60+
└── key4hep-spack.sh # Key4HEP-spack repository configuration
61+
```
62+
63+
## Support
64+
65+
For questions or issues, please visit:
66+
- [GitHub Issues](https://github.com/eic/containers/issues)
67+
- [EIC Software Working Group](https://github.com/eic)

‎docs/_sidebar.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
<!-- docs/_sidebar.md -->
2+
3+
* [Home](README.md)
4+
* **Architecture**
5+
* [Overview](architecture.md)
6+
* [Build Pipeline](build-pipeline.md)
7+
* **Building**
8+
* [Building Locally](building-locally.md)
9+
* [Spack Environment](spack-environment.md)

‎docs/architecture.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Architecture Overview
2+
3+
The EIC container infrastructure uses a multi-stage build approach with separate builder and runtime tracks. This design ensures efficient builds while producing lightweight final images.
4+
5+
## Build Strategy
6+
7+
The container build follows a two-track approach:
8+
9+
```mermaid
10+
flowchart TB
11+
subgraph "Builder Track"
12+
A[builder_image<br/>debian_stable_base] --> B[builder_concretization_default<br/>Concretize spack environment]
13+
B --> C[builder_installation_default<br/>Build packages]
14+
C --> D[builder_concretization_custom<br/>Concretize custom versions]
15+
D --> E[builder_installation_custom<br/>Build custom packages]
16+
end
17+
18+
subgraph "Runtime Track"
19+
F[runtime_image<br/>debian_stable_base] --> G[runtime_concretization_default<br/>Copy spack.lock from builder]
20+
G --> H[runtime_installation_default<br/>Install from buildcache]
21+
H --> I[runtime_concretization_custom<br/>Copy custom spack.lock]
22+
I --> J[runtime_installation_custom<br/>Install custom from buildcache]
23+
J --> K[Final Image<br/>eic_ci / eic_xl]
24+
end
25+
26+
C -.->|spack.lock| G
27+
C -.->|buildcache| H
28+
E -.->|spack.lock| I
29+
E -.->|buildcache| J
30+
```
31+
32+
## Multi-Architecture Support
33+
34+
The infrastructure supports both `amd64` and `arm64` architectures through parallel builds:
35+
36+
```mermaid
37+
flowchart LR
38+
subgraph "Build Phase"
39+
A1[Build amd64]
40+
A2[Build arm64]
41+
end
42+
43+
subgraph "Manifest Phase"
44+
M[Create Multi-Arch Manifest]
45+
end
46+
47+
A1 --> D1[amd64 digest]
48+
A2 --> D2[arm64 digest]
49+
D1 --> M
50+
D2 --> M
51+
M --> R[Registry<br/>ghcr.io/eic/*]
52+
```
53+
54+
## Container Layer Structure
55+
56+
### Base Image (debian_stable_base)
57+
58+
```mermaid
59+
graph TB
60+
subgraph "debian_stable_base"
61+
D[Debian Stable Slim]
62+
D --> P1[System Packages<br/>git, curl, compilers, etc.]
63+
P1 --> G[GCC + Clang Compilers]
64+
G --> S[Spack Installation]
65+
S --> SR1[spack/spack repository]
66+
S --> SR2[spack/spack-packages repository]
67+
S --> SR3[key4hep/key4hep-spack repository]
68+
S --> SR4[eic/eic-spack repository]
69+
SR4 --> M[Spack Mirrors<br/>binaries.spack.io, ghcr.io/eic/spack-*]
70+
end
71+
```
72+
73+
### EIC Image (eic_ci / eic_xl)
74+
75+
```mermaid
76+
graph TB
77+
subgraph "EIC Container"
78+
B[debian_stable_base]
79+
B --> E1[Default Spack Environment<br/>ci or xl]
80+
E1 --> E2[Custom Environment<br/>epic, eicrecon, etc.]
81+
E2 --> F[Final Configuration]
82+
F --> BM[Benchmarks]
83+
F --> CP[Campaigns]
84+
F --> SC[Scripts & Entrypoint]
85+
end
86+
```
87+
88+
## Spack Repository Hierarchy
89+
90+
The Spack package definitions come from multiple repositories with priority ordering:
91+
92+
```mermaid
93+
flowchart TB
94+
subgraph "Priority Order (High to Low)"
95+
E[eic/eic-spack<br/>EIC-specific packages]
96+
K[key4hep/key4hep-spack<br/>Key4HEP packages]
97+
SP[spack/spack-packages<br/>Community packages]
98+
end
99+
100+
E --> K --> SP
101+
102+
P[Package Resolution] --> E
103+
```
104+
105+
## Caching Architecture
106+
107+
Multiple caching layers are used to optimize build times:
108+
109+
```mermaid
110+
flowchart TB
111+
subgraph "Build Cache Types"
112+
RC[Registry Cache<br/>Docker layer cache in ghcr.io]
113+
GC[GitHub Actions Cache<br/>ccache, apt, spack source]
114+
BC[Spack Buildcache<br/>Pre-built binaries on ghcr.io]
115+
end
116+
117+
subgraph "Cache Locations"
118+
RC --> R1[ghcr.io/eic/buildcache:*]
119+
GC --> G1[ccache]
120+
GC --> G2[/var/cache/apt]
121+
GC --> G3[/var/cache/spack]
122+
BC --> B1[ghcr.io/eic/spack-v2025.07.0]
123+
end
124+
```
125+
126+
## Environment Variants
127+
128+
### CI Environment (eic_ci)
129+
130+
Minimal environment for continuous integration:
131+
- Core HEP packages (ROOT, Geant4, DD4hep)
132+
- Essential reconstruction tools
133+
- No GUI dependencies (`-opengl`, `-webgui`)
134+
135+
### XL Environment (eic_xl)
136+
137+
Full development environment:
138+
- All CI packages plus GUI support
139+
- Development tools (emacs, gdb, valgrind, etc.)
140+
- Additional Python packages
141+
- Machine learning tools (TensorFlow, PyTorch, ONNX)
142+
- Jupyter notebook support
143+
144+
## Version Configuration
145+
146+
Package versions are controlled through several configuration files:
147+
148+
| File | Purpose |
149+
|------|---------|
150+
| `spack.sh` | Spack core version and cherry-picks |
151+
| `spack-packages.sh` | Spack-packages version and cherry-picks |
152+
| `key4hep-spack.sh` | Key4HEP-spack version |
153+
| `eic-spack.sh` | EIC-spack version |
154+
| `spack-environment/packages.yaml` | Package version preferences and variants |

0 commit comments

Comments
 (0)