Skip to content

Commit ee195d1

Browse files
ZihaoZhouZihaoZhou
committed
Add docker in LXC
1 parent 4ab9c72 commit ee195d1

File tree

5 files changed

+242
-47
lines changed

5 files changed

+242
-47
lines changed

‎docs/.vitepress/config.mts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,10 @@ export default defineConfig({
8989
text: 'Remote Desktop',
9090
link: '/guide/rdp',
9191
},
92+
{
93+
text: 'Docker in LXC',
94+
link: '/guide/docker',
95+
},
9296
{
9397
text: 'Password and Security',
9498
link: '/guide/security',

‎docs/guide/docker.md‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
# Running Docker in LXC Containers
2+
3+
This guide explains how to run Docker inside LXC containers using the fuse-overlayfs storage driver. This is necessary because LXC containers cannot use Docker's default overlay2 storage driver due to kernel limitations.
4+
5+
::: warning Prerequisites
6+
- LXC container with FUSE support enabled
7+
- Root or sudo access inside the container
8+
- Basic understanding of Docker and systemctl
9+
:::
10+
11+
## Background: Why fuse-overlayfs?
12+
13+
Docker's default `overlay2` storage driver requires kernel-level permissions that LXC containers don't have. When you try to use overlay2 inside an LXC container, Docker fails because:
14+
15+
1. LXC mounts its filesystem using overlay, and the kernel doesn't allow overlay-on-overlay mounting
16+
2. The container lacks permissions to create kernel-level overlay filesystems
17+
18+
Without proper configuration, Docker falls back to the `vfs` driver, which creates full copies of each filesystem layer instead of using efficient copy-on-write. This can consume massive amounts of disk space.
19+
20+
## Prerequisites Check
21+
22+
### 1. Verify FUSE Support
23+
24+
First, check if your container has FUSE support:
25+
26+
```bash
27+
ls -la /dev/fuse
28+
```
29+
30+
If `/dev/fuse` doesn't exist, contact your system administrator to enable FUSE support for your container.
31+
32+
::: tip Enabling FUSE in Proxmox
33+
If using Proxmox, set `features: fuse=1` in the LXC config or check "FUSE" under Options in the web interface. Restart the container after making this change.
34+
:::
35+
36+
### 2. Install Required Packages
37+
38+
First, install Docker following the official installation guide:
39+
40+
::: tip Docker Installation
41+
Follow the official [Docker Engine installation guide for Ubuntu](https://docs.docker.com/engine/install/ubuntu/) to install Docker from the official repository. This ensures you get the latest stable version with proper support.
42+
:::
43+
44+
After installing Docker, install fuse-overlayfs:
45+
46+
```bash
47+
sudo apt update
48+
sudo apt install fuse-overlayfs
49+
```
50+
51+
## Configuration Steps
52+
53+
### 1. Configure Docker Storage Driver
54+
55+
Create or edit the Docker daemon configuration:
56+
57+
```bash
58+
sudo vim /etc/docker/daemon.json
59+
```
60+
61+
Add the following configuration:
62+
63+
```json
64+
{
65+
"storage-driver": "fuse-overlayfs"
66+
}
67+
```
68+
69+
::: warning NVIDIA Runtime
70+
If you have NVIDIA container runtime installed, preserve it in the configuration:
71+
```json
72+
{
73+
"storage-driver": "fuse-overlayfs",
74+
"runtimes": {
75+
"nvidia": {
76+
"args": [],
77+
"path": "nvidia-container-runtime"
78+
}
79+
}
80+
}
81+
```
82+
:::
83+
84+
### 2. Stop Existing Docker Service
85+
86+
Before applying changes, stop any running Docker service:
87+
88+
```bash
89+
sudo systemctl stop docker
90+
sudo systemctl stop docker.socket
91+
```
92+
93+
### 3. Test Configuration
94+
95+
Test the Docker daemon with the new storage driver:
96+
97+
```bash
98+
sudo dockerd --debug
99+
```
100+
101+
Look for messages confirming fuse-overlayfs is being used. Press `Ctrl+C` to stop once verified.
102+
103+
::: tip Troubleshooting Startup
104+
If you see an error about conflicting directives:
105+
```
106+
unable to configure the Docker daemon with file /etc/docker/daemon.json:
107+
the following directives are specified both as a flag and in the configuration file:
108+
storage-driver: (from flag: fuse-overlayfs, from file: overlay2)
109+
```
110+
This means there's a conflict in your configuration. Check `/etc/default/docker` or systemd service files for conflicting storage driver settings.
111+
:::
112+
113+
### 4. Enable and Start Docker Service
114+
115+
Once configuration is verified:
116+
117+
```bash
118+
sudo systemctl daemon-reload
119+
sudo systemctl enable docker
120+
sudo systemctl start docker
121+
```
122+
123+
## Common Issues and Solutions
124+
125+
### Permission Denied on Docker Socket
126+
127+
If you encounter:
128+
```
129+
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
130+
```
131+
132+
**Solution**: Run Docker commands with `sudo` or add your user to the docker group:
133+
```bash
134+
sudo usermod -aG docker $USER
135+
# Log out and back in for changes to take effect
136+
```
137+
138+
### Service Failed to Start
139+
140+
To debug Docker service issues:
141+
142+
```bash
143+
# Check service status
144+
sudo systemctl status docker
145+
146+
# View detailed logs
147+
sudo journalctl -xeu docker.service
148+
149+
# Check Docker daemon directly
150+
sudo dockerd --debug
151+
```
152+
153+
Common causes:
154+
- Syntax errors in `/etc/docker/daemon.json`
155+
- Missing fuse-overlayfs package
156+
- FUSE not enabled in container
157+
158+
### Storage Driver Conflicts
159+
160+
If Docker complains about conflicting storage drivers:
161+
162+
1. Check all configuration sources:
163+
```bash
164+
cat /etc/docker/daemon.json
165+
cat /etc/default/docker
166+
systemctl cat docker.service
167+
```
168+
169+
2. Remove any `--storage-driver` flags from systemd service files
170+
3. Ensure only one storage driver is specified in `daemon.json`
171+
172+
## Performance Considerations
173+
174+
::: warning Performance Impact
175+
fuse-overlayfs operates in userspace and has performance overhead compared to kernel-based overlay2. However, it's significantly more efficient than the vfs fallback:
176+
- **vfs**: Creates full copies of filesystem layers (can use 3-4x more space)
177+
- **fuse-overlayfs**: Uses copy-on-write like overlay2 but with ~10-20% performance overhead
178+
- **overlay2**: Native kernel driver (not available in LXC)
179+
:::
180+
181+
## Verification
182+
183+
Verify Docker is using the correct storage driver:
184+
185+
```bash
186+
sudo docker info | grep "Storage Driver"
187+
# Should output: Storage Driver: fuse-overlayfs
188+
```
189+
190+
Test with a simple container:
191+
192+
```bash
193+
sudo docker run hello-world
194+
```
195+
196+
## Advanced Configuration
197+
198+
### Systemd Service Debugging
199+
200+
For persistent issues, create a systemd override:
201+
202+
```bash
203+
sudo mkdir -p /etc/systemd/system/docker.service.d
204+
sudo vim /etc/systemd/system/docker.service.d/override.conf
205+
```
206+
207+
Add debugging options:
208+
```ini
209+
[Service]
210+
ExecStart=
211+
ExecStart=/usr/bin/dockerd --debug
212+
```
213+
214+
### Alternative: Recent Kernel Support
215+
216+
::: tip Kernel 6.1.10+ Users
217+
Recent kernels (6.1.10+) may support overlay2 in LXC containers natively, especially on ZFS. Test with:
218+
```bash
219+
sudo dockerd --storage-driver=overlay2
220+
```
221+
If this works, you may not need fuse-overlayfs.
222+
:::
223+
224+
## Related Documentation
225+
226+
- [Container Limitations](./limit.md#nested-container) - General limitations of nested containers
227+
- [Troubleshooting Guide](./troubleshooting.md#docker-in-container) - Additional Docker troubleshooting steps
228+
- [Security Considerations](./security.md) - Security implications of running Docker in LXC
229+
230+
## Summary
231+
232+
Running Docker inside LXC containers requires using fuse-overlayfs as a storage driver due to kernel limitations. While this adds some performance overhead, it's a reliable solution that avoids the disk space issues of the vfs driver. Always ensure FUSE support is enabled in your container and carefully manage the Docker daemon configuration to avoid conflicts.

‎docs/guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -233,6 +233,7 @@ Congratulations on setting up your RoseLab container! Here are some next steps:
233233
2. Review the [differences](./limit) between containers and dedicated servers.
234234
3. Enhance your [security](./security) by updating passwords and keys.
235235
4. If you encounter performance issues, consult the [Troubleshooting](./troubleshooting) section.
236+
5. If you need to run Docker inside your container, see our [Docker in LXC guide](./docker).
236237

237238
## Support
238239

‎docs/guide/limit.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ It is technically feasible to create a Docker container inside your container, b
7272

7373
> LXC upstream's position is that those containers aren't and cannot be root-safe. They are still valuable in an environment where you are running trusted workloads or where no untrusted task is running as root in the container. —— [Linux Containers - LXC - Security](https://linuxcontainers.org/lxc/security/)
7474
75-
Our position is Docker-based development should be deployed to the Nautilus cluster. However, you may request privilege on a short term basis with valid reason (e.g., to build a Docker image for your accepted paper). However, docker overlay2 storage driver [does not work well with LXC](https://discuss.linuxcontainers.org/t/problem-running-docker-inside-lxc-container/10392), so you may need to downgrade docker to 2.3.0 in order to use the removed `overlay` driver. See [Troubleshooting](./troubleshooting) for more information.
75+
Our position is Docker-based development should be deployed to the Nautilus cluster. However, you may request privilege on a short term basis with valid reason (e.g., to build a Docker image for your accepted paper). Note that docker overlay2 storage driver [does not work well with LXC](https://discuss.linuxcontainers.org/t/problem-running-docker-inside-lxc-container/10392) due to kernel limitations. See our [Docker in LXC guide](./docker.md) for instructions on using fuse-overlayfs with current Docker versions.
7676

7777

7878
## Firewall

‎docs/guide/troubleshooting.md‎

Lines changed: 4 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -47,53 +47,11 @@ Pulling netflow ... extracting (100.0%)
4747
ERROR: failed to register layer: Error ...
4848
```
4949

50-
This is because the Docker's latest overlay2 storage driver does not work well with LXC. However, the legacy `overlay` driver has been removed from Docker 2.4.0. You can downgrade Docker to 2.3.0 to use the `overlay` driver.
50+
This is because Docker's overlay2 storage driver does not work well with LXC containers due to kernel limitations.
5151

52-
#### Solution
53-
54-
First, remove your existing Docker installation.
55-
56-
```bash
57-
sudo systemctl stop docker
58-
sudo apt-get remove docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
59-
```
60-
61-
Then, modifies the Docker daemon configuration file:
62-
63-
```bash
64-
sudo nano /etc/docker/daemon.json
65-
```
66-
67-
add or modify the "storage-driver" option in the file:
68-
69-
```json
70-
{
71-
...,
72-
"storage-driver": "overlay"
73-
}
74-
```
75-
76-
install Docker 2.3.0.
77-
78-
```bash
79-
sudo apt-get update
80-
sudo apt-get install docker-ce=5:23.0.0-1~ubuntu.22.04~jammy docker-ce-cli=5:23.0.0-1~ubuntu.22.04~jammy containerd.io docker-buildx-plugin docker-compose-plugin
81-
```
82-
83-
Start the Docker service and verify the installation.
84-
85-
```bash
86-
sudo systemctl start docker
87-
docker --version
88-
```
89-
90-
Pin the package version to prevent automatic upgrades:
91-
92-
```bash
93-
sudo apt-mark hold docker-ce docker-ce-cli
94-
```
95-
96-
Now you can pull Docker images in your container.
52+
::: tip Solution
53+
To run Docker with current versions inside LXC containers, use the fuse-overlayfs storage driver. See our comprehensive [Docker in LXC guide](./docker.md) for detailed setup instructions.
54+
:::
9755

9856
## Machine Learning Threshold
9957

0 commit comments

Comments
 (0)