Failure to get data in Prometheus on latest Docker

[zacharyshale95]

Ubuntu 24.04.3 LTS
Docker version 29.0.0, build 3d4129b

I1111 00:05:15.302059 1 manager.go:233] Version: {KernelVersion:6.14.0-35-generic ContainerOsVersion:Alpine Linux v3.18 DockerVersion: DockerAPIVersion: CadvisorVersion:v0.49.1 CadvisorRevision:6f3f25ba}
I1111 00:05:15.302274 1 factory.go:219] Registration of the podman container factory failed: failed to validate Podman info: response not present: Get "http://d/v1.0.0/info": dial unix /var/run/podman/podman.sock: connect: no such file or directory
I1111 00:05:15.304095 1 factory.go:145] Registering containerd factory
I1111 00:05:15.304109 1 factory.go:221] Registration of the containerd container factory successfully
I1111 00:05:15.304114 1 factory.go:55] Registering systemd factory
I1111 00:05:15.304117 1 factory.go:221] Registration of the systemd container factory successfully
I1111 00:05:15.304220 1 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I1111 00:05:15.305482 1 factory.go:219] Registration of the mesos container factory failed: unable to create mesos agent client: failed to get version
I1111 00:05:15.305757 1 factory.go:219] Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: Error response from daemon: client version 1.41 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version
I1111 00:05:15.305951 1 factory.go:103] Registering Raw factory
I1111 00:05:15.305975 1 manager.go:1196] Started watching for new ooms in manager
W1111 00:05:15.305997 1 manager.go:306] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
I1111 00:05:15.306260 1 manager.go:319] Starting recovery of all containers
I1111 00:05:15.350626 1 manager.go:324] Recovery completed
I1111 00:05:15.361028 1 cadvisor.go:176] Starting cAdvisor version: v0.49.1-6f3f25ba on port 8080

Data for prom/node-exporter working fine, but I get no data from cAdvisor on nodes that I have updated docker to 29.0.0

Seeing similar errors in other containers as well, e.g.: I1111 00:05:15.305757 1 factory.go:219] Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: Error response from daemon: client version 1.41 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version

[tovarischbr]

Same here, had to rollback to docker 28.5.2 for now (on Debian 13)

[SuzukiHonoka]

Same here.

Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: Error response from daemon: client version 1.41 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version

[vigenere23]

This is because their documentation and versioning is poorly done.

1. The latest tag on gcr.io is pinned at 0.49.1, which is a year old
2. The latest pushed image on gcr.io is 0.52.0, which dates back to april
3. The new images are pushed on ghcr.io (github container registry instead of google cloud registry)
4. The new images on ghcr.io do not have the latest tag

Using the newer ghcr.io/google/cadvisor:v0.53.0 image resolves this issue. However, I still have other problems on my end, but they are most probably specific to my setup.

Ongoing PR for a fix in the README : #3740

[jrgn9]

Using the newer ghcr.io/google/cadvisor:v0.53.0 image resolves this issue. However, I still have other problems on my end, but they are most probably specific to my setup.

I have this issue, and have explicitly been using ghcr.io/google/cadvisor:v0.53.0

[nouse]

I have upgraded to ghcr.io/google/cadvisor:v0.53.0 , but it doesn't work docker 29.

I can confirm docker 29 enables "containerd-snapshotter" by default, but #3709 is not merged neither released.

[tmelikoff]

May be related to this breaking change in Docker 29? I have a similar issue with Portainer.

[andrelucca]

The comment from @tmelikoff is indeed the cause of this issue.
Docker Engine 29 introduced breaking changes to Docker API, so any containers that need Docker API is most likely to have problems.
There is a workaround though, at least until the official fix is released on cadivsor, I already tested on my end and it is working.
This fix consist in editing the docker.service systemd file to add an environment variable forcing it to specify an older version of the Docker API. This solution is simple and easier compared to downgrading the Docker Engine.
The steps are the following:

1. Run sudo systemctl edit docker.service to edit the docker.service systemd file.

2. Paste the following lines before the comment ### Lines below this comment will be discarded :

[Service]
Environment=DOCKER_MIN_API_VERSION=1.24

3. Restart docker.service by running sudo systemctl restart docker.service

This solution was originally posted on Portainer GitHub Issues (wich is also suffering from the breaking changes on Docker API), and also solves the problems happening to cAdvisor for now.

[vigenere23]

I think there are 2 unrelated errors described in this issue and their comments.

1 - Docker API version new requirements

Docker 29 did drop support for api versions below 1.44, which cadvisor v0.49.1 (latest on gcr.io) still use (using v1.41 as described in the error below).

Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: Error response from daemon: client version 1.41 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version

This is easily fixed by migrating to the latest cadvisor v0.53.0. In my logs after update (event on Docker 29) :

Registration of the docker container factory successfully

2 - New containerd-snapshotter image store

As mentioned by @nouse :

I can confirm docker 29 enables "containerd-snapshotter" by default, but #3709 is not merged neither released.

This is related to the errors you get once using docker 29 :

 Failed to create existing container: /system.slice/docker-e97b4c581c672f9ffbb4cb14adf5115899ac4c8929102d9f649dd9671b8fca11.scope: failed to identify the read-write layer ID for container "e97b4c581c672f9ffbb4cb14adf5115899ac4c8929102d9f649dd9671b8fca11". - open /rootfs/var/lib/docker/image/overlayfs/layerdb/mounts/e97b4c581c672f9ffbb4cb14adf5115899ac4c8929102d9f649dd9671b8fca11/mount-id: no such file or directory

Modifying the /etc/docker/daemon.json and putting these entry does fix the problem :

  "features": {
    "cdi": true,
    "containerd-snapshotter": false
  }

(you most certainly need to restart the docker daemon for changes to take effect, and I'm not sure if a migration is needed since I boot fresh from a VM).

[wollomatic]

Situation here: Cadvisor v0.53.0 works with Docker 29 with migrated Docker store, but not with Docker 29 with containerd-snapshotter enabled.
(Edit) I think this confirms what @vigenere23 wrote.