Grafana bridge container does not return metrics for sensors restricted to a subset of nodes

Grafana-bridge container does not return metrics for sensors restricted to a subset of nodes

If grafana-bridge is running in a container separately from pmcollector, it does not have direct access to the pmcollector sensor configuration data. This is why either the ZIMonSensors.cfg or the mmsdrfs file must be mounted in the grafana-bridge container at startup.

There is a slight difference between the two files. ZIMonSensors.cfg includes the sensor configuration information for a local node, whereas mmsdrfs provides a cluster-wide view. For example, on a non-nsdServer node, the GPFSNSDDisk sensor is disabled, so the period for that sensor is set to 0 in the local ZIMonSensors.cfg file. The mmsdrfs file on the same node will show a sensor period greater than 0 if the sensor is enabled on any node in the cluster.

Solution

To avoid any missing data, mount mmsdrfs instead of ZIMonSensors.cfg in the grafana-bridge container at startup.

# podman run -dt -p 4242:4242,9250:9250 -e "SERVER=9.1XX.XXX.X54" -e "APIKEYVALUE=c234c7af-c2d7-43c2-97b5-3bcbddbd7a1d" -e "PORT=4242" -e "PROMETHEUS=9250" -e "PROTOCOL=http" -e "BASICAUTH=True" -e "BASICAUTHUSER=scale_admin" -e "BASICAUTHPASSW=TXlWZXJ5U3Ryb25nUGFzc3cwcmQhCg==" -e "LOGLEVEL=5" -e "RAWCOUNTERS=False" -v /tmp:/var/log/ibm_bridge_for_grafana --mount type=bind,src=mmsdrfs,target=/var/mmfs/gen/mmsdrfs,ro=true --tz CET --pod new:my-bridge-basic-auth-test-pod --name bridge-basic-auth-test scale_bridge:test_9.0.1_dev
7d735c2df8e5026209d1c817063e79949d1f07ed88af94e02983be8c7e704ec2

You can verify that the expected metrics data has been returned correctly by querying the corresponding REST API metrics endpoint via a curl call.

# podman container inspect -f '{{.NetworkSettings.IPAddress}}' bridge-basic-auth-test
10.88.0.120
# curl http://10.88.0.120:9250/metrics_gpfs_nsddisk -u scale_admin:TXlWZXJ5U3Ryb25nUGFzc3cwcmQhCg==
# HELP gpfs_nsdds_bytes_read Desc not found
# TYPE gpfs_nsdds_bytes_read counter
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk01"} 29320192.0 1764176090000
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk02"} 3262899200.0 1764176090000
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk03"} 3239584768.0 1764176090000
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk04"} 8704.0 1764176090000
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk05"} 8704.0 1764176090000
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk06"} 8704.0 1764176090000
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk07"} 8704.0 1764176090000
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk08"} 8704.0 1764176090000
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk09"} 8704.0 1764176090000
gpfs_nsdds_bytes_read{node="scale-11",gpfs_disk_name="disk10"} 8704.0 1764176090000
...