CLOUDP-321076: expose prometheus metrics settings in MongoDBSearch #578

lsierant · 2025-11-07T15:51:08Z

Summary

Expose prometheus settings in MongoDBSearch spec in a new spec.prometheus field.

By default, if spec.prometheus field is not provided then metrics endpoint in mongot is disabled. This is a breaking change. Previously the metrics endpoing was always enabled on port 9946.
To enable prometheus metrics endpoint specify empty spec.prometheus: field. It will enable metrics endpoint on a default port (9946). To change the port, set it in spec.prometheus.port field.

Proof of Work

Tested in unit and e2e tests.

For e2e prometheus endpoint checks were hooked into existing e2e_search_enterprise_tls.

github-actions · 2025-11-07T15:52:08Z

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.6.0 Release Notes

New Features

MongoDBCommunity: Added support to configure custom cluster domain via newly introduced spec.clusterDomain resource field. If spec.clusterDomain is not set, environment variable CLUSTER_DOMAIN is used as cluster domain. If the environment variable CLUSTER_DOMAIN is also not set, operator falls back to cluster.local as default cluster domain.
Helm Chart: Introduced two new helm fields operator.podSecurityContext and operator.securityContext that can be used to configure securityContext for Operator deployment through Helm Chart.
MongoDBSearch:
- Switched to gRPC and mTLS for internal communication between mongod and mongot.
  - Since MCK 1.4 the mongod and mongot processess communicated using the MongoDB Wire Protocol and used keyfile authentication. This release switches that to gRPC with mTLS authentication. gRPC will allow for load-balancing search queries against multiple mongot processes in the future, and mTLS decouples the internal cluster authentication mode and credentials among mongod processes from the connection to the mongot process. The Operator will automatically enable gRPC for existing and new workloads, and will enable mTLS authentication if both Database Server and MongoDBSearch resource are configured for TLS.
- Exposed configuration settings for mongot's prometheus metrics endpoint.
  - By default, if spec.prometheus field is not provided then metrics endpoint in mongot is disabled. This is a breaking change. Previously the metrics endpoing was always enabled on port 9946.
  - To enable prometheus metrics endpoint specify empty spec.prometheus: field. It will enable metrics endpoint on a default port (9946). To change the port, set it in spec.prometheus.port field.
- Simplified MongoDB Search setup: Removed the custom Search Coordinator polyfill (a piece of compatibility code previously needed to add the required permissions), as MongoDB 8.2.0 and later now include the necessary permissions via the built-in searchCoordinator role.
- Updated the default mongodb/mongodb-search image version to 0.55.0. This is the version MCK uses if .spec.version is not specified.
- MongoDB deployments using X509 internal cluster authentication are now supported. Previously MongoDB Search required SCRAM authentication among members of a MongoDB replica set. Note: SCRAM client authentication is still required, this change merely relaxes the requirements on internal cluster authentication.

Bug Fixes

Fixed parsing of the customEnvVars Helm value when values contain = characters.
ReplicaSet: Blocked disabling TLS and changing member count simultaneously. These operations must now be applied separately to prevent configuration inconsistencies.

Other Changes

kubectl-mongodb plugin: cosign, the signing tool that is used to sign kubectl-mongodb plugin binaries, has been updated to version 3.0.2. With this change, released binaries will be bundled with .bundle files containing both signature and certificate information. For more information on how to verify signatures using new cosign version please refer to -> https://github.com/sigstore/cosign/blob/v3.0.2/doc/cosign_verify-blob.md

fealebenpae

Looks good! Couple of suggestions, and please fix the incorrect value in the Port field's comment in the CRD.

api/v1/search/mongodbsearch_types.go

fealebenpae · 2025-11-07T16:22:36Z

controllers/searchcontroller/mongodbsearch_reconcile_helper.go

-			Address: fmt.Sprintf("0.0.0.0:%d", search.GetMongotMetricsPort()),
+
+		if prometheus := search.GetPrometheus(); prometheus != nil {
+			port := search.GetMongotMetricsPort()


Perhaps we should remove GetMongotMetricsPort() from the MongoDBSearch struct and implement it on the Prometheus struct so this will read something like:

if prometheus := search.GetPrometheus(); prometheus != nil { port := prometheus.GetPort() config.Metrics = mongot.ConfigMetrics{...} }

Same thing in buildSearchHeadlessService() above.

excellent suggestion! Thanks!

fealebenpae · 2025-11-07T16:33:49Z

docker/mongodb-kubernetes-tests/tests/search/search_enterprise_tls.py

+
+def assert_search_pod_prometheus_endpoint(mdbs: MongoDBSearch, should_be_accessible: bool, port: int = 9946):
+    pod_name = f"{mdbs.name}-search-0"
+    url = f"http://localhost:{port}/metrics"


If we hit http://{mdbs.name}-search-svc:{port} directly we could omit explicitly asserting properties of the Service object in assert_search_service_prometheus_port() altogether, I think. I don't know if it's worth it to maintain the distinction between "the port was configured on the Service" and "prometheus was enabled in mongot configuration" since the former is already tested in the search reconcile helper unit test.

Agree 💯 I've reworked it a bit. I'm deploying a tools/bastion pod (community image like in snippets) to connect to the probe. Might seem like an overkill for now, but I wanted to take that opportunity and test this approach in practice as we're planning to get rid of the e2e pod entirely and we'll need some sort of proxy for the connectivity checks soon anyway.

lsierant · 2025-11-07T21:03:17Z

controllers/operator/appdbreplicaset_controller_test.go


 // getReleaseJsonPath searches for a specified target directory by traversing the directory tree backwards from the current working directory
 func getReleaseJsonPath() (string, error) {
-	repositoryRootDirName := "mongodb-kubernetes"


this failed tests when running from a different dir, e.g. a git worktree

lsierant · 2025-11-07T21:05:52Z

controllers/operator/mongodbsearch_controller_test.go

-				ctx,
-				reconcile.Request{NamespacedName: types.NamespacedName{Name: search.Name, Namespace: search.Namespace}},
-			)
-			expected, _ := workflow.OK().ReconcileResult()


thanks to the fix here it started to fail.

In order to reconcile successfully it must be always done by two steps: first reconcile is Pending (sts not ready), second time we simulate sts readiness and then it's Running. This is what checkSearchReconcileSuccessful is doing here.

Co-authored-by: Yavor Georgiev <fealebenpae@users.noreply.github.com>

lsierant · 2025-11-07T21:24:44Z

changelog/20251030_feature_update_mongodb_search_2nd_preview.md

@vinilage I've aggregated all the search-related change logs for this release under one point as those were scattered randomly across release notes. We might think about adding some additional grouping per CRD for example to make it automatic.

The new entry for this PR is the one starting from "Exposed configuration settings..."

CLOUDP-321076: expose prometheus metrics settings in MongoDBSearch

069997e

fealebenpae approved these changes Nov 7, 2025

View reviewed changes

lsierant force-pushed the lsierant/search-prom-settings branch from 3987745 to c7e1d7c Compare November 7, 2025 20:57

lsierant commented Nov 7, 2025

View reviewed changes

lsierant force-pushed the lsierant/search-prom-settings branch from c7e1d7c to 161ebd4 Compare November 7, 2025 21:14

lsierant marked this pull request as ready for review November 7, 2025 21:16

lsierant requested review from a team and vinilage as code owners November 7, 2025 21:16

lsierant requested review from MaciejKaras and anandsyncs November 7, 2025 21:16

lsierant and others added 5 commits November 7, 2025 22:21

Updated release notes for search

f17e896

Running prom endpoint test in tools pod

ff1dfa2

Running prom endpoint test in tools pod

d41c71e

Apply suggestion from @fealebenpae

6ab6bd6

Co-authored-by: Yavor Georgiev <fealebenpae@users.noreply.github.com>

Review fixes

6fc1067

lsierant force-pushed the lsierant/search-prom-settings branch from 161ebd4 to 6fc1067 Compare November 7, 2025 21:21

lsierant commented Nov 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CLOUDP-321076: expose prometheus metrics settings in MongoDBSearch #578

CLOUDP-321076: expose prometheus metrics settings in MongoDBSearch #578

Uh oh!

lsierant commented Nov 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 7, 2025 •

edited

Loading

Uh oh!

fealebenpae left a comment

Uh oh!

Uh oh!

fealebenpae Nov 7, 2025

Uh oh!

lsierant Nov 7, 2025

Uh oh!

fealebenpae Nov 7, 2025

Uh oh!

lsierant Nov 7, 2025 •

edited

Loading

Uh oh!

lsierant Nov 7, 2025

Uh oh!

lsierant Nov 7, 2025

Uh oh!

lsierant Nov 7, 2025

Uh oh!

lsierant Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CLOUDP-321076: expose prometheus metrics settings in MongoDBSearch #578

Are you sure you want to change the base?

CLOUDP-321076: expose prometheus metrics settings in MongoDBSearch #578

Uh oh!

Conversation

lsierant commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of Work

Uh oh!

github-actions bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MCK 1.6.0 Release Notes

New Features

Bug Fixes

Other Changes

Uh oh!

fealebenpae left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fealebenpae Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

lsierant Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

fealebenpae Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

lsierant Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lsierant Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

lsierant Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

lsierant Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

lsierant Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lsierant commented Nov 7, 2025 •

edited

Loading

github-actions bot commented Nov 7, 2025 •

edited

Loading

lsierant Nov 7, 2025 •

edited

Loading