fix: clean up listeners and document the intended purpose

[razvan]

Description

This issue is a pre-requisite to continue working on the Kafka 4 issue.

Currently the Kafka op creates a whole bunch of listeners (both at Kafka as well as Kubernetes level) without a clear understanding of their purpose.

The term listener here is a compound term including:

• a Kafka listener and advertised listerner property
• the Kubernetes listener volume associated with it
• the Kubernetes service associated with the listener volume.

The problem is that the code has become extremely complicated to maintain and extend, there are no tests for external access and no clear understanding if the status quo is actually correct or not.

For example:

• There are client and client_auth listeners. This is redundant. The name of the client listeners should not change because a different authentication mechanism is used.

• The naming is used interchangeably. In one part of the code client refers to client listeners and in others to internal broker listeners.

• Bug: The bootstrap SVC is always defined but the corresponding Kafka (advertised) listener is only defined when Kerberos enabled.

  • ⚠️ This seems to be intentional but there are inconsistencies in the Kafka configuration.

• When kerberos enabled, client listeners are left hanging. There is no client jaas configuration, no keytab, etc.

  • ✔️ The CLIENT listener does have a corresponding JAAS configuration and uses a different principal than the BOOTSTRAP listener.

• There is no bootstrap-controller listener. This would be client interface for kraft controllers similar to the bootstrap listener for brokers.

  • ✔️ It's not clear is this is actually needed for any practical reasons.

Implementation Questions

• ✔️ Bootstrap service questions
  • ✔️ In TLS mode, the bootstrap service scope is not included in the generated certs.
    • It is included.
  • ✔️ There is one internal service per pod but only one bootstrap service per role . Is this correct ?
    • This is intentional. The bootstrap SVC is supposed to be the only (stable?) address needed by Kafka clients.
• ✔️ Kafka service properties are passed both via command line and configuration files. This is unnecessary complicated and hard to reuse. Fixed in : refactor: move server config props from cmd line to config files #911
• ✔️ In Kerberos mode, the Java "jaas" properties are passed by command line. A better solution is to pass them as config files and also include the KafkaClient section. Partially fixed in : refactor: move server config props from cmd line to config files #911 . There is no KafkaClient entry in the JAAS file yet.
• ✔️ Clarify what enabling Kerberos actually means?
  • Client/Broker authentication? Yes ✔️
  • Broker/Broker authentication ? Yes ✔️
  • Broker/Zookeeper authentication ? 🛑 No. The Kafka client for Zookeeper doesn't use Kerberos for authentication.
  • Kraft controllers authentication ? 🛑 No. Kraft controllers only have an SSL listener.
  • Kraft controllers/Brokers ? 🛑 No. Kraft controllers only have an SSL listener.
• ❓ Kafka clients that connect using a bootstrap server will receive addresses of brokers from the associated (advertised) listener. In my understanding for every bootstrap service there should be a corresponding Kafka advertised listener.
  • In Kerberos mode, the BOOTSTRAP advertised listener is identical with the CLIENT advertised listener. Even the ports are the same (9093) even though the BOOTSTRAP listener binds to 0.0.0.0:9093. This is confusing and raises the question if the BOOTSTRAP advertised listener is actually used (because the tests are 🟢 ).
• ❓ The Kafka advertised bootstrap listener configuration is actually using the broker advertised listener configuration for the host name. Bug ?
  • See above. I tried "fixing" the BOOTSTRAP advertised listener to use the bootstrap SVC address and the correct port (9095) but this doesn't work. The brokers won't even start because they are complaining that they all try to advertise the same (single) bootrstrap SVC address (which is true) and this is illegal.
• ✔️ There is a cluster role called kafka-operator-kafka-broker-clusterrole and an associated command line argument to the operator that are not used anywhere. Fixed chore: remove unused RBAC role #914

Acceptance criteria

• There is a clear minimal list of (advertised) listeners and they are documented. Done in refactor: clean up Kafka listeners #915
• The questions (bugs) from above are addressed