Configuration property 'catalog.config-dir' was not used

[larandvit]

Hello,

When I try to use catalog.config-dir property for coordinator, "Configuration property 'catalog.config-dir' was not used" error is encountered.

Sample of the configuration

image:
  tag: "474"
server:
  workers: 2
  node:
    environment: test
  coordinatorExtraConfig: |
    catalog.config-dir=/data/trino/etc/catalog

Thanks.

[nineinchnick]

I can't reproduce this using the following commands:

% cat test.yaml
image:
  tag: "474"
server:
  workers: 2
  node:
    environment: test
  coordinatorExtraConfig: |
    catalog.config-dir=/data/trino/etc/catalog

% helm install my-trino trino/trino --values test.yaml
% kubectl get pod
NAME                                         READY   STATUS    RESTARTS   AGE
my-trino-trino-coordinator-c5c4cccdf-l8898   1/1     Running   0          66s
my-trino-trino-worker-7c598989f5-nggbg       1/1     Running   0          66s
my-trino-trino-worker-7c598989f5-rgkzn       1/1     Running   0          66s

[larandvit]

Hi @nineinchnick

Thank you for looking to the issue.

What might be the issue from my side? I want to use the dynamic catalog functionality and I receive the error on a coordinator.

My full yaml

image:
  tag: "474"
server:
  workers: 2
  node:
    environment: test
  coordinatorExtraConfig: |
    catalog.config-dir=/data/trino/etc/catalog
service:
  port: 443
additionalConfigProperties:
  - http-server.process-forwarded=true
  - catalog.management=dynamic
catalogs:
  tpch: |
    connector.name=tpch
    tpch.splits-per-node=4
  tpcds: |
    connector.name=tpcds
    tpcds.splits-per-node=4
  memory: |
    connector.name=memory
    memory.max-data-per-node=128MB

Kubernetes error. The K8 cluster is run in Rancher.

+ launcher_opts=(--etc-dir /etc/trino)
+ grep -s -q node.id /etc/trino/node.properties
+ launcher_opts+=("-Dnode.id=${HOSTNAME}")
+ exec /usr/lib/trino/bin/launcher run --etc-dir /etc/trino -Dnode.id=trino-cluster-dev-trino-coordinator-65f788d49-75sc9
WARNING: Using incubator modules: jdk.incubator.vector
# WARNING: Unable to attach Serviceability Agent. You can try again with escalated privileges. Two options: a) use -Djol.tryWithSudo=true to try with sudo; b) echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::arrayBaseOffset has been called by org.openjdk.jol.vm.HotspotUnsafe (file:/usr/lib/trino/lib/org.openjdk.jol_jol-core-0.17.jar)
WARNING: Please consider reporting this to the maintainers of class org.openjdk.jol.vm.HotspotUnsafe
WARNING: sun.misc.Unsafe::arrayBaseOffset will be removed in a future release
Apr 17, 2025 1:48:00 PM io.airlift.log.Logger info
INFO: Java version: 24
2025-04-17T13:48:00.315Z	INFO	main	io.airlift.log.Logging	Logging to stderr
2025-04-17T13:48:00.318Z	INFO	main	Bootstrap	Loading secrets plugins
2025-04-17T13:48:00.318Z	INFO	main	Bootstrap	Loading configuration
2025-04-17T13:48:00.424Z	INFO	main	org.hibernate.validator.internal.util.Version	HV000001: Hibernate Validator 8.0.2.Final
2025-04-17T13:48:00.581Z	INFO	main	Bootstrap	Initializing logging
2025-04-17T13:48:00.648Z	INFO	main	io.airlift.log.Logging	Disabling stderr output
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::load has been called by com.sun.jna.Native in an unnamed module (file:/usr/lib/trino/lib/net.java.dev.jna_jna-5.17.0.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

2025-04-17T13:48:01.426Z	WARN	main	oshi.software.os.linux.LinuxOperatingSystem	Did not find udev library in operating system. Some features may not work.
2025-04-17T13:48:01.734Z	ERROR	main	io.trino.server.Server	Configuration is invalid
==========

Errors:

1) Configuration property 'catalog.config-dir' was not used. Did you mean to use 'catalog.store'?

==========

kubectl get pod --namespace test
NAME                                                  READY   STATUS             RESTARTS     AGE
trino-cluster-dev-trino-coordinator-65f788d49-75sc9   0/1     CrashLoopBackOff   6 (2m ago)   8m25s
trino-cluster-dev-trino-worker-c7c748c8c-fxm9q        1/1     Running            0            8m25s
trino-cluster-dev-trino-worker-c7c748c8c-ztrfp        1/1     Running            0            8m25s

Thank you.

[nineinchnick]

The docs for this property say:

Requires catalog.management to be set to static or catalog.store to be set to file. The directory with catalog property files.

See #178 for an example how to enable dynamic catalogs.

[larandvit]

Hi @nineinchnick,

Your sample was critical for me to understand the dynamic catalog setup. I appreciate your information!

The issue is in something else. I'm still receiving the same issue. It seems Trino coordinator doesn't recognize catalog.config-dir property.

I'm missing something what I can't get. If you could look your fresh eyes?

Persistent storage: catalogs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: catalogs-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 30Mi

Create persistent storage

kubectl -n dts-trino-dev create -f catalogs-pvc.yaml

Values: values-dev.yaml

image:
  tag: "474"
server:
  workers: 2
  node:
    environment: test
  coordinatorExtraConfig: |
    catalog.config-dir=/etc/trino/dynamic-catalog
service:
  port: 443
additionalConfigProperties:
  - http-server.process-forwarded=true
  - catalog.management=dynamic
coordinator:
  additionalVolumes:
    - name: catalogs
      persistentVolumeClaim:
        claimName: catalogs-pvc
  additionalVolumeMounts:
    - name: catalogs
      mountPath: /etc/trino/dynamic-catalog
      readOnly: false

Create deployment

helm install -f values-dev.yaml --namespace dts-trino-dev --create-namespace trino-cluster-dev trino/trino

Validate pods

kubectl get pod --namespace dts-trino-dev

Output

NAME                                                   READY   STATUS             RESTARTS      AGE
trino-cluster-dev-trino-coordinator-67c58f66d9-487b2   0/1     CrashLoopBackOff   5 (15s ago)   3m26s
trino-cluster-dev-trino-worker-fdcb7dcf7-22dbc         1/1     Running            0             3m26s
trino-cluster-dev-trino-worker-fdcb7dcf7-9kvsw         1/1     Running            0             3m26s

Coordinator log

+ launcher_opts=(--etc-dir /etc/trino)
+ grep -s -q node.id /etc/trino/node.properties
+ launcher_opts+=("-Dnode.id=${HOSTNAME}")
+ exec /usr/lib/trino/bin/launcher run --etc-dir /etc/trino -Dnode.id=trino-cluster-dev-trino-coordinator-67c58f66d9-487b2
WARNING: Using incubator modules: jdk.incubator.vector
# WARNING: Unable to attach Serviceability Agent. You can try again with escalated privileges. Two options: a) use -Djol.tryWithSudo=true to try with sudo; b) echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::arrayBaseOffset has been called by org.openjdk.jol.vm.HotspotUnsafe (file:/usr/lib/trino/lib/org.openjdk.jol_jol-core-0.17.jar)
WARNING: Please consider reporting this to the maintainers of class org.openjdk.jol.vm.HotspotUnsafe
WARNING: sun.misc.Unsafe::arrayBaseOffset will be removed in a future release
Apr 17, 2025 3:08:27 PM io.airlift.log.Logger info
INFO: Java version: 24
2025-04-17T15:08:27.285Z	INFO	main	io.airlift.log.Logging	Logging to stderr
2025-04-17T15:08:27.287Z	INFO	main	Bootstrap	Loading secrets plugins
2025-04-17T15:08:27.287Z	INFO	main	Bootstrap	Loading configuration
2025-04-17T15:08:27.384Z	INFO	main	org.hibernate.validator.internal.util.Version	HV000001: Hibernate Validator 8.0.2.Final
2025-04-17T15:08:27.550Z	INFO	main	Bootstrap	Initializing logging
2025-04-17T15:08:27.602Z	INFO	main	io.airlift.log.Logging	Disabling stderr output
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::load has been called by com.sun.jna.Native in an unnamed module (file:/usr/lib/trino/lib/net.java.dev.jna_jna-5.17.0.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

2025-04-17T15:08:28.218Z	WARN	main	oshi.software.os.linux.LinuxOperatingSystem	Did not find udev library in operating system. Some features may not work.
2025-04-17T15:08:28.470Z	ERROR	main	io.trino.server.Server	Configuration is invalid
==========

Errors:

1) Configuration property 'catalog.config-dir' was not used. Did you mean to use 'catalog.store'?

==========

Thank you!

[larandvit]

@nineinchnick

I have success with the configuration to store catalogs in memory

image:
  tag: "474"
server:
  workers: 2
  node:
    environment: test
  coordinatorExtraConfig: |
    catalog.store=memory
service:
  port: 443
additionalConfigProperties:
  - http-server.process-forwarded=true
  - catalog.management=dynamic
coordinator:
  additionalVolumes:
    - name: catalogs
      persistentVolumeClaim:
        claimName: catalogs-pvc
  additionalVolumeMounts:
    - name: catalogs
      mountPath: /etc/trino/dynamic-catalog
      readOnly: false

[larandvit]

@nineinchnick

What would be the best approach to resolve my issue with dynamic catalog file storage (catalog.store=file)? If you have any thoughts on it? Memory option doesn't persist catalog configurations.

[IcaroG]

I just encountered the same issue when trying to get it to work in kubernetes.
When setting this up with docker, I tried adding catalog.config-dir=/etc/trino/dynamic-catalog inside the config.properties file and it was also not working. It only started working when I created a new file catalog-store.properties (not documented) with the specific catalog.config-dir.

Can we reopen this issue please?

[nineinchnick]

@IcaroG I was not aware of that, thanks for pointing it out!

[nineinchnick]

The confusion comes from catalog.config-dir being defined in two places:

• StaticCatalogManagerConfig.java - which reads it from config.properties
• FileCatalogStoreConfig.java - which reads it from catalog-store.properties here: https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStoreManager.java#L84, and this is only used when dynamic catalogs are enabled

[IcaroG]

I would argue that this is not an issue with the chart itself, but with the documentation at https://trino.io/docs/current/admin/properties-catalog.html

You can correctly set it up with the chart with this config:

    coordinator:
      additionalConfigFiles:
        catalog-store.properties: |
          catalog.config-dir=/etc/trino/dynamic-catalog

[larandvit]

Thank you, @IcaroG, for your research. I tried it and it seems working but I couldn't make it work completely.

When I create a dynamic catalog, I get "Could not store catalog properties" error. The error is caused by the mount permissions.

drwxr-xr-x 3 root root 1024 May 14 01:07 dynamic-catalog

Root only has write access to the mount. I'm not sure how to change it.

[IcaroG]

To fix the permission issue you just need to add

    securityContext:
      fsGroup: 1000

There are other ways too, like initContainers, but this is how I did it.