|
| 1 | +--- |
| 2 | +layout: docs |
| 3 | +page_title: Vault Secrets Operator CSI driver troubleshooting |
| 4 | +description: >- |
| 5 | + Understand why your installation of the |
| 6 | + Vault Secrets Operator CSI driver might not be working properly. |
| 7 | +--- |
| 8 | + |
| 9 | +# Troubleshooting the Vault Secrets Operator CSI driver |
| 10 | + |
| 11 | +## Driver not appearing in cluster |
| 12 | + |
| 13 | +Verify that you have enabled it with the `csi.enabled` field on the Vault Secrets Operator Helm chart. You can do this by either adding the field to your `values.yaml` file, |
| 14 | +or by passing it directly with the `--set` flag: |
| 15 | + |
| 16 | +```shell-session |
| 17 | +$ helm upgrade --install \ |
| 18 | + --set "csi.enabled=true" \ |
| 19 | + --namespace vault-secrets-operator-system \ |
| 20 | + vault-secrets-operator hashicorp/vault-secrets-operator |
| 21 | +``` |
| 22 | + |
| 23 | +## Application won't start |
| 24 | + |
| 25 | +If your application Pod won't start up after mounting a secret from the CSI driver, check your Pod events for a `FailedMount` message: |
| 26 | + |
| 27 | +<CodeBlockConfig hideClipboard> |
| 28 | + |
| 29 | +```shell |
| 30 | +Warning FailedMount 7s (x8 over 71s) kubelet |
| 31 | +MountVolume.SetUp failed for volume "csi-secrets" : |
| 32 | +... |
| 33 | +``` |
| 34 | + |
| 35 | +</CodeBlockConfig> |
| 36 | + |
| 37 | +The next line shows the specific error. |
| 38 | + |
| 39 | +### Not found |
| 40 | + |
| 41 | +If you see the following error message: |
| 42 | + |
| 43 | +<CodeBlockConfig hideClipboard> |
| 44 | + |
| 45 | +```shell |
| 46 | +rpc error: code = Internal desc = csisecrets.secrets.hashicorp.com "csi-secrets" not found |
| 47 | +``` |
| 48 | + |
| 49 | +</CodeBlockConfig> |
| 50 | + |
| 51 | +This likely means that you didn't reference the `CSISecrets` volume correctly. |
| 52 | +Check that you have specified the correct name and namespace for your CSISecrets volume in your application Pod spec. |
| 53 | + |
| 54 | +```yaml |
| 55 | + volumes: |
| 56 | + - name: csi-secrets |
| 57 | + csi: |
| 58 | + driver: csi.vso.hashicorp.com |
| 59 | + volumeAttributes: |
| 60 | + csiSecretsName: csi-secrets |
| 61 | + csiSecretsNamespace: admin |
| 62 | +``` |
| 63 | +
|
| 64 | +The `csiSecretsName` field should match the one in the `metadata.name` field of the CSISecrets resource that you have deployed. |
| 65 | +The `csiSecretsNamespace` field should match the one in the `metadata.namespace` field of the CSISecrets resource. |
| 66 | +Confirm that you are not accidentally setting it to the value in the `spec.namespace` field, which refers to a Vault namespace. |
| 67 | + |
| 68 | +### Not authorized |
| 69 | + |
| 70 | +If you see the following error message: |
| 71 | + |
| 72 | +<CodeBlockConfig hideClipboard> |
| 73 | + |
| 74 | +```shell |
| 75 | +rpc error: code = PermissionDenied desc = not authorized for secret sync |
| 76 | +``` |
| 77 | + |
| 78 | +</CodeBlockConfig> |
| 79 | + |
| 80 | +This likely means that your requesting application Pod name, namespace, service account does not |
| 81 | +match the regex pattern defined in the CSISecrets resource `accessControl` stanza, or that |
| 82 | +your container name does not match the pattern defined in the `containerState` stanza. |
| 83 | + |
| 84 | +```yaml |
| 85 | +accessControl: |
| 86 | + serviceAccountPattern: "default" |
| 87 | + namespacePatterns: |
| 88 | + - "default" |
| 89 | + podNamePatterns: |
| 90 | + - "^csi-test-app-" |
| 91 | +syncConfig: |
| 92 | + containerState: |
| 93 | + namePattern: "^(app|sidecar)$" |
| 94 | +``` |
| 95 | + |
| 96 | +These values should be [Go regular expressions](https://pkg.go.dev/regexp/syntax). |
| 97 | + |
| 98 | +### Incorrect auth configuration |
| 99 | + |
| 100 | +If you see one of the following error messages: |
| 101 | + |
| 102 | +<CodeBlockConfig hideClipboard> |
| 103 | + |
| 104 | +```shell |
| 105 | +rpc error: code = Internal desc = vaultauths.secrets.hashicorp.com "default" not found |
| 106 | +``` |
| 107 | + |
| 108 | +</CodeBlockConfig> |
| 109 | + |
| 110 | +or |
| 111 | + |
| 112 | +<CodeBlockConfig hideClipboard> |
| 113 | + |
| 114 | +```shell |
| 115 | +rpc error: code = Internal desc = failed getting default/default, err= │ |
| 116 | +│ vaultauthglobals.secrets.hashicorp.com "default" not found |
| 117 | +``` |
| 118 | + |
| 119 | +</CodeBlockConfig> |
| 120 | + |
| 121 | +you may have set up your authentication incorrectly. |
| 122 | + |
| 123 | + |
| 124 | +The Vault Secrets Operator CSI driver requires a VaultAuth and a VaultConnection to authenticate. |
| 125 | +For more detailed information about these resources, refer to the Vault Secrets Operator guide for |
| 126 | +[Vault authentication](/vault/docs/deploy/kubernetes/vso/sources/vault/auth). |
| 127 | + |
| 128 | +Verify that the following statements about your authentication setup are true: |
| 129 | +- The CSISecrets is referencing the correct name and namespace for a VaultAuth in its `vaultAuthRef`. |
| 130 | +- The VaultAuth is able to find the VaultAuthGlobal in the right namespace. (Try explicitly setting `namespace` rather than relying on defaults.) |
| 131 | +- The VaultAuthGlobal has allowedNamespaces set to include the namespace of the VaultAuth. |
| 132 | +- The VaultAuthGlobal references a valid VaultConnection. |
| 133 | +- The auth method used by the VaultAuth has been set up in Vault with a valid policy. |
| 134 | + |
| 135 | +## Application can't interact with Vault |
| 136 | + |
| 137 | +Application Pods do not need to connect to Vault for KV secrets; the Vault Secrets Operator CSI driver does this on behalf of the requesting Pod at container startup. |
| 138 | + |
| 139 | +However, if your application Pod is using the Vault Secrets Operator CSI driver to generate an AppRole secret ID for subsequent logins to Vault, you have |
| 140 | +to set up the [AppRole auth method](/vault/docs/auth/approle) in Vault ahead of time. |
| 141 | + |
| 142 | +If you see errors when logging into Vault with your secret ID, verify the following: |
| 143 | +- If using response-wrapping, the wrapping token's TTL has not expired. |
| 144 | +- If using response-wrapping, you did not already unwrap the wrapping token, which may have been set to one-time use. |
| 145 | + |
| 146 | +If you are able to log in, but cannot perform expected actions after that, verify the following: |
| 147 | +- The AppRole role has a Vault policy that would allow it to do whatever Vault operations the application needs to do next, like read from a certain KV mount. |
| 148 | +- You have some method for renewing your tokens after you've logged into Vault with AppRole, |
| 149 | +such as a [lifetime watcher](https://pkg.go.dev/github.com/hashicorp/vault/api#LifetimeWatcher) or short-lived [periodic tokens](/vault/docs/concepts/tokens#periodic-tokens). |
| 150 | +If the token is truly expired, you must restart the pod to have the CSI driver regenerate a new secret ID to log in with. |
| 151 | + |
| 152 | +You should also verify that your Vault policy for the CSI driver contains the following permissions, so that it can generate secret IDs: |
| 153 | + |
| 154 | +```hcl |
| 155 | +path "auth/approle/role/my-role/secret-id" { |
| 156 | + capabilities = ["update"] |
| 157 | +} |
| 158 | +
|
| 159 | +path "auth/approle/role/my-role/role-id" { |
| 160 | + capabilities = ["read"] |
| 161 | +} |
| 162 | +
|
| 163 | +path "sys/license/status" { |
| 164 | + capabilities = ["read"] |
| 165 | +} |
| 166 | +``` |
0 commit comments