feat(web,worker): Environment overrides (#597)

Author: brendan-kellam

.env.development

@@ -84,7 +84,7 @@ SOURCEBOT_TELEMETRY_DISABLED=true # Disables telemetry collection
 # NEXT_PUBLIC_SOURCEBOT_VERSION=
 
 # CONFIG_MAX_REPOS_NO_TOKEN=
-# NODE_ENV=
+NODE_ENV=development
 # SOURCEBOT_TENANCY_MODE=single
 
 # NEXT_PUBLIC_SOURCEBOT_CLOUD_ENVIRONMENT=

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## Added
 - [Experimental][Sourcebot EE] Added GitLab permission syncing. [#585](https://github.com/sourcebot-dev/sourcebot/pull/585)
 - [Sourcebot EE] Added external identity provider config and support for multiple accounts. [#595](https://github.com/sourcebot-dev/sourcebot/pull/595)
+- Added ability to configure environment variables from the config. [#597](https://github.com/sourcebot-dev/sourcebot/pull/597)
 
 ### Fixed
 - [ask sb] Fixed issue where reasoning tokens would appear in `text` content for openai compatible models. [#582](https://github.com/sourcebot-dev/sourcebot/pull/582)

Dockerfile

@@ -42,16 +42,10 @@ COPY package.json yarn.lock* .yarnrc.yml ./
 COPY .yarn ./.yarn
 COPY ./packages/db ./packages/db
 COPY ./packages/schemas ./packages/schemas
-COPY ./packages/crypto ./packages/crypto
-COPY ./packages/error ./packages/error
-COPY ./packages/logger ./packages/logger
 COPY ./packages/shared ./packages/shared
 
 RUN yarn workspace @sourcebot/db install
 RUN yarn workspace @sourcebot/schemas install
-RUN yarn workspace @sourcebot/crypto install
-RUN yarn workspace @sourcebot/error install
-RUN yarn workspace @sourcebot/logger install
 RUN yarn workspace @sourcebot/shared install
 # ------------------------------------
 
@@ -97,9 +91,6 @@ COPY ./packages/web ./packages/web
 COPY --from=shared-libs-builder /app/node_modules ./node_modules
 COPY --from=shared-libs-builder /app/packages/db ./packages/db
 COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas
-COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto
-COPY --from=shared-libs-builder /app/packages/error ./packages/error
-COPY --from=shared-libs-builder /app/packages/logger ./packages/logger
 COPY --from=shared-libs-builder /app/packages/shared ./packages/shared
 
 # Fixes arm64 timeouts
@@ -138,9 +129,6 @@ COPY ./packages/backend ./packages/backend
 COPY --from=shared-libs-builder /app/node_modules ./node_modules
 COPY --from=shared-libs-builder /app/packages/db ./packages/db
 COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas
-COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto
-COPY --from=shared-libs-builder /app/packages/error ./packages/error
-COPY --from=shared-libs-builder /app/packages/logger ./packages/logger
 COPY --from=shared-libs-builder /app/packages/shared ./packages/shared
 RUN yarn workspace @sourcebot/backend install
 RUN yarn workspace @sourcebot/backend build
@@ -185,7 +173,6 @@ ENV DATA_DIR=/data
 ENV DATA_CACHE_DIR=$DATA_DIR/.sourcebot
 ENV DATABASE_DATA_DIR=$DATA_CACHE_DIR/db
 ENV REDIS_DATA_DIR=$DATA_CACHE_DIR/redis
-ENV REDIS_URL="redis://localhost:6379"
 ENV SRC_TENANT_ENFORCEMENT_MODE=strict
 ENV SOURCEBOT_PUBLIC_KEY_PATH=/app/public.pem
 
@@ -225,9 +212,6 @@ COPY --from=backend-builder /app/packages/backend ./packages/backend
 COPY --from=shared-libs-builder /app/node_modules ./node_modules
 COPY --from=shared-libs-builder /app/packages/db ./packages/db
 COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas
-COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto
-COPY --from=shared-libs-builder /app/packages/error ./packages/error
-COPY --from=shared-libs-builder /app/packages/logger ./packages/logger
 COPY --from=shared-libs-builder /app/packages/shared ./packages/shared
 
 # Configure dependencies

Makefile

@@ -28,10 +28,6 @@ clean:
 		packages/db/dist \
 		packages/schemas/node_modules \
 		packages/schemas/dist \
-		packages/crypto/node_modules \
-		packages/crypto/dist \
-		packages/error/node_modules \
-		packages/error/dist \
 		packages/mcp/node_modules \
 		packages/mcp/dist \
 		packages/shared/node_modules \

docs/docs/configuration/config-file.mdx

@@ -3,6 +3,9 @@ title: Config File
 sidebarTitle: Config file
 ---
 
+import ConfigSchema from '/snippets/schemas/v3/index.schema.mdx'
+import EnvironmentOverridesSchema from '/snippets/schemas/v3/environmentOverrides.schema.mdx'
+
 When self-hosting Sourcebot, you **must** provide it a config file. This is done by defining a config file in a volume that's mounted to Sourcebot, and providing the path to this
 file in the `CONFIG_PATH` environment variable. For example:
 
@@ -49,3 +52,103 @@ The following are settings that can be provided in your config file to modify So
 | `enablePublicAccess` **(deprecated)**           | boolean | false      | —       | Use the `FORCE_ENABLE_ANONYMOUS_ACCESS` environment variable instead.                  |
 | `experiment_repoDrivenPermissionSyncIntervalMs` | number  | 24 hours   | 1       | Interval at which the repo permission syncer should run.                               |
 | `experiment_userDrivenPermissionSyncIntervalMs` | number  | 24 hours   | 1       | Interval at which the user permission syncer should run.                               |
+
+# Tokens
+
+Tokens are used to securely pass secrets to Sourcebot in a config file. They are used in various places, including connections, language model providers, auth providers, etc. Tokens can be passed as either environment variables or Google Cloud secrets:
+
+<AccordionGroup>
+  <Accordion title="Environment Variables">
+    ```json
+    {
+        "token": {
+            "env": "TOKEN_NAME"
+        }
+    }
+    ```
+  </Accordion>
+  <Accordion title="Google Cloud Secrets">
+    ```json
+    {
+        "token": {
+            "googleCloudSecret": "projects/<project-id>/secrets/<secret-name>/versions/<version-id>"
+        }
+    }
+    ```
+  </Accordion>
+</AccordionGroup>
+
+# Overriding environment variables from the config
+
+You can override / set environment variables from the config file by using the `environmentOverrides` property. Overrides can be of type `string`, `number`, `boolean`, or a [token](/docs/configuration/config-file#tokens). Tokens are useful when you want to configure a environment variable using a Google Cloud Secret or other supported secret management service.
+
+<AccordionGroup>
+    <Accordion title="Token">
+    ```jsonc
+    {
+        "environmentOverrides": {
+            "DATABASE_URL": {
+                "type": "token",
+                "value": {
+                    "googleCloudSecret": "projects/<id>/secrets/postgres-connection-string/versions/latest"
+                }
+            },
+            "REDIS_URL": {
+                "type": "token",
+                "value": {
+                    "googleCloudSecret": "projects/<id>/secrets/redis-connection-string/versions/latest"
+                }
+            }
+        },
+    }
+    ```
+    </Accordion>
+
+    <Accordion title="String">
+    ```jsonc
+    {
+        "environmentOverrides": {
+            "EMAIL_FROM_ADDRESS": {
+                "type": "string",
+                "value": "hello@sourcebot.dev"
+            }
+        }
+    }
+    ```
+    </Accordion>
+
+    <Accordion title="Number">
+    ```jsonc
+    {
+        "environmentOverrides": {
+            "SOURCEBOT_CHAT_MODEL_TEMPERATURE": {
+                "type": "number",
+                "value": 0.5
+            }
+        }
+    }
+    ```
+    </Accordion>
+    
+    <Accordion title="Boolean">
+    ```jsonc
+    {
+        "environmentOverrides": {
+            "SOURCEBOT_TELEMETRY_DISABLED": {
+                "type": "boolean",
+                "value": false
+            }
+        }
+    }
+    ```
+    </Accordion>
+</AccordionGroup>
+
+
+**Note:** Overrides are **not** set as system environment variables, and instead are resolved at runtime on startup and stored in memory.
+
+<Accordion title="Schema reference">
+[schemas/v3/environmentOverrides.json](https://github.com/sourcebot-dev/sourcebot/blob/main/schemas/v3/environmentOverrides.json)
+
+<EnvironmentOverridesSchema />
+</Accordion>

docs/docs/configuration/environment-variables.mdx

@@ -1,7 +1,6 @@
 ---
 title: Environment variables 
 sidebarTitle: Environment variables 
-mode: "wide"
 ---
 
 <Note>This page provides a detailed reference of all environment variables supported by Sourcebot. If you're just looking to get up and running, we recommend starting with the [deployment guide](/docs/deployment-guide) instead.</Note>
@@ -71,3 +70,6 @@ The following environment variables allow you to configure your Sourcebot deploy
 | `REVIEW_AGENT_LOGGING_ENABLED` | `true` | <p>Enables/disables logging for the review agent. Logs are saved in `DATA_CACHE_DIR/review-agent`</p> |
 | `REVIEW_AGENT_REVIEW_COMMAND` | `review` | <p>The command used to trigger a code review by the review agent.</p> |
 
+### Overriding environment variables from the config
+
+You can override environment variables from the config file by using the `environmentOverrides` property. See [this doc](/docs/configuration/config-file#overriding-environment-variables-from-the-config) for more info.

docs/docs/configuration/idp.mdx

@@ -20,11 +20,6 @@ External identity providers can be used for [authentication](/docs/configuration
             "provider": "github",
             "purpose": "account_linking",
             "accountLinkingRequired": true,
-/*
-Secrets are provided through environment variables. Set the secret into
-an env var and provide the name here to tell Sourcebot where to get
-the value
-*/
             "clientId": {
                 "env": "GITHUB_IDENTITY_PROVIDER_CLIENT_ID"
             },
@@ -45,6 +40,8 @@ the value
 }
 ```
 
+Secret values (such as `clientId` and `clientSecret`) can be provided as environment variables or Google Cloud secrets via [tokens](/docs/configuration/config-file#tokens).
+
 # Supported External Identity Providers
 
 Sourcebot uses [Auth.js](https://authjs.dev/) to connect to external identity providers. If there's a provider supported by Auth.js that you don't see below, please submit a

docs/docs/connections/ado-cloud.mdx

@@ -86,7 +86,7 @@ If you're not familiar with Sourcebot [connections](/docs/connections/overview),
 Azure Devops Cloud requires you to provide a PAT in order to index your repositories. To learn how to create PAT, check out the [Azure Devops docs](https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Windows).
 Sourcebot needs the `Read` access for the `Code` scope in order to find and clone your repos.
 
-Next, provide the access token via an environment variable which is referenced in the `token` property:
+Next, provide the access [token](/docs/configuration/config-file#tokens) via an environment variable which is referenced in the `token` property:
 
 <Tabs>
     <Tab title="Environment Variable">

docs/docs/connections/ado-server.mdx

@@ -100,7 +100,7 @@ If you're not familiar with Sourcebot [connections](/docs/connections/overview),
 Azure Devops Server requires you to provide a PAT in order to index your repositories. To learn how to create PAT, check out the [Azure Devops docs](https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Windows).
 Sourcebot needs the `Read` access for the `Code` scope in order to find and clone your repos.
 
-Next, provide the access token via an environment variable which is referenced in the `token` property:
+Next, provide the access [token](/docs/configuration/config-file#tokens) via an environment variable which is referenced in the `token` property:
 
 <Tabs>
     <Tab title="Environment Variable">

docs/docs/connections/bitbucket-cloud.mdx

@@ -78,7 +78,7 @@ If you're not familiar with Sourcebot [connections](/docs/connections/overview),
 
 ## Authenticating with Bitbucket Cloud
 
-In order to index private repositories, you'll need to provide authentication credentials. You can do this using an `App Password` or an `Access Token`
+In order to index private repositories, you'll need to provide authentication credentials via a [token](/docs/configuration/config-file#tokens). You can do this using an `App Password` or an `Access Token`
 
 <Tabs>
     <Tab title="App Password">