From e96a679024b9ece4a9e1a1408833419e354e2d5b Mon Sep 17 00:00:00 2001 From: Nitwel Date: Fri, 26 Sep 2025 14:28:22 +0200 Subject: [PATCH 1/4] initial commit --- .../{3.testing.md => 3.testing-2.md} | 2 +- content/community/3.codebase/4.testing.md | 104 ++++++++++++++++++ content/community/3.codebase/5.sandbox-cli.md | 73 ++++++++++++ 3 files changed, 178 insertions(+), 1 deletion(-) rename content/community/3.codebase/{3.testing.md => 3.testing-2.md} (99%) create mode 100644 content/community/3.codebase/4.testing.md create mode 100644 content/community/3.codebase/5.sandbox-cli.md diff --git a/content/community/3.codebase/3.testing.md b/content/community/3.codebase/3.testing-2.md similarity index 99% rename from content/community/3.codebase/3.testing.md rename to content/community/3.codebase/3.testing-2.md index 6a2bf559..caf27cc2 100644 --- a/content/community/3.codebase/3.testing.md +++ b/content/community/3.codebase/3.testing-2.md @@ -23,7 +23,7 @@ pnpm --filter app test # Start tests in watch mode pnpm --filter api test -- --watch -# Enable coverage report +# Enable coverage reporwt pnpm --filter api test -- --coverage # Run specific test files using a filter pattern diff --git a/content/community/3.codebase/4.testing.md b/content/community/3.codebase/4.testing.md new file mode 100644 index 00000000..f2ef2e07 --- /dev/null +++ b/content/community/3.codebase/4.testing.md @@ -0,0 +1,104 @@ +--- +title: Testing +description: How to run unit and blackbox tests in the Directus codebase. +--- + +Directus has two main methods of testing, being through unit and blackbox tests. Unit tests are located in each package and can be run from the pacakage itself. Blackbox tests on the other hand are located in `tests/blackbox`. + +## Running tests + +### Unit tests +```bash +# Run from the package that you want to start the tests from +pnpm test +``` + +### Blackbox tests + +```bash +# Test against directus running with postgres +pnpm test + +# Test against all databases +pnpm test:all + +# Test against a specific database. The project option can be used multiple times to test against multiple different databases at the same time. +pnpm vitest --project sqlite +``` + +### Vitest options + +Both unit and blackbox tests are running through [Vitest](https://vitest.dev) and thus support all options that vitest has to offer for customizing what tests to run. + +```bash +# Run all tests that have "permission" in their filename. +pnpm test permissions + +# Run all tests and watch for changes in the test files. +pnpm test -w +``` + +For more options, see [here](https://vitest.dev/guide/cli.html). + +## Writing tests + +The basic test structure goes as follows. + +1. (optionally) Create a new folder for your tests +2. Create your test file, ending with `.test.ts` +3. You can start with this template: + +```ts +import { createDirectus, rest, serverPing, staticToken } from '@directus/sdk'; +import { useSnapshot } from '@utils/useSnapshot.js'; +import { expect, test } from 'vitest'; + +import { port } from '@utils/constants.js'; +import type { Schema } from './schema.js'; + +const api = createDirectus(`http://localhost:${port}`) + .with(rest()) + .with(staticToken('admin')); + +const { collections } = await useSnapshot(api); + +test('ping', async () => { + const result = await api.request(serverPing()); + + expect(result).toBe('pong'); +}); +``` + +### Using a custom schema + +Blackbox tests allow you to easily setup custom schemas for testing using the `useSnapshot` function. By default, the function uses the `snapshot.json` file that should be located on the same level as your test file. + +In oder to create such a schema and types for said schema quickly, use the sandbox cli: + +```bash +# Run this in the same folder as your tests +sandbox -s -x postgres +``` + +The `-x` option hereby exports the schema every 2s and the `-s` starts the directus instance using a preexisting `snapshot.json` so editing an exisitng snapshot is also quickly possible. + +When creating collections, make that each collection ends with `_1234` as this is required to ensure that each collection will have a uniqe name in the blackbox tests. + + + + + +1. While writing tests, make sure that reruning a test doesn't cause conflicts. E.g. using `randomUUID()` to avoid user + or permission collisions. + +> The blackbox tests rely on the `@directus/sandbox` package for starting up the api so have a look there for more +> configuration options. + +### Creating a custom schema + +If a custom schema is required for your tests, make sure that each collection ends with `_1234`, that way a unique +collection name can be guaranteed. + +A custom schema can be created by running the sandbox cli: `sandbox -x postgres` in the folder of your tests. This +automatically writes the `schema.d.ts` and `snapshot.json` into the current folder. If you quickly want to modify an +existing schema, use `sandbox -s -x postgres` to start up a sandbox with the schema preloaded. diff --git a/content/community/3.codebase/5.sandbox-cli.md b/content/community/3.codebase/5.sandbox-cli.md new file mode 100644 index 00000000..720b8a99 --- /dev/null +++ b/content/community/3.codebase/5.sandbox-cli.md @@ -0,0 +1,73 @@ +--- +title: Sandbox Cli +description: How to make use of the Sandbox Cli and Api when working with directus. +--- + +Utility functions for quickly spinning up and down instances of directus for usage such as testing or development. + +## Usage + +The package offers two ways of interacting, either through calling JS functions or through accessing the command line +interface. + +### CLI + +``` +Usage: sandbox [options] + +Arguments: + database What database to start the api with (choices: "maria", "cockroachdb", "mssql", "mysql", "oracle", "postgres", "sqlite") + +Options: + -b, --build Rebuild directus from source + -d, --dev Start directus in developer mode. Not compatible with build + -w, --watch Restart the api when changes are made + --inspect Start the api with debugger (default: true) + -p, --port Port to start the api on + -x, --export Export the schema to a file every 2 seconds + -s, --schema [schema] Load an additional schema snapshot on startup + --docker.basePort Minimum port number to use for docker containers + --docker.keep Keep containers running when stopping the sandbox + --docker.name Overwrite the name of the docker project + -e, --extras Enable redis,maildev,saml or other extras + -i, --instances Horizontally scale directus to a given number of instances (default: "1") + --killPorts Forcefully kills all processes that occupy ports that the api would use + -h, --help display help for command +``` + +### API + +The api is accessed through the following two functions: + +- [sandbox](docs/functions/sandbox.md) +- [sandboxes](docs/functions/sandboxes.md) + +#### Example + +```ts +import { sandbox } from '@directus/sandbox'; + +const sb = await sandbox('postgres', { dev: true }); + +// Interact via Rest, GQL or WebSockets +const result = await fetch(sb.env.PUBLIC_URL + '/items/articles'); + +console.log(await result.json()); + +await sb.close(); +``` + +## Inner workings + +Depending on what is set in the configuration, some of these steps might be skipped: + +1. **Building of the api**: If enabled, the api is freshly build each time the sandbox is started. Use the `watch` + option to quickly iterate on changes. +2. **Starting of the docker containers**: All required docker containers like databases or extras like redis are spun up + and awaited until healthy. If the docker containers are still running, they will be reused instead of starting up new + ones. +3. **Bootstrapping of the database**: If not already bootstrapped, the sandbox will make sure that all necessary + database tables are created. +4. **Loading of a schema snapshot**: In case the `schema` option is set, the database will also the snapshot applied + before starting directus. +5. **Starting of the api**: Finally, the api(s) are spun up with the right environment variables configured. From 6453a4fab53bdcbc366ddb98d2fcdeb109f0ee7c Mon Sep 17 00:00:00 2001 From: Nitwel Date: Fri, 26 Sep 2025 18:20:23 +0200 Subject: [PATCH 2/4] work on docs --- content/community/3.codebase/3.testing-2.md | 199 ------------------ .../3.codebase/{4.testing.md => 3.testing.md} | 66 ++++-- .../{5.sandbox-cli.md => 4.sandbox-cli.md} | 4 +- 3 files changed, 52 insertions(+), 217 deletions(-) delete mode 100644 content/community/3.codebase/3.testing-2.md rename content/community/3.codebase/{4.testing.md => 3.testing.md} (55%) rename content/community/3.codebase/{5.sandbox-cli.md => 4.sandbox-cli.md} (94%) diff --git a/content/community/3.codebase/3.testing-2.md b/content/community/3.codebase/3.testing-2.md deleted file mode 100644 index caf27cc2..00000000 --- a/content/community/3.codebase/3.testing-2.md +++ /dev/null @@ -1,199 +0,0 @@ ---- -title: Testing -description: How to run unit and blackbox tests in the Directus codebase. ---- - -The current test strategy for Directus consists of blackbox tests, which test the overall functionality of the platform, as well as unit tests, which test individual parts of the codebase. - -## Running Unit Tests - -Use the following command to perform unit tests in all packages: - -```bash -pnpm --workspace-root test -``` - -Use one of the following commands to perform more specific actions with unit tests (mix and match as desired): - -```bash -# Run tests for a specific package (for example only in the api or app package) -pnpm --filter api test -pnpm --filter app test - -# Start tests in watch mode -pnpm --filter api test -- --watch - -# Enable coverage reporwt -pnpm --filter api test -- --coverage - -# Run specific test files using a filter pattern -pnpm --filter api test -- app.test.ts -pnpm --filter api test -- utils -``` - -::callout{icon="material-symbols:info-outline"} -**Relative Commands** - -If you are already in a directory of a specific package, you may omit the `--filter` flag in `pnpm` commands since the commands will be executed relative to the current directory. - -```bash -# Run API tests, from within the "/api" directory -pnpm test -``` - -:: - -## Running Blackbox Tests - -Install [Docker](https://docs.docker.com/get-docker/) and ensure that the service is up and running. Run the following commands to start the blackbox tests: - -```bash -# Ensure that you are testing against the lastest state of the codebase -pnpm --workspace-root build - -# Clean up in case you ran the tests before -pnpm --filter tests-blackbox exec docker compose down --volumes -# Start the containers required for the tests -pnpm --filter tests-blackbox exec docker compose up --detach --wait - -# Deploy Directus and run the tests -## Run common tests unrelated to database -pnpm --workspace-root test:blackbox -- --project common -## Run database specific tests -pnpm --workspace-root test:blackbox -- --project db -``` - -Subsequent test runs can be issued with the following command, if only modifications to the blackbox tests themselves have been made: - -```bash -## Run common tests unrelated to database -pnpm --filter tests-blackbox test --project common -## Run database specific tests -pnpm --filter tests-blackbox test --project db -``` - -### Testing Specific Database Vendors - -Provide a CSV of database vendors via the `TEST_DB` environment variable to target only a specific subset: - -```bash -# Example targeting multiple vendors -TEST_DB=cockroachdb,postgres pnpm --workspace-root test:blackbox -- --project db - -# Example targeting a single vendor -TEST_DB=sqlite3 pnpm --workspace-root test:blackbox -- --project db -``` - -If tests are only run against a subset of databases, it also makes sense to only start the corresponding containers: - -```bash -# Start the containers that are always required -pnpm --filter tests-blackbox exec docker compose up auth-saml redis minio minio-mc --detach --wait - -# Start the specific database container (for example 'postgres') -pnpm --filter tests-blackbox exec docker compose up postgres --detach --wait -``` - -### Using an Existing Directus Project - -Usually, the test suite will spin up a fresh copy of the Directus API built from the current state of the codebase. To use an already running instance of Directus instead, enable the `TEST_LOCAL` flag: - -```bash -TEST_DB=cockroachdb TEST_LOCAL=true pnpm --workspace-root test:blackbox -- --project db -``` - -Note: The tests expect the instance running at `localhost:8055`. Make sure to connect the instance to the test database container found in the `tests/blackbox/docker-compose.yml` file. - -### Server Logs - -For debugging purposes, server logs can be enabled by specifying a log level using the `TEST_SAVE_LOGS` flag, for example: - -```bash -TEST_SAVE_LOGS=info pnpm --workspace-root test:blackbox -- --project db -``` - -The log files will be available under `tests/blackbox/server-logs-*`. - -## Writing Unit Tests - -Unit Tests are written throughout the codebase in a vite native unit test framework called [Vitest](https://vitest.dev). - -### Example - -```ts [/directus/api/src/utils/get-date-formatted.test.ts] -import { afterEach, beforeEach, expect, test, vi } from 'vitest'; - -import { getDateFormatted } from './get-date-formatted.js'; - -beforeEach(() => { - vi.useFakeTimers(); -}); - -afterEach(() => { - vi.useRealTimers(); -}); - -function getUtcDateForString(date: string) { - const now = new Date(date); - - // account for timezone difference depending on the machine where this test is ran - const timezoneOffsetInMinutes = now.getTimezoneOffset(); - const timezoneOffsetInMilliseconds = timezoneOffsetInMinutes * 60 * 1000; - const nowUTC = new Date(now.valueOf() + timezoneOffsetInMilliseconds); - - return nowUTC; -} - -test.each([ - { utc: '2023-01-01T01:23:45.678Z', expected: '20230101-12345' }, - { utc: '2023-01-11T01:23:45.678Z', expected: '20230111-12345' }, - { utc: '2023-11-01T01:23:45.678Z', expected: '20231101-12345' }, - { utc: '2023-11-11T12:34:56.789Z', expected: '20231111-123456' }, - { utc: '2023-06-01T01:23:45.678Z', expected: '20230601-12345' }, - { utc: '2023-06-11T12:34:56.789Z', expected: '20230611-123456' }, -])('should format $utc into "$expected"', ({ utc, expected }) => { - const nowUTC = getUtcDateForString(utc); - - vi.setSystemTime(nowUTC); - - expect(getDateFormatted()).toBe(expected); -}); -``` - -## Writing Blackbox Tests - -### Example - -```ts [/directus/tests/blackbox/routes/server/ping.test.ts] -import { getUrl } from '@common/config'; -import request from 'supertest'; -import vendors from '@common/get-dbs-to-test'; -import { requestGraphQL } from '@common/transport'; - -describe('/server', () => { - describe('GET /ping', () => { - it.each(vendors)('%s', async (vendor) => { - // Action - const response = await request(getUrl(vendor)) - .get('/server/ping') - .expect('Content-Type', /text\/html/) - .expect(200); - - const gqlResponse = await requestGraphQL(getUrl(vendor), true, null, { - query: { - server_ping: true, - }, - }); - - // Assert - expect(response.text).toBe('pong'); - expect(gqlResponse.body.data.server_ping).toBe('pong'); - }); - }); -}); -``` - - - - - diff --git a/content/community/3.codebase/4.testing.md b/content/community/3.codebase/3.testing.md similarity index 55% rename from content/community/3.codebase/4.testing.md rename to content/community/3.codebase/3.testing.md index f2ef2e07..d6bc9ae9 100644 --- a/content/community/3.codebase/4.testing.md +++ b/content/community/3.codebase/3.testing.md @@ -15,6 +15,10 @@ pnpm test ### Blackbox tests +::callout{icon="material-symbols:warning-rounded" color="info"} +[Docker](https://docs.docker.com/get-docker/) is required to run extensions locally. +:: + ```bash # Test against directus running with postgres pnpm test @@ -40,7 +44,23 @@ pnpm test -w For more options, see [here](https://vitest.dev/guide/cli.html). -## Writing tests +## Writing unit tests + +Unit Tests are written throughout the codebase in a vite native unit test framework called [Vitest](https://vitest.dev). + +```ts +import { afterEach, beforeEach, expect, test, vi } from 'vitest'; + +import { formatTitle } from './format-title.js'; + +test('should format utc string', () => { + const result = formatTitle('hello-world') + + expect(result).toBe('Hello World'); +}); +``` + +## Writing blackbox tests The basic test structure goes as follows. @@ -56,9 +76,7 @@ import { expect, test } from 'vitest'; import { port } from '@utils/constants.js'; import type { Schema } from './schema.js'; -const api = createDirectus(`http://localhost:${port}`) - .with(rest()) - .with(staticToken('admin')); +const api = createDirectus(`http://localhost:${port}`).with(rest()).with(staticToken('admin')); const { collections } = await useSnapshot(api); @@ -84,21 +102,35 @@ The `-x` option hereby exports the schema every 2s and the `-s` starts the direc When creating collections, make that each collection ends with `_1234` as this is required to ensure that each collection will have a uniqe name in the blackbox tests. +### Avoiding naming conflicts +If you manually want to create collections or other things like users, you have to make sure that these are unique across tests runs and different tests. +A good way to do that is either to use the `getUID()` or `randomUUID()` functions. `getUID()` returns a string unique to the file that you're currently running in but returns the same string for there same file whereas `randomUUID()` will awalys return a random UUID. +### Launching a custom directus instance +It is possible to spin up separate directus instances inside of a test file itself. This is useful if you want to test against cases with e.g. horizontal scaling. -1. While writing tests, make sure that reruning a test doesn't cause conflicts. E.g. using `randomUUID()` to avoid user - or permission collisions. - -> The blackbox tests rely on the `@directus/sandbox` package for starting up the api so have a look there for more -> configuration options. - -### Creating a custom schema - -If a custom schema is required for your tests, make sure that each collection ends with `_1234`, that way a unique -collection name can be guaranteed. +```ts +import { sandbox } from '@directus/sandbox'; +import { database } from '@utils/constants.js'; +import getPort from 'get-port'; +import { getUID } from '@utils/getUID.js'; + +const port = await getPort() + +const directus = await sandbox(database, { + port, + schema: join(import.meta.dirname, 'snapshot.json'), // Custom schema to start the instance with + inspect: false, + silent: true, + docker: { + basePort: port + 1, + suffix: getUID(), // make sure this is unique as it could collide with other docker project names + }, + }); +``` -A custom schema can be created by running the sandbox cli: `sandbox -x postgres` in the folder of your tests. This -automatically writes the `schema.d.ts` and `snapshot.json` into the current folder. If you quickly want to modify an -existing schema, use `sandbox -s -x postgres` to start up a sandbox with the schema preloaded. +::callout{icon="material-symbols:info-outline" color="info"} +For more information about the configuration options for the sandbox cli, see [here](/community/codebase/sandbox-cli). +:: diff --git a/content/community/3.codebase/5.sandbox-cli.md b/content/community/3.codebase/4.sandbox-cli.md similarity index 94% rename from content/community/3.codebase/5.sandbox-cli.md rename to content/community/3.codebase/4.sandbox-cli.md index 720b8a99..d98ad682 100644 --- a/content/community/3.codebase/5.sandbox-cli.md +++ b/content/community/3.codebase/4.sandbox-cli.md @@ -12,7 +12,7 @@ interface. ### CLI -``` +```bash Usage: sandbox [options] Arguments: @@ -29,7 +29,9 @@ Options: --docker.basePort Minimum port number to use for docker containers --docker.keep Keep containers running when stopping the sandbox --docker.name Overwrite the name of the docker project + --docker.suffix Adds a suffix to the docker project. Can be used to ensure uniqueness -e, --extras Enable redis,maildev,saml or other extras + --silent Silence all logs except for errors -i, --instances Horizontally scale directus to a given number of instances (default: "1") --killPorts Forcefully kills all processes that occupy ports that the api would use -h, --help display help for command From 7243d0dfd774f7b69af4191381110191dceef852 Mon Sep 17 00:00:00 2001 From: Nitwel Date: Mon, 6 Oct 2025 09:18:24 +0200 Subject: [PATCH 3/4] update docs on custom schema --- content/community/3.codebase/3.testing.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/content/community/3.codebase/3.testing.md b/content/community/3.codebase/3.testing.md index d6bc9ae9..7792c3e2 100644 --- a/content/community/3.codebase/3.testing.md +++ b/content/community/3.codebase/3.testing.md @@ -60,6 +60,10 @@ test('should format utc string', () => { }); ``` +::callout{icon="material-symbols:info-outline" color="success"} +Please follow [these guidelines](https://github.com/goldbergyoni/nodejs-testing-best-practices/blob/master/README.md) as they form a good and extensive baseline on how tests should be structured, organized and explains a lot of useful concepts. +:: + ## Writing blackbox tests The basic test structure goes as follows. @@ -87,9 +91,20 @@ test('ping', async () => { }); ``` +::callout{icon="material-symbols:info-outline" color="success"} +Please follow [these guidelines](https://github.com/goldbergyoni/nodejs-testing-best-practices/blob/master/README.md) as they form a good and extensive baseline on how tests should be structured, organized and explains a lot of useful concepts. +:: + ### Using a custom schema -Blackbox tests allow you to easily setup custom schemas for testing using the `useSnapshot` function. By default, the function uses the `snapshot.json` file that should be located on the same level as your test file. +Blackbox tests allow you to easily setup custom schemas for testing using the `useSnapshot` function. By default, the function uses the `snapshot.json` file that should be located on the same level as your test file. The function ensures that collection names are always unique by mapping the name in the schema to a name that is unique to the test. The function returns the mapped collection names as well as the used schema in json format. + +```ts +useSnapshot(api: DirectusClient & RestClient, file?: string = 'snapshot.json'): Promise<{ + collections: Collections; + snapshot: Snapshot; +}> +``` In oder to create such a schema and types for said schema quickly, use the sandbox cli: From 55c8c9db7021715b16045975ea43f8e55cec914c Mon Sep 17 00:00:00 2001 From: Nitwel Date: Mon, 6 Oct 2025 09:34:02 +0200 Subject: [PATCH 4/4] update sandbox cli --- content/community/3.codebase/4.sandbox-cli.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/content/community/3.codebase/4.sandbox-cli.md b/content/community/3.codebase/4.sandbox-cli.md index d98ad682..1a4cc63d 100644 --- a/content/community/3.codebase/4.sandbox-cli.md +++ b/content/community/3.codebase/4.sandbox-cli.md @@ -41,8 +41,15 @@ Options: The api is accessed through the following two functions: -- [sandbox](docs/functions/sandbox.md) -- [sandboxes](docs/functions/sandboxes.md) +```ts +function sandbox(database: Database, options?: DeepPartial): Promise + +function sandboxes(sandboxes: SandboxesOptions, options?: { + build?: boolean; + dev?: boolean; + watch?: boolean; +}): Promise +``` #### Example