feat: index json schemas on tasks and schemaTask (#2351)

* Add payload schema handling for task indexing

This change introduces support for handling payload schemas during task indexing. By incorporating the `payloadSchema` attribute into various components, we ensure that each task's payload structure is clearly defined and can be validated before processing.

- Updated the TaskManifest and task metadata structures to include an optional `payloadSchema` attribute. This addition allows for more robust validation and handling of task payloads.
- Enhanced several core modules to export and utilize the new `getSchemaToJsonSchema` function, providing easier conversion of schema types to JSON schemas.
- Modified the database schema to store the `payloadSchema` attribute, ensuring that the payload schema information is persisted.
- The change helps in maintaining consistency in data handling and improves the integrity of task data across the application.

* Refactor: Remove getSchemaToJsonSchema in favor of schemaToJsonSchema

The `getSchemaToJsonSchema` function was removed and replaced with `schemaToJsonSchema` across the codebase. This update introduces a new `@trigger.dev/schema-to-json` package to handle conversions of schema validation libraries to JSON Schema format, centralizing the functionality and improving maintainability.

- Removed `getSchemaToJsonSchema` exports and references.
- Added new schema conversion utility `@trigger.dev/schema-to-json`.
- Updated `trigger-sdk` package to utilize `schemaToJsonSchema` for payloads.
- Extensive testing coverage included to ensure conversion accuracy across various schema libraries including Zod, Yup, ArkType, Effect, and TypeBox.
- The update ensures consistent and reliable schema conversions, facilitating future enhancements and supporting additional schema libraries.

* Add support for Zod 4 in schema-to-json

This change enhances the schema-to-json package by adding support for Zod version 4, which introduces the native `toJsonSchema` method. This method facilitates a direct conversion of Zod schemas to JSON Schema format, improving performance and reducing reliance on the `zod-to-json-schema` library.

- Updated README to reflect Zod 4 support with native method and retained support for Zod 3 via existing library.
- Modified package.json to allow installation of both Zod 3 and 4 versions.
- Implemented handling for Zod 4 schemas in `src/index.ts` using their native method.
- Added a test case to verify the proper conversion of Zod 4 schemas to JSON Schema.
- Included a script for updating the package version based on the root package.json.
- Introduced a specific TypeScript config for source files.

* Revise schema-to-json for bundle safety and tests

The package @trigger.dev/schema-to-json has been revised to ensure bundle safety by removing direct dependencies on schema libraries such as Zod, Yup, and Effect. This change minimizes bundle size and enhances tree-shaking by allowing external conversion libraries to be utilized only at runtime if necessary. As a result, the README was updated to reflect this usage pattern.

- Introduced `initializeSchemaConverters` function to load necessary conversion libraries at runtime, keeping the base package slim.
- Adjusted test suite to initialize converters before tests, ensuring accurate testing of schema conversion capabilities.
- Updated `schemaToJsonSchema` function to dynamically check for availability of conversion libraries, improving flexibility without increasing the package size.
- Added configuration files for Vitest to support the new testing framework, reflecting the transition from previous test setups.

These enhancements ensure that only the schema libraries actively used in an application are bundled, optimizing performance and resource usage.

* Refine JSON Schema typing across packages

The changes introduce stricter typing for JSON Schema-related definitions, specifically replacing vague types with more precise ones, such as using `z.record(z.unknown())` instead of `z.any()` and `Record<string, unknown>` in place of `any`. This is part of an effort to better align with common practices and improve type safety in the packages.

- Updated the `payloadSchema` in several files to use `z.record(z.unknown())`, enhancing the type strictness and consistency with JSON Schema Draft 7 recommendations.
- Added `@types/json-schema` as a dependency, utilizing its definitions for improved type clarity and adherence to best practices in TypeScript.
- Modified various comments to explicitly mention JSON Schema Draft 7, ensuring developers are aware of the JSON Schema version being implemented.
- These adjustments are informed by research into how popular libraries and tools handle JSON Schema typing, aiming to integrate best practices for improved maintainability and interoperability.

* Add JSON Schema examples using various libraries

The change introduces extensive examples of using JSON Schemas in the 'references/hello-world' project within the 'trigger.dev' repository. These examples utilize libraries like Zod, Yup, and TypeBox for JSON Schema conversion and validation. The new examples demonstrate different use cases, including automatic conversion with schemaTask, manual schema provision, and schema conversion at build time. We also updated the dependencies in 'package.json' to include the necessary libraries for schema conversion and validation.

- Included examples of processing tasks with JSON Schema using libraries such as Zod, Yup, TypeBox, and ArkType.
- Showcased schema conversion techniques and type-safe JSON Schema creation.
- Updated 'package.json' to ensure all necessary dependencies for schema operations are available.
- Created illustrative scripts that cover task management from user processing to complex schema implementations.

* Refactor SDK to encapsulate schema-to-json package

The previous implementation required users to directly import and initialize functions from the `@trigger.dev/schema-to-json` package, which was not the intended user experience. This change refactors the SDK so that all necessary functions and types from `@trigger.dev/schema-to-json` are encapsulated within the `@trigger.dev/*` packages.

- The examples in `usage.ts` have been updated to clearly mark `@trigger.dev/schema-to-json` as an internal-only package.
- Re-export JSON Schema types and conversions in the SDK to improve developer experience (DX).
- Removed unnecessary direct dependencies on `@trigger.dev/schema-to-json` from user-facing code, ensuring initialization and conversion logic is handled internally.
- Replaced instances where users were required to manually perform schema conversions with automatic handling within the SDK for simplification and better maintainability.

* Add JSONSchema type for payloadSchema in tasks

The change was necessary to improve type safety by using a proper JSONSchema type definition instead of a generic Record<string, unknown>. This enhances the developer experience and ensures that task payloads conform to the JSON Schema Draft 7 specification. The JSONSchema type is now re-exported from the SDK for user convenience, hiding internal complexity and maintaining a seamless developer experience.

- Added JSONSchema type based on Draft 7 specification
- Updated task metadata and options to use JSONSchema type
- Hid internal schema conversion logic from users by re-exporting types from SDK
- Improved bundle safety and dependency management

* Add JSON schema testing and revert package dependencies

This commit introduces a comprehensive set of JSON schema testing within the monorepo, specifically adding a new test project in `references/json-schema-test`. This includes a variety of schema definitions and tasks utilizing multiple validation libraries to ensure robust type-checking and runtime validation.

Additionally, the dependency versions for `@effect/schema` have been adjusted from `^0.76.5` to `^0.75.5` to maintain compatibility across the project components. This ensures consistent behavior and compatibility with existing code bases without introducing breaking changes or unexpected behavior due to version discrepancies.

Key updates include:
- Added new test project with extensive schema validation tests.
- Ensured type safety across various task implementations.
- Reverted dependency versions to ensure compatibility.
- Created multiple schema tasks using libraries like Zod, Yup, and others for thorough testing.

* Refactor JSON Schema test files for clarity

Whitespace and formatting changes were applied across the `json-schema-test` reference project to enhance code readability and cohesion. This included removing unnecessary trailing spaces and ensuring consistent indentation patterns, which improves maintainability and readability by following the project's code style guidelines.

- Renamed JSONSchema type annotations to adhere to TypeScript conventions, ensuring that all schema definitions properly satisfy the JSONSchema interface.
- Restructured some object declarations for improved clarity, especially within complex schema definitions.
- These adjustments are crucial for better future maintainability, reducing potential developer errors when interacting with these test schemas.

* Fixed some stuff

* WIP

* we now convert schema to jsonSchema on the CLI side via the indexing

* Remove the json-schema-test reference project

* Improve schema-to-json peer deps and fix effect schema

* Explain the casting and match the version numbers

* Fixed a bunch more schema stuff

* Don't clean files that might be written to

* Don't use a custom version of vitest in the new package

* fix attw in schema-to-json

Author: ericallam

apps/webapp/app/v3/services/createBackgroundWorker.server.ts

@@ -77,7 +77,8 @@ export class CreateBackgroundWorkerService extends BaseService {
           version: nextVersion,
           runtimeEnvironmentId: environment.id,
           projectId: project.id,
-          metadata: body.metadata,
+          // body.metadata has an index signature that Prisma doesn't like (from the JSONSchema type) so we are safe to just cast it
+          metadata: body.metadata as Prisma.InputJsonValue,
           contentHash: body.metadata.contentHash,
           cliVersion: body.metadata.cliPackageVersion,
           sdkVersion: body.metadata.packageVersion,
@@ -280,6 +281,7 @@ async function createWorkerTask(
         fileId: tasksToBackgroundFiles?.get(task.id) ?? null,
         maxDurationInSeconds: task.maxDuration ? clampMaxDuration(task.maxDuration) : null,
         queueId: queue.id,
+        payloadSchema: task.payloadSchema as any,
       },
     });
   } catch (error) {

apps/webapp/app/v3/services/createDeploymentBackgroundWorkerV3.server.ts

@@ -1,5 +1,5 @@
 import { CreateBackgroundWorkerRequestBody } from "@trigger.dev/core/v3";
-import type { BackgroundWorker } from "@trigger.dev/database";
+import type { BackgroundWorker, Prisma } from "@trigger.dev/database";
 import { AuthenticatedEnvironment } from "~/services/apiAuth.server";
 import { logger } from "~/services/logger.server";
 import { socketIo } from "../handleSocketIo.server";
@@ -48,7 +48,8 @@ export class CreateDeploymentBackgroundWorkerServiceV3 extends BaseService {
           version: deployment.version,
           runtimeEnvironmentId: environment.id,
           projectId: environment.projectId,
-          metadata: body.metadata,
+          // body.metadata has an index signature that Prisma doesn't like (from the JSONSchema type) so we are safe to just cast it
+          metadata: body.metadata as Prisma.InputJsonValue,
           contentHash: body.metadata.contentHash,
           cliVersion: body.metadata.cliPackageVersion,
           sdkVersion: body.metadata.packageVersion,

apps/webapp/app/v3/services/createDeploymentBackgroundWorkerV4.server.ts

@@ -1,6 +1,6 @@
 import { CreateBackgroundWorkerRequestBody, logger, tryCatch } from "@trigger.dev/core/v3";
 import { BackgroundWorkerId } from "@trigger.dev/core/v3/isomorphic";
-import type { BackgroundWorker, WorkerDeployment } from "@trigger.dev/database";
+import type { BackgroundWorker, Prisma, WorkerDeployment } from "@trigger.dev/database";
 import { AuthenticatedEnvironment } from "~/services/apiAuth.server";
 import { BaseService, ServiceValidationError } from "./baseService.server";
 import {
@@ -65,7 +65,8 @@ export class CreateDeploymentBackgroundWorkerServiceV4 extends BaseService {
           version: deployment.version,
           runtimeEnvironmentId: environment.id,
           projectId: environment.projectId,
-          metadata: body.metadata,
+          // body.metadata has an index signature that Prisma doesn't like (from the JSONSchema type) so we are safe to just cast it
+          metadata: body.metadata as Prisma.InputJsonValue,
           contentHash: body.metadata.contentHash,
           cliVersion: body.metadata.cliPackageVersion,
           sdkVersion: body.metadata.packageVersion,

internal-packages/database/prisma/migrations/20250730084611_add_payload_schema_to_background_worker_task/migration.sql

@@ -0,0 +1,2 @@
+-- AlterTable
+ALTER TABLE "BackgroundWorkerTask" ADD COLUMN     "payloadSchema" JSONB;

internal-packages/database/prisma/schema.prisma

@@ -510,6 +510,8 @@ model BackgroundWorkerTask {
 
   triggerSource TaskTriggerSource @default(STANDARD)
 
+  payloadSchema Json?
+
   @@unique([workerId, slug])
   // Quick lookup of task identifiers
   @@index([projectId, slug])

packages/cli-v3/package.json

@@ -91,6 +91,7 @@
     "@opentelemetry/semantic-conventions": "1.36.0",
     "@trigger.dev/build": "workspace:4.0.0-v4-beta.26",
     "@trigger.dev/core": "workspace:4.0.0-v4-beta.26",
+    "@trigger.dev/schema-to-json": "workspace:4.0.0-v4-beta.26",
     "ansi-escapes": "^7.0.0",
     "braces": "^3.0.3",
     "c12": "^1.11.1",

packages/cli-v3/src/entryPoints/dev-index-worker.ts

@@ -18,6 +18,7 @@ import { registerResources } from "../indexing/registerResources.js";
 import { env } from "std-env";
 import { normalizeImportPath } from "../utilities/normalizeImportPath.js";
 import { detectRuntimeVersion } from "@trigger.dev/core/v3/build";
+import { schemaToJsonSchema, initializeSchemaConverters } from "@trigger.dev/schema-to-json";
 
 sourceMapSupport.install({
   handleUncaughtExceptions: false,
@@ -100,7 +101,7 @@ async function bootstrap() {
 
 const { buildManifest, importErrors, config, timings } = await bootstrap();
 
-let tasks = resourceCatalog.listTaskManifests();
+let tasks = await convertSchemasToJsonSchemas(resourceCatalog.listTaskManifests());
 
 // If the config has retry defaults, we need to apply them to all tasks that don't have any retry settings
 if (config.retries?.default) {
@@ -190,3 +191,24 @@ await new Promise<void>((resolve) => {
     resolve();
   }, 10);
 });
+
+async function convertSchemasToJsonSchemas(tasks: TaskManifest[]): Promise<TaskManifest[]> {
+  await initializeSchemaConverters();
+
+  const convertedTasks = tasks.map((task) => {
+    const schema = resourceCatalog.getTaskSchema(task.id);
+
+    if (schema) {
+      try {
+        const result = schemaToJsonSchema(schema);
+        return { ...task, payloadSchema: result?.jsonSchema };
+      } catch {
+        return task;
+      }
+    }
+
+    return task;
+  });
+
+  return convertedTasks;
+}

packages/cli-v3/src/entryPoints/managed-index-worker.ts

@@ -18,6 +18,7 @@ import { registerResources } from "../indexing/registerResources.js";
 import { env } from "std-env";
 import { normalizeImportPath } from "../utilities/normalizeImportPath.js";
 import { detectRuntimeVersion } from "@trigger.dev/core/v3/build";
+import { schemaToJsonSchema, initializeSchemaConverters } from "@trigger.dev/schema-to-json";
 
 sourceMapSupport.install({
   handleUncaughtExceptions: false,
@@ -100,7 +101,7 @@ async function bootstrap() {
 
 const { buildManifest, importErrors, config, timings } = await bootstrap();
 
-let tasks = resourceCatalog.listTaskManifests();
+let tasks = await convertSchemasToJsonSchemas(resourceCatalog.listTaskManifests());
 
 // If the config has retry defaults, we need to apply them to all tasks that don't have any retry settings
 if (config.retries?.default) {
@@ -196,3 +197,24 @@ await new Promise<void>((resolve) => {
     resolve();
   }, 10);
 });
+
+async function convertSchemasToJsonSchemas(tasks: TaskManifest[]): Promise<TaskManifest[]> {
+  await initializeSchemaConverters();
+
+  const convertedTasks = tasks.map((task) => {
+    const schema = resourceCatalog.getTaskSchema(task.id);
+
+    if (schema) {
+      try {
+        const result = schemaToJsonSchema(schema);
+        return { ...task, payloadSchema: result?.jsonSchema };
+      } catch {
+        return task;
+      }
+    }
+
+    return task;
+  });
+
+  return convertedTasks;
+}

packages/core/src/v3/resource-catalog/catalog.ts

@@ -1,5 +1,5 @@
 import { QueueManifest, TaskManifest, WorkerManifest } from "../schemas/index.js";
-import { TaskMetadataWithFunctions } from "../types/index.js";
+import { TaskMetadataWithFunctions, TaskSchema } from "../types/index.js";
 
 export interface ResourceCatalog {
   setCurrentFileContext(filePath: string, entryPoint: string): void;
@@ -13,4 +13,5 @@ export interface ResourceCatalog {
   registerWorkerManifest(workerManifest: WorkerManifest): void;
   registerQueueMetadata(queue: QueueManifest): void;
   listQueueManifests(): Array<QueueManifest>;
+  getTaskSchema(id: string): TaskSchema | undefined;
 }

packages/core/src/v3/resource-catalog/index.ts

@@ -1,7 +1,7 @@
 const API_NAME = "resource-catalog";
 
 import { QueueManifest, TaskManifest, WorkerManifest } from "../schemas/index.js";
-import { TaskMetadataWithFunctions } from "../types/index.js";
+import { TaskMetadataWithFunctions, TaskSchema } from "../types/index.js";
 import { getGlobal, registerGlobal, unregisterGlobal } from "../utils/globals.js";
 import { type ResourceCatalog } from "./catalog.js";
 import { NoopResourceCatalog } from "./noopResourceCatalog.js";
@@ -65,6 +65,10 @@ export class ResourceCatalogAPI {
     return this.#getCatalog().getTask(id);
   }
 
+  public getTaskSchema(id: string): TaskSchema | undefined {
+    return this.#getCatalog().getTaskSchema(id);
+  }
+
   public taskExists(id: string): boolean {
     return this.#getCatalog().taskExists(id);
   }