feat: validate lexicons #5
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n ingest route - Use LexiconSchemaRecord type and type guard for incoming records - Resolve and check NSID DID authority matches record DID - Return error if record is invalid or authority does not match - Refactor to use new @atproto/lexicon and @atproto/lexicon-resolver utilities
…r NSID resolution Remove extension from @atproto/lexicon-resolver type and explicitly define LexiconSchemaRecord with required 'id' property. Update type guard and documentation to clarify purpose for DID authority resolution.
…nd zod in ingest route Parse incoming lexicon records with parseLexiconDoc, validate with zod, and improve error handling for invalid records. Store parsed LexiconDoc in the database instead of raw input.
- Replace single lexicons table with valid_lexicons and invalid_lexicons tables - Add repo_did to primary key [nsid, cid, repo_did] to track lexicon migrations - Store validation errors in invalid_lexicons for developer debugging - Use different column names (data vs raw_data) for semantic clarity - Add indexes on nsid and repo_did for both tables - Include repo_rev field to track repository state at ingestion time
- Store valid lexicons in valid_lexicons table after successful parsing - Store invalid lexicons in invalid_lexicons table with validation errors - DNS validation acts as gate before any storage (prevents DDOS) - Include repo_did and repo_rev in all stored records - Improve logging to distinguish valid vs invalid ingestion events - Remove onConflictDoUpdate logic (primary key now includes repo_did)
- Combine initial lexicons table and updated schema into one migration - Remove intermediate migration file (0001) - Clean schema shows final valid_lexicons and invalid_lexicons tables
…om Nexus and Postgres settings Provides a Docker Compose override for development, disabling lexhub in Docker, reducing resource usage for Nexus and Postgres, and enabling debug logging and SQL query logging.
…thing in ingest route
- Check for error shape (has 'issues' array) instead of instanceof check - parseLexiconDoc throws errors that may not pass instanceof z.ZodError check - Add onConflictDoNothing to invalid_lexicons insert - Ensures invalid lexicons are now properly stored for debugging
… handling in ingest route
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive lexicon validation to the ATProto lexicon hub by implementing DNS-based DID authority verification and Zod schema validation. The changes transform the simple lexicon storage system into a robust validation pipeline that separates valid from invalid lexicons while maintaining detailed debugging information.
Key Changes
- Two-tier validation: DNS authority check followed by schema validation using
@atproto/lexiconparser - Dual table architecture: Separate
valid_lexiconsandinvalid_lexiconstables for production data and debugging - Enhanced schema: Added
repo_didandrepo_revtracking with composite primary keys to handle lexicon migrations
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/util/lexicon.ts | New type guard and interface for lexicon schema records |
| src/db/schema.ts | Complete schema refactor with separate valid/invalid lexicon tables and migration tracking |
| src/app/api/ingest/route.ts | Added DNS validation and schema parsing with error handling for invalid lexicons |
| package.json | Added @atproto dependencies for lexicon parsing and DNS resolution |
| package-lock.json | Dependency lockfile updates for new @atproto packages |
| drizzle/0000_init.sql | Squashed migration creating dual table structure with appropriate indexes |
| drizzle/meta/_journal.json | Updated migration timestamp |
| compose.override.yaml | New development configuration for local development workflow |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
elijaharita
reviewed
Dec 2, 2025
| @@ -1,26 +1,31 @@ | |||
| CREATE TABLE "lexicons" ( | |||
| "id" varchar(317) NOT NULL, | |||
| -- Create valid_lexicons table | |||
Contributor
There was a problem hiding this comment.
probably don't need this comment 😄
Contributor
There was a problem hiding this comment.
same with a couple others
elijaharita
reviewed
Dec 2, 2025
Contributor
There was a problem hiding this comment.
should this be in .gitignore?
elijaharita
approved these changes
Dec 2, 2025
Contributor
elijaharita
left a comment
elijaharita
left a comment
There was a problem hiding this comment.
2 potential things otherwise looks good!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Validate lexions DID authority through a DNS check, and validate that the lexicon matches the zod schema from the official atproto sdk.
Also added a dev compose file, and squashed the db schema since there is no production data yet.