[SYNPY-1672] Extract JSON Schema creation code from schematic (#1266)

* Extract convert and create json schema functions from schematic, renamed to create json schemas

Author: BryanFauble

.github/workflows/build.yml

@@ -84,15 +84,15 @@ jobs:
           path: |
             ${{ steps.get-dependencies.outputs.site_packages_loc }}
             ${{ steps.get-dependencies.outputs.site_bin_dir }}
-          key: ${{ runner.os }}-${{ matrix.python }}-build-${{ env.cache-name }}-${{ hashFiles('setup.py') }}-v27
+          key: ${{ runner.os }}-${{ matrix.python }}-build-${{ env.cache-name }}-${{ hashFiles('setup.py') }}-v28
 
       - name: Install py-dependencies
         if: steps.cache-dependencies.outputs.cache-hit != 'true'
         shell: bash
         run: |
           python -m pip install --upgrade pip
 
-          pip install -e ".[boto3,pandas,pysftp,tests]"
+          pip install -e ".[boto3,pandas,pysftp,tests,curator]"
 
           # ensure that numpy c extensions are installed on windows
           # https://stackoverflow.com/a/59346525

Pipfile

@@ -10,4 +10,4 @@ synapseclient = {file = ".", path = "."}
 python_version = "3.12.6"
 
 [dev-packages]
-synapseclient = {file = ".", editable = true, path = ".", extras = ["dev", "tests", "pandas", "pysftp", "boto3", "docs"]}
+synapseclient = {file = ".", editable = true, path = ".", extras = ["dev", "tests", "pandas", "pysftp", "boto3", "docs", "curator"]}

setup.cfg

@@ -107,6 +107,11 @@ pandas =
 
 curator =
     %(pandas)s
+    pandarallel>=1.6.4
+    inflection>=0.5.1
+    networkx>=2.2.8
+    dataclasses-json>=0.6.1
+    rdflib>=6.0.0
 
 pysftp =
     pysftp>=0.2.8,<0.3

synapseclient/extensions/curator/__init__.py

@@ -6,10 +6,13 @@
 
 from .file_based_metadata_task import create_file_based_metadata_task
 from .record_based_metadata_task import create_record_based_metadata_task
+from .schema_generation import generate_jsonld, generate_jsonschema
 from .schema_registry import query_schema_registry
 
 __all__ = [
     "create_file_based_metadata_task",
     "create_record_based_metadata_task",
     "query_schema_registry",
+    "generate_jsonld",
+    "generate_jsonschema",
 ]

synapseclient/extensions/curator/readme.md

@@ -12,24 +12,30 @@ The curator extension is designed around three core principles:
 
 ## Module Structure
 
-The curator extension consists of three focused modules:
+The curator extension consists of four focused modules:
 
 ```
 synapseclient/extensions/curator/
 ├── __init__.py                      # Clean public API surface
 ├── file_based_metadata_task.py      # File-annotation workflows
 ├── record_based_metadata_task.py    # Structured record workflows
-└── schema_registry.py               # Schema discovery and validation
+├── schema_registry.py               # Schema discovery and validation
+└── schema_generation.py             # Data model and JSON Schema generation
 ```
 
 ## Public API Design
 
-The module exposes three main functions that follow consistent design patterns:
+The module exposes five main functions that follow consistent design patterns:
 
+**Metadata Curation Workflows:**
 - **`create_file_based_metadata_task()`** - Configurable file-annotation curation workflows
 - **`create_record_based_metadata_task()`** - Configurable structured-record curation workflows
 - **`query_schema_registry()`** - Flexible schema discovery with custom filtering
 
+**Data Model and Schema Generation:**
+- **`generate_jsonld()`** - Convert CSV data models to JSON-LD format with validation
+- **`generate_jsonschema()`** - Generate JSON Schema validation files from data models
+
 ## Configuration and Flexibility
 
 ### Extensive Parameter Control
@@ -167,6 +173,56 @@ The module provides composable building blocks that can be combined to create so
 - Version filtering (latest-only or all versions)
 - Dynamic filter construction using keyword arguments
 
+### Data Model and Schema Generation
+
+**Purpose**: Create and validate data models, then generate JSON Schema validation files.
+
+The schema generation workflow consists of two key functions that work together:
+
+#### JSON-LD Data Model Generation (`generate_jsonld`)
+
+Converts CSV-based data model specifications into standardized JSON-LD format with comprehensive validation:
+
+**Input Requirements**:
+- CSV file with attributes, validation rules, dependencies, and valid values
+- Columns defining display names, descriptions, requirements, and relationships
+
+**Validation Performed**:
+- Required field presence checks
+- Dependency cycle detection (ensures valid DAG structure)
+- Blacklisted character detection in display names
+- Reserved name conflict checking
+- Graph structure validation
+
+**Configuration Levers**:
+- Label format selection (`class_label` vs `display_label`)
+- Custom output path or automatic naming
+- Comprehensive error and warning logging
+
+**Output**: JSON-LD file suitable for schema generation and other data model operations
+
+#### JSON Schema Generation (`generate_jsonschema`)
+
+Generates JSON Schema validation files from JSON-LD data models, translating validation rules into schema constraints:
+
+**Supported Validation Rules**:
+- Type validation (string, number, integer, boolean)
+- Enum constraints from valid values
+- Required field enforcement (including component-specific requirements)
+- Range constraints (`inRange` → min/max)
+- Pattern matching (`regex` → JSON Schema patterns)
+- Format validation (`date`, `url`)
+- Array handling (`list` rules)
+- Conditional dependencies (if/then schemas)
+
+**Configuration Levers**:
+- Component selection (specific data types or all components)
+- Label format for property names
+- Custom output directory structure
+- Component-based rule application using `#Component` syntax
+
+**Output**: JSON Schema files for each component, enabling validation of submitted manifests
+
 ## Development Philosophy
 
 ### Fail Fast with Clear Messages