[registry] Fix ArtifactHub sourceURL to persist actual package URL

[Copilot]

Problem Statement

When generating components from ArtifactHub, the model's sourceURL was set to the search query instead of the chosen package's actual URL. This caused subsequent updates to re-search ArtifactHub rather than directly using the resolved package URL.

Solution Overview

The fix ensures that:

1. When a package is chosen from ArtifactHub, its actual URL (ChartUrl) is stored in:
   • Model's metadata as source_uri
   • CSV model's SourceURL field
2. The updated SourceURL is persisted to the Google Spreadsheet
3. Subsequent updates use the CSV's SourceURL directly (no disk reading needed)
4. The model is written to filesystem only once with the correct source_uri

Changes Made

1. generators/artifacthub/package_manager.go

Modified GetPackage() method:

• Added logic to detect if SourceURL is an actual URL (starts with http://, https://, or oci://)
• If it's a URL, create an AhPackage directly with that ChartUrl instead of searching
• If it's not a URL (i.e., a package name), perform the search as before

2. registry/model.go

Enhanced UpdateModelDefinition:

• Added sourceURL to modelMetadataValues array
• Added handling for sourceURL case to store it as source_uri in model metadata
• This ensures source_uri is automatically included when the model is created

Simplified model generation flow:

• Removed redundant disk reading logic (26 lines)
• Now directly uses model.SourceURL from the CSV
• Updates model.SourceURL to actual package URL BEFORE calling writeModelDefToFileSystem
• writeModelDefToFileSystem creates model with source_uri already set via UpdateModelDefinition
• Removed duplicate model write - model is written only once with correct source_uri
• Sends updated CSV to spreadsheet channel for persistence

Modified GenerateComponentsFromPkg() function:

• Updated to set the model's source_uri from the package's actual URL

3. generators/artifacthub/package_test.go

Added TestGetPackageWithDirectURL():

• Tests that when SourceURL is an actual URL (HTTP, HTTPS, or OCI), it is used directly
• Verifies the package's GetSourceURL() returns the correct URL
• All tests pass successfully

4. generators/readme.md

Added comprehensive documentation:

• Documented the complete ArtifactHub generator flow
• Explained first generation vs subsequent updates
• Described URL detection logic
• Listed benefits of the approach
• Provided code structure overview

How It Works

First Generation (New Model)

1. CSV contains SourceURL as a search query (e.g., "consul")
2. System searches ArtifactHub and finds packages
3. Best package is chosen (e.g., "https://charts.bitnami.com/bitnami/consul-1.0.0.tgz")
4. Components are generated from that package
5. CSV model.SourceURL updated to actual package URL
6. writeModelDefToFileSystem creates model with source_uri from SourceURL
7. Model written once to disk with correct source_uri
8. Updated CSV sent to spreadsheet with actual URL

Subsequent Updates

1. CSV read from spreadsheet has SourceURL = actual URL (not search query)
2. Use model.SourceURL directly (no disk reading!)
3. ArtifactHub package manager detects it's a URL and uses it directly
4. Package fetched from URL, components generated
5. Model written with existing source_uri preserved

Benefits

1. Consistency: The actual package URL is stored and used everywhere (model metadata, CSV, spreadsheet)
2. Efficiency: No disk I/O for reading existing models, no duplicate writes
3. Simplicity: Removed 26 lines of redundant disk reading logic and duplicate write
4. Reliability: Using direct URLs ensures we get the same package every time
5. Backward Compatible: Still supports search queries for new models
6. Well Documented: Clear documentation for maintainers and contributors
7. Single Write: Model is written to filesystem only once with correct metadata

Testing

• Added unit test TestGetPackageWithDirectURL to verify direct URL handling
• Test verifies HTTP, HTTPS, and OCI protocol URLs work correctly
• All tests pass successfully
• Code compiles without errors

Original prompt

when generating a component from a model with registrant as artifactHub we use the name to search artifacthub and choose a pacakge from there and generate component . but we set the sourceURL of modle to be the search query rather the choosen package url .

update the logic to such that once choose the source url is the actual package url and also make sure it is properly consumed next time or during the updater flow

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[aabidsofi19]

@copilot looking good . but we are still missing upsert the sourceURL to actual package url into csv . so during first generation search by name and once package is resolved set the SourceURL of csv to actual url not search url and on subsequent updates use this url

[Copilot]

@copilot looking good . but we are still missing upsert the sourceURL to actual package url into csv . so during first generation search by name and once package is resolved set the SourceURL of csv ...

Updated in commit 2d3f1ad. The CSV model's SourceURL field is now set to the actual package URL after resolution. This ensures that when the SpreadsheetData is sent to the update channel, it contains the resolved URL which will be persisted back to the Google Spreadsheet.

The flow now works as:

1. First generation: Search by name → Choose package → Update both model metadata source_uri AND CSV SourceURL to actual URL
2. Spreadsheet update: The updated SourceURL is written back to the spreadsheet
3. Subsequent updates: Read existing source_uri from disk and use it directly (no search)

[aabidsofi19]

@copilot why are you still reading from disk if the sourceURL in spreadsheet is set to resolved packkage url

[Copilot]

@copilot why are you still reading from disk if the sourceURL in spreadsheet is set to resolved packkage url

You're right! The disk reading is now redundant. Since we're updating the CSV's SourceURL to the actual package URL and it gets persisted to the spreadsheet, on subsequent runs the model.SourceURL from the CSV will already contain the resolved URL. The disk reading was a workaround before we updated the CSV field. I'll remove this redundant logic.

[aabidsofi19]

good do it

[aabidsofi19]

@copilot after done also update documentation with correct flow

[Copilot]

@copilot after done also update documentation with correct flow

Documentation added in commit 2c258b1. Created comprehensive documentation in generators/readme.md that explains:

• Complete ArtifactHub generator flow for first generation and subsequent updates
• URL detection logic
• Benefits of the approach
• Code structure

The documentation clearly shows how the sourceURL field is used as a search query initially, then updated to the actual package URL, and reused directly in subsequent updates.

[aabidsofi19]

@copilot use the already existing step for writing to fs . we shouldnt write the model again

[Copilot]

@copilot use the already existing step for writing to fs . we shouldnt write the model again

Fixed in commit 56606e1. Removed the duplicate model write by:

1. Adding sourceURL to modelMetadataValues array
2. Updating UpdateModelDefinition to handle sourceURL and store it as source_uri in metadata
3. Updating model.SourceURL to actual package URL BEFORE calling writeModelDefToFileSystem
4. Removing the redundant second write - the model is now written only once with the correct source_uri included via the existing UpdateModelDefinition flow