feat: last copy pipe executions monitoring endpoint #3520
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Creates a new tinybird pipe for copy pipe executions
Monitors Tinybird copy job executions and exposes their status as Prometheus gauge metrics. Consumed by datadog scraper (
dd-tinybird-copy-pipe-executions-scraper)Tracks the latest execution status for each copy pipe (starting today) and generates one-hot encoded metrics where each pipe gets a value of
1for its current status ('ok','error','cancelled','queued','working') and0for all other statuses.Metric:
copy_pipes_latest_execution_status(gauge)Labels:
pipe_name,statusNote: The
'queued'status is virtual—it's derived by detecting the error message"You have reached the maximum number of copy jobs"rather than being a native Tinybird status. This is because tinybird retries these again when possible but returns an error status. Similarly,'ok'is mapped from the native'done'status for datadog color-scheme convention.