roachprod-microbench: post GitHub issues for performance regressions #157045

ibreakthecloud · 2025-11-07T08:10:10Z

Previously, performance regressions detected during the weekly microbenchmark comparison were only reported via Slack notifications. This made it difficult to track and ensure timely follow-up on regressions, as they were often discussed informally without formal issue tracking.

This change extends the existing --post-issues flag to work with the compare command. When enabled, the system automatically creates GitHub issues for performance regressions that exceed 20% (the "red" regression threshold). Each issue includes:

Package name and list of regressed benchmarks
Regression percentages and formatted deltas
Link to the Google Sheet with detailed comparison data
Labels: O-microbench and C-performance for easy filtering

The implementation reuses the same GitHub posting infrastructure and environment variables (GITHUB_BRANCH, GITHUB_SHA, GITHUB_BINARY) as the existing benchmark failure reporting. Issues are created per package to avoid spam, with up to 10 regressions listed in each issue summary.

Example GitHub Issue screenshot:

Epic: None
Release note: None

cockroach-teamcity · 2025-11-07T08:10:24Z

This change is

Previously, performance regressions detected during the weekly microbenchmark comparison were only reported via Slack notifications. This made it difficult to track and ensure timely follow-up on regressions, as they were often discussed informally without formal issue tracking. This change extends the existing `--post-issues` flag to work with the compare command. When enabled, the system automatically creates GitHub issues for performance regressions that exceed 20% (the "red" regression threshold). Each issue includes: - Package name and list of regressed benchmarks - Regression percentages and formatted deltas - Link to the Google Sheet with detailed comparison data - Labels: O-microbench and C-performance for easy filtering The implementation reuses the same GitHub posting infrastructure and environment variables (GITHUB_BRANCH, GITHUB_SHA, GITHUB_BINARY) as the existing benchmark failure reporting. Issues are created per package to avoid spam, with up to 10 regressions listed in each issue summary. This change also renames `postBenchmarkIssue` to `postIssuesToGitHub` for consistency, as it now handles both execution failures and performance regressions. Epic: None Release note: None

rishabh7m

How was this tested? Can you paste a screenshot or the link of any generated issue?
Does this change needs to be backported?

rishabh7m · 2025-11-11T06:18:11Z

build/teamcity/cockroach/nightlies/microbenchmark_weekly.sh

    --sheet-desc="$sheet_description" \
    ${influx_token:+--influx-token="$influx_token"} \
    ${influx_host:+--influx-host="$influx_host"} \
+    ${TRIGGERED_BUILD:+--post-issues} \


Why this is required and cannot be the default setting? Is there any similar flag available for the slack notification?

This flag already exists, I am just using it. See here.

Yes, we'd only want this to happen on a triggered build. Since people are allowed to run custom comparisons which should not post (although it's not done often).

rishabh7m · 2025-11-11T06:23:24Z

pkg/cmd/roachprod-microbench/executor.go

 				artifactsDir := fmt.Sprintf("%s/%s", e.outputDir, benchmarkResponse.key)
 				formatter, req := createBenchmarkPostRequest(artifactsDir, response, timeout)
-				err = postBenchmarkIssue(context.Background(), e.log, formatter, req)
+				err = postIssuesToGitHub(context.Background(), e.log, formatter, req)


Why this renaming is required?

This function is a generic, and can be used to post any issues to github.
In the beginning, I thought we are posting issue for two different events, benchmark and regression, but now I realise both are benchmark issues, and naming was correct. I will now revert this change.

rishabh7m · 2025-11-11T06:26:41Z

pkg/cmd/roachprod-microbench/github.go

+}
+
+// postIssuesToGitHub posts a benchmark issue to github.
+func postIssuesToGitHub(


nit: postIssueToGitHub. It is taking post request for a single issue.

rishabh7m · 2025-11-11T06:37:05Z

pkg/cmd/roachprod-microbench/compare.go

+			formatter, req := createRegressionPostRequest(pkgName, regressions, sheetLink, c.sheetDesc)
+			err := postIssuesToGitHub(c.ctx, l, formatter, req)
+			if err != nil {
+				return errors.Wrapf(err, "failed to post regression issue for package %s", pkgName)


This will also not create the GitHub issue for the remaining regression packages if the earlier one failed.

Fair point. I will log the error and continue.

…ssions

ibreakthecloud · 2025-11-11T07:18:30Z

@rishabh7m

How was this tested? Can you paste a screenshot or the link of any generated issue?

No, it was not, let me test and update this PR.

Does this change needs to be backported?

No I don't think so.

…ssions

ibreakthecloud · 2025-11-11T10:33:14Z

How was this tested? Can you paste a screenshot or the link of any generated issue?

I've added the unit test that verifies the format of issue posted during regression.

srosenberg · 2025-11-13T03:55:04Z

How was this tested? Can you paste a screenshot or the link of any generated issue?

I've added the unit test that verifies the format of issue posted during regression.

The unit test is great, but it would still be nice to test it end-to-end. You can create a dummy issue to serve as an example.

When enabled, the system automatically creates GitHub issues for performance regressions that exceed 20% (the "red" regression threshold). Each issue includes:

Package name and list of regressed benchmarks

In case of a misconfiguration (or other bug), what if every package results in a regression? We should limit the total (possible) number of created GH issues.

ibreakthecloud · 2025-11-13T07:14:30Z

@srosenberg

How was this tested? Can you paste a screenshot or the link of any generated issue?

I've added the unit test that verifies the format of issue posted during regression.

The unit test is great, but it would still be nice to test it end-to-end. You can create a dummy issue to serve as an example.

Fair point, I will create a dummy issue and update the description.

In case of a misconfiguration (or other bug), what if every package results in a regression? We should limit the total (possible) number of created GH issues.

~~I will limit it to 5 issues.~~

This changes creates one GitHub issue per package with all severe regressions (skips creating issue, incase it there's none in the pkg). Currently there are 23 packages and from the historical slack messages in #perf-ops, I think on an average we get ~8 pkg with atleast one regression, it would be safe to put 10 issues as limit on stop creating. LMK your thoughts.

Also, would like to understand how exactly do you want to limit. Stop creating issues after 10 issues or donot even create one if there's more than 10.

herkolategan · 2025-11-25T12:10:57Z

pkg/cmd/roachprod-microbench/github_test.go

+		PackageNameShort: req.PackageName,
+	}
+	title := formatter.Title(data)
+	if !strings.Contains(title, "pkg/sql") {


Nit / Personal preference to some degree. but we use the require package to simplify these assertions a bit.

For instance could replace it with one of these:

require.Contains(t, title, "pkg/sql")

It already generates the message saying it expect X to contain Y. But you can add another arg in require.Contains(..., "should have contained ...") if you want to customize the message.

herkolategan

I would like to understand which team labels will be applied to each package regression issue. Since this adds the release-blocker label, it will become the duty of those team(s) to address or close this issue. Although an issue could end up containing benchmarks owned by multiple teams.

herkolategan · 2025-11-25T12:29:52Z

pkg/cmd/roachprod-microbench/github.go

+	title := fmt.Sprintf("%s: performance regression", pkgName)
+	f := githubpost.MicrobenchmarkFailure(
+		pkgName,
+		title,


The title here replaces the benchmarkName. This becomes the testName when we try to resolve the owning team (See: https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/bazci/githubpost/githubpost.go#L689)

The question here is in which team's bucket does this all end up? Should we be more intelligent here and determine all the regressed benchmark's teams?

ibreakthecloud marked this pull request as ready for review November 11, 2025 05:04

ibreakthecloud requested review from a team as code owners November 11, 2025 05:04

ibreakthecloud requested review from golgeek and srosenberg and removed request for a team November 11, 2025 05:04

ibreakthecloud force-pushed the harsh-codesys-106 branch from dbaabf1 to f71f5f2 Compare November 11, 2025 06:36

rishabh7m requested changes Nov 11, 2025

View reviewed changes

fixup! roachprod-microbench: post GitHub issues for performance regre…

be14bbe

…ssions

fixup! roachprod-microbench: post GitHub issues for performance regre…

5852034

…ssions

ibreakthecloud requested a review from rishabh7m November 11, 2025 10:33

rishabh7m approved these changes Nov 12, 2025

View reviewed changes

herkolategan self-requested a review November 12, 2025 15:55

herkolategan reviewed Nov 25, 2025

View reviewed changes

herkolategan requested changes Nov 25, 2025

View reviewed changes

herkolategan reviewed Nov 25, 2025

View reviewed changes

roachprod-microbench: post GitHub issues for performance regressions #157045

Are you sure you want to change the base?

roachprod-microbench: post GitHub issues for performance regressions #157045

Conversation

ibreakthecloud commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cockroach-teamcity commented Nov 7, 2025

Uh oh!

rishabh7m left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ibreakthecloud commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ibreakthecloud commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srosenberg commented Nov 13, 2025

Uh oh!

ibreakthecloud commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

herkolategan left a comment

Choose a reason for hiding this comment

Uh oh!

herkolategan Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ibreakthecloud commented Nov 7, 2025 •

edited

Loading

ibreakthecloud commented Nov 11, 2025 •

edited

Loading

ibreakthecloud commented Nov 11, 2025 •

edited

Loading

ibreakthecloud commented Nov 13, 2025 •

edited

Loading

herkolategan Nov 25, 2025 •

edited

Loading