-
Notifications
You must be signed in to change notification settings - Fork 315
Increase load benchmark repetitions to 5 #9898
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
sarahchen6
wants to merge
8
commits into
master
Choose a base branch
from
sarahchen6/increase-load-benchmark-reps
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
f7ce7fb to
f68d358
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 53 metrics, 12 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.56.0-SNAPSHOT~4243ccc14e, baseline=1.56.0-SNAPSHOT~1397f25f48
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1054176
Total [baseline] (8.62 s) : 0, 8619808
Agent [candidate] (1.061 s) : 0, 1060820
Total [candidate] (8.647 s) : 0, 8646692
section iast
Agent [baseline] (1.18 s) : 0, 1180089
Total [baseline] (9.234 s) : 0, 9234094
Agent [candidate] (1.193 s) : 0, 1192611
Total [candidate] (9.323 s) : 0, 9322587
gantt
title insecure-bank - break down per module: candidate=1.56.0-SNAPSHOT~4243ccc14e, baseline=1.56.0-SNAPSHOT~1397f25f48
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.481 ms) : 0, 1481
crashtracking [candidate] (1.479 ms) : 0, 1479
BytebuddyAgent [baseline] (709.651 ms) : 0, 709651
BytebuddyAgent [candidate] (713.975 ms) : 0, 713975
GlobalTracer [baseline] (247.235 ms) : 0, 247235
GlobalTracer [candidate] (249.069 ms) : 0, 249069
AppSec [baseline] (32.492 ms) : 0, 32492
AppSec [candidate] (32.867 ms) : 0, 32867
Debugger [baseline] (6.431 ms) : 0, 6431
Debugger [candidate] (6.55 ms) : 0, 6550
Remote Config [baseline] (711.393 µs) : 0, 711
Remote Config [candidate] (720.253 µs) : 0, 720
Telemetry [baseline] (13.932 ms) : 0, 13932
Telemetry [candidate] (11.557 ms) : 0, 11557
Flare Poller [baseline] (7.336 ms) : 0, 7336
Flare Poller [candidate] (9.591 ms) : 0, 9591
section iast
crashtracking [baseline] (1.457 ms) : 0, 1457
crashtracking [candidate] (1.476 ms) : 0, 1476
BytebuddyAgent [baseline] (828.552 ms) : 0, 828552
BytebuddyAgent [candidate] (839.62 ms) : 0, 839620
GlobalTracer [baseline] (234.766 ms) : 0, 234766
GlobalTracer [candidate] (235.345 ms) : 0, 235345
AppSec [baseline] (28.844 ms) : 0, 28844
AppSec [candidate] (28.391 ms) : 0, 28391
Debugger [baseline] (6.049 ms) : 0, 6049
Debugger [candidate] (6.04 ms) : 0, 6040
Remote Config [baseline] (623.049 µs) : 0, 623
Remote Config [candidate] (603.064 µs) : 0, 603
Telemetry [baseline] (8.478 ms) : 0, 8478
Telemetry [candidate] (8.48 ms) : 0, 8480
Flare Poller [baseline] (4.09 ms) : 0, 4090
Flare Poller [candidate] (4.12 ms) : 0, 4120
IAST [baseline] (32.346 ms) : 0, 32346
IAST [candidate] (33.523 ms) : 0, 33523
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.56.0-SNAPSHOT~4243ccc14e, baseline=1.56.0-SNAPSHOT~1397f25f48
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1048653
Total [baseline] (10.919 s) : 0, 10918532
Agent [candidate] (1.06 s) : 0, 1060259
Total [candidate] (10.776 s) : 0, 10775511
section appsec
Agent [baseline] (1.222 s) : 0, 1221992
Total [baseline] (10.912 s) : 0, 10912300
Agent [candidate] (1.222 s) : 0, 1221632
Total [candidate] (10.913 s) : 0, 10912967
section iast
Agent [baseline] (1.181 s) : 0, 1181077
Total [baseline] (11.157 s) : 0, 11156533
Agent [candidate] (1.19 s) : 0, 1189681
Total [candidate] (11.171 s) : 0, 11170749
section profiling
Agent [baseline] (1.191 s) : 0, 1191232
Total [baseline] (10.921 s) : 0, 10921346
Agent [candidate] (1.195 s) : 0, 1195011
Total [candidate] (10.868 s) : 0, 10867785
gantt
title petclinic - break down per module: candidate=1.56.0-SNAPSHOT~4243ccc14e, baseline=1.56.0-SNAPSHOT~1397f25f48
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.461 ms) : 0, 1461
crashtracking [candidate] (1.48 ms) : 0, 1480
BytebuddyAgent [baseline] (705.92 ms) : 0, 705920
BytebuddyAgent [candidate] (713.746 ms) : 0, 713746
GlobalTracer [baseline] (246.057 ms) : 0, 246057
GlobalTracer [candidate] (248.895 ms) : 0, 248895
AppSec [baseline] (32.294 ms) : 0, 32294
AppSec [candidate] (32.716 ms) : 0, 32716
Debugger [baseline] (6.393 ms) : 0, 6393
Debugger [candidate] (6.557 ms) : 0, 6557
Remote Config [baseline] (713.14 µs) : 0, 713
Remote Config [candidate] (720.331 µs) : 0, 720
Telemetry [baseline] (15.301 ms) : 0, 15301
Telemetry [candidate] (14.54 ms) : 0, 14540
Flare Poller [baseline] (5.701 ms) : 0, 5701
Flare Poller [candidate] (6.533 ms) : 0, 6533
section appsec
crashtracking [baseline] (1.448 ms) : 0, 1448
crashtracking [candidate] (1.462 ms) : 0, 1462
BytebuddyAgent [baseline] (729.131 ms) : 0, 729131
BytebuddyAgent [candidate] (729.145 ms) : 0, 729145
GlobalTracer [baseline] (237.669 ms) : 0, 237669
GlobalTracer [candidate] (237.497 ms) : 0, 237497
AppSec [baseline] (174.848 ms) : 0, 174848
AppSec [candidate] (174.66 ms) : 0, 174660
Debugger [baseline] (5.989 ms) : 0, 5989
Debugger [candidate] (5.985 ms) : 0, 5985
Remote Config [baseline] (650.022 µs) : 0, 650
Remote Config [candidate] (638.291 µs) : 0, 638
Telemetry [baseline] (8.46 ms) : 0, 8460
Telemetry [candidate] (8.501 ms) : 0, 8501
Flare Poller [baseline] (3.997 ms) : 0, 3997
Flare Poller [candidate] (3.987 ms) : 0, 3987
IAST [baseline] (24.957 ms) : 0, 24957
IAST [candidate] (24.859 ms) : 0, 24859
section iast
crashtracking [baseline] (1.463 ms) : 0, 1463
crashtracking [candidate] (1.465 ms) : 0, 1465
BytebuddyAgent [baseline] (829.461 ms) : 0, 829461
BytebuddyAgent [candidate] (836.138 ms) : 0, 836138
GlobalTracer [baseline] (235.043 ms) : 0, 235043
GlobalTracer [candidate] (236.217 ms) : 0, 236217
AppSec [baseline] (27.948 ms) : 0, 27948
AppSec [candidate] (28.068 ms) : 0, 28068
Debugger [baseline] (6.006 ms) : 0, 6006
Debugger [candidate] (6.048 ms) : 0, 6048
Remote Config [baseline] (590.523 µs) : 0, 591
Remote Config [candidate] (597.263 µs) : 0, 597
Telemetry [baseline] (8.333 ms) : 0, 8333
Telemetry [candidate] (8.55 ms) : 0, 8550
Flare Poller [baseline] (4.121 ms) : 0, 4121
Flare Poller [candidate] (4.204 ms) : 0, 4204
IAST [baseline] (33.302 ms) : 0, 33302
IAST [candidate] (33.488 ms) : 0, 33488
section profiling
ProfilingAgent [baseline] (110.579 ms) : 0, 110579
ProfilingAgent [candidate] (111.242 ms) : 0, 111242
crashtracking [baseline] (1.434 ms) : 0, 1434
crashtracking [candidate] (1.436 ms) : 0, 1436
BytebuddyAgent [baseline] (727.819 ms) : 0, 727819
BytebuddyAgent [candidate] (729.785 ms) : 0, 729785
GlobalTracer [baseline] (222.021 ms) : 0, 222021
GlobalTracer [candidate] (222.823 ms) : 0, 222823
AppSec [baseline] (32.032 ms) : 0, 32032
AppSec [candidate] (32.094 ms) : 0, 32094
Debugger [baseline] (9.126 ms) : 0, 9126
Debugger [candidate] (7.409 ms) : 0, 7409
Remote Config [baseline] (711.891 µs) : 0, 712
Remote Config [candidate] (668.068 µs) : 0, 668
Telemetry [baseline] (13.648 ms) : 0, 13648
Telemetry [candidate] (15.472 ms) : 0, 15472
Flare Poller [baseline] (4.135 ms) : 0, 4135
Flare Poller [candidate] (4.12 ms) : 0, 4120
Profiling [baseline] (111.224 ms) : 0, 111224
Profiling [candidate] (111.893 ms) : 0, 111893
LoadParameters
See matching parameters
SummaryFound 2 performance improvements and 0 performance regressions! Performance is the same for 7 metrics, 9 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.56.0-SNAPSHOT~4243ccc14e, baseline=1.56.0-SNAPSHOT~1397f25f48
dateFormat X
axisFormat %s
section baseline
no_agent (1.186 ms) : 1178, 1193
. : milestone, 1186,
iast (3.188 ms) : 3160, 3216
. : milestone, 3188,
iast_FULL (5.925 ms) : 5887, 5963
. : milestone, 5925,
iast_GLOBAL (3.647 ms) : 3610, 3685
. : milestone, 3647,
profiling (1.994 ms) : 1982, 2006
. : milestone, 1994,
tracing (1.79 ms) : 1781, 1799
. : milestone, 1790,
section candidate
no_agent (1.192 ms) : 1184, 1199
. : milestone, 1192,
iast (3.177 ms) : 3153, 3201
. : milestone, 3177,
iast_FULL (5.673 ms) : 5638, 5709
. : milestone, 5673,
iast_GLOBAL (3.609 ms) : 3575, 3643
. : milestone, 3609,
profiling (2.052 ms) : 2040, 2063
. : milestone, 2052,
tracing (1.845 ms) : 1835, 1855
. : milestone, 1845,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~4243ccc14e, baseline=1.56.0-SNAPSHOT~1397f25f48
dateFormat X
axisFormat %s
section baseline
no_agent (1.467 ms) : 1456, 1478
. : milestone, 1467,
appsec (2.445 ms) : 2394, 2496
. : milestone, 2445,
iast (2.192 ms) : 2129, 2255
. : milestone, 2192,
iast_GLOBAL (2.24 ms) : 2176, 2303
. : milestone, 2240,
profiling (2.455 ms) : 2299, 2610
. : milestone, 2455,
tracing (2.02 ms) : 1971, 2069
. : milestone, 2020,
section candidate
no_agent (1.472 ms) : 1460, 1483
. : milestone, 1472,
appsec (3.704 ms) : 3486, 3923
. : milestone, 3704,
iast (2.199 ms) : 2136, 2263
. : milestone, 2199,
iast_GLOBAL (2.242 ms) : 2179, 2305
. : milestone, 2242,
profiling (2.045 ms) : 1994, 2096
. : milestone, 2045,
tracing (2.011 ms) : 1962, 2060
. : milestone, 2011,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~4243ccc14e, baseline=1.56.0-SNAPSHOT~1397f25f48
dateFormat X
axisFormat %s
section baseline
no_agent (15.599 s) : 15599000, 15599000
. : milestone, 15599000,
appsec (14.866 s) : 14866000, 14866000
. : milestone, 14866000,
iast (18.646 s) : 18646000, 18646000
. : milestone, 18646000,
iast_GLOBAL (18.307 s) : 18307000, 18307000
. : milestone, 18307000,
profiling (15.641 s) : 15641000, 15641000
. : milestone, 15641000,
tracing (14.857 s) : 14857000, 14857000
. : milestone, 14857000,
section candidate
no_agent (15.347 s) : 15347000, 15347000
. : milestone, 15347000,
appsec (14.802 s) : 14802000, 14802000
. : milestone, 14802000,
iast (18.478 s) : 18478000, 18478000
. : milestone, 18478000,
iast_GLOBAL (17.978 s) : 17978000, 17978000
. : milestone, 17978000,
profiling (15.271 s) : 15271000, 15271000
. : milestone, 15271000,
tracing (14.559 s) : 14559000, 14559000
. : milestone, 14559000,
|
f68d358 to
140dd23
Compare
eec9091 to
9e108de
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
comp: tooling
Build & Tooling
tag: no release notes
Changes to exclude from release notes
type: enhancement
Enhancements and improvements
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Separate the
petclinicandinsecure-bankload benchmarks into two separate jobs that run in parallel in order to reduce CI time while increasing the repetition count to 5.Motivation
Improve load benchmark stability without increasing CI time.
Additional Notes
WIP
Contributor Checklist
type:and (comp:orinst:) labels in addition to any useful labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]