Skip to content

Commit 8fed488

Browse files
silkfiretimcassell
silkfire
and
timcassell
authored
Add Roslyn analyzers to detect incorrect usage of BenchmarkDotNet (#2837)
* Add Roslyn analyzers to detect incorrect usage of BenchmarkDotNet * Unify C# language version * Remove Analyzers package projects * Revert BenchmarkDotNet.Disassembler changes * Reference Analyzers project from Annotations * Move Benchmark.Analyzers and Benchmark.Analyzers.Tests to correct directories * Remove accidentally added package Microsoft.CodeAnalysis.NetAnalyzers from BenchmarkDotNet.Annotations * * Benchmark classes annotated with a [GenericTypeArguments] attribute must be non-abstract and generic * Benchmark classes are allowed to be generic if they are either abstract or annotated with at least one [GenericTypeArguments] attribute * Assume that a class can be annotated with more than one [GenericTypeArguments] attribute * * Change diagnostic ID increment ordering * Add a rule that the benchmark class referenced in the type argument of the BenchmarkRunner.Run method cannot be abstract * When determining whether a class has any benchmark methods, iterate through its ancestors * Move "Benchmark class cannot be sealed" to Run analyzer * Move "Benchmark class must be public" to Run analyzer * Support analyzing overload of BenchmarkRunner.Run that takes a Type parameter created by using a typeof expression * Remove requirement that a class must have at least one method annotated with the Benchmark attribute when analyzing GenericTypeArguments attribute rules * * Integer attribute values that fit within target type range should not trigger mismatching type diagnostics * Test all valid attribute value types when performing type matching * Use a dummy syntax tree to test whether types are implicitly convertible * Move "Generic class must be abstract or annotated with a [GenericTypeArgumentsAttribute]" to Run analyzer and remove abstract modifier requirement * Add support to analyze implicit conversion from an array to a Span of said array for [Arguments] attribute values * Add support to analyze implicit conversion when using constant values for [Params] and [Arguments] attribute values * * Add support to analyze a boolean constant value for the Baseline property on BenchmarkAttribute * Split logic for baseline method analyzer into two rules, also verifying whether they're unique per category * Add test to OnlyOneMethodCanBeBaselinePerCategory rule verifying that it works correctly with invalid string values * Disable warnings for "Missing XML comment for publicly visible type or member" (CS1591) * Correct resource strings for OnlyOneMethodCanBeBaselinePerCategory rule * Build error fixes * Add analyzer projects to main sln. Run analyzer tests in CI. Fix compile errors. Ensure analyzers run against test projects. * Revert annotations csproj. Add analyzers to samples. * * Change severity level of "UnnecessarySingleValuePassedToAttribute" rule from warning to information * Move diagnostic "MethodWithoutAttributeMustHaveNoParameters" to its own analyzer and also take [ArgumentsSource] attribute into consideration * Suppress analyzer errors for incorrect BenchmarkDotNet usages in tests * Adjust wording of "ClassWithGenericTypeArgumentsAttributeMustBeGenericRule" diagnostic and move trigger location to the attribute level * Revert suppressing of BDN1401 in ArgumentsTests.cs * Revert suppressing of BDN1401 in FullNameProviderTests.cs, PriorityTests.cs and TypeFilterTests.cs * Fix Directory.build.props included in autogenerated projects. * Add message to failing CI test. * Revert sdk update. * Suppress warning. * Skip failing test * Adjust wording of "ClassWithGenericTypeArgumentsAttributeMustBeNonAbstract" diagnostic and move trigger location to the attribute level * Add diagnostics for unintendedly passing a single null value to the [Arguments] or [Params] attribute * Revert commit d760312 * Formatting --------- Co-authored-by: Tim Cassell <cassell.timothy@gmail.com>
1 parent 24a357b commit 8fed488

File tree

52 files changed

+11009
-73
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+11009
-73
lines changed

‎.github/workflows/run-tests.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,12 @@ jobs:
4545
- name: Run task 'build'
4646
shell: cmd
4747
run: ./build.cmd build
48+
- name: Run task 'unit-tests'
49+
shell: cmd
50+
run: ./build.cmd unit-tests -e
51+
- name: Run task 'analyzer-tests'
52+
shell: cmd
53+
run: ./build.cmd analyzer-tests -e
4854
- name: Run task 'in-tests-full'
4955
shell: cmd
5056
run: ./build.cmd in-tests-full -e
@@ -79,6 +85,8 @@ jobs:
7985
# Build and Test
8086
- name: Run task 'build'
8187
run: ./build.cmd build
88+
- name: Run task 'analyzer-tests'
89+
run: ./build.cmd analyzer-tests -e
8290
- name: Run task 'unit-tests'
8391
run: ./build.cmd unit-tests -e
8492
- name: Run task 'in-tests-core'
@@ -116,6 +124,8 @@ jobs:
116124
# Build and Test
117125
- name: Run task 'build'
118126
run: ./build.cmd build
127+
- name: Run task 'analyzer-tests'
128+
run: ./build.cmd analyzer-tests -e
119129
- name: Run task 'unit-tests'
120130
run: ./build.cmd unit-tests -e
121131
- name: Run task 'in-tests-core'

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,9 @@ src/BenchmarkDotNet/Disassemblers/BenchmarkDotNet.Disassembler.*.nupkg
5454
# Visual Studio 2015 cache/options directory
5555
.vs/
5656

57+
# VSCode directory
58+
.vscode/
59+
5760
# Cake
5861
tools/**
5962
.dotnet

‎BenchmarkDotNet.sln‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BenchmarkDotNet.Exporters.P
5959
EndProject
6060
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BenchmarkDotNet.Exporters.Plotting.Tests", "tests\BenchmarkDotNet.Exporters.Plotting.Tests\BenchmarkDotNet.Exporters.Plotting.Tests.csproj", "{199AC83E-30BD-40CD-87CE-0C838AC0320D}"
6161
EndProject
62+
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BenchmarkDotNet.Analyzers", "src\BenchmarkDotNet.Analyzers\BenchmarkDotNet.Analyzers.csproj", "{AA4DDCA0-C1D8-ADA8-69FE-2F67C4CA96B1}"
63+
EndProject
64+
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BenchmarkDotNet.Analyzers.Tests", "tests\BenchmarkDotNet.Analyzers.Tests\BenchmarkDotNet.Analyzers.Tests.csproj", "{7DE89F16-2160-42E3-004E-1F5064732121}"
65+
EndProject
6266
Global
6367
GlobalSection(SolutionConfigurationPlatforms) = preSolution
6468
Debug|Any CPU = Debug|Any CPU
@@ -161,6 +165,14 @@ Global
161165
{199AC83E-30BD-40CD-87CE-0C838AC0320D}.Debug|Any CPU.Build.0 = Debug|Any CPU
162166
{199AC83E-30BD-40CD-87CE-0C838AC0320D}.Release|Any CPU.ActiveCfg = Release|Any CPU
163167
{199AC83E-30BD-40CD-87CE-0C838AC0320D}.Release|Any CPU.Build.0 = Release|Any CPU
168+
{AA4DDCA0-C1D8-ADA8-69FE-2F67C4CA96B1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
169+
{AA4DDCA0-C1D8-ADA8-69FE-2F67C4CA96B1}.Debug|Any CPU.Build.0 = Debug|Any CPU
170+
{AA4DDCA0-C1D8-ADA8-69FE-2F67C4CA96B1}.Release|Any CPU.ActiveCfg = Release|Any CPU
171+
{AA4DDCA0-C1D8-ADA8-69FE-2F67C4CA96B1}.Release|Any CPU.Build.0 = Release|Any CPU
172+
{7DE89F16-2160-42E3-004E-1F5064732121}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
173+
{7DE89F16-2160-42E3-004E-1F5064732121}.Debug|Any CPU.Build.0 = Debug|Any CPU
174+
{7DE89F16-2160-42E3-004E-1F5064732121}.Release|Any CPU.ActiveCfg = Release|Any CPU
175+
{7DE89F16-2160-42E3-004E-1F5064732121}.Release|Any CPU.Build.0 = Release|Any CPU
164176
EndGlobalSection
165177
GlobalSection(SolutionProperties) = preSolution
166178
HideSolutionNode = FALSE
@@ -190,6 +202,8 @@ Global
190202
{2E2283A3-6DA6-4482-8518-99D6D9F689AB} = {D6597E3A-6892-4A68-8E14-042FC941FDA2}
191203
{B92ECCEF-7C27-4012-9E19-679F3C40A6A6} = {D6597E3A-6892-4A68-8E14-042FC941FDA2}
192204
{199AC83E-30BD-40CD-87CE-0C838AC0320D} = {14195214-591A-45B7-851A-19D3BA2413F9}
205+
{AA4DDCA0-C1D8-ADA8-69FE-2F67C4CA96B1} = {D6597E3A-6892-4A68-8E14-042FC941FDA2}
206+
{7DE89F16-2160-42E3-004E-1F5064732121} = {14195214-591A-45B7-851A-19D3BA2413F9}
193207
EndGlobalSection
194208
GlobalSection(ExtensibilityGlobals) = postSolution
195209
SolutionGuid = {4D9AF12B-1F7F-45A7-9E8C-E4E46ADCBD1F}

‎NuGet.Config‎

Lines changed: 21 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,21 @@
1-
<?xml version="1.0" encoding="utf-8"?>
2-
<configuration>
3-
<solution>
4-
<add key="disableSourceControlIntegration" value="true" />
5-
</solution>
6-
<packageSources>
7-
<!--To inherit the global NuGet package sources remove the <clear/> line below -->
8-
<clear />
9-
10-
<add key="api.nuget.org" value="https://api.nuget.org/v3/index.json" />
11-
<!-- reuquired to run Mono AOT benchmarks -->
12-
<add key="dotnet6" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json" />
13-
<add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />
14-
<add key="dotnet8" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json" />
15-
<add key="dotnet9" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9/nuget/v3/index.json" />
16-
<add key="dotnet10" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet10/nuget/v3/index.json" />
17-
</packageSources>
18-
</configuration>
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<configuration>
3+
<solution>
4+
<add key="disableSourceControlIntegration" value="true" />
5+
</solution>
6+
<packageSources>
7+
<!--To inherit the global NuGet package sources remove the <clear/> line below -->
8+
<clear />
9+
10+
<add key="api.nuget.org" value="https://api.nuget.org/v3/index.json" />
11+
<!-- required to run Mono AOT benchmarks -->
12+
<add key="dotnet6" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json" />
13+
<add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />
14+
<add key="dotnet8" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json" />
15+
<add key="dotnet9" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9/nuget/v3/index.json" />
16+
<add key="dotnet10" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet10/nuget/v3/index.json" />
17+
18+
<!-- required for Roslyn analyzers -->
19+
<add key="dotnet-tools" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json" />
20+
</packageSources>
21+
</configuration>

‎build/BenchmarkDotNet.Build/Program.cs‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,28 @@ public HelpInfo GetHelp()
9898
}
9999
}
100100

101+
[TaskName(Name)]
102+
[TaskDescription("Run analyzer tests")]
103+
[IsDependentOn(typeof(BuildTask))]
104+
public class AnalyzerTestsTask : FrostingTask<BuildContext>, IHelpProvider
105+
{
106+
private const string Name = "analyzer-tests";
107+
public override void Run(BuildContext context) => context.UnitTestRunner.RunAnalyzerTests();
108+
109+
public HelpInfo GetHelp()
110+
{
111+
return new HelpInfo
112+
{
113+
Examples =
114+
[
115+
new Example(Name)
116+
.WithArgument(KnownOptions.Exclusive)
117+
.WithArgument(KnownOptions.Verbosity, "Diagnostic")
118+
]
119+
};
120+
}
121+
}
122+
101123
[TaskName(Name)]
102124
[TaskDescription("Run integration tests using .NET Framework 4.6.2+ (slow)")]
103125
[IsDependentOn(typeof(BuildTask))]
@@ -123,8 +145,9 @@ public class InTestsCoreTask : FrostingTask<BuildContext>, IHelpProvider
123145
}
124146

125147
[TaskName(Name)]
126-
[TaskDescription("Run all unit and integration tests (slow)")]
148+
[TaskDescription("Run all unit, analyzer, and integration tests (slow)")]
127149
[IsDependentOn(typeof(UnitTestsTask))]
150+
[IsDependentOn(typeof(AnalyzerTestsTask))]
128151
[IsDependentOn(typeof(InTestsFullTask))]
129152
[IsDependentOn(typeof(InTestsCoreTask))]
130153
public class AllTestsTask : FrostingTask<BuildContext>, IHelpProvider

‎build/BenchmarkDotNet.Build/Runners/UnitTestRunner.cs‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,11 @@ public class UnitTestRunner(BuildContext context)
2020
.Combine("BenchmarkDotNet.Exporters.Plotting.Tests")
2121
.CombineWithFilePath("BenchmarkDotNet.Exporters.Plotting.Tests.csproj");
2222

23+
private FilePath AnalyzerTestsProjectFile { get; } = context.RootDirectory
24+
.Combine("tests")
25+
.Combine("BenchmarkDotNet.Analyzers.Tests")
26+
.CombineWithFilePath("BenchmarkDotNet.Analyzers.Tests.csproj");
27+
2328
private FilePath IntegrationTestsProjectFile { get; } = context.RootDirectory
2429
.Combine("tests")
2530
.Combine("BenchmarkDotNet.IntegrationTests")
@@ -70,5 +75,12 @@ public void RunUnitTests()
7075
RunUnitTests(targetFramework);
7176
}
7277

78+
public void RunAnalyzerTests()
79+
{
80+
string[] targetFrameworks = context.IsRunningOnWindows() ? ["net462", "net8.0"] : ["net8.0"];
81+
foreach (var targetFramework in targetFrameworks)
82+
RunTests(AnalyzerTestsProjectFile, "analyzer", targetFramework);
83+
}
84+
7385
public void RunInTests(string tfm) => RunTests(IntegrationTestsProjectFile, "integration", tfm);
7486
}

‎build/common.props‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,11 @@
2121
<CheckEolTargetFramework>false</CheckEolTargetFramework>
2222
<CodeAnalysisRuleSet>$(MSBuildThisFileDirectory)CodingStyle.ruleset</CodeAnalysisRuleSet>
2323
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
24-
24+
<WarningsNotAsErrors>NU1900</WarningsNotAsErrors>
2525
<Nullable>annotations</Nullable>
2626
<!-- Suppress warning for nuget package used in old (unsupported) tfm. -->
2727
<SuppressTfmSupportBuildWarnings>true</SuppressTfmSupportBuildWarnings>
28+
<NoWarn>CS9057</NoWarn>
2829
</PropertyGroup>
2930

3031
<ItemGroup>

‎samples/BenchmarkDotNet.Samples/BenchmarkDotNet.Samples.csproj‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,4 +40,8 @@
4040
<ProjectReference Include="..\..\src\BenchmarkDotNet.Diagnostics.Windows\BenchmarkDotNet.Diagnostics.Windows.csproj" />
4141
<ProjectReference Include="..\..\src\BenchmarkDotNet.TestAdapter\BenchmarkDotNet.TestAdapter.csproj" />
4242
</ItemGroup>
43+
<ItemGroup>
44+
<!-- Enables analyzers for this project (this is required since ProjectReference is not transitive). -->
45+
<ProjectReference Include="..\..\src\BenchmarkDotNet.Analyzers\BenchmarkDotNet.Analyzers.csproj" ReferenceOutputAssembly="false" OutputItemType="Analyzer"/>
46+
</ItemGroup>
4347
</Project>

0 commit comments

Comments
 (0)