Merge branch 'main' into serializable-fence

Author: nedtwigg

.gitignore

@@ -19,9 +19,6 @@ gradle-app.setting
 # Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)
 !gradle-wrapper.jar
 
-#Gradle-Lock files in _ext just serving as an input for lib-extra
-_ext/*/gradle/dependency-locks/*.lockfile
-
 ### Eclipse ###
 .metadata
 .gradle

CHANGES.md

@@ -10,6 +10,30 @@ This document is intended for Spotless developers.
 We adhere to the [keepachangelog](https://keepachangelog.com/en/1.0.0/) format (starting after version `1.27.0`).
 
 ## [Unreleased]
+### Added
+* `FileSignature.Promised` and `JarState.Promised` to facilitate round-trip serialization for the Gradle configuration cache. ([#1945](https://github.com/diffplug/spotless/pull/1945)) 
+### Removed
+* **BREAKING** Remove `JarState.getMavenCoordinate(String prefix)`. ([#1945](https://github.com/diffplug/spotless/pull/1945))
+### Fixed
+* Ignore system git config when running tests ([#1990](https://github.com/diffplug/spotless/issues/1990))
+
+## [2.45.0] - 2024-01-23
+### Added
+* Support for `gofmt` ([#2001](https://github.com/diffplug/spotless/pull/2001))
+* Support for formatting Java Docs for the Palantir formatter ([#2009](https://github.com/diffplug/spotless/pull/2009))
+
+## [2.44.0] - 2024-01-15
+### Added
+* New static method to `DiffMessageFormatter` which allows to retrieve diffs with their line numbers ([#1960](https://github.com/diffplug/spotless/issues/1960))
+* Gradle - Support for formatting shell scripts via [shfmt](https://github.com/mvdan/sh). ([#1994](https://github.com/diffplug/spotless/pull/1994))
+### Fixed
+* Fix empty files with biome >= 1.5.0 when formatting files that are in the ignore list of the biome configuration file. ([#1989](https://github.com/diffplug/spotless/pull/1989) fixes [#1987](https://github.com/diffplug/spotless/issues/1987))
+* Fix a regression in BufStep where the same arguments were being provided to every `buf` invocation. ([#1976](https://github.com/diffplug/spotless/issues/1976))
+### Changes
+* Use palantir-java-format 2.39.0 on Java 21. ([#1948](https://github.com/diffplug/spotless/pull/1948))
+* Bump default `ktlint` version to latest `1.0.1` -> `1.1.1`. ([#1973](https://github.com/diffplug/spotless/pull/1973))
+* Bump default `googleJavaFormat` version to latest `1.18.1` -> `1.19.2`. ([#1971](https://github.com/diffplug/spotless/pull/1971))
+* Bump default `diktat` version to latest `1.2.5` -> `2.0.0`. ([#1972](https://github.com/diffplug/spotless/pull/1972))
 
 ## [2.43.1] - 2023-12-04
 ### Fixed

CONTRIBUTING.md

@@ -36,7 +36,6 @@ For the folders below in monospace text, they are published on MavenCentral at t
 | `lib-extra` | Contains the optional parts of Spotless which require external dependencies.  `LineEnding.GIT_ATTRIBUTES` won't work unless `lib-extra` is available. |
 | `plugin-gradle` | Integrates spotless and all of its formatters into Gradle. |
 | `plugin-maven` | Integrates spotless and all of its formatters into Maven. |
-| `_ext` | Folder for generating glue jars (specifically packaging Eclipse jars from p2 for consumption using maven).
 
 ## How to add a new FormatterStep
 
@@ -95,10 +94,31 @@ Here's a checklist for creating a new step for Spotless:
 
 - [ ] Class name ends in Step, `SomeNewStep`.
 - [ ] Class has a public static method named `create` that returns a `FormatterStep`.
-- [ ] Has a test class named `SomeNewStepTest`.
+- [ ] Has a test class named `SomeNewStepTest` that uses `StepHarness` or `StepHarnessWithFile` to test the step.
+  - [ ] Start with `StepHarness.forStep(myStep).supportsRoundTrip(false)`, and then add round trip support as described in the next section. 
 - [ ] Test class has test methods to verify behavior.
 - [ ] Test class has a test method `equality()` which tests equality using `StepEqualityTester` (see existing methods for examples).
 
+### Serialization roundtrip
+
+In order to support Gradle's configuration cache, all `FormatterStep` must be round-trip serializable. This is a bit tricky because step equality is based on the serialized form of the state, and `transient` is used to take absolute paths out of the equality check. To make this work, roundtrip compatible steps actually have *two* states:
+
+- `RoundtripState` which must be roundtrip serializable but has no equality constraints
+  - `FileSignature.Promised` for settings files and `JarState.Promised` for the classpath 
+- `EqualityState` which will never be reserialized and its serialized form is used for equality / hashCode checks
+  - `FileSignature` for settings files and `JarState` for the classpath
+
+```java
+FormatterStep create(String name,
+  RoundtripState roundTrip,
+  SerializedFunction<RoundtripState, EqualityState> equalityFunc,
+  SerializedFunction<EqualityState, FormatterFunc> formatterFunc)
+FormatterStep createLazy(String name,
+  Supplier<RoundtripState> roundTrip,
+  SerializedFunction<RoundtripState, EqualityState> equalityFunc,
+  SerializedFunction<EqualityState, FormatterFunc> formatterFunc)
+```
+
 ### Third-party dependencies via reflection or compile-only source sets
 
 Most formatters are going to use some kind of third-party jar. Spotless integrates with many formatters, some of which have incompatible transitive dependencies. To address this, we resolve third-party dependencies using [`JarState`](https://github.com/diffplug/spotless/blob/b26f0972b185995d7c6a7aefa726c146d24d9a82/lib/src/main/java/com/diffplug/spotless/kotlin/KtfmtStep.java#L118). To call methods on the classes in that `JarState`, you can either use reflection or a compile-only source set. See [#524](https://github.com/diffplug/spotless/issues/524) for examples of both approaches.
@@ -119,24 +139,6 @@ There are many great formatters (prettier, clang-format, black, etc.) which live
 
 Because of Spotless' up-to-date checking and [git ratcheting](https://github.com/diffplug/spotless/tree/main/plugin-gradle#ratchet), Spotless actually doesn't have to call formatters very often, so even an expensive shell call for every single invocation isn't that bad.  Anything that works is better than nothing, and we can always speed things up later if it feels too slow (but it probably won't).
 
-## How to enable the `_ext` projects
-
-The `_ext` projects are disabled per default, since:
-
-* some of the projects perform vast downloads at configuration time
-* the downloaded content may change on server side and break CI builds
-
-
-The `_ext` can be activated via the root project property `com.diffplug.spotless.include.ext`.
-
-Activate the property via command line, like for example:
-
-```
-gradlew -Pcom.diffplug.spotless.include.ext=true build
-```
-
-Or set the property in your user `gradle.properties` file, which is especially recommended if you like to work with the `_ext` projects using IDEs.
-
 ## How to add a new plugin for a build system
 
 The gist of it is that you will have to:
@@ -151,6 +153,37 @@ The gist of it is that you will have to:
 
 If you get something running, we'd love to host your plugin within this repo as a peer to `plugin-gradle` and `plugin-maven`.
 
+## Run tests
+
+To run all tests, simply do
+
+> gradlew test
+
+Since that takes some time, you might only want to run the tests
+concerning what you are working on:
+
+```shell
+# Run only from test from the "lib" project
+gradlew :testlib:test --tests com.diffplug.spotless.generic.IndentStepTest
+
+# Run only one test from the "plugin-maven" project
+gradlew :plugin-maven:test --tests com.diffplug.spotless.maven.pom.SortPomMavenTest
+
+# Run only one test from the "plugin-gradle" project
+gradlew :plugin-gradle:test --tests com.diffplug.gradle.spotless.FreshMarkExtensionTest
+```
+
+## Check and format code
+
+Before creating a pull request, you might want to format (yes, spotless is  formatted by spotless)
+the code and check for possible bugs
+
+* `./gradlew spotlessApply`
+* `./gradlew spotbugsMain`
+
+These checks are also run by the automated pipeline when you submit a pull request, if
+the pipeline fails, first check if the code is formatted and no bugs were found.
+
 ## Integration testing
 
 ### Gradle - locally

README.md

@@ -4,7 +4,7 @@
 [![Maven Plugin](https://img.shields.io/maven-central/v/com.diffplug.spotless/spotless-maven-plugin?color=blue&label=maven%20plugin)](plugin-maven)
 [![SBT Plugin](https://img.shields.io/badge/sbt%20plugin-0.1.3-blue)](https://github.com/moznion/sbt-spotless)
 
-Spotless can format <antlr | c | c# | c++ | css | flow | graphql | groovy | html | java | javascript | json | jsx | kotlin | less | license headers | markdown | objective-c | protobuf | python | scala | scss | sql | typeScript | vue | yaml | anything> using <gradle | maven | sbt | anything>.
+Spotless can format <antlr | c | c# | c++ | css | flow | graphql | groovy | html | java | javascript | json | jsx | kotlin | less | license headers | markdown | objective-c | protobuf | python | scala | scss | shell | sql | typeScript | vue | yaml | anything> using <gradle | maven | sbt | anything>.
 
 You probably want one of the links below:
 
@@ -75,8 +75,10 @@ lib('generic.ReplaceRegexStep')                  +'{{yes}}       | {{yes}}
 lib('generic.ReplaceStep')                       +'{{yes}}       | {{yes}}      | {{no}}       | {{no}}  |',
 lib('generic.TrimTrailingWhitespaceStep')        +'{{yes}}       | {{yes}}      | {{no}}       | {{no}}  |',
 lib('antlr4.Antlr4FormatterStep')                +'{{yes}}       | {{yes}}      | {{no}}       | {{no}}  |',
+lib('biome.BiomeStep')                           +'{{yes}}       | {{yes}}      | {{no}}       | {{no}}  |',
 lib('cpp.ClangFormatStep')                       +'{{yes}}       | {{no}}       | {{no}}       | {{no}}  |',
 extra('cpp.EclipseFormatterStep')                +'{{yes}}       | {{yes}}      | {{yes}}      | {{no}}  |',
+lib('go.GofmtFormatStep')                        +'{{yes}}       | {{no}}       | {{no}}       | {{no}}  |',
 lib('gherkin.GherkinUtilsStep')                  +'{{yes}}       | {{yes}}      | {{no}}       | {{no}}  |',
 extra('groovy.GrEclipseFormatterStep')           +'{{yes}}       | {{yes}}      | {{yes}}      | {{no}}  |',
 lib('java.GoogleJavaFormatStep')                 +'{{yes}}       | {{yes}}      | {{yes}}      | {{no}}  |',
@@ -101,8 +103,8 @@ lib('npm.TsFmtFormatterStep')                    +'{{yes}}       | {{yes}}
 lib('pom.SortPomStepStep')                       +'{{no}}        | {{yes}}      | {{no}}       | {{no}}  |',
 lib('protobuf.BufStep')                          +'{{yes}}       | {{no}}       | {{no}}       | {{no}}  |',
 lib('python.BlackStep')                          +'{{yes}}       | {{no}}       | {{no}}       | {{no}}  |',
-lib('biome.BiomeStep')                           +'{{yes}}       | {{yes}}      | {{no}}       | {{no}}  |',
 lib('scala.ScalaFmtStep')                        +'{{yes}}       | {{yes}}      | {{yes}}      | {{no}}  |',
+lib('shell.ShfmtStep')                           +'{{yes}}       | {{yes}}      | {{no}}       | {{no}}  |',
 lib('sql.DBeaverSQLFormatterStep')               +'{{yes}}       | {{yes}}      | {{yes}}      | {{no}}  |',
 extra('wtp.EclipseWtpFormatterStep')             +'{{yes}}       | {{yes}}      | {{no}}       | {{no}}  |',
 lib('yaml.JacksonYamlStep')                      +'{{yes}}       | {{yes}}      | {{no}}       | {{no}}  |',
@@ -127,8 +129,10 @@ lib('yaml.JacksonYamlStep')                      +'{{yes}}       | {{yes}}
 | [`generic.ReplaceStep`](lib/src/main/java/com/diffplug/spotless/generic/ReplaceStep.java) | :+1:       | :+1:      | :white_large_square:       | :white_large_square:  |
 | [`generic.TrimTrailingWhitespaceStep`](lib/src/main/java/com/diffplug/spotless/generic/TrimTrailingWhitespaceStep.java) | :+1:       | :+1:      | :white_large_square:       | :white_large_square:  |
 | [`antlr4.Antlr4FormatterStep`](lib/src/main/java/com/diffplug/spotless/antlr4/Antlr4FormatterStep.java) | :+1:       | :+1:      | :white_large_square:       | :white_large_square:  |
+| [`biome.BiomeStep`](lib/src/main/java/com/diffplug/spotless/biome/BiomeStep.java) | :+1:       | :+1:      | :white_large_square:       | :white_large_square:  |
 | [`cpp.ClangFormatStep`](lib/src/main/java/com/diffplug/spotless/cpp/ClangFormatStep.java) | :+1:       | :white_large_square:       | :white_large_square:       | :white_large_square:  |
 | [`cpp.EclipseFormatterStep`](lib-extra/src/main/java/com/diffplug/spotless/extra/cpp/EclipseFormatterStep.java) | :+1:       | :+1:      | :+1:      | :white_large_square:  |
+| [`go.GofmtFormatStep`](lib/src/main/java/com/diffplug/spotless/go/GofmtFormatStep.java) | :+1:       | :white_large_square:       | :white_large_square:       | :white_large_square:  |
 | [`gherkin.GherkinUtilsStep`](lib/src/main/java/com/diffplug/spotless/gherkin/GherkinUtilsStep.java) | :+1:       | :+1:      | :white_large_square:       | :white_large_square:  |
 | [`groovy.GrEclipseFormatterStep`](lib-extra/src/main/java/com/diffplug/spotless/extra/groovy/GrEclipseFormatterStep.java) | :+1:       | :+1:      | :+1:      | :white_large_square:  |
 | [`java.GoogleJavaFormatStep`](lib/src/main/java/com/diffplug/spotless/java/GoogleJavaFormatStep.java) | :+1:       | :+1:      | :+1:      | :white_large_square:  |
@@ -153,8 +157,8 @@ lib('yaml.JacksonYamlStep')                      +'{{yes}}       | {{yes}}
 | [`pom.SortPomStepStep`](lib/src/main/java/com/diffplug/spotless/pom/SortPomStepStep.java) | :white_large_square:        | :+1:      | :white_large_square:       | :white_large_square:  |
 | [`protobuf.BufStep`](lib/src/main/java/com/diffplug/spotless/protobuf/BufStep.java) | :+1:       | :white_large_square:       | :white_large_square:       | :white_large_square:  |
 | [`python.BlackStep`](lib/src/main/java/com/diffplug/spotless/python/BlackStep.java) | :+1:       | :white_large_square:       | :white_large_square:       | :white_large_square:  |
-| [`biome.BiomeStep`](lib/src/main/java/com/diffplug/spotless/biome/BiomeStep.java) | :+1:       | :+1:      | :white_large_square:       | :white_large_square:  |
 | [`scala.ScalaFmtStep`](lib/src/main/java/com/diffplug/spotless/scala/ScalaFmtStep.java) | :+1:       | :+1:      | :+1:      | :white_large_square:  |
+| [`shell.ShfmtStep`](lib/src/main/java/com/diffplug/spotless/shell/ShfmtStep.java) | :+1:       | :+1:      | :white_large_square:       | :white_large_square:  |
 | [`sql.DBeaverSQLFormatterStep`](lib/src/main/java/com/diffplug/spotless/sql/DBeaverSQLFormatterStep.java) | :+1:       | :+1:      | :+1:      | :white_large_square:  |
 | [`wtp.EclipseWtpFormatterStep`](lib-extra/src/main/java/com/diffplug/spotless/extra/wtp/EclipseWtpFormatterStep.java) | :+1:       | :+1:      | :white_large_square:       | :white_large_square:  |
 | [`yaml.JacksonYamlStep`](lib/src/main/java/com/diffplug/spotless/yaml/JacksonYamlStep.java) | :+1:       | :+1:      | :white_large_square:       | :white_large_square:  |

gradle/special-tests.gradle

@@ -1,17 +1,17 @@
 apply plugin: 'com.adarshr.test-logger'
+
+// See com.diffplug.spotless.tag package for available JUnit 5 @Tag annotations
 def special = [
-	'Npm',
-	'Black',
-	'Clang',
-	'Buf'
+	'black',
+	'buf',
+	'clang',
+	'gofmt',
+	'npm',
+	'shfmt'
 ]
 
 boolean isCiServer = System.getenv().containsKey("CI")
 tasks.withType(Test).configureEach {
-	// See com.diffplug.spotless.tag package for available JUnit 5 @Tag annotations
-	useJUnitPlatform {
-		excludeTags special as String[]
-	}
 	if (isCiServer) {
 		retry {
 			maxRetries = 2
@@ -25,9 +25,13 @@ tasks.withType(Test).configureEach {
 		maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1
 	}
 }
-
+tasks.named('test').configure {
+	useJUnitPlatform {
+		excludeTags special as String[]
+	}
+}
 special.forEach { tag ->
-	tasks.register("test${tag}", Test) {
+	tasks.register("test${tag.capitalize()}", Test) {
 		useJUnitPlatform { includeTags tag }
 	}
 }

lib-extra/build.gradle

@@ -7,7 +7,7 @@ version = rootProject.spotlessChangelog.versionNext
 apply from: rootProject.file('gradle/java-setup.gradle')
 apply from: rootProject.file('gradle/java-publish.gradle')
 
-String VER_SOLSTICE = '1.7.4'
+String VER_SOLSTICE = '1.7.5'
 dependencies {
 	api projects.lib
 	// misc useful utilities

lib-extra/src/main/java/com/diffplug/spotless/extra/EclipseBasedStepBuilder.java

@@ -1,5 +1,5 @@
 /*
- * Copyright 2016-2023 DiffPlug
+ * Copyright 2016-2024 DiffPlug
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.

lib-extra/src/main/java/com/diffplug/spotless/extra/EquoBasedStepBuilder.java

@@ -1,5 +1,5 @@
 /*
- * Copyright 2016-2023 DiffPlug
+ * Copyright 2016-2024 DiffPlug
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -113,7 +113,7 @@ public FormatterStep build() {
 			}
 			var classpath = new ArrayList<File>();
 			var mavenDeps = new ArrayList<String>();
-			mavenDeps.add("dev.equo.ide:solstice:1.7.4");
+			mavenDeps.add("dev.equo.ide:solstice:1.7.5");
 			mavenDeps.add("com.diffplug.durian:durian-swt.os:4.2.0");
 			mavenDeps.addAll(query.getJarsOnMavenCentral());
 			classpath.addAll(mavenProvisioner.provisionWithTransitives(false, mavenDeps));

lib-extra/src/main/java/com/diffplug/spotless/extra/integration/DiffMessageFormatter.java

@@ -24,9 +24,11 @@
 import java.nio.file.Path;
 import java.util.List;
 import java.util.ListIterator;
+import java.util.Map;
 import java.util.Objects;
 
 import org.eclipse.jgit.diff.DiffFormatter;
+import org.eclipse.jgit.diff.Edit;
 import org.eclipse.jgit.diff.EditList;
 import org.eclipse.jgit.diff.MyersDiff;
 import org.eclipse.jgit.diff.RawText;
@@ -234,6 +236,19 @@ private void addIntendedLine(String indent, String line) {
 	 * sequence (\n, \r, \r\n).
 	 */
 	private String diff(File file) throws IOException {
+		return diff(formatter, file).getValue();
+	}
+
+	/**
+	 * Returns a map entry with value being a git-style diff between the contents of the given file and what those contents would
+	 * look like if formatted using the given formatter. Does not end with any newline
+	 * sequence (\n, \r, \r\n). The key of the map entry is the 0-based line where the first difference occurred.
+	 */
+	public static Map.Entry<Integer, String> diff(Formatter formatter, File file) throws IOException {
+		return diff(new CleanProviderFormatter(formatter), file);
+	}
+
+	private static Map.Entry<Integer, String> diff(CleanProvider formatter, File file) throws IOException {
 		String raw = new String(Files.readAllBytes(file.toPath()), formatter.getEncoding());
 		String rawUnix = LineEnding.toUnix(raw);
 		String formatted = formatter.getFormatted(file, rawUnix);
@@ -248,13 +263,13 @@ private String diff(File file) throws IOException {
 	}
 
 	/**
-	 * Returns a git-style diff between the two unix strings.
+	 * Returns a map entry with value being a git-style diff between the two unix strings and key being the 0-based line of the first difference (in the dirty string)
 	 * <p>
 	 * Output has no trailing newlines.
 	 * <p>
 	 * Boolean args determine whether whitespace or line endings will be visible.
 	 */
-	private static String diffWhitespaceLineEndings(String dirty, String clean, boolean whitespace, boolean lineEndings) throws IOException {
+	private static Map.Entry<Integer, String> diffWhitespaceLineEndings(String dirty, String clean, boolean whitespace, boolean lineEndings) throws IOException {
 		dirty = visibleWhitespaceLineEndings(dirty, whitespace, lineEndings);
 		clean = visibleWhitespaceLineEndings(clean, whitespace, lineEndings);
 
@@ -271,7 +286,11 @@ private static String diffWhitespaceLineEndings(String dirty, String clean, bool
 
 		// we don't need the diff to show this, since we display newlines ourselves
 		formatted = formatted.replace("\\ No newline at end of file\n", "");
-		return NEWLINE_MATCHER.trimTrailingFrom(formatted);
+		return Map.entry(getLineOfFirstDifference(edits), NEWLINE_MATCHER.trimTrailingFrom(formatted));
+	}
+
+	private static int getLineOfFirstDifference(EditList edits) {
+		return edits.stream().mapToInt(Edit::getBeginA).min().getAsInt();
 	}
 
 	private static final CharMatcher NEWLINE_MATCHER = CharMatcher.is('\n');