|
5 | 5 | </div> |
6 | 6 |
|
7 | 7 | <p align="center"> |
8 | | - <a href="#General-set-up">General set up</a> • |
| 8 | + <a href="#General-setup">General setup</a> • |
9 | 9 | <a href="#Benchmark-heap-allocation-of-several-Maven-releases">Benchmark heap allocation of several Maven releases</a> |
10 | 10 | </p> |
11 | 11 | <p align="center"> |
|
14 | 14 | <a href="#License">License</a> |
15 | 15 | </p> |
16 | 16 |
|
17 | | -At this moment, this project allows to benchmark and investigate the origin of heap allocation caused by *mvn validate*. |
| 17 | +At this moment, this project allows to benchmark and investigate the origin of heap allocation caused by `mvn validate` (the first phase before lanching any plugin: see [Lifecycles Reference](https://maven.apache.org/ref/current/maven-core/lifecycles.html)). |
18 | 18 |
|
19 | | -This project is based on [QuickPerf](https://github.com/quick-perf/quickperf) to measure and investigate the heap allocation level. |
| 19 | +This project uses [QuickPerf](https://github.com/quick-perf/quickperf) to measure and investigate the heap allocation level. |
20 | 20 |
|
21 | | -Measures have been done executing *mvn validate* on Apache Camel project. |
| 21 | +Measures have been done executing `mvn validate` on Apache Camel source code. |
22 | 22 |
|
23 | 23 | Feel free to use this project and contribute to it! |
24 | 24 |
|
25 | | -# General set up |
| 25 | +# General setup |
26 | 26 |
|
27 | | -This project contains two types of test. *MvnValidateAllocationByMaven3VersionTest* can be used to evaluate the heap allocation level for a range of Maven versions. For a given Maven version, *MvnValidateProfilingTest* can be used to investigate the origin of allocation. |
| 27 | +This project contains two types of test: |
| 28 | +- `MvnValidateAllocationByMaven3VersionTest` can be used to evaluate the heap allocation level for a range of Maven versions, |
| 29 | +- for a given Maven version, `MvnValidateProfilingTest` can be used to investigate the origin of allocation. |
28 | 30 |
|
29 | | -This general set up part describes configurations common to both tests. |
| 31 | +This general setup part describes configurations common to both tests. |
30 | 32 |
|
31 | | -You have to give values to the *project-under-test.path* and *maven.binaries.path* properties contained in the *maven-bench.properties* file. The other properties are only used by *MvnValidateAllocationByMaven3VersionTest*. |
| 33 | +You have to define values to the `project-under-test.path` and `maven.binaries.path` properties contained in the `maven-bench.properties` file. The other properties are only used by `MvnValidateAllocationByMaven3VersionTest`. |
32 | 34 |
|
33 | | -The *project-under-test.path* represents the path of the project on which *mvn validate* will be applied. |
34 | | -Our measures are based on the Apache Camel project. A specific version of this project was chosen to be able to apply *mvn validate* from Maven 3.2.5 to 3.6.2: |
| 35 | +The `project-under-test.path` represents the path of the project on which `mvn validate` will be applied. |
| 36 | +Our next measures are based on the Apache Camel project, but you can choose your own target. For reproducibility of our measure, a precisely defined version of this project was chosen: |
35 | 37 | ``` |
36 | 38 | git clone -n https://github.com/apache/camel.git |
37 | 39 | git checkout c409ab7aabb971065fc8384a861904d2a2819be5 |
38 | 40 | ``` |
39 | | -This Apache Camel version contains 841 modules. |
| 41 | +This Apache Camel version contains 841 modules: such a huge build is perfect to get significant measures. |
40 | 42 |
|
41 | | -The *maven.binaries.path* property corresponds to the path where the needed Maven distributions will be automatically downloaded by the tests. Downloads are performed during *@Before* execution. |
42 | | -If you want to apply measures on Maven head, you can execute the following commands where {maven-distrib-location} has to be replaced with the url given by the *maven.binaries.path* property of *maven-bench.properties* file: |
| 43 | +The `maven.binaries.path` property corresponds to the path where the needed Maven distributions will be automatically downloaded by the tests. Downloads are performed during *@Before* execution. |
| 44 | +If you want to apply measures on Maven HEAD, you can execute the following commands where {maven-distrib-location} has to be replaced with the url given by the `maven.binaries.path` property of `maven-bench.properties` file: |
43 | 45 | ``` |
44 | 46 | git clone https://github.com/apache/maven.git |
45 | 47 | cd maven |
46 | 48 | mvn -DdistributionTargetDir="{maven-distrib-location}/apache-maven-head" clean package |
47 | 49 | ``` |
48 | 50 |
|
49 | | -Heap size can be fixed with the help of [@HeapSize](https://github.com/quick-perf/doc/wiki/JVM-annotations#heapsize). As we are going to see thereafter, in the range of 3.2.5 to 3.6.2 Maven versions, heap allocation level is the highest with Maven 3.2.5 and the smallest with Maven 3.6.2. |
50 | | -With [our execution context](measures/execution-context-2019-09-01-18-48-41.txt), Maven 3.2.5 and a heap size between 6 Go and 9 Go, one measure of heap allocation lasts around one minute. The test length is about one minute and a half with a 5 Go heap size, probably due to more garbage collection. With Maven 3.6.2, the test length is around 15 s for a heap size between 1 Go to 9 Go. |
51 | | - |
52 | 51 | # Benchmark heap allocation of several Maven releases |
53 | 52 |
|
54 | | -*MvnValidateAllocationByMaven3VersionTest* test allows to benchmark the heap allocation level on several Maven 3 distributions. |
| 53 | +`MvnValidateAllocationByMaven3VersionTest` test allows to benchmark the heap allocation level on several Maven 3 distributions. |
55 | 54 |
|
56 | 55 | Heap allocation level is measured with the help of [@MeasureHeapAllocation](https://github.com/quick-perf/doc/wiki/JVM-annotations#Verify-heap-allocation) QuickPerf annotation. This annotation measures the heap allocation level of the thread running the method annotated with @Test. |
57 | | -Feel free to contribute to QuickPerf by adding a feature allowing to measure the allocation level aggregated across all the threads! With *mvn validate*, we have checked that Maven code is not multithreaded by profiling the JVM with the help of [@ProfileJvm](https://github.com/quick-perf/doc/wiki/JVM-annotations#ProfileJvm). |
| 56 | +Feel free to contribute to QuickPerf by adding a feature allowing to measure the allocation level aggregated across all the threads! With `mvn validate`, we have checked that Maven code is not multithreaded during this validate phase by profiling the JVM with the help of [@ProfileJvm](https://github.com/quick-perf/doc/wiki/JVM-annotations#ProfileJvm). |
58 | 57 |
|
59 | | -Please read [General set up](#General-set-up) to get some of the set up requirements. |
| 58 | +Please read [General setup](#General-setup) to get some of the setup requirements. |
60 | 59 |
|
61 | 60 | You also have to give a value for the following properties contained in the [maven-bench.properties](src/test/resources/maven-bench.properties) file: |
62 | | -* maven.version.from |
63 | | -* maven.version.to |
64 | | -* warmup.number |
65 | | -* measures.number-by-maven-version |
| 61 | +* `maven.version.from` |
| 62 | +* `maven.version.to` |
| 63 | +* `warmup.number` |
| 64 | +* `measures.number-by-maven-version` |
66 | 65 |
|
67 | 66 | The meaning of these properties is given in the [maven-bench.properties](src/test/resources/maven-bench.properties) file. |
68 | 67 |
|
69 | 68 | Measures can be launched with this command line: ```mvn -Dtest=MvnValidateAllocationByMaven3VersionTest test```. |
70 | 69 | Before doing it, you can close your IDE, web browser or other applications to free memory. |
71 | 70 |
|
72 | | -The benchmark results are exported into a *maven-memory-allocation-{date-time}.csv* file. The execution context (processor, OS, ...) is reported in an *execution-context-{date-time}.txt* file. |
| 71 | +The benchmark results are exported into a `maven-memory-allocation-{date-time}.csv` file. The execution context (processor, OS, ...) is reported in an `execution-context-{date-time}.txt` file. |
73 | 72 |
|
74 | | -For several Maven versions, the following graphic gives the average of ten heap allocations caused by the application of *mvn validate* on Apache Camel: |
| 73 | +For several Maven versions, the following graph gives the average of ten heap allocations caused by the application of `mvn validate` on Apache Camel: |
75 | 74 | <p align="center"> |
76 | 75 | <img src="measures/mvn-validate-on-camel.png"> |
77 | 76 | </p> |
78 | 77 |
|
79 | | -For this graphic you can consult: |
| 78 | +For this graph, you can consult: |
80 | 79 | * [the measures](measures/maven-memory-allocation-2019-09-01-18-48-41.csv) |
81 | 80 | * [the execution context](measures/execution-context-2019-09-01-18-48-41.txt) |
82 | 81 |
|
83 | | -Measures took 1 hour and 12 minutes. |
| 82 | +Measures took 1 hour and 12 minutes. From Maven versions 3.2.5 to 3.6.2, heap allocation level is the highest with Maven 3.2.5 and the smallest with Maven 3.6.2. And __the heap allocation decreases from ~7 GB with Maven 3.6.1 to ~3 GB with Maven 3.6.2__. |
| 83 | + |
| 84 | +JVM heap size can be fixed with the help of [@HeapSize](https://github.com/quick-perf/doc/wiki/JVM-annotations#heapsize): |
| 85 | +* with Maven 3.2.5 and a JVM heap size between 6 and 9 GB, one measure of heap allocation takes around one minute: the test duration is about 1.5 minute with a 5 GB heap size, probably due to more garbage collection, |
| 86 | +* __with Maven 3.6.2, the test duration drops to 15 seconds for a heap size between 1 and 9 GB__ |
84 | 87 |
|
85 | | -In these measures, the heap allocation decreases from ~7 Go to ~3 Go between Maven 3.6.1 to Maven 3.6.2. |
| 88 | +Less heap allocation means you can allocate less memory, and given a memory allocation you get less garbage collection, then it takes less time: with QuickPerf, we were able to measure the improvement precisely. |
86 | 89 |
|
87 | 90 | # Investigate where heap allocation comes from |
88 | 91 |
|
89 | | -You can use *MvnValidateProfilingTest* to understand the origin of heap allocation. |
90 | | -Some of the set up requirements can be found in [General set up](#General-set-up) part. |
| 92 | +You can use `MvnValidateProfilingTest` to understand the origin of heap allocation. |
| 93 | +Some of the set up requirements can be found in [General setup](#General-setup) part. |
91 | 94 |
|
92 | | -The Maven version under test can be set with the MAVEN_3_VERSION constant: |
| 95 | +The Maven version under test can be set with the `MAVEN_3_VERSION` constant: |
93 | 96 | ``` java |
94 | 97 | public static Maven3Version MAVEN_3_VERSION = Maven3Version.V_3_6_2; |
95 | 98 | ``` |
|
0 commit comments