Skip to content

Commit bc47cbe

Browse files
Add PGO documentation section to crate configuration (#18959)
Add section explaining Profile Guided Optimization can provide up to 25% performance improvements. Includes three-stage build process instructions and tips for effective PGO usage. References issue #9507. ## Which issue does this PR close? Closes #9561 ## Rationale for this change Adds documentation for Profile Guided Optimization (PGO) as requested. PGO can provide up to 25% performance improvements for DataFusion workloads, and users need clear guidance on how to use it. ## What changes are included in this PR? - Added "Profile Guided Optimization (PGO)" section to `docs/source/user-guide/crate-configuration.md` - Three-stage build process instructions (instrumentation, profiling, recompilation) - Tips for effective PGO usage (representative workloads, multiple iterations, combining with other optimizations) - Links to Rust compiler guide and issue #9507 ## Are these changes tested? Yes. Documentation changes are validated by the CI workflow which builds the docs and checks for errors. The markdown syntax is valid and follows existing patterns. ## Are there any user-facing changes? Yes. This adds documentation that will be published on the DataFusion website under "Crate Configuration" > "Optimizing Builds". Users will find guidance on using PGO to improve performance.
1 parent 3b6e2fb commit bc47cbe

File tree

1 file changed

+30
-0
lines changed

1 file changed

+30
-0
lines changed

docs/source/user-guide/crate-configuration.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,36 @@ lto = true
9292
codegen-units = 1
9393
```
9494

95+
### Profile Guided Optimization (PGO)
96+
97+
Profile Guided Optimization can improve DataFusion performance by up to 25%. It works by compiling with instrumentation, running representative workloads to collect profile data, then recompiling with optimizations based on that data.
98+
99+
Build with instrumentation:
100+
101+
```shell
102+
RUSTFLAGS="-C profile-generate=/tmp/pgo-data" cargo build --release
103+
```
104+
105+
Run your workloads to collect profile data. Use benchmarks like TPCH or Clickbench, or your actual production queries:
106+
107+
```shell
108+
./target/release/your-datafusion-app --benchmark
109+
```
110+
111+
Rebuild using the collected profile:
112+
113+
```shell
114+
RUSTFLAGS="-C profile-use=/tmp/pgo-data" cargo build --release
115+
```
116+
117+
Tips:
118+
119+
- Use workloads that match your production patterns
120+
- Run multiple iterations during profiling for better coverage
121+
- Combine with LTO and CPU-specific optimizations for best results
122+
123+
See the [Rust compiler guide](https://rustc-dev-guide.rust-lang.org/building/optimized-build.html#profile-guided-optimization) for more details. Discussion and results in [issue #9507](https://github.com/apache/datafusion/issues/9507).
124+
95125
### Alternate Allocator: `snmalloc`
96126

97127
You can also use [snmalloc-rs](https://crates.io/crates/snmalloc-rs) crate as

0 commit comments

Comments
 (0)