Skip to content

Commit 7faf271

Browse files
FrancoGiachettaFrancoGiachetta
authored
add workflow to measure execute time (#789)
* add workflow for linux * add hyperfine and comments * add runs * fix run * fix run * fix run * fix run * fix command * show errors * change how handles project building * typo in command * forgot issue-comment-id in update comment * better result printing * change approach * versions fix * build binaries correctly * require binaries to run benches * typo * remove unwanted programs * fix run command * checkout base before running * checkout base before running * fix run hyperfine command * fix run hyperfine command * typos * fix OUTPUT_DIR env var * fix ROOT_DIR var * forgot to make the output's dir * fix * fix hyperfine command * fix * change ouput's dir * change ouput's dir * fix * check with one branch * check with other branch * remove checkout * use checkout before running bench * use onw branh * create cache againt * clean unwanted dir * check errors * change command * remove unwanted dir * cache program to run * fix caching * fix caching * forgot to add env var * typo * typo * typo * checkout first * revert * on pull_request all branches * change from base to head * show error * make deps * run both branches * run both branches * typo * make corrections * remove unwanted dirs * fix bench command * fix bench command * add warmup * add warmup * change hyperfine command * showoutput * add runtime library * cache hits * fix typo * fix typo * remove conditional temporarily * typos * debug * debug * debug * debug * typo * enable shell * debug * debug * debug * debug * typo * typo * typo * change cache files name * cache everything again * cache everything again * debug * debug * debug * debug * debug * debug * add head variant * add head variant * add head variant * add more programs * add more programs * add more programs * typo * better title * better title
1 parent 0eb9d45 commit 7faf271

File tree

2 files changed

+168
-0
lines changed

2 files changed

+168
-0
lines changed

‎.github/workflows/bench-hyperfine.yml‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,3 +86,169 @@ jobs:
8686
issue-number: ${{ github.event.pull_request.number }}
8787
body-path: bench-hyperfine.md
8888
edit-mode: replace
89+
90+
build-binaries:
91+
env:
92+
MLIR_SYS_180_PREFIX: /usr/lib/llvm-18/
93+
LLVM_SYS_181_PREFIX: /usr/lib/llvm-18/
94+
TABLEGEN_180_PREFIX: /usr/lib/llvm-18/
95+
RUST_LOG: cairo_native=debug,cairo_native_test=debug
96+
strategy:
97+
matrix:
98+
branch: [ base, head ]
99+
name: Build cairo-native-run for ${{ matrix.branch }}
100+
runs-on: ubuntu-latest
101+
steps:
102+
- name: Cache binary
103+
uses: actions/cache@v3
104+
id: cache-binary
105+
with:
106+
path: bin/cairo-native-run-${{ matrix.branch }}
107+
key: binary-${{ github.event.pull_request[matrix.branch].sha }}
108+
109+
- name: Cache runtime-library
110+
uses: actions/cache@v3
111+
id: cache-library
112+
with:
113+
path: lib/libcairo-native-runtime-${{ matrix.branch }}.a
114+
key: library-${{ github.event.pull_request[matrix.branch].sha }}
115+
116+
- name: check and free hdd space left
117+
if: ${{ steps.cache-binary.outputs.cache-hit != 'true' }}
118+
run: |
119+
echo "Listing 20 largest packages"
120+
dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 20
121+
df -h
122+
sudo apt-get update
123+
sudo apt-get remove -y '^llvm-.*'
124+
sudo apt-get remove -y 'php.*'
125+
sudo apt-get remove -y '^dotnet-.*'
126+
sudo apt-get remove -y '^temurin-.*'
127+
sudo apt-get remove -y azure-cli google-cloud-cli microsoft-edge-stable google-chrome-stable firefox powershell mono-devel
128+
sudo apt-get autoremove -y
129+
sudo apt-get clean
130+
df -h
131+
echo "Removing large directories"
132+
# deleting 15GB
133+
sudo rm -rf /usr/share/dotnet/
134+
sudo rm -rf /usr/local/lib/android
135+
df -h
136+
137+
- name: Install Rust
138+
if: ${{ steps.cache-binary.outputs.cache-hit != 'true' || steps.cache-library.outputs.cache-hit != 'true' }}
139+
uses: dtolnay/rust-toolchain@1.81.0
140+
141+
- name: add llvm deb repository
142+
if: ${{ steps.cache-binary.outputs.cache-hit != 'true' || steps.cache-library.outputs.cache-hit != 'true' }}
143+
uses: myci-actions/add-deb-repo@11
144+
with:
145+
repo: deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-18 main
146+
repo-name: llvm-repo
147+
keys-asc: https://apt.llvm.org/llvm-snapshot.gpg.key
148+
149+
- name: Install LLVM
150+
if: ${{ steps.cache-binary.outputs.cache-hit != 'true' || steps.cache-library.outputs.cache-hit != 'true' }}
151+
run: sudo apt-get update && sudo apt-get install llvm-18 llvm-18-dev llvm-18-runtime clang-18 clang-tools-18 lld-18 libpolly-18-dev libmlir-18-dev mlir-18-tools
152+
153+
- name: Checkout base commit
154+
if: ${{ steps.cache-binary.outputs.cache-hit != 'true' || steps.cache-library.outputs.cache-hit != 'true' }}
155+
uses: actions/checkout@v4
156+
with:
157+
ref: ${{ github.event.pull_request[matrix.branch].sha }}
158+
159+
- name: Fetch Rust cache
160+
if: ${{ steps.cache-binary.outputs.cache-hit != 'true' || steps.cache-library.outputs.cache-hit != 'true' }}
161+
uses: Swatinem/rust-cache@v2
162+
163+
- name: Build binary
164+
if: ${{ steps.cache-binary.outputs.cache-hit != 'true' || steps.cache-library.outputs.cache-hit != 'true' }}
165+
run: |
166+
make build
167+
mkdir lib bin
168+
cp target/release/cairo-native-run bin/cairo-native-run-${{ matrix.branch }}
169+
cp target/release/libcairo_native_runtime.a lib/libcairo-native-runtime-${{ matrix.branch }}.a
170+
171+
hyperfine-prs:
172+
name: Bench PR (linux, amd64)
173+
needs: [ build-binaries ]
174+
runs-on: ubuntu-latest
175+
env:
176+
PROGRAM: fib_2M
177+
OUTPUT_DIR: bench-outputs
178+
steps:
179+
- uses: actions/checkout@v4
180+
181+
- name: Install Hyperfine
182+
uses: taiki-e/install-action@v2
183+
with:
184+
tool: hyperfine@1.16
185+
186+
- name: Fetch base binary
187+
uses: actions/cache/restore@v3
188+
with:
189+
path: bin/cairo-native-run-base
190+
key: binary-${{ github.event.pull_request.base.sha }}
191+
192+
- name: Fetch HEAD binary
193+
uses: actions/cache/restore@v3
194+
with:
195+
path: bin/cairo-native-run-head
196+
key: binary-${{ github.event.pull_request.head.sha }}
197+
198+
- name: Fetch base runtime-library
199+
uses: actions/cache/restore@v3
200+
with:
201+
path: lib/libcairo-native-runtime-base.a
202+
key: library-${{ github.event.pull_request.base.sha }}
203+
204+
- name: Fetch head runtime-library
205+
uses: actions/cache/restore@v3
206+
with:
207+
path: lib/libcairo-native-runtime-head.a
208+
key: library-${{ github.event.pull_request.head.sha }}
209+
210+
- name: Build Dependencies
211+
run: make deps
212+
213+
- name: Run Program
214+
run: |
215+
mkdir $OUTPUT_DIR
216+
cp programs/benches/*.cairo $OUTPUT_DIR
217+
for f in $OUTPUT_DIR/*.cairo; do
218+
hyperfine \
219+
--export-markdown "${f%.*}.md" \
220+
--export-json "${f%.*}.json" \
221+
--warmup 3 \
222+
-n "head $(basename $f) (JIT)" "./bin/cairo-native-run-head --run-mode=jit -s $f --opt-level 3 --available-gas 18446744073709551615" \
223+
-n "base $(basename $f) (JIT)" "./bin/cairo-native-run-base --run-mode=jit -s $f --opt-level 3 --available-gas 18446744073709551615" \
224+
-n "head $(basename $f) (AOT)" "CAIRO_NATIVE_RUNTIME_LIBRARY=lib/libcairo-native-runtime-head.a ./bin/cairo-native-run-head --run-mode=aot -s $f --opt-level 3 --available-gas 18446744073709551615" \
225+
-n "base $(basename $f) (AOT)" "CAIRO_NATIVE_RUNTIME_LIBRARY=lib/libcairo-native-runtime-base.a ./bin/cairo-native-run-base --run-mode=aot -s $f --opt-level 3 --available-gas 18446744073709551615" \
226+
>> /dev/stderr
227+
done
228+
- name: Print tables
229+
run: |
230+
{
231+
echo "Benchmark results Main vs HEAD"
232+
for f in $OUTPUT_DIR/*.md; do
233+
echo
234+
cat $f
235+
done
236+
} | tee -a comment_body.md
237+
238+
- name: Find Bench Comment
239+
continue-on-error: true
240+
uses: peter-evans/find-comment@v3
241+
id: fc
242+
with:
243+
issue-number: ${{ github.event.pull_request.number }}
244+
comment-author: "github-actions[bot]"
245+
body-includes: Benchmarking
246+
247+
- name: Create or update bench comment
248+
continue-on-error: true
249+
uses: peter-evans/create-or-update-comment@v4
250+
with:
251+
comment-id: ${{ steps.fc.outputs.comment-id }}
252+
issue-number: ${{ github.event.pull_request.number }}
253+
body-path: comment_body.md
254+
edit-mode: replace

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,3 +34,5 @@ cairo-*.tar
3434

3535
*.ipynb
3636
cairo-native-stress-logs.jsonl
37+
38+
.DS_Store

0 commit comments

Comments
 (0)