Skip to content

Commit cacf243

Browse files
authored
xet-uploads: re-compile xet wasm with fix for compute_file_hash function (#1736)
Updates the wasm used by the xet upload portion of the library to include latest fixes to compute_file_hash function to process large files correctly. Also update the build-xet-wasm.sh script to (either can be reverted just comment below) - work on macos (base64 command requires `-i` flag on macos) - adds to the comment at the top of the `chunker_wasm_bg.wasm.base64.ts` to include some information about when the file was created and with what parameters. WASM binary is 99KB
1 parent 402c323 commit cacf243

File tree

2 files changed

+1377
-1675
lines changed

2 files changed

+1377
-1675
lines changed

packages/hub/scripts/build-xet-wasm.sh

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -222,15 +222,31 @@ if [[ -n "$OUTPUT_DIR" ]]; then
222222
log "Output copied to: $OUTPUT_DIR"
223223
fi
224224

225+
# Detect operating system for base64 command compatibility
226+
OS=$(uname)
227+
if [[ "$OS" == "Darwin" ]]; then
228+
BASE64_CMD="base64 -i"
229+
log "Detected macOS - using base64 -i flag"
230+
else
231+
BASE64_CMD="base64"
232+
log "Detected Linux/Unix - using standard base64"
233+
fi
234+
225235
# copy the generated hf_xet_thin_wasm_bg.js to the hub package and hf_xet_thin_wasm_bg.wasm to the hub package
226236
cp "$CLONE_DIR/$PACKAGE/pkg/hf_xet_thin_wasm_bg.js" "./src/vendor/xet-chunk/chunker_wasm_bg.js"
227237
cp "$CLONE_DIR/$PACKAGE/pkg/hf_xet_thin_wasm_bg.wasm.d.ts" "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.d.ts"
228-
cat << 'EOF' > "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.base64.ts"
238+
cat << EOF > "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.base64.ts"
229239
// Generated by build-xet-wasm.sh
240+
// Repository: $REPO_URL
241+
// Branch: $BRANCH
242+
// Package: $PACKAGE
243+
// JS Target: $JS_TARGET
244+
// Build Date: $(date -u +"%Y-%m-%d %H:%M:%S UTC")
245+
230246
export const wasmBase64 = atob(
231-
`
247+
\`
232248
EOF
233-
base64 "$CLONE_DIR/$PACKAGE/pkg/hf_xet_thin_wasm_bg.wasm" | fold -w 100 >> "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.base64.ts"
249+
$BASE64_CMD "$CLONE_DIR/$PACKAGE/pkg/hf_xet_thin_wasm_bg.wasm" | fold -w 100 >> "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.base64.ts"
234250
cat << 'EOF' >> "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.base64.ts"
235251
`
236252
.trim()

0 commit comments

Comments
 (0)