Add support for UV dependency management (#2601)

Closes #2553

# Rationale for this change

During the release process cleanup, I also had prototyped a migration
from Poetry to UV. I used the
https://github.com/mkniewallner/migrate-to-uv package to migrate to UV
and ran a release workflow and some others like doc serve for
validation.

This PR also migrates the Makefile to use UV. I'll still need to upgrade
the docs to reflect the UV contribution workflow. I think it would be a
good start to push up as a prototype, and have others checkout and give
it a try.

## Build System Changes

### Build Backend
UV doesn't have a Poetry-equivalent build backend with build hooks, so
I've switched to __setuptools__ with native Cython support for the avro
changes.

```
[build-system]
requires = ["setuptools>=80", "wheel", "Cython>=3.0.0"]
build-backend = "setuptools.build_meta"
```

### Packaging for release

__PEP 639 License Compliance__

By going to the test above the license is now properly included in the
release metadata

```toml
[project]
license = "Apache-2.0"  # SPDX identifier
license-files = ["LICEN[CS]E*", "NOTICE*"]  # Automatic inclusion
```



### Build Verification

__Release builds verified:__

```bash
uv build --sdist  # Source distribution with correct metadata
uv build --wheel  # Wheel with compiled Cython extensions
check-wheel-contents dist/  # Quality checks pass
```

__Wheel vs Poetry:__

The migration addresses a packaging issue where Poetry was building
non-standard "fat" wheels. Binary comparison using diff revealed
significant differences in wheel structure between Poetry and UV builds.

```
diff -rq uv-wheel poetry-wheel

Only in poetry-wheel/pyiceberg/avro: decoder_basic.c <-- built into .so so not needed :)
Only in poetry-wheel/pyiceberg/avro: decoder_fast.cpython-310-darwin.so
Only in poetry-wheel/pyiceberg/avro: decoder_fast.cpython-311-darwin.so
Only in poetry-wheel: NOTICE
Files uv-wheel/pyiceberg/avro/decoder_fast.cpython-312-darwin.so and poetry-wheel/pyiceberg/avro/decoder_fast.cpython-312-darwin.so differ
```
Poetry's wheels tagged for a specific Python version (e.g.,
`cp312-cp312`) contained compiled extensions for multiple Python
versions simultaneously. The comparison shows that a Python 3.12 wheel
included `decoder_fast.cpython-310-darwin.so` and
`decoder_fast.cpython-311-darwin.so` alongside the correct binary. These
wheels also included source files and placed NOTICE at the wheel root
instead of `dist-info/licenses/`.

The new build process with `cibuildwheel` creates standard
PyPI-compliant wheels where each wheel contains only its corresponding
Python version's binary. This reduces wheel size while ensuring proper
PEP 639 license file placement in `dist-info/licenses/`.

Test builds are deployed also can be checked against the repo above. 

## Are these changes tested?

Yes

- [x] `make install` Environment setup and dependency installation
- [x] `make test` - Unit tests  
- [x] `make test-integration` - Integration tests with all extras
- [x] `make lint` - Linting and code quality checks
- [x] `make doc-install && make doc-serve` - Documentation 

## Are there any user-facing changes?

No user facing changes. Contributors will need to use UV instead of
Poetry for development, but all make targets remain the same. So not
really?

---------

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>

Author: geruh

.github/dependabot.yml

@@ -19,7 +19,7 @@
 
 version: 2
 updates:
-  - package-ecosystem: "pip"
+  - package-ecosystem: "uv"
     directory: "/"
     schedule:
       interval: "weekly"

.github/workflows/nightly-pypi-build.yml

@@ -39,13 +39,13 @@ jobs:
         with:
           python-version: 3.12
 
-      - name: Install Poetry
-        run: make install-poetry
+      - name: Install UV
+        uses: astral-sh/setup-uv@v6
 
       - name: Set version
         id: set-version
         run: |
-          CURRENT_VERSION=$(poetry version --short)
+          CURRENT_VERSION=$(uv version --short)
           TIMESTAMP=$(date +%Y%m%d%H%M%S)
           echo "VERSION=${CURRENT_VERSION}.dev${TIMESTAMP}" >> "$GITHUB_OUTPUT"
 

.github/workflows/pypi-build-artifacts.yml

@@ -46,18 +46,18 @@ jobs:
             3.11
             3.12
 
-      - name: Install poetry
-        run: make install-poetry
+      - name: Install UV
+        uses: astral-sh/setup-uv@v6
 
       - name: Set version with RC
         env:
           VERSION: ${{ inputs.VERSION }}
-        run: python -m poetry version "${{ env.VERSION }}"
+        run: uv version "${{ env.VERSION }}"
 
       # Publish the source distribution with the version that's in
       # the repository, otherwise the tests will fail
       - name: Compile source distribution
-        run: python3 -m poetry build --format=sdist
+        run: uv build --sdist
         if: startsWith(matrix.os, 'ubuntu')
 
       - name: Build wheels
@@ -77,6 +77,7 @@ jobs:
           # Skip free-threaded (PEP 703) builds until we evaluate decoder_fast support
           CIBW_SKIP: "cp3*t-*"
 
+
       - name: Add source distribution
         if: startsWith(matrix.os, 'ubuntu')
         run: ls -lah dist/* && cp dist/* wheelhouse/

.github/workflows/python-ci-docs.yml

@@ -39,8 +39,8 @@ jobs:
       - uses: actions/setup-python@v6
         with:
           python-version: 3.12
-      - name: Install poetry
-        run: make install-poetry
+      - name: Install UV
+        uses: astral-sh/setup-uv@v6
       - name: Install
         run: make docs-install
       - name: Build docs

.github/workflows/python-ci.yml

@@ -54,8 +54,8 @@ jobs:
     - uses: actions/setup-python@v6
       with:
         python-version: ${{ matrix.python }}
-    - name: Install poetry
-      run: make install-poetry
+    - name: Install UV
+      uses: astral-sh/setup-uv@v6
     - name: Install system dependencies
       run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
     - name: Install

.github/workflows/python-release-docs.yml

@@ -34,8 +34,8 @@ jobs:
       - uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python }}
-      - name: Install poetry
-        run: make install-poetry
+      - name: Install UV
+        uses: astral-sh/setup-uv@v6
       - name: Install docs
         run: make docs-install
       - name: Build docs

.github/workflows/python-release.yml

@@ -97,20 +97,20 @@ jobs:
         with:
           python-version: 3.12
 
-      - name: Install Poetry
-        run: make install-poetry
+      - name: Install UV
+        uses: astral-sh/setup-uv@v6
 
       - name: Validate current pyiceberg version
         env:
           VERSION: ${{ needs.validate-inputs.outputs.VERSION }}
         run: |
-          # Extract the current version from Poetry
-          current_pyiceberg_version=$(poetry version --short)
-          echo "Detected Poetry version: $current_pyiceberg_version"
+          # Extract the current version from UV
+          current_pyiceberg_version=$(uv version --short)
+          echo "Detected UV version: $current_pyiceberg_version"
 
-          # Compare the input version with the Poetry version
+          # Compare the input version with the UV version
           if [[ "$VERSION" != "$current_pyiceberg_version" ]]; then
-            echo "Error: Input version ($VERSION) does not match the Poetry version ($current_pyiceberg_version)"
+            echo "Error: Input version ($VERSION) does not match the UV version ($current_pyiceberg_version)"
             exit 1
           fi
 

.github/workflows/svn-build-artifacts.yml

@@ -46,13 +46,13 @@ jobs:
             3.11
             3.12
 
-      - name: Install poetry
-        run: make install-poetry
+      - name: Install UV
+        uses: astral-sh/setup-uv@v6
 
       # Publish the source distribution with the version that's in
       # the repository, otherwise the tests will fail
       - name: Compile source distribution
-        run: python3 -m poetry build --format=sdist
+        run: uv build --sdist
         if: startsWith(matrix.os, 'ubuntu')
 
       - name: Build wheels

MANIFEST.in

@@ -15,4 +15,22 @@
 # specific language governing permissions and limitations
 # under the License.
 
-graft src
+# Include Cython source files for building from source
+recursive-include pyiceberg *.pyx *.c
+
+# Exclude generated Cython C file
+exclude pyiceberg/avro/decoder_fast.c
+
+# Include test files in sdist
+recursive-include tests *
+
+# Include development files
+include Makefile
+recursive-include dev *
+
+# Exclude build artifacts
+global-exclude */__pycache__/*
+prune .venv
+prune build
+prune dist
+prune .pytest_cache

Makefile

@@ -18,20 +18,23 @@
 # Configuration Variables
 # ========================
 
+PYTHON ?=  # Override with e.g. PYTHON=3.11 to use specific Python version
 PYTEST_ARGS ?= -v -x  # Override with e.g. PYTEST_ARGS="-vv --tb=short"
 COVERAGE ?= 0      # Set COVERAGE=1 to enable coverage: make test COVERAGE=1
 COVERAGE_FAIL_UNDER ?= 85  # Minimum coverage % to pass: make coverage-report COVERAGE_FAIL_UNDER=70
 KEEP_COMPOSE ?= 0  # Set KEEP_COMPOSE=1 to keep containers after integration tests
 
-PIP = python -m pip
-
-POETRY_VERSION = 2.2.1
-POETRY = python -m poetry
+# Set Python argument for uv commands if PYTHON is specified
+ifneq ($(PYTHON),)
+  PYTHON_ARG = --python $(PYTHON)
+else
+  PYTHON_ARG =
+endif
 
 ifeq ($(COVERAGE),1)
-  TEST_RUNNER = $(POETRY) run coverage run --parallel-mode --source=pyiceberg -m
+  TEST_RUNNER = uv run python -m coverage run --parallel-mode --source=pyiceberg -m
 else
-  TEST_RUNNER = $(POETRY) run
+  TEST_RUNNER = uv run python -m
 endif
 
 ifeq ($(KEEP_COMPOSE),1)
@@ -55,24 +58,21 @@ help: ## Display this help message
 
 ##@ Setup
 
-install-poetry: ## Ensure Poetry is installed at the specified version
-	@if ! command -v ${POETRY} &> /dev/null; then \
-		echo "Poetry not found. Installing..."; \
-		${PIP} install poetry==$(POETRY_VERSION); \
+install-uv: ## Ensure uv is installed
+	@if ! command -v uv &> /dev/null; then \
+		echo "uv not found. Installing..."; \
+		curl -LsSf https://astral.sh/uv/install.sh | sh; \
 	else \
-		INSTALLED_VERSION=$$(${PIP} show poetry | grep Version | awk '{print $$2}'); \
-		if [ "$$INSTALLED_VERSION" != "$(POETRY_VERSION)" ]; then \
-			echo "Updating Poetry to version $(POETRY_VERSION)..."; \
-			${PIP} install --upgrade poetry==$(POETRY_VERSION); \
-		else \
-			echo "Poetry version $(POETRY_VERSION) already installed."; \
-		fi; \
+		echo "uv is already installed."; \
 	fi
 
-install-dependencies: ## Install all dependencies including extras
-	$(POETRY) install --all-extras
+setup-venv: ## Create virtual environment
+	uv venv $(PYTHON_ARG)
+
+install-dependencies: setup-venv ## Install all dependencies including extras
+	uv sync --all-extras
 
-install: install-poetry install-dependencies ## Install Poetry and dependencies
+install: install-uv install-dependencies ## Install uv and dependencies
 
 # ===============
 # Code Validation
@@ -84,7 +84,7 @@ check-license: ## Check license headers
 	./dev/check-license
 
 lint: ## Run code linters via prek (pre-commit hooks)
-	$(POETRY) run prek run -a
+	uv run prek run -a
 
 # ===============
 # Testing Section
@@ -101,7 +101,7 @@ test-integration-setup: ## Start Docker services for integration tests
 	docker compose -f dev/docker-compose-integration.yml kill
 	docker compose -f dev/docker-compose-integration.yml rm -f
 	docker compose -f dev/docker-compose-integration.yml up -d --wait
-	$(POETRY) run python dev/provision.py
+	uv run python dev/provision.py
 
 test-integration-exec: ## Run integration tests (excluding provision)
 	$(TEST_RUNNER) pytest tests/ -m integration $(PYTEST_ARGS)
@@ -133,25 +133,25 @@ test-coverage: COVERAGE=1
 test-coverage: test test-integration test-s3 test-adls test-gcs coverage-report ## Run all tests with coverage and report
 
 coverage-report: ## Combine and report coverage
-	${POETRY} run coverage combine
-	${POETRY} run coverage report -m --fail-under=$(COVERAGE_FAIL_UNDER)
-	${POETRY} run coverage html
-	${POETRY} run coverage xml
+	uv run coverage combine
+	uv run coverage report -m --fail-under=$(COVERAGE_FAIL_UNDER)
+	uv run coverage html
+	uv run coverage xml
 
 # ================
 # Documentation
 # ================
 
 ##@ Documentation
 
-docs-install: ## Install docs dependencies
-	${POETRY} install --with docs
+docs-install: ## Install docs dependencies (included in default groups)
+	uv sync --group docs
 
 docs-serve: ## Serve local docs preview (hot reload)
-	${POETRY} run mkdocs serve -f mkdocs/mkdocs.yml
+	uv run mkdocs serve -f mkdocs/mkdocs.yml
 
 docs-build: ## Build the static documentation site
-	${POETRY} run mkdocs build -f mkdocs/mkdocs.yml --strict
+	uv run mkdocs build -f mkdocs/mkdocs.yml --strict
 
 # ===================
 # Project Maintenance
@@ -161,7 +161,7 @@ docs-build: ## Build the static documentation site
 
 clean: ## Remove build artifacts and caches
 	@echo "Cleaning up Cython and Python cached files..."
-	@rm -rf build dist *.egg-info
+	@rm -rf build dist *.egg-info .venv
 	@find . -name "*.so" -exec echo Deleting {} \; -delete
 	@find . -name "*.pyc" -exec echo Deleting {} \; -delete
 	@find . -name "__pycache__" -exec echo Deleting {} \; -exec rm -rf {} +