Skip to content

Commit b2b45de

Browse files
kevinjqliusumanth-manchala
authored andcommitted
test: pyiceberg-core, pin datafusion and unpin pyarrow (apache#1648)
## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Follow up to apache#1635 We dont need to pin pyarrow specifically, just grab it from pyiceberg. Revert datafusion version threshold to 45. Version 45 is the lowest version which will still work. The underlying issue for apache#1635 is with the **newer** versions of datafusion, mainly datafusion 49 due to a breaking change described in apache#1647 So pyiceberg-core works with datafusion (python) library 45, 46, 47, and 48. just not 49 I verified locally with all the versions above: ``` hatch env prune && hatch run dev:test ``` i double checked. `insert_into` is available in datafusion 45. apache/datafusion@7f9a8c0 ## Are these changes tested?
1 parent bd228fe commit b2b45de

File tree

3 files changed

+5
-3
lines changed

3 files changed

+5
-3
lines changed

bindings/python/LICENSE

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../LICENSE

bindings/python/pyproject.toml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ name = "pyiceberg-core"
3434
readme = "project-description.md"
3535
requires-python = "~=3.9"
3636
dynamic = ["version"]
37+
license = { file = "LICENSE" }
3738

3839
[tool.maturin]
3940
features = ["pyo3/extension-module"]
@@ -44,7 +45,7 @@ python-source = "python"
4445
ignore = ["F403", "F405"]
4546

4647
[tool.hatch.envs.dev]
47-
dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "pyarrow==16.*", "datafusion==47.*", "pyiceberg[sql-sqlite]>=0.9.1"]
48+
dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==45.*", "pyiceberg[sql-sqlite,pyarrow]>=0.9.1"]
4849

4950
[tool.hatch.envs.dev.scripts]
5051
build = "maturin build --out dist --sdist"

bindings/python/tests/test_datafusion_table_provider.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,8 @@
2727
import datafusion
2828

2929
assert (
30-
datafusion.__version__ >= "47"
31-
) # iceberg table provider only works for datafusion >= 47
30+
datafusion.__version__ >= "45"
31+
) # iceberg table provider only works for datafusion >= 45
3232

3333

3434
@pytest.fixture(scope="session")

0 commit comments

Comments
 (0)