Skip to content

feat: support GEOMETRY/GEOGRAPHY data types in Databricks #11733

@danielsparing

Description

@danielsparing

Is your feature request related to a problem?

Currently trying to read a Databricks table with GEOMETRY or GEOGRAPHY columns in them fails, no other columns can be read either.

(The workaround currently is to create a view in Databricks, without Ibis, without the GEO* columns, and then read that into Ibis.)

The exact error message is the following, but the solution probably would take a few more steps than just sidestepping this issue:

  • _from_sqlglot_GEOMETRY expects a typename like GEOMETRY(POINT) etc, a la Postgres, but Databricks instead uses GEOMETRY(<srid>) such as GEOMETRY(0) for undefined srid. therefore you get the error, when attempting to read a Databricks table with a GEOMETRY:
con.table('e', database='workspace.duckgeo')

# Returns:
# File c:\Users\[...]\venv\Lib\site-packages\ibis\backends\sql\datatypes.py:360, in SqlglotType._from_sqlglot_GEOMETRY(cls, arg, srid, nullable)
#     352 @classmethod
#     353 def _from_sqlglot_GEOMETRY(
#     354     cls,
#    (...)    357     nullable: bool | None = None,
#     358 ) -> sge.DataType:
#     359     if arg is not None:
# --> [360](file:///C:/Users/[...]/venv/Lib/site-packages/ibis/backends/sql/datatypes.py:360)         typeclass = _geotypes[arg.this.this]
#     361     else:
#     362         typeclass = dt.GeoSpatial

# KeyError: '0'

What is the motivation behind your request?

Being able to read these tables (given that there's already Ibis support for Geometries in Postgres), or at least gracefully ignoring these columns in the meantime.

Describe the solution you'd like

Read the columns similarly to reading Postgres geometries.

What version of ibis are you running?

11.0.0

What backend(s) are you using, if any?

Databricks

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureFeatures or general enhancements

    Type

    No type

    Projects

    Status

    backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions