-
Notifications
You must be signed in to change notification settings - Fork 682
Description
What happened?
The problem
Arithmetic operations fail when a numeric or string literal is on the LEFT side and a Deferred column reference (ibis._['column']) is on the RIGHT side:
It works for Boolean literals, as demonstrated below.
Illustrative example
# Just an illustrative example - not working, just easy to understand!
lit = ibis.literal(5)
col = ibis._['value']
# This works ✅
col + lit
# This fails ❌
lit + col
# InputTypeError: Unable to infer datatype of value _['value']
# with type <class 'ibis.common.deferred.Deferred'>Working Reproduction of the bug
import ibis
import pandas as pd
# Setup backends
polars_backend = ibis.polars.connect()
duckdb_backend = ibis.duckdb.connect()
sqlite_backend = ibis.sqlite.connect()
# Create test data
pl_df = pd.DataFrame({'value': [10]})
# Create tables
df_polars = polars_backend.create_table('test', pl_df, overwrite=True)
df_duckdb = duckdb_backend.create_table('test', pl_df, overwrite=True)
df_sqlite = sqlite_backend.create_table('test', pl_df, overwrite=True)
# Define expressions using Deferred syntax
lit = ibis.literal(5)
col = ibis._['value'] # Deferred column reference
# Test each backend
results = []
for backend_name, table in [
('polars', df_polars),
('duckdb', df_duckdb),
('sqlite', df_sqlite),
]:
result = {'backend': backend_name}
# Test: col + lit (should work)
try:
expr = col + lit
res = table.select(expr.name('result')).execute()
result['col+lit'] = res.result[0]
except Exception as e:
result['col+lit'] = f'{type(e).__name__}'
# Test: lit + col (fails with InputTypeError)
try:
expr = lit + col
res = table.select(expr.name('result')).execute()
result['lit+col'] = res.result[0]
except Exception as e:
result['lit+col'] = f'{type(e).__name__}'
results.append(result)
# Display results
df_results = pd.DataFrame(results)
print(df_results.to_markdown(index=False))Output
| backend | col+lit | lit+col |
|---|---|---|
| polars | 15 | InputTypeError |
| duckdb | 15 | InputTypeError |
| sqlite | 15 | InputTypeError |
Affected Scalar Types and Operations
IntegerScalar & FloatingScalar (BROKEN):
- Addition:
literal(5) + ibis._['x']→ ❌ InputTypeError - Subtraction:
literal(100) - ibis._['x']→ ❌ InputTypeError - Multiplication:
literal(2) * ibis._['x']→ ❌ InputTypeError - Division:
literal(100) / ibis._['x']→ ❌ InputTypeError - Modulo:
literal(100) % ibis._['x']→ ❌ InputTypeError - Power:
literal(2) ** ibis._['x']→ ❌ InputTypeError - Floor division:
literal(100) // ibis._['x']→ ❌ InputTypeError
BooleanScalar (ALREADY WORKS):
- AND:
literal(True) & ibis._['x']→ ✅ Works correctly - OR:
literal(False) | ibis._['x']→ ✅ Works correctly
StringScalar (BROKEN - different error):
- Concatenation:
literal('x') + ibis._['y']→ ❌ SignatureValidationError (not InputTypeError)
Impact
This breaks expression building frameworks that generate expressions programmatically, where operand order may not be predictable. It also violates Python's numeric type behavior where a + b and b + a should both work.
Bug location and suggested fixes
ibis/ibis/expr/types/generic.py
Line 3168 in 484776f
| except (ValidationError, NotImplementedError): |
The error occurs for float and integer literals on the LHS of arithmetic operators.
A similar issue occurs for string literals, but requires a different fix.
Boolean literals work as they return a SignatureValidationError which is a subclass of ValidationError.
Fix for Integer and String Literals
This fix would align with the principles stated in the comments for _binop:
"returns NotImplemented if we aren't sure:"
File: ibis/expr/types/core.py
Function: _binop
Change: Add InputTypeError to the exception tuple (one line)
Current code:
except (ValidationError, NotImplementedError):
return NotImplementedSuggested fix:
except (ValidationError, NotImplementedError, InputTypeError):
return NotImplementedWhy StringScalar still fails (different reason):**
StringValue uses a completely different code path that bypasses _binop:
# NumericValue.__add__ (goes through _binop)
def __add__(self, other):
return _binop(ops.Add, self, other) # ✅ Will be fixed
# BooleanValue.__and__ (goes through _binop)
def __and__(self, other):
return _binop(ops.And, self, other) # ✅ Already works
# StringValue.__add__ (BYPASSES _binop!)
def __add__(self, other):
return self.concat(other) # ❌ Won't be fixed by this change
def concat(self, other, *args):
return ops.StringConcat((self, other, *args)).to_expr() # No _binop!Scripts
See attached scripts (written by Claude) identifying:
- cases where the errors occur
- code paths taken by each error
These highlight why integer and float arithmetic ops will be fixed by this change, but string ops will require a different fix ( that I haven't yet identified! )
ibis_reverse_operator_comprehensive_repro.py
ibis_reverse_operator_code_path_analysis.py
What version of ibis are you using?
11.0.0
What backend(s) are you using, if any?
Error is consistent on Polars, SQLite and Duckdb.
Relevant log output
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
Type
Projects
Status