Skip to content
Open
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
8a0e6df
feat: Add support for decimal.Decimal object
RaphDal Oct 13, 2025
4826301
feat: add support to arrow's decimals
RaphDal Oct 15, 2025
7aae3a8
tests: add serialization tests for decimals
RaphDal Oct 16, 2025
856c809
feat: uses binary format for decimal object
RaphDal Oct 16, 2025
5772e0e
fix: correct enum value for col_source_decimal_pyobj
RaphDal Oct 17, 2025
1d4c69f
Merge branch 'main' into rd_decimal
RaphDal Oct 20, 2025
b62520a
fix: hold GIL when working with py_obj
RaphDal Nov 7, 2025
5d7ee5e
tests: fix test to work with new protocol version 3
RaphDal Nov 7, 2025
0d7017f
adding print to debug failing test
RaphDal Nov 7, 2025
48e9709
Remove GIL requirement for arrow decimals
RaphDal Nov 7, 2025
5b4cbfa
fix: update decimal source enum values for compatibility
RaphDal Nov 7, 2025
c2c19d7
test: add debug print for bad protocol version handling
RaphDal Nov 7, 2025
9b00b39
refactor: remove debug print statements for protocol version handling
RaphDal Nov 7, 2025
cad427c
fix: restore conditional for expected timestamp type based on QuestDB…
RaphDal Nov 7, 2025
674e83c
tests: add decimal support tests for pyarrow and decimal types
RaphDal Nov 7, 2025
784d8b9
feat: support decimal serialization directly from column
RaphDal Nov 7, 2025
94dba61
fix: adjust byte length calculation for unscaled object in decimal co…
RaphDal Nov 7, 2025
3a57ea4
fix: add decimal repo test in ci
RaphDal Nov 7, 2025
00ff8a4
fix: correct byte length calculation for unscaled object in decimal c…
RaphDal Nov 7, 2025
6990a20
fix: update pyarrow import handling
RaphDal Nov 7, 2025
0de7bfd
fix: remove decimal branch compilation from CI pipeline
RaphDal Nov 11, 2025
ec21a54
fix: add table creation for decimal tests in system tests
RaphDal Nov 11, 2025
ac4350a
fix: update subproject commit reference for c-questdb-client
RaphDal Nov 11, 2025
6b1f263
fix: update decimal column name in table creation for system tests
RaphDal Nov 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion c-questdb-client
Submodule c-questdb-client updated 51 files
+10 −0 CMakeLists.txt
+42 −39 cpp_test/test_line_sender.cpp
+7 −1 doc/C.md
+5 −1 doc/CPP.md
+5 −2 examples/line_sender_c_example.c
+1 −0 examples/line_sender_c_example_array_byte_strides.c
+2 −1 examples/line_sender_c_example_array_c_major.c
+2 −1 examples/line_sender_c_example_array_elem_strides.c
+5 −2 examples/line_sender_c_example_auth.c
+5 −2 examples/line_sender_c_example_auth_tls.c
+129 −0 examples/line_sender_c_example_decimal_binary.c
+5 −2 examples/line_sender_c_example_from_conf.c
+4 −1 examples/line_sender_c_example_from_env.c
+4 −1 examples/line_sender_c_example_http.c
+5 −2 examples/line_sender_c_example_tls_ca.c
+3 −2 examples/line_sender_cpp_example.cpp
+1 −1 examples/line_sender_cpp_example_array_c_major.cpp
+1 −1 examples/line_sender_cpp_example_array_elem_strides.cpp
+3 −2 examples/line_sender_cpp_example_auth.cpp
+3 −2 examples/line_sender_cpp_example_auth_tls.cpp
+84 −0 examples/line_sender_cpp_example_decimal_binary.cpp
+137 −0 examples/line_sender_cpp_example_decimal_custom.cpp
+3 −2 examples/line_sender_cpp_example_from_conf.cpp
+2 −1 examples/line_sender_cpp_example_from_env.cpp
+9 −7 examples/line_sender_cpp_example_http.cpp
+3 −2 examples/line_sender_cpp_example_tls_ca.cpp
+51 −0 include/questdb/ingress/line_sender.h
+103 −602 include/questdb/ingress/line_sender.hpp
+266 −0 include/questdb/ingress/line_sender_array.hpp
+432 −0 include/questdb/ingress/line_sender_core.hpp
+282 −0 include/questdb/ingress/line_sender_decimal.hpp
+136 −0 questdb-rs-ffi/src/decimal.rs
+71 −2 questdb-rs-ffi/src/lib.rs
+21 −5 questdb-rs/Cargo.toml
+39 −9 questdb-rs/build.rs
+3 −3 questdb-rs/examples/basic.rs
+8 −2 questdb-rs/examples/http.rs
+10 −6 questdb-rs/examples/protocol_version.rs
+3 −0 questdb-rs/src/error.rs
+96 −2 questdb-rs/src/ingress/buffer.rs
+196 −0 questdb-rs/src/ingress/decimal.rs
+16 −1 questdb-rs/src/ingress/mod.md
+34 −13 questdb-rs/src/ingress/mod.rs
+1 −0 questdb-rs/src/ingress/sender/http.rs
+584 −0 questdb-rs/src/tests/decimal.rs
+2 −2 questdb-rs/src/tests/http.rs
+44 −1 questdb-rs/src/tests/interop/ilp-client-interop-test.json
+1 −0 questdb-rs/src/tests/mod.rs
+8 −5 questdb-rs/src/tests/sender.rs
+17 −2 system_test/questdb_line_sender.py
+42 −4 system_test/test.py
55 changes: 55 additions & 0 deletions src/questdb/dataframe.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,13 +93,68 @@ We need to extract:
* 64-bit floats
* UTF-8 string buffers
* Nanosecond-precision UTC unix epoch 64-bit signed int timestamps
* decimals

```python
import pandas as pd
import pyarrow as pa
import datetime as dt
```

### Decimals

Decimals aren't natively supported by pandas nor numpy, they use the `decimal.Decimal` objects.

#### Pandas

Decimals stored as Python objects in an 'object' dtype column.

```python
>>> df = pd.DataFrame({ 'decimals': [Decimal('123.456')] })
>>> df.dtypes
decimals object
dtype: object
```

#### Numpy

Similarly, numpy stores decimals as Python objects.

```python
>>> arr = numpy.array([Decimal('123.456')])
>>> arr
array([Decimal('123.456')], dtype=object)
```

#### PyArrow

PyArrow provides native decimal support with configurable precision and scale.
The data is stored in a fixed-width binary format.

```python
import pyarrow as pa
from decimal import Decimal

# Create decimal array: decimal128(precision, scale)
# precision = total digits, scale = digits after decimal point
decimal_array = pa.array(
[Decimal('123.456'), Decimal('789.012'), Decimal('-456.789'), None],
type=pa.decimal128(10, 3) # 10 total digits, 3 after decimal
)

# Use in DataFrame with ArrowDtype
df = pd.DataFrame({
'prices': pd.array(
[Decimal('123.45'), Decimal('678.90'), None],
dtype=pd.ArrowDtype(pa.decimal128(10, 2))
)
})
```

Notes:
- 4 datatypes: `decimal32`, `decimal64`, `decimal128` and `decimal256`
- Nulls are supported via Arrow's validity bitmap

### Booleans

```python
Expand Down
Loading
Loading