Compare commits
7 Commits
v2026.09.0
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| c85aaf2ecb | |||
| acc21b8b81 | |||
| 9fbb97199f | |||
| ce5a582048 | |||
| 805fa58eb2 | |||
| 088171325d | |||
| 7ffc148112 |
44
CHANGELOG.md
44
CHANGELOG.md
@ -2,6 +2,50 @@
|
|||||||
|
|
||||||
All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440.
|
All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440.
|
||||||
|
|
||||||
|
## 2026.09.03 — A placeholder rewriter that could not see a string literal, and named row access
|
||||||
|
|
||||||
|
### Upgrade if you write string literals containing a colon
|
||||||
|
|
||||||
|
The driver rewrites `:1` placeholders to the `?` the wire protocol takes. That rewrite was a bare regular expression, and a regular expression cannot see a string literal, so it rewrote the inside of one:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
|
||||||
|
-- stored as 'http://host?/x'
|
||||||
|
```
|
||||||
|
|
||||||
|
Any `HH:MM` time, any URL with a port, any aspect ratio, any `key:value` string. It wrote wrong data and said nothing about it.
|
||||||
|
|
||||||
|
Two things made it wider than it looks. It was never opt-in: the rewrite runs whenever a statement has parameters, whatever placeholder style you actually used, so writing `?` everywhere and never touching numeric style gave no protection. And it changed the placeholder *count* while the driver still told the server there were `len(params)` of them, leaving the two disagreeing about how many binds exist.
|
||||||
|
|
||||||
|
It is now a single pass that substitutes only outside quotes and comments. The lexical rules are Informix's own, measured against all three servers rather than assumed from standard SQL, and one of them would have been got wrong from habit: a backslash escapes nothing, so `'a\'b'` is an unterminated string and draws `-282`. A scanner written to Postgres reflexes would have desynced there and corrupted everything after it. Block comments do not nest, the first `*/` closes them. Braces are a comment. `::` is stepped over as a unit so a cast can never be read as the start of a placeholder.
|
||||||
|
|
||||||
|
An unterminated quote or comment consumes the rest and substitutes nothing further. Under-substituting hands malformed SQL to the server to reject; guessing would corrupt a literal.
|
||||||
|
|
||||||
|
This was the last place the driver inferred meaning from SQL text instead of handling it properly.
|
||||||
|
|
||||||
|
### Rows that answer to a column name
|
||||||
|
|
||||||
|
Requested by a user running this alongside SQL Server, where both `pyodbc` and `mssql-python` hand back rows addressable three ways at once. The argument is readability on a wide projection: `row[11]` tells a reader nothing, and stays correct only until somebody adds a column in the middle.
|
||||||
|
|
||||||
|
```python
|
||||||
|
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||||
|
cur.execute("SELECT tabid, tabname FROM systables")
|
||||||
|
row = cur.fetchone()
|
||||||
|
row[0], row["tabname"], row.tabname
|
||||||
|
```
|
||||||
|
|
||||||
|
Set on the connection, so it is one line for an application rather than per query. A cursor can override it. Pools and the async API forward it unchanged.
|
||||||
|
|
||||||
|
`Row` subclasses `tuple`, so `row == (1, "x")` is still true and code that treats rows as sequences keeps working. Slices degrade to plain tuples, since a slice has no column map.
|
||||||
|
|
||||||
|
**It is opt-in, and here is the number.** On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` is a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript. Defaulting it on would move the published 1.05-1.15x ratio against IfxPy to roughly 1.15-1.25x. That is a reasonable trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is a choice rather than a decision made on your behalf.
|
||||||
|
|
||||||
|
Details that were measured rather than assumed. Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`, and the `.lower()` in the workaround people write by hand is already a no-op. Expression columns get server-generated names like `(count(*))` that cannot be attributes, so they are subscript-only. Duplicate names resolve to the first occurrence, matching `pyodbc`. And a column always beats a method of the same name: `tuple` defines `count` and `index`, `Row` adds `keys`, `_asdict` and `_fields`, and any column with one of those names wins, because otherwise it would hand back a bound method in silence. That reserved set is computed from the class rather than hand-listed, which is what caught it: the hand-listed version covered the two `tuple` methods and missed all three of `Row`'s own.
|
||||||
|
|
||||||
|
### Verified
|
||||||
|
|
||||||
|
**472** integration tests on 15.0.1.0.3DE and 14.10.FC7W1DE, **471** on 12.10.FC12W1DE (one skip, no common table expressions before 14.10), up from 457. Both changes have tests that fail against `2026.09.02`.
|
||||||
|
|
||||||
## 2026.09.02 — A systematic review, and eleven bugs it found
|
## 2026.09.02 — A systematic review, and eleven bugs it found
|
||||||
|
|
||||||
No new features. This is the result of going back over the driver looking for the *shape* of past bugs rather than for new symptoms, and it turned up more than expected — including two that silently returned wrong data and one where `rollback()` did nothing at all.
|
No new features. This is the result of going back over the driver looking for the *shape* of past bugs rather than for new symptoms, and it turned up more than expected — including two that silently returned wrong data and one where `rollback()` did nothing at all.
|
||||||
|
|||||||
@ -13,7 +13,7 @@
|
|||||||
</h1>
|
</h1>
|
||||||
<p class="ifx-hero__lede">
|
<p class="ifx-hero__lede">
|
||||||
Every other Informix driver wraps IBM's C SDK or the JDBC JAR. We weren't into that.
|
Every other Informix driver wraps IBM's C SDK or the JDBC JAR. We weren't into that.
|
||||||
So we read the protocol and wrote it ourselves — PEP 249, sync + async, pooled, TLS.
|
So we read the protocol and wrote it ourselves: PEP 249, sync and async, pooled, TLS.
|
||||||
Within 10% of IBM's own C driver on bulk fetches, <strong>1.6× faster</strong> on
|
Within 10% of IBM's own C driver on bulk fetches, <strong>1.6× faster</strong> on
|
||||||
bulk inserts. No compile step. No <code>LD_LIBRARY_PATH</code> ritual. No
|
bulk inserts. No compile step. No <code>LD_LIBRARY_PATH</code> ritual. No
|
||||||
<code>libcrypt.so.1</code> from 2018.
|
<code>libcrypt.so.1</code> from 2018.
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Architecture overview
|
title: Architecture overview
|
||||||
description: How the layers stack — socket, framing, codec, resultset, cursor, connection, pool.
|
description: How the layers stack, from socket through framing, codec, resultset, cursor, connection, and pool.
|
||||||
sidebar:
|
sidebar:
|
||||||
order: 2
|
order: 2
|
||||||
---
|
---
|
||||||
@ -28,13 +28,13 @@ The driver is six layers, each with a single responsibility, each testable in is
|
|||||||
|
|
||||||
The lowest layer. Wraps `socket.socket` with a connection-scoped read buffer (Phase 39). One `recv(64K)` per ~64 KB of incoming data; parsers read into the buffer via `struct.unpack_from(buf, offset)` rather than slicing copies.
|
The lowest layer. Wraps `socket.socket` with a connection-scoped read buffer (Phase 39). One `recv(64K)` per ~64 KB of incoming data; parsers read into the buffer via `struct.unpack_from(buf, offset)` rather than slicing copies.
|
||||||
|
|
||||||
Everything above this layer is `bytes` and `bytearray` arithmetic — no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
|
Everything above this layer is `bytes` and `bytearray` arithmetic, with no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
|
||||||
|
|
||||||
See [The buffered reader →](/explain/buffered-reader/) for why the buffer lives here and not on the parser.
|
See [The buffered reader →](/explain/buffered-reader/) for why the buffer lives here and not on the parser.
|
||||||
|
|
||||||
## Protocol / PDU framing
|
## Protocol / PDU framing
|
||||||
|
|
||||||
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean* — only how to read and write the byte shapes.
|
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean*, only how to read and write the byte shapes.
|
||||||
|
|
||||||
The PDU types and their fields were reverse-engineered from three sources:
|
The PDU types and their fields were reverse-engineered from three sources:
|
||||||
|
|
||||||
@ -44,9 +44,9 @@ The PDU types and their fields were reverse-engineered from three sources:
|
|||||||
|
|
||||||
## Codec / Per-column readers
|
## Codec / Per-column readers
|
||||||
|
|
||||||
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types — see [SQL ↔ Python types](/reference/types/) for the full table.
|
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types. See [SQL ↔ Python types](/reference/types/) for the full table.
|
||||||
|
|
||||||
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]` — no per-column dispatch overhead.
|
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]`, with no per-column dispatch overhead.
|
||||||
|
|
||||||
Phase 38 went further with `exec()`-based codegen: for the hottest tables, the driver generates a flat decoder function with all readers inlined and dispatch decisions baked in. The generated function is the equivalent of unrolling the per-column dispatch into straight-line code.
|
Phase 38 went further with `exec()`-based codegen: for the hottest tables, the driver generates a flat decoder function with all readers inlined and dispatch decisions baked in. The generated function is the equivalent of unrolling the per-column dispatch into straight-line code.
|
||||||
|
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Async strategy
|
title: Async strategy
|
||||||
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async — and what that costs.
|
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async, and what that costs.
|
||||||
sidebar:
|
sidebar:
|
||||||
order: 4
|
order: 4
|
||||||
---
|
---
|
||||||
@ -19,13 +19,13 @@ Three options for adding async support to a sync database driver:
|
|||||||
|
|
||||||
2. **Thread-pool wrapping.** Keep the sync core. Wrap each public method with `loop.run_in_executor()`. ~250 lines of code, sync tests still apply, no protocol-layer changes.
|
2. **Thread-pool wrapping.** Keep the sync core. Wrap each public method with `loop.run_in_executor()`. ~250 lines of code, sync tests still apply, no protocol-layer changes.
|
||||||
|
|
||||||
3. **Dual implementations.** Maintain two parallel code paths — one sync, one async. Most code duplicated. Worst of both worlds.
|
3. **Dual implementations.** Maintain two parallel code paths, one sync and one async. Most code duplicated. Worst of both worlds.
|
||||||
|
|
||||||
We picked option 2.
|
We picked option 2.
|
||||||
|
|
||||||
## Why option 2 was the right call
|
## Why option 2 was the right call
|
||||||
|
|
||||||
For typical database workloads — request-scoped connections, mostly waiting on I/O — the practical difference between option 1 and option 2 is small:
|
For typical database workloads, meaning request-scoped connections that mostly wait on I/O, the practical difference between option 1 and option 2 is small:
|
||||||
|
|
||||||
- **Latency**: option 1 has a slight edge (no thread context switch), but the difference is dwarfed by the actual database round-trip (~80 µs LAN, ~ms WAN). For a single query, option 2 adds ~5–10 µs of executor overhead.
|
- **Latency**: option 1 has a slight edge (no thread context switch), but the difference is dwarfed by the actual database round-trip (~80 µs LAN, ~ms WAN). For a single query, option 2 adds ~5–10 µs of executor overhead.
|
||||||
- **Throughput under concurrency**: option 1 wins when you have N coroutines on M physical cores with M < N. The thread pool needs to context-switch between threads; the async loop just runs the next coroutine. For 10–100 concurrent FastAPI requests on a 4-core box, this difference is small.
|
- **Throughput under concurrency**: option 1 wins when you have N coroutines on M physical cores with M < N. The thread pool needs to context-switch between threads; the async loop just runs the next coroutine. For 10–100 concurrent FastAPI requests on a 4-core box, this difference is small.
|
||||||
@ -39,7 +39,7 @@ The honest costs:
|
|||||||
|
|
||||||
- **One worker thread per concurrent in-flight query.** With 100 concurrent queries, you have 100 threads. This is fine for I/O-bound work (Python releases the GIL during socket reads) but doesn't scale beyond a few hundred concurrent queries on a single process.
|
- **One worker thread per concurrent in-flight query.** With 100 concurrent queries, you have 100 threads. This is fine for I/O-bound work (Python releases the GIL during socket reads) but doesn't scale beyond a few hundred concurrent queries on a single process.
|
||||||
- **Thread-pool sizing matters.** The default executor size (5 × CPU count) is fine for most workloads. For high-concurrency workloads, you may want a larger executor.
|
- **Thread-pool sizing matters.** The default executor size (5 × CPU count) is fine for most workloads. For high-concurrency workloads, you may want a larger executor.
|
||||||
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe — cancelled workers cannot leak onto recycled pool connections — but the underlying syscall does still complete.
|
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe, in that cancelled workers cannot leak onto recycled pool connections, but the underlying syscall does still complete.
|
||||||
|
|
||||||
## What it doesn't cost
|
## What it doesn't cost
|
||||||
|
|
||||||
|
|||||||
@ -7,7 +7,7 @@ sidebar:
|
|||||||
|
|
||||||
import { Aside } from '@astrojs/starlight/components';
|
import { Aside } from '@astrojs/starlight/components';
|
||||||
|
|
||||||
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39 — a connection-scoped buffered reader — closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging.
|
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39, a connection-scoped buffered reader, closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging.
|
||||||
|
|
||||||
This page is about both the technical change and the failure mode that hid the win for two phases.
|
This page is about both the technical change and the failure mode that hid the win for two phases.
|
||||||
|
|
||||||
@ -26,7 +26,7 @@ The headline "I/O dominated" was true. The interesting half is the breakdown of
|
|||||||
- Actual `recv()` syscalls: ~153 ms
|
- Actual `recv()` syscalls: ~153 ms
|
||||||
- Python wrapper overhead: ~400 ms
|
- Python wrapper overhead: ~400 ms
|
||||||
|
|
||||||
That ~400 ms was our own buffer abstraction — a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
|
That ~400 ms was our own buffer abstraction: a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
|
||||||
|
|
||||||
The kernel was doing maybe 25–30 ms of work. The other 130 ms of the gap-vs-IfxPy was friction we had introduced ourselves.
|
The kernel was doing maybe 25–30 ms of work. The other 130 ms of the gap-vs-IfxPy was friction we had introduced ourselves.
|
||||||
|
|
||||||
@ -63,21 +63,21 @@ Result: **one `recv()` per ~64 KB of incoming data**, not per field.
|
|||||||
|
|
||||||
The natural thing to call this is "BufferedSocketReader". The natural thing to do is put the bytearray on the reader. That's what I did first.
|
The natural thing to call this is "BufferedSocketReader". The natural thing to do is put the bytearray on the reader. That's what I did first.
|
||||||
|
|
||||||
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken` — process blocked in `recv()` waiting for bytes that weren't coming.
|
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken`, meaning the process was blocked in `recv()` waiting for bytes that weren't coming.
|
||||||
|
|
||||||
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes — 200 bytes for response 1, 400 bytes for response 2 — it consumed bytes for response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
|
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes, 200 for response 1 and 400 for response 2, it consumed bytes belonging to response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
|
||||||
|
|
||||||
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket` (the connection-scoped wrapper) — readers are short-lived parser-views, the buffer outlives them.
|
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket`, the connection-scoped wrapper. Readers are short-lived parser-views, and the buffer outlives them.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# WRONG (first pass) — buffer scoped to reader
|
# WRONG (first pass): buffer scoped to reader
|
||||||
class BufferedSocketReader:
|
class BufferedSocketReader:
|
||||||
def __init__(self, sock):
|
def __init__(self, sock):
|
||||||
self.sock = sock
|
self.sock = sock
|
||||||
self.buf = bytearray() # ← dies with the reader
|
self.buf = bytearray() # ← dies with the reader
|
||||||
self.offset = 0
|
self.offset = 0
|
||||||
|
|
||||||
# RIGHT (Phase 39) — buffer scoped to connection
|
# RIGHT (Phase 39): buffer scoped to connection
|
||||||
class IfxSocket:
|
class IfxSocket:
|
||||||
def __init__(self, sock):
|
def __init__(self, sock):
|
||||||
self.sock = sock
|
self.sock = sock
|
||||||
@ -122,7 +122,7 @@ The buffered reader ships **enabled by default** in version 2026.05.05.12. To op
|
|||||||
IFX_BUFFERED_READER=0 python my_app.py
|
IFX_BUFFERED_READER=0 python my_app.py
|
||||||
```
|
```
|
||||||
|
|
||||||
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime — close and reopen the pool to flip behavior.
|
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime, so close and reopen the pool to flip behavior.
|
||||||
|
|
||||||
<Aside type="note">
|
<Aside type="note">
|
||||||
The flag exists to make A/B measurement easy. There's no expected reason to disable it in production. If you hit a workload where the buffered reader is slower, that's a bug and we'd like to know.
|
The flag exists to make A/B measurement easy. There's no expected reason to disable it in production. If you hit a workload where the buffered reader is slower, that's a bug and we'd like to know.
|
||||||
@ -132,9 +132,9 @@ The flag is read once at connection construction. Existing connections in a pool
|
|||||||
|
|
||||||
The general pattern: **what's visible gets optimization attention; what's invisible gets written off as irreducible**.
|
The general pattern: **what's visible gets optimization attention; what's invisible gets written off as irreducible**.
|
||||||
|
|
||||||
The codec is visible — there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
|
The codec is visible: there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
|
||||||
|
|
||||||
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads — `read_short`, `read_int`, `read_exact(payload_size)` — are one-liners. What could be slow about that?
|
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads (`read_short`, `read_int`, `read_exact(payload_size)`) are one-liners. What could be slow about that?
|
||||||
|
|
||||||
It was carrying ~30% of total wall time. Two phases of changelogs implicitly blamed "the protocol" for the remaining gap. The actual culprit was a few lines of `bytes.join` in a wrapper from Phase 1 that nobody had revisited.
|
It was carrying ~30% of total wall time. Two phases of changelogs implicitly blamed "the protocol" for the remaining gap. The actual culprit was a few lines of `bytes.join` in a wrapper from Phase 1 that nobody had revisited.
|
||||||
|
|
||||||
@ -142,6 +142,6 @@ The lesson is small and easy to state: a profile turns vibes into an attack surf
|
|||||||
|
|
||||||
## Read more
|
## Read more
|
||||||
|
|
||||||
- **[Architecture overview →](/explain/architecture/)** — where the buffered reader sits in the layer stack.
|
- **[Architecture overview →](/explain/architecture/)**: where the buffered reader sits in the layer stack.
|
||||||
- **[Phase log →](/explain/phase-log/)** — the full progression from Phase 1 through Phase 39+.
|
- **[Phase log →](/explain/phase-log/)**: the full progression from Phase 1 through Phase 39+.
|
||||||
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)** — Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.
|
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)**: Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.
|
||||||
|
|||||||
@ -43,11 +43,11 @@ The driver was built across 39+ phases, each with a focused scope and a decision
|
|||||||
| 23 | Health checks | Pool validates idle connections before return |
|
| 23 | Health checks | Pool validates idle connections before return |
|
||||||
| 24 | Statement caching | Per-connection prepared-statement cache |
|
| 24 | Statement caching | Per-connection prepared-statement cache |
|
||||||
| 25 | Fast-path call (`SQ_FPROUTINE`) | Direct UDF/SPL invocation, bypassing PREPARE |
|
| 25 | Fast-path call (`SQ_FPROUTINE`) | Direct UDF/SPL invocation, bypassing PREPARE |
|
||||||
| 26 | **CRITICAL** | Pool returned connections with open transactions — fixed |
|
| 26 | **CRITICAL** | Pool returned connections with open transactions (fixed) |
|
||||||
| 27 | **CRITICAL** | Per-connection wire lock + async cancellation safety |
|
| 27 | **CRITICAL** | Per-connection wire lock + async cancellation safety |
|
||||||
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync — fixed |
|
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync (fixed) |
|
||||||
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise — fixed |
|
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise (fixed) |
|
||||||
| 30 | Hardening pass | 5 medium-severity audit findings — all closed |
|
| 30 | Hardening pass | 5 medium-severity audit findings, all closed |
|
||||||
|
|
||||||
After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver is production-ready.
|
After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver is production-ready.
|
||||||
|
|
||||||
@ -67,21 +67,84 @@ After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver i
|
|||||||
|
|
||||||
The Phase 37–39 trajectory is documented in detail at [The buffered reader →](/explain/buffered-reader/), including the architectural mistake the first pass got wrong.
|
The Phase 37–39 trajectory is documented in detail at [The buffered reader →](/explain/buffered-reader/), including the architectural mistake the first pass got wrong.
|
||||||
|
|
||||||
|
## Field reports and systematic review (2026.08–2026.09)
|
||||||
|
|
||||||
|
Phase 30's audit found nothing left. Then real schemas arrived, and the audit's clean bill of health turned out to be a statement about the questions it had asked rather than about the driver.
|
||||||
|
|
||||||
|
A field report from a user running Informix 12 surfaced three type-framing bugs in a single afternoon: `INT8` / `SERIAL8` never decoded at all, `NCHAR` losing its first character, and `BOOLEAN` corrupting every column after it. Fuzzing the type matrix found more, and each one had the same shape as the last.
|
||||||
|
|
||||||
|
`2026.09.02` went back over the driver looking for that shape rather than for new symptoms, and found eleven bugs.
|
||||||
|
|
||||||
|
| Area | What was wrong |
|
||||||
|
|---|---|
|
||||||
|
| Row framing | Nothing ever checked that a row consumed its own payload. Fourteen framing bugs had reached users, every one detectable for free |
|
||||||
|
| Smart LOBs | Read as a flat 72-byte field when they occupy 149, shifting every column after a non-final `BLOB` |
|
||||||
|
| Composites | A NULL collection skipped its length field, the same bug as NULL `LVARCHAR` twelve lines away |
|
||||||
|
| Transactions | `rollback()` after a SQL `BEGIN WORK` sent nothing and reported success |
|
||||||
|
| Classification | Comments, CTEs and parenthesized selects were run as DML and failed with `-260` |
|
||||||
|
| Scrollable cursors | A second statement on the connection got `-285` and destroyed the cursor too |
|
||||||
|
| Statement release | Two of eight exits leaked a failed statement, bricking the connection |
|
||||||
|
| Cursor finalizers | Cleanup could land inside another statement, and the lock probe was blind to its own thread |
|
||||||
|
| Async | The whole layer shared the process-wide thread pool, so cancellations starved it |
|
||||||
|
| Socket | Two readers on one stream agreed only by accident of how the server replies |
|
||||||
|
|
||||||
|
The pattern behind almost all of them: a hazard that was understood and guarded at the site where it was first observed, rather than at the abstraction that owned it. The guard then never travelled to its siblings. Four separate hand-written copies of the same UDT envelope, three of them wrong. Six hand-rolled copies of the same statement-release cleanup, two missing entirely.
|
||||||
|
|
||||||
|
Two of the fixes came from asking the server instead of guessing. `statement_type` and `statement_id` had both been parsed out of every DESCRIBE response into a metadata dict that nothing read, while the code that needed them inferred the answer from the first word of the SQL and got it wrong five ways.
|
||||||
|
|
||||||
|
The cheapest fix was also the most valuable. Asserting that a row decoder lands exactly on the end of its payload costs one integer comparison, and it would have caught all fourteen framing bugs at the byte where each happened rather than three releases later in somebody's result set.
|
||||||
|
|
||||||
|
Test count went from 241 to 457 across three server versions over this stretch.
|
||||||
|
|
||||||
|
### 2026.09.03
|
||||||
|
|
||||||
|
Two more, one of each kind.
|
||||||
|
|
||||||
|
The `:N` placeholder rewriter was a bare regular expression, so it
|
||||||
|
rewrote the inside of string literals: `'http://host:8080/x'` was stored
|
||||||
|
as `'http://host?/x'`. Same family as the statement-classification bug
|
||||||
|
above, and the last place the driver inferred meaning from SQL text
|
||||||
|
instead of handling it. Replaced with a scanner using Informix's own
|
||||||
|
lexical rules, measured rather than assumed. One of those rules would
|
||||||
|
have been got wrong from habit: a backslash escapes nothing, so `'a\'b'`
|
||||||
|
is an unterminated string and draws `-282`. A scanner written to
|
||||||
|
Postgres reflexes desyncs there and corrupts everything after it.
|
||||||
|
|
||||||
|
`row_factory=Row` adds column-name and attribute access, on request from
|
||||||
|
a user coming from `pyodbc`. Opt-in, because it costs about 9% on bulk
|
||||||
|
fetch and that is not a trade to make for someone who never reads a
|
||||||
|
column by name.
|
||||||
|
|
||||||
|
Testing the second one found the pattern again, this time in code
|
||||||
|
written an hour earlier. The names a column could shadow were
|
||||||
|
hand-listed as `count` and `index`, the two `tuple` methods; then `keys`,
|
||||||
|
`_asdict` and `_fields` were added to the same class and the list was
|
||||||
|
never revisited. A column called `keys` returned a bound method, in
|
||||||
|
silence. The fix was to compute the set from the class rather than
|
||||||
|
lengthen the list, and to parametrize the test over that computed set so
|
||||||
|
it grows with the class.
|
||||||
|
|
||||||
|
The interesting part is where testing found it. A differential test
|
||||||
|
across twenty columns covering every awkward type, populated and fully
|
||||||
|
NULL, found nothing. The bug was in six lines of name bookkeeping that
|
||||||
|
looked obviously correct, and had been correct when written. Anything
|
||||||
|
that must stay in sync with a class by hand eventually will not.
|
||||||
|
|
||||||
## Notable architectural pivots
|
## Notable architectural pivots
|
||||||
|
|
||||||
The decision log calls out four moments where the obvious choice would have been wrong:
|
The decision log calls out four moments where the obvious choice would have been wrong:
|
||||||
|
|
||||||
1. **Phase 10/11** — abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
|
1. **Phase 10/11**: abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
|
||||||
2. **Phase 16** — thread-pool async instead of full async refactor. ~88% less code, same FastAPI surface.
|
2. **Phase 16**: thread-pool async instead of a full async refactor. ~88% less code, same FastAPI surface.
|
||||||
3. **Phase 27** — adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
|
3. **Phase 27**: adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
|
||||||
4. **Phase 39** — buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
|
4. **Phase 39**: buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
|
||||||
|
|
||||||
## What's next
|
## What's next
|
||||||
|
|
||||||
The roadmap (loose, not committed):
|
The roadmap (loose, not committed):
|
||||||
|
|
||||||
- **Phase 40+ (codec)**: Numpy-backed bulk decode for homogeneous columns. ~5× speedup target on analytical workloads.
|
- **Phase 40+ (codec)**: Numpy-backed bulk decode for homogeneous columns. ~5× speedup target on analytical workloads.
|
||||||
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python" — gated behind a build flag.
|
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python", so it would sit behind a build flag.
|
||||||
- **Phase 5x (API)**: Native `callproc` with named parameters, IBM-specific scrollable cursor extensions for full IfxPy parity.
|
- **Phase 5x (API)**: Native `callproc` with named parameters, IBM-specific scrollable cursor extensions for full IfxPy parity.
|
||||||
|
|
||||||
The phase log is updated as work lands. The repo's [`CHANGELOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md) is the source of truth for shipped changes.
|
The phase log is updated as work lands. The repo's [`CHANGELOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md) is the source of truth for shipped changes.
|
||||||
|
|||||||
@ -26,14 +26,14 @@ The order-of-magnitude intuition: pure-Python is ~2× slower than C-bound for **
|
|||||||
|
|
||||||
The benefits are mostly deployment, not performance:
|
The benefits are mostly deployment, not performance:
|
||||||
|
|
||||||
- **50 KB wheel** — installable in a slim Docker image without a build toolchain.
|
- **50 KB wheel**: installable in a slim Docker image without a build toolchain.
|
||||||
- **No `libcrypt.so.1`** — works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
|
- **No `libcrypt.so.1`**: works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
|
||||||
- **Python 3.10–3.14** — no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
|
- **Python 3.10–3.14**: no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
|
||||||
- **Type annotations everywhere** — `py.typed` flag, full coverage in mypy / pyright.
|
- **Type annotations everywhere**: `py.typed` flag, full coverage in mypy / pyright.
|
||||||
- **Auditable codepaths** — every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
|
- **Auditable codepaths**: every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
|
||||||
- **Async without `run_in_executor`** — native `async def` API, FastAPI-compatible.
|
- **Async without `run_in_executor`**: native `async def` API, FastAPI-compatible.
|
||||||
|
|
||||||
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1` — those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
|
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1`: those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
|
||||||
|
|
||||||
## Where the ceiling sits
|
## Where the ceiling sits
|
||||||
|
|
||||||
@ -48,7 +48,7 @@ Five fields × ~250 ns/field + ~250 ns overhead = ~1.5 µs. We're at ~2.0 µs wh
|
|||||||
|
|
||||||
Strategies for closing further:
|
Strategies for closing further:
|
||||||
|
|
||||||
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim — there'd be a build step.
|
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim, because there'd be a build step.
|
||||||
- **Bytecode optimization via `exec()`-codegen** (the Phase 38 approach). Marginal further wins; we've already extracted most of what's available.
|
- **Bytecode optimization via `exec()`-codegen** (the Phase 38 approach). Marginal further wins; we've already extracted most of what's available.
|
||||||
- **Numpy-backed bulk decode** for homogeneous columns. Promising for analytical workloads. ~5× speedup possible for `SELECT col FROM huge_table` over the current per-row approach. Probably Phase 41+.
|
- **Numpy-backed bulk decode** for homogeneous columns. Promising for analytical workloads. ~5× speedup possible for `SELECT col FROM huge_table` over the current per-row approach. Probably Phase 41+.
|
||||||
|
|
||||||
@ -58,6 +58,6 @@ For I/O-bound workloads we're already at the ceiling. The buffered reader closed
|
|||||||
|
|
||||||
Pure-Python costs us ~5–15% on bulk-fetch workloads and zero (or favorable) on everything else. The deployment, async, and modern-Python wins are large and don't depend on workload.
|
Pure-Python costs us ~5–15% on bulk-fetch workloads and zero (or favorable) on everything else. The deployment, async, and modern-Python wins are large and don't depend on workload.
|
||||||
|
|
||||||
If the codec gap matters for your case — analytical reporting against a wide table, pulling millions of rows in a single SELECT — IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
|
If the codec gap matters for your case, meaning analytical reporting against a wide table or pulling millions of rows in a single SELECT, IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
|
||||||
|
|
||||||
The driver chose the goal — *first pure-socket Informix driver in any language* — over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.
|
The driver chose the goal, *first pure-socket Informix driver in any language*, over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.
|
||||||
|
|||||||
@ -1,13 +1,13 @@
|
|||||||
---
|
---
|
||||||
title: The SQLI wire protocol
|
title: The SQLI wire protocol
|
||||||
description: A short tour of Informix's SQLI protocol — PDU framing, the handshake, statement execution, fetch.
|
description: A short tour of Informix's SQLI protocol, covering PDU framing, the handshake, statement execution, and fetch.
|
||||||
sidebar:
|
sidebar:
|
||||||
order: 1
|
order: 1
|
||||||
---
|
---
|
||||||
|
|
||||||
import { Aside } from '@astrojs/starlight/components';
|
import { Aside } from '@astrojs/starlight/components';
|
||||||
|
|
||||||
SQLI is Informix's wire protocol — the same protocol IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
|
SQLI is Informix's wire protocol, the same one IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
|
||||||
|
|
||||||
This page is a short tour. The byte-level reference (with hex annotations) lives in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) in the repo.
|
This page is a short tour. The byte-level reference (with hex annotations) lives in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) in the repo.
|
||||||
|
|
||||||
@ -71,14 +71,14 @@ The full lifecycle for `SELECT id FROM users WHERE id = ?`:
|
|||||||
← SQ_ID
|
← SQ_ID
|
||||||
```
|
```
|
||||||
|
|
||||||
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts — see [Bulk inserts](/how-to/executemany/).
|
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts. See [Bulk inserts](/how-to/executemany/).
|
||||||
|
|
||||||
## Smart-LOB transfer
|
## Smart-LOB transfer
|
||||||
|
|
||||||
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side — no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
|
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side, with no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
|
||||||
|
|
||||||
This was the architectural pivot in [Phase 10/11](/explain/phase-log/) that made smart-LOBs work end-to-end in pure Python. Reading and writing GB-sized BLOBs goes through the same socket as any other query.
|
This was the architectural pivot in [Phase 10/11](/explain/phase-log/) that made smart-LOBs work end-to-end in pure Python. Reading and writing GB-sized BLOBs goes through the same socket as any other query.
|
||||||
|
|
||||||
<Aside type="note">
|
<Aside type="note">
|
||||||
The protocol has many more PDU types than this page covers — mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
|
The protocol has many more PDU types than this page covers, mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
|
||||||
</Aside>
|
</Aside>
|
||||||
|
|||||||
@ -56,7 +56,7 @@ async def get_user(user_id: int, conn = Depends(get_conn)):
|
|||||||
|
|
||||||
## Cancellation
|
## Cancellation
|
||||||
|
|
||||||
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe — the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean (transactions rolled back). You don't need to wrap anything in `try/finally`.
|
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe: the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean, with transactions rolled back. You don't need to wrap anything in `try/finally`.
|
||||||
|
|
||||||
<Aside type="note">
|
<Aside type="note">
|
||||||
This is a Phase 27 invariant: async cancellation cannot leak running workers onto recycled connections. The earlier behavior was a `High` audit finding; the fix is a CI tripwire test that's been green every commit since.
|
This is a Phase 27 invariant: async cancellation cannot leak running workers onto recycled connections. The earlier behavior was a `High` audit finding; the fix is a CI tripwire test that's been green every commit since.
|
||||||
@ -89,5 +89,5 @@ async def create_order(order: OrderIn, conn = Depends(get_conn)):
|
|||||||
`HTTPException` is an `Exception`, so raising one inside the block rolls back before FastAPI turns it into a response.
|
`HTTPException` is an `Exception`, so raising one inside the block rolls back before FastAPI turns it into a response.
|
||||||
|
|
||||||
<Aside type="note">
|
<Aside type="note">
|
||||||
There is no `conn.transaction()` context manager on either the sync or async connection — an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
|
There is no `conn.transaction()` context manager on either the sync or async connection, though an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
|
||||||
</Aside>
|
</Aside>
|
||||||
|
|||||||
@ -1,11 +1,11 @@
|
|||||||
---
|
---
|
||||||
title: Optimize bulk SELECT
|
title: Optimize bulk SELECT
|
||||||
description: How the buffered reader works in practice — when it's on, when it isn't, how to A/B-measure your workload.
|
description: How the buffered reader works in practice, when it's on, when it isn't, and how to A/B-measure your workload.
|
||||||
sidebar:
|
sidebar:
|
||||||
order: 5
|
order: 5
|
||||||
---
|
---
|
||||||
|
|
||||||
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything — the bulk-fetch gap against IfxPy is now ~5–15% rather than ~140%.
|
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything, since the bulk-fetch gap against IfxPy is now ~5–15% rather than ~140%.
|
||||||
|
|
||||||
For the architectural rationale, see [The buffered reader →](/explain/buffered-reader/).
|
For the architectural rationale, see [The buffered reader →](/explain/buffered-reader/).
|
||||||
|
|
||||||
@ -33,7 +33,7 @@ For typical bulk-SELECT workloads expect a 30–40% wall-time reduction. For wor
|
|||||||
|
|
||||||
## When the speedup is largest
|
## When the speedup is largest
|
||||||
|
|
||||||
Workloads where every column read makes ~4–5 small `recv()` calls — i.e. tabular data, narrow rows, large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
|
Workloads where every column read makes ~4–5 small `recv()` calls, meaning tabular data, narrow rows, and large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
|
||||||
|
|
||||||
| Workload shape | Speedup |
|
| Workload shape | Speedup |
|
||||||
|---|---:|
|
|---|---:|
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Run the dev container
|
title: Run the dev container
|
||||||
description: IBM Informix Developer Edition in Docker — first-time setup, sbspace for smart-LOBs, common troubleshooting.
|
description: IBM Informix Developer Edition in Docker, covering first-time setup, sbspace for smart-LOBs, and common troubleshooting.
|
||||||
sidebar:
|
sidebar:
|
||||||
order: 8
|
order: 8
|
||||||
---
|
---
|
||||||
@ -20,9 +20,9 @@ docker run -d --name informix-dev \
|
|||||||
icr.io/informix/informix-developer-database:15.0.1.0.3DE
|
icr.io/informix/informix-developer-database:15.0.1.0.3DE
|
||||||
```
|
```
|
||||||
|
|
||||||
- **`9088`** — clear-text SQLI listener
|
- **`9088`**: clear-text SQLI listener
|
||||||
- **`9089`** — TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
|
- **`9089`**: TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
|
||||||
- **`--privileged`** — required for the dev image's shared-memory tuning
|
- **`--privileged`**: required for the dev image's shared-memory tuning
|
||||||
|
|
||||||
The image takes ~90 seconds to initialize. Watch for `oninit running`:
|
The image takes ~90 seconds to initialize. Watch for `oninit running`:
|
||||||
|
|
||||||
@ -84,10 +84,10 @@ pytest -m integration
|
|||||||
|
|
||||||
**"Connection refused" on 9088**: the image is still initializing. Wait for `oninit running` in the logs.
|
**"Connection refused" on 9088**: the image is still initializing. Wait for `oninit running` in the logs.
|
||||||
|
|
||||||
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing — it always exists.
|
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing, since it always exists.
|
||||||
|
|
||||||
**`-908` (system error / shared memory)**: the container needs `--privileged`. Restart with that flag.
|
**`-908` (system error / shared memory)**: the container needs `--privileged`. Restart with that flag.
|
||||||
|
|
||||||
<Aside type="tip">
|
<Aside type="tip">
|
||||||
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart — the database survives container shutdown.
|
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart, and the database survives container shutdown.
|
||||||
</Aside>
|
</Aside>
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Bulk inserts (executemany)
|
title: Bulk inserts (executemany)
|
||||||
description: How to bulk-load with executemany — and the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
description: How to bulk-load with executemany, including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||||
sidebar:
|
sidebar:
|
||||||
order: 4
|
order: 4
|
||||||
---
|
---
|
||||||
@ -49,7 +49,7 @@ The default is `autocommit=False`, so this only catches you if you've explicitly
|
|||||||
|
|
||||||
## Why it's faster than IfxPy
|
## Why it's faster than IfxPy
|
||||||
|
|
||||||
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row — for 10,000 rows on a 80 µs RTT, that's 800 ms of just waiting for ACKs.
|
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row, so for 10,000 rows on an 80 µs RTT it adds up to 800 ms of just waiting for ACKs.
|
||||||
|
|
||||||
Phase 33 changed our `executemany` to **pipeline** the BIND+EXECUTE PDUs:
|
Phase 33 changed our `executemany` to **pipeline** the BIND+EXECUTE PDUs:
|
||||||
|
|
||||||
@ -78,7 +78,7 @@ with conn:
|
|||||||
cur = conn.cursor()
|
cur = conn.cursor()
|
||||||
for batch in chunks(huge_iterator, 10_000):
|
for batch in chunks(huge_iterator, 10_000):
|
||||||
cur.executemany("INSERT INTO logs ...", batch)
|
cur.executemany("INSERT INTO logs ...", batch)
|
||||||
# one transaction, many batched executemany calls — one commit at the end
|
# one transaction, many batched executemany calls, one commit at the end
|
||||||
```
|
```
|
||||||
|
|
||||||
10,000 rows per chunk is a reasonable default; the per-chunk Python memory cost is `~ N × bytes_per_row`. For 10k tuples of 5 small fields that's a few MB.
|
10,000 rows per chunk is a reasonable default; the per-chunk Python memory cost is `~ N × bytes_per_row`. For 10k tuples of 5 small fields that's a few MB.
|
||||||
@ -95,7 +95,7 @@ new_id = cur.fetchone()[0]
|
|||||||
|
|
||||||
`sqlca.sqlerrd1` is where Informix records the serial value from the most recent INSERT on this connection, so read it before running any other statement.
|
`sqlca.sqlerrd1` is where Informix records the serial value from the most recent INSERT on this connection, so read it before running any other statement.
|
||||||
|
|
||||||
There is no `cursor.lastrowid` — an earlier version of this page said there was.
|
There is no `cursor.lastrowid`, though an earlier version of this page said there was.
|
||||||
|
|
||||||
For batch inserts that need the IDs, the idiomatic pattern is:
|
For batch inserts that need the IDs, the idiomatic pattern is:
|
||||||
|
|
||||||
|
|||||||
@ -27,11 +27,11 @@ conn = informix_db.connect(
|
|||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported (deliberately — they're a security and parsing footgun).
|
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported, deliberately: they're a security and parsing footgun.
|
||||||
|
|
||||||
## The DB-API surface is the same
|
## The DB-API surface is the same
|
||||||
|
|
||||||
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, `close()` — all behave per PEP 249.
|
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, and `close()` all behave per PEP 249.
|
||||||
|
|
||||||
The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `DatabaseError`, `DataError`, `OperationalError`, `IntegrityError`, `InternalError`, `ProgrammingError`, `NotSupportedError`.
|
The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `DatabaseError`, `DataError`, `OperationalError`, `IntegrityError`, `InternalError`, `ProgrammingError`, `NotSupportedError`.
|
||||||
|
|
||||||
@ -45,9 +45,39 @@ The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `Dat
|
|||||||
|
|
||||||
- **Async API** (`from informix_db import aio`)
|
- **Async API** (`from informix_db import aio`)
|
||||||
- **Connection pool** (`informix_db.create_pool` / `aio.create_pool`)
|
- **Connection pool** (`informix_db.create_pool` / `aio.create_pool`)
|
||||||
- **Type-safe annotations** — `informix-driver` ships with `py.typed`
|
- **Type-safe annotations**: `informix-driver` ships with `py.typed`
|
||||||
- **Python 3.12+ support**
|
- **Python 3.12+ support**
|
||||||
- **Pipelined `executemany`** — 1.6× faster than IfxPy's per-row implementation
|
- **Pipelined `executemany`**: 1.6× faster than IfxPy's per-row implementation
|
||||||
|
- **Rows addressable by name**: `row["col"]` and `row.col`, not just `row[0]`
|
||||||
|
|
||||||
|
## Reading rows by name
|
||||||
|
|
||||||
|
IfxPy gives you positional access and nothing else, which is why most
|
||||||
|
IfxPy codebases grow a helper like this:
|
||||||
|
|
||||||
|
```python
|
||||||
|
cols = [c[0].lower() for c in cur.description]
|
||||||
|
row_dict = dict(zip(cols, cur.fetchone()))
|
||||||
|
```
|
||||||
|
|
||||||
|
That gets you a dict and still no attribute access. Ask for `Row`
|
||||||
|
instead and all three work at once:
|
||||||
|
|
||||||
|
```python
|
||||||
|
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||||
|
cur.execute("SELECT config_key, config_value FROM settings")
|
||||||
|
row = cur.fetchone()
|
||||||
|
|
||||||
|
row[0], row["config_key"], row.config_key
|
||||||
|
```
|
||||||
|
|
||||||
|
Set on the connection, so it applies to every cursor from it. The
|
||||||
|
`.lower()` in the hand-written version is already a no-op, incidentally:
|
||||||
|
Informix folds unquoted identifiers to lower case.
|
||||||
|
|
||||||
|
It is opt-in because it costs about 9% on bulk fetch. See
|
||||||
|
[the API reference](/reference/api/#rows) for the numbers and the
|
||||||
|
edge cases.
|
||||||
|
|
||||||
## Migrating incrementally
|
## Migrating incrementally
|
||||||
|
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Use the connection pool
|
title: Use the connection pool
|
||||||
description: Sync and async connection pools — sizing, timeouts, lifecycle, threading.
|
description: Sync and async connection pools, covering sizing, timeouts, lifecycle, and threading.
|
||||||
sidebar:
|
sidebar:
|
||||||
order: 2
|
order: 2
|
||||||
---
|
---
|
||||||
@ -32,7 +32,7 @@ with pool.connection() as conn:
|
|||||||
pool.close()
|
pool.close()
|
||||||
```
|
```
|
||||||
|
|
||||||
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically — you never see a dirty connection from `pool.connection()`.
|
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically, so you never see a dirty connection from `pool.connection()`.
|
||||||
|
|
||||||
## Async pool
|
## Async pool
|
||||||
|
|
||||||
@ -56,16 +56,16 @@ async def main():
|
|||||||
asyncio.run(main())
|
asyncio.run(main())
|
||||||
```
|
```
|
||||||
|
|
||||||
Same semantics, `async`-aware. Cancellation is cancellation-safe — a cancelled task does not leak an in-flight worker onto a recycled connection.
|
Same semantics, `async`-aware. Cancellation is safe here too: a cancelled task does not leak an in-flight worker onto a recycled connection.
|
||||||
|
|
||||||
## Sizing
|
## Sizing
|
||||||
|
|
||||||
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes" — usually 8–16 for typical web/API services.
|
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes", usually 8–16 for typical web/API services.
|
||||||
|
|
||||||
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`** — the server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
|
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`**. The server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
|
||||||
|
|
||||||
## Threading
|
## Threading
|
||||||
|
|
||||||
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally — `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
|
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally, since `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
|
||||||
|
|
||||||
Phase 27 added a per-connection wire lock that makes accidental sharing safe (interleaved PDUs serialize correctly), but you should still give each thread its own connection. The lock is a backstop, not a license.
|
Phase 27 added a per-connection wire lock that makes accidental sharing safe (interleaved PDUs serialize correctly), but you should still give each thread its own connection. The lock is a backstop, not a license.
|
||||||
|
|||||||
@ -34,7 +34,7 @@ The `BLOB_PLACEHOLDER` token in the SQL marks where the BLOB data goes. Other pa
|
|||||||
|
|
||||||
## Reading a CLOB
|
## Reading a CLOB
|
||||||
|
|
||||||
CLOBs use the same methods as BLOBs — there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
|
CLOBs use the same methods as BLOBs, and there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
raw: bytes = cur.read_blob_column(
|
raw: bytes = cur.read_blob_column(
|
||||||
|
|||||||
@ -30,7 +30,7 @@ conn = informix_db.connect(
|
|||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
Bring-your-own context is the recommended production pattern — you get full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
|
Bring-your-own context is the recommended production pattern, giving you full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
|
||||||
|
|
||||||
## Dev / self-signed: tls=True
|
## Dev / self-signed: tls=True
|
||||||
|
|
||||||
@ -38,7 +38,7 @@ Bring-your-own context is the recommended production pattern — you get full co
|
|||||||
informix_db.connect(host="127.0.0.1", port=9089, ..., tls=True)
|
informix_db.connect(host="127.0.0.1", port=9089, ..., tls=True)
|
||||||
```
|
```
|
||||||
|
|
||||||
`tls=True` is a convenience for development — it builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
|
`tls=True` is a convenience for development. It builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
|
||||||
|
|
||||||
## Server-side configuration
|
## Server-side configuration
|
||||||
|
|
||||||
@ -48,7 +48,7 @@ The Informix server needs a TLS listener entry in `sqlhosts`:
|
|||||||
informix_tls onsoctcp myhost 9089
|
informix_tls onsoctcp myhost 9089
|
||||||
```
|
```
|
||||||
|
|
||||||
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089` — no configuration needed.
|
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089`, with no configuration needed.
|
||||||
|
|
||||||
<Aside type="tip">
|
<Aside type="tip">
|
||||||
If your connection hangs at the handshake, you've probably pointed at the non-TLS port (`9088` instead of `9089`). The non-TLS listener will accept the TCP connection but won't speak TLS, so the handshake stalls.
|
If your connection hangs at the handshake, you've probably pointed at the non-TLS port (`9088` instead of `9089`). The non-TLS listener will accept the TCP connection but won't speak TLS, so the handshake stalls.
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: informix-driver
|
title: informix-driver
|
||||||
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket — no CSDK, no JVM, no native libraries.
|
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket, with no CSDK, no JVM, and no native libraries.
|
||||||
template: splash
|
template: splash
|
||||||
hero:
|
hero:
|
||||||
tagline: ''
|
tagline: ''
|
||||||
@ -21,12 +21,12 @@ import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
|
|||||||
<div class="ifx-feature">
|
<div class="ifx-feature">
|
||||||
<Icon name="seti:python" class="ifx-feature__icon" />
|
<Icon name="seti:python" class="ifx-feature__icon" />
|
||||||
<h3>~10% behind IfxPy on bulk fetches</h3>
|
<h3>~10% behind IfxPy on bulk fetches</h3>
|
||||||
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics — for now.</p>
|
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics, for now.</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="ifx-feature">
|
<div class="ifx-feature">
|
||||||
<Icon name="puzzle" class="ifx-feature__icon" />
|
<Icon name="puzzle" class="ifx-feature__icon" />
|
||||||
<h3>50 KB wheel. Zero native deps.</h3>
|
<h3>50 KB wheel. Zero native deps.</h3>
|
||||||
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.10–3.14 — including the versions IfxPy doesn't.</p>
|
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.10–3.14, including the versions IfxPy doesn't.</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="ifx-feature">
|
<div class="ifx-feature">
|
||||||
<Icon name="sun" class="ifx-feature__icon" />
|
<Icon name="sun" class="ifx-feature__icon" />
|
||||||
@ -36,7 +36,7 @@ import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
|
|||||||
<div class="ifx-feature">
|
<div class="ifx-feature">
|
||||||
<Icon name="approve-check" class="ifx-feature__icon" />
|
<Icon name="approve-check" class="ifx-feature__icon" />
|
||||||
<h3>PEP 249, no surprises</h3>
|
<h3>PEP 249, no surprises</h3>
|
||||||
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy — and threadsafe sharing through a per-connection wire lock.</p>
|
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy, plus threadsafe sharing through a per-connection wire lock.</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="ifx-feature">
|
<div class="ifx-feature">
|
||||||
<Icon name="document" class="ifx-feature__icon" />
|
<Icon name="document" class="ifx-feature__icon" />
|
||||||
@ -66,7 +66,7 @@ That's it. No `IBM_DB_HOME`. No DSN file. No `libcrypt.so.1`.
|
|||||||
|
|
||||||
The existing tools were not my style.
|
The existing tools were not my style.
|
||||||
|
|
||||||
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix` — all of them. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
|
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That means all of them: `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
|
||||||
|
|
||||||
The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9). It needs four `LD_LIBRARY_PATH` entries. It needs `setuptools < 58`. And IfxPy itself is broken on Python 3.12+. For containerized deployments, ETL pipelines, FastAPI services, or anywhere a build toolchain on the runtime is friction, this driver is the alternative that didn't previously exist. Now it does.
|
The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9). It needs four `LD_LIBRARY_PATH` entries. It needs `setuptools < 58`. And IfxPy itself is broken on Python 3.12+. For containerized deployments, ETL pipelines, FastAPI services, or anywhere a build toolchain on the runtime is friction, this driver is the alternative that didn't previously exist. Now it does.
|
||||||
|
|
||||||
@ -82,7 +82,7 @@ The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, mi
|
|||||||
[Read →](/start/vs-ifxpy/)
|
[Read →](/start/vs-ifxpy/)
|
||||||
</Card>
|
</Card>
|
||||||
<Card title="The buffered reader" icon="information">
|
<Card title="The buffered reader" icon="information">
|
||||||
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1× — and the architectural mistake the first pass got wrong.
|
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1×, and the architectural mistake the first pass got wrong.
|
||||||
[Read →](/explain/buffered-reader/)
|
[Read →](/explain/buffered-reader/)
|
||||||
</Card>
|
</Card>
|
||||||
<Card title="Architecture" icon="puzzle">
|
<Card title="Architecture" icon="puzzle">
|
||||||
|
|||||||
@ -51,7 +51,7 @@ informix_db.connect(
|
|||||||
|
|
||||||
| Method / property | Description |
|
| Method / property | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `cursor(scrollable=False)` | Returns a new `Cursor`. |
|
| `cursor(scrollable=False)` | Returns a new `Cursor`. Only one scrollable cursor may be open per connection; see below. |
|
||||||
| `commit()` | Commits the current transaction. |
|
| `commit()` | Commits the current transaction. |
|
||||||
| `rollback()` | Rolls back the current transaction. |
|
| `rollback()` | Rolls back the current transaction. |
|
||||||
| `close()` | Closes the connection. Idempotent. |
|
| `close()` | Closes the connection. Idempotent. |
|
||||||
@ -64,7 +64,7 @@ informix_db.connect(
|
|||||||
|
|
||||||
`Connection` is also a context manager (`with informix_db.connect(...) as conn:`), which closes on exit.
|
`Connection` is also a context manager (`with informix_db.connect(...) as conn:`), which closes on exit.
|
||||||
|
|
||||||
There is no `conn.transaction()` helper and no `conn.autocommit` attribute — earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
|
There is no `conn.transaction()` helper and no `conn.autocommit` attribute. Earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
conn = informix_db.connect(..., autocommit=False)
|
conn = informix_db.connect(..., autocommit=False)
|
||||||
@ -78,12 +78,67 @@ except Exception:
|
|||||||
raise
|
raise
|
||||||
```
|
```
|
||||||
|
|
||||||
|
:::caution[One statement per session]
|
||||||
|
Informix gives a session a single statement slot. A non-scrollable cursor never trips over this, because it materializes its rows and releases the statement before `execute()` returns. A **scrollable** cursor holds the slot open on purpose, so while one is open, any other statement on that same connection raises `ProgrammingError`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
scroll = conn.cursor(scrollable=True)
|
||||||
|
scroll.execute("SELECT id, body FROM big_table ORDER BY id")
|
||||||
|
scroll.fetch_first()
|
||||||
|
|
||||||
|
other = conn.cursor()
|
||||||
|
other.execute("SELECT COUNT(*) FROM audit") # ProgrammingError
|
||||||
|
```
|
||||||
|
|
||||||
|
Close the scrollable cursor first, or use a second connection for the other query. Re-running `execute()` on the scrollable cursor itself is fine; it closes its own server-side cursor first.
|
||||||
|
|
||||||
|
Before `2026.09.02` this was not refused. The server answered the second statement with `-285` and destroyed the scrollable cursor as well, whose next fetch then returned `-267`, "the transaction has been rolled back". Two errors, neither naming the cause.
|
||||||
|
:::
|
||||||
|
|
||||||
|
:::note[Transaction control written as SQL]
|
||||||
|
`commit()` and `rollback()` are the supported way to end a transaction, but `cursor.execute("BEGIN WORK")` and its `COMMIT` / `ROLLBACK` counterparts work too, and the connection tracks them from `2026.09.02` onward.
|
||||||
|
|
||||||
|
That matters because it used not to. Under `autocommit=True`, a SQL `BEGIN WORK` opened a real transaction the connection never learned about, and since `rollback()` is guarded by that state, **it returned successfully having sent nothing**. The rows it was asked to discard survived, and a pooled connection went back into circulation still holding the transaction and its locks.
|
||||||
|
:::
|
||||||
|
|
||||||
:::note[Two version numbers, and why]
|
:::note[Two version numbers, and why]
|
||||||
The login response carries Informix's *internal* protocol version, not the release you installed: 12.10 announces itself as `9.56`, 14.10 as `9.59`, and 15 as `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers, which is why all three speak an identical SQLI dialect.
|
The login response carries Informix's *internal* protocol version, not the release you installed: 12.10 announces itself as `9.56`, 14.10 as `9.59`, and 15 as `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers, which is why all three speak an identical SQLI dialect.
|
||||||
|
|
||||||
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')` — one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
|
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')`. That is one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
|
||||||
:::
|
:::
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Fetch methods return plain tuples by default. Pass `row_factory` to get rows that also answer to a column name and an attribute, the shape `pyodbc` and `mssql-python` provide:
|
||||||
|
|
||||||
|
```python
|
||||||
|
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("SELECT tabid, tabname FROM systables")
|
||||||
|
row = cur.fetchone()
|
||||||
|
|
||||||
|
row[0] # 1
|
||||||
|
row["tabname"] # 'systables'
|
||||||
|
row.tabname # 'systables'
|
||||||
|
```
|
||||||
|
|
||||||
|
Set on the connection, so it applies to every cursor from it. A cursor can override it with `cur.row_factory = informix_db.Row`. Pools and the async API forward it unchanged.
|
||||||
|
|
||||||
|
`Row` subclasses `tuple`, so `row == (1, "systables")` is still true and anything treating rows as sequences keeps working. Slices return plain tuples, since a slice has no column map. `row.keys()` gives the column names and `row._asdict()` gives a dict.
|
||||||
|
|
||||||
|
:::note[Why it is opt-in]
|
||||||
|
On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` becomes a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript.
|
||||||
|
|
||||||
|
Defaulting it on would move the [1.05-1.15x ratio against IfxPy](/start/vs-ifxpy/) to roughly 1.15-1.25x. That is a good trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is yours to make.
|
||||||
|
:::
|
||||||
|
|
||||||
|
Four behaviours worth knowing:
|
||||||
|
|
||||||
|
- Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`. The `.lower()` people add by hand when building this themselves is already a no-op.
|
||||||
|
- Expression columns get server-generated names like `(count(*))`, which cannot be Python attributes. Those are reachable by subscript only.
|
||||||
|
- Duplicate column names resolve to the first occurrence, matching `pyodbc`. Positional access still reaches both.
|
||||||
|
- **A column beats a method of the same name.** `tuple` defines `count` and `index`; `Row` adds `keys`, `_asdict` and `_fields`. A column with any of those names wins, because otherwise it would hand back a bound method instead of a value. The methods stay reachable as `tuple.count(row, x)` and `Row.keys(row)`.
|
||||||
|
|
||||||
## Cursor
|
## Cursor
|
||||||
|
|
||||||
| Method / property | Description |
|
| Method / property | Description |
|
||||||
@ -105,7 +160,7 @@ That string is available for free as `server_version_internal`. Because it reads
|
|||||||
| `rownumber` | Current 0-indexed position, or `None` before the first row. |
|
| `rownumber` | Current 0-indexed position, or `None` before the first row. |
|
||||||
| `arraysize` | Default `fetchmany()` size. |
|
| `arraysize` | Default `fetchmany()` size. |
|
||||||
|
|
||||||
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute — earlier versions of this page listed all three and none have existed.
|
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute. Earlier versions of this page listed all three, and none have existed.
|
||||||
|
|
||||||
CLOBs go through the BLOB methods. `read_blob_column` returns `bytes` either way, so decode it yourself with the column's encoding:
|
CLOBs go through the BLOB methods. `read_blob_column` returns `bytes` either way, so decode it yourself with the column's encoding:
|
||||||
|
|
||||||
@ -120,7 +175,7 @@ raw = cur.read_blob_column("SELECT txt FROM docs WHERE id = ?", (1,))
|
|||||||
text = raw.decode("iso-8859-1")
|
text = raw.decode("iso-8859-1")
|
||||||
```
|
```
|
||||||
|
|
||||||
For a server-assigned `SERIAL` after an INSERT, ask the server — Informix exposes it through `DBINFO`:
|
For a server-assigned `SERIAL` after an INSERT, ask the server. Informix exposes it through `DBINFO`:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
cur.execute("INSERT INTO people (name) VALUES (?)", ("ada",))
|
cur.execute("INSERT INTO people (name) VALUES (?)", ("ada",))
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Performance baselines
|
title: Performance baselines
|
||||||
description: Single-connection benchmark results — codec, framing, end-to-end queries, vs IfxPy.
|
description: Single-connection benchmark results for codec, framing, end-to-end queries, and IfxPy.
|
||||||
sidebar:
|
sidebar:
|
||||||
order: 5
|
order: 5
|
||||||
---
|
---
|
||||||
|
|||||||
@ -13,7 +13,7 @@ sidebar:
|
|||||||
| `IFX_DEBUG_WIRE` | unset | When set to a truthy value, log wire-level PDU framing to stderr. Verbose; for debugging only. |
|
| `IFX_DEBUG_WIRE` | unset | When set to a truthy value, log wire-level PDU framing to stderr. Verbose; for debugging only. |
|
||||||
| `IFX_PROTOCOL_TRACE` | unset | When set to a path, write annotated wire captures to the file. |
|
| `IFX_PROTOCOL_TRACE` | unset | When set to a path, write annotated wire captures to the file. |
|
||||||
| `IFX_DISABLE_PIPELINE` | unset | Disables pipelined `executemany` (Phase 33). Use only to A/B-measure. |
|
| `IFX_DISABLE_PIPELINE` | unset | Disables pipelined `executemany` (Phase 33). Use only to A/B-measure. |
|
||||||
| `INFORMIXSERVER` | — | Read by `connect()` if `server=` is not provided. |
|
| `INFORMIXSERVER` | *(none)* | Read by `connect()` if `server=` is not provided. |
|
||||||
|
|
||||||
Environment variables are read at connection construction. Changing them at runtime doesn't affect existing connections.
|
Environment variables are read at connection construction. Changing them at runtime doesn't affect existing connections.
|
||||||
|
|
||||||
@ -64,4 +64,4 @@ informix_db.connect(
|
|||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
`CLIENT_LOCALE` is set automatically from `client_locale=` — don't put it in `env=`.
|
`CLIENT_LOCALE` is set automatically from `client_locale=`, so don't put it in `env=`.
|
||||||
|
|||||||
@ -9,7 +9,7 @@ sidebar:
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `SMALLINT` / `INT` / `SERIAL` | `int` | Arbitrary precision on the Python side. |
|
| `SMALLINT` / `INT` / `SERIAL` | `int` | Arbitrary precision on the Python side. |
|
||||||
| `BIGINT` / `BIGSERIAL` | `int` | 8-byte two's complement on the wire. |
|
| `BIGINT` / `BIGSERIAL` | `int` | 8-byte two's complement on the wire. |
|
||||||
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer — a different wire format from `BIGINT`, not an alias. See below. |
|
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer, a different wire format from `BIGINT` rather than an alias. See below. |
|
||||||
| `FLOAT` / `SMALLFLOAT` | `float` | IEEE 754 double / single. |
|
| `FLOAT` / `SMALLFLOAT` | `float` | IEEE 754 double / single. |
|
||||||
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | Exact precision preserved. |
|
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | Exact precision preserved. |
|
||||||
| `CHAR` / `NCHAR` | `str` | Fixed width, space-padded; trailing spaces stripped on decode. |
|
| `CHAR` / `NCHAR` | `str` | Fixed width, space-padded; trailing spaces stripped on decode. |
|
||||||
@ -18,11 +18,11 @@ sidebar:
|
|||||||
| `DATE` | `datetime.date` | |
|
| `DATE` | `datetime.date` | |
|
||||||
| `DATETIME YEAR TO …` | `datetime.datetime` / `datetime.time` / `datetime.date` | The Python type depends on the field range. |
|
| `DATETIME YEAR TO …` | `datetime.datetime` / `datetime.time` / `datetime.date` | The Python type depends on the field range. |
|
||||||
| `INTERVAL DAY TO FRACTION` | `datetime.timedelta` | |
|
| `INTERVAL DAY TO FRACTION` | `datetime.timedelta` | |
|
||||||
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type — `datetime.timedelta` can't represent year-month intervals. |
|
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type, because `datetime.timedelta` can't represent year-month intervals. |
|
||||||
| `BYTE` / `TEXT` (legacy in-row blobs) | `bytes` / `str` | |
|
| `BYTE` / `TEXT` (legacy in-row blobs) | `bytes` / `str` | |
|
||||||
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
|
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators, 72 bytes. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
|
||||||
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string — **not** decomposed into fields. See below. |
|
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string, **not** decomposed into fields. See below. |
|
||||||
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema — **not** iterable. See below. |
|
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema, **not** iterable. See below. |
|
||||||
| `NULL` | `None` | |
|
| `NULL` | `None` | |
|
||||||
|
|
||||||
## INT8 and SERIAL8 are not BIGINT
|
## INT8 and SERIAL8 are not BIGINT
|
||||||
@ -34,16 +34,20 @@ Informix has two unrelated 64-bit integer types and they share nothing on the wi
|
|||||||
| `BIGINT` / `BIGSERIAL` (type 52 / 53) | 8 bytes, two's complement, big-endian |
|
| `BIGINT` / `BIGSERIAL` (type 52 / 53) | 8 bytes, two's complement, big-endian |
|
||||||
| `INT8` / `SERIAL8` (type 17 / 18) | 10 bytes, sign-magnitude, halves stored high-last |
|
| `INT8` / `SERIAL8` (type 17 / 18) | 10 bytes, sign-magnitude, halves stored high-last |
|
||||||
|
|
||||||
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `−n` with *identical* magnitude bytes and the sign in a leading word — decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
|
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `−n` with *identical* magnitude bytes and the sign in a leading word. Decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
|
||||||
|
|
||||||
`INT8`/`SERIAL8` are common in schemas predating Informix 11.50, which is when `BIGINT` arrived.
|
`INT8`/`SERIAL8` are common in schemas predating Informix 11.50, which is when `BIGINT` arrived.
|
||||||
|
|
||||||
:::caution[Upgrade if you use these types]
|
:::caution[Upgrade if you use these types]
|
||||||
`2026.08.27` fixed three decoding bugs: `INT8`/`SERIAL8` weren't decoded at all and returned raw `bytes`; `NCHAR` lost its first character; and `BOOLEAN` corrupted every column after it.
|
`2026.08.27` fixed three decoding bugs: `INT8`/`SERIAL8` weren't decoded at all and returned raw `bytes`; `NCHAR` lost its first character; and `BOOLEAN` corrupted every column after it.
|
||||||
|
|
||||||
`2026.08.31` fixed two more in `LVARCHAR` framing — a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
|
`2026.08.31` fixed two more in `LVARCHAR` framing: a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
|
||||||
|
|
||||||
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe — upgrade to `2026.08.31`.
|
`2026.09.02` fixed two more, both found by a new check that verifies each row consumed exactly its own payload. Smart LOBs were read as a flat 72-byte field when they actually occupy 149 bytes populated and 5 when NULL, so a `BLOB` or `CLOB` anywhere but the **last** column shifted every column after it. A NULL `SET` / `MULTISET` / `LIST` / `ROW` skipped its length field and did the same. That release also made a mismatch raise where it happens, naming the column and the byte delta, instead of handing back plausible-looking wrong values.
|
||||||
|
|
||||||
|
`2026.09.03` fixed one on the **write** path rather than the read path, which makes it the only entry here that affects data already stored. The `:N` placeholder rewriter was a regular expression and could not see a string literal, so it rewrote the inside of one: `'http://host:8080/x'` was stored as `'http://host?/x'`. Any `HH:MM` time, URL with a port, or `key:value` string was affected, whichever placeholder style you used. If you have written such values through this driver, they are worth checking.
|
||||||
|
|
||||||
|
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe. If you select a `BLOB` or `CLOB` in any position other than last, or a nullable collection column, nothing before `2026.09.02` is safe. Upgrade.
|
||||||
:::
|
:::
|
||||||
|
|
||||||
## LVARCHAR wire framing
|
## LVARCHAR wire framing
|
||||||
@ -54,7 +58,7 @@ Worth knowing if you're reading captures. Every Informix server we've tested des
|
|||||||
[1-byte null indicator][4-byte length][content]
|
[1-byte null indicator][4-byte length][content]
|
||||||
```
|
```
|
||||||
|
|
||||||
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte — both carry a length of zero.
|
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte, since both carry a length of zero.
|
||||||
|
|
||||||
## DATETIME field ranges
|
## DATETIME field ranges
|
||||||
|
|
||||||
@ -68,17 +72,17 @@ Informix's `DATETIME YEAR TO X` is field-range typed. The Python type returned d
|
|||||||
|
|
||||||
## Decimal precision
|
## Decimal precision
|
||||||
|
|
||||||
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly — no float intermediate.
|
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly, with no float intermediate.
|
||||||
|
|
||||||
For binding `Decimal` values into INSERT/UPDATE, the driver uses the column's declared scale. Pass `Decimal` for exact values; `float` works but may cause rounding at the column scale.
|
For binding `Decimal` values into INSERT/UPDATE, the driver uses the column's declared scale. Pass `Decimal` for exact values; `float` works but may cause rounding at the column scale.
|
||||||
|
|
||||||
## NULL
|
## NULL
|
||||||
|
|
||||||
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL — `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
|
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL, because `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
|
||||||
|
|
||||||
## Type extensions
|
## Type extensions
|
||||||
|
|
||||||
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count** — mirroring the server's own representation — with `years` and `remainder_months` available as derived properties:
|
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count**, mirroring the server's own representation, with `years` and `remainder_months` available as derived properties:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from informix_db import IntervalYM
|
from informix_db import IntervalYM
|
||||||
@ -100,8 +104,8 @@ Negative intervals are supported; the sign lives on `months` and propagates to b
|
|||||||
`informix_db.RowValue` and `informix_db.CollectionValue` are returned for `ROW`, `SET`, `MULTISET`, and `LIST` columns. Both are **opaque wrappers, not decomposed values**:
|
`informix_db.RowValue` and `informix_db.CollectionValue` are returned for `ROW`, `SET`, `MULTISET`, and `LIST` columns. Both are **opaque wrappers, not decomposed values**:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
row_val.raw # bytes — the server's textual representation, e.g. b"ROW('Alice',30)"
|
row_val.raw # bytes: the server's textual representation, e.g. b"ROW('Alice',30)"
|
||||||
row_val.schema # str — the column's declared schema
|
row_val.schema # str: the column's declared schema
|
||||||
|
|
||||||
coll_val.raw # bytes, e.g. b'LIST{10,20,30}'
|
coll_val.raw # bytes, e.g. b'LIST{10,20,30}'
|
||||||
coll_val.kind # 'set' | 'multiset' | 'list' | 'collection'
|
coll_val.kind # 'set' | 'multiset' | 'list' | 'collection'
|
||||||
@ -110,7 +114,7 @@ coll_val.element_schema # str
|
|||||||
|
|
||||||
They are not iterable and do not expose fields by name. Fully parsing a composite type means reimplementing JDBC's `IfxComplexInput`, which we haven't done.
|
They are not iterable and do not expose fields by name. Fully parsing a composite type means reimplementing JDBC's `IfxComplexInput`, which we haven't done.
|
||||||
|
|
||||||
If you need individual fields today, project them in SQL — the server does the decomposition for you and you get ordinary typed columns back:
|
If you need individual fields today, project them in SQL. The server does the decomposition for you and you get ordinary typed columns back:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
cur.execute("SELECT person.name, person.age FROM staff") # str, int
|
cur.execute("SELECT person.name, person.age FROM staff") # str, int
|
||||||
|
|||||||
@ -55,7 +55,7 @@ docker logs -f informix-dev
|
|||||||
```
|
```
|
||||||
|
|
||||||
<Aside type="tip">
|
<Aside type="tip">
|
||||||
The `--privileged` flag is required by the dev image — it tries to manage shared memory limits. For production servers this isn't a thing.
|
The `--privileged` flag is required by the dev image, which tries to manage shared memory limits. For production servers this isn't a thing.
|
||||||
</Aside>
|
</Aside>
|
||||||
|
|
||||||
## 3. Run your first query
|
## 3. Run your first query
|
||||||
@ -88,7 +88,7 @@ Then:
|
|||||||
python hello.py
|
python hello.py
|
||||||
```
|
```
|
||||||
|
|
||||||
You should see five rows from Informix's system catalog. If you do, congratulations — you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
|
You should see five rows from Informix's system catalog. If you do, congratulations: you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
|
||||||
|
|
||||||
## What just happened
|
## What just happened
|
||||||
|
|
||||||
@ -119,7 +119,12 @@ with informix_db.connect(host="127.0.0.1", port=9088, user="informix",
|
|||||||
print(cur.fetchone())
|
print(cur.fetchone())
|
||||||
```
|
```
|
||||||
|
|
||||||
`?` and `:1` both work — Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
|
`?` and `:1` both work. Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
|
||||||
|
|
||||||
|
Rows come back as plain tuples. If you would rather read them by column
|
||||||
|
name, pass `row_factory=informix_db.Row` to `connect()` and you get
|
||||||
|
`row["name"]` and `row.name` alongside `row[0]`. See
|
||||||
|
[the API reference](/reference/api/#rows).
|
||||||
|
|
||||||
## 5. Use the connection pool
|
## 5. Use the connection pool
|
||||||
|
|
||||||
@ -144,7 +149,7 @@ with pool.connection() as conn:
|
|||||||
pool.close()
|
pool.close()
|
||||||
```
|
```
|
||||||
|
|
||||||
The pool is thread-safe and has a per-connection wire lock — accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
|
The pool is thread-safe and has a per-connection wire lock, so accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
|
||||||
|
|
||||||
## What's next
|
## What's next
|
||||||
|
|
||||||
@ -154,7 +159,7 @@ The pool is thread-safe and has a per-connection wire lock — accidental sharin
|
|||||||
|
|
||||||
2. **Connecting to production?** [Connect with TLS →](/how-to/tls/) covers TLS-listener configuration and bring-your-own-context patterns.
|
2. **Connecting to production?** [Connect with TLS →](/how-to/tls/) covers TLS-listener configuration and bring-your-own-context patterns.
|
||||||
|
|
||||||
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/) — and the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/), including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||||
|
|
||||||
4. **Migrating from IfxPy?** [Migrate from IfxPy →](/how-to/migrate-from-ifxpy/) covers the API differences and the things IfxPy does that we don't (yet).
|
4. **Migrating from IfxPy?** [Migrate from IfxPy →](/how-to/migrate-from-ifxpy/) covers the API differences and the things IfxPy does that we don't (yet).
|
||||||
|
|
||||||
|
|||||||
@ -7,7 +7,7 @@ sidebar:
|
|||||||
|
|
||||||
import { Aside } from '@astrojs/starlight/components';
|
import { Aside } from '@astrojs/starlight/components';
|
||||||
|
|
||||||
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver — a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
|
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver, a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
|
||||||
|
|
||||||
Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Informix Developer Edition Docker container on the same host. Methodology and reproduction steps live in [`tests/benchmarks/compare/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare) in the repo.
|
Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Informix Developer Edition Docker container on the same host. Methodology and reproduction steps live in [`tests/benchmarks/compare/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare) in the repo.
|
||||||
|
|
||||||
@ -33,9 +33,9 @@ Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Inf
|
|||||||
|
|
||||||
### Bulk inserts at scale
|
### Bulk inserts at scale
|
||||||
|
|
||||||
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms** — `informix-driver` is 1.6× faster.
|
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms**, so `informix-driver` is 1.6× faster.
|
||||||
|
|
||||||
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row — N RTTs at ~80 µs each adds up to the 100 ms gap.
|
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row, and N RTTs at ~80 µs each adds up to the 100 ms gap.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# Both drivers
|
# Both drivers
|
||||||
@ -44,8 +44,8 @@ cur.executemany(
|
|||||||
rows, # list of 10_000 tuples
|
rows, # list of 10_000 tuples
|
||||||
)
|
)
|
||||||
|
|
||||||
# informix-driver: 161 ms — 10k PDUs sent, then 10k responses drained
|
# informix-driver: 161 ms (10k PDUs sent, then 10k responses drained)
|
||||||
# IfxPy: 259 ms — 10k round-trips, each blocking on response
|
# IfxPy: 259 ms (10k round-trips, each blocking on response)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Containerized deployment
|
### Containerized deployment
|
||||||
@ -62,7 +62,7 @@ IfxPy's deployment surface is dramatically larger:
|
|||||||
- 92 MB IBM OneDB Client tarball
|
- 92 MB IBM OneDB Client tarball
|
||||||
- `setuptools < 58` build pin
|
- `setuptools < 58` build pin
|
||||||
- `LD_LIBRARY_PATH` configuration for four directories
|
- `LD_LIBRARY_PATH` configuration for four directories
|
||||||
- `libcrypt.so.1` (deprecated 2018 — missing on Arch, Fedora 35+, RHEL 9)
|
- `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9)
|
||||||
- C compiler in the build image
|
- C compiler in the build image
|
||||||
|
|
||||||
For slim images, multi-stage builds, FaaS deployments, or anywhere build-toolchain-on-the-runtime is friction, `informix-driver` is the only reasonable option.
|
For slim images, multi-stage builds, FaaS deployments, or anywhere build-toolchain-on-the-runtime is friction, `informix-driver` is the only reasonable option.
|
||||||
@ -88,13 +88,13 @@ async def main():
|
|||||||
rows = await cur.fetchall()
|
rows = await cur.fetchall()
|
||||||
```
|
```
|
||||||
|
|
||||||
IfxPy has no async support — every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
|
IfxPy has no async support, so every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
|
||||||
|
|
||||||
## When IfxPy wins
|
## When IfxPy wins
|
||||||
|
|
||||||
### Large analytical fetches
|
### Large analytical fetches
|
||||||
|
|
||||||
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 5–15% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock — meaningful but not disqualifying.
|
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 5–15% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock, which is meaningful but not disqualifying.
|
||||||
|
|
||||||
The gap is closing phase by phase:
|
The gap is closing phase by phase:
|
||||||
|
|
||||||
@ -109,7 +109,7 @@ If you're running analytical reports that pull millions of rows in a single SELE
|
|||||||
|
|
||||||
### Workloads built around CSDK extensions
|
### Workloads built around CSDK extensions
|
||||||
|
|
||||||
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API — see [the migration guide](/how-to/migrate-from-ifxpy/).
|
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API. See [the migration guide](/how-to/migrate-from-ifxpy/).
|
||||||
|
|
||||||
## Methodology
|
## Methodology
|
||||||
|
|
||||||
@ -117,7 +117,7 @@ Benchmarks are pytest-benchmark fixtures in `tests/benchmarks/compare/` against
|
|||||||
|
|
||||||
Reported numbers are **median over 10+ rounds**, with IQR included. Why median over mean: the first round of any run includes JIT warmup, page-cache miss, and a TCP slow-start round-trip. The mean is contaminated by these one-shot costs in a way that misrepresents steady-state behavior. Median + IQR is what we report.
|
Reported numbers are **median over 10+ rounds**, with IQR included. Why median over mean: the first round of any run includes JIT warmup, page-cache miss, and a TCP slow-start round-trip. The mean is contaminated by these one-shot costs in a way that misrepresents steady-state behavior. Median + IQR is what we report.
|
||||||
|
|
||||||
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise — a fair reading is "5–15% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
|
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise. A fair reading is "5–15% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
|
||||||
|
|
||||||
To reproduce:
|
To reproduce:
|
||||||
|
|
||||||
@ -146,4 +146,4 @@ Use IfxPy when:
|
|||||||
- You're running large analytical SELECTs and the 5–15% decode-side gap matters
|
- You're running large analytical SELECTs and the 5–15% decode-side gap matters
|
||||||
- You're constrained to Python ≤ 3.11 anyway
|
- You're constrained to Python ≤ 3.11 anyway
|
||||||
|
|
||||||
For everything else — the cost-benefit favors `pip install informix-driver`.
|
For everything else, the cost-benefit favors `pip install informix-driver`.
|
||||||
|
|||||||
@ -8,7 +8,7 @@ sidebar:
|
|||||||
|
|
||||||
The existing tools were not my style.
|
The existing tools were not my style.
|
||||||
|
|
||||||
Every Informix driver in any language — `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix` — wraps either IBM's C Client SDK or the JDBC JAR. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
|
Every Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That covers `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
|
||||||
|
|
||||||
## The problem with IBM's C SDK
|
## The problem with IBM's C SDK
|
||||||
|
|
||||||
@ -19,19 +19,19 @@ The IBM Informix Client SDK (CSDK), now packaged as part of OneDB Client, is a 9
|
|||||||
- Permissive `CFLAGS` for the C extension build
|
- Permissive `CFLAGS` for the C extension build
|
||||||
- Manual download of the 92 MB ODBC tarball
|
- Manual download of the 92 MB ODBC tarball
|
||||||
- Four `LD_LIBRARY_PATH` directories
|
- Four `LD_LIBRARY_PATH` directories
|
||||||
- `libcrypt.so.1` — deprecated in 2018, missing on Arch, Fedora 35+, RHEL 9
|
- `libcrypt.so.1`, deprecated in 2018 and missing on Arch, Fedora 35+, RHEL 9
|
||||||
|
|
||||||
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db` — the distribution name dodges PyPI's 2008-vintage `informixdb` package, the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
|
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db`; the distribution name dodges PyPI's 2008-vintage `informixdb` package, while the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
|
||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly — the same protocol IBM's JDBC driver uses, the same protocol the CSDK speaks under the hood. No native code is in the thread of execution.
|
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly. It is the same protocol IBM's JDBC driver uses, and the same one the CSDK speaks under the hood. No native code is in the thread of execution.
|
||||||
|
|
||||||
The wire protocol was reverse-engineered through three sources:
|
The wire protocol was reverse-engineered through three sources:
|
||||||
|
|
||||||
1. **Decompiled IBM JDBC driver** (`com.informix.jdbc.IfxConnection` and friends), used as a clean-room reference for PDU shapes and protocol semantics.
|
1. **Decompiled IBM JDBC driver** (`com.informix.jdbc.IfxConnection` and friends), used as a clean-room reference for PDU shapes and protocol semantics.
|
||||||
2. **Annotated `socat` captures** of real client/server traffic against the IBM Informix Developer Edition Docker image.
|
2. **Annotated `socat` captures** of real client/server traffic against the IBM Informix Developer Edition Docker image.
|
||||||
3. **Differential testing** against `IfxPy` — every codec path is tested against the C driver's behavior on the same data.
|
3. **Differential testing** against `IfxPy`, so every codec path is checked against the C driver's behavior on the same data.
|
||||||
|
|
||||||
The result is a PEP 249 compliant driver with a sync API, an async API (FastAPI / asyncio compatible), a connection pool, TLS support, smart-LOB read/write, scrollable cursors, fast-path stored procedure invocation, and bulk-insert / bulk-fetch performance within ~10–60% of the C driver depending on workload.
|
The result is a PEP 249 compliant driver with a sync API, an async API (FastAPI / asyncio compatible), a connection pool, TLS support, smart-LOB read/write, scrollable cursors, fast-path stored procedure invocation, and bulk-insert / bulk-fetch performance within ~10–60% of the C driver depending on workload.
|
||||||
|
|
||||||
@ -76,12 +76,12 @@ Every finding from a system-wide failure-mode audit (data correctness, wire safe
|
|||||||
|
|
||||||
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
|
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
|
||||||
|
|
||||||
400+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 414/414 on **all three** of 12.10.FC12W1DE, 14.10.FC7W1DE, and 15.0.1.0.3DE — `make test-matrix` runs the lot.
|
470+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 472/472 on 15.0.1.0.3DE and 14.10.FC7W1DE, and 471/472 on 12.10.FC12W1DE (the single skip is a common table expression, which 12.10 predates). `make test-matrix` runs the lot.
|
||||||
|
|
||||||
That matrix exists because it turned out to be needed. A user reported corrupted result sets on Informix 12; the cause was three framing bugs that affected every version including the one we tested against, and they'd survived because no fixture used the affected types. Testing one server and inferring the rest is how that happens.
|
That matrix exists because it turned out to be needed. A user reported corrupted result sets on Informix 12; the cause was three framing bugs that affected every version including the one we tested against, and they'd survived because no fixture used the affected types. Testing one server and inferring the rest is how that happens.
|
||||||
|
|
||||||
## Read next
|
## Read next
|
||||||
|
|
||||||
- **[Install & first query →](/start/quickstart/)** — five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
|
- **[Install & first query →](/start/quickstart/)**: five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
|
||||||
- **[Compared to IfxPy →](/start/vs-ifxpy/)** — full head-to-head benchmarks, methodology, and reproduction.
|
- **[Compared to IfxPy →](/start/vs-ifxpy/)**: full head-to-head benchmarks, methodology, and reproduction.
|
||||||
- **[Architecture →](/explain/architecture/)** — how the layers stack: socket, framing, codec, resultset, cursor.
|
- **[Architecture →](/explain/architecture/)**: how the layers stack, from socket through framing, codec, resultset, and cursor.
|
||||||
|
|||||||
@ -243,7 +243,7 @@
|
|||||||
color: var(--sl-color-gray-2);
|
color: var(--sl-color-gray-2);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Supported Systems "joint" badge — appears below every page's footer */
|
/* Supported Systems "joint" badge, appears below every page's footer */
|
||||||
.ifx-ss-badge {
|
.ifx-ss-badge {
|
||||||
margin-top: 3.5rem;
|
margin-top: 3.5rem;
|
||||||
padding: 0;
|
padding: 0;
|
||||||
@ -337,7 +337,7 @@
|
|||||||
* ============================================================ */
|
* ============================================================ */
|
||||||
|
|
||||||
@media (max-width: 640px) {
|
@media (max-width: 640px) {
|
||||||
/* Defensive guard against any descendant forcing horizontal scroll —
|
/* Defensive guard against any descendant forcing horizontal scroll:
|
||||||
the wire-dump's white-space: pre content was overflowing the
|
the wire-dump's white-space: pre content was overflowing the
|
||||||
hero column and propagating up to the page. overflow-x: hidden
|
hero column and propagating up to the page. overflow-x: hidden
|
||||||
on .ifx-hero contains it without affecting page-level scroll. */
|
on .ifx-hero contains it without affecting page-level scroll. */
|
||||||
|
|||||||
@ -1,7 +1,7 @@
|
|||||||
/*
|
/*
|
||||||
* informix-driver docs theme
|
* informix-driver docs theme
|
||||||
* - Charcoal base (no purple gradients, ever)
|
* - Charcoal base (no purple gradients, ever)
|
||||||
* - Amber accent — CRT-monitor nod, distinct from sibling sites' cyan
|
* - Amber accent, a CRT-monitor nod distinct from sibling sites' cyan
|
||||||
* - Inter for body, IBM Plex Mono for technical bytes
|
* - Inter for body, IBM Plex Mono for technical bytes
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@ -82,7 +82,7 @@
|
|||||||
--sl-color-hairline-shade: rgba(120, 80, 12, 0.28);
|
--sl-color-hairline-shade: rgba(120, 80, 12, 0.28);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Tighten heading rhythm — Starlight defaults are a touch loose for technical docs */
|
/* Tighten heading rhythm; Starlight defaults are a touch loose for technical docs */
|
||||||
.sl-markdown-content h2 {
|
.sl-markdown-content h2 {
|
||||||
margin-top: 2.5rem;
|
margin-top: 2.5rem;
|
||||||
border-top: 1px solid var(--sl-color-hairline-light);
|
border-top: 1px solid var(--sl-color-hairline-light);
|
||||||
@ -104,7 +104,7 @@
|
|||||||
border-radius: 4px;
|
border-radius: 4px;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Tables: dense, technical, with amber column rules — for type-mapping & benchmark tables */
|
/* Tables: dense, technical, amber column rules, for type-mapping & benchmark tables */
|
||||||
.sl-markdown-content table {
|
.sl-markdown-content table {
|
||||||
border-collapse: collapse;
|
border-collapse: collapse;
|
||||||
font-variant-numeric: tabular-nums;
|
font-variant-numeric: tabular-nums;
|
||||||
@ -124,7 +124,7 @@
|
|||||||
padding: 0.5rem 0.75rem;
|
padding: 0.5rem 0.75rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Anchor links — underline, no rainbow */
|
/* Anchor links: underline, no rainbow */
|
||||||
.sl-markdown-content a:not(.sl-anchor-link) {
|
.sl-markdown-content a:not(.sl-anchor-link) {
|
||||||
text-decoration: underline;
|
text-decoration: underline;
|
||||||
text-decoration-color: var(--sl-color-accent);
|
text-decoration-color: var(--sl-color-accent);
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "informix-driver"
|
name = "informix-driver"
|
||||||
version = "2026.09.02"
|
version = "2026.09.03"
|
||||||
description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries."
|
description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
license = { text = "MIT" }
|
license = { text = "MIT" }
|
||||||
|
|||||||
@ -50,6 +50,7 @@ from .pool import (
|
|||||||
PoolTimeoutError,
|
PoolTimeoutError,
|
||||||
create_pool,
|
create_pool,
|
||||||
)
|
)
|
||||||
|
from .rows import Row
|
||||||
|
|
||||||
# PEP 249 module-level globals
|
# PEP 249 module-level globals
|
||||||
apilevel = "2.0"
|
apilevel = "2.0"
|
||||||
@ -85,6 +86,7 @@ __all__ = [
|
|||||||
"PoolClosedError",
|
"PoolClosedError",
|
||||||
"PoolTimeoutError",
|
"PoolTimeoutError",
|
||||||
"ProgrammingError",
|
"ProgrammingError",
|
||||||
|
"Row",
|
||||||
"RowValue",
|
"RowValue",
|
||||||
"ServerCapabilities",
|
"ServerCapabilities",
|
||||||
"Warning",
|
"Warning",
|
||||||
@ -111,6 +113,7 @@ def connect(
|
|||||||
client_locale: str = "en_US.8859-1",
|
client_locale: str = "en_US.8859-1",
|
||||||
env: dict[str, str] | None = None,
|
env: dict[str, str] | None = None,
|
||||||
autocommit: bool = False,
|
autocommit: bool = False,
|
||||||
|
row_factory: object | None = None,
|
||||||
tls: bool | ssl.SSLContext = False,
|
tls: bool | ssl.SSLContext = False,
|
||||||
tls_server_hostname: str | None = None,
|
tls_server_hostname: str | None = None,
|
||||||
) -> Connection:
|
) -> Connection:
|
||||||
@ -153,4 +156,5 @@ def connect(
|
|||||||
client_locale=client_locale,
|
client_locale=client_locale,
|
||||||
env=env,
|
env=env,
|
||||||
autocommit=autocommit,
|
autocommit=autocommit,
|
||||||
|
row_factory=row_factory,
|
||||||
)
|
)
|
||||||
|
|||||||
@ -283,6 +283,7 @@ class Connection:
|
|||||||
client_locale: str = "en_US.8859-1",
|
client_locale: str = "en_US.8859-1",
|
||||||
env: dict[str, str] | None = None,
|
env: dict[str, str] | None = None,
|
||||||
autocommit: bool = False, # honored from Phase 3 onward
|
autocommit: bool = False, # honored from Phase 3 onward
|
||||||
|
row_factory: object | None = None,
|
||||||
tls: bool | ssl.SSLContext = False,
|
tls: bool | ssl.SSLContext = False,
|
||||||
tls_server_hostname: str | None = None,
|
tls_server_hostname: str | None = None,
|
||||||
):
|
):
|
||||||
@ -337,6 +338,10 @@ class Connection:
|
|||||||
# before the next DML in non-autocommit mode. We default to "no
|
# before the next DML in non-autocommit mode. We default to "no
|
||||||
# open txn" — the first DML will trigger SQ_BEGIN.
|
# open txn" — the first DML will trigger SQ_BEGIN.
|
||||||
self._in_transaction = False
|
self._in_transaction = False
|
||||||
|
# Default row type for cursors from this connection. None means
|
||||||
|
# plain tuples, which is the zero-cost default; see
|
||||||
|
# informix_db.rows for the opt-in named-access type.
|
||||||
|
self.row_factory = row_factory
|
||||||
# Tri-state: True after first successful SQ_BEGIN, False after
|
# Tri-state: True after first successful SQ_BEGIN, False after
|
||||||
# an unlogged-DB rejection (-201). None until we've tried.
|
# an unlogged-DB rejection (-201). None until we've tried.
|
||||||
# Used to avoid repeatedly probing on unlogged DBs.
|
# Used to avoid repeatedly probing on unlogged DBs.
|
||||||
|
|||||||
@ -50,6 +50,7 @@ from .exceptions import (
|
|||||||
NotSupportedError,
|
NotSupportedError,
|
||||||
ProgrammingError,
|
ProgrammingError,
|
||||||
)
|
)
|
||||||
|
from .rows import Row, make_row_class
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from .connections import Connection
|
from .connections import Connection
|
||||||
@ -60,7 +61,6 @@ if TYPE_CHECKING:
|
|||||||
_cursor_counter = itertools.count(1)
|
_cursor_counter = itertools.count(1)
|
||||||
|
|
||||||
|
|
||||||
_NUMERIC_PLACEHOLDER_RE = __import__("re").compile(r":(\d+)")
|
|
||||||
|
|
||||||
|
|
||||||
# Phase 28: pre-built CLOSE and RELEASE PDU bytes for cursor finalizers.
|
# Phase 28: pre-built CLOSE and RELEASE PDU bytes for cursor finalizers.
|
||||||
@ -247,17 +247,103 @@ def _finalize_cursor(
|
|||||||
conn._wire_lock.release()
|
conn._wire_lock.release()
|
||||||
|
|
||||||
|
|
||||||
|
_ASCII_DIGITS = frozenset("0123456789")
|
||||||
|
|
||||||
|
|
||||||
def _rewrite_numeric_to_qmark(sql: str) -> str:
|
def _rewrite_numeric_to_qmark(sql: str) -> str:
|
||||||
"""Convert ``:1`` / ``:2`` placeholders (paramstyle="numeric") to ``?``.
|
"""Convert ``:1`` / ``:2`` placeholders (paramstyle="numeric") to ``?``.
|
||||||
|
|
||||||
Informix's wire protocol uses ``?`` natively. Since we expose
|
Informix's wire protocol uses ``?`` natively, and we advertise
|
||||||
``paramstyle="numeric"`` in the public API (matches Informix
|
``paramstyle="numeric"`` to match the ESQL/C convention, so the
|
||||||
ESQL/C convention), we rewrite before sending. Trivial cases only
|
placeholders are rewritten on the way out.
|
||||||
— strings and comments are NOT escaped, so SQL containing literal
|
|
||||||
``:1`` inside string literals will be wrongly substituted. Phase 5
|
This used to be ``re.sub(r":(\\d+)", "?", sql)``, which cannot see a
|
||||||
can add a proper SQL tokenizer.
|
string literal and so rewrote the contents of one. Any ``HH:MM``
|
||||||
|
time, any URL with a port, any aspect ratio::
|
||||||
|
|
||||||
|
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
|
||||||
|
stored as 'http://host?/x'
|
||||||
|
|
||||||
|
Nothing about that is opt-in. The rewrite runs whenever a statement
|
||||||
|
has parameters, whatever placeholder style the caller actually used,
|
||||||
|
so writing ``?`` everywhere and never touching numeric style did not
|
||||||
|
protect you. It also changed the placeholder *count* while
|
||||||
|
``num_qmarks`` was still computed from ``len(params)``, leaving the
|
||||||
|
driver and the server disagreeing about how many binds exist.
|
||||||
|
|
||||||
|
So: a single pass that substitutes only outside quotes and comments.
|
||||||
|
The lexical rules are Informix's own, measured against 12.10, 14.10
|
||||||
|
and 15 rather than assumed from standard SQL:
|
||||||
|
|
||||||
|
* ``''`` doubling escapes a quote inside ``'...'``. A backslash does
|
||||||
|
**not** escape anything; ``'a\'b'`` is an unterminated string and
|
||||||
|
the server answers ``-282``. A Postgres-style scanner that honours
|
||||||
|
``\'`` would desync here and corrupt everything after it.
|
||||||
|
* ``"..."`` is a delimited identifier or a string depending on
|
||||||
|
``DELIMIDENT``. Either way its contents are not ours to touch.
|
||||||
|
* ``--`` runs to end of line, ``/* */`` does **not** nest (the first
|
||||||
|
``*/`` closes it; nesting is a syntax error), and ``{ }`` is a
|
||||||
|
comment too.
|
||||||
|
* ``::`` is the cast operator and is stepped over as a unit, so it
|
||||||
|
can never be read as the start of a placeholder.
|
||||||
|
|
||||||
|
An unterminated quote or comment consumes the rest of the string and
|
||||||
|
substitutes nothing further. That is deliberate: under-substituting
|
||||||
|
leaves the server to reject SQL that was already malformed, while
|
||||||
|
guessing would corrupt a literal.
|
||||||
"""
|
"""
|
||||||
return _NUMERIC_PLACEHOLDER_RE.sub("?", sql)
|
if ":" not in sql:
|
||||||
|
return sql
|
||||||
|
out: list[str] = []
|
||||||
|
i = 0
|
||||||
|
n = len(sql)
|
||||||
|
while i < n:
|
||||||
|
ch = sql[i]
|
||||||
|
if ch in ("'", '"'):
|
||||||
|
j = i + 1
|
||||||
|
while j < n:
|
||||||
|
if sql[j] == ch:
|
||||||
|
if j + 1 < n and sql[j + 1] == ch:
|
||||||
|
j += 2 # doubled quote, still inside
|
||||||
|
continue
|
||||||
|
j += 1
|
||||||
|
break
|
||||||
|
j += 1
|
||||||
|
out.append(sql[i:j])
|
||||||
|
i = j
|
||||||
|
elif ch == "-" and sql.startswith("--", i):
|
||||||
|
j = sql.find("\n", i)
|
||||||
|
j = n if j == -1 else j
|
||||||
|
out.append(sql[i:j])
|
||||||
|
i = j
|
||||||
|
elif ch == "/" and sql.startswith("/*", i):
|
||||||
|
j = sql.find("*/", i + 2)
|
||||||
|
j = n if j == -1 else j + 2
|
||||||
|
out.append(sql[i:j])
|
||||||
|
i = j
|
||||||
|
elif ch == "{":
|
||||||
|
j = sql.find("}", i)
|
||||||
|
j = n if j == -1 else j + 1
|
||||||
|
out.append(sql[i:j])
|
||||||
|
i = j
|
||||||
|
elif ch == ":":
|
||||||
|
if sql.startswith("::", i):
|
||||||
|
out.append("::")
|
||||||
|
i += 2
|
||||||
|
continue
|
||||||
|
j = i + 1
|
||||||
|
while j < n and sql[j] in _ASCII_DIGITS:
|
||||||
|
j += 1
|
||||||
|
if j > i + 1:
|
||||||
|
out.append("?")
|
||||||
|
i = j
|
||||||
|
else:
|
||||||
|
out.append(ch)
|
||||||
|
i += 1
|
||||||
|
else:
|
||||||
|
out.append(ch)
|
||||||
|
i += 1
|
||||||
|
return "".join(out)
|
||||||
|
|
||||||
|
|
||||||
def _generate_cursor_name() -> str:
|
def _generate_cursor_name() -> str:
|
||||||
@ -288,6 +374,9 @@ class Cursor:
|
|||||||
# manipulation. Two-mode cursor; the same surface API works
|
# manipulation. Two-mode cursor; the same surface API works
|
||||||
# for both.
|
# for both.
|
||||||
self._scrollable = scrollable
|
self._scrollable = scrollable
|
||||||
|
# Inherited from the connection, overridable per cursor. See
|
||||||
|
# informix_db.rows for what this costs and why it is opt-in.
|
||||||
|
self.row_factory = connection.row_factory
|
||||||
self._description: list[tuple] | None = None
|
self._description: list[tuple] | None = None
|
||||||
self._columns: list[ColumnInfo] = []
|
self._columns: list[ColumnInfo] = []
|
||||||
self._column_readers: list[tuple] | None = None # Phase 37
|
self._column_readers: list[tuple] | None = None # Phase 37
|
||||||
@ -328,6 +417,8 @@ class Cursor:
|
|||||||
# DESCRIBE's statement-type field. Decides whether a cursor is
|
# DESCRIBE's statement-type field. Decides whether a cursor is
|
||||||
# opened -- see _produces_result_set.
|
# opened -- see _produces_result_set.
|
||||||
self._statement_type: int = 0
|
self._statement_type: int = 0
|
||||||
|
# Per-result-set row class when row_factory is set, else None.
|
||||||
|
self._row_class: type | None = None
|
||||||
# Phase 10: smart-LOB read via ``lotofile(col, path, 'client')``.
|
# Phase 10: smart-LOB read via ``lotofile(col, path, 'client')``.
|
||||||
# The server orchestrates a SQ_FILE (98) protocol where it tells
|
# The server orchestrates a SQ_FILE (98) protocol where it tells
|
||||||
# us to "open file X, write these bytes, close". We emulate the
|
# us to "open file X, write these bytes, close". We emulate the
|
||||||
@ -429,6 +520,7 @@ class Cursor:
|
|||||||
self._rowcount = -1
|
self._rowcount = -1
|
||||||
self._rows = []
|
self._rows = []
|
||||||
self._row_index = -1 # before-first-row
|
self._row_index = -1 # before-first-row
|
||||||
|
self._row_class = None
|
||||||
self._statement_type = 0
|
self._statement_type = 0
|
||||||
self._statement_already_done = False
|
self._statement_already_done = False
|
||||||
|
|
||||||
@ -468,6 +560,8 @@ class Cursor:
|
|||||||
else:
|
else:
|
||||||
self._execute_dml()
|
self._execute_dml()
|
||||||
|
|
||||||
|
self._row_class = self._resolve_row_class()
|
||||||
|
|
||||||
# The statement succeeded. If it was transaction control, the
|
# The statement succeeded. If it was transaction control, the
|
||||||
# server's transaction state just changed and the connection has
|
# server's transaction state just changed and the connection has
|
||||||
# to know, or commit() and rollback() silently do nothing.
|
# to know, or commit() and rollback() silently do nothing.
|
||||||
@ -481,6 +575,25 @@ class Cursor:
|
|||||||
if self._description is not None:
|
if self._description is not None:
|
||||||
self._row_index = -1
|
self._row_index = -1
|
||||||
|
|
||||||
|
def _resolve_row_class(self) -> type | None:
|
||||||
|
"""Pick the class this result set's rows are handed back as.
|
||||||
|
|
||||||
|
``row_factory`` is either :class:`informix_db.Row` (or a subclass),
|
||||||
|
in which case the per-shape class is built and cached from the
|
||||||
|
column names, or any callable taking the name tuple and returning
|
||||||
|
something that takes a values tuple.
|
||||||
|
|
||||||
|
Returns ``None`` when no factory is set, which is the default and
|
||||||
|
keeps plain tuples on the hot path at zero cost.
|
||||||
|
"""
|
||||||
|
factory = self.row_factory
|
||||||
|
if factory is None or self._description is None:
|
||||||
|
return None
|
||||||
|
names = tuple(d[0] for d in self._description)
|
||||||
|
if isinstance(factory, type) and issubclass(factory, Row):
|
||||||
|
return make_row_class(names)
|
||||||
|
return factory(names)
|
||||||
|
|
||||||
def _note_transaction_control(self) -> None:
|
def _note_transaction_control(self) -> None:
|
||||||
"""Sync the connection's transaction flag after a successful execute.
|
"""Sync the connection's transaction flag after a successful execute.
|
||||||
|
|
||||||
@ -1302,7 +1415,8 @@ class Cursor:
|
|||||||
self._row_index = len(self._rows) # past-last
|
self._row_index = len(self._rows) # past-last
|
||||||
return None
|
return None
|
||||||
self._row_index = nxt
|
self._row_index = nxt
|
||||||
return self._rows[nxt]
|
row = self._rows[nxt]
|
||||||
|
return self._row_class(row) if self._row_class is not None else row
|
||||||
|
|
||||||
def fetchmany(self, size: int | None = None) -> list[tuple]:
|
def fetchmany(self, size: int | None = None) -> list[tuple]:
|
||||||
self._check_open()
|
self._check_open()
|
||||||
@ -1333,6 +1447,8 @@ class Cursor:
|
|||||||
return []
|
return []
|
||||||
start = self._row_index + 1
|
start = self._row_index + 1
|
||||||
out = self._rows[start:]
|
out = self._rows[start:]
|
||||||
|
if self._row_class is not None:
|
||||||
|
out = [self._row_class(r) for r in out]
|
||||||
self._row_index = len(self._rows)
|
self._row_index = len(self._rows)
|
||||||
return list(out)
|
return list(out)
|
||||||
|
|
||||||
@ -1513,7 +1629,7 @@ class Cursor:
|
|||||||
if scrolltype == 4 or is_last_probe:
|
if scrolltype == 4 or is_last_probe:
|
||||||
# SFETCH(LAST) — TUPID == total row count
|
# SFETCH(LAST) — TUPID == total row count
|
||||||
self._scroll_total_rows = self._last_tupid
|
self._scroll_total_rows = self._last_tupid
|
||||||
return row
|
return self._row_class(row) if self._row_class is not None else row
|
||||||
|
|
||||||
def close(self) -> None:
|
def close(self) -> None:
|
||||||
"""Close the cursor.
|
"""Close the cursor.
|
||||||
|
|||||||
180
src/informix_db/rows.py
Normal file
180
src/informix_db/rows.py
Normal file
@ -0,0 +1,180 @@
|
|||||||
|
"""Rows that can be read by position, by column name, or by attribute.
|
||||||
|
|
||||||
|
PEP 249 only requires a sequence, and a sequence is what the driver
|
||||||
|
returns by default. That is fine for ``SELECT a, b`` and steadily worse
|
||||||
|
as the projection grows: ``row[11]`` tells a reader nothing, and stays
|
||||||
|
correct only until somebody adds a column in the middle.
|
||||||
|
|
||||||
|
Opting in with ``row_factory=Row`` gives the shape ``pyodbc`` and
|
||||||
|
``mssql-python`` provide, all three at once::
|
||||||
|
|
||||||
|
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||||
|
cur.execute("SELECT tabid, tabname FROM systables")
|
||||||
|
row = cur.fetchone()
|
||||||
|
row[0], row["tabname"], row.tabname
|
||||||
|
|
||||||
|
It is opt-in rather than the default because it is not free, and the
|
||||||
|
driver's whole argument is that pure Python can stay within noise of the
|
||||||
|
C driver on bulk fetch. Measured on a 20,000-row five-column fetch:
|
||||||
|
**37.2 ms with tuples, 40.8 ms with Row**, so about 9%. Defaulting it on
|
||||||
|
would move the published 1.05-1.15x ratio against IfxPy to roughly
|
||||||
|
1.15-1.25x, which is not a trade to make on everybody's behalf.
|
||||||
|
|
||||||
|
Where the 9% goes, against a plain tuple: about 30 ns per row to
|
||||||
|
construct, about 39 ns per ``row[0]`` because supporting ``row["name"]``
|
||||||
|
means ``__getitem__`` is a Python method rather than C-level tuple
|
||||||
|
indexing, and about 13 ns per unpack since CPython's fast path for
|
||||||
|
``a, b = row`` applies to exact tuples and not to subclasses.
|
||||||
|
|
||||||
|
Worth it for readability in application code. Not worth paying in a bulk
|
||||||
|
export that never looks at a column by name, which is exactly why it is
|
||||||
|
a choice.
|
||||||
|
|
||||||
|
``Row`` subclasses ``tuple``, so ``row == (1, "x")`` is still true and
|
||||||
|
existing code keeps working unchanged.
|
||||||
|
|
||||||
|
**A column always beats a method of the same name.** ``tuple`` defines
|
||||||
|
``count`` and ``index``; this class adds ``keys``, ``_asdict`` and
|
||||||
|
``_fields``. A column named any of those would otherwise resolve to the
|
||||||
|
method and hand back a bound method instead of a value, which is exactly
|
||||||
|
the shape of the framing bugs this driver spent a fortnight removing: a
|
||||||
|
plausible-looking wrong answer, in silence. So every such name gets a
|
||||||
|
descriptor and the column wins, and the reserved set is *computed* from
|
||||||
|
the class rather than hand-listed, because hand-listing it is what
|
||||||
|
missed ``keys``, ``_asdict`` and ``_fields`` the first time round. The
|
||||||
|
methods stay reachable through the class: ``tuple.count(row, x)``,
|
||||||
|
``Row.keys(row)``.
|
||||||
|
|
||||||
|
The machinery itself is name-mangled (``__fields`` / ``__map``) so that
|
||||||
|
shadowing ``_fields`` cannot break ``repr`` or ``_asdict``.
|
||||||
|
|
||||||
|
Column names come from ``cursor.description``. Informix folds unquoted
|
||||||
|
identifiers to lower case, so ``SELECT Config_Key`` is reachable as
|
||||||
|
``row.config_key``. Expressions get server-generated names that are not
|
||||||
|
Python identifiers, like ``(count(*))``; those are reachable by
|
||||||
|
subscript but not as attributes. Duplicate names resolve to the first
|
||||||
|
occurrence, matching ``pyodbc``. Dunder names cannot be shadowed and
|
||||||
|
stay subscript-only, which no real schema should notice.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import operator
|
||||||
|
from functools import lru_cache
|
||||||
|
from typing import ClassVar
|
||||||
|
|
||||||
|
__all__ = ["Row", "make_row_class"]
|
||||||
|
|
||||||
|
|
||||||
|
class Row(tuple):
|
||||||
|
"""A result row addressable by position, name, or attribute.
|
||||||
|
|
||||||
|
Used as a ``row_factory``. The concrete class handed to each result
|
||||||
|
set is a subclass carrying that query's column names, built by
|
||||||
|
:func:`make_row_class`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
__slots__ = ()
|
||||||
|
|
||||||
|
# Name-mangled to ``_Row__fields`` / ``_Row__map`` so that a column
|
||||||
|
# called "_fields" can be shadowed without breaking the machinery
|
||||||
|
# that reads it. Set on the per-result-set subclass.
|
||||||
|
__fields: ClassVar[tuple[str, ...]] = ()
|
||||||
|
__map: ClassVar[dict[str, int]] = {}
|
||||||
|
|
||||||
|
def __getitem__(self, key):
|
||||||
|
# ``key.__class__ is str`` rather than isinstance: this runs on
|
||||||
|
# every subscript.
|
||||||
|
if key.__class__ is str:
|
||||||
|
try:
|
||||||
|
return tuple.__getitem__(self, self.__map[key])
|
||||||
|
except KeyError:
|
||||||
|
raise KeyError(
|
||||||
|
f"no column named {key!r}; this row has "
|
||||||
|
f"{list(self.__fields)}"
|
||||||
|
) from None
|
||||||
|
return tuple.__getitem__(self, key)
|
||||||
|
|
||||||
|
def __getattr__(self, name):
|
||||||
|
# Only reached when normal attribute lookup has already failed,
|
||||||
|
# so this costs nothing for names that do not collide.
|
||||||
|
try:
|
||||||
|
return tuple.__getitem__(self, self.__map[name])
|
||||||
|
except KeyError:
|
||||||
|
raise AttributeError(
|
||||||
|
f"no column named {name!r}; this row has "
|
||||||
|
f"{list(self.__fields)}"
|
||||||
|
) from None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def _fields(self) -> tuple[str, ...]:
|
||||||
|
"""Column names, in select order. Mirrors ``namedtuple._fields``."""
|
||||||
|
return self.__fields
|
||||||
|
|
||||||
|
def keys(self) -> tuple[str, ...]:
|
||||||
|
"""Column names, in select order."""
|
||||||
|
return self.__fields
|
||||||
|
|
||||||
|
def _asdict(self) -> dict:
|
||||||
|
"""A plain ``dict`` of the row.
|
||||||
|
|
||||||
|
On duplicate column names the last occurrence wins here, while
|
||||||
|
subscript access gives the first. A dict cannot represent both,
|
||||||
|
and quietly dropping a duplicate is better than raising on a
|
||||||
|
query that is otherwise fine.
|
||||||
|
"""
|
||||||
|
return dict(zip(self.__fields, self, strict=True))
|
||||||
|
|
||||||
|
def __repr__(self) -> str:
|
||||||
|
fields = self.__fields
|
||||||
|
if len(fields) != len(self):
|
||||||
|
return tuple.__repr__(self)
|
||||||
|
body = ", ".join(
|
||||||
|
f"{name}={value!r}"
|
||||||
|
for name, value in zip(fields, self, strict=True)
|
||||||
|
)
|
||||||
|
return f"Row({body})"
|
||||||
|
|
||||||
|
def __reduce__(self):
|
||||||
|
# The per-result-set class is created at runtime and cannot be
|
||||||
|
# pickled by reference, so rebuild it from the field names.
|
||||||
|
return (_rebuild_row, (self.__fields, tuple(self)))
|
||||||
|
|
||||||
|
|
||||||
|
# Every non-dunder attribute a Row already answers to. A column with one
|
||||||
|
# of these names gets a descriptor so the column wins. Computed, not
|
||||||
|
# hand-listed: the hand-listed version covered ``count`` and ``index``
|
||||||
|
# and silently missed ``keys``, ``_asdict`` and ``_fields``.
|
||||||
|
_RESERVED = frozenset(
|
||||||
|
name for name in dir(Row) if not name.startswith("__")
|
||||||
|
) - {"_Row__fields", "_Row__map"}
|
||||||
|
|
||||||
|
|
||||||
|
def _rebuild_row(fields: tuple[str, ...], values: tuple):
|
||||||
|
return make_row_class(fields)(values)
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=256)
|
||||||
|
def make_row_class(fields: tuple[str, ...]) -> type[Row]:
|
||||||
|
"""Build (and cache) the row class for one column-name shape.
|
||||||
|
|
||||||
|
Cached because a class per ``execute()`` would put a ``type()`` call
|
||||||
|
on the path of every small query, and applications run the same
|
||||||
|
handful of statement shapes over and over. Keyed on the names alone,
|
||||||
|
so two queries selecting the same columns share a class.
|
||||||
|
"""
|
||||||
|
namespace: dict = {
|
||||||
|
"__slots__": (),
|
||||||
|
"_Row__fields": fields,
|
||||||
|
# First occurrence wins on duplicates, matching pyodbc. Building
|
||||||
|
# the map in reverse and letting earlier entries overwrite later
|
||||||
|
# ones is the shortest way to say that.
|
||||||
|
"_Row__map": {
|
||||||
|
name: i for i, name in reversed(list(enumerate(fields)))
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for i, name in enumerate(fields):
|
||||||
|
if name in _RESERVED:
|
||||||
|
# A column of this name would otherwise resolve to a method.
|
||||||
|
namespace[name] = property(operator.itemgetter(i))
|
||||||
|
return type("Row", (Row,), namespace)
|
||||||
@ -214,3 +214,52 @@ async def test_closing_a_connection_stops_its_thread(
|
|||||||
f"connection threads outlived their connections "
|
f"connection threads outlived their connections "
|
||||||
f"({before} -> {threading.active_count()})"
|
f"({before} -> {threading.active_count()})"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_row_factory_reaches_the_async_paths(
|
||||||
|
conn_params: ConnParams,
|
||||||
|
) -> None:
|
||||||
|
"""The async layer wraps the sync cursor, so row_factory should flow
|
||||||
|
through untouched. Each of the four async fetch routes is a separate
|
||||||
|
call site, and async iteration goes through __anext__ rather than
|
||||||
|
__next__."""
|
||||||
|
import informix_db
|
||||||
|
|
||||||
|
conn = await aio.connect(row_factory=informix_db.Row, **_kw(conn_params))
|
||||||
|
try:
|
||||||
|
cur = await conn.cursor()
|
||||||
|
sql = "SELECT FIRST 2 tabid, tabname FROM systables ORDER BY tabid"
|
||||||
|
|
||||||
|
await cur.execute(sql)
|
||||||
|
one = await cur.fetchone()
|
||||||
|
assert one[0] == one["tabid"] == one.tabid
|
||||||
|
|
||||||
|
await cur.execute(sql)
|
||||||
|
assert [r.tabid for r in await cur.fetchall()] == [1, 2]
|
||||||
|
|
||||||
|
await cur.execute(sql)
|
||||||
|
assert all(r.tabname for r in await cur.fetchmany(2))
|
||||||
|
|
||||||
|
await cur.execute(sql)
|
||||||
|
assert [r.tabid async for r in cur] == [1, 2]
|
||||||
|
finally:
|
||||||
|
await conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_async_pool_forwards_the_row_factory(
|
||||||
|
conn_params: ConnParams,
|
||||||
|
) -> None:
|
||||||
|
import informix_db
|
||||||
|
|
||||||
|
pool = await aio.create_pool(
|
||||||
|
row_factory=informix_db.Row, min_size=1, max_size=2, **_kw(conn_params)
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
async with pool.connection() as conn:
|
||||||
|
cur = await conn.cursor()
|
||||||
|
await cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||||
|
assert (await cur.fetchone()).tabid == 1
|
||||||
|
finally:
|
||||||
|
await pool.close()
|
||||||
|
|||||||
248
tests/test_placeholder_rewrite.py
Normal file
248
tests/test_placeholder_rewrite.py
Normal file
@ -0,0 +1,248 @@
|
|||||||
|
"""Rewriting :N placeholders without rewriting the SQL around them.
|
||||||
|
|
||||||
|
We advertise ``paramstyle="numeric"`` to match Informix's ESQL/C
|
||||||
|
convention, and the wire protocol takes ``?``, so placeholders get
|
||||||
|
rewritten on the way out. That was ``re.sub(r":(\\d+)", "?", sql)``,
|
||||||
|
which cannot see a string literal and therefore rewrote the inside of
|
||||||
|
one::
|
||||||
|
|
||||||
|
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
|
||||||
|
stored as 'http://host?/x'
|
||||||
|
|
||||||
|
Any ``HH:MM`` time, any URL with a port, any aspect ratio, any
|
||||||
|
``key:value`` string. It wrote wrong data and said nothing.
|
||||||
|
|
||||||
|
Nothing about it was opt-in either. The rewrite runs whenever a
|
||||||
|
statement has parameters, whatever placeholder style the caller actually
|
||||||
|
used, so writing ``?`` everywhere and never touching numeric style did
|
||||||
|
not protect you. And it changed the placeholder *count* while
|
||||||
|
``num_qmarks`` was still computed from ``len(params)``, leaving driver
|
||||||
|
and server disagreeing about how many binds exist.
|
||||||
|
|
||||||
|
The lexical rules below are Informix's own, measured against 12.10,
|
||||||
|
14.10 and 15 rather than assumed from standard SQL. The one that matters
|
||||||
|
most is the backslash: ``'a\\'b'`` is an *unterminated string* to
|
||||||
|
Informix (``-282``), not an escaped quote. A scanner written to
|
||||||
|
Postgres habits would desync on it and corrupt everything after.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import informix_db
|
||||||
|
from informix_db.cursors import _rewrite_numeric_to_qmark as rewrite
|
||||||
|
from tests.conftest import ConnParams
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Substitution happens where it should
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("sql", "expected"),
|
||||||
|
[
|
||||||
|
("SELECT * FROM t WHERE a = :1", "SELECT * FROM t WHERE a = ?"),
|
||||||
|
(
|
||||||
|
"SELECT * FROM t WHERE a = :1 AND b = :2",
|
||||||
|
"SELECT * FROM t WHERE a = ? AND b = ?",
|
||||||
|
),
|
||||||
|
("SELECT * FROM t WHERE a = :10", "SELECT * FROM t WHERE a = ?"),
|
||||||
|
# Already-? SQL is returned untouched, and cheaply.
|
||||||
|
("SELECT * FROM t WHERE a = ?", "SELECT * FROM t WHERE a = ?"),
|
||||||
|
("", ""),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_placeholders_are_rewritten(sql: str, expected: str) -> None:
|
||||||
|
assert rewrite(sql) == expected
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# ...and nowhere else
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("label", "sql", "expected"),
|
||||||
|
[
|
||||||
|
(
|
||||||
|
"url-with-port",
|
||||||
|
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = :1",
|
||||||
|
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"time-of-day",
|
||||||
|
"INSERT INTO log VALUES ('started at 09:15:30', :1)",
|
||||||
|
"INSERT INTO log VALUES ('started at 09:15:30', ?)",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"aspect-ratio",
|
||||||
|
"UPDATE t SET ratio = '16:9' WHERE id = :1",
|
||||||
|
"UPDATE t SET ratio = '16:9' WHERE id = ?",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"doubled-quote-escape",
|
||||||
|
"SELECT 'it''s 10:30' FROM t WHERE a = :1",
|
||||||
|
"SELECT 'it''s 10:30' FROM t WHERE a = ?",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"delimited-identifier",
|
||||||
|
'SELECT "col:1" FROM t WHERE a = :1',
|
||||||
|
'SELECT "col:1" FROM t WHERE a = ?',
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"line-comment",
|
||||||
|
"-- ticket :99\nSELECT * FROM t WHERE a = :1",
|
||||||
|
"-- ticket :99\nSELECT * FROM t WHERE a = ?",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"block-comment",
|
||||||
|
"SELECT /* not :99 */ * FROM t WHERE a = :1",
|
||||||
|
"SELECT /* not :99 */ * FROM t WHERE a = ?",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"brace-comment",
|
||||||
|
"SELECT { not :99 } * FROM t WHERE a = :1",
|
||||||
|
"SELECT { not :99 } * FROM t WHERE a = ?",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"cast-operator",
|
||||||
|
"SELECT a::INT FROM t WHERE a = :1",
|
||||||
|
"SELECT a::INT FROM t WHERE a = ?",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"colon-not-a-placeholder",
|
||||||
|
"SELECT a FROM t WHERE b = ':x' AND c = :1",
|
||||||
|
"SELECT a FROM t WHERE b = ':x' AND c = ?",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"literal-after-placeholder",
|
||||||
|
"SELECT * FROM t WHERE a = :1 AND b = '10:30'",
|
||||||
|
"SELECT * FROM t WHERE a = ? AND b = '10:30'",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_quotes_and_comments_are_left_alone(
|
||||||
|
label: str, sql: str, expected: str
|
||||||
|
) -> None:
|
||||||
|
assert rewrite(sql) == expected, label
|
||||||
|
|
||||||
|
|
||||||
|
def test_backslash_does_not_escape_a_quote() -> None:
|
||||||
|
"""Informix answers -282 for ``'a\\'b'``: the backslash is an ordinary
|
||||||
|
character and the string is unterminated. A scanner that treated it
|
||||||
|
as an escape would think it was still inside the literal and stop
|
||||||
|
substituting, or worse, resume in the wrong place."""
|
||||||
|
sql = "SELECT 'a\\' FROM t WHERE x = :1"
|
||||||
|
# The quote closes at the character after the backslash, so :1 is
|
||||||
|
# outside the literal and gets substituted.
|
||||||
|
assert rewrite(sql) == "SELECT 'a\\' FROM t WHERE x = ?"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unterminated_quote_substitutes_nothing_further() -> None:
|
||||||
|
"""Under-substituting leaves the server to reject SQL that was
|
||||||
|
already malformed. Guessing would corrupt a literal."""
|
||||||
|
assert rewrite("SELECT 'oops :1 FROM t") == "SELECT 'oops :1 FROM t"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unterminated_block_comment_substitutes_nothing_further() -> None:
|
||||||
|
assert rewrite("SELECT /* oops :1 FROM t") == "SELECT /* oops :1 FROM t"
|
||||||
|
|
||||||
|
|
||||||
|
def test_block_comments_do_not_nest() -> None:
|
||||||
|
"""Measured: ``/* a /* b */ c */`` is a syntax error on all three
|
||||||
|
servers, so the first ``*/`` closes the comment. Treating them as
|
||||||
|
nesting would swallow live SQL."""
|
||||||
|
assert (
|
||||||
|
rewrite("SELECT /* a /* b */ :1 FROM t")
|
||||||
|
== "SELECT /* a /* b */ ? FROM t"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Against a real server
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||||
|
return informix_db.connect(
|
||||||
|
host=conn_params.host,
|
||||||
|
port=conn_params.port,
|
||||||
|
user=conn_params.user,
|
||||||
|
password=conn_params.password,
|
||||||
|
database=conn_params.database,
|
||||||
|
server=conn_params.server,
|
||||||
|
connect_timeout=10.0,
|
||||||
|
read_timeout=25.0,
|
||||||
|
autocommit=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_colon_literals_round_trip(conn_params: ConnParams) -> None:
|
||||||
|
"""The bug as a user meets it: the value stored is not the value
|
||||||
|
written. Both placeholder styles, because the rewrite ran for both."""
|
||||||
|
with _connect(conn_params) as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("CREATE TEMP TABLE t_rw (id INT, s VARCHAR(60))")
|
||||||
|
cur.execute(
|
||||||
|
"INSERT INTO t_rw VALUES (?, 'http://host:8080/path')", (1,)
|
||||||
|
)
|
||||||
|
cur.execute("INSERT INTO t_rw VALUES (:1, 'at 09:15:30')", (2,))
|
||||||
|
cur.execute("SELECT id, s FROM t_rw ORDER BY id")
|
||||||
|
assert cur.fetchall() == [
|
||||||
|
(1, "http://host:8080/path"),
|
||||||
|
(2, "at 09:15:30"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_colon_literals_round_trip_through_executemany(
|
||||||
|
conn_params: ConnParams,
|
||||||
|
) -> None:
|
||||||
|
"""executemany rewrites unconditionally, so it had the bug too."""
|
||||||
|
with _connect(conn_params) as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("CREATE TEMP TABLE t_rw2 (id INT, s VARCHAR(40))")
|
||||||
|
cur.executemany(
|
||||||
|
"INSERT INTO t_rw2 VALUES (?, '16:9')", [(1,), (2,)]
|
||||||
|
)
|
||||||
|
cur.execute("SELECT DISTINCT s FROM t_rw2")
|
||||||
|
assert cur.fetchall() == [("16:9",)]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_comment_bearing_sql_still_binds(conn_params: ConnParams) -> None:
|
||||||
|
"""A leading comment reaches PREPARE now that classification asks the
|
||||||
|
server. Make sure the rewriter agrees and doesn't eat a placeholder
|
||||||
|
that follows one."""
|
||||||
|
with _connect(conn_params) as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute(
|
||||||
|
"/* report: daily */ SELECT FIRST 1 tabid FROM systables "
|
||||||
|
"WHERE tabid > :1",
|
||||||
|
(0,),
|
||||||
|
)
|
||||||
|
assert cur.fetchone() is not None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_placeholder_count_matches_after_rewrite(
|
||||||
|
conn_params: ConnParams,
|
||||||
|
) -> None:
|
||||||
|
"""The old regex could add placeholders the driver never counted,
|
||||||
|
leaving num_qmarks (from len(params)) disagreeing with the SQL. A
|
||||||
|
literal containing three colon-digit sequences is the shape that
|
||||||
|
used to break it."""
|
||||||
|
with _connect(conn_params) as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("CREATE TEMP TABLE t_rw3 (id INT, s VARCHAR(40))")
|
||||||
|
with contextlib.suppress(Exception):
|
||||||
|
cur.execute("DELETE FROM t_rw3")
|
||||||
|
cur.execute(
|
||||||
|
"INSERT INTO t_rw3 VALUES (:1, 'a:1 b:2 c:3')", (7,)
|
||||||
|
)
|
||||||
|
cur.execute("SELECT id, s FROM t_rw3")
|
||||||
|
assert cur.fetchone() == (7, "a:1 b:2 c:3")
|
||||||
435
tests/test_rows.py
Normal file
435
tests/test_rows.py
Normal file
@ -0,0 +1,435 @@
|
|||||||
|
"""Named row access, and the cost of it.
|
||||||
|
|
||||||
|
A field request: `row[11]` tells a reader nothing on a wide projection,
|
||||||
|
and `pyodbc` / `mssql-python` both hand back rows that answer to
|
||||||
|
position, column name, and attribute at once. `row_factory=Row` gives
|
||||||
|
the same three.
|
||||||
|
|
||||||
|
It is opt-in. Defaulting it on would tax the thing this driver is
|
||||||
|
measured against, since supporting `row["name"]` means `__getitem__`
|
||||||
|
becomes a Python method rather than C-level tuple indexing, which costs
|
||||||
|
roughly 39 ns on every subscript. Users who want readable column access
|
||||||
|
in application code should pay that; a bulk export that never looks at a
|
||||||
|
column by name should not.
|
||||||
|
|
||||||
|
`Row` subclasses `tuple`, so `row == (1, "x")` still holds and nothing
|
||||||
|
that already worked stops working. That constraint drove the design: the
|
||||||
|
existing suite compares fetched rows against plain tuples in hundreds of
|
||||||
|
places, and so does everybody's code.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
|
import pickle
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import informix_db
|
||||||
|
from informix_db.rows import _RESERVED, Row, make_row_class
|
||||||
|
from tests.conftest import ConnParams
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The type itself
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _row(fields, values):
|
||||||
|
return make_row_class(tuple(fields))(values)
|
||||||
|
|
||||||
|
|
||||||
|
def test_three_ways_to_reach_a_column() -> None:
|
||||||
|
row = _row(("tabid", "tabname"), (1, "systables"))
|
||||||
|
assert row[0] == 1
|
||||||
|
assert row["tabname"] == "systables"
|
||||||
|
assert row.tabname == "systables"
|
||||||
|
|
||||||
|
|
||||||
|
def test_still_equal_to_a_plain_tuple() -> None:
|
||||||
|
"""The compatibility constraint. Existing code and the existing test
|
||||||
|
suite compare fetched rows against tuples everywhere."""
|
||||||
|
row = _row(("a", "b"), (1, "x"))
|
||||||
|
plain = (1, "x")
|
||||||
|
assert row == plain
|
||||||
|
# Both directions: tuple.__eq__ on the left has to accept a subclass
|
||||||
|
# on the right, or `expected == fetched` assertions break.
|
||||||
|
assert plain == row
|
||||||
|
assert list(row) == [1, "x"]
|
||||||
|
assert len(row) == 2
|
||||||
|
a, b = row
|
||||||
|
assert (a, b) == (1, "x")
|
||||||
|
assert row in [(1, "x")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_slice_degrades_to_a_plain_tuple() -> None:
|
||||||
|
"""A slice has no meaningful column mapping, so it should not pretend
|
||||||
|
to be a Row."""
|
||||||
|
row = _row(("a", "b", "c"), (1, 2, 3))
|
||||||
|
assert row[0:2] == (1, 2)
|
||||||
|
assert type(row[0:2]) is tuple
|
||||||
|
|
||||||
|
|
||||||
|
def test_negative_index_still_works() -> None:
|
||||||
|
assert _row(("a", "b"), (1, 2))[-1] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_keys_and_asdict() -> None:
|
||||||
|
row = _row(("a", "b"), (1, "x"))
|
||||||
|
assert row.keys() == ("a", "b")
|
||||||
|
assert row._asdict() == {"a": 1, "b": "x"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_repr_names_the_columns() -> None:
|
||||||
|
assert repr(_row(("a", "b"), (1, "x"))) == "Row(a=1, b='x')"
|
||||||
|
|
||||||
|
|
||||||
|
def test_pickles() -> None:
|
||||||
|
"""The per-shape class is built at runtime, so it cannot be pickled by
|
||||||
|
reference. Multiprocessing users would hit that immediately."""
|
||||||
|
row = _row(("a", "b"), (1, "x"))
|
||||||
|
restored = pickle.loads(pickle.dumps(row))
|
||||||
|
assert restored == (1, "x")
|
||||||
|
assert restored.b == "x"
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_column_says_what_is_there() -> None:
|
||||||
|
row = _row(("tabid", "tabname"), (1, "x"))
|
||||||
|
with pytest.raises(KeyError, match="tabid"):
|
||||||
|
_ = row["nope"]
|
||||||
|
with pytest.raises(AttributeError, match="tabid"):
|
||||||
|
_ = row.nope
|
||||||
|
|
||||||
|
|
||||||
|
def test_class_is_cached_per_shape() -> None:
|
||||||
|
"""A type() call per execute() would land on every small query."""
|
||||||
|
assert make_row_class(("a", "b")) is make_row_class(("a", "b"))
|
||||||
|
assert make_row_class(("a", "b")) is not make_row_class(("a", "c"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_duplicate_names_resolve_to_the_first() -> None:
|
||||||
|
"""``SELECT tabid, tabid`` is legal and both columns are named
|
||||||
|
``tabid``. pyodbc gives the first; so do we."""
|
||||||
|
row = _row(("tabid", "tabid"), (1, 2))
|
||||||
|
assert row["tabid"] == 1
|
||||||
|
assert row.tabid == 1
|
||||||
|
assert row[1] == 2, "positional access must still reach the second"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("name", sorted(_RESERVED))
|
||||||
|
def test_a_column_always_beats_a_method_of_the_same_name(name: str) -> None:
|
||||||
|
"""Parametrized over the *computed* reserved set rather than a
|
||||||
|
hand-written list, because the hand-written list is what missed
|
||||||
|
``keys``, ``_asdict`` and ``_fields`` on the first attempt. Adding a
|
||||||
|
method to Row later cannot silently reopen the hole: this test grows
|
||||||
|
with it.
|
||||||
|
|
||||||
|
Without the guard these return a bound method, which is the exact
|
||||||
|
failure shape this driver spent a fortnight removing from its
|
||||||
|
decoders: a plausible-looking wrong answer, in silence."""
|
||||||
|
row = _row((name, "other"), (7, 9))
|
||||||
|
assert getattr(row, name) == 7
|
||||||
|
assert row[name] == 7
|
||||||
|
|
||||||
|
|
||||||
|
def test_shadowed_methods_stay_reachable_through_the_base_class() -> None:
|
||||||
|
row = _row(("count", "keys", "_asdict"), (1, 2, 3))
|
||||||
|
assert tuple.count(row, 1) == 1
|
||||||
|
assert Row.keys(row) == ("count", "keys", "_asdict")
|
||||||
|
assert Row._asdict(row) == {"count": 1, "keys": 2, "_asdict": 3}
|
||||||
|
|
||||||
|
|
||||||
|
def test_shadowing_fields_does_not_break_repr_or_asdict() -> None:
|
||||||
|
"""The machinery reads its own field list, so if a column named
|
||||||
|
``_fields`` shadowed it, repr and _asdict would report the column
|
||||||
|
value instead of the names. Mangled attributes keep them separate."""
|
||||||
|
row = _row(("_fields", "x"), ("not the names", 2))
|
||||||
|
assert row._fields == "not the names"
|
||||||
|
assert Row.keys(row) == ("_fields", "x")
|
||||||
|
assert repr(row) == "Row(_fields='not the names', x=2)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_identifier_names_are_subscript_only() -> None:
|
||||||
|
"""Informix names expression columns things like ``(count(*))``,
|
||||||
|
which cannot be an attribute."""
|
||||||
|
row = _row(("(count(*))", "ok"), (5, 1))
|
||||||
|
assert row["(count(*))"] == 5
|
||||||
|
assert row.ok == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Against a real server
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||||
|
return informix_db.connect(
|
||||||
|
host=conn_params.host,
|
||||||
|
port=conn_params.port,
|
||||||
|
user=conn_params.user,
|
||||||
|
password=conn_params.password,
|
||||||
|
database=conn_params.database,
|
||||||
|
server=conn_params.server,
|
||||||
|
connect_timeout=10.0,
|
||||||
|
read_timeout=25.0,
|
||||||
|
autocommit=True,
|
||||||
|
**kw,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_default_is_still_a_plain_tuple(conn_params: ConnParams) -> None:
|
||||||
|
"""The opt-in has to be genuinely opt-in. Anyone who does not ask for
|
||||||
|
Row should not pay for it or see any behaviour change."""
|
||||||
|
with _connect(conn_params) as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("SELECT FIRST 1 tabid, tabname FROM systables")
|
||||||
|
row = cur.fetchone()
|
||||||
|
assert type(row) is tuple
|
||||||
|
with pytest.raises(TypeError):
|
||||||
|
_ = row["tabname"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_row_factory_on_the_connection(conn_params: ConnParams) -> None:
|
||||||
|
"""One line at connect time, then every cursor from it, which is what
|
||||||
|
'without any additional coding' has to mean in practice."""
|
||||||
|
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute(
|
||||||
|
"SELECT FIRST 1 tabid, tabname FROM systables ORDER BY tabid"
|
||||||
|
)
|
||||||
|
row = cur.fetchone()
|
||||||
|
assert row[0] == row["tabid"] == row.tabid
|
||||||
|
assert row[1] == row["tabname"] == row.tabname
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_every_fetch_path_returns_rows(conn_params: ConnParams) -> None:
|
||||||
|
"""fetchone, fetchmany, fetchall and iteration each return rows by a
|
||||||
|
different route through the cursor."""
|
||||||
|
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||||
|
sql = "SELECT FIRST 4 tabid, tabname FROM systables ORDER BY tabid"
|
||||||
|
cur = conn.cursor()
|
||||||
|
|
||||||
|
cur.execute(sql)
|
||||||
|
assert cur.fetchone().tabname is not None
|
||||||
|
|
||||||
|
cur.execute(sql)
|
||||||
|
assert all(r.tabname is not None for r in cur.fetchmany(2))
|
||||||
|
|
||||||
|
cur.execute(sql)
|
||||||
|
assert all(r.tabname is not None for r in cur.fetchall())
|
||||||
|
|
||||||
|
cur.execute(sql)
|
||||||
|
assert all(r.tabname is not None for r in cur)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_scrollable_cursor_returns_rows(conn_params: ConnParams) -> None:
|
||||||
|
"""Scrollable cursors return each row straight from the wire rather
|
||||||
|
than from the materialized list, so they are a separate path."""
|
||||||
|
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||||
|
cur = conn.cursor(scrollable=True)
|
||||||
|
cur.execute("SELECT tabid, tabname FROM systables ORDER BY tabid")
|
||||||
|
assert cur.fetch_first().tabname is not None
|
||||||
|
assert cur.fetch_absolute(1).tabname is not None
|
||||||
|
cur.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_per_cursor_override(conn_params: ConnParams) -> None:
|
||||||
|
with _connect(conn_params) as conn:
|
||||||
|
named = conn.cursor()
|
||||||
|
named.row_factory = informix_db.Row
|
||||||
|
plain = conn.cursor()
|
||||||
|
sql = "SELECT FIRST 1 tabid FROM systables"
|
||||||
|
named.execute(sql)
|
||||||
|
plain.execute(sql)
|
||||||
|
assert isinstance(named.fetchone(), Row)
|
||||||
|
assert type(plain.fetchone()) is tuple
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_informix_lowercases_so_the_obvious_name_works(
|
||||||
|
conn_params: ConnParams,
|
||||||
|
) -> None:
|
||||||
|
"""Informix folds unquoted identifiers, which is why the .lower() in
|
||||||
|
the workaround people write by hand is a no-op."""
|
||||||
|
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("CREATE TEMP TABLE t_rows (Config_Key INT, Cnt INT)")
|
||||||
|
cur.execute("INSERT INTO t_rows VALUES (1, 2)")
|
||||||
|
cur.execute("SELECT Config_Key, Cnt FROM t_rows")
|
||||||
|
row = cur.fetchone()
|
||||||
|
assert row.config_key == 1
|
||||||
|
assert row.cnt == 2
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_pool_forwards_the_factory(conn_params: ConnParams) -> None:
|
||||||
|
pool = informix_db.create_pool(
|
||||||
|
host=conn_params.host,
|
||||||
|
port=conn_params.port,
|
||||||
|
user=conn_params.user,
|
||||||
|
password=conn_params.password,
|
||||||
|
database=conn_params.database,
|
||||||
|
server=conn_params.server,
|
||||||
|
autocommit=True,
|
||||||
|
row_factory=informix_db.Row,
|
||||||
|
min_size=1,
|
||||||
|
max_size=2,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with pool.connection() as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||||
|
assert cur.fetchone().tabid is not None
|
||||||
|
finally:
|
||||||
|
pool.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_zero_column_row() -> None:
|
||||||
|
row = make_row_class(())(())
|
||||||
|
assert row == ()
|
||||||
|
assert row.keys() == ()
|
||||||
|
assert repr(row) == "Row()"
|
||||||
|
|
||||||
|
|
||||||
|
def test_wide_row() -> None:
|
||||||
|
"""500 columns: the name map is a dict, so this should be flat, but
|
||||||
|
it is the shape most likely to expose an off-by-one."""
|
||||||
|
fields = tuple(f"c{i}" for i in range(500))
|
||||||
|
row = make_row_class(fields)(tuple(range(500)))
|
||||||
|
assert row["c499"] == row.c499 == row[499] == 499
|
||||||
|
assert row["c0"] == row[0] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_names_that_cannot_be_attributes_are_subscript_only() -> None:
|
||||||
|
row = _row(("col with space", "1leading_digit", ""), (1, 2, 3))
|
||||||
|
assert row["col with space"] == 1
|
||||||
|
assert row["1leading_digit"] == 2
|
||||||
|
assert row[""] == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_unicode_column_name() -> None:
|
||||||
|
row = _row(("café", "x"), (1, 2))
|
||||||
|
assert row["café"] == 1
|
||||||
|
assert row.café == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_dunder_named_column_is_subscript_only() -> None:
|
||||||
|
"""A dunder cannot be shadowed without breaking the object protocol,
|
||||||
|
so the column is reachable by subscript and the attribute keeps its
|
||||||
|
ordinary meaning. No real schema should notice."""
|
||||||
|
row = _row(("__class__", "ok"), (1, 2))
|
||||||
|
assert row["__class__"] == 1
|
||||||
|
assert row.__class__.__name__ == "Row"
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_outlives_eviction_of_its_class_from_the_cache() -> None:
|
||||||
|
"""The class cache is bounded. A row already handed to the caller
|
||||||
|
holds its class alive, so eviction must not affect it."""
|
||||||
|
row = make_row_class(("z1", "z2"))((9, 8))
|
||||||
|
for i in range(300):
|
||||||
|
make_row_class((f"evict{i}", "b"))
|
||||||
|
assert (row.z1, row["z2"], row[0]) == (9, 8, 9)
|
||||||
|
|
||||||
|
|
||||||
|
def test_class_cache_is_bounded() -> None:
|
||||||
|
for i in range(400):
|
||||||
|
make_row_class((f"bounded{i}", "b"))
|
||||||
|
assert make_row_class.cache_info().currsize <= 256
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Row must not change a single value
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_WIDE_DDL = """CREATE TABLE t_rowdiff (
|
||||||
|
c_int INT, c_small SMALLINT, c_big BIGINT, c_int8 INT8, c_serial SERIAL,
|
||||||
|
c_float FLOAT, c_smallfloat SMALLFLOAT, c_dec DECIMAL(16,4),
|
||||||
|
c_decu DECIMAL(16), c_money MONEY(12,2), c_char CHAR(10),
|
||||||
|
c_vchar VARCHAR(40), c_nchar NCHAR(8), c_lvar LVARCHAR(200), c_date DATE,
|
||||||
|
c_dt DATETIME YEAR TO FRACTION(5), c_ivl INTERVAL YEAR TO MONTH,
|
||||||
|
c_bool BOOLEAN, c_set SET(INT NOT NULL), c_tail INT)"""
|
||||||
|
|
||||||
|
_WIDE_COLS = (
|
||||||
|
"c_int,c_small,c_big,c_int8,c_serial,c_float,c_smallfloat,c_dec,c_decu,"
|
||||||
|
"c_money,c_char,c_vchar,c_nchar,c_lvar,c_date,c_dt,c_ivl,c_bool,c_set,"
|
||||||
|
"c_tail"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_rows_are_value_identical_to_tuples(conn_params: ConnParams) -> None:
|
||||||
|
"""The strongest statement available: across every awkward type, with
|
||||||
|
a fully populated row and a fully NULL one, wrapping must change
|
||||||
|
nothing. If it does, Row is not a presentation layer, it is a bug."""
|
||||||
|
sql = f"SELECT {_WIDE_COLS} FROM t_rowdiff ORDER BY c_tail"
|
||||||
|
with _connect(conn_params) as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
with contextlib.suppress(Exception):
|
||||||
|
cur.execute("DROP TABLE t_rowdiff")
|
||||||
|
cur.execute(_WIDE_DDL)
|
||||||
|
try:
|
||||||
|
cur.execute(
|
||||||
|
"INSERT INTO t_rowdiff VALUES (1,2,3,4,0,1.5,2.5,12.34,99,"
|
||||||
|
"9.99,'ch','vc','nc','lv',TODAY,CURRENT,"
|
||||||
|
"INTERVAL(1-2) YEAR TO MONTH,'t',SET{1,2},77)"
|
||||||
|
)
|
||||||
|
cur.execute(
|
||||||
|
"INSERT INTO t_rowdiff VALUES (NULL,NULL,NULL,NULL,0,NULL,"
|
||||||
|
"NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,"
|
||||||
|
"NULL,NULL,88)"
|
||||||
|
)
|
||||||
|
cur.execute(sql)
|
||||||
|
plain = cur.fetchall()
|
||||||
|
|
||||||
|
with _connect(conn_params, row_factory=informix_db.Row) as named_conn:
|
||||||
|
named = named_conn.cursor()
|
||||||
|
named.execute(sql)
|
||||||
|
rows = named.fetchall()
|
||||||
|
|
||||||
|
assert rows == plain, "wrapping changed a value"
|
||||||
|
names = _WIDE_COLS.split(",")
|
||||||
|
first = rows[0]
|
||||||
|
for i, name in enumerate(names):
|
||||||
|
assert first[i] == first[name] == getattr(first, name), name
|
||||||
|
assert rows[1]["c_int"] is None
|
||||||
|
assert rows[1].c_tail == 88
|
||||||
|
finally:
|
||||||
|
with contextlib.suppress(Exception):
|
||||||
|
cur.execute("DROP TABLE t_rowdiff")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.integration
|
||||||
|
def test_threads_share_one_class_per_shape(conn_params: ConnParams) -> None:
|
||||||
|
"""The cache is process-wide and the pool hands connections to many
|
||||||
|
threads, so two threads running the same query must land on the same
|
||||||
|
class rather than racing to build competing ones."""
|
||||||
|
import threading
|
||||||
|
|
||||||
|
seen: list[type] = []
|
||||||
|
errors: list[Exception] = []
|
||||||
|
|
||||||
|
def worker() -> None:
|
||||||
|
try:
|
||||||
|
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
for _ in range(5):
|
||||||
|
cur.execute(
|
||||||
|
"SELECT FIRST 1 tabid, tabname FROM systables"
|
||||||
|
)
|
||||||
|
row = cur.fetchone()
|
||||||
|
assert row.tabname == row["tabname"] == row[1]
|
||||||
|
seen.append(type(row))
|
||||||
|
except Exception as exc:
|
||||||
|
errors.append(exc)
|
||||||
|
|
||||||
|
threads = [threading.Thread(target=worker) for _ in range(6)]
|
||||||
|
for t in threads:
|
||||||
|
t.start()
|
||||||
|
for t in threads:
|
||||||
|
t.join(60)
|
||||||
|
assert not errors, errors[:2]
|
||||||
|
assert len({id(c) for c in seen}) == 1, "same shape built more than once"
|
||||||
Loading…
x
Reference in New Issue
Block a user