Compare commits
24 Commits
v2026.08.3
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| c85aaf2ecb | |||
| acc21b8b81 | |||
| 9fbb97199f | |||
| ce5a582048 | |||
| 805fa58eb2 | |||
| 088171325d | |||
| 7ffc148112 | |||
| 8f341c4b2a | |||
| 429a3910e4 | |||
| 5ad296419c | |||
| c67bbc4766 | |||
| 2eb5ac8f0f | |||
| ac00a25cb7 | |||
| 0d8bd57ba9 | |||
| 66120cf6f3 | |||
| 2e30ceacfb | |||
| 2896f7e93e | |||
| 488773f078 | |||
| ac9075ca9a | |||
| 3d19ca0f1b | |||
| 4af1c90c6f | |||
| 4e186bb890 | |||
| 15d75dd052 | |||
| 5c6991efba |
252
CHANGELOG.md
252
CHANGELOG.md
@ -2,6 +2,258 @@
|
||||
|
||||
All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440.
|
||||
|
||||
## 2026.09.03 — A placeholder rewriter that could not see a string literal, and named row access
|
||||
|
||||
### Upgrade if you write string literals containing a colon
|
||||
|
||||
The driver rewrites `:1` placeholders to the `?` the wire protocol takes. That rewrite was a bare regular expression, and a regular expression cannot see a string literal, so it rewrote the inside of one:
|
||||
|
||||
```sql
|
||||
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
|
||||
-- stored as 'http://host?/x'
|
||||
```
|
||||
|
||||
Any `HH:MM` time, any URL with a port, any aspect ratio, any `key:value` string. It wrote wrong data and said nothing about it.
|
||||
|
||||
Two things made it wider than it looks. It was never opt-in: the rewrite runs whenever a statement has parameters, whatever placeholder style you actually used, so writing `?` everywhere and never touching numeric style gave no protection. And it changed the placeholder *count* while the driver still told the server there were `len(params)` of them, leaving the two disagreeing about how many binds exist.
|
||||
|
||||
It is now a single pass that substitutes only outside quotes and comments. The lexical rules are Informix's own, measured against all three servers rather than assumed from standard SQL, and one of them would have been got wrong from habit: a backslash escapes nothing, so `'a\'b'` is an unterminated string and draws `-282`. A scanner written to Postgres reflexes would have desynced there and corrupted everything after it. Block comments do not nest, the first `*/` closes them. Braces are a comment. `::` is stepped over as a unit so a cast can never be read as the start of a placeholder.
|
||||
|
||||
An unterminated quote or comment consumes the rest and substitutes nothing further. Under-substituting hands malformed SQL to the server to reject; guessing would corrupt a literal.
|
||||
|
||||
This was the last place the driver inferred meaning from SQL text instead of handling it properly.
|
||||
|
||||
### Rows that answer to a column name
|
||||
|
||||
Requested by a user running this alongside SQL Server, where both `pyodbc` and `mssql-python` hand back rows addressable three ways at once. The argument is readability on a wide projection: `row[11]` tells a reader nothing, and stays correct only until somebody adds a column in the middle.
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||
cur.execute("SELECT tabid, tabname FROM systables")
|
||||
row = cur.fetchone()
|
||||
row[0], row["tabname"], row.tabname
|
||||
```
|
||||
|
||||
Set on the connection, so it is one line for an application rather than per query. A cursor can override it. Pools and the async API forward it unchanged.
|
||||
|
||||
`Row` subclasses `tuple`, so `row == (1, "x")` is still true and code that treats rows as sequences keeps working. Slices degrade to plain tuples, since a slice has no column map.
|
||||
|
||||
**It is opt-in, and here is the number.** On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` is a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript. Defaulting it on would move the published 1.05-1.15x ratio against IfxPy to roughly 1.15-1.25x. That is a reasonable trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is a choice rather than a decision made on your behalf.
|
||||
|
||||
Details that were measured rather than assumed. Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`, and the `.lower()` in the workaround people write by hand is already a no-op. Expression columns get server-generated names like `(count(*))` that cannot be attributes, so they are subscript-only. Duplicate names resolve to the first occurrence, matching `pyodbc`. And a column always beats a method of the same name: `tuple` defines `count` and `index`, `Row` adds `keys`, `_asdict` and `_fields`, and any column with one of those names wins, because otherwise it would hand back a bound method in silence. That reserved set is computed from the class rather than hand-listed, which is what caught it: the hand-listed version covered the two `tuple` methods and missed all three of `Row`'s own.
|
||||
|
||||
### Verified
|
||||
|
||||
**472** integration tests on 15.0.1.0.3DE and 14.10.FC7W1DE, **471** on 12.10.FC12W1DE (one skip, no common table expressions before 14.10), up from 457. Both changes have tests that fail against `2026.09.02`.
|
||||
|
||||
## 2026.09.02 — A systematic review, and eleven bugs it found
|
||||
|
||||
No new features. This is the result of going back over the driver looking for the *shape* of past bugs rather than for new symptoms, and it turned up more than expected — including two that silently returned wrong data and one where `rollback()` did nothing at all.
|
||||
|
||||
### Rows never checked that they consumed their own payload
|
||||
|
||||
Fourteen framing bugs had reached users before this release. Every one was the same defect — a column reading the wrong number of bytes — and not one of them raised where the mistake happened. The wrong width produced a plausible value and corrupted whatever came *next*, so the damage always surfaced in a different column, a different row, or a different statement.
|
||||
|
||||
All fourteen were detectable for free. The payload is a fully extracted `bytes` of known length, so after decoding N columns the offset has to land exactly on the end. Now it's checked, on every row, in all three decode paths, and a mismatch names the column shape and the byte delta instead of handing back plausible garbage.
|
||||
|
||||
Turning it on found two more bugs on the first run:
|
||||
|
||||
**Smart LOBs were read as a flat 72-byte field.** They actually use the UDT envelope — 149 bytes when populated, 5 when NULL. The driver consumed 72 and left 77 behind, so a `BLOB` or `CLOB` anywhere but the final column position shifted every column after it. The 144 envelope bytes are the 72-byte locator *hex-encoded*, which means `BlobLocator.raw` had never held a locator, only the first half of the hex text. Nobody noticed because `read_blob_column` resolves LOBs server-side and never reads it.
|
||||
|
||||
**NULL composite UDTs skipped their length field**, byte for byte the LVARCHAR NULL bug in a branch twelve lines away. Both are now the one `_read_udt_envelope`, which also rejects a negative length rather than rewinding into bytes it already decoded.
|
||||
|
||||
### `rollback()` could silently do nothing
|
||||
|
||||
`cursor.execute("BEGIN WORK")` is a reasonable thing to write, and with autocommit on it opened a real transaction that the connection never learned about. `commit()` and `rollback()` are both guarded by that flag, so **`rollback()` returned successfully having sent nothing, and the rows it was asked to discard survived.** The connection then went back to the pool holding an open transaction and its locks, because the pool's cleanup is guarded by the same flag.
|
||||
|
||||
With autocommit off it failed instead, and for a sillier reason: the driver's implicit `SQ_BEGIN` fired first, so the caller's `BEGIN WORK` got `-535`, "already in transaction". The driver and the user competing to open the same transaction, and the user losing.
|
||||
|
||||
The server labels these statements and always has. Both spellings of all three now update the connection's state, and the implicit begin is skipped when the caller is doing the job themselves.
|
||||
|
||||
### Five ordinary query forms failed with a nonsense error
|
||||
|
||||
The driver decided whether to open a cursor by checking if the first word of the SQL was `SELECT`. That gets wrong:
|
||||
|
||||
```sql
|
||||
-- a leading comment → -260
|
||||
/* of any of the three flavours */ → -260
|
||||
{ that Informix accepts } → -260
|
||||
(SELECT parenthesized) → -260
|
||||
WITH cte AS (...) SELECT ... → -260
|
||||
(SELECT a) UNION (SELECT b) → -260
|
||||
```
|
||||
|
||||
`-260` is "Cursor name already in use", which describes neither the cause nor anything the caller did. It says "cursor" because the driver sent `SQ_EXECUTE` where the server was waiting to open one.
|
||||
|
||||
The server reports the statement type in the first field of every DESCRIBE response, and the driver had been parsing it into a metadata dict that nothing read. The decision now uses it. `EXECUTE FUNCTION` gets fixed as a side effect — it used to run down the DML path and discard its return value, and now yields it.
|
||||
|
||||
### A scrollable cursor took the connection down with it
|
||||
|
||||
Informix gives a session one statement slot. A scrollable cursor holds it open on purpose, so any other statement on that connection got `-285` — **and destroyed the scrollable cursor too**, whose next fetch came back `-267`, "the transaction has been rolled back, all locks released". Two unattributable failures from code that reads as entirely ordinary: iterate a large result set, run a lookup partway through.
|
||||
|
||||
It is now a `ProgrammingError` that explains the constraint and leaves the cursor alone. Re-executing the *same* scrollable cursor is still allowed and now closes its own server-side cursor first, because it collided with itself as well.
|
||||
|
||||
### Statements that failed were sometimes never released
|
||||
|
||||
A failed statement stays allocated server-side and collides with the next `PREPARE`, after which every call on that connection returns a nonsense error pointing at the *previous* SQL. That guard existed at six exits and was missing from two, both easy to reach:
|
||||
|
||||
- The parameterized-`SELECT` bind drain. Passing a string where the column is an `INT` encodes cleanly, so the rejection comes from the server (`-1213`, `-415`) *after* the guarded step.
|
||||
- Opening a scrollable cursor — no guard at all, and the worst place to lack one, since the GC-time fallback is armed on the line after.
|
||||
|
||||
All six now share one implementation, which also fixed a defect in the copies that had a cursor to close: `CLOSE` and `RELEASE` shared a single suppression, so a failing `CLOSE` skipped the `RELEASE` — losing the half that actually matters.
|
||||
|
||||
### Cleanup could land inside somebody else's statement
|
||||
|
||||
`SQ_CLOSE` and `SQ_RELEASE` act on whatever statement is current; neither names one. Two consequences.
|
||||
|
||||
A cursor finalizer that can't take the wire lock queues its cleanup for the next operation to flush — but that queue was flushed before *every* PDU, and a finalizer queues precisely because another thread is mid-statement. The flush landed inside that thread's own statement and released it (`-208` before the first fetch, `-267` between fetch batches). Flushing only at a statement boundary is both correct and sufficient.
|
||||
|
||||
And the finalizer's own lock probe couldn't see its own thread. `RLock.acquire(blocking=False)` grants a reentrant acquire to the owner, so when GC fired on a thread that was mid-statement, the finalizer concluded it had exclusive access and sent `CLOSE`/`RELEASE` into the running query. Refcounting hides this — a dropped cursor is freed at the drop — but a cursor caught in a reference cycle waits for a collection, and cycles are ordinary. Any traceback holding a cursor makes one.
|
||||
|
||||
A stale queue entry was separately fatal: the leftover `CLOSE` draws `-267`, an `OperationalError`, which was treated as a dead wire and force-closed a perfectly healthy connection.
|
||||
|
||||
### The async layer borrowed the whole process's thread pool
|
||||
|
||||
Every blocking call went through `asyncio.to_thread`, which runs on the event loop's default executor — sized `min(32, cpu_count + 4)`, so six threads on a two-CPU container.
|
||||
|
||||
`asyncio.to_thread` cannot interrupt a worker, so a cancelled await leaves the thread running the wire call until the read timeout. Cancellation is ordinary in a web app; a client disconnect cancels the request task. Measured: six cancelled calls against a six-worker default executor starve an unrelated `to_thread` indefinitely, and with the executor held, **four concurrent driver queries never ran at all**. Pool concurrency was capped by the same unrelated number — `max_size=20` on a two-CPU box ran six queries at a time.
|
||||
|
||||
Each connection now owns one thread. That's the right size rather than a compromise, since the connection serializes every wire operation on its own lock anyway, and it avoids a deadlock a shared pool-sized executor invites.
|
||||
|
||||
### Two readers shared one stream without agreeing
|
||||
|
||||
`IfxSocket` owns a read-ahead buffer that the buffered reader fills, while the login path and the connection-level drain read straight from the socket without looking at it. Buffered bytes would be skipped, and skipped bytes in a length-framed protocol don't announce themselves.
|
||||
|
||||
Nothing triggers it today, because the server sends one response per request. That's a property of the traffic, not of the code — pipelined `executemany` already puts several responses in flight, and the buffer is connection-scoped precisely so read-ahead *can* cross response boundaries. The two paths now agree by construction.
|
||||
|
||||
Also here: `fill_recv_buf` believed whatever byte count it was handed, and that count is almost always a length field straight off the wire — a garbage `0x7FFFFFFF` read as a 2 GB request and sat in `recv` until the read timeout. It now refuses above `IFX_MAX_READ_BYTES` (256 MiB default) with an error naming the number.
|
||||
|
||||
### Verified
|
||||
|
||||
**457** integration tests on 15.0.1.0.3DE and 14.10.FC7W1DE, **456** on 12.10.FC12W1DE (one skip — 12.10 has no CTEs) — up from 414. Every fix has at least one test that fails against the previous release.
|
||||
|
||||
## 2026.09.01 — TLS traffic fuzzed; no bugs found
|
||||
|
||||
Tests and docs only — no behaviour change. TLS was the last untested surface, and it came back clean.
|
||||
|
||||
### Why it needed testing separately
|
||||
|
||||
Previous TLS coverage stopped at the handshake. Everything after it ran only over plain sockets, and `SSLSocket.recv` is not `socket.recv`: it returns at most one TLS record's worth of plaintext however much you ask for, it can return fewer bytes than are available, and plaintext buffered inside the SSL object is invisible to the OS. The Phase 39 buffered reader asks for up to 64 KB per call and loops until satisfied — that loop is the thing which has to be right, and nothing in the plain-socket suite put the same pressure on it.
|
||||
|
||||
`tests/test_tls_traffic.py` runs real SQLI traffic through a TLS-terminating proxy: the framing-bug types end-to-end, payloads at 1 / 4096 / **16383 / 16384 / 16385** / 32000 bytes (straddling the ~16 KB TLS record boundary), 500- and 5000-row bulk fetches, error recovery, concurrent TLS sessions, and three negative cases — TLS client against a plaintext port, plaintext client against a TLS port (must raise rather than hang), and a verifying context correctly rejecting a self-signed certificate.
|
||||
|
||||
**All clean on all three servers.** The buffered reader handles `SSLSocket` semantics correctly.
|
||||
|
||||
### Scope, stated plainly
|
||||
|
||||
The proxy supplies the TLS half, so this exercises the driver's TLS path — the half we own. It does **not** exercise IBM's server-side TLS listener. Setting one up on the developer-edition image was attempted and abandoned: Informix 15 wants a PKCS#12 keystore (`onkstash` takes a `.p12`, not the older CMS `.kdb`), and the engine kept rejecting the stash with `GSK_ERROR_BAD_KEYFILE_PASSWORD` even with a keystore GSKit itself could open. That half is IBM's code; everything below `ssl.wrap_socket` is identical either way.
|
||||
|
||||
### Verified
|
||||
|
||||
**414/414** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 399.
|
||||
|
||||
With this, every surface has been fuzzed: type framing, fetch batching, error recovery, cursor lifecycle, threads, pooling, transactions, async cancellation, `executemany` partial failure, scrollable cursors, smart LOBs, and TLS.
|
||||
|
||||
## 2026.08.31.3 — An encoding failure inside `executemany` leaked the statement
|
||||
|
||||
The last of the untested surface: pipelined `executemany`, scrollable cursors, and smart LOBs. One bug, in the first of those.
|
||||
|
||||
### The bug
|
||||
|
||||
`executemany` builds all its BIND+EXECUTE PDUs *after* the PREPARE and before anything is drained — that batching is what makes the pipeline fast. If encoding a row raises there (a value the connection's codec cannot represent), the exception escaped the whole block **without sending the RELEASE**, leaking the prepared statement. The next PREPARE collided with it and every later call on that connection failed with an error pointing at the previous SQL.
|
||||
|
||||
Same failure as `2026.08.31.2`, in a sibling path. `_execute_dml_with_params` has guarded this exact case for the single-row path for a long time; the pipelined path was simply never given the same treatment. That is the recurring shape of these: a hazard understood in one place and not carried across to the code next to it.
|
||||
|
||||
### What was already sound
|
||||
|
||||
Worth recording, because it's the larger part of the result:
|
||||
|
||||
- **`executemany` constraint failures.** Duplicate-key and NOT NULL violations at the first, middle, and last position of batches from 2 to 1000 rows all recover cleanly, and `COUNT(*)` always agrees with a full fetch. The pipelined drain-N-responses invariant holds under partial failure.
|
||||
- **Scrollable cursors.** `fetch_first` / `fetch_last` / `fetch_prior` / `fetch_relative` / `fetch_absolute` are correct at 0, 1, 2, 5, 50 and 300 rows, including off both ends (`None`, not a wrap or a crash) and a full forward walk after arbitrary positioning. Twenty abandoned scroll cursors leak nothing.
|
||||
- **Smart LOBs.** Round-trip at 0, 1, 255, 256, 1023, 1024, 4095, 4096, 65535 and 65536 bytes — straddling the 4096-byte `SQ_FILE` chunk and the 64K mark — plus recovery from failed reads.
|
||||
|
||||
### Verified
|
||||
|
||||
**399/399** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 356.
|
||||
|
||||
### Honest note on what is still not fuzzed
|
||||
|
||||
TLS is covered by handshake tests against a self-signed local socket, which exercises the transport wrapper but not a real Informix TLS listener (that needs server-side keystore and `onconfig` SSL setup we don't have in the test containers). The SQLI layer above the socket is identical either way and is now heavily fuzzed, so the residual risk is confined to the handshake itself.
|
||||
|
||||
## 2026.08.31.2 — A failed statement killed the connection; cancelled acquires starved the pool
|
||||
|
||||
Two more found by fuzzing rather than by users, this time outside the type system. Both are ordinary-path bugs that a happy-path test cannot reach.
|
||||
|
||||
### A failed statement was never released
|
||||
|
||||
Successful DML sent `PREPARE → EXECUTE → RELEASE`. **Failing DML sent `PREPARE → EXECUTE` and stopped.** The prepared statement stayed allocated server-side, collided with the next `PREPARE`, and from then on every call on that connection returned a nonsense error — `-255 "Not in transaction"` under autocommit, `-285` otherwise — whose reported offset pointed back at the *failed* SQL rather than the statement that actually failed.
|
||||
|
||||
The practical shape of this: **one duplicate-key violation bricked the connection.** An `INSERT` that trips a unique constraint is about the most routine error an application can hit — every "insert if not exists" pattern produces them — and afterwards nothing on that connection worked again.
|
||||
|
||||
The docstring on the parameterised path already described this exact hazard and guarded the *parameter-encoding* failure. It just never covered the case where EXECUTE itself failed. Fixed in all three paths (DML, parameterised DML, and SELECT, whose fetch loop had the same gap and whose GC-time finalizer only covers scrollable cursors).
|
||||
|
||||
### Cancelling a pool acquire leaked the connection permanently
|
||||
|
||||
`asyncio.to_thread` cannot interrupt its worker. When a task awaiting `pool.acquire()` was cancelled *while the worker was still blocked waiting for a free connection*, the worker went on to succeed and hand back a connection **nobody owned** — checked out, never returned. Each occurrence cost one pool slot until the pool was dead.
|
||||
|
||||
This is not a theoretical race for anyone serving HTTP. A client disconnecting cancels the request task, and under load those cancellations land precisely while waiting for a connection. The pool dies one slot at a time, only under load, and the eventual symptom (`PoolTimeoutError`) points nowhere near the cause.
|
||||
|
||||
`acquire()` now shields the inner future and, if the caller goes away, returns whatever the worker produced to the pool. `add_done_callback` fires immediately on an already-resolved future, so "the worker finished a moment before the cancellation" is the same code path rather than a separate race.
|
||||
|
||||
### What the harness looked for
|
||||
|
||||
Three areas the type-matrix fuzzer cannot see:
|
||||
|
||||
- **Fetch batching.** `NFETCH` is a 4096-*byte* budget, so rows per batch moves with row width. Row counts from 0 to 1025 across narrow/medium/wide schemas, every fetch style (`fetchall`, `fetchone` loop, `fetchmany` at several sizes, iteration) required to agree exactly.
|
||||
- **Error recovery.** Eight error classes, repeated, each followed by a known-good query on the same cursor *and* a fresh one. Repetition matters: a leak accumulates.
|
||||
- **Concurrency.** Threads sharing one connection, pooled threads that deliberately fail before releasing, transaction isolation across pool checkouts, and async cancellation. Every worker reads back a value only it supplied, so a crossed wire fails on *data* even when nothing raises.
|
||||
|
||||
### Two harness false positives, recorded because they're easy to re-trip
|
||||
|
||||
Informix **silently truncates** over-long strings — a 500-character value into `VARCHAR(8)` stores 8 characters and raises nothing. Verified that a literal SQL insert behaves identically, so the driver matches the server; the test expectation was wrong, not the code.
|
||||
|
||||
Informix also **rejects a bare `?` in a projection** (`SELECT ? FROM t` is a syntax error — no type to infer). `SELECT ?::INT FROM t` works. Worth knowing before concluding the driver mishandles parameters.
|
||||
|
||||
### Verified
|
||||
|
||||
**356/356** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 326.
|
||||
|
||||
## 2026.08.31.1 — Three more bugs, found by a fuzzer instead of a user
|
||||
|
||||
Six framing bugs had reached users across three reports. Rather than wait for a seventh, this release adds a harness built specifically to find that class of bug — and it immediately found three more, one of which **hangs the connection**.
|
||||
|
||||
### What it found
|
||||
|
||||
**`BOOLEAN` could not be bound as a parameter at all.** `_encode_bool` emitted type code 45, which the server does not accept as a bind type. It simply stopped responding, so *any* `execute()` with a `bool` parameter hung until the read timeout — or forever, if none was set. Type 41 (how BOOLEAN is *described* in results) hangs identically; the descriptor type and the bind type are not interchangeable. We now bind `'t'`/`'f'` as CHAR and let the server cast, mirroring Informix's own literal syntax.
|
||||
|
||||
This one is worth dwelling on: a hang is worse than wrong data, and it had shipped in every release that claimed BOOLEAN support.
|
||||
|
||||
**Unscaled `DECIMAL` was read one byte short**, shifting every following column. The width formula is `(precision + (scale & 1) + 3) // 2` per `IfxColumnInfo.adjustedColumnLength`; we had dropped the `scale & 1` term. That only matters when precision is even and scale is odd — and an unscaled `DECIMAL(p)` reports scale **255**. So `DECIMAL(16)` is 10 bytes on the wire, not 9. Verified against the wire for ten DECIMAL/MONEY shapes.
|
||||
|
||||
The same formula governs DATETIME and INTERVAL, whose qualifier parity happens to track their digit count, so those agreed by coincidence. All four types now share one helper taken from the reference — one rule beats two that coincide for reasons nobody wrote down.
|
||||
|
||||
**NULL `CHAR`/`NCHAR` came back as `''`**, indistinguishable from a genuinely empty column, so `WHERE c IS NULL` disagreed with what the driver returned. The wire distinguishes them plainly:
|
||||
|
||||
```
|
||||
'ab' -> 61 62 20 20 20 20
|
||||
'' -> 20 20 20 20 20 20 all spaces
|
||||
NULL -> 00 20 20 20 20 20 leading nul
|
||||
```
|
||||
|
||||
### The harness
|
||||
|
||||
`tests/test_type_matrix.py` encodes three principles, each derived from how a real bug escaped:
|
||||
|
||||
1. **Always put a column after the value under test.** Every bug so far corrupted the *next* column; a trailing value can be mis-sized with no visible effect. Every case now ends in a sentinel column, and projections are rotated so each type is exercised in each position.
|
||||
2. **Vary the data, not just the type.** Branch coverage means nothing if no value takes the branch. The LVARCHAR fixture was `'lv value'` — 8 characters, even, never NULL — so both of its broken branches sat unexecuted through 247 tests. The corpus now carries odd/even lengths, empty vs NULL, min/max, negative, scaled vs unscaled.
|
||||
3. **Use an oracle our own codecs don't share.** A Python round-trip cannot catch a symmetric encode/decode bug: our DATETIME encoder wrote zeros and our decoder read them back in perfect agreement. Asking the *server* to render the stored value as text breaks that symmetry, and that check is what would have caught the DATETIME bug on day one.
|
||||
|
||||
It runs exhaustively over the corpus plus seeded random multi-type rows, so failures are reproducible.
|
||||
|
||||
### Verified
|
||||
|
||||
**326/326** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 281. The fuzzer additionally reports clean over 80-round runs at several seeds and widths on all three servers.
|
||||
|
||||
## 2026.08.31 — Fix LVARCHAR tuple framing; DATETIME fractions on bind
|
||||
|
||||
More data corruption, from the same field report that produced `2026.05.08.2`. **If your schema has `LVARCHAR` columns, upgrade** — anything selected after one could be wrong, and `2026.08.27` is not safe.
|
||||
|
||||
@ -27,7 +27,7 @@ Imports as `informix_db` (the distribution name is `informix-driver` because the
|
||||
|
||||
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
|
||||
|
||||
**Test coverage:** 400+ tests across unit / integration / benchmark suites. The integration suite passes 281/281 against **each** of Informix 12.10, 14.10, and 15 — `make test-matrix` runs all three.
|
||||
**Test coverage:** 400+ tests across unit / integration / benchmark suites. The integration suite passes 414/414 against **each** of Informix 12.10, 14.10, and 15 — `make test-matrix` runs all three.
|
||||
|
||||
## Quick start
|
||||
|
||||
@ -154,9 +154,9 @@ All three tested against the official IBM developer-edition Docker images, full
|
||||
|
||||
| Server | Image | Integration suite |
|
||||
|---|---|---|
|
||||
| **15.0.1.0.3DE** | `icr.io/informix/informix-developer-database` | **281 / 281** |
|
||||
| **14.10.FC7W1DE** | `ibmcom/informix-developer-database` | **281 / 281** |
|
||||
| **12.10.FC12W1DE** | `ibmcom/informix-developer-database` | **281 / 281** |
|
||||
| **15.0.1.0.3DE** | `icr.io/informix/informix-developer-database` | **414 / 414** |
|
||||
| **14.10.FC7W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
|
||||
| **12.10.FC12W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
|
||||
|
||||
Reproduce the whole matrix:
|
||||
|
||||
|
||||
@ -13,7 +13,7 @@
|
||||
</h1>
|
||||
<p class="ifx-hero__lede">
|
||||
Every other Informix driver wraps IBM's C SDK or the JDBC JAR. We weren't into that.
|
||||
So we read the protocol and wrote it ourselves — PEP 249, sync + async, pooled, TLS.
|
||||
So we read the protocol and wrote it ourselves: PEP 249, sync and async, pooled, TLS.
|
||||
Within 10% of IBM's own C driver on bulk fetches, <strong>1.6× faster</strong> on
|
||||
bulk inserts. No compile step. No <code>LD_LIBRARY_PATH</code> ritual. No
|
||||
<code>libcrypt.so.1</code> from 2018.
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Architecture overview
|
||||
description: How the layers stack — socket, framing, codec, resultset, cursor, connection, pool.
|
||||
description: How the layers stack, from socket through framing, codec, resultset, cursor, connection, and pool.
|
||||
sidebar:
|
||||
order: 2
|
||||
---
|
||||
@ -28,13 +28,13 @@ The driver is six layers, each with a single responsibility, each testable in is
|
||||
|
||||
The lowest layer. Wraps `socket.socket` with a connection-scoped read buffer (Phase 39). One `recv(64K)` per ~64 KB of incoming data; parsers read into the buffer via `struct.unpack_from(buf, offset)` rather than slicing copies.
|
||||
|
||||
Everything above this layer is `bytes` and `bytearray` arithmetic — no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
|
||||
Everything above this layer is `bytes` and `bytearray` arithmetic, with no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
|
||||
|
||||
See [The buffered reader →](/explain/buffered-reader/) for why the buffer lives here and not on the parser.
|
||||
|
||||
## Protocol / PDU framing
|
||||
|
||||
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean* — only how to read and write the byte shapes.
|
||||
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean*, only how to read and write the byte shapes.
|
||||
|
||||
The PDU types and their fields were reverse-engineered from three sources:
|
||||
|
||||
@ -44,9 +44,9 @@ The PDU types and their fields were reverse-engineered from three sources:
|
||||
|
||||
## Codec / Per-column readers
|
||||
|
||||
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types — see [SQL ↔ Python types](/reference/types/) for the full table.
|
||||
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types. See [SQL ↔ Python types](/reference/types/) for the full table.
|
||||
|
||||
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]` — no per-column dispatch overhead.
|
||||
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]`, with no per-column dispatch overhead.
|
||||
|
||||
Phase 38 went further with `exec()`-based codegen: for the hottest tables, the driver generates a flat decoder function with all readers inlined and dispatch decisions baked in. The generated function is the equivalent of unrolling the per-column dispatch into straight-line code.
|
||||
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Async strategy
|
||||
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async — and what that costs.
|
||||
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async, and what that costs.
|
||||
sidebar:
|
||||
order: 4
|
||||
---
|
||||
@ -19,13 +19,13 @@ Three options for adding async support to a sync database driver:
|
||||
|
||||
2. **Thread-pool wrapping.** Keep the sync core. Wrap each public method with `loop.run_in_executor()`. ~250 lines of code, sync tests still apply, no protocol-layer changes.
|
||||
|
||||
3. **Dual implementations.** Maintain two parallel code paths — one sync, one async. Most code duplicated. Worst of both worlds.
|
||||
3. **Dual implementations.** Maintain two parallel code paths, one sync and one async. Most code duplicated. Worst of both worlds.
|
||||
|
||||
We picked option 2.
|
||||
|
||||
## Why option 2 was the right call
|
||||
|
||||
For typical database workloads — request-scoped connections, mostly waiting on I/O — the practical difference between option 1 and option 2 is small:
|
||||
For typical database workloads, meaning request-scoped connections that mostly wait on I/O, the practical difference between option 1 and option 2 is small:
|
||||
|
||||
- **Latency**: option 1 has a slight edge (no thread context switch), but the difference is dwarfed by the actual database round-trip (~80 µs LAN, ~ms WAN). For a single query, option 2 adds ~5–10 µs of executor overhead.
|
||||
- **Throughput under concurrency**: option 1 wins when you have N coroutines on M physical cores with M < N. The thread pool needs to context-switch between threads; the async loop just runs the next coroutine. For 10–100 concurrent FastAPI requests on a 4-core box, this difference is small.
|
||||
@ -39,7 +39,7 @@ The honest costs:
|
||||
|
||||
- **One worker thread per concurrent in-flight query.** With 100 concurrent queries, you have 100 threads. This is fine for I/O-bound work (Python releases the GIL during socket reads) but doesn't scale beyond a few hundred concurrent queries on a single process.
|
||||
- **Thread-pool sizing matters.** The default executor size (5 × CPU count) is fine for most workloads. For high-concurrency workloads, you may want a larger executor.
|
||||
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe — cancelled workers cannot leak onto recycled pool connections — but the underlying syscall does still complete.
|
||||
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe, in that cancelled workers cannot leak onto recycled pool connections, but the underlying syscall does still complete.
|
||||
|
||||
## What it doesn't cost
|
||||
|
||||
|
||||
@ -7,7 +7,7 @@ sidebar:
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39 — a connection-scoped buffered reader — closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging.
|
||||
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39, a connection-scoped buffered reader, closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging.
|
||||
|
||||
This page is about both the technical change and the failure mode that hid the win for two phases.
|
||||
|
||||
@ -26,7 +26,7 @@ The headline "I/O dominated" was true. The interesting half is the breakdown of
|
||||
- Actual `recv()` syscalls: ~153 ms
|
||||
- Python wrapper overhead: ~400 ms
|
||||
|
||||
That ~400 ms was our own buffer abstraction — a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
|
||||
That ~400 ms was our own buffer abstraction: a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
|
||||
|
||||
The kernel was doing maybe 25–30 ms of work. The other 130 ms of the gap-vs-IfxPy was friction we had introduced ourselves.
|
||||
|
||||
@ -63,21 +63,21 @@ Result: **one `recv()` per ~64 KB of incoming data**, not per field.
|
||||
|
||||
The natural thing to call this is "BufferedSocketReader". The natural thing to do is put the bytearray on the reader. That's what I did first.
|
||||
|
||||
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken` — process blocked in `recv()` waiting for bytes that weren't coming.
|
||||
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken`, meaning the process was blocked in `recv()` waiting for bytes that weren't coming.
|
||||
|
||||
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes — 200 bytes for response 1, 400 bytes for response 2 — it consumed bytes for response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
|
||||
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes, 200 for response 1 and 400 for response 2, it consumed bytes belonging to response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
|
||||
|
||||
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket` (the connection-scoped wrapper) — readers are short-lived parser-views, the buffer outlives them.
|
||||
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket`, the connection-scoped wrapper. Readers are short-lived parser-views, and the buffer outlives them.
|
||||
|
||||
```python
|
||||
# WRONG (first pass) — buffer scoped to reader
|
||||
# WRONG (first pass): buffer scoped to reader
|
||||
class BufferedSocketReader:
|
||||
def __init__(self, sock):
|
||||
self.sock = sock
|
||||
self.buf = bytearray() # ← dies with the reader
|
||||
self.offset = 0
|
||||
|
||||
# RIGHT (Phase 39) — buffer scoped to connection
|
||||
# RIGHT (Phase 39): buffer scoped to connection
|
||||
class IfxSocket:
|
||||
def __init__(self, sock):
|
||||
self.sock = sock
|
||||
@ -122,7 +122,7 @@ The buffered reader ships **enabled by default** in version 2026.05.05.12. To op
|
||||
IFX_BUFFERED_READER=0 python my_app.py
|
||||
```
|
||||
|
||||
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime — close and reopen the pool to flip behavior.
|
||||
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime, so close and reopen the pool to flip behavior.
|
||||
|
||||
<Aside type="note">
|
||||
The flag exists to make A/B measurement easy. There's no expected reason to disable it in production. If you hit a workload where the buffered reader is slower, that's a bug and we'd like to know.
|
||||
@ -132,9 +132,9 @@ The flag is read once at connection construction. Existing connections in a pool
|
||||
|
||||
The general pattern: **what's visible gets optimization attention; what's invisible gets written off as irreducible**.
|
||||
|
||||
The codec is visible — there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
|
||||
The codec is visible: there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
|
||||
|
||||
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads — `read_short`, `read_int`, `read_exact(payload_size)` — are one-liners. What could be slow about that?
|
||||
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads (`read_short`, `read_int`, `read_exact(payload_size)`) are one-liners. What could be slow about that?
|
||||
|
||||
It was carrying ~30% of total wall time. Two phases of changelogs implicitly blamed "the protocol" for the remaining gap. The actual culprit was a few lines of `bytes.join` in a wrapper from Phase 1 that nobody had revisited.
|
||||
|
||||
@ -142,6 +142,6 @@ The lesson is small and easy to state: a profile turns vibes into an attack surf
|
||||
|
||||
## Read more
|
||||
|
||||
- **[Architecture overview →](/explain/architecture/)** — where the buffered reader sits in the layer stack.
|
||||
- **[Phase log →](/explain/phase-log/)** — the full progression from Phase 1 through Phase 39+.
|
||||
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)** — Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.
|
||||
- **[Architecture overview →](/explain/architecture/)**: where the buffered reader sits in the layer stack.
|
||||
- **[Phase log →](/explain/phase-log/)**: the full progression from Phase 1 through Phase 39+.
|
||||
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)**: Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.
|
||||
|
||||
@ -43,11 +43,11 @@ The driver was built across 39+ phases, each with a focused scope and a decision
|
||||
| 23 | Health checks | Pool validates idle connections before return |
|
||||
| 24 | Statement caching | Per-connection prepared-statement cache |
|
||||
| 25 | Fast-path call (`SQ_FPROUTINE`) | Direct UDF/SPL invocation, bypassing PREPARE |
|
||||
| 26 | **CRITICAL** | Pool returned connections with open transactions — fixed |
|
||||
| 26 | **CRITICAL** | Pool returned connections with open transactions (fixed) |
|
||||
| 27 | **CRITICAL** | Per-connection wire lock + async cancellation safety |
|
||||
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync — fixed |
|
||||
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise — fixed |
|
||||
| 30 | Hardening pass | 5 medium-severity audit findings — all closed |
|
||||
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync (fixed) |
|
||||
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise (fixed) |
|
||||
| 30 | Hardening pass | 5 medium-severity audit findings, all closed |
|
||||
|
||||
After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver is production-ready.
|
||||
|
||||
@ -67,21 +67,84 @@ After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver i
|
||||
|
||||
The Phase 37–39 trajectory is documented in detail at [The buffered reader →](/explain/buffered-reader/), including the architectural mistake the first pass got wrong.
|
||||
|
||||
## Field reports and systematic review (2026.08–2026.09)
|
||||
|
||||
Phase 30's audit found nothing left. Then real schemas arrived, and the audit's clean bill of health turned out to be a statement about the questions it had asked rather than about the driver.
|
||||
|
||||
A field report from a user running Informix 12 surfaced three type-framing bugs in a single afternoon: `INT8` / `SERIAL8` never decoded at all, `NCHAR` losing its first character, and `BOOLEAN` corrupting every column after it. Fuzzing the type matrix found more, and each one had the same shape as the last.
|
||||
|
||||
`2026.09.02` went back over the driver looking for that shape rather than for new symptoms, and found eleven bugs.
|
||||
|
||||
| Area | What was wrong |
|
||||
|---|---|
|
||||
| Row framing | Nothing ever checked that a row consumed its own payload. Fourteen framing bugs had reached users, every one detectable for free |
|
||||
| Smart LOBs | Read as a flat 72-byte field when they occupy 149, shifting every column after a non-final `BLOB` |
|
||||
| Composites | A NULL collection skipped its length field, the same bug as NULL `LVARCHAR` twelve lines away |
|
||||
| Transactions | `rollback()` after a SQL `BEGIN WORK` sent nothing and reported success |
|
||||
| Classification | Comments, CTEs and parenthesized selects were run as DML and failed with `-260` |
|
||||
| Scrollable cursors | A second statement on the connection got `-285` and destroyed the cursor too |
|
||||
| Statement release | Two of eight exits leaked a failed statement, bricking the connection |
|
||||
| Cursor finalizers | Cleanup could land inside another statement, and the lock probe was blind to its own thread |
|
||||
| Async | The whole layer shared the process-wide thread pool, so cancellations starved it |
|
||||
| Socket | Two readers on one stream agreed only by accident of how the server replies |
|
||||
|
||||
The pattern behind almost all of them: a hazard that was understood and guarded at the site where it was first observed, rather than at the abstraction that owned it. The guard then never travelled to its siblings. Four separate hand-written copies of the same UDT envelope, three of them wrong. Six hand-rolled copies of the same statement-release cleanup, two missing entirely.
|
||||
|
||||
Two of the fixes came from asking the server instead of guessing. `statement_type` and `statement_id` had both been parsed out of every DESCRIBE response into a metadata dict that nothing read, while the code that needed them inferred the answer from the first word of the SQL and got it wrong five ways.
|
||||
|
||||
The cheapest fix was also the most valuable. Asserting that a row decoder lands exactly on the end of its payload costs one integer comparison, and it would have caught all fourteen framing bugs at the byte where each happened rather than three releases later in somebody's result set.
|
||||
|
||||
Test count went from 241 to 457 across three server versions over this stretch.
|
||||
|
||||
### 2026.09.03
|
||||
|
||||
Two more, one of each kind.
|
||||
|
||||
The `:N` placeholder rewriter was a bare regular expression, so it
|
||||
rewrote the inside of string literals: `'http://host:8080/x'` was stored
|
||||
as `'http://host?/x'`. Same family as the statement-classification bug
|
||||
above, and the last place the driver inferred meaning from SQL text
|
||||
instead of handling it. Replaced with a scanner using Informix's own
|
||||
lexical rules, measured rather than assumed. One of those rules would
|
||||
have been got wrong from habit: a backslash escapes nothing, so `'a\'b'`
|
||||
is an unterminated string and draws `-282`. A scanner written to
|
||||
Postgres reflexes desyncs there and corrupts everything after it.
|
||||
|
||||
`row_factory=Row` adds column-name and attribute access, on request from
|
||||
a user coming from `pyodbc`. Opt-in, because it costs about 9% on bulk
|
||||
fetch and that is not a trade to make for someone who never reads a
|
||||
column by name.
|
||||
|
||||
Testing the second one found the pattern again, this time in code
|
||||
written an hour earlier. The names a column could shadow were
|
||||
hand-listed as `count` and `index`, the two `tuple` methods; then `keys`,
|
||||
`_asdict` and `_fields` were added to the same class and the list was
|
||||
never revisited. A column called `keys` returned a bound method, in
|
||||
silence. The fix was to compute the set from the class rather than
|
||||
lengthen the list, and to parametrize the test over that computed set so
|
||||
it grows with the class.
|
||||
|
||||
The interesting part is where testing found it. A differential test
|
||||
across twenty columns covering every awkward type, populated and fully
|
||||
NULL, found nothing. The bug was in six lines of name bookkeeping that
|
||||
looked obviously correct, and had been correct when written. Anything
|
||||
that must stay in sync with a class by hand eventually will not.
|
||||
|
||||
## Notable architectural pivots
|
||||
|
||||
The decision log calls out four moments where the obvious choice would have been wrong:
|
||||
|
||||
1. **Phase 10/11** — abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
|
||||
2. **Phase 16** — thread-pool async instead of full async refactor. ~88% less code, same FastAPI surface.
|
||||
3. **Phase 27** — adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
|
||||
4. **Phase 39** — buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
|
||||
1. **Phase 10/11**: abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
|
||||
2. **Phase 16**: thread-pool async instead of a full async refactor. ~88% less code, same FastAPI surface.
|
||||
3. **Phase 27**: adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
|
||||
4. **Phase 39**: buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
|
||||
|
||||
## What's next
|
||||
|
||||
The roadmap (loose, not committed):
|
||||
|
||||
- **Phase 40+ (codec)**: Numpy-backed bulk decode for homogeneous columns. ~5× speedup target on analytical workloads.
|
||||
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python" — gated behind a build flag.
|
||||
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python", so it would sit behind a build flag.
|
||||
- **Phase 5x (API)**: Native `callproc` with named parameters, IBM-specific scrollable cursor extensions for full IfxPy parity.
|
||||
|
||||
The phase log is updated as work lands. The repo's [`CHANGELOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md) is the source of truth for shipped changes.
|
||||
|
||||
@ -26,14 +26,14 @@ The order-of-magnitude intuition: pure-Python is ~2× slower than C-bound for **
|
||||
|
||||
The benefits are mostly deployment, not performance:
|
||||
|
||||
- **50 KB wheel** — installable in a slim Docker image without a build toolchain.
|
||||
- **No `libcrypt.so.1`** — works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
|
||||
- **Python 3.10–3.14** — no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
|
||||
- **Type annotations everywhere** — `py.typed` flag, full coverage in mypy / pyright.
|
||||
- **Auditable codepaths** — every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
|
||||
- **Async without `run_in_executor`** — native `async def` API, FastAPI-compatible.
|
||||
- **50 KB wheel**: installable in a slim Docker image without a build toolchain.
|
||||
- **No `libcrypt.so.1`**: works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
|
||||
- **Python 3.10–3.14**: no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
|
||||
- **Type annotations everywhere**: `py.typed` flag, full coverage in mypy / pyright.
|
||||
- **Auditable codepaths**: every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
|
||||
- **Async without `run_in_executor`**: native `async def` API, FastAPI-compatible.
|
||||
|
||||
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1` — those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
|
||||
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1`: those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
|
||||
|
||||
## Where the ceiling sits
|
||||
|
||||
@ -48,7 +48,7 @@ Five fields × ~250 ns/field + ~250 ns overhead = ~1.5 µs. We're at ~2.0 µs wh
|
||||
|
||||
Strategies for closing further:
|
||||
|
||||
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim — there'd be a build step.
|
||||
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim, because there'd be a build step.
|
||||
- **Bytecode optimization via `exec()`-codegen** (the Phase 38 approach). Marginal further wins; we've already extracted most of what's available.
|
||||
- **Numpy-backed bulk decode** for homogeneous columns. Promising for analytical workloads. ~5× speedup possible for `SELECT col FROM huge_table` over the current per-row approach. Probably Phase 41+.
|
||||
|
||||
@ -58,6 +58,6 @@ For I/O-bound workloads we're already at the ceiling. The buffered reader closed
|
||||
|
||||
Pure-Python costs us ~5–15% on bulk-fetch workloads and zero (or favorable) on everything else. The deployment, async, and modern-Python wins are large and don't depend on workload.
|
||||
|
||||
If the codec gap matters for your case — analytical reporting against a wide table, pulling millions of rows in a single SELECT — IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
|
||||
If the codec gap matters for your case, meaning analytical reporting against a wide table or pulling millions of rows in a single SELECT, IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
|
||||
|
||||
The driver chose the goal — *first pure-socket Informix driver in any language* — over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.
|
||||
The driver chose the goal, *first pure-socket Informix driver in any language*, over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.
|
||||
|
||||
@ -1,13 +1,13 @@
|
||||
---
|
||||
title: The SQLI wire protocol
|
||||
description: A short tour of Informix's SQLI protocol — PDU framing, the handshake, statement execution, fetch.
|
||||
description: A short tour of Informix's SQLI protocol, covering PDU framing, the handshake, statement execution, and fetch.
|
||||
sidebar:
|
||||
order: 1
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
SQLI is Informix's wire protocol — the same protocol IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
|
||||
SQLI is Informix's wire protocol, the same one IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
|
||||
|
||||
This page is a short tour. The byte-level reference (with hex annotations) lives in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) in the repo.
|
||||
|
||||
@ -71,14 +71,14 @@ The full lifecycle for `SELECT id FROM users WHERE id = ?`:
|
||||
← SQ_ID
|
||||
```
|
||||
|
||||
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts — see [Bulk inserts](/how-to/executemany/).
|
||||
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts. See [Bulk inserts](/how-to/executemany/).
|
||||
|
||||
## Smart-LOB transfer
|
||||
|
||||
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side — no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
|
||||
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side, with no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
|
||||
|
||||
This was the architectural pivot in [Phase 10/11](/explain/phase-log/) that made smart-LOBs work end-to-end in pure Python. Reading and writing GB-sized BLOBs goes through the same socket as any other query.
|
||||
|
||||
<Aside type="note">
|
||||
The protocol has many more PDU types than this page covers — mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
|
||||
The protocol has many more PDU types than this page covers, mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
|
||||
</Aside>
|
||||
|
||||
@ -56,7 +56,7 @@ async def get_user(user_id: int, conn = Depends(get_conn)):
|
||||
|
||||
## Cancellation
|
||||
|
||||
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe — the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean (transactions rolled back). You don't need to wrap anything in `try/finally`.
|
||||
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe: the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean, with transactions rolled back. You don't need to wrap anything in `try/finally`.
|
||||
|
||||
<Aside type="note">
|
||||
This is a Phase 27 invariant: async cancellation cannot leak running workers onto recycled connections. The earlier behavior was a `High` audit finding; the fix is a CI tripwire test that's been green every commit since.
|
||||
@ -89,5 +89,5 @@ async def create_order(order: OrderIn, conn = Depends(get_conn)):
|
||||
`HTTPException` is an `Exception`, so raising one inside the block rolls back before FastAPI turns it into a response.
|
||||
|
||||
<Aside type="note">
|
||||
There is no `conn.transaction()` context manager on either the sync or async connection — an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
|
||||
There is no `conn.transaction()` context manager on either the sync or async connection, though an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
|
||||
</Aside>
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Optimize bulk SELECT
|
||||
description: How the buffered reader works in practice — when it's on, when it isn't, how to A/B-measure your workload.
|
||||
description: How the buffered reader works in practice, when it's on, when it isn't, and how to A/B-measure your workload.
|
||||
sidebar:
|
||||
order: 5
|
||||
---
|
||||
|
||||
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything — the bulk-fetch gap against IfxPy is now ~5–15% rather than ~140%.
|
||||
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything, since the bulk-fetch gap against IfxPy is now ~5–15% rather than ~140%.
|
||||
|
||||
For the architectural rationale, see [The buffered reader →](/explain/buffered-reader/).
|
||||
|
||||
@ -33,7 +33,7 @@ For typical bulk-SELECT workloads expect a 30–40% wall-time reduction. For wor
|
||||
|
||||
## When the speedup is largest
|
||||
|
||||
Workloads where every column read makes ~4–5 small `recv()` calls — i.e. tabular data, narrow rows, large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
|
||||
Workloads where every column read makes ~4–5 small `recv()` calls, meaning tabular data, narrow rows, and large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
|
||||
|
||||
| Workload shape | Speedup |
|
||||
|---|---:|
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Run the dev container
|
||||
description: IBM Informix Developer Edition in Docker — first-time setup, sbspace for smart-LOBs, common troubleshooting.
|
||||
description: IBM Informix Developer Edition in Docker, covering first-time setup, sbspace for smart-LOBs, and common troubleshooting.
|
||||
sidebar:
|
||||
order: 8
|
||||
---
|
||||
@ -20,9 +20,9 @@ docker run -d --name informix-dev \
|
||||
icr.io/informix/informix-developer-database:15.0.1.0.3DE
|
||||
```
|
||||
|
||||
- **`9088`** — clear-text SQLI listener
|
||||
- **`9089`** — TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
|
||||
- **`--privileged`** — required for the dev image's shared-memory tuning
|
||||
- **`9088`**: clear-text SQLI listener
|
||||
- **`9089`**: TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
|
||||
- **`--privileged`**: required for the dev image's shared-memory tuning
|
||||
|
||||
The image takes ~90 seconds to initialize. Watch for `oninit running`:
|
||||
|
||||
@ -84,10 +84,10 @@ pytest -m integration
|
||||
|
||||
**"Connection refused" on 9088**: the image is still initializing. Wait for `oninit running` in the logs.
|
||||
|
||||
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing — it always exists.
|
||||
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing, since it always exists.
|
||||
|
||||
**`-908` (system error / shared memory)**: the container needs `--privileged`. Restart with that flag.
|
||||
|
||||
<Aside type="tip">
|
||||
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart — the database survives container shutdown.
|
||||
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart, and the database survives container shutdown.
|
||||
</Aside>
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Bulk inserts (executemany)
|
||||
description: How to bulk-load with executemany — and the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
description: How to bulk-load with executemany, including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
sidebar:
|
||||
order: 4
|
||||
---
|
||||
@ -49,7 +49,7 @@ The default is `autocommit=False`, so this only catches you if you've explicitly
|
||||
|
||||
## Why it's faster than IfxPy
|
||||
|
||||
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row — for 10,000 rows on a 80 µs RTT, that's 800 ms of just waiting for ACKs.
|
||||
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row, so for 10,000 rows on an 80 µs RTT it adds up to 800 ms of just waiting for ACKs.
|
||||
|
||||
Phase 33 changed our `executemany` to **pipeline** the BIND+EXECUTE PDUs:
|
||||
|
||||
@ -78,7 +78,7 @@ with conn:
|
||||
cur = conn.cursor()
|
||||
for batch in chunks(huge_iterator, 10_000):
|
||||
cur.executemany("INSERT INTO logs ...", batch)
|
||||
# one transaction, many batched executemany calls — one commit at the end
|
||||
# one transaction, many batched executemany calls, one commit at the end
|
||||
```
|
||||
|
||||
10,000 rows per chunk is a reasonable default; the per-chunk Python memory cost is `~ N × bytes_per_row`. For 10k tuples of 5 small fields that's a few MB.
|
||||
@ -95,7 +95,7 @@ new_id = cur.fetchone()[0]
|
||||
|
||||
`sqlca.sqlerrd1` is where Informix records the serial value from the most recent INSERT on this connection, so read it before running any other statement.
|
||||
|
||||
There is no `cursor.lastrowid` — an earlier version of this page said there was.
|
||||
There is no `cursor.lastrowid`, though an earlier version of this page said there was.
|
||||
|
||||
For batch inserts that need the IDs, the idiomatic pattern is:
|
||||
|
||||
|
||||
@ -27,11 +27,11 @@ conn = informix_db.connect(
|
||||
)
|
||||
```
|
||||
|
||||
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported (deliberately — they're a security and parsing footgun).
|
||||
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported, deliberately: they're a security and parsing footgun.
|
||||
|
||||
## The DB-API surface is the same
|
||||
|
||||
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, `close()` — all behave per PEP 249.
|
||||
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, and `close()` all behave per PEP 249.
|
||||
|
||||
The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `DatabaseError`, `DataError`, `OperationalError`, `IntegrityError`, `InternalError`, `ProgrammingError`, `NotSupportedError`.
|
||||
|
||||
@ -45,9 +45,39 @@ The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `Dat
|
||||
|
||||
- **Async API** (`from informix_db import aio`)
|
||||
- **Connection pool** (`informix_db.create_pool` / `aio.create_pool`)
|
||||
- **Type-safe annotations** — `informix-driver` ships with `py.typed`
|
||||
- **Type-safe annotations**: `informix-driver` ships with `py.typed`
|
||||
- **Python 3.12+ support**
|
||||
- **Pipelined `executemany`** — 1.6× faster than IfxPy's per-row implementation
|
||||
- **Pipelined `executemany`**: 1.6× faster than IfxPy's per-row implementation
|
||||
- **Rows addressable by name**: `row["col"]` and `row.col`, not just `row[0]`
|
||||
|
||||
## Reading rows by name
|
||||
|
||||
IfxPy gives you positional access and nothing else, which is why most
|
||||
IfxPy codebases grow a helper like this:
|
||||
|
||||
```python
|
||||
cols = [c[0].lower() for c in cur.description]
|
||||
row_dict = dict(zip(cols, cur.fetchone()))
|
||||
```
|
||||
|
||||
That gets you a dict and still no attribute access. Ask for `Row`
|
||||
instead and all three work at once:
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||
cur.execute("SELECT config_key, config_value FROM settings")
|
||||
row = cur.fetchone()
|
||||
|
||||
row[0], row["config_key"], row.config_key
|
||||
```
|
||||
|
||||
Set on the connection, so it applies to every cursor from it. The
|
||||
`.lower()` in the hand-written version is already a no-op, incidentally:
|
||||
Informix folds unquoted identifiers to lower case.
|
||||
|
||||
It is opt-in because it costs about 9% on bulk fetch. See
|
||||
[the API reference](/reference/api/#rows) for the numbers and the
|
||||
edge cases.
|
||||
|
||||
## Migrating incrementally
|
||||
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Use the connection pool
|
||||
description: Sync and async connection pools — sizing, timeouts, lifecycle, threading.
|
||||
description: Sync and async connection pools, covering sizing, timeouts, lifecycle, and threading.
|
||||
sidebar:
|
||||
order: 2
|
||||
---
|
||||
@ -32,7 +32,7 @@ with pool.connection() as conn:
|
||||
pool.close()
|
||||
```
|
||||
|
||||
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically — you never see a dirty connection from `pool.connection()`.
|
||||
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically, so you never see a dirty connection from `pool.connection()`.
|
||||
|
||||
## Async pool
|
||||
|
||||
@ -56,16 +56,16 @@ async def main():
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
Same semantics, `async`-aware. Cancellation is cancellation-safe — a cancelled task does not leak an in-flight worker onto a recycled connection.
|
||||
Same semantics, `async`-aware. Cancellation is safe here too: a cancelled task does not leak an in-flight worker onto a recycled connection.
|
||||
|
||||
## Sizing
|
||||
|
||||
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes" — usually 8–16 for typical web/API services.
|
||||
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes", usually 8–16 for typical web/API services.
|
||||
|
||||
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`** — the server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
|
||||
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`**. The server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
|
||||
|
||||
## Threading
|
||||
|
||||
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally — `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
|
||||
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally, since `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
|
||||
|
||||
Phase 27 added a per-connection wire lock that makes accidental sharing safe (interleaved PDUs serialize correctly), but you should still give each thread its own connection. The lock is a backstop, not a license.
|
||||
|
||||
@ -34,7 +34,7 @@ The `BLOB_PLACEHOLDER` token in the SQL marks where the BLOB data goes. Other pa
|
||||
|
||||
## Reading a CLOB
|
||||
|
||||
CLOBs use the same methods as BLOBs — there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
|
||||
CLOBs use the same methods as BLOBs, and there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
|
||||
|
||||
```python
|
||||
raw: bytes = cur.read_blob_column(
|
||||
|
||||
@ -30,7 +30,7 @@ conn = informix_db.connect(
|
||||
)
|
||||
```
|
||||
|
||||
Bring-your-own context is the recommended production pattern — you get full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
|
||||
Bring-your-own context is the recommended production pattern, giving you full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
|
||||
|
||||
## Dev / self-signed: tls=True
|
||||
|
||||
@ -38,7 +38,7 @@ Bring-your-own context is the recommended production pattern — you get full co
|
||||
informix_db.connect(host="127.0.0.1", port=9089, ..., tls=True)
|
||||
```
|
||||
|
||||
`tls=True` is a convenience for development — it builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
|
||||
`tls=True` is a convenience for development. It builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
|
||||
|
||||
## Server-side configuration
|
||||
|
||||
@ -48,7 +48,7 @@ The Informix server needs a TLS listener entry in `sqlhosts`:
|
||||
informix_tls onsoctcp myhost 9089
|
||||
```
|
||||
|
||||
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089` — no configuration needed.
|
||||
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089`, with no configuration needed.
|
||||
|
||||
<Aside type="tip">
|
||||
If your connection hangs at the handshake, you've probably pointed at the non-TLS port (`9088` instead of `9089`). The non-TLS listener will accept the TCP connection but won't speak TLS, so the handshake stalls.
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: informix-driver
|
||||
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket — no CSDK, no JVM, no native libraries.
|
||||
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket, with no CSDK, no JVM, and no native libraries.
|
||||
template: splash
|
||||
hero:
|
||||
tagline: ''
|
||||
@ -21,12 +21,12 @@ import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
|
||||
<div class="ifx-feature">
|
||||
<Icon name="seti:python" class="ifx-feature__icon" />
|
||||
<h3>~10% behind IfxPy on bulk fetches</h3>
|
||||
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics — for now.</p>
|
||||
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics, for now.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="puzzle" class="ifx-feature__icon" />
|
||||
<h3>50 KB wheel. Zero native deps.</h3>
|
||||
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.10–3.14 — including the versions IfxPy doesn't.</p>
|
||||
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.10–3.14, including the versions IfxPy doesn't.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="sun" class="ifx-feature__icon" />
|
||||
@ -36,7 +36,7 @@ import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
|
||||
<div class="ifx-feature">
|
||||
<Icon name="approve-check" class="ifx-feature__icon" />
|
||||
<h3>PEP 249, no surprises</h3>
|
||||
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy — and threadsafe sharing through a per-connection wire lock.</p>
|
||||
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy, plus threadsafe sharing through a per-connection wire lock.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="document" class="ifx-feature__icon" />
|
||||
@ -66,7 +66,7 @@ That's it. No `IBM_DB_HOME`. No DSN file. No `libcrypt.so.1`.
|
||||
|
||||
The existing tools were not my style.
|
||||
|
||||
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix` — all of them. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
|
||||
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That means all of them: `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
|
||||
|
||||
The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9). It needs four `LD_LIBRARY_PATH` entries. It needs `setuptools < 58`. And IfxPy itself is broken on Python 3.12+. For containerized deployments, ETL pipelines, FastAPI services, or anywhere a build toolchain on the runtime is friction, this driver is the alternative that didn't previously exist. Now it does.
|
||||
|
||||
@ -82,7 +82,7 @@ The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, mi
|
||||
[Read →](/start/vs-ifxpy/)
|
||||
</Card>
|
||||
<Card title="The buffered reader" icon="information">
|
||||
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1× — and the architectural mistake the first pass got wrong.
|
||||
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1×, and the architectural mistake the first pass got wrong.
|
||||
[Read →](/explain/buffered-reader/)
|
||||
</Card>
|
||||
<Card title="Architecture" icon="puzzle">
|
||||
|
||||
@ -51,7 +51,7 @@ informix_db.connect(
|
||||
|
||||
| Method / property | Description |
|
||||
|---|---|
|
||||
| `cursor(scrollable=False)` | Returns a new `Cursor`. |
|
||||
| `cursor(scrollable=False)` | Returns a new `Cursor`. Only one scrollable cursor may be open per connection; see below. |
|
||||
| `commit()` | Commits the current transaction. |
|
||||
| `rollback()` | Rolls back the current transaction. |
|
||||
| `close()` | Closes the connection. Idempotent. |
|
||||
@ -64,7 +64,7 @@ informix_db.connect(
|
||||
|
||||
`Connection` is also a context manager (`with informix_db.connect(...) as conn:`), which closes on exit.
|
||||
|
||||
There is no `conn.transaction()` helper and no `conn.autocommit` attribute — earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
|
||||
There is no `conn.transaction()` helper and no `conn.autocommit` attribute. Earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(..., autocommit=False)
|
||||
@ -78,12 +78,67 @@ except Exception:
|
||||
raise
|
||||
```
|
||||
|
||||
:::caution[One statement per session]
|
||||
Informix gives a session a single statement slot. A non-scrollable cursor never trips over this, because it materializes its rows and releases the statement before `execute()` returns. A **scrollable** cursor holds the slot open on purpose, so while one is open, any other statement on that same connection raises `ProgrammingError`:
|
||||
|
||||
```python
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT id, body FROM big_table ORDER BY id")
|
||||
scroll.fetch_first()
|
||||
|
||||
other = conn.cursor()
|
||||
other.execute("SELECT COUNT(*) FROM audit") # ProgrammingError
|
||||
```
|
||||
|
||||
Close the scrollable cursor first, or use a second connection for the other query. Re-running `execute()` on the scrollable cursor itself is fine; it closes its own server-side cursor first.
|
||||
|
||||
Before `2026.09.02` this was not refused. The server answered the second statement with `-285` and destroyed the scrollable cursor as well, whose next fetch then returned `-267`, "the transaction has been rolled back". Two errors, neither naming the cause.
|
||||
:::
|
||||
|
||||
:::note[Transaction control written as SQL]
|
||||
`commit()` and `rollback()` are the supported way to end a transaction, but `cursor.execute("BEGIN WORK")` and its `COMMIT` / `ROLLBACK` counterparts work too, and the connection tracks them from `2026.09.02` onward.
|
||||
|
||||
That matters because it used not to. Under `autocommit=True`, a SQL `BEGIN WORK` opened a real transaction the connection never learned about, and since `rollback()` is guarded by that state, **it returned successfully having sent nothing**. The rows it was asked to discard survived, and a pooled connection went back into circulation still holding the transaction and its locks.
|
||||
:::
|
||||
|
||||
:::note[Two version numbers, and why]
|
||||
The login response carries Informix's *internal* protocol version, not the release you installed: 12.10 announces itself as `9.56`, 14.10 as `9.59`, and 15 as `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers, which is why all three speak an identical SQLI dialect.
|
||||
|
||||
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')` — one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
|
||||
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')`. That is one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
|
||||
:::
|
||||
|
||||
## Rows
|
||||
|
||||
Fetch methods return plain tuples by default. Pass `row_factory` to get rows that also answer to a column name and an attribute, the shape `pyodbc` and `mssql-python` provide:
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT tabid, tabname FROM systables")
|
||||
row = cur.fetchone()
|
||||
|
||||
row[0] # 1
|
||||
row["tabname"] # 'systables'
|
||||
row.tabname # 'systables'
|
||||
```
|
||||
|
||||
Set on the connection, so it applies to every cursor from it. A cursor can override it with `cur.row_factory = informix_db.Row`. Pools and the async API forward it unchanged.
|
||||
|
||||
`Row` subclasses `tuple`, so `row == (1, "systables")` is still true and anything treating rows as sequences keeps working. Slices return plain tuples, since a slice has no column map. `row.keys()` gives the column names and `row._asdict()` gives a dict.
|
||||
|
||||
:::note[Why it is opt-in]
|
||||
On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` becomes a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript.
|
||||
|
||||
Defaulting it on would move the [1.05-1.15x ratio against IfxPy](/start/vs-ifxpy/) to roughly 1.15-1.25x. That is a good trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is yours to make.
|
||||
:::
|
||||
|
||||
Four behaviours worth knowing:
|
||||
|
||||
- Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`. The `.lower()` people add by hand when building this themselves is already a no-op.
|
||||
- Expression columns get server-generated names like `(count(*))`, which cannot be Python attributes. Those are reachable by subscript only.
|
||||
- Duplicate column names resolve to the first occurrence, matching `pyodbc`. Positional access still reaches both.
|
||||
- **A column beats a method of the same name.** `tuple` defines `count` and `index`; `Row` adds `keys`, `_asdict` and `_fields`. A column with any of those names wins, because otherwise it would hand back a bound method instead of a value. The methods stay reachable as `tuple.count(row, x)` and `Row.keys(row)`.
|
||||
|
||||
## Cursor
|
||||
|
||||
| Method / property | Description |
|
||||
@ -105,7 +160,7 @@ That string is available for free as `server_version_internal`. Because it reads
|
||||
| `rownumber` | Current 0-indexed position, or `None` before the first row. |
|
||||
| `arraysize` | Default `fetchmany()` size. |
|
||||
|
||||
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute — earlier versions of this page listed all three and none have existed.
|
||||
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute. Earlier versions of this page listed all three, and none have existed.
|
||||
|
||||
CLOBs go through the BLOB methods. `read_blob_column` returns `bytes` either way, so decode it yourself with the column's encoding:
|
||||
|
||||
@ -120,7 +175,7 @@ raw = cur.read_blob_column("SELECT txt FROM docs WHERE id = ?", (1,))
|
||||
text = raw.decode("iso-8859-1")
|
||||
```
|
||||
|
||||
For a server-assigned `SERIAL` after an INSERT, ask the server — Informix exposes it through `DBINFO`:
|
||||
For a server-assigned `SERIAL` after an INSERT, ask the server. Informix exposes it through `DBINFO`:
|
||||
|
||||
```python
|
||||
cur.execute("INSERT INTO people (name) VALUES (?)", ("ada",))
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Performance baselines
|
||||
description: Single-connection benchmark results — codec, framing, end-to-end queries, vs IfxPy.
|
||||
description: Single-connection benchmark results for codec, framing, end-to-end queries, and IfxPy.
|
||||
sidebar:
|
||||
order: 5
|
||||
---
|
||||
|
||||
@ -13,7 +13,7 @@ sidebar:
|
||||
| `IFX_DEBUG_WIRE` | unset | When set to a truthy value, log wire-level PDU framing to stderr. Verbose; for debugging only. |
|
||||
| `IFX_PROTOCOL_TRACE` | unset | When set to a path, write annotated wire captures to the file. |
|
||||
| `IFX_DISABLE_PIPELINE` | unset | Disables pipelined `executemany` (Phase 33). Use only to A/B-measure. |
|
||||
| `INFORMIXSERVER` | — | Read by `connect()` if `server=` is not provided. |
|
||||
| `INFORMIXSERVER` | *(none)* | Read by `connect()` if `server=` is not provided. |
|
||||
|
||||
Environment variables are read at connection construction. Changing them at runtime doesn't affect existing connections.
|
||||
|
||||
@ -64,4 +64,4 @@ informix_db.connect(
|
||||
)
|
||||
```
|
||||
|
||||
`CLIENT_LOCALE` is set automatically from `client_locale=` — don't put it in `env=`.
|
||||
`CLIENT_LOCALE` is set automatically from `client_locale=`, so don't put it in `env=`.
|
||||
|
||||
@ -9,7 +9,7 @@ sidebar:
|
||||
|---|---|---|
|
||||
| `SMALLINT` / `INT` / `SERIAL` | `int` | Arbitrary precision on the Python side. |
|
||||
| `BIGINT` / `BIGSERIAL` | `int` | 8-byte two's complement on the wire. |
|
||||
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer — a different wire format from `BIGINT`, not an alias. See below. |
|
||||
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer, a different wire format from `BIGINT` rather than an alias. See below. |
|
||||
| `FLOAT` / `SMALLFLOAT` | `float` | IEEE 754 double / single. |
|
||||
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | Exact precision preserved. |
|
||||
| `CHAR` / `NCHAR` | `str` | Fixed width, space-padded; trailing spaces stripped on decode. |
|
||||
@ -18,11 +18,11 @@ sidebar:
|
||||
| `DATE` | `datetime.date` | |
|
||||
| `DATETIME YEAR TO …` | `datetime.datetime` / `datetime.time` / `datetime.date` | The Python type depends on the field range. |
|
||||
| `INTERVAL DAY TO FRACTION` | `datetime.timedelta` | |
|
||||
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type — `datetime.timedelta` can't represent year-month intervals. |
|
||||
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type, because `datetime.timedelta` can't represent year-month intervals. |
|
||||
| `BYTE` / `TEXT` (legacy in-row blobs) | `bytes` / `str` | |
|
||||
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
|
||||
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string — **not** decomposed into fields. See below. |
|
||||
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema — **not** iterable. See below. |
|
||||
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators, 72 bytes. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
|
||||
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string, **not** decomposed into fields. See below. |
|
||||
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema, **not** iterable. See below. |
|
||||
| `NULL` | `None` | |
|
||||
|
||||
## INT8 and SERIAL8 are not BIGINT
|
||||
@ -34,16 +34,20 @@ Informix has two unrelated 64-bit integer types and they share nothing on the wi
|
||||
| `BIGINT` / `BIGSERIAL` (type 52 / 53) | 8 bytes, two's complement, big-endian |
|
||||
| `INT8` / `SERIAL8` (type 17 / 18) | 10 bytes, sign-magnitude, halves stored high-last |
|
||||
|
||||
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `−n` with *identical* magnitude bytes and the sign in a leading word — decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
|
||||
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `−n` with *identical* magnitude bytes and the sign in a leading word. Decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
|
||||
|
||||
`INT8`/`SERIAL8` are common in schemas predating Informix 11.50, which is when `BIGINT` arrived.
|
||||
|
||||
:::caution[Upgrade if you use these types]
|
||||
`2026.08.27` fixed three decoding bugs: `INT8`/`SERIAL8` weren't decoded at all and returned raw `bytes`; `NCHAR` lost its first character; and `BOOLEAN` corrupted every column after it.
|
||||
|
||||
`2026.08.31` fixed two more in `LVARCHAR` framing — a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
|
||||
`2026.08.31` fixed two more in `LVARCHAR` framing: a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
|
||||
|
||||
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe — upgrade to `2026.08.31`.
|
||||
`2026.09.02` fixed two more, both found by a new check that verifies each row consumed exactly its own payload. Smart LOBs were read as a flat 72-byte field when they actually occupy 149 bytes populated and 5 when NULL, so a `BLOB` or `CLOB` anywhere but the **last** column shifted every column after it. A NULL `SET` / `MULTISET` / `LIST` / `ROW` skipped its length field and did the same. That release also made a mismatch raise where it happens, naming the column and the byte delta, instead of handing back plausible-looking wrong values.
|
||||
|
||||
`2026.09.03` fixed one on the **write** path rather than the read path, which makes it the only entry here that affects data already stored. The `:N` placeholder rewriter was a regular expression and could not see a string literal, so it rewrote the inside of one: `'http://host:8080/x'` was stored as `'http://host?/x'`. Any `HH:MM` time, URL with a port, or `key:value` string was affected, whichever placeholder style you used. If you have written such values through this driver, they are worth checking.
|
||||
|
||||
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe. If you select a `BLOB` or `CLOB` in any position other than last, or a nullable collection column, nothing before `2026.09.02` is safe. Upgrade.
|
||||
:::
|
||||
|
||||
## LVARCHAR wire framing
|
||||
@ -54,7 +58,7 @@ Worth knowing if you're reading captures. Every Informix server we've tested des
|
||||
[1-byte null indicator][4-byte length][content]
|
||||
```
|
||||
|
||||
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte — both carry a length of zero.
|
||||
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte, since both carry a length of zero.
|
||||
|
||||
## DATETIME field ranges
|
||||
|
||||
@ -68,17 +72,17 @@ Informix's `DATETIME YEAR TO X` is field-range typed. The Python type returned d
|
||||
|
||||
## Decimal precision
|
||||
|
||||
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly — no float intermediate.
|
||||
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly, with no float intermediate.
|
||||
|
||||
For binding `Decimal` values into INSERT/UPDATE, the driver uses the column's declared scale. Pass `Decimal` for exact values; `float` works but may cause rounding at the column scale.
|
||||
|
||||
## NULL
|
||||
|
||||
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL — `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
|
||||
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL, because `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
|
||||
|
||||
## Type extensions
|
||||
|
||||
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count** — mirroring the server's own representation — with `years` and `remainder_months` available as derived properties:
|
||||
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count**, mirroring the server's own representation, with `years` and `remainder_months` available as derived properties:
|
||||
|
||||
```python
|
||||
from informix_db import IntervalYM
|
||||
@ -100,8 +104,8 @@ Negative intervals are supported; the sign lives on `months` and propagates to b
|
||||
`informix_db.RowValue` and `informix_db.CollectionValue` are returned for `ROW`, `SET`, `MULTISET`, and `LIST` columns. Both are **opaque wrappers, not decomposed values**:
|
||||
|
||||
```python
|
||||
row_val.raw # bytes — the server's textual representation, e.g. b"ROW('Alice',30)"
|
||||
row_val.schema # str — the column's declared schema
|
||||
row_val.raw # bytes: the server's textual representation, e.g. b"ROW('Alice',30)"
|
||||
row_val.schema # str: the column's declared schema
|
||||
|
||||
coll_val.raw # bytes, e.g. b'LIST{10,20,30}'
|
||||
coll_val.kind # 'set' | 'multiset' | 'list' | 'collection'
|
||||
@ -110,7 +114,7 @@ coll_val.element_schema # str
|
||||
|
||||
They are not iterable and do not expose fields by name. Fully parsing a composite type means reimplementing JDBC's `IfxComplexInput`, which we haven't done.
|
||||
|
||||
If you need individual fields today, project them in SQL — the server does the decomposition for you and you get ordinary typed columns back:
|
||||
If you need individual fields today, project them in SQL. The server does the decomposition for you and you get ordinary typed columns back:
|
||||
|
||||
```python
|
||||
cur.execute("SELECT person.name, person.age FROM staff") # str, int
|
||||
|
||||
@ -55,7 +55,7 @@ docker logs -f informix-dev
|
||||
```
|
||||
|
||||
<Aside type="tip">
|
||||
The `--privileged` flag is required by the dev image — it tries to manage shared memory limits. For production servers this isn't a thing.
|
||||
The `--privileged` flag is required by the dev image, which tries to manage shared memory limits. For production servers this isn't a thing.
|
||||
</Aside>
|
||||
|
||||
## 3. Run your first query
|
||||
@ -88,7 +88,7 @@ Then:
|
||||
python hello.py
|
||||
```
|
||||
|
||||
You should see five rows from Informix's system catalog. If you do, congratulations — you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
|
||||
You should see five rows from Informix's system catalog. If you do, congratulations: you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
|
||||
|
||||
## What just happened
|
||||
|
||||
@ -119,7 +119,12 @@ with informix_db.connect(host="127.0.0.1", port=9088, user="informix",
|
||||
print(cur.fetchone())
|
||||
```
|
||||
|
||||
`?` and `:1` both work — Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
|
||||
`?` and `:1` both work. Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
|
||||
|
||||
Rows come back as plain tuples. If you would rather read them by column
|
||||
name, pass `row_factory=informix_db.Row` to `connect()` and you get
|
||||
`row["name"]` and `row.name` alongside `row[0]`. See
|
||||
[the API reference](/reference/api/#rows).
|
||||
|
||||
## 5. Use the connection pool
|
||||
|
||||
@ -144,7 +149,7 @@ with pool.connection() as conn:
|
||||
pool.close()
|
||||
```
|
||||
|
||||
The pool is thread-safe and has a per-connection wire lock — accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
|
||||
The pool is thread-safe and has a per-connection wire lock, so accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
|
||||
|
||||
## What's next
|
||||
|
||||
@ -154,7 +159,7 @@ The pool is thread-safe and has a per-connection wire lock — accidental sharin
|
||||
|
||||
2. **Connecting to production?** [Connect with TLS →](/how-to/tls/) covers TLS-listener configuration and bring-your-own-context patterns.
|
||||
|
||||
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/) — and the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/), including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
|
||||
4. **Migrating from IfxPy?** [Migrate from IfxPy →](/how-to/migrate-from-ifxpy/) covers the API differences and the things IfxPy does that we don't (yet).
|
||||
|
||||
|
||||
@ -7,7 +7,7 @@ sidebar:
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver — a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
|
||||
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver, a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
|
||||
|
||||
Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Informix Developer Edition Docker container on the same host. Methodology and reproduction steps live in [`tests/benchmarks/compare/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare) in the repo.
|
||||
|
||||
@ -33,9 +33,9 @@ Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Inf
|
||||
|
||||
### Bulk inserts at scale
|
||||
|
||||
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms** — `informix-driver` is 1.6× faster.
|
||||
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms**, so `informix-driver` is 1.6× faster.
|
||||
|
||||
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row — N RTTs at ~80 µs each adds up to the 100 ms gap.
|
||||
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row, and N RTTs at ~80 µs each adds up to the 100 ms gap.
|
||||
|
||||
```python
|
||||
# Both drivers
|
||||
@ -44,8 +44,8 @@ cur.executemany(
|
||||
rows, # list of 10_000 tuples
|
||||
)
|
||||
|
||||
# informix-driver: 161 ms — 10k PDUs sent, then 10k responses drained
|
||||
# IfxPy: 259 ms — 10k round-trips, each blocking on response
|
||||
# informix-driver: 161 ms (10k PDUs sent, then 10k responses drained)
|
||||
# IfxPy: 259 ms (10k round-trips, each blocking on response)
|
||||
```
|
||||
|
||||
### Containerized deployment
|
||||
@ -62,7 +62,7 @@ IfxPy's deployment surface is dramatically larger:
|
||||
- 92 MB IBM OneDB Client tarball
|
||||
- `setuptools < 58` build pin
|
||||
- `LD_LIBRARY_PATH` configuration for four directories
|
||||
- `libcrypt.so.1` (deprecated 2018 — missing on Arch, Fedora 35+, RHEL 9)
|
||||
- `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9)
|
||||
- C compiler in the build image
|
||||
|
||||
For slim images, multi-stage builds, FaaS deployments, or anywhere build-toolchain-on-the-runtime is friction, `informix-driver` is the only reasonable option.
|
||||
@ -88,13 +88,13 @@ async def main():
|
||||
rows = await cur.fetchall()
|
||||
```
|
||||
|
||||
IfxPy has no async support — every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
|
||||
IfxPy has no async support, so every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
|
||||
|
||||
## When IfxPy wins
|
||||
|
||||
### Large analytical fetches
|
||||
|
||||
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 5–15% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock — meaningful but not disqualifying.
|
||||
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 5–15% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock, which is meaningful but not disqualifying.
|
||||
|
||||
The gap is closing phase by phase:
|
||||
|
||||
@ -109,7 +109,7 @@ If you're running analytical reports that pull millions of rows in a single SELE
|
||||
|
||||
### Workloads built around CSDK extensions
|
||||
|
||||
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API — see [the migration guide](/how-to/migrate-from-ifxpy/).
|
||||
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API. See [the migration guide](/how-to/migrate-from-ifxpy/).
|
||||
|
||||
## Methodology
|
||||
|
||||
@ -117,7 +117,7 @@ Benchmarks are pytest-benchmark fixtures in `tests/benchmarks/compare/` against
|
||||
|
||||
Reported numbers are **median over 10+ rounds**, with IQR included. Why median over mean: the first round of any run includes JIT warmup, page-cache miss, and a TCP slow-start round-trip. The mean is contaminated by these one-shot costs in a way that misrepresents steady-state behavior. Median + IQR is what we report.
|
||||
|
||||
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise — a fair reading is "5–15% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
|
||||
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise. A fair reading is "5–15% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
|
||||
|
||||
To reproduce:
|
||||
|
||||
@ -146,4 +146,4 @@ Use IfxPy when:
|
||||
- You're running large analytical SELECTs and the 5–15% decode-side gap matters
|
||||
- You're constrained to Python ≤ 3.11 anyway
|
||||
|
||||
For everything else — the cost-benefit favors `pip install informix-driver`.
|
||||
For everything else, the cost-benefit favors `pip install informix-driver`.
|
||||
|
||||
@ -8,7 +8,7 @@ sidebar:
|
||||
|
||||
The existing tools were not my style.
|
||||
|
||||
Every Informix driver in any language — `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix` — wraps either IBM's C Client SDK or the JDBC JAR. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
|
||||
Every Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That covers `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
|
||||
|
||||
## The problem with IBM's C SDK
|
||||
|
||||
@ -19,19 +19,19 @@ The IBM Informix Client SDK (CSDK), now packaged as part of OneDB Client, is a 9
|
||||
- Permissive `CFLAGS` for the C extension build
|
||||
- Manual download of the 92 MB ODBC tarball
|
||||
- Four `LD_LIBRARY_PATH` directories
|
||||
- `libcrypt.so.1` — deprecated in 2018, missing on Arch, Fedora 35+, RHEL 9
|
||||
- `libcrypt.so.1`, deprecated in 2018 and missing on Arch, Fedora 35+, RHEL 9
|
||||
|
||||
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db` — the distribution name dodges PyPI's 2008-vintage `informixdb` package, the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
|
||||
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db`; the distribution name dodges PyPI's 2008-vintage `informixdb` package, while the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
|
||||
|
||||
## What it does
|
||||
|
||||
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly — the same protocol IBM's JDBC driver uses, the same protocol the CSDK speaks under the hood. No native code is in the thread of execution.
|
||||
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly. It is the same protocol IBM's JDBC driver uses, and the same one the CSDK speaks under the hood. No native code is in the thread of execution.
|
||||
|
||||
The wire protocol was reverse-engineered through three sources:
|
||||
|
||||
1. **Decompiled IBM JDBC driver** (`com.informix.jdbc.IfxConnection` and friends), used as a clean-room reference for PDU shapes and protocol semantics.
|
||||
2. **Annotated `socat` captures** of real client/server traffic against the IBM Informix Developer Edition Docker image.
|
||||
3. **Differential testing** against `IfxPy` — every codec path is tested against the C driver's behavior on the same data.
|
||||
3. **Differential testing** against `IfxPy`, so every codec path is checked against the C driver's behavior on the same data.
|
||||
|
||||
The result is a PEP 249 compliant driver with a sync API, an async API (FastAPI / asyncio compatible), a connection pool, TLS support, smart-LOB read/write, scrollable cursors, fast-path stored procedure invocation, and bulk-insert / bulk-fetch performance within ~10–60% of the C driver depending on workload.
|
||||
|
||||
@ -76,12 +76,12 @@ Every finding from a system-wide failure-mode audit (data correctness, wire safe
|
||||
|
||||
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
|
||||
|
||||
400+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 247/247 on **all three** of 12.10.FC12W1DE, 14.10.FC7W1DE, and 15.0.1.0.3DE — `make test-matrix` runs the lot.
|
||||
470+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 472/472 on 15.0.1.0.3DE and 14.10.FC7W1DE, and 471/472 on 12.10.FC12W1DE (the single skip is a common table expression, which 12.10 predates). `make test-matrix` runs the lot.
|
||||
|
||||
That matrix exists because it turned out to be needed. A user reported corrupted result sets on Informix 12; the cause was three framing bugs that affected every version including the one we tested against, and they'd survived because no fixture used the affected types. Testing one server and inferring the rest is how that happens.
|
||||
|
||||
## Read next
|
||||
|
||||
- **[Install & first query →](/start/quickstart/)** — five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
|
||||
- **[Compared to IfxPy →](/start/vs-ifxpy/)** — full head-to-head benchmarks, methodology, and reproduction.
|
||||
- **[Architecture →](/explain/architecture/)** — how the layers stack: socket, framing, codec, resultset, cursor.
|
||||
- **[Install & first query →](/start/quickstart/)**: five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
|
||||
- **[Compared to IfxPy →](/start/vs-ifxpy/)**: full head-to-head benchmarks, methodology, and reproduction.
|
||||
- **[Architecture →](/explain/architecture/)**: how the layers stack, from socket through framing, codec, resultset, and cursor.
|
||||
|
||||
@ -243,7 +243,7 @@
|
||||
color: var(--sl-color-gray-2);
|
||||
}
|
||||
|
||||
/* Supported Systems "joint" badge — appears below every page's footer */
|
||||
/* Supported Systems "joint" badge, appears below every page's footer */
|
||||
.ifx-ss-badge {
|
||||
margin-top: 3.5rem;
|
||||
padding: 0;
|
||||
@ -337,7 +337,7 @@
|
||||
* ============================================================ */
|
||||
|
||||
@media (max-width: 640px) {
|
||||
/* Defensive guard against any descendant forcing horizontal scroll —
|
||||
/* Defensive guard against any descendant forcing horizontal scroll:
|
||||
the wire-dump's white-space: pre content was overflowing the
|
||||
hero column and propagating up to the page. overflow-x: hidden
|
||||
on .ifx-hero contains it without affecting page-level scroll. */
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
/*
|
||||
* informix-driver docs theme
|
||||
* - Charcoal base (no purple gradients, ever)
|
||||
* - Amber accent — CRT-monitor nod, distinct from sibling sites' cyan
|
||||
* - Amber accent, a CRT-monitor nod distinct from sibling sites' cyan
|
||||
* - Inter for body, IBM Plex Mono for technical bytes
|
||||
*/
|
||||
|
||||
@ -82,7 +82,7 @@
|
||||
--sl-color-hairline-shade: rgba(120, 80, 12, 0.28);
|
||||
}
|
||||
|
||||
/* Tighten heading rhythm — Starlight defaults are a touch loose for technical docs */
|
||||
/* Tighten heading rhythm; Starlight defaults are a touch loose for technical docs */
|
||||
.sl-markdown-content h2 {
|
||||
margin-top: 2.5rem;
|
||||
border-top: 1px solid var(--sl-color-hairline-light);
|
||||
@ -104,7 +104,7 @@
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
/* Tables: dense, technical, with amber column rules — for type-mapping & benchmark tables */
|
||||
/* Tables: dense, technical, amber column rules, for type-mapping & benchmark tables */
|
||||
.sl-markdown-content table {
|
||||
border-collapse: collapse;
|
||||
font-variant-numeric: tabular-nums;
|
||||
@ -124,7 +124,7 @@
|
||||
padding: 0.5rem 0.75rem;
|
||||
}
|
||||
|
||||
/* Anchor links — underline, no rainbow */
|
||||
/* Anchor links: underline, no rainbow */
|
||||
.sl-markdown-content a:not(.sl-anchor-link) {
|
||||
text-decoration: underline;
|
||||
text-decoration-color: var(--sl-color-accent);
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "informix-driver"
|
||||
version = "2026.08.31"
|
||||
version = "2026.09.03"
|
||||
description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries."
|
||||
readme = "README.md"
|
||||
license = { text = "MIT" }
|
||||
|
||||
@ -50,6 +50,7 @@ from .pool import (
|
||||
PoolTimeoutError,
|
||||
create_pool,
|
||||
)
|
||||
from .rows import Row
|
||||
|
||||
# PEP 249 module-level globals
|
||||
apilevel = "2.0"
|
||||
@ -85,6 +86,7 @@ __all__ = [
|
||||
"PoolClosedError",
|
||||
"PoolTimeoutError",
|
||||
"ProgrammingError",
|
||||
"Row",
|
||||
"RowValue",
|
||||
"ServerCapabilities",
|
||||
"Warning",
|
||||
@ -111,6 +113,7 @@ def connect(
|
||||
client_locale: str = "en_US.8859-1",
|
||||
env: dict[str, str] | None = None,
|
||||
autocommit: bool = False,
|
||||
row_factory: object | None = None,
|
||||
tls: bool | ssl.SSLContext = False,
|
||||
tls_server_hostname: str | None = None,
|
||||
) -> Connection:
|
||||
@ -153,4 +156,5 @@ def connect(
|
||||
client_locale=client_locale,
|
||||
env=env,
|
||||
autocommit=autocommit,
|
||||
row_factory=row_factory,
|
||||
)
|
||||
|
||||
@ -29,6 +29,29 @@ class ProtocolError(Exception):
|
||||
"""Raised when wire bytes can't be parsed (truncated stream, bad framing)."""
|
||||
|
||||
|
||||
# Every way a wire read/write can fail, as one tuple.
|
||||
#
|
||||
# This exists because the tuple was written out by hand in six places and
|
||||
# drifted: four of them caught ``(ProtocolError, OSError)`` — which
|
||||
# ``IfxSocket`` never raises. It converts *every* socket failure, including
|
||||
# clean EOF, into ``OperationalError``, and that is a ``DatabaseError``, not
|
||||
# an ``OSError``. So those four handlers could not catch the thing they
|
||||
# existed to catch. The worst of them sat in a ``weakref.finalize`` callback,
|
||||
# where the escaping exception is printed to stderr and swallowed, leaving a
|
||||
# desynchronised connection to be returned to the pool marked healthy.
|
||||
#
|
||||
# Import this rather than re-typing the members. A tuple in one place cannot
|
||||
# drift out of sync with itself.
|
||||
def _wire_error_types() -> tuple[type[BaseException], ...]:
|
||||
# Imported lazily: exceptions.py must stay free of protocol imports.
|
||||
from .exceptions import InterfaceError, OperationalError
|
||||
|
||||
return (ProtocolError, OSError, OperationalError, InterfaceError)
|
||||
|
||||
|
||||
WIRE_ERRORS: tuple[type[BaseException], ...] = _wire_error_types()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Writer
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -269,6 +292,13 @@ class BufferedSocketReader(IfxStreamReader):
|
||||
return b
|
||||
|
||||
def skip(self, n: int) -> None:
|
||||
# The base reader's skip delegates to read_exact, which returns
|
||||
# b"" for n <= 0. This one advances the cursor arithmetically, so
|
||||
# without the guard a negative n *rewinds* it — silently
|
||||
# re-decoding bytes already consumed as if they were the next
|
||||
# field. Same divergence as read_exact, which does guard.
|
||||
if n <= 0:
|
||||
return
|
||||
sock = self._sock
|
||||
sock.fill_recv_buf(n)
|
||||
sock._recv_pos += n
|
||||
|
||||
@ -25,7 +25,7 @@ from dataclasses import dataclass
|
||||
from datetime import timedelta as _timedelta
|
||||
from types import MappingProxyType
|
||||
|
||||
from ._protocol import IfxStreamReader
|
||||
from ._protocol import IfxStreamReader, ProtocolError
|
||||
from ._types import IfxType, base_type, is_nullable
|
||||
from .converters import (
|
||||
_DOUBLE_NULL,
|
||||
@ -247,6 +247,140 @@ _COMPOSITE_UDT_TYPES = frozenset({
|
||||
|
||||
_NUMERIC_TYPES = frozenset({_TC_DECIMAL, _TC_MONEY})
|
||||
|
||||
|
||||
def _read_udt_envelope(payload: bytes, offset: int) -> tuple[int, bytes | None]:
|
||||
"""Read the UDT wire envelope: ``[1-byte indicator][int32 length][data]``.
|
||||
|
||||
Returns ``(new_offset, body)`` with ``body`` ``None`` when the
|
||||
indicator says NULL. The length field is present either way — it
|
||||
belongs to the envelope, not the value — so the offset advances by
|
||||
5 for a NULL and 5 + length otherwise.
|
||||
|
||||
This exists because the same envelope was hand-written four times and
|
||||
three of the copies were wrong, each in a different way:
|
||||
|
||||
* BOOLEAN read ``encoded_length`` (1 byte) instead of the envelope,
|
||||
leaving 5 bytes behind.
|
||||
* The composite-UDT branch returned on the indicator without reading
|
||||
the length, leaving 4 bytes behind.
|
||||
* BLOB/CLOB read a flat 72 bytes, leaving 77 behind.
|
||||
|
||||
Only the UDTVAR(lvarchar) copy was right, and only after two separate
|
||||
fixes. Four copies of one three-line format is four chances to get it
|
||||
wrong; this is one.
|
||||
"""
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
length = int.from_bytes(payload[offset : offset + 4], "big", signed=True)
|
||||
offset += 4
|
||||
if indicator == 1:
|
||||
return offset, None
|
||||
if length < 0:
|
||||
raise ProtocolError(
|
||||
f"UDT envelope declared a negative length ({length}) at payload "
|
||||
f"offset {offset - 4}; the wire is desynchronised"
|
||||
)
|
||||
body = payload[offset : offset + length]
|
||||
offset += length
|
||||
return offset, body
|
||||
|
||||
|
||||
def _decode_lob_locator(body: bytes, extended_id: int):
|
||||
"""Turn a smart-LOB envelope body into a Blob/ClobLocator.
|
||||
|
||||
The body is the 72-byte locator **hex-encoded as 144 ASCII characters**,
|
||||
not the raw locator. We previously consumed a flat 72 bytes, which took
|
||||
the first half of that hex text and handed it to ``BlobLocator`` as if
|
||||
it were binary — so ``BlobLocator.raw`` never actually held a locator.
|
||||
It went unnoticed because ``read_blob_column`` resolves LOBs through
|
||||
the server-side ``lotofile`` function and never uses the locator.
|
||||
"""
|
||||
cls = BlobLocator if extended_id == 10 else ClobLocator
|
||||
try:
|
||||
raw = bytes.fromhex(body.decode("ascii"))
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
# Not hex — surface what arrived rather than guessing, but only if
|
||||
# it is already the right size for a locator.
|
||||
raw = bytes(body)
|
||||
if len(raw) != 72:
|
||||
raise ProtocolError(
|
||||
f"smart-LOB locator decoded to {len(raw)} bytes, expected 72 "
|
||||
f"(envelope body was {len(body)} bytes)"
|
||||
)
|
||||
return cls(raw=raw)
|
||||
|
||||
|
||||
def _row_not_consumed(
|
||||
offset: int, payload_len: int, columns: list[ColumnInfo]
|
||||
) -> None:
|
||||
"""Raise when a decoded row didn't consume exactly its payload.
|
||||
|
||||
THIS IS THE CHECK THAT WOULD HAVE CAUGHT EVERY FRAMING BUG WE SHIPPED.
|
||||
|
||||
Thirteen of them reached users, and all thirteen were the same defect:
|
||||
a column consumed the wrong number of bytes. Not one raised at the
|
||||
point of the mistake — the wrong width produced a plausible value and
|
||||
corrupted whatever came *next*, so the damage surfaced in a different
|
||||
column, a different row, or a different statement.
|
||||
|
||||
Every one was detectable here for free. ``payload`` is a fully
|
||||
extracted ``bytes`` object of known length (the wire's even-alignment
|
||||
pad is consumed separately by the caller), so after decoding N columns
|
||||
the offset must land exactly on the end. Anything else means the codec
|
||||
and the server disagree about the row's shape, and continuing can only
|
||||
produce wrong answers.
|
||||
|
||||
A previous attempt at this check was reverted, and the comment
|
||||
explaining why survived long after its reason did: it cited the
|
||||
LVARCHAR odd-length pad, which was itself a bug and has since been
|
||||
removed. With that gone the invariant is exact.
|
||||
|
||||
The message names the column shape because "row decode failed" is not
|
||||
actionable — the whole point is to say *which* column shape and by how
|
||||
many bytes, so the next report arrives with the answer in it.
|
||||
"""
|
||||
delta = offset - payload_len
|
||||
direction = "over-read" if delta > 0 else "under-read"
|
||||
shape = ", ".join(
|
||||
f"{c.name}:tc={c.type_code}"
|
||||
f"{'/ext=' + c.extended_name if c.extended_name else ''}"
|
||||
f"/enclen={c.encoded_length}"
|
||||
for c in columns
|
||||
)
|
||||
raise ProtocolError(
|
||||
f"row decoder {direction} by {abs(delta)} byte(s): consumed "
|
||||
f"{offset} of {payload_len} payload bytes. This means the driver "
|
||||
f"and the server disagree about a column's wire width, so every "
|
||||
f"column after the offending one is unreliable. Column shape: "
|
||||
f"[{shape}]"
|
||||
)
|
||||
|
||||
|
||||
def _packed_width(encoded_length: int) -> int:
|
||||
"""On-wire byte width for the four types whose ``encoded_length``
|
||||
packs two fields into ``(high << 8) | low``: DECIMAL, MONEY,
|
||||
DATETIME and INTERVAL.
|
||||
|
||||
Mirrors ``IfxColumnInfo.adjustedColumnLength`` / ``IfxDecimal.decLength``::
|
||||
|
||||
((ColLength >> 8 & 0xFF) + (ColLength & 0xFF & 1) + 3) / 2
|
||||
|
||||
The ``low & 1`` term is the part that is easy to miss, and dropping
|
||||
it is wrong exactly when the high byte is even and the low byte is
|
||||
odd. For DECIMAL/MONEY the low byte is the scale, and an unscaled
|
||||
``DECIMAL(p)`` reports scale **255** — odd — so every floating
|
||||
DECIMAL with even precision was read one byte short, desyncing every
|
||||
column after it. ``DECIMAL(16)`` is 10 bytes on the wire, not 9.
|
||||
|
||||
For DATETIME and INTERVAL the low byte is the qualifier, whose
|
||||
parity happens to track the digit count, so the two formulas agree
|
||||
there in practice. They are unified here anyway: one rule taken from
|
||||
the reference beats two that coincide for reasons nobody wrote down.
|
||||
"""
|
||||
high = (encoded_length >> 8) & 0xFF
|
||||
low = encoded_length & 0xFF
|
||||
return (high + (low & 1) + 3) // 2
|
||||
|
||||
# Types that are fixed-width on the wire AND have a registered decoder
|
||||
# in ``FIXED_WIDTHS``: SMALLINT, INT, SERIAL, SMFLOAT, FLOAT, BIGINT,
|
||||
# BIGSERIAL, DATE, BOOL. These are the most common types in any real
|
||||
@ -319,20 +453,17 @@ def compile_column_readers(columns: list[ColumnInfo]) -> list[tuple]:
|
||||
continue
|
||||
|
||||
if tc in _NUMERIC_TYPES:
|
||||
precision = (col.encoded_length >> 8) & 0xFF
|
||||
width = (precision + 1) // 2 + 1
|
||||
width = _packed_width(col.encoded_length)
|
||||
readers.append((_RK_DECIMAL, width, DECODERS[tc]))
|
||||
continue
|
||||
|
||||
if tc == _TC_DATETIME:
|
||||
digit_count = (col.encoded_length >> 8) & 0xFF
|
||||
width = (digit_count + 1) // 2 + 1
|
||||
width = _packed_width(col.encoded_length)
|
||||
readers.append((_RK_DATETIME, width, col.encoded_length))
|
||||
continue
|
||||
|
||||
if tc == _TC_INTERVAL:
|
||||
digit_count = (col.encoded_length >> 8) & 0xFF
|
||||
width = (digit_count + 1) // 2 + 1
|
||||
width = _packed_width(col.encoded_length)
|
||||
readers.append((_RK_INTERVAL, width, col.encoded_length))
|
||||
continue
|
||||
|
||||
@ -517,6 +648,12 @@ def compile_row_decoder(
|
||||
# Unknown kind — abort codegen, caller falls back.
|
||||
return None
|
||||
|
||||
# Same end-of-row reconciliation the interpreted paths do. The
|
||||
# generated function is the hot path, so this must be emitted here
|
||||
# too — a check that only guards the slow path guards nothing in
|
||||
# production.
|
||||
lines.append(" if offset != len(payload):")
|
||||
lines.append(" _row_short(offset, len(payload))")
|
||||
if val_names:
|
||||
lines.append(f" return ({', '.join(val_names)},)")
|
||||
else:
|
||||
@ -541,6 +678,13 @@ def compile_row_decoder(
|
||||
"_decode_datetime": _decode_datetime,
|
||||
"_decode_interval": _decode_interval,
|
||||
"_legacy_dispatch_one_column": _legacy_dispatch_one_column,
|
||||
# Bound to this shape's columns so the raised message can name them
|
||||
# without the generated source having to carry the list itself.
|
||||
"_row_short": (
|
||||
lambda offset, payload_len, _cols=columns: _row_not_consumed(
|
||||
offset, payload_len, _cols
|
||||
)
|
||||
),
|
||||
"_UNPACK_SHORT": _UNPACK_SHORT,
|
||||
"_UNPACK_INT": _UNPACK_INT,
|
||||
"_UNPACK_LONG": _UNPACK_LONG,
|
||||
@ -587,11 +731,16 @@ def _legacy_dispatch_one_column(
|
||||
"""
|
||||
# BLOB / CLOB locator (UDTFIXED + extended_id 10/11)
|
||||
if tc == _TC_UDTFIXED and col.extended_id in (10, 11):
|
||||
width = col.encoded_length
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
cls = BlobLocator if col.extended_id == 10 else ClobLocator
|
||||
return offset, cls(raw=bytes(raw))
|
||||
# Smart LOBs use the UDT envelope, NOT a flat encoded_length field.
|
||||
# Measured on Informix 15: a populated BLOB column is 149 bytes
|
||||
# ([ind=0][len=144][144 hex chars]) and a NULL one is 5, while
|
||||
# encoded_length reports 72. Consuming 72 left 77 bytes behind and
|
||||
# corrupted every column after a LOB. Found by the end-of-row
|
||||
# reconciliation check, which fired on its first run.
|
||||
offset, body = _read_udt_envelope(payload, offset)
|
||||
if body is None:
|
||||
return offset, None
|
||||
return offset, _decode_lob_locator(body, col.extended_id)
|
||||
|
||||
# BOOLEAN. The server describes it as UDTFIXED (41) with
|
||||
# extended_name='boolean' and encoded_length=1, but ``encoded_length``
|
||||
@ -605,24 +754,28 @@ def _legacy_dispatch_one_column(
|
||||
if tc == _TC_UDTFIXED and (
|
||||
col.extended_name == "boolean" or col.extended_id == 5
|
||||
):
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
|
||||
offset += 4
|
||||
raw = payload[offset:offset + length]
|
||||
offset += length
|
||||
if indicator == 1:
|
||||
offset, body = _read_udt_envelope(payload, offset)
|
||||
if body is None:
|
||||
return offset, None
|
||||
return offset, bool(raw and raw[0] in (ord("t"), ord("T"), 1))
|
||||
return offset, bool(body and body[0] in (ord("t"), ord("T"), 1))
|
||||
|
||||
# ROW / COLLECTION composite UDT
|
||||
if tc in _COMPOSITE_UDT_TYPES:
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
if indicator == 1:
|
||||
return offset, None
|
||||
# The 4-byte length is part of the envelope and is present even when
|
||||
# the indicator says NULL — identical to the UDTVAR(lvarchar) branch
|
||||
# below, which decodes the same `[ind][int32 len][data]` shape.
|
||||
# Returning early on the indicator left those 4 bytes unread and
|
||||
# corrupted the following column. Wire evidence, Informix 15,
|
||||
# (INT, SET(INT), VARCHAR) with a NULL set:
|
||||
# 00 00 00 08 | 01 | 00 00 00 00 | 03 78 79 7a
|
||||
# a = 8 | ind | length = 0 | [3]"xyz"
|
||||
# Before the fix the VARCHAR decoded as '' instead of 'xyz'.
|
||||
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
|
||||
offset += 4
|
||||
if indicator == 1:
|
||||
return offset, None
|
||||
raw = bytes(payload[offset:offset + length])
|
||||
offset += length
|
||||
if tc == _TC_ROW:
|
||||
@ -807,6 +960,8 @@ def parse_tuple_payload(
|
||||
payload, offset, tc, col, encoding
|
||||
)
|
||||
values.append(value)
|
||||
if offset != len(payload):
|
||||
_row_not_consumed(offset, len(payload), columns)
|
||||
return tuple(values)
|
||||
|
||||
# Legacy slow path (no pre-compiled readers).
|
||||
@ -874,8 +1029,7 @@ def parse_tuple_payload(
|
||||
# the high byte of encoded_length (packed as (precision << 8) | scale).
|
||||
# Per IfxRowColumn.loadColumnData and IfxToJavaDecimal byte sizing.
|
||||
if tc in _NUMERIC_TYPES:
|
||||
precision = (col.encoded_length >> 8) & 0xFF
|
||||
width = (precision + 1) // 2 + 1
|
||||
width = _packed_width(col.encoded_length)
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
try:
|
||||
@ -889,8 +1043,7 @@ def parse_tuple_payload(
|
||||
# (start_TU << 4) | end_TU). The decoder needs the qualifier too,
|
||||
# so we call it directly here rather than via the dispatch.
|
||||
if tc == _TC_DATETIME:
|
||||
digit_count = (col.encoded_length >> 8) & 0xFF
|
||||
width = (digit_count + 1) // 2 + 1
|
||||
width = _packed_width(col.encoded_length)
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
values.append(_decode_datetime(raw, col.encoded_length))
|
||||
@ -903,8 +1056,7 @@ def parse_tuple_payload(
|
||||
# qualifier is needed at decode time, so we bypass the generic
|
||||
# dispatch.
|
||||
if tc == _TC_INTERVAL:
|
||||
digit_count = (col.encoded_length >> 8) & 0xFF
|
||||
width = (digit_count + 1) // 2 + 1
|
||||
width = _packed_width(col.encoded_length)
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
values.append(_decode_interval(raw, col.encoded_length))
|
||||
@ -916,11 +1068,13 @@ def parse_tuple_payload(
|
||||
# we read here are an opaque server-side reference, NOT the
|
||||
# actual data. Phase 10 lets users fetch via lotofile + SQ_FILE.
|
||||
if tc == _TC_UDTFIXED and col.extended_id in (10, 11):
|
||||
width = col.encoded_length
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
cls = BlobLocator if col.extended_id == 10 else ClobLocator
|
||||
values.append(cls(raw=bytes(raw)))
|
||||
# UDT envelope, not a flat encoded_length field — see the
|
||||
# matching branch in _legacy_dispatch_one_column.
|
||||
offset, body = _read_udt_envelope(payload, offset)
|
||||
values.append(
|
||||
None if body is None
|
||||
else _decode_lob_locator(body, col.extended_id)
|
||||
)
|
||||
continue
|
||||
|
||||
# BOOLEAN — UDT envelope, not a bare byte. See the matching branch
|
||||
@ -928,18 +1082,11 @@ def parse_tuple_payload(
|
||||
if tc == _TC_UDTFIXED and (
|
||||
col.extended_name == "boolean" or col.extended_id == 5
|
||||
):
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
length = int.from_bytes(
|
||||
payload[offset:offset + 4], "big", signed=True
|
||||
offset, body = _read_udt_envelope(payload, offset)
|
||||
values.append(
|
||||
None if body is None
|
||||
else bool(body and body[0] in (ord("t"), ord("T"), 1))
|
||||
)
|
||||
offset += 4
|
||||
raw = payload[offset:offset + length]
|
||||
offset += length
|
||||
if indicator == 1:
|
||||
values.append(None)
|
||||
else:
|
||||
values.append(bool(raw and raw[0] in (ord("t"), ord("T"), 1)))
|
||||
continue
|
||||
|
||||
# ROW / COLLECTION (Phase 12): composite UDTs. Wire format is
|
||||
@ -956,13 +1103,15 @@ def parse_tuple_payload(
|
||||
if tc in _COMPOSITE_UDT_TYPES:
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
if indicator == 1: # null
|
||||
values.append(None)
|
||||
continue
|
||||
# Length is present even when NULL — see the matching branch in
|
||||
# _legacy_dispatch_one_column for the wire evidence.
|
||||
length = int.from_bytes(
|
||||
payload[offset:offset + 4], "big", signed=True
|
||||
)
|
||||
offset += 4
|
||||
if indicator == 1: # null
|
||||
values.append(None)
|
||||
continue
|
||||
raw = bytes(payload[offset:offset + length])
|
||||
offset += length
|
||||
if tc == _TC_ROW:
|
||||
@ -1028,4 +1177,6 @@ def parse_tuple_payload(
|
||||
# by Python's slicing semantics for strings — short = harmless).
|
||||
# If a future protocol message produces actual garbage here, add a
|
||||
# branch-local check at the offending dispatch path.
|
||||
if offset != len(payload):
|
||||
_row_not_consumed(offset, len(payload), columns)
|
||||
return tuple(values)
|
||||
|
||||
@ -17,11 +17,35 @@ the rest of the protocol layer.
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import socket
|
||||
import ssl
|
||||
|
||||
from ._protocol import ProtocolError
|
||||
from .exceptions import InterfaceError, OperationalError
|
||||
|
||||
|
||||
def _max_read_bytes() -> int:
|
||||
"""Ceiling for a single length-prefixed read, from IFX_MAX_READ_BYTES.
|
||||
|
||||
256 MiB by default: comfortably above any row, string table, or blob
|
||||
chunk a real query produces, and far below the values a desynced
|
||||
stream invents. A genuinely larger single value is possible (a very
|
||||
large TEXT column read in one go), which is why the knob exists.
|
||||
"""
|
||||
raw = os.environ.get("IFX_MAX_READ_BYTES")
|
||||
if raw:
|
||||
try:
|
||||
value = int(raw)
|
||||
except ValueError:
|
||||
value = 0
|
||||
if value > 0:
|
||||
return value
|
||||
return 256 * 1024 * 1024
|
||||
|
||||
|
||||
MAX_READ_BYTES = _max_read_bytes()
|
||||
|
||||
# A ``tls`` parameter to ``IfxSocket`` accepts:
|
||||
# False (default) — plain TCP
|
||||
# True — TLS with verification disabled (dev / self-signed)
|
||||
@ -124,10 +148,42 @@ class IfxSocket:
|
||||
raise OperationalError(f"write failed: {e}") from e
|
||||
|
||||
def read_exact(self, n: int) -> bytes:
|
||||
"""Read exactly ``n`` bytes or raise on EOF / timeout."""
|
||||
"""Read exactly ``n`` bytes or raise on EOF / timeout.
|
||||
|
||||
Consumes the read-ahead buffer before touching the socket. This
|
||||
matters because two readers share one stream: ``BufferedSocketReader``
|
||||
fills ``_recv_buf`` (over-reading by design, up to ``_recv_size``),
|
||||
while ``Connection._drain_to_eot``, ``_raise_sq_err`` and the
|
||||
login path call this method directly. Recv'ing here while bytes
|
||||
sat unconsumed in the buffer would skip them, and skipped bytes
|
||||
in a length-framed protocol don't announce themselves — the next
|
||||
read lands mid-field and every subsequent one is wrong.
|
||||
|
||||
No workload triggers it today: the server sends one response per
|
||||
request, so recv returns exactly that response and the buffered
|
||||
reader consumes all of it before control returns here. That is a
|
||||
property of the traffic, not of the code. Pipelined executemany
|
||||
already puts multiple responses in flight, and the buffer is
|
||||
connection-scoped precisely so read-ahead can cross response
|
||||
boundaries. Making the two paths agree by construction costs one
|
||||
branch on a cold path.
|
||||
"""
|
||||
if self._sock is None:
|
||||
raise InterfaceError("socket is closed")
|
||||
if n <= 0:
|
||||
return b""
|
||||
wanted = n
|
||||
chunks: list[bytes] = []
|
||||
buffered = len(self._recv_buf) - self._recv_pos
|
||||
if buffered > 0:
|
||||
take = min(buffered, n)
|
||||
chunks.append(
|
||||
bytes(self._recv_buf[self._recv_pos : self._recv_pos + take])
|
||||
)
|
||||
self._recv_pos += take
|
||||
n -= take
|
||||
if n == 0:
|
||||
return chunks[0]
|
||||
remaining = n
|
||||
while remaining > 0:
|
||||
try:
|
||||
@ -138,7 +194,8 @@ class IfxSocket:
|
||||
if not chunk:
|
||||
self._force_close()
|
||||
raise OperationalError(
|
||||
f"server closed connection mid-read (wanted {n} bytes, got {n - remaining})"
|
||||
f"server closed connection mid-read "
|
||||
f"(wanted {wanted} bytes, got {wanted - remaining})"
|
||||
)
|
||||
chunks.append(chunk)
|
||||
remaining -= len(chunk)
|
||||
@ -155,6 +212,21 @@ class IfxSocket:
|
||||
"""
|
||||
if self._sock is None:
|
||||
raise InterfaceError("socket is closed")
|
||||
if need > MAX_READ_BYTES:
|
||||
# ``need`` is almost always a length field straight off the
|
||||
# wire. Trusting it means a corrupt or desynced stream turns
|
||||
# into an allocation of whatever that field happened to say —
|
||||
# a garbage 0x7FFFFFFF reads as a 2 GB request, and the loop
|
||||
# below sits in recv until the read timeout while the buffer
|
||||
# grows. Refusing names the number instead, which is the
|
||||
# diagnostic: a length that absurd means framing is already
|
||||
# lost upstream, not that the row is genuinely that big.
|
||||
raise ProtocolError(
|
||||
f"refusing to read {need} bytes in one field (limit "
|
||||
f"{MAX_READ_BYTES}). Either the wire has desynced, or a "
|
||||
f"single value really is this large — raise the limit "
|
||||
f"with the IFX_MAX_READ_BYTES environment variable."
|
||||
)
|
||||
avail = len(self._recv_buf) - self._recv_pos
|
||||
if avail >= need:
|
||||
return
|
||||
|
||||
@ -62,7 +62,10 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import contextlib
|
||||
import functools
|
||||
import threading
|
||||
import weakref
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from typing import Any, TypeVar
|
||||
|
||||
from . import connect as _sync_connect
|
||||
@ -79,6 +82,61 @@ def _to_thread(fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
|
||||
return asyncio.to_thread(fn, *args, **kwargs)
|
||||
|
||||
|
||||
def _run_on(
|
||||
executor: ThreadPoolExecutor,
|
||||
fn: Callable[..., T],
|
||||
*args: Any,
|
||||
**kwargs: Any,
|
||||
) -> Awaitable[T]:
|
||||
"""Await ``fn`` on a specific executor rather than the loop's default."""
|
||||
loop = asyncio.get_running_loop()
|
||||
return loop.run_in_executor(executor, functools.partial(fn, *args, **kwargs))
|
||||
|
||||
|
||||
def _connection_executor(conn: _SyncConnection) -> ThreadPoolExecutor:
|
||||
"""The dedicated worker thread for one connection, created on demand.
|
||||
|
||||
``asyncio.to_thread`` runs on the event loop's default executor, which
|
||||
the whole process shares and which is sized from the CPU count —
|
||||
``min(32, cpu_count + 4)``, so six threads on a two-CPU container.
|
||||
Two consequences, both silent.
|
||||
|
||||
A cancelled await does not stop its worker. ``asyncio.to_thread``
|
||||
cannot interrupt the thread, so it keeps running the wire call until
|
||||
the read timeout expires. Cancellation is ordinary in a web app — a
|
||||
client disconnect cancels the request task — so a handful of them
|
||||
pins every thread in the shared pool and unrelated ``to_thread`` work
|
||||
anywhere else in the process stops dead. Measured: six cancelled
|
||||
calls against a six-worker default executor starve an unrelated
|
||||
``to_thread`` indefinitely.
|
||||
|
||||
And pool concurrency was capped by the same number without saying so.
|
||||
A pool with ``max_size=20`` on a two-CPU box ran six queries at once.
|
||||
|
||||
One thread per connection is the right size rather than a compromise:
|
||||
the sync connection serializes every wire operation on its own lock,
|
||||
so a second thread could do nothing but wait for the first. It also
|
||||
rules out a deadlock that a shared pool-sized executor invites — with
|
||||
N threads and N connections, N tasks blocked in ``acquire`` occupy
|
||||
every thread while the connection they are waiting for is held by a
|
||||
task that now needs a thread of its own to finish and release it.
|
||||
|
||||
The executor lives on the sync connection so it survives being
|
||||
returned to the pool and handed out again. The finalizer is the
|
||||
backstop for a connection dropped without ``close()``: a
|
||||
``ThreadPoolExecutor`` that is never shut down leaves its worker
|
||||
parked on the work queue for the life of the process.
|
||||
"""
|
||||
executor = getattr(conn, "_async_executor", None)
|
||||
if executor is None:
|
||||
executor = ThreadPoolExecutor(
|
||||
max_workers=1, thread_name_prefix="informix-conn"
|
||||
)
|
||||
conn._async_executor = executor
|
||||
weakref.finalize(conn, executor.shutdown, wait=False)
|
||||
return executor
|
||||
|
||||
|
||||
class AsyncCursor:
|
||||
"""Async wrapper over a sync :class:`Cursor`. Each I/O call awaits
|
||||
a thread-offloaded version of the sync operation.
|
||||
@ -88,10 +146,14 @@ class AsyncCursor:
|
||||
paying the thread-hop cost.
|
||||
"""
|
||||
|
||||
__slots__ = ("_cur",)
|
||||
__slots__ = ("_cur", "_run")
|
||||
|
||||
def __init__(self, cur: _SyncCursor):
|
||||
def __init__(self, cur: _SyncCursor, run: Callable[..., Awaitable[Any]]):
|
||||
self._cur = cur
|
||||
# The owning connection's runner, so cursor I/O lands on that
|
||||
# connection's dedicated thread rather than the shared default
|
||||
# executor. See _connection_executor.
|
||||
self._run = run
|
||||
|
||||
# -- Pass-through synchronous attributes (no I/O) ---------------------
|
||||
|
||||
@ -120,32 +182,32 @@ class AsyncCursor:
|
||||
async def execute(
|
||||
self, operation: str, parameters: Any = None
|
||||
) -> None:
|
||||
await _to_thread(self._cur.execute, operation, parameters)
|
||||
await self._run(self._cur.execute, operation, parameters)
|
||||
|
||||
async def executemany(
|
||||
self, operation: str, seq_of_parameters: Any
|
||||
) -> None:
|
||||
await _to_thread(
|
||||
await self._run(
|
||||
self._cur.executemany, operation, list(seq_of_parameters)
|
||||
)
|
||||
|
||||
async def fetchone(self) -> tuple | None:
|
||||
return await _to_thread(self._cur.fetchone)
|
||||
return await self._run(self._cur.fetchone)
|
||||
|
||||
async def fetchmany(self, size: int | None = None) -> list[tuple]:
|
||||
return await _to_thread(self._cur.fetchmany, size)
|
||||
return await self._run(self._cur.fetchmany, size)
|
||||
|
||||
async def fetchall(self) -> list[tuple]:
|
||||
return await _to_thread(self._cur.fetchall)
|
||||
return await self._run(self._cur.fetchall)
|
||||
|
||||
async def close(self) -> None:
|
||||
await _to_thread(self._cur.close)
|
||||
await self._run(self._cur.close)
|
||||
|
||||
# Phase 10/11 BLOB helpers (preserve the sync API surface)
|
||||
async def read_blob_column(
|
||||
self, sql: str, params: tuple = ()
|
||||
) -> bytes | None:
|
||||
return await _to_thread(self._cur.read_blob_column, sql, params)
|
||||
return await self._run(self._cur.read_blob_column, sql, params)
|
||||
|
||||
async def write_blob_column(
|
||||
self,
|
||||
@ -155,7 +217,7 @@ class AsyncCursor:
|
||||
*,
|
||||
clob: bool = False,
|
||||
) -> None:
|
||||
await _to_thread(
|
||||
await self._run(
|
||||
functools.partial(
|
||||
self._cur.write_blob_column,
|
||||
sql, blob_data, params, clob=clob,
|
||||
@ -177,14 +239,23 @@ class AsyncCursor:
|
||||
class AsyncConnection:
|
||||
"""Async wrapper over a sync :class:`Connection`."""
|
||||
|
||||
__slots__ = ("_conn",)
|
||||
__slots__ = ("_conn", "_executor")
|
||||
|
||||
def __init__(self, conn: _SyncConnection):
|
||||
self._conn = conn
|
||||
self._executor = _connection_executor(conn)
|
||||
|
||||
def _run(self, fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
|
||||
"""Run a blocking connection call on this connection's own thread."""
|
||||
return _run_on(self._executor, fn, *args, **kwargs)
|
||||
|
||||
@classmethod
|
||||
async def connect(cls, *args: Any, **kwargs: Any) -> AsyncConnection:
|
||||
"""Open a connection. Same parameters as :func:`informix_db.connect`."""
|
||||
"""Open a connection. Same parameters as :func:`informix_db.connect`.
|
||||
|
||||
The connect itself still goes to the default executor -- there is
|
||||
no connection yet to own a thread, and it is one bounded call.
|
||||
"""
|
||||
sync_conn = await _to_thread(
|
||||
functools.partial(_sync_connect, *args, **kwargs)
|
||||
)
|
||||
@ -195,22 +266,46 @@ class AsyncConnection:
|
||||
return self._conn.closed
|
||||
|
||||
async def cursor(self) -> AsyncCursor:
|
||||
sync_cur = await _to_thread(self._conn.cursor)
|
||||
return AsyncCursor(sync_cur)
|
||||
sync_cur = await self._run(self._conn.cursor)
|
||||
return AsyncCursor(sync_cur, self._run)
|
||||
|
||||
async def commit(self) -> None:
|
||||
await _to_thread(self._conn.commit)
|
||||
await self._run(self._conn.commit)
|
||||
|
||||
async def rollback(self) -> None:
|
||||
await _to_thread(self._conn.rollback)
|
||||
await self._run(self._conn.rollback)
|
||||
|
||||
async def close(self) -> None:
|
||||
await _to_thread(self._conn.close)
|
||||
"""Close the connection and stop the thread that served it.
|
||||
|
||||
One thread per connection is only affordable if the thread goes
|
||||
away with the connection. ``ThreadPoolExecutor`` workers park on
|
||||
the work queue rather than exiting when idle, so an executor that
|
||||
is never shut down leaks its thread for the life of the process —
|
||||
the ``weakref.finalize`` in ``_connection_executor`` is a backstop
|
||||
for connections dropped without ``close()``, not a substitute for
|
||||
closing here.
|
||||
|
||||
``wait=False`` because we are on the event loop: the worker has
|
||||
just finished the close and needs no waiting, and blocking the
|
||||
loop to confirm that would be the one thing this module exists to
|
||||
avoid.
|
||||
"""
|
||||
try:
|
||||
await self._run(self._conn.close)
|
||||
finally:
|
||||
self._executor.shutdown(wait=False)
|
||||
# Drop the reference too. A shut-down executor rejects new
|
||||
# work, so leaving it attached would turn any later wrap of
|
||||
# this sync connection into a RuntimeError rather than a
|
||||
# fresh thread.
|
||||
with contextlib.suppress(AttributeError):
|
||||
del self._conn._async_executor
|
||||
|
||||
async def fast_path_call(
|
||||
self, signature: str, *params: object
|
||||
) -> list[object]:
|
||||
return await _to_thread(self._conn.fast_path_call, signature, *params)
|
||||
return await self._run(self._conn.fast_path_call, signature, *params)
|
||||
|
||||
# Async context-manager support
|
||||
async def __aenter__(self) -> AsyncConnection:
|
||||
@ -249,14 +344,69 @@ class AsyncConnectionPool:
|
||||
return self._pool.idle_count
|
||||
|
||||
async def acquire(self, timeout: float | None = None) -> AsyncConnection:
|
||||
sync_conn = await _to_thread(self._pool.acquire, timeout)
|
||||
"""Acquire a connection, surviving cancellation of the waiter.
|
||||
|
||||
``asyncio.to_thread`` cannot interrupt its worker. If the task
|
||||
awaiting an acquire is cancelled while the worker is still
|
||||
blocked waiting for a free connection, the worker eventually
|
||||
succeeds and hands back a connection **that nobody owns** — it
|
||||
is checked out of the pool and never returned. Repeat that and
|
||||
the pool starves to death.
|
||||
|
||||
This is not hypothetical for anyone serving HTTP: a client
|
||||
disconnecting cancels the request task, and under load those
|
||||
cancellations land precisely while waiting for a connection. The
|
||||
pool then dies one slot at a time, and only under load.
|
||||
|
||||
So the inner future is shielded — cancelling the caller must not
|
||||
orphan a result we still need to see — and if the caller does go
|
||||
away, a callback returns whatever the worker produced to the
|
||||
pool. ``add_done_callback`` fires immediately when the future has
|
||||
already resolved, so the "worker finished a moment before the
|
||||
cancellation" race is covered by the same code path.
|
||||
"""
|
||||
fut = asyncio.ensure_future(_to_thread(self._pool.acquire, timeout))
|
||||
try:
|
||||
sync_conn = await asyncio.shield(fut)
|
||||
except asyncio.CancelledError:
|
||||
fut.add_done_callback(self._return_orphan)
|
||||
raise
|
||||
return AsyncConnection(sync_conn)
|
||||
|
||||
def _return_orphan(self, fut: asyncio.Future) -> None:
|
||||
"""Give back a connection whose acquirer was cancelled.
|
||||
|
||||
Runs on the event loop, so it must not block: ``release`` takes
|
||||
the connection's wire lock and can wait. Hand off to a short-lived
|
||||
daemon thread rather than the running loop — this path also has to
|
||||
work while the loop is shutting down, which is exactly when
|
||||
``create_task`` is unavailable.
|
||||
"""
|
||||
if fut.cancelled() or fut.exception() is not None:
|
||||
return
|
||||
conn = fut.result()
|
||||
|
||||
def _release() -> None:
|
||||
# broken=False: the connection was never handed to anyone, so
|
||||
# its wire is untouched and it is safe to reuse. Evicting here
|
||||
# would trade a leak for needless reconnect churn.
|
||||
with contextlib.suppress(Exception):
|
||||
self._pool.release(conn, broken=False)
|
||||
|
||||
threading.Thread(target=_release, daemon=True).start()
|
||||
|
||||
async def release(
|
||||
self, conn: AsyncConnection, *, broken: bool = False
|
||||
) -> None:
|
||||
await _to_thread(
|
||||
functools.partial(self._pool.release, conn._conn, broken=broken)
|
||||
# On the connection's own thread, not the default executor. That
|
||||
# thread is idle by definition — the caller is done with the
|
||||
# connection — and it keeps the release off a shared pool that
|
||||
# tasks blocked in ``acquire`` may have filled. Release has to
|
||||
# win that race: it is what frees the connection they are
|
||||
# waiting for.
|
||||
await _run_on(
|
||||
conn._executor,
|
||||
functools.partial(self._pool.release, conn._conn, broken=broken),
|
||||
)
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
|
||||
@ -18,6 +18,7 @@ import socket as socket_mod
|
||||
import ssl
|
||||
import struct
|
||||
import threading
|
||||
import weakref
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
|
||||
@ -38,10 +39,16 @@ from ._messages import (
|
||||
SLHeader,
|
||||
StmtOptions,
|
||||
)
|
||||
from ._protocol import IfxStreamReader, IfxStreamWriter, ProtocolError, make_pdu_writer
|
||||
from ._protocol import (
|
||||
WIRE_ERRORS,
|
||||
IfxStreamReader,
|
||||
IfxStreamWriter,
|
||||
ProtocolError,
|
||||
make_pdu_writer,
|
||||
)
|
||||
from ._socket import IfxSocket
|
||||
from .cursors import Cursor
|
||||
from .exceptions import InterfaceError, OperationalError
|
||||
from .exceptions import InterfaceError, OperationalError, ProgrammingError
|
||||
|
||||
# Default capability bits the JDBC reference sends. Validated against
|
||||
# 01-connect-only.socat.log via the PDU diff in tests/test_pdu_match.py:
|
||||
@ -199,6 +206,62 @@ _DEFAULT_ENV: dict[str, str] = {
|
||||
}
|
||||
|
||||
|
||||
class _WireLock:
|
||||
"""A reentrant lock that can say whether *this* thread already holds it.
|
||||
|
||||
``threading.RLock`` cannot answer that question in public API, and
|
||||
the difference is not academic. The cursor finalizer runs at GC time
|
||||
on whatever thread happened to allocate — including a thread that is
|
||||
at that moment mid-statement holding this lock. It probes with
|
||||
``acquire(blocking=False)`` intending to mean "is anyone using the
|
||||
wire?", but an RLock grants a reentrant acquire to its own owner, so
|
||||
the probe returns True and the finalizer sends CLOSE/RELEASE into the
|
||||
middle of the statement it interrupted. The victim gets ``-208``.
|
||||
|
||||
Reachability is not exotic. Refcounting frees a dropped cursor
|
||||
immediately, before the next statement, which is why this went
|
||||
unnoticed — but a cursor caught in a reference cycle waits for a
|
||||
collection instead, and cycles are ordinary in Python. Any traceback
|
||||
that holds a cursor makes one.
|
||||
|
||||
``held_by_current_thread`` reads ``_depth``/``_owner`` without the
|
||||
lock, which is safe: both are mutated only under it, and the only
|
||||
values another thread can leave behind are a depth of zero or an
|
||||
owner that isn't us. Either way the answer is False, correctly.
|
||||
"""
|
||||
|
||||
__slots__ = ("_depth", "_lock", "_owner")
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._lock = threading.RLock()
|
||||
self._owner: int | None = None
|
||||
self._depth = 0
|
||||
|
||||
def acquire(self, blocking: bool = True, timeout: float = -1) -> bool:
|
||||
acquired = self._lock.acquire(blocking, timeout)
|
||||
if acquired:
|
||||
self._owner = threading.get_ident()
|
||||
self._depth += 1
|
||||
return acquired
|
||||
|
||||
def release(self) -> None:
|
||||
self._depth -= 1
|
||||
if self._depth == 0:
|
||||
self._owner = None
|
||||
self._lock.release()
|
||||
|
||||
def __enter__(self) -> _WireLock:
|
||||
self.acquire()
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc_info: object) -> None:
|
||||
self.release()
|
||||
|
||||
@property
|
||||
def held_by_current_thread(self) -> bool:
|
||||
return self._depth > 0 and self._owner == threading.get_ident()
|
||||
|
||||
|
||||
class Connection:
|
||||
"""A SQLI session. Owns one TCP socket and the post-login state.
|
||||
|
||||
@ -220,6 +283,7 @@ class Connection:
|
||||
client_locale: str = "en_US.8859-1",
|
||||
env: dict[str, str] | None = None,
|
||||
autocommit: bool = False, # honored from Phase 3 onward
|
||||
row_factory: object | None = None,
|
||||
tls: bool | ssl.SSLContext = False,
|
||||
tls_server_hostname: str | None = None,
|
||||
):
|
||||
@ -243,7 +307,7 @@ class Connection:
|
||||
# this lock with a timeout, then calls ``conn.rollback()`` —
|
||||
# which itself acquires the lock. Same thread, two acquires.
|
||||
# Reentrance must be cheap and correct.
|
||||
self._wire_lock = threading.RLock()
|
||||
self._wire_lock = _WireLock()
|
||||
# Phase 29: deferred-cleanup queue for cursor finalizers that
|
||||
# couldn't acquire the wire lock at GC time. Each entry is a
|
||||
# PDU's worth of bytes (typically a CLOSE or RELEASE) that
|
||||
@ -263,12 +327,21 @@ class Connection:
|
||||
# under ``_wire_lock``.
|
||||
self._pending_cleanup: list[bytes] = []
|
||||
self._cleanup_lock = threading.Lock()
|
||||
# Weak ref to the scrollable cursor currently holding the
|
||||
# session's statement slot, or None. Weak so that abandoning a
|
||||
# scrollable cursor still lets its finalizer run — a strong ref
|
||||
# here would keep the very object alive whose GC we depend on.
|
||||
self._open_scroll_cursor: weakref.ref | None = None
|
||||
# Logged-DB transaction state: True iff there's an open server-side
|
||||
# transaction (SQ_BEGIN sent, not yet committed/rolled-back). The
|
||||
# cursor uses this to decide whether to send an implicit SQ_BEGIN
|
||||
# before the next DML in non-autocommit mode. We default to "no
|
||||
# open txn" — the first DML will trigger SQ_BEGIN.
|
||||
self._in_transaction = False
|
||||
# Default row type for cursors from this connection. None means
|
||||
# plain tuples, which is the zero-cost default; see
|
||||
# informix_db.rows for the opt-in named-access type.
|
||||
self.row_factory = row_factory
|
||||
# Tri-state: True after first successful SQ_BEGIN, False after
|
||||
# an unlogged-DB rejection (-201). None until we've tried.
|
||||
# Used to avoid repeatedly probing on unlogged DBs.
|
||||
@ -357,22 +430,80 @@ class Connection:
|
||||
raise InterfaceError("connection is closed")
|
||||
return Cursor(self, scrollable=scrollable)
|
||||
|
||||
def _send_pdu(self, pdu: bytes) -> None:
|
||||
def _send_pdu(self, pdu: bytes, *, statement_boundary: bool = False) -> None:
|
||||
"""Send an assembled PDU. Used by Cursor.
|
||||
|
||||
Phase 29: opportunistically drains any pending cleanup PDUs
|
||||
from the deferred-cleanup queue *before* sending the new PDU.
|
||||
Caller must hold ``_wire_lock`` (every actual call site already
|
||||
does — execute/executemany/_sfetch_at, commit, rollback,
|
||||
fast_path_call, etc.). The drain happens under that lock so
|
||||
the queued cleanup atomically completes before the next op.
|
||||
Caller must hold ``_wire_lock`` (every actual call site does).
|
||||
|
||||
``statement_boundary=True`` additionally drains the deferred
|
||||
cleanup queue first. **Only pass it when this PDU is the first
|
||||
of a new statement**, meaning no statement is currently open
|
||||
server-side.
|
||||
|
||||
The queue previously drained before *every* PDU, which was wrong
|
||||
in a way that took a repro to see. ``SQ_CLOSE`` and
|
||||
``SQ_RELEASE`` carry no statement identifier — they act on the
|
||||
server's *current* statement. A finalizer enqueues precisely
|
||||
because it lost the race for the wire lock, which means another
|
||||
thread is mid-statement; that thread's next ``_send_pdu`` was
|
||||
then its own ``CURNAME``/``NFETCH``, and the drain released the
|
||||
statement out from under it. The caller saw a nonsense error on
|
||||
valid SQL: ``-208`` when injected before the first fetch,
|
||||
``-267`` "transaction has been rolled back" between fetch
|
||||
batches. Both point nowhere near the actual cause, and the
|
||||
window is exactly the window in which enqueueing happens.
|
||||
|
||||
Draining only at a boundary costs nothing: cleanup that misses
|
||||
one statement is picked up by the next.
|
||||
"""
|
||||
if self._closed:
|
||||
raise InterfaceError("connection is closed")
|
||||
if self._pending_cleanup:
|
||||
if statement_boundary and self._pending_cleanup:
|
||||
self._drain_pending_cleanup()
|
||||
self._sock.write_all(pdu)
|
||||
|
||||
def _check_scroll_cursor_conflict(self, requester: object) -> None:
|
||||
"""Refuse to start a statement while a scrollable cursor is open.
|
||||
|
||||
A server-side scrollable cursor occupies the session's statement
|
||||
slot, and SQLI gives us no way to address around it: ``SQ_CLOSE``,
|
||||
``SQ_RELEASE`` and ``SQ_SFETCH`` all act on the session's current
|
||||
statement.
|
||||
|
||||
The server does not refuse politely. Starting another statement
|
||||
returns ``-285``, and the scrollable cursor is collateral damage
|
||||
— its next fetch comes back ``-267`` "the transaction has been
|
||||
rolled back, all locks released". Two unattributable failures
|
||||
from code that looks entirely ordinary: iterate a large result
|
||||
set with a scrollable cursor, run a lookup query partway through.
|
||||
|
||||
Multiplexing is presumably expressible — JDBC prefixes every
|
||||
statement-scoped PDU with the statement id, which the server does
|
||||
assign distinctly (0 and 1 for two concurrent cursors). But the
|
||||
id alone is not sufficient: addressing the second cursor's
|
||||
``SQ_SFETCH`` to its own id returns ``-259`` "cursor not open".
|
||||
Until that is understood, refusing is the honest behaviour. It
|
||||
costs the caller a second connection and it never corrupts.
|
||||
"""
|
||||
ref = self._open_scroll_cursor
|
||||
if ref is None:
|
||||
return
|
||||
other = ref()
|
||||
if (
|
||||
other is None
|
||||
or other is requester
|
||||
or not getattr(other, "_server_cursor_open", False)
|
||||
):
|
||||
self._open_scroll_cursor = None
|
||||
return
|
||||
raise ProgrammingError(
|
||||
"a scrollable cursor is open on this connection; Informix "
|
||||
"allows only one statement per session, so running another "
|
||||
"statement here would fail with -285 and destroy the "
|
||||
"scrollable cursor as well. Close the scrollable cursor "
|
||||
"first, or use a separate connection for the other statement."
|
||||
)
|
||||
|
||||
def _enqueue_cleanup(self, pdus: list[bytes]) -> None:
|
||||
"""Append cleanup PDUs to the deferred queue.
|
||||
|
||||
@ -402,7 +533,6 @@ class Connection:
|
||||
discarded; the server-side resources they would have released
|
||||
are freed when the session ends anyway.
|
||||
"""
|
||||
from ._protocol import ProtocolError
|
||||
|
||||
with self._cleanup_lock:
|
||||
if not self._pending_cleanup:
|
||||
@ -413,11 +543,42 @@ class Connection:
|
||||
try:
|
||||
self._sock.write_all(pdu)
|
||||
self._drain_to_eot()
|
||||
except (ProtocolError, OSError, OperationalError):
|
||||
# Wire is unrecoverable; force-close. Subsequent
|
||||
# ``_send_pdu`` will raise InterfaceError. Server
|
||||
# cleanup of the remaining queued entries happens
|
||||
# implicitly at session end.
|
||||
except Exception as exc:
|
||||
if getattr(exc, "sqlcode", None) is not None:
|
||||
# The *server* rejected the cleanup — a stale entry
|
||||
# for a cursor it no longer has. Queued cleanup goes
|
||||
# stale routinely: the finalizer enqueues, then the
|
||||
# cursor gets closed properly before the drain runs.
|
||||
#
|
||||
# This is not a wire problem. ``_raise_sq_err``
|
||||
# self-drains the trailing SQ_EOT, so the wire is
|
||||
# still aligned and the remaining entries are still
|
||||
# worth sending.
|
||||
#
|
||||
# It must not escape, and it must not be treated as
|
||||
# fatal. Both were wrong before: a stale CLOSE draws
|
||||
# ``-267``, which is an OperationalError, which is in
|
||||
# WIRE_ERRORS — so a stale queue entry force-closed a
|
||||
# perfectly healthy connection. And this runs at the
|
||||
# start of somebody else's statement, so letting it
|
||||
# out would fail their good SQL with an error about a
|
||||
# cursor they never opened.
|
||||
_log.debug(
|
||||
"deferred cleanup rejected by server (stale entry), "
|
||||
"continuing: %r",
|
||||
exc,
|
||||
)
|
||||
continue
|
||||
if not isinstance(exc, WIRE_ERRORS):
|
||||
_log.warning(
|
||||
"unexpected error draining deferred cleanup: %r", exc
|
||||
)
|
||||
# No sqlcode means the failure came from the wire, not
|
||||
# the server: the socket died, or framing desynced and we
|
||||
# can no longer say where a response ends. Force-close.
|
||||
# Subsequent ``_send_pdu`` raises InterfaceError. The
|
||||
# server-side resources the remaining entries would have
|
||||
# freed are released when the session ends anyway.
|
||||
self._closed = True
|
||||
with contextlib.suppress(Exception):
|
||||
self._sock.close()
|
||||
@ -596,7 +757,7 @@ class Connection:
|
||||
# method but stable across versions; cheap (~50ns) and only
|
||||
# checks the current thread. If it ever changes shape, drop
|
||||
# this assert — the doc still names the precondition.
|
||||
assert self._wire_lock._is_owned(), (
|
||||
assert self._wire_lock.held_by_current_thread, (
|
||||
"_ensure_transaction called without _wire_lock held; "
|
||||
"the cursor method that called it must wrap its body in "
|
||||
"`with self._conn._wire_lock:`"
|
||||
@ -835,7 +996,6 @@ class Connection:
|
||||
[short near_token_len][bytes name][optional pad][short SQ_EOT]
|
||||
"""
|
||||
from . import _errcodes
|
||||
from ._protocol import ProtocolError
|
||||
|
||||
sqlcode = struct.unpack("!h", self._sock.read_exact(2))[0]
|
||||
isamcode = struct.unpack("!h", self._sock.read_exact(2))[0]
|
||||
@ -851,7 +1011,7 @@ class Connection:
|
||||
if name_len & 1:
|
||||
self._sock.read_exact(1)
|
||||
near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace")
|
||||
except (ProtocolError, OSError):
|
||||
except WIRE_ERRORS:
|
||||
pass
|
||||
# Phase 28: drain failure means wire desync — force-close so
|
||||
# subsequent operations don't inherit the broken state.
|
||||
@ -865,7 +1025,7 @@ class Connection:
|
||||
next_tag = struct.unpack("!h", self._sock.read_exact(2))[0]
|
||||
if next_tag == MessageType.SQ_EOT:
|
||||
break
|
||||
except (ProtocolError, OSError, OperationalError):
|
||||
except WIRE_ERRORS:
|
||||
self._closed = True
|
||||
with contextlib.suppress(Exception):
|
||||
self._sock.close()
|
||||
|
||||
@ -204,8 +204,25 @@ def _decode_float(raw: bytes) -> float | None:
|
||||
return _UNPACK_DOUBLE(raw)[0]
|
||||
|
||||
|
||||
def _decode_char(raw: bytes, encoding: str = "iso-8859-1") -> str:
|
||||
"""Strip trailing spaces (CHAR is space-padded to declared length)."""
|
||||
def _decode_char(raw: bytes, encoding: str = "iso-8859-1") -> str | None:
|
||||
"""Decode CHAR / NCHAR: fixed width, space-padded to the declared length.
|
||||
|
||||
A leading ``0x00`` is Informix's NULL marker for these types. Without
|
||||
that check a NULL CHAR came back as ``''``, indistinguishable from a
|
||||
genuinely empty one — and ``WHERE c IS NULL`` disagreeing with what
|
||||
the driver hands you is a nasty thing to debug. Wire evidence for
|
||||
``CHAR(6)``::
|
||||
|
||||
'ab' -> 61 62 20 20 20 20
|
||||
'' -> 20 20 20 20 20 20 all spaces
|
||||
NULL -> 00 20 20 20 20 20 leading nul
|
||||
|
||||
The two are distinguishable, so we distinguish them. A real value
|
||||
cannot begin with a nul: character data has no use for one, and the
|
||||
server reserves it precisely as this marker.
|
||||
"""
|
||||
if raw[:1] == b"\x00":
|
||||
return None
|
||||
return raw.rstrip(b" \x00").decode(encoding)
|
||||
|
||||
|
||||
@ -744,8 +761,28 @@ def _encode_float(value: float) -> EncodedParam:
|
||||
|
||||
|
||||
def _encode_bool(value: bool) -> EncodedParam:
|
||||
"""Encode a Python bool as Informix BOOLEAN (type=45, 1 byte)."""
|
||||
return (45, 0, b"\x01" if value else b"\x00")
|
||||
"""Encode a Python bool by binding the literal ``'t'`` / ``'f'``.
|
||||
|
||||
Informix BOOLEAN has no bindable binary form we could find. This used
|
||||
to send ``(45, 0, b"\\x01")``, and type 45 is not something the server
|
||||
accepts as a bind type — it simply stopped responding, so any
|
||||
``execute`` with a bool parameter **hung until the read timeout**, or
|
||||
forever if none was set. Type 41 (which is how BOOLEAN is *described*
|
||||
in results) hangs the same way; the descriptor type and the bind type
|
||||
are not interchangeable.
|
||||
|
||||
Sending ``'t'`` / ``'f'`` as CHAR and letting the server cast on
|
||||
assignment is what works, and it mirrors Informix's own literal
|
||||
syntax (``INSERT ... VALUES ('t')``). It also matches how
|
||||
:func:`_encode_str` already leans on server-side conversion rather
|
||||
than trying to match the destination column's exact type.
|
||||
|
||||
Note the asymmetry with the read path, which is correct and stays
|
||||
as it is: BOOLEAN comes *back* as UDTFIXED (41) wrapped in a UDT
|
||||
envelope whose payload byte is ``0x74``/``0x66`` — ASCII ``t``/``f``,
|
||||
the same characters we send here.
|
||||
"""
|
||||
return _encode_str("t" if value else "f")
|
||||
|
||||
|
||||
def _encode_date(value: datetime.date) -> EncodedParam:
|
||||
|
||||
@ -30,7 +30,12 @@ from typing import TYPE_CHECKING, Any
|
||||
|
||||
from . import _errcodes
|
||||
from ._messages import MessageType
|
||||
from ._protocol import BufferedSocketReader, IfxStreamReader, make_pdu_writer
|
||||
from ._protocol import (
|
||||
WIRE_ERRORS,
|
||||
BufferedSocketReader,
|
||||
IfxStreamReader,
|
||||
make_pdu_writer,
|
||||
)
|
||||
from ._resultset import (
|
||||
ColumnInfo,
|
||||
compile_column_readers,
|
||||
@ -45,6 +50,7 @@ from .exceptions import (
|
||||
NotSupportedError,
|
||||
ProgrammingError,
|
||||
)
|
||||
from .rows import Row, make_row_class
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .connections import Connection
|
||||
@ -55,7 +61,6 @@ if TYPE_CHECKING:
|
||||
_cursor_counter = itertools.count(1)
|
||||
|
||||
|
||||
_NUMERIC_PLACEHOLDER_RE = __import__("re").compile(r":(\d+)")
|
||||
|
||||
|
||||
# Phase 28: pre-built CLOSE and RELEASE PDU bytes for cursor finalizers.
|
||||
@ -97,6 +102,48 @@ def _make_socket_reader(sock):
|
||||
return _SocketReader(sock)
|
||||
|
||||
|
||||
# Statement types from the DESCRIBE response's first field. The server
|
||||
# tells us exactly what it prepared; these are the two values that mean
|
||||
# "this will produce rows".
|
||||
_ST_SELECT = 2
|
||||
_ST_ROUTINE = 56 # EXECUTE PROCEDURE / EXECUTE FUNCTION
|
||||
|
||||
# Transaction control run as SQL. JDBC reads the same three values off
|
||||
# the describe and calls setTxBeginState / setTxEndState (IfxSqli, the
|
||||
# TxStmt field). Both ``BEGIN`` and ``BEGIN WORK`` report 34; likewise
|
||||
# the WORK-less spellings of the other two.
|
||||
_ST_TX_BEGIN = 34
|
||||
_ST_TX_COMMIT = 35
|
||||
_ST_TX_ROLLBACK = 36
|
||||
_TX_CONTROL_TYPES = frozenset({_ST_TX_BEGIN, _ST_TX_COMMIT, _ST_TX_ROLLBACK})
|
||||
|
||||
|
||||
def _produces_result_set(statement_type: int, ncolumns: int) -> bool:
|
||||
"""Whether a prepared statement needs a cursor opened for it.
|
||||
|
||||
This is JDBC's ``IfxSqli.isResultSet`` predicate, and it replaces a
|
||||
first-word check for ``SELECT`` that got five ordinary forms wrong.
|
||||
A leading comment of any of the three Informix flavours, a
|
||||
parenthesized select, a parenthesized UNION, and a CTE were all
|
||||
classified as DML, so the driver sent SQ_EXECUTE where the server
|
||||
expected a cursor. The report back was ``-260 Cursor name already in
|
||||
use``, which describes neither the cause nor anything the caller did.
|
||||
|
||||
The original comment justified the heuristic on the grounds that
|
||||
``nfields`` can't distinguish these, because ``INSERT INTO t VALUES
|
||||
(?)`` also describes a column. True, and irrelevant: ``statement_type``
|
||||
distinguishes them exactly. That INSERT reports 6 with one field; every
|
||||
SELECT form above reports 2. The value was being parsed into the
|
||||
describe metadata and thrown away.
|
||||
|
||||
``EXECUTE PROCEDURE``/``FUNCTION`` (56) is the one type that depends on
|
||||
the column count, since a routine may or may not return rows.
|
||||
"""
|
||||
if statement_type == _ST_SELECT:
|
||||
return True
|
||||
return statement_type == _ST_ROUTINE and ncolumns > 0
|
||||
|
||||
|
||||
def _finalize_cursor(
|
||||
conn_ref: weakref.ReferenceType,
|
||||
state: list,
|
||||
@ -120,13 +167,30 @@ def _finalize_cursor(
|
||||
a list (not the cursor object itself) keeps the finalizer's closure
|
||||
weak — the cursor remains GC'd-able.
|
||||
"""
|
||||
from ._protocol import ProtocolError
|
||||
|
||||
if not state[0]:
|
||||
return # nothing to release
|
||||
conn = conn_ref()
|
||||
if conn is None or conn.closed:
|
||||
return
|
||||
if conn._wire_lock.held_by_current_thread:
|
||||
# GC fired on the very thread that is mid-statement. The
|
||||
# non-blocking acquire below would *succeed* here — an RLock
|
||||
# grants a reentrant acquire to its own owner — and we would
|
||||
# send CLOSE/RELEASE into the middle of that statement, killing
|
||||
# it with -208. Defer instead, exactly as for another thread.
|
||||
#
|
||||
# Refcounting hides this: a dropped cursor is freed at the drop,
|
||||
# before the next statement. A cursor caught in a reference cycle
|
||||
# waits for a collection instead, and cycles are ordinary — any
|
||||
# traceback holding a cursor makes one.
|
||||
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
|
||||
_log.debug(
|
||||
"cursor finalizer: GC ran on the thread holding the wire lock; "
|
||||
"enqueued CLOSE+RELEASE for deferred cleanup on conn %s",
|
||||
id(conn),
|
||||
)
|
||||
return
|
||||
if not conn._wire_lock.acquire(blocking=False):
|
||||
# Another thread is mid-operation on this connection. Don't
|
||||
# deadlock; instead, hand the cleanup bytes to the connection's
|
||||
@ -147,19 +211,33 @@ def _finalize_cursor(
|
||||
conn._drain_to_eot()
|
||||
conn._send_pdu(_RELEASE_PDU)
|
||||
conn._drain_to_eot()
|
||||
except (ProtocolError, OSError) as exc:
|
||||
# Wire desync during cleanup — same doctrine as
|
||||
# ``_raise_sq_err``: the wire is unrecoverable, force-close
|
||||
# the connection. Asymmetric handling of the same failure
|
||||
# mode would be a Hamilton smell.
|
||||
_log.warning(
|
||||
"cursor finalizer: wire desync during cleanup; "
|
||||
"force-closing connection: %r",
|
||||
exc,
|
||||
)
|
||||
conn._closed = True
|
||||
with contextlib.suppress(Exception):
|
||||
conn._sock.close()
|
||||
except WIRE_ERRORS as exc:
|
||||
if getattr(exc, "sqlcode", None) is not None:
|
||||
# The *server* rejected the cleanup — typically a stale
|
||||
# CLOSE for a cursor it no longer has, which answers
|
||||
# -267 "the transaction has been rolled back". That is
|
||||
# an OperationalError, which is in WIRE_ERRORS, so this
|
||||
# branch used to force-close a perfectly healthy
|
||||
# connection over a no-op. ``_raise_sq_err`` self-drains
|
||||
# the trailing SQ_EOT, so the wire is still aligned.
|
||||
# Same distinction as _drain_pending_cleanup.
|
||||
_log.debug(
|
||||
"cursor finalizer: server rejected cleanup (stale): %r",
|
||||
exc,
|
||||
)
|
||||
else:
|
||||
# No sqlcode: the socket died or framing desynced and we
|
||||
# can no longer say where a response ends. Force-close —
|
||||
# same doctrine as ``_raise_sq_err``. Asymmetric handling
|
||||
# of the same failure mode would be a Hamilton smell.
|
||||
_log.warning(
|
||||
"cursor finalizer: wire desync during cleanup; "
|
||||
"force-closing connection: %r",
|
||||
exc,
|
||||
)
|
||||
conn._closed = True
|
||||
with contextlib.suppress(Exception):
|
||||
conn._sock.close()
|
||||
except InterfaceError:
|
||||
# Connection was closed by another thread between our
|
||||
# ``conn.closed`` check above and the actual write. No-op:
|
||||
@ -169,17 +247,103 @@ def _finalize_cursor(
|
||||
conn._wire_lock.release()
|
||||
|
||||
|
||||
_ASCII_DIGITS = frozenset("0123456789")
|
||||
|
||||
|
||||
def _rewrite_numeric_to_qmark(sql: str) -> str:
|
||||
"""Convert ``:1`` / ``:2`` placeholders (paramstyle="numeric") to ``?``.
|
||||
|
||||
Informix's wire protocol uses ``?`` natively. Since we expose
|
||||
``paramstyle="numeric"`` in the public API (matches Informix
|
||||
ESQL/C convention), we rewrite before sending. Trivial cases only
|
||||
— strings and comments are NOT escaped, so SQL containing literal
|
||||
``:1`` inside string literals will be wrongly substituted. Phase 5
|
||||
can add a proper SQL tokenizer.
|
||||
Informix's wire protocol uses ``?`` natively, and we advertise
|
||||
``paramstyle="numeric"`` to match the ESQL/C convention, so the
|
||||
placeholders are rewritten on the way out.
|
||||
|
||||
This used to be ``re.sub(r":(\\d+)", "?", sql)``, which cannot see a
|
||||
string literal and so rewrote the contents of one. Any ``HH:MM``
|
||||
time, any URL with a port, any aspect ratio::
|
||||
|
||||
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
|
||||
stored as 'http://host?/x'
|
||||
|
||||
Nothing about that is opt-in. The rewrite runs whenever a statement
|
||||
has parameters, whatever placeholder style the caller actually used,
|
||||
so writing ``?`` everywhere and never touching numeric style did not
|
||||
protect you. It also changed the placeholder *count* while
|
||||
``num_qmarks`` was still computed from ``len(params)``, leaving the
|
||||
driver and the server disagreeing about how many binds exist.
|
||||
|
||||
So: a single pass that substitutes only outside quotes and comments.
|
||||
The lexical rules are Informix's own, measured against 12.10, 14.10
|
||||
and 15 rather than assumed from standard SQL:
|
||||
|
||||
* ``''`` doubling escapes a quote inside ``'...'``. A backslash does
|
||||
**not** escape anything; ``'a\'b'`` is an unterminated string and
|
||||
the server answers ``-282``. A Postgres-style scanner that honours
|
||||
``\'`` would desync here and corrupt everything after it.
|
||||
* ``"..."`` is a delimited identifier or a string depending on
|
||||
``DELIMIDENT``. Either way its contents are not ours to touch.
|
||||
* ``--`` runs to end of line, ``/* */`` does **not** nest (the first
|
||||
``*/`` closes it; nesting is a syntax error), and ``{ }`` is a
|
||||
comment too.
|
||||
* ``::`` is the cast operator and is stepped over as a unit, so it
|
||||
can never be read as the start of a placeholder.
|
||||
|
||||
An unterminated quote or comment consumes the rest of the string and
|
||||
substitutes nothing further. That is deliberate: under-substituting
|
||||
leaves the server to reject SQL that was already malformed, while
|
||||
guessing would corrupt a literal.
|
||||
"""
|
||||
return _NUMERIC_PLACEHOLDER_RE.sub("?", sql)
|
||||
if ":" not in sql:
|
||||
return sql
|
||||
out: list[str] = []
|
||||
i = 0
|
||||
n = len(sql)
|
||||
while i < n:
|
||||
ch = sql[i]
|
||||
if ch in ("'", '"'):
|
||||
j = i + 1
|
||||
while j < n:
|
||||
if sql[j] == ch:
|
||||
if j + 1 < n and sql[j + 1] == ch:
|
||||
j += 2 # doubled quote, still inside
|
||||
continue
|
||||
j += 1
|
||||
break
|
||||
j += 1
|
||||
out.append(sql[i:j])
|
||||
i = j
|
||||
elif ch == "-" and sql.startswith("--", i):
|
||||
j = sql.find("\n", i)
|
||||
j = n if j == -1 else j
|
||||
out.append(sql[i:j])
|
||||
i = j
|
||||
elif ch == "/" and sql.startswith("/*", i):
|
||||
j = sql.find("*/", i + 2)
|
||||
j = n if j == -1 else j + 2
|
||||
out.append(sql[i:j])
|
||||
i = j
|
||||
elif ch == "{":
|
||||
j = sql.find("}", i)
|
||||
j = n if j == -1 else j + 1
|
||||
out.append(sql[i:j])
|
||||
i = j
|
||||
elif ch == ":":
|
||||
if sql.startswith("::", i):
|
||||
out.append("::")
|
||||
i += 2
|
||||
continue
|
||||
j = i + 1
|
||||
while j < n and sql[j] in _ASCII_DIGITS:
|
||||
j += 1
|
||||
if j > i + 1:
|
||||
out.append("?")
|
||||
i = j
|
||||
else:
|
||||
out.append(ch)
|
||||
i += 1
|
||||
else:
|
||||
out.append(ch)
|
||||
i += 1
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def _generate_cursor_name() -> str:
|
||||
@ -210,6 +374,9 @@ class Cursor:
|
||||
# manipulation. Two-mode cursor; the same surface API works
|
||||
# for both.
|
||||
self._scrollable = scrollable
|
||||
# Inherited from the connection, overridable per cursor. See
|
||||
# informix_db.rows for what this costs and why it is opt-in.
|
||||
self.row_factory = connection.row_factory
|
||||
self._description: list[tuple] | None = None
|
||||
self._columns: list[ColumnInfo] = []
|
||||
self._column_readers: list[tuple] | None = None # Phase 37
|
||||
@ -247,6 +414,11 @@ class Cursor:
|
||||
# from. Empirically the server accepts 0 here even when a real
|
||||
# ID was assigned, so this is best-effort tracking.
|
||||
self._statement_id: int = 0
|
||||
# DESCRIBE's statement-type field. Decides whether a cursor is
|
||||
# opened -- see _produces_result_set.
|
||||
self._statement_type: int = 0
|
||||
# Per-result-set row class when row_factory is set, else None.
|
||||
self._row_class: type | None = None
|
||||
# Phase 10: smart-LOB read via ``lotofile(col, path, 'client')``.
|
||||
# The server orchestrates a SQ_FILE (98) protocol where it tells
|
||||
# us to "open file X, write these bytes, close". We emulate the
|
||||
@ -331,6 +503,15 @@ class Cursor:
|
||||
|
||||
def _execute_under_wire_lock(self, sql: str, params: tuple) -> None:
|
||||
"""Wire-bound body of ``execute``. Caller MUST hold ``_wire_lock``."""
|
||||
# Someone else's scrollable cursor owns the session's statement
|
||||
# slot -- refuse rather than draw a -285 and destroy their cursor.
|
||||
self._conn._check_scroll_cursor_conflict(self)
|
||||
# Our own previous scrollable cursor is still open server-side.
|
||||
# Re-executing over the top of it collides exactly the same way,
|
||||
# so close it first. This is the one caller allowed past the
|
||||
# check above, and it is only safe because of this.
|
||||
if self._scrollable and self._server_cursor_open:
|
||||
self._close_server_cursor()
|
||||
# Reset previous-execute state.
|
||||
self._description = None
|
||||
self._columns = []
|
||||
@ -339,27 +520,37 @@ class Cursor:
|
||||
self._rowcount = -1
|
||||
self._rows = []
|
||||
self._row_index = -1 # before-first-row
|
||||
self._row_class = None
|
||||
self._statement_type = 0
|
||||
self._statement_already_done = False
|
||||
|
||||
# Step 1: PREPARE — send SQL with numQmarks = len(params).
|
||||
# statement_boundary: nothing is open server-side yet, so this is
|
||||
# the one safe moment to flush a finalizer's deferred cleanup.
|
||||
self._conn._send_pdu(
|
||||
self._build_prepare_pdu(sql, num_qmarks=len(params)),
|
||||
statement_boundary=True,
|
||||
)
|
||||
self._read_describe_response()
|
||||
|
||||
# On a logged DB in non-autocommit mode, the server requires an
|
||||
# explicit SQ_BEGIN before the first DML in each transaction.
|
||||
# _ensure_transaction is a no-op for autocommit / unlogged DBs,
|
||||
# and idempotent within an open transaction.
|
||||
self._conn._ensure_transaction()
|
||||
#
|
||||
# This runs *after* the describe, matching JDBC's
|
||||
# initiateTransaction placement, because until the describe lands
|
||||
# we don't know whether the caller's statement is itself
|
||||
# transaction control. Opening a transaction on their behalf and
|
||||
# then executing their BEGIN WORK gets -535, "already in
|
||||
# transaction" — the driver competing with the user for the same
|
||||
# job and the user losing.
|
||||
if self._statement_type not in _TX_CONTROL_TYPES:
|
||||
self._conn._ensure_transaction()
|
||||
|
||||
# Step 1: PREPARE — send SQL with numQmarks = len(params).
|
||||
self._conn._send_pdu(self._build_prepare_pdu(sql, num_qmarks=len(params)))
|
||||
self._read_describe_response()
|
||||
|
||||
# Branch on the SQL keyword. We can't use ``self._columns`` /
|
||||
# ``nfields`` here because a parameterized INSERT also returns
|
||||
# a non-empty DESCRIBE (server describes the would-be inserted
|
||||
# row's columns). The SQL-keyword heuristic is what JDBC effectively
|
||||
# does too via its IfxStatement / IfxPreparedStatement subclassing.
|
||||
first_word = sql.lstrip().split(None, 1)[0].upper() if sql.strip() else ""
|
||||
is_select = first_word == "SELECT"
|
||||
|
||||
if is_select:
|
||||
# Ask the server what it just prepared, rather than guessing from
|
||||
# the first word of the SQL.
|
||||
if _produces_result_set(self._statement_type, len(self._columns)):
|
||||
if params:
|
||||
self._execute_select_with_params(params)
|
||||
else:
|
||||
@ -369,6 +560,13 @@ class Cursor:
|
||||
else:
|
||||
self._execute_dml()
|
||||
|
||||
self._row_class = self._resolve_row_class()
|
||||
|
||||
# The statement succeeded. If it was transaction control, the
|
||||
# server's transaction state just changed and the connection has
|
||||
# to know, or commit() and rollback() silently do nothing.
|
||||
self._note_transaction_control()
|
||||
|
||||
# SELECT path: position cursor before the first row so the next
|
||||
# ``fetchone()`` returns ``rows[0]``. DML paths leave _row_index
|
||||
# at -1 too (no rows to iterate).
|
||||
@ -377,6 +575,98 @@ class Cursor:
|
||||
if self._description is not None:
|
||||
self._row_index = -1
|
||||
|
||||
def _resolve_row_class(self) -> type | None:
|
||||
"""Pick the class this result set's rows are handed back as.
|
||||
|
||||
``row_factory`` is either :class:`informix_db.Row` (or a subclass),
|
||||
in which case the per-shape class is built and cached from the
|
||||
column names, or any callable taking the name tuple and returning
|
||||
something that takes a values tuple.
|
||||
|
||||
Returns ``None`` when no factory is set, which is the default and
|
||||
keeps plain tuples on the hot path at zero cost.
|
||||
"""
|
||||
factory = self.row_factory
|
||||
if factory is None or self._description is None:
|
||||
return None
|
||||
names = tuple(d[0] for d in self._description)
|
||||
if isinstance(factory, type) and issubclass(factory, Row):
|
||||
return make_row_class(names)
|
||||
return factory(names)
|
||||
|
||||
def _note_transaction_control(self) -> None:
|
||||
"""Sync the connection's transaction flag after a successful execute.
|
||||
|
||||
``BEGIN WORK`` run through ``execute()`` opens a real transaction
|
||||
on the server, and the connection had no idea. With autocommit on
|
||||
nothing stopped it reaching the server, so ``_in_transaction``
|
||||
stayed False while a transaction was open — and both ``commit()``
|
||||
and ``rollback()`` are guarded by that flag. ``rollback()``
|
||||
returned successfully having sent nothing, and the rows it was
|
||||
asked to discard were still there.
|
||||
|
||||
The pool reads the same flag to decide whether a returned
|
||||
connection needs cleaning up, so the connection went back into
|
||||
circulation holding an open transaction and its locks.
|
||||
|
||||
Called only on the success path: a statement that failed did not
|
||||
change the server's transaction state.
|
||||
"""
|
||||
statement_type = self._statement_type
|
||||
if statement_type == _ST_TX_BEGIN:
|
||||
self._conn._in_transaction = True
|
||||
elif statement_type in (_ST_TX_COMMIT, _ST_TX_ROLLBACK):
|
||||
self._conn._in_transaction = False
|
||||
|
||||
def _close_server_cursor(self) -> None:
|
||||
"""Free the server-side scrollable cursor. Caller MUST hold ``_wire_lock``.
|
||||
|
||||
Best-effort: a wire failure here is swallowed. Both callers are
|
||||
already past the point where reporting it would help — one is
|
||||
closing the cursor, the other is about to run a new statement
|
||||
that will report its own failure if the wire is really gone.
|
||||
"""
|
||||
try:
|
||||
self._conn._send_pdu(self._build_close_pdu())
|
||||
self._drain_to_eot()
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
pass
|
||||
self._server_cursor_open = False
|
||||
self._conn._open_scroll_cursor = None
|
||||
|
||||
def _release_after_failure(self, *, close_cursor: bool = False) -> None:
|
||||
"""Best-effort server-side cleanup after a statement fails.
|
||||
|
||||
A statement that failed is still allocated. Skipping the release
|
||||
bricks the connection: the next PREPARE collides with the leaked
|
||||
one and every subsequent call returns a nonsense error whose
|
||||
offset points back at the *failed* SQL. A duplicate-key violation
|
||||
is about the most ordinary error an application can hit, so "one
|
||||
constraint violation kills the connection" was easy to reach and
|
||||
hard to attribute.
|
||||
|
||||
CLOSE and RELEASE get **separate** suppressions. Putting both in
|
||||
one ``with contextlib.suppress(Exception)`` block reads as "clean
|
||||
up both", but a CLOSE that raises skips the RELEASE entirely —
|
||||
and RELEASE is the one that matters. Failing to close a cursor
|
||||
wastes a handle; failing to release the statement is what breaks
|
||||
the next call.
|
||||
|
||||
Everything here is swallowed rather than propagated. If the wire
|
||||
is genuinely desynced then the cleanup fails too, and the caller
|
||||
is far better served by the real SQL error than by a secondary
|
||||
failure from the cleanup path.
|
||||
"""
|
||||
if close_cursor:
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn._send_pdu(self._build_close_pdu())
|
||||
self._drain_to_eot()
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
|
||||
def _execute_select_with_params(self, params: tuple) -> None:
|
||||
"""Parameterized SELECT: SQ_BIND → CURNAME+NFETCH → drain → CLOSE+RELEASE.
|
||||
|
||||
@ -395,12 +685,20 @@ class Cursor:
|
||||
try:
|
||||
pdu = self._build_bind_only_pdu(params)
|
||||
except Exception:
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
self._release_after_failure()
|
||||
raise
|
||||
self._conn._send_pdu(pdu)
|
||||
self._drain_to_eot()
|
||||
try:
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
# The server can reject the BIND itself — a value whose type
|
||||
# doesn't match the described parameter, a bind against a
|
||||
# statement the server has since invalidated. This was the
|
||||
# last unguarded door of the six: the build was covered and
|
||||
# the drain was not, so a server-side bind rejection left the
|
||||
# statement allocated and the *next* execute() failed instead.
|
||||
self._release_after_failure()
|
||||
raise
|
||||
# Now open the cursor and fetch — the bound values are in scope
|
||||
# for the prepared statement.
|
||||
self._execute_select()
|
||||
@ -423,9 +721,23 @@ class Cursor:
|
||||
self._conn._send_pdu(
|
||||
self._build_curname_scroll_open_pdu(cursor_name)
|
||||
)
|
||||
self._drain_to_eot()
|
||||
try:
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
# Opening a scrollable cursor fails like any other
|
||||
# statement — a bad ORDER BY, a permission error, a table
|
||||
# dropped between PREPARE and OPEN. This branch had no
|
||||
# guard at all, and it is the worst place to lack one:
|
||||
# the GC-time finalizer is armed on the line *after* the
|
||||
# drain, so a failure here left the statement allocated
|
||||
# with no fallback whatsoever to reclaim it.
|
||||
self._release_after_failure(close_cursor=True)
|
||||
raise
|
||||
self._server_cursor_open = True
|
||||
self._finalizer_state[0] = True # arm the GC-time fallback
|
||||
# The connection needs to know its statement slot is taken,
|
||||
# so the next statement can refuse instead of drawing -285.
|
||||
self._conn._open_scroll_cursor = weakref.ref(self)
|
||||
self._scroll_total_rows = None
|
||||
return # don't close; cursor stays live for SQ_SFETCH
|
||||
# Phase 35: NFETCH loop — keep fetching until a response yields
|
||||
@ -435,20 +747,32 @@ class Cursor:
|
||||
# This bug was latent for ~30 phases because no test used a
|
||||
# large enough result set to trigger it.
|
||||
self._conn._send_pdu(self._build_curname_nfetch_pdu(cursor_name))
|
||||
rows_before = len(self._rows)
|
||||
self._read_fetch_response()
|
||||
rows_received = len(self._rows) - rows_before
|
||||
|
||||
while rows_received > 0:
|
||||
self._conn._send_pdu(self._build_nfetch_pdu())
|
||||
try:
|
||||
rows_before = len(self._rows)
|
||||
self._read_fetch_response()
|
||||
rows_received = len(self._rows) - rows_before
|
||||
|
||||
# Dereference BYTE/TEXT blob descriptors BEFORE CLOSE — the
|
||||
# locators are only valid while the cursor is open. No-op when
|
||||
# no BYTE/TEXT columns are present.
|
||||
self._dereference_blob_columns()
|
||||
while rows_received > 0:
|
||||
self._conn._send_pdu(self._build_nfetch_pdu())
|
||||
rows_before = len(self._rows)
|
||||
self._read_fetch_response()
|
||||
rows_received = len(self._rows) - rows_before
|
||||
|
||||
# Dereference BYTE/TEXT blob descriptors BEFORE CLOSE — the
|
||||
# locators are only valid while the cursor is open. No-op when
|
||||
# no BYTE/TEXT columns are present.
|
||||
self._dereference_blob_columns()
|
||||
except Exception:
|
||||
# A raise anywhere in the fetch loop leaves the cursor and the
|
||||
# statement allocated server-side. The GC-time finalizer only
|
||||
# covers scrollable cursors (it is armed above, in the branch
|
||||
# that returns early), so this path needs its own cleanup —
|
||||
# otherwise a mid-fetch failure leaks and the next statement
|
||||
# collides with it. Same failure mode as the DML path; see
|
||||
# _release_after_failure for what that looks like from the
|
||||
# caller's side.
|
||||
self._release_after_failure(close_cursor=True)
|
||||
raise
|
||||
|
||||
self._conn._send_pdu(self._build_close_pdu())
|
||||
self._drain_to_eot()
|
||||
@ -834,12 +1158,17 @@ class Cursor:
|
||||
try:
|
||||
pdu = self._build_bind_execute_pdu(params)
|
||||
except Exception:
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
self._release_after_failure()
|
||||
raise
|
||||
self._conn._send_pdu(pdu)
|
||||
self._drain_to_eot()
|
||||
try:
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
# The statement is still allocated server-side even though it
|
||||
# failed. See _execute_dml for why skipping this bricks the
|
||||
# connection.
|
||||
self._release_after_failure()
|
||||
raise
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
|
||||
@ -859,7 +1188,27 @@ class Cursor:
|
||||
let the optimization-looking response confuse you.
|
||||
"""
|
||||
self._conn._send_pdu(self._build_execute_pdu())
|
||||
self._drain_to_eot() # reads DONE + COST + EOT, populates rowcount
|
||||
try:
|
||||
self._drain_to_eot() # reads DONE + COST + EOT, populates rowcount
|
||||
except Exception:
|
||||
# A statement that FAILS is still allocated on the server, and
|
||||
# skipping the release here bricked the whole connection: the
|
||||
# next PREPARE collided with the leaked statement and every
|
||||
# subsequent call returned a nonsense error (-255 "Not in
|
||||
# transaction" in autocommit, -285 otherwise) whose offset
|
||||
# pointed back at the *failed* SQL, not the new statement.
|
||||
#
|
||||
# A duplicate-key violation is about the most ordinary error an
|
||||
# application can hit, so "one constraint violation kills the
|
||||
# connection" was easy to reach and hard to attribute. Send the
|
||||
# release, then re-raise the original error.
|
||||
#
|
||||
# Suppressed rather than propagated: if the wire is genuinely
|
||||
# desynced the release will fail too, and the caller is far
|
||||
# better served by the real SQL error than by a secondary
|
||||
# failure from the cleanup path.
|
||||
self._release_after_failure()
|
||||
raise
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
|
||||
@ -916,7 +1265,10 @@ class Cursor:
|
||||
f"expected {first_len} (matching set [0])"
|
||||
)
|
||||
|
||||
# Detect SELECT — not supported in executemany.
|
||||
# Cheap pre-flight reject for the obvious case, so the common
|
||||
# mistake costs no round-trip. The authoritative check is after
|
||||
# PREPARE, below — this one shares the first-word heuristic's
|
||||
# blind spots (leading comments, CTEs, parenthesized selects).
|
||||
first_word = operation.lstrip().split(None, 1)[0].upper() if operation.strip() else ""
|
||||
if first_word == "SELECT":
|
||||
raise NotSupportedError("executemany on SELECT is not supported")
|
||||
@ -927,6 +1279,9 @@ class Cursor:
|
||||
# under the wire lock — N rows commit atomically with respect
|
||||
# to other threads on the connection.
|
||||
with self._conn._wire_lock:
|
||||
self._conn._check_scroll_cursor_conflict(self)
|
||||
if self._scrollable and self._server_cursor_open:
|
||||
self._close_server_cursor()
|
||||
# Reset per-execute state.
|
||||
self._description = None
|
||||
self._columns = []
|
||||
@ -934,27 +1289,61 @@ class Cursor:
|
||||
self._rowcount = -1
|
||||
self._rows = []
|
||||
self._row_index = -1
|
||||
self._statement_type = 0
|
||||
self._statement_already_done = False
|
||||
|
||||
# Logged-DB transaction guard — same as execute(). Idempotent
|
||||
# within an open transaction.
|
||||
self._conn._ensure_transaction()
|
||||
|
||||
# PREPARE once.
|
||||
self._conn._send_pdu(
|
||||
self._build_prepare_pdu(sql, num_qmarks=first_len)
|
||||
self._build_prepare_pdu(sql, num_qmarks=first_len),
|
||||
statement_boundary=True,
|
||||
)
|
||||
self._read_describe_response()
|
||||
|
||||
# Now the server has told us what it prepared. A result-set
|
||||
# statement that slipped past the first-word check above --
|
||||
# a CTE, a leading comment, a parenthesized select -- would
|
||||
# otherwise be executed N times down the DML path, which
|
||||
# opens no cursor and answers -260.
|
||||
if _produces_result_set(self._statement_type, len(self._columns)):
|
||||
self._release_after_failure()
|
||||
raise NotSupportedError(
|
||||
"executemany on a statement that returns rows is not "
|
||||
"supported"
|
||||
)
|
||||
|
||||
# Logged-DB transaction guard — same as execute(), and for the
|
||||
# same reason placed after the describe rather than before it.
|
||||
if self._statement_type not in _TX_CONTROL_TYPES:
|
||||
self._conn._ensure_transaction()
|
||||
|
||||
# Phase 33: pipeline — build all BIND+EXECUTE PDUs first
|
||||
# (Python work, no I/O), then send them back-to-back, then
|
||||
# drain all responses. Eliminates the per-row round-trip
|
||||
# the older serial loop paid.
|
||||
pdus = [
|
||||
self._build_bind_execute_pdu(tuple(p)) for p in seq
|
||||
]
|
||||
for pdu in pdus:
|
||||
self._conn._send_pdu(pdu)
|
||||
try:
|
||||
pdus = [
|
||||
self._build_bind_execute_pdu(tuple(p)) for p in seq
|
||||
]
|
||||
for pdu in pdus:
|
||||
self._conn._send_pdu(pdu)
|
||||
except Exception:
|
||||
# Encoding a row can fail client-side (DataError for a
|
||||
# value the connection's codec can't represent), and it
|
||||
# fails here — after the PREPARE, before anything is
|
||||
# drained. Escaping without the RELEASE leaks the
|
||||
# statement server-side, and the next PREPARE then
|
||||
# collides with it: every later call on this connection
|
||||
# returns a nonsense error pointing at the *previous*
|
||||
# SQL. ``_execute_dml_with_params`` guards the identical
|
||||
# case for the single-row path; the pipelined path was
|
||||
# simply missed.
|
||||
#
|
||||
# A send failure lands here too. The wire is likely
|
||||
# already unusable in that case, but attempting the
|
||||
# release costs nothing and the original error is what
|
||||
# propagates either way.
|
||||
self._release_after_failure()
|
||||
raise
|
||||
|
||||
# Drain N responses. The first error is captured but we
|
||||
# still drain the rest (they're SQ_ERRs for the aborted
|
||||
@ -1026,7 +1415,8 @@ class Cursor:
|
||||
self._row_index = len(self._rows) # past-last
|
||||
return None
|
||||
self._row_index = nxt
|
||||
return self._rows[nxt]
|
||||
row = self._rows[nxt]
|
||||
return self._row_class(row) if self._row_class is not None else row
|
||||
|
||||
def fetchmany(self, size: int | None = None) -> list[tuple]:
|
||||
self._check_open()
|
||||
@ -1057,6 +1447,8 @@ class Cursor:
|
||||
return []
|
||||
start = self._row_index + 1
|
||||
out = self._rows[start:]
|
||||
if self._row_class is not None:
|
||||
out = [self._row_class(r) for r in out]
|
||||
self._row_index = len(self._rows)
|
||||
return list(out)
|
||||
|
||||
@ -1237,7 +1629,7 @@ class Cursor:
|
||||
if scrolltype == 4 or is_last_probe:
|
||||
# SFETCH(LAST) — TUPID == total row count
|
||||
self._scroll_total_rows = self._last_tupid
|
||||
return row
|
||||
return self._row_class(row) if self._row_class is not None else row
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the cursor.
|
||||
@ -1249,20 +1641,8 @@ class Cursor:
|
||||
if self._closed:
|
||||
return
|
||||
if self._scrollable and self._server_cursor_open:
|
||||
# Phase 27: hold the wire lock during CLOSE+RELEASE so we
|
||||
# don't interleave with another thread's pending op on the
|
||||
# connection. Best-effort: any wire failure here is
|
||||
# swallowed (the caller is closing; we don't want to mask
|
||||
# whatever caused them to close).
|
||||
try:
|
||||
with self._conn._wire_lock:
|
||||
self._conn._send_pdu(self._build_close_pdu())
|
||||
self._drain_to_eot()
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
pass
|
||||
self._server_cursor_open = False
|
||||
with self._conn._wire_lock:
|
||||
self._close_server_cursor()
|
||||
# Phase 28: explicit close has handled the server-side resources
|
||||
# (or tried to). Disarm the finalizer so it doesn't fire later
|
||||
# for nothing — and clear the state flag as a belt-and-suspenders
|
||||
@ -1566,6 +1946,7 @@ class Cursor:
|
||||
elif tag == MessageType.SQ_DESCRIBE:
|
||||
self._columns, meta = parse_describe(reader)
|
||||
self._statement_id = meta.get("statement_id", 0)
|
||||
self._statement_type = meta.get("statement_type", 0)
|
||||
self._description = (
|
||||
[c.to_description_tuple() for c in self._columns] if self._columns else None
|
||||
)
|
||||
@ -1697,7 +2078,6 @@ class Cursor:
|
||||
(e.g. table or column name for "not found" errors). Empty for
|
||||
most syntax errors.
|
||||
"""
|
||||
from ._protocol import ProtocolError
|
||||
|
||||
sqlcode = reader.read_short()
|
||||
isamcode = reader.read_short()
|
||||
@ -1715,7 +2095,7 @@ class Cursor:
|
||||
if name_len & 1:
|
||||
reader.read_exact(1) # pad to even
|
||||
near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace")
|
||||
except (ProtocolError, OSError):
|
||||
except WIRE_ERRORS:
|
||||
pass
|
||||
# Drain remaining bytes until SQ_EOT. Phase 28: a (ProtocolError,
|
||||
# OSError) during drain means the wire is in an unknown state —
|
||||
@ -1727,7 +2107,7 @@ class Cursor:
|
||||
t = reader.read_short()
|
||||
if t == MessageType.SQ_EOT:
|
||||
break
|
||||
except (ProtocolError, OSError):
|
||||
except WIRE_ERRORS:
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn.close()
|
||||
|
||||
|
||||
180
src/informix_db/rows.py
Normal file
180
src/informix_db/rows.py
Normal file
@ -0,0 +1,180 @@
|
||||
"""Rows that can be read by position, by column name, or by attribute.
|
||||
|
||||
PEP 249 only requires a sequence, and a sequence is what the driver
|
||||
returns by default. That is fine for ``SELECT a, b`` and steadily worse
|
||||
as the projection grows: ``row[11]`` tells a reader nothing, and stays
|
||||
correct only until somebody adds a column in the middle.
|
||||
|
||||
Opting in with ``row_factory=Row`` gives the shape ``pyodbc`` and
|
||||
``mssql-python`` provide, all three at once::
|
||||
|
||||
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||
cur.execute("SELECT tabid, tabname FROM systables")
|
||||
row = cur.fetchone()
|
||||
row[0], row["tabname"], row.tabname
|
||||
|
||||
It is opt-in rather than the default because it is not free, and the
|
||||
driver's whole argument is that pure Python can stay within noise of the
|
||||
C driver on bulk fetch. Measured on a 20,000-row five-column fetch:
|
||||
**37.2 ms with tuples, 40.8 ms with Row**, so about 9%. Defaulting it on
|
||||
would move the published 1.05-1.15x ratio against IfxPy to roughly
|
||||
1.15-1.25x, which is not a trade to make on everybody's behalf.
|
||||
|
||||
Where the 9% goes, against a plain tuple: about 30 ns per row to
|
||||
construct, about 39 ns per ``row[0]`` because supporting ``row["name"]``
|
||||
means ``__getitem__`` is a Python method rather than C-level tuple
|
||||
indexing, and about 13 ns per unpack since CPython's fast path for
|
||||
``a, b = row`` applies to exact tuples and not to subclasses.
|
||||
|
||||
Worth it for readability in application code. Not worth paying in a bulk
|
||||
export that never looks at a column by name, which is exactly why it is
|
||||
a choice.
|
||||
|
||||
``Row`` subclasses ``tuple``, so ``row == (1, "x")`` is still true and
|
||||
existing code keeps working unchanged.
|
||||
|
||||
**A column always beats a method of the same name.** ``tuple`` defines
|
||||
``count`` and ``index``; this class adds ``keys``, ``_asdict`` and
|
||||
``_fields``. A column named any of those would otherwise resolve to the
|
||||
method and hand back a bound method instead of a value, which is exactly
|
||||
the shape of the framing bugs this driver spent a fortnight removing: a
|
||||
plausible-looking wrong answer, in silence. So every such name gets a
|
||||
descriptor and the column wins, and the reserved set is *computed* from
|
||||
the class rather than hand-listed, because hand-listing it is what
|
||||
missed ``keys``, ``_asdict`` and ``_fields`` the first time round. The
|
||||
methods stay reachable through the class: ``tuple.count(row, x)``,
|
||||
``Row.keys(row)``.
|
||||
|
||||
The machinery itself is name-mangled (``__fields`` / ``__map``) so that
|
||||
shadowing ``_fields`` cannot break ``repr`` or ``_asdict``.
|
||||
|
||||
Column names come from ``cursor.description``. Informix folds unquoted
|
||||
identifiers to lower case, so ``SELECT Config_Key`` is reachable as
|
||||
``row.config_key``. Expressions get server-generated names that are not
|
||||
Python identifiers, like ``(count(*))``; those are reachable by
|
||||
subscript but not as attributes. Duplicate names resolve to the first
|
||||
occurrence, matching ``pyodbc``. Dunder names cannot be shadowed and
|
||||
stay subscript-only, which no real schema should notice.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import operator
|
||||
from functools import lru_cache
|
||||
from typing import ClassVar
|
||||
|
||||
__all__ = ["Row", "make_row_class"]
|
||||
|
||||
|
||||
class Row(tuple):
|
||||
"""A result row addressable by position, name, or attribute.
|
||||
|
||||
Used as a ``row_factory``. The concrete class handed to each result
|
||||
set is a subclass carrying that query's column names, built by
|
||||
:func:`make_row_class`.
|
||||
"""
|
||||
|
||||
__slots__ = ()
|
||||
|
||||
# Name-mangled to ``_Row__fields`` / ``_Row__map`` so that a column
|
||||
# called "_fields" can be shadowed without breaking the machinery
|
||||
# that reads it. Set on the per-result-set subclass.
|
||||
__fields: ClassVar[tuple[str, ...]] = ()
|
||||
__map: ClassVar[dict[str, int]] = {}
|
||||
|
||||
def __getitem__(self, key):
|
||||
# ``key.__class__ is str`` rather than isinstance: this runs on
|
||||
# every subscript.
|
||||
if key.__class__ is str:
|
||||
try:
|
||||
return tuple.__getitem__(self, self.__map[key])
|
||||
except KeyError:
|
||||
raise KeyError(
|
||||
f"no column named {key!r}; this row has "
|
||||
f"{list(self.__fields)}"
|
||||
) from None
|
||||
return tuple.__getitem__(self, key)
|
||||
|
||||
def __getattr__(self, name):
|
||||
# Only reached when normal attribute lookup has already failed,
|
||||
# so this costs nothing for names that do not collide.
|
||||
try:
|
||||
return tuple.__getitem__(self, self.__map[name])
|
||||
except KeyError:
|
||||
raise AttributeError(
|
||||
f"no column named {name!r}; this row has "
|
||||
f"{list(self.__fields)}"
|
||||
) from None
|
||||
|
||||
@property
|
||||
def _fields(self) -> tuple[str, ...]:
|
||||
"""Column names, in select order. Mirrors ``namedtuple._fields``."""
|
||||
return self.__fields
|
||||
|
||||
def keys(self) -> tuple[str, ...]:
|
||||
"""Column names, in select order."""
|
||||
return self.__fields
|
||||
|
||||
def _asdict(self) -> dict:
|
||||
"""A plain ``dict`` of the row.
|
||||
|
||||
On duplicate column names the last occurrence wins here, while
|
||||
subscript access gives the first. A dict cannot represent both,
|
||||
and quietly dropping a duplicate is better than raising on a
|
||||
query that is otherwise fine.
|
||||
"""
|
||||
return dict(zip(self.__fields, self, strict=True))
|
||||
|
||||
def __repr__(self) -> str:
|
||||
fields = self.__fields
|
||||
if len(fields) != len(self):
|
||||
return tuple.__repr__(self)
|
||||
body = ", ".join(
|
||||
f"{name}={value!r}"
|
||||
for name, value in zip(fields, self, strict=True)
|
||||
)
|
||||
return f"Row({body})"
|
||||
|
||||
def __reduce__(self):
|
||||
# The per-result-set class is created at runtime and cannot be
|
||||
# pickled by reference, so rebuild it from the field names.
|
||||
return (_rebuild_row, (self.__fields, tuple(self)))
|
||||
|
||||
|
||||
# Every non-dunder attribute a Row already answers to. A column with one
|
||||
# of these names gets a descriptor so the column wins. Computed, not
|
||||
# hand-listed: the hand-listed version covered ``count`` and ``index``
|
||||
# and silently missed ``keys``, ``_asdict`` and ``_fields``.
|
||||
_RESERVED = frozenset(
|
||||
name for name in dir(Row) if not name.startswith("__")
|
||||
) - {"_Row__fields", "_Row__map"}
|
||||
|
||||
|
||||
def _rebuild_row(fields: tuple[str, ...], values: tuple):
|
||||
return make_row_class(fields)(values)
|
||||
|
||||
|
||||
@lru_cache(maxsize=256)
|
||||
def make_row_class(fields: tuple[str, ...]) -> type[Row]:
|
||||
"""Build (and cache) the row class for one column-name shape.
|
||||
|
||||
Cached because a class per ``execute()`` would put a ``type()`` call
|
||||
on the path of every small query, and applications run the same
|
||||
handful of statement shapes over and over. Keyed on the names alone,
|
||||
so two queries selecting the same columns share a class.
|
||||
"""
|
||||
namespace: dict = {
|
||||
"__slots__": (),
|
||||
"_Row__fields": fields,
|
||||
# First occurrence wins on duplicates, matching pyodbc. Building
|
||||
# the map in reverse and letting earlier entries overwrite later
|
||||
# ones is the shortest way to say that.
|
||||
"_Row__map": {
|
||||
name: i for i, name in reversed(list(enumerate(fields)))
|
||||
},
|
||||
}
|
||||
for i, name in enumerate(fields):
|
||||
if name in _RESERVED:
|
||||
# A column of this name would otherwise resolve to a method.
|
||||
namespace[name] = property(operator.itemgetter(i))
|
||||
return type("Row", (Row,), namespace)
|
||||
265
tests/test_async_threads.py
Normal file
265
tests/test_async_threads.py
Normal file
@ -0,0 +1,265 @@
|
||||
"""The async layer used a thread pool it shared with the whole process.
|
||||
|
||||
``asyncio.to_thread`` runs on the event loop's default executor. That
|
||||
executor belongs to the process, not to us, and it is sized from the CPU
|
||||
count — ``min(32, cpu_count + 4)``, which is six threads on a two-CPU
|
||||
container. Every blocking call in ``informix_db.aio`` went through it.
|
||||
|
||||
Two consequences, both silent.
|
||||
|
||||
**Cancelled calls hold their threads.** ``asyncio.to_thread`` cannot
|
||||
interrupt a worker, so a cancelled await leaves the thread running the
|
||||
wire call until the read timeout expires. Cancellation is ordinary in a
|
||||
web app — a client disconnect cancels the request task — so a handful of
|
||||
them pins every thread in the shared pool. Unrelated ``to_thread`` work
|
||||
anywhere else in the process then stops dead, and so does the driver.
|
||||
|
||||
**Pool concurrency was capped by an unrelated number.** A pool with
|
||||
``max_size=20`` on a two-CPU box ran six queries at a time, and nothing
|
||||
said so.
|
||||
|
||||
Each connection now owns one thread. That is the right size rather than
|
||||
a compromise: the sync connection serializes every wire operation on its
|
||||
own lock, so a second thread could do nothing but wait for the first. It
|
||||
also avoids a deadlock that a shared pool-sized executor invites — with
|
||||
N threads and N connections, N tasks blocked in ``acquire`` occupy every
|
||||
thread while the connection they wait for is held by a task that now
|
||||
needs a thread to finish and release it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
import pytest
|
||||
|
||||
from informix_db import aio
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _kw(conn_params: ConnParams) -> dict:
|
||||
return {
|
||||
"host": conn_params.host,
|
||||
"port": conn_params.port,
|
||||
"user": conn_params.user,
|
||||
"password": conn_params.password,
|
||||
"database": conn_params.database,
|
||||
"server": conn_params.server,
|
||||
"connect_timeout": 10.0,
|
||||
"read_timeout": 20.0,
|
||||
"autocommit": True,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_queries_run_with_the_default_executor_fully_held(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The starvation case, with real queries. Four unrelated jobs occupy
|
||||
every thread of a four-worker default executor; the driver must not
|
||||
be waiting behind them."""
|
||||
loop = asyncio.get_running_loop()
|
||||
previous = loop._default_executor
|
||||
loop.set_default_executor(ThreadPoolExecutor(max_workers=4))
|
||||
conns = [await aio.connect(**_kw(conn_params)) for _ in range(4)]
|
||||
cursors = [await c.cursor() for c in conns]
|
||||
|
||||
gate = threading.Event()
|
||||
hogs = [
|
||||
asyncio.create_task(asyncio.to_thread(gate.wait, 30)) for _ in range(4)
|
||||
]
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
async def query(cursor, i: int):
|
||||
await cursor.execute(f"SELECT FIRST 1 tabid + {i} FROM systables")
|
||||
return await cursor.fetchone()
|
||||
|
||||
try:
|
||||
rows = await asyncio.wait_for(
|
||||
asyncio.gather(*(query(c, i) for i, c in enumerate(cursors))),
|
||||
timeout=10.0,
|
||||
)
|
||||
assert len(rows) == 4
|
||||
assert all(r is not None for r in rows)
|
||||
finally:
|
||||
gate.set()
|
||||
await asyncio.gather(*hogs, return_exceptions=True)
|
||||
for c in conns:
|
||||
await c.close()
|
||||
# set_default_executor rejects None, which is what the loop
|
||||
# starts with before anything has used to_thread.
|
||||
loop.set_default_executor(previous or ThreadPoolExecutor())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_each_connection_gets_its_own_thread(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
a = await aio.connect(**_kw(conn_params))
|
||||
b = await aio.connect(**_kw(conn_params))
|
||||
try:
|
||||
assert a._executor is not b._executor
|
||||
assert a._executor._max_workers == 1, (
|
||||
"more than one thread per connection cannot help — the wire "
|
||||
"lock serializes them anyway"
|
||||
)
|
||||
finally:
|
||||
await a.close()
|
||||
await b.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cursor_runs_on_its_connections_thread(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""A cursor must not fall back to the default executor, or half the
|
||||
work goes back to being shared."""
|
||||
conn = await aio.connect(**_kw(conn_params))
|
||||
try:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
names: list[str] = []
|
||||
await conn._run(lambda: names.append(threading.current_thread().name))
|
||||
assert names[0].startswith("informix-conn")
|
||||
assert cur._run.__self__ is conn
|
||||
finally:
|
||||
await conn.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pool_reuses_one_thread_per_connection(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The executor lives on the sync connection, so a connection handed
|
||||
out, returned, and handed out again keeps the same thread instead of
|
||||
spawning one per acquire."""
|
||||
pool = await aio.create_pool(**_kw(conn_params), min_size=1, max_size=2)
|
||||
try:
|
||||
seen = []
|
||||
for _ in range(4):
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert await cur.fetchone() is not None
|
||||
seen.append(id(conn._executor))
|
||||
assert len(set(seen)) <= 2, (
|
||||
f"expected at most one executor per pooled connection, saw "
|
||||
f"{len(set(seen))}"
|
||||
)
|
||||
finally:
|
||||
await pool.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pool_concurrency_is_not_capped_by_the_default_executor(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Six concurrent pooled queries against a two-worker default
|
||||
executor. Under the old arrangement at most two could run."""
|
||||
loop = asyncio.get_running_loop()
|
||||
previous = loop._default_executor
|
||||
pool = await aio.create_pool(**_kw(conn_params), min_size=6, max_size=6)
|
||||
loop.set_default_executor(ThreadPoolExecutor(max_workers=2))
|
||||
try:
|
||||
|
||||
async def one(i: int):
|
||||
async with pool.connection(timeout=15.0) as conn:
|
||||
cur = await conn.cursor()
|
||||
# Long enough that serialized execution would be obvious.
|
||||
await cur.execute(
|
||||
"SELECT FIRST 200 a.tabid FROM systables a, systables b"
|
||||
)
|
||||
return len(await cur.fetchall())
|
||||
|
||||
started = time.monotonic()
|
||||
results = await asyncio.wait_for(
|
||||
asyncio.gather(*(one(i) for i in range(6))), timeout=25.0
|
||||
)
|
||||
assert all(r > 0 for r in results)
|
||||
assert len(results) == 6
|
||||
elapsed = time.monotonic() - started
|
||||
assert elapsed < 20.0, f"queries appear serialized ({elapsed:.1f}s)"
|
||||
finally:
|
||||
await pool.close()
|
||||
# set_default_executor rejects None, which is what the loop
|
||||
# starts with before anything has used to_thread.
|
||||
loop.set_default_executor(previous or ThreadPoolExecutor())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_closing_a_connection_stops_its_thread(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""One thread per connection is only affordable if the thread goes
|
||||
away with the connection."""
|
||||
before = threading.active_count()
|
||||
conns = [await aio.connect(**_kw(conn_params)) for _ in range(5)]
|
||||
for c in conns:
|
||||
cur = await c.cursor()
|
||||
await cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
for c in conns:
|
||||
await c.close()
|
||||
del conns
|
||||
# Threads exit asynchronously after shutdown; give them a moment.
|
||||
for _ in range(50):
|
||||
if threading.active_count() <= before + 1:
|
||||
break
|
||||
await asyncio.sleep(0.1)
|
||||
assert threading.active_count() <= before + 1, (
|
||||
f"connection threads outlived their connections "
|
||||
f"({before} -> {threading.active_count()})"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_row_factory_reaches_the_async_paths(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The async layer wraps the sync cursor, so row_factory should flow
|
||||
through untouched. Each of the four async fetch routes is a separate
|
||||
call site, and async iteration goes through __anext__ rather than
|
||||
__next__."""
|
||||
import informix_db
|
||||
|
||||
conn = await aio.connect(row_factory=informix_db.Row, **_kw(conn_params))
|
||||
try:
|
||||
cur = await conn.cursor()
|
||||
sql = "SELECT FIRST 2 tabid, tabname FROM systables ORDER BY tabid"
|
||||
|
||||
await cur.execute(sql)
|
||||
one = await cur.fetchone()
|
||||
assert one[0] == one["tabid"] == one.tabid
|
||||
|
||||
await cur.execute(sql)
|
||||
assert [r.tabid for r in await cur.fetchall()] == [1, 2]
|
||||
|
||||
await cur.execute(sql)
|
||||
assert all(r.tabname for r in await cur.fetchmany(2))
|
||||
|
||||
await cur.execute(sql)
|
||||
assert [r.tabid async for r in cur] == [1, 2]
|
||||
finally:
|
||||
await conn.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_pool_forwards_the_row_factory(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
import informix_db
|
||||
|
||||
pool = await aio.create_pool(
|
||||
row_factory=informix_db.Row, min_size=1, max_size=2, **_kw(conn_params)
|
||||
)
|
||||
try:
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert (await cur.fetchone()).tabid == 1
|
||||
finally:
|
||||
await pool.close()
|
||||
258
tests/test_batch_scroll_lob.py
Normal file
258
tests/test_batch_scroll_lob.py
Normal file
@ -0,0 +1,258 @@
|
||||
"""Regression tests for pipelined executemany, scrollable cursors, and LOBs.
|
||||
|
||||
Found here by fuzzing: **a client-side encoding failure inside an
|
||||
``executemany`` batch leaked the prepared statement.** The PDUs are built
|
||||
after the PREPARE, so a value the connection's codec cannot represent
|
||||
raises there — before anything is drained — and the exception escaped
|
||||
without the RELEASE. The leaked statement then collided with the next
|
||||
PREPARE and every later call on that connection failed with an error
|
||||
pointing at the *previous* SQL.
|
||||
|
||||
``_execute_dml_with_params`` already guarded exactly this case for the
|
||||
single-row path. The pipelined path was simply missed, which is the
|
||||
recurring shape of these bugs: a hazard understood in one place and not
|
||||
carried to its sibling.
|
||||
|
||||
The rest of this file is the coverage that proved the neighbouring paths
|
||||
sound — batch failures at every position, scroll boundaries, LOB sizes —
|
||||
kept so they stay that way. Each failure case ends in a health check,
|
||||
because the interesting damage is never in the statement that failed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=15.0,
|
||||
read_timeout=30.0,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
def _assert_healthy(cur) -> None:
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None, "connection unusable"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# executemany — the encoding-failure leak, and the neighbours
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [3, 50])
|
||||
@pytest.mark.parametrize("position", ["first", "mid", "last"])
|
||||
def test_encoding_failure_in_batch_does_not_leak(
|
||||
conn_params: ConnParams, n: int, position: str
|
||||
) -> None:
|
||||
"""The bug. A value the codec can't encode raises after the PREPARE;
|
||||
escaping without the RELEASE bricked the connection."""
|
||||
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_em_enc (s VARCHAR(32))")
|
||||
rows: list[tuple] = [(f"s{i}",) for i in range(n)]
|
||||
rows[idx] = ("中文",) # not representable in iso-8859-1
|
||||
|
||||
with pytest.raises(Exception): # noqa: B017 — DataError or UnicodeError
|
||||
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
|
||||
|
||||
# The connection must survive, repeatedly.
|
||||
for _ in range(3):
|
||||
_assert_healthy(cur)
|
||||
with pytest.raises(Exception): # noqa: B017
|
||||
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
|
||||
_assert_healthy(cur)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [2, 3, 10, 100])
|
||||
@pytest.mark.parametrize("position", ["first", "mid", "last"])
|
||||
def test_constraint_violation_in_batch_recovers(
|
||||
conn_params: ConnParams, n: int, position: str
|
||||
) -> None:
|
||||
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_em_dup (k INT PRIMARY KEY)")
|
||||
cur.execute("INSERT INTO t_em_dup VALUES (?)", (10_000,))
|
||||
rows = [(i,) for i in range(n)]
|
||||
rows[idx] = (10_000,)
|
||||
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.executemany("INSERT INTO t_em_dup VALUES (?)", rows)
|
||||
_assert_healthy(cur)
|
||||
|
||||
# Whatever the partial-batch semantics, COUNT(*) and a full fetch
|
||||
# must agree — a disagreement means the row decoder and the server
|
||||
# have different ideas about what is in the table.
|
||||
cur.execute("SELECT COUNT(*) FROM t_em_dup")
|
||||
(counted,) = cur.fetchone()
|
||||
cur.execute("SELECT k FROM t_em_dup")
|
||||
assert counted == len(cur.fetchall())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [1, 2, 3, 10, 100, 1000])
|
||||
def test_executemany_inserts_exactly_n_rows(
|
||||
conn_params: ConnParams, n: int
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_em_ok (k INT, v VARCHAR(16))")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_em_ok VALUES (?, ?)",
|
||||
[(i, f"v{i}") for i in range(n)],
|
||||
)
|
||||
cur.execute("SELECT COUNT(*) FROM t_em_ok")
|
||||
assert cur.fetchone() == (n,)
|
||||
|
||||
|
||||
def test_executemany_empty_and_single(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_em_edge (k INT)")
|
||||
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [])
|
||||
_assert_healthy(cur)
|
||||
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [(1,)])
|
||||
cur.execute("SELECT COUNT(*) FROM t_em_edge")
|
||||
assert cur.fetchone() == (1,)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scrollable cursors — boundaries get their own round-trip each
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [0, 1, 2, 5, 50, 300])
|
||||
def test_scroll_boundaries(conn_params: ConnParams, n: int) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
setup = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
setup.execute("DROP TABLE t_scroll")
|
||||
setup.execute("CREATE TABLE t_scroll (k INT)")
|
||||
try:
|
||||
if n:
|
||||
setup.executemany(
|
||||
"INSERT INTO t_scroll VALUES (?)", [(i,) for i in range(n)]
|
||||
)
|
||||
|
||||
cur = conn.cursor(scrollable=True)
|
||||
try:
|
||||
cur.execute("SELECT k FROM t_scroll ORDER BY k")
|
||||
assert cur.fetch_first() == ((0,) if n else None)
|
||||
assert cur.fetch_last() == ((n - 1,) if n else None)
|
||||
|
||||
if n >= 3:
|
||||
assert cur.fetch_absolute(2) == (2,)
|
||||
assert cur.fetch_prior() == (1,)
|
||||
assert cur.fetch_relative(2) == (3,)
|
||||
|
||||
# Off both ends must be None — not a crash, not a wrap.
|
||||
assert cur.fetch_absolute(n + 50) is None
|
||||
cur.fetch_first()
|
||||
assert cur.fetch_prior() is None
|
||||
|
||||
if n:
|
||||
cur.fetch_first()
|
||||
seen = [(0,)]
|
||||
while (row := cur.fetchone()) is not None:
|
||||
seen.append(row)
|
||||
assert seen == [(i,) for i in range(n)]
|
||||
finally:
|
||||
cur.close()
|
||||
_assert_healthy(setup)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
setup.execute("DROP TABLE t_scroll")
|
||||
|
||||
|
||||
def test_abandoned_scroll_cursors_do_not_leak(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Scroll cursors stay open server-side, so abandoning one leaks
|
||||
unless the finalizer runs."""
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
setup = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
setup.execute("DROP TABLE t_scroll_ab")
|
||||
setup.execute("CREATE TABLE t_scroll_ab (k INT)")
|
||||
try:
|
||||
setup.executemany(
|
||||
"INSERT INTO t_scroll_ab VALUES (?)", [(i,) for i in range(20)]
|
||||
)
|
||||
for i in range(20):
|
||||
c = conn.cursor(scrollable=True)
|
||||
c.execute("SELECT k FROM t_scroll_ab ORDER BY k")
|
||||
c.fetch_first()
|
||||
if i % 2:
|
||||
c.close()
|
||||
else:
|
||||
del c # rely on the finalizer
|
||||
_assert_healthy(setup)
|
||||
setup.execute("SELECT COUNT(*) FROM t_scroll_ab")
|
||||
assert setup.fetchone() == (20,)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
setup.execute("DROP TABLE t_scroll_ab")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Smart LOBs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"size", [0, 1, 255, 256, 1023, 1024, 4095, 4096, 65535, 65536]
|
||||
)
|
||||
def test_blob_round_trip_sizes(conn_params: ConnParams, size: int) -> None:
|
||||
"""Sizes straddle the 4096-byte SQ_FILE chunk and the 64K mark."""
|
||||
payload = bytes((i * 7 + size) % 256 for i in range(size))
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_blob_rt")
|
||||
try:
|
||||
cur.execute("CREATE TABLE t_blob_rt (k INT, b BLOB)")
|
||||
except informix_db.Error:
|
||||
pytest.skip("no sbspace configured; see make ifx-spaces")
|
||||
try:
|
||||
cur.write_blob_column(
|
||||
"INSERT INTO t_blob_rt VALUES (?, BLOB_PLACEHOLDER)",
|
||||
payload, (1,),
|
||||
)
|
||||
got = cur.read_blob_column(
|
||||
"SELECT b FROM t_blob_rt WHERE k = ?", (1,)
|
||||
)
|
||||
if size == 0:
|
||||
assert got in (b"", None)
|
||||
else:
|
||||
assert got == payload
|
||||
_assert_healthy(cur)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_blob_rt")
|
||||
|
||||
|
||||
def test_failed_blob_read_recovers(conn_params: ConnParams) -> None:
|
||||
"""The SQ_FILE path involves a server-side temp file; a failure part
|
||||
way through must not strand the connection."""
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
for _ in range(5):
|
||||
with contextlib.suppress(Exception):
|
||||
cur.read_blob_column("SELECT b FROM t_no_such_blob_tbl", ())
|
||||
_assert_healthy(cur)
|
||||
248
tests/test_placeholder_rewrite.py
Normal file
248
tests/test_placeholder_rewrite.py
Normal file
@ -0,0 +1,248 @@
|
||||
"""Rewriting :N placeholders without rewriting the SQL around them.
|
||||
|
||||
We advertise ``paramstyle="numeric"`` to match Informix's ESQL/C
|
||||
convention, and the wire protocol takes ``?``, so placeholders get
|
||||
rewritten on the way out. That was ``re.sub(r":(\\d+)", "?", sql)``,
|
||||
which cannot see a string literal and therefore rewrote the inside of
|
||||
one::
|
||||
|
||||
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
|
||||
stored as 'http://host?/x'
|
||||
|
||||
Any ``HH:MM`` time, any URL with a port, any aspect ratio, any
|
||||
``key:value`` string. It wrote wrong data and said nothing.
|
||||
|
||||
Nothing about it was opt-in either. The rewrite runs whenever a
|
||||
statement has parameters, whatever placeholder style the caller actually
|
||||
used, so writing ``?`` everywhere and never touching numeric style did
|
||||
not protect you. And it changed the placeholder *count* while
|
||||
``num_qmarks`` was still computed from ``len(params)``, leaving driver
|
||||
and server disagreeing about how many binds exist.
|
||||
|
||||
The lexical rules below are Informix's own, measured against 12.10,
|
||||
14.10 and 15 rather than assumed from standard SQL. The one that matters
|
||||
most is the backslash: ``'a\\'b'`` is an *unterminated string* to
|
||||
Informix (``-282``), not an escaped quote. A scanner written to
|
||||
Postgres habits would desync on it and corrupt everything after.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _rewrite_numeric_to_qmark as rewrite
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Substitution happens where it should
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("sql", "expected"),
|
||||
[
|
||||
("SELECT * FROM t WHERE a = :1", "SELECT * FROM t WHERE a = ?"),
|
||||
(
|
||||
"SELECT * FROM t WHERE a = :1 AND b = :2",
|
||||
"SELECT * FROM t WHERE a = ? AND b = ?",
|
||||
),
|
||||
("SELECT * FROM t WHERE a = :10", "SELECT * FROM t WHERE a = ?"),
|
||||
# Already-? SQL is returned untouched, and cheaply.
|
||||
("SELECT * FROM t WHERE a = ?", "SELECT * FROM t WHERE a = ?"),
|
||||
("", ""),
|
||||
],
|
||||
)
|
||||
def test_placeholders_are_rewritten(sql: str, expected: str) -> None:
|
||||
assert rewrite(sql) == expected
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ...and nowhere else
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("label", "sql", "expected"),
|
||||
[
|
||||
(
|
||||
"url-with-port",
|
||||
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = :1",
|
||||
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?",
|
||||
),
|
||||
(
|
||||
"time-of-day",
|
||||
"INSERT INTO log VALUES ('started at 09:15:30', :1)",
|
||||
"INSERT INTO log VALUES ('started at 09:15:30', ?)",
|
||||
),
|
||||
(
|
||||
"aspect-ratio",
|
||||
"UPDATE t SET ratio = '16:9' WHERE id = :1",
|
||||
"UPDATE t SET ratio = '16:9' WHERE id = ?",
|
||||
),
|
||||
(
|
||||
"doubled-quote-escape",
|
||||
"SELECT 'it''s 10:30' FROM t WHERE a = :1",
|
||||
"SELECT 'it''s 10:30' FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"delimited-identifier",
|
||||
'SELECT "col:1" FROM t WHERE a = :1',
|
||||
'SELECT "col:1" FROM t WHERE a = ?',
|
||||
),
|
||||
(
|
||||
"line-comment",
|
||||
"-- ticket :99\nSELECT * FROM t WHERE a = :1",
|
||||
"-- ticket :99\nSELECT * FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"block-comment",
|
||||
"SELECT /* not :99 */ * FROM t WHERE a = :1",
|
||||
"SELECT /* not :99 */ * FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"brace-comment",
|
||||
"SELECT { not :99 } * FROM t WHERE a = :1",
|
||||
"SELECT { not :99 } * FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"cast-operator",
|
||||
"SELECT a::INT FROM t WHERE a = :1",
|
||||
"SELECT a::INT FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"colon-not-a-placeholder",
|
||||
"SELECT a FROM t WHERE b = ':x' AND c = :1",
|
||||
"SELECT a FROM t WHERE b = ':x' AND c = ?",
|
||||
),
|
||||
(
|
||||
"literal-after-placeholder",
|
||||
"SELECT * FROM t WHERE a = :1 AND b = '10:30'",
|
||||
"SELECT * FROM t WHERE a = ? AND b = '10:30'",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_quotes_and_comments_are_left_alone(
|
||||
label: str, sql: str, expected: str
|
||||
) -> None:
|
||||
assert rewrite(sql) == expected, label
|
||||
|
||||
|
||||
def test_backslash_does_not_escape_a_quote() -> None:
|
||||
"""Informix answers -282 for ``'a\\'b'``: the backslash is an ordinary
|
||||
character and the string is unterminated. A scanner that treated it
|
||||
as an escape would think it was still inside the literal and stop
|
||||
substituting, or worse, resume in the wrong place."""
|
||||
sql = "SELECT 'a\\' FROM t WHERE x = :1"
|
||||
# The quote closes at the character after the backslash, so :1 is
|
||||
# outside the literal and gets substituted.
|
||||
assert rewrite(sql) == "SELECT 'a\\' FROM t WHERE x = ?"
|
||||
|
||||
|
||||
def test_unterminated_quote_substitutes_nothing_further() -> None:
|
||||
"""Under-substituting leaves the server to reject SQL that was
|
||||
already malformed. Guessing would corrupt a literal."""
|
||||
assert rewrite("SELECT 'oops :1 FROM t") == "SELECT 'oops :1 FROM t"
|
||||
|
||||
|
||||
def test_unterminated_block_comment_substitutes_nothing_further() -> None:
|
||||
assert rewrite("SELECT /* oops :1 FROM t") == "SELECT /* oops :1 FROM t"
|
||||
|
||||
|
||||
def test_block_comments_do_not_nest() -> None:
|
||||
"""Measured: ``/* a /* b */ c */`` is a syntax error on all three
|
||||
servers, so the first ``*/`` closes the comment. Treating them as
|
||||
nesting would swallow live SQL."""
|
||||
assert (
|
||||
rewrite("SELECT /* a /* b */ :1 FROM t")
|
||||
== "SELECT /* a /* b */ ? FROM t"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_colon_literals_round_trip(conn_params: ConnParams) -> None:
|
||||
"""The bug as a user meets it: the value stored is not the value
|
||||
written. Both placeholder styles, because the rewrite ran for both."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_rw (id INT, s VARCHAR(60))")
|
||||
cur.execute(
|
||||
"INSERT INTO t_rw VALUES (?, 'http://host:8080/path')", (1,)
|
||||
)
|
||||
cur.execute("INSERT INTO t_rw VALUES (:1, 'at 09:15:30')", (2,))
|
||||
cur.execute("SELECT id, s FROM t_rw ORDER BY id")
|
||||
assert cur.fetchall() == [
|
||||
(1, "http://host:8080/path"),
|
||||
(2, "at 09:15:30"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_colon_literals_round_trip_through_executemany(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""executemany rewrites unconditionally, so it had the bug too."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_rw2 (id INT, s VARCHAR(40))")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_rw2 VALUES (?, '16:9')", [(1,), (2,)]
|
||||
)
|
||||
cur.execute("SELECT DISTINCT s FROM t_rw2")
|
||||
assert cur.fetchall() == [("16:9",)]
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_comment_bearing_sql_still_binds(conn_params: ConnParams) -> None:
|
||||
"""A leading comment reaches PREPARE now that classification asks the
|
||||
server. Make sure the rewriter agrees and doesn't eat a placeholder
|
||||
that follows one."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"/* report: daily */ SELECT FIRST 1 tabid FROM systables "
|
||||
"WHERE tabid > :1",
|
||||
(0,),
|
||||
)
|
||||
assert cur.fetchone() is not None
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_placeholder_count_matches_after_rewrite(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The old regex could add placeholders the driver never counted,
|
||||
leaving num_qmarks (from len(params)) disagreeing with the SQL. A
|
||||
literal containing three colon-digit sequences is the shape that
|
||||
used to break it."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_rw3 (id INT, s VARCHAR(40))")
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DELETE FROM t_rw3")
|
||||
cur.execute(
|
||||
"INSERT INTO t_rw3 VALUES (:1, 'a:1 b:2 c:3')", (7,)
|
||||
)
|
||||
cur.execute("SELECT id, s FROM t_rw3")
|
||||
assert cur.fetchone() == (7, "a:1 b:2 c:3")
|
||||
459
tests/test_recovery_and_concurrency.py
Normal file
459
tests/test_recovery_and_concurrency.py
Normal file
@ -0,0 +1,459 @@
|
||||
"""Regression tests for error recovery, cursor lifecycle, and concurrency.
|
||||
|
||||
The type-matrix suite covers what a value looks like on the wire. This
|
||||
covers everything around it: what happens after a statement fails, what
|
||||
happens to abandoned cursors, and what happens when more than one thread
|
||||
or task is involved.
|
||||
|
||||
Two bugs here were found by fuzzing, and both are the kind that never
|
||||
show up in a single-threaded happy-path test:
|
||||
|
||||
* **A failed statement was never released.** Successful DML sent
|
||||
PREPARE → EXECUTE → RELEASE; failing DML sent PREPARE → EXECUTE and
|
||||
stopped. The leaked statement collided with the next PREPARE, and
|
||||
every subsequent call on that connection returned a nonsense error
|
||||
whose offset pointed back at the *failed* SQL. A duplicate-key
|
||||
violation — about the most ordinary error an application can hit —
|
||||
killed the connection outright.
|
||||
|
||||
* **Cancelling a pool acquire leaked the connection.** ``asyncio.to_thread``
|
||||
cannot interrupt its worker, so a cancelled waiter left the worker to
|
||||
finish and hand back a connection nobody owned. Under HTTP load, where
|
||||
client disconnects cancel request tasks precisely while they wait for
|
||||
a connection, the pool dies one slot at a time.
|
||||
|
||||
The tests that follow assert on *data* wherever they can, not merely the
|
||||
absence of an exception: a worker reads back a value only it wrote, so a
|
||||
crossed wire fails even when nothing raises.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db import aio
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
# Bounded: a leaked statement used to manifest as a hang.
|
||||
read_timeout=25.0,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
def _pool_kw(conn_params: ConnParams) -> dict:
|
||||
return {
|
||||
"host": conn_params.host,
|
||||
"port": conn_params.port,
|
||||
"user": conn_params.user,
|
||||
"password": conn_params.password,
|
||||
"database": conn_params.database,
|
||||
"server": conn_params.server,
|
||||
"connect_timeout": 15.0,
|
||||
"read_timeout": 30.0,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Error recovery — a failed statement must not poison the connection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
FAILING_SQL = [
|
||||
pytest.param("SELECT FROM WHERE", id="syntax"),
|
||||
pytest.param("SELECT * FROM no_such_table_xyz", id="no_such_table"),
|
||||
pytest.param("SELECT no_such_col FROM systables", id="no_such_column"),
|
||||
pytest.param("SELECT no_such_function_xyz(1) FROM systables", id="bad_func"),
|
||||
pytest.param("SELECT 1/0 FROM systables", id="div_zero"),
|
||||
pytest.param("SELECT 'notanumber'::INT FROM systables", id="bad_cast"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("sql", FAILING_SQL)
|
||||
def test_connection_survives_failed_statement(
|
||||
conn_params: ConnParams, sql: str
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
for _ in range(4): # repeat: a leak accumulates
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute(sql)
|
||||
cur.fetchall()
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables ORDER BY tabid")
|
||||
assert cur.fetchone() is not None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("autocommit", [True, False])
|
||||
def test_duplicate_key_does_not_kill_the_connection(
|
||||
conn_params: ConnParams, autocommit: bool
|
||||
) -> None:
|
||||
"""The bug that started this file. A unique-constraint violation left
|
||||
the prepared statement allocated server-side; the next statement then
|
||||
failed with a nonsense error (-255 "Not in transaction" in autocommit,
|
||||
-285 otherwise) and the connection never recovered."""
|
||||
with _connect(conn_params, autocommit=autocommit) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_dup (k INT PRIMARY KEY)")
|
||||
cur.execute("INSERT INTO t_dup VALUES (1)")
|
||||
if not autocommit:
|
||||
conn.commit()
|
||||
|
||||
for i in range(5):
|
||||
with pytest.raises(informix_db.IntegrityError):
|
||||
cur.execute("INSERT INTO t_dup VALUES (1)")
|
||||
if not autocommit:
|
||||
conn.rollback()
|
||||
cur.execute("SELECT k FROM t_dup")
|
||||
assert cur.fetchall() == [(1,)], f"broken after {i + 1} violations"
|
||||
|
||||
|
||||
def test_failed_dml_with_params_releases_statement(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_nn (s VARCHAR(8) NOT NULL, k INT)")
|
||||
for _ in range(4):
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("INSERT INTO t_nn VALUES (?, ?)", (None, 1))
|
||||
cur.execute("SELECT COUNT(*) FROM t_nn")
|
||||
assert cur.fetchone() == (0,)
|
||||
|
||||
|
||||
def test_new_cursor_works_after_error(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("SELECT * FROM no_such_table_xyz")
|
||||
other = conn.cursor()
|
||||
other.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert other.fetchone() is not None
|
||||
other.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fetch batching — NFETCH is a byte budget, so batch edges move with width
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [0, 1, 2, 3, 63, 64, 65, 199, 200, 201, 512, 1025])
|
||||
def test_row_count_exact_across_fetch_styles(
|
||||
conn_params: ConnParams, n: int
|
||||
) -> None:
|
||||
"""Every fetch style must agree, at counts that straddle plausible
|
||||
server batch boundaries. A batching off-by-one shows up as a wrong
|
||||
count, a duplicate, or a dropped row."""
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_batch "
|
||||
"(k INT, a VARCHAR(32), b INT, c VARCHAR(32), d INT)"
|
||||
)
|
||||
if n:
|
||||
cur.executemany(
|
||||
"INSERT INTO t_batch VALUES (?, ?, ?, ?, ?)",
|
||||
[(i, f"a{i}", i * 2, f"c{i}", i * 3) for i in range(n)],
|
||||
)
|
||||
want = [(i,) for i in range(n)]
|
||||
sql = "SELECT k FROM t_batch ORDER BY k"
|
||||
|
||||
cur.execute(sql)
|
||||
assert cur.fetchall() == want
|
||||
|
||||
cur.execute(sql)
|
||||
got = []
|
||||
while (row := cur.fetchone()) is not None:
|
||||
got.append(row)
|
||||
assert got == want
|
||||
|
||||
for size in (1, 7, 100):
|
||||
cur.execute(sql)
|
||||
got = []
|
||||
while chunk := cur.fetchmany(size):
|
||||
got.extend(chunk)
|
||||
assert got == want, f"fetchmany({size}) disagreed at n={n}"
|
||||
|
||||
cur.execute(sql)
|
||||
assert list(cur) == want
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cursor lifecycle
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_abandoned_partial_fetches_do_not_leak(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_aband (k INT)")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_aband VALUES (?)", [(i,) for i in range(200)]
|
||||
)
|
||||
for i in range(25):
|
||||
c = conn.cursor()
|
||||
c.execute("SELECT k FROM t_aband ORDER BY k")
|
||||
c.fetchone()
|
||||
if i % 2:
|
||||
c.close()
|
||||
else:
|
||||
del c # rely on the finalizer
|
||||
cur.execute("SELECT COUNT(*) FROM t_aband")
|
||||
assert cur.fetchone() == (200,)
|
||||
|
||||
|
||||
def test_re_execute_mid_fetch(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_reex (k INT)")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_reex VALUES (?)", [(i,) for i in range(100)]
|
||||
)
|
||||
for _ in range(10):
|
||||
cur.execute("SELECT k FROM t_reex ORDER BY k")
|
||||
cur.fetchone() # abandon mid-fetch
|
||||
cur.execute("SELECT k FROM t_reex ORDER BY k")
|
||||
assert len(cur.fetchall()) == 100
|
||||
|
||||
|
||||
def test_interleaved_cursors_on_one_connection(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_inter (k INT)")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_inter VALUES (?)", [(i,) for i in range(50)]
|
||||
)
|
||||
a, b = conn.cursor(), conn.cursor()
|
||||
a.execute("SELECT k FROM t_inter ORDER BY k")
|
||||
b.execute("SELECT k FROM t_inter ORDER BY k DESC")
|
||||
assert [a.fetchone() for _ in range(5)] == [(i,) for i in range(5)]
|
||||
assert [b.fetchone() for _ in range(5)] == [
|
||||
(i,) for i in range(49, 44, -1)
|
||||
]
|
||||
a.close()
|
||||
b.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Concurrency
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_threads_sharing_one_connection_do_not_interleave(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""One connection is one socket. Each thread reads back a value only
|
||||
it supplied, so crossed wires fail even if nothing raises."""
|
||||
failures: list[str] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
def worker(tid: int) -> None:
|
||||
try:
|
||||
for r in range(10):
|
||||
cur = conn.cursor()
|
||||
token = tid * 1000 + r
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 ?::INT, ?::VARCHAR(16) FROM systables",
|
||||
(token, f"t{tid}"),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if row != (token, f"t{tid}"):
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: got {row!r}")
|
||||
return
|
||||
cur.close()
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
assert not failures, failures
|
||||
|
||||
|
||||
def test_pool_hands_out_clean_connections_under_load(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Borrowers that fail must still return a usable connection."""
|
||||
failures: list[str] = []
|
||||
lock = threading.Lock()
|
||||
pool = informix_db.create_pool(
|
||||
**_pool_kw(conn_params), min_size=2, max_size=4,
|
||||
acquire_timeout=30.0, autocommit=True,
|
||||
)
|
||||
try:
|
||||
def worker(tid: int) -> None:
|
||||
try:
|
||||
for r in range(8):
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
if r % 3 == 0:
|
||||
with contextlib.suppress(informix_db.Error):
|
||||
cur.execute("SELECT * FROM no_such_tbl_xyz")
|
||||
token = tid * 1000 + r
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 ?::INT FROM systables", (token,)
|
||||
)
|
||||
got = cur.fetchone()
|
||||
if got != (token,):
|
||||
with lock:
|
||||
failures.append(
|
||||
f"thread {tid}: got {got!r}, want {token}"
|
||||
)
|
||||
return
|
||||
cur.close()
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
pool.close()
|
||||
assert not failures, failures
|
||||
|
||||
|
||||
def test_pool_does_not_leak_transactions_between_borrowers(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""max_size=1 so both borrowers get the same underlying connection."""
|
||||
pool = informix_db.create_pool(
|
||||
**_pool_kw(conn_params), min_size=1, max_size=1,
|
||||
acquire_timeout=30.0, autocommit=False,
|
||||
)
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_pool_txn")
|
||||
cur.execute("CREATE TABLE t_pool_txn (k INT)")
|
||||
conn.commit()
|
||||
try:
|
||||
for i in range(5):
|
||||
with pool.connection() as conn: # dirties, never commits
|
||||
conn.cursor().execute(
|
||||
"INSERT INTO t_pool_txn VALUES (?)", (i,)
|
||||
)
|
||||
with pool.connection() as conn: # must not see it
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT COUNT(*) FROM t_pool_txn")
|
||||
assert cur.fetchone() == (0,), "transaction leaked"
|
||||
conn.rollback()
|
||||
finally:
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_pool_txn")
|
||||
conn.commit()
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
pool.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Async
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_async_pool_survives_cancelled_acquires(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Cancelling a task blocked on acquire used to destroy that pool slot
|
||||
permanently: ``asyncio.to_thread`` cannot interrupt its worker, so the
|
||||
worker finished and handed back a connection nobody owned.
|
||||
|
||||
Two holders keep both connections checked out; three waiters block on
|
||||
acquire and are cancelled there. Afterwards the pool must still serve.
|
||||
"""
|
||||
pool = await aio.create_pool(
|
||||
**_pool_kw(conn_params), min_size=1, max_size=2, autocommit=True
|
||||
)
|
||||
try:
|
||||
release = asyncio.Event()
|
||||
|
||||
async def holder(i: int) -> None:
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute(
|
||||
"SELECT FIRST 1 ?::INT FROM systables", (i,)
|
||||
)
|
||||
await cur.fetchone()
|
||||
await release.wait()
|
||||
|
||||
async def waiter(i: int) -> tuple | None:
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute(
|
||||
"SELECT FIRST 1 ?::INT FROM systables", (i,)
|
||||
)
|
||||
return await cur.fetchone()
|
||||
|
||||
for cycle in range(2):
|
||||
release.clear()
|
||||
holders = [asyncio.create_task(holder(i)) for i in range(2)]
|
||||
await asyncio.sleep(0.5) # both connections held
|
||||
waiters = [asyncio.create_task(waiter(100 + i)) for i in range(3)]
|
||||
await asyncio.sleep(0.5) # all blocked on acquire
|
||||
for t in waiters:
|
||||
t.cancel()
|
||||
await asyncio.gather(*waiters, return_exceptions=True)
|
||||
release.set()
|
||||
await asyncio.gather(*holders, return_exceptions=True)
|
||||
await asyncio.sleep(0.3) # orphan returns land
|
||||
|
||||
got = await asyncio.wait_for(waiter(200 + cycle), timeout=15)
|
||||
assert got == (200 + cycle,), (
|
||||
f"pool starved after cycle {cycle} — cancelled acquires leaked"
|
||||
)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
await pool.close()
|
||||
|
||||
|
||||
async def test_async_pool_concurrent_tasks_get_their_own_data(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
pool = await aio.create_pool(
|
||||
**_pool_kw(conn_params), min_size=2, max_size=4, autocommit=True
|
||||
)
|
||||
try:
|
||||
async def worker(tid: int) -> None:
|
||||
for r in range(6):
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
token = tid * 1000 + r
|
||||
await cur.execute(
|
||||
"SELECT FIRST 1 ?::INT FROM systables", (token,)
|
||||
)
|
||||
assert await cur.fetchone() == (token,)
|
||||
|
||||
await asyncio.gather(*(worker(i) for i in range(6)))
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
await pool.close()
|
||||
230
tests/test_row_reconciliation.py
Normal file
230
tests/test_row_reconciliation.py
Normal file
@ -0,0 +1,230 @@
|
||||
"""End-of-row reconciliation, and the two bugs it found immediately.
|
||||
|
||||
Fourteen framing bugs reached users before this check existed. Every one
|
||||
was the same defect — a column consuming the wrong number of bytes — and
|
||||
not one raised at the point of the mistake. The wrong width produced a
|
||||
plausible value and corrupted whatever came *next*, so the damage always
|
||||
surfaced in a different column, row, or statement.
|
||||
|
||||
All fourteen were detectable for free. ``payload`` is a fully extracted
|
||||
``bytes`` object of known length, so after decoding N columns the offset
|
||||
must land exactly on the end. ``_row_not_consumed`` raises when it
|
||||
doesn't, naming the column shape and the byte delta.
|
||||
|
||||
Turning the check on found two more bugs on its first run against the
|
||||
existing suite:
|
||||
|
||||
* **Smart LOBs were read as a flat 72-byte field.** They use the UDT
|
||||
envelope: 149 bytes populated (``[ind=0][len=144][144 hex chars]``),
|
||||
5 bytes NULL. We consumed 72 and left 77 behind, so any LOB not in the
|
||||
final column position corrupted everything after it. The 144 bytes are
|
||||
the 72-byte locator *hex-encoded*, which means ``BlobLocator.raw`` had
|
||||
never actually held a locator — only the first half of the hex text.
|
||||
Nobody noticed because ``read_blob_column`` resolves LOBs through the
|
||||
server-side ``lotofile`` function and never uses the locator.
|
||||
|
||||
* **NULL composite UDTs skipped their length field**, byte-for-byte the
|
||||
LVARCHAR NULL bug in the branch twelve lines away.
|
||||
|
||||
Both were the same ``[indicator][int32 length][data]`` envelope, written
|
||||
out by hand a third and fourth time. ``_read_udt_envelope`` is now the
|
||||
single implementation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import io
|
||||
import struct
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db._protocol import IfxStreamReader, ProtocolError
|
||||
from informix_db._resultset import (
|
||||
ColumnInfo,
|
||||
_read_udt_envelope,
|
||||
parse_tuple_payload,
|
||||
)
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The check itself — unit level, no server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _tuple_pdu(body: bytes) -> IfxStreamReader:
|
||||
"""Wrap a payload in the SQ_TUPLE framing parse_tuple_payload expects."""
|
||||
return IfxStreamReader(
|
||||
io.BytesIO(struct.pack("!h", 0) + struct.pack("!i", len(body)) + body)
|
||||
)
|
||||
|
||||
|
||||
def test_under_read_is_detected() -> None:
|
||||
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
|
||||
with pytest.raises(ProtocolError, match="under-read by 6"):
|
||||
parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5) + b"LEFTOV"), cols)
|
||||
|
||||
|
||||
def test_exact_consumption_passes() -> None:
|
||||
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
|
||||
assert parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5)), cols) == (5,)
|
||||
|
||||
|
||||
def test_error_names_the_column_shape() -> None:
|
||||
""""row decode failed" is not actionable. The message has to say which
|
||||
shape and by how much, so the next report arrives with the answer."""
|
||||
cols = [
|
||||
ColumnInfo(name="ident", type_code=2, raw_type_code=2, encoded_length=4),
|
||||
ColumnInfo(name="qty", type_code=2, raw_type_code=2, encoded_length=4),
|
||||
]
|
||||
body = struct.pack("!i", 1) + struct.pack("!i", 2) + b"UNCONSUMED"
|
||||
with pytest.raises(ProtocolError) as exc:
|
||||
parse_tuple_payload(_tuple_pdu(body), cols)
|
||||
msg = str(exc.value)
|
||||
assert "ident" in msg and "qty" in msg, "message must name the columns"
|
||||
assert "tc=2" in msg
|
||||
assert "consumed 8 of 18" in msg, "message must give the byte counts"
|
||||
assert "under-read by 10" in msg
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The shared UDT envelope
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_envelope_reads_length_even_when_null() -> None:
|
||||
"""The length belongs to the envelope, not the value. Returning early
|
||||
on the indicator is the bug that hit LVARCHAR and then composites."""
|
||||
payload = bytes([1]) + struct.pack("!i", 0) + b"NEXT"
|
||||
offset, body = _read_udt_envelope(payload, 0)
|
||||
assert body is None
|
||||
assert offset == 5, "NULL envelope must still consume its length field"
|
||||
assert payload[offset:] == b"NEXT"
|
||||
|
||||
|
||||
def test_envelope_reads_body_when_present() -> None:
|
||||
payload = bytes([0]) + struct.pack("!i", 3) + b"abc" + b"NEXT"
|
||||
offset, body = _read_udt_envelope(payload, 0)
|
||||
assert body == b"abc"
|
||||
assert offset == 8
|
||||
assert payload[offset:] == b"NEXT"
|
||||
|
||||
|
||||
def test_envelope_rejects_negative_length() -> None:
|
||||
"""A negative length off the wire would rewind the offset and silently
|
||||
re-decode earlier bytes as a later column."""
|
||||
payload = bytes([0]) + struct.pack("!i", -4) + b"abc"
|
||||
with pytest.raises(ProtocolError, match="negative length"):
|
||||
_read_udt_envelope(payload, 0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
pytestmark_integration = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host, port=conn_params.port, user=conn_params.user,
|
||||
password=conn_params.password, database=conn_params.database,
|
||||
server=conn_params.server, connect_timeout=15.0, read_timeout=30.0,
|
||||
autocommit=True,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_null_composite_does_not_shift_next_column(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""A NULL SET made the following VARCHAR return '' instead of its value."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_nullset")
|
||||
cur.execute(
|
||||
"CREATE TABLE t_nullset (a INT, s SET(INT NOT NULL), b VARCHAR(10))"
|
||||
)
|
||||
try:
|
||||
cur.execute("INSERT INTO t_nullset VALUES (7, SET{1,2}, 'abc')")
|
||||
cur.execute("INSERT INTO t_nullset VALUES (8, NULL, 'xyz')")
|
||||
cur.execute("SELECT a, s, b FROM t_nullset ORDER BY a")
|
||||
rows = cur.fetchall()
|
||||
assert rows[0][0] == 7 and rows[0][2] == "abc"
|
||||
assert rows[1] == (8, None, "xyz"), (
|
||||
"NULL composite shifted the following column"
|
||||
)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_nullset")
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.parametrize("populated", [False, True])
|
||||
def test_lob_in_non_final_position(
|
||||
conn_params: ConnParams, populated: bool
|
||||
) -> None:
|
||||
"""A LOB was read as 72 bytes when it occupies 149 (or 5 when NULL), so
|
||||
anything selected after it was garbage."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_lobmid")
|
||||
try:
|
||||
cur.execute(
|
||||
"CREATE TABLE t_lobmid (a INT, b BLOB, c INT, d VARCHAR(10))"
|
||||
)
|
||||
except informix_db.Error:
|
||||
pytest.skip("no sbspace configured; see make ifx-spaces")
|
||||
try:
|
||||
if populated:
|
||||
cur.write_blob_column(
|
||||
"INSERT INTO t_lobmid VALUES (?, BLOB_PLACEHOLDER, ?, ?)",
|
||||
b"hello", (1, 222222, "tail"),
|
||||
)
|
||||
else:
|
||||
cur.execute(
|
||||
"INSERT INTO t_lobmid VALUES (1, NULL, 222222, 'tail')"
|
||||
)
|
||||
cur.execute("SELECT a, b, c, d FROM t_lobmid")
|
||||
a, b, c, d = cur.fetchone()
|
||||
assert (a, c, d) == (1, 222222, "tail"), (
|
||||
"columns after the LOB were shifted"
|
||||
)
|
||||
if populated:
|
||||
assert isinstance(b, informix_db.BlobLocator)
|
||||
assert len(b.raw) == 72, (
|
||||
"locator must be the decoded 72 bytes, not hex text"
|
||||
)
|
||||
else:
|
||||
assert b is None
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_lobmid")
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_wide_mixed_row_reconciles(conn_params: ConnParams) -> None:
|
||||
"""Belt and braces: a row using most of the tricky types must consume
|
||||
its payload exactly. The check runs on every fetch, so merely getting
|
||||
a row back proves reconciliation held."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_recon ("
|
||||
" a INT8 NOT NULL, k LVARCHAR(512), n NCHAR(8), v VARCHAR(32),"
|
||||
" d DECIMAL(16), b BOOLEAN, t DATETIME YEAR TO FRACTION(5),"
|
||||
" tail INT)"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_recon VALUES (?, ?, ?, ?, ?, ?, CURRENT, ?)",
|
||||
(1, "PackageRoot", "nch", "v", None, True, 424242),
|
||||
)
|
||||
cur.execute(
|
||||
"SELECT a, k, n, v, d, b, t, tail FROM t_recon"
|
||||
)
|
||||
row = cur.fetchone()
|
||||
assert row[0] == 1
|
||||
assert row[-1] == 424242
|
||||
435
tests/test_rows.py
Normal file
435
tests/test_rows.py
Normal file
@ -0,0 +1,435 @@
|
||||
"""Named row access, and the cost of it.
|
||||
|
||||
A field request: `row[11]` tells a reader nothing on a wide projection,
|
||||
and `pyodbc` / `mssql-python` both hand back rows that answer to
|
||||
position, column name, and attribute at once. `row_factory=Row` gives
|
||||
the same three.
|
||||
|
||||
It is opt-in. Defaulting it on would tax the thing this driver is
|
||||
measured against, since supporting `row["name"]` means `__getitem__`
|
||||
becomes a Python method rather than C-level tuple indexing, which costs
|
||||
roughly 39 ns on every subscript. Users who want readable column access
|
||||
in application code should pay that; a bulk export that never looks at a
|
||||
column by name should not.
|
||||
|
||||
`Row` subclasses `tuple`, so `row == (1, "x")` still holds and nothing
|
||||
that already worked stops working. That constraint drove the design: the
|
||||
existing suite compares fetched rows against plain tuples in hundreds of
|
||||
places, and so does everybody's code.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import pickle
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.rows import _RESERVED, Row, make_row_class
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The type itself
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _row(fields, values):
|
||||
return make_row_class(tuple(fields))(values)
|
||||
|
||||
|
||||
def test_three_ways_to_reach_a_column() -> None:
|
||||
row = _row(("tabid", "tabname"), (1, "systables"))
|
||||
assert row[0] == 1
|
||||
assert row["tabname"] == "systables"
|
||||
assert row.tabname == "systables"
|
||||
|
||||
|
||||
def test_still_equal_to_a_plain_tuple() -> None:
|
||||
"""The compatibility constraint. Existing code and the existing test
|
||||
suite compare fetched rows against tuples everywhere."""
|
||||
row = _row(("a", "b"), (1, "x"))
|
||||
plain = (1, "x")
|
||||
assert row == plain
|
||||
# Both directions: tuple.__eq__ on the left has to accept a subclass
|
||||
# on the right, or `expected == fetched` assertions break.
|
||||
assert plain == row
|
||||
assert list(row) == [1, "x"]
|
||||
assert len(row) == 2
|
||||
a, b = row
|
||||
assert (a, b) == (1, "x")
|
||||
assert row in [(1, "x")]
|
||||
|
||||
|
||||
def test_slice_degrades_to_a_plain_tuple() -> None:
|
||||
"""A slice has no meaningful column mapping, so it should not pretend
|
||||
to be a Row."""
|
||||
row = _row(("a", "b", "c"), (1, 2, 3))
|
||||
assert row[0:2] == (1, 2)
|
||||
assert type(row[0:2]) is tuple
|
||||
|
||||
|
||||
def test_negative_index_still_works() -> None:
|
||||
assert _row(("a", "b"), (1, 2))[-1] == 2
|
||||
|
||||
|
||||
def test_keys_and_asdict() -> None:
|
||||
row = _row(("a", "b"), (1, "x"))
|
||||
assert row.keys() == ("a", "b")
|
||||
assert row._asdict() == {"a": 1, "b": "x"}
|
||||
|
||||
|
||||
def test_repr_names_the_columns() -> None:
|
||||
assert repr(_row(("a", "b"), (1, "x"))) == "Row(a=1, b='x')"
|
||||
|
||||
|
||||
def test_pickles() -> None:
|
||||
"""The per-shape class is built at runtime, so it cannot be pickled by
|
||||
reference. Multiprocessing users would hit that immediately."""
|
||||
row = _row(("a", "b"), (1, "x"))
|
||||
restored = pickle.loads(pickle.dumps(row))
|
||||
assert restored == (1, "x")
|
||||
assert restored.b == "x"
|
||||
|
||||
|
||||
def test_missing_column_says_what_is_there() -> None:
|
||||
row = _row(("tabid", "tabname"), (1, "x"))
|
||||
with pytest.raises(KeyError, match="tabid"):
|
||||
_ = row["nope"]
|
||||
with pytest.raises(AttributeError, match="tabid"):
|
||||
_ = row.nope
|
||||
|
||||
|
||||
def test_class_is_cached_per_shape() -> None:
|
||||
"""A type() call per execute() would land on every small query."""
|
||||
assert make_row_class(("a", "b")) is make_row_class(("a", "b"))
|
||||
assert make_row_class(("a", "b")) is not make_row_class(("a", "c"))
|
||||
|
||||
|
||||
def test_duplicate_names_resolve_to_the_first() -> None:
|
||||
"""``SELECT tabid, tabid`` is legal and both columns are named
|
||||
``tabid``. pyodbc gives the first; so do we."""
|
||||
row = _row(("tabid", "tabid"), (1, 2))
|
||||
assert row["tabid"] == 1
|
||||
assert row.tabid == 1
|
||||
assert row[1] == 2, "positional access must still reach the second"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", sorted(_RESERVED))
|
||||
def test_a_column_always_beats_a_method_of_the_same_name(name: str) -> None:
|
||||
"""Parametrized over the *computed* reserved set rather than a
|
||||
hand-written list, because the hand-written list is what missed
|
||||
``keys``, ``_asdict`` and ``_fields`` on the first attempt. Adding a
|
||||
method to Row later cannot silently reopen the hole: this test grows
|
||||
with it.
|
||||
|
||||
Without the guard these return a bound method, which is the exact
|
||||
failure shape this driver spent a fortnight removing from its
|
||||
decoders: a plausible-looking wrong answer, in silence."""
|
||||
row = _row((name, "other"), (7, 9))
|
||||
assert getattr(row, name) == 7
|
||||
assert row[name] == 7
|
||||
|
||||
|
||||
def test_shadowed_methods_stay_reachable_through_the_base_class() -> None:
|
||||
row = _row(("count", "keys", "_asdict"), (1, 2, 3))
|
||||
assert tuple.count(row, 1) == 1
|
||||
assert Row.keys(row) == ("count", "keys", "_asdict")
|
||||
assert Row._asdict(row) == {"count": 1, "keys": 2, "_asdict": 3}
|
||||
|
||||
|
||||
def test_shadowing_fields_does_not_break_repr_or_asdict() -> None:
|
||||
"""The machinery reads its own field list, so if a column named
|
||||
``_fields`` shadowed it, repr and _asdict would report the column
|
||||
value instead of the names. Mangled attributes keep them separate."""
|
||||
row = _row(("_fields", "x"), ("not the names", 2))
|
||||
assert row._fields == "not the names"
|
||||
assert Row.keys(row) == ("_fields", "x")
|
||||
assert repr(row) == "Row(_fields='not the names', x=2)"
|
||||
|
||||
|
||||
def test_non_identifier_names_are_subscript_only() -> None:
|
||||
"""Informix names expression columns things like ``(count(*))``,
|
||||
which cannot be an attribute."""
|
||||
row = _row(("(count(*))", "ok"), (5, 1))
|
||||
assert row["(count(*))"] == 5
|
||||
assert row.ok == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_default_is_still_a_plain_tuple(conn_params: ConnParams) -> None:
|
||||
"""The opt-in has to be genuinely opt-in. Anyone who does not ask for
|
||||
Row should not pay for it or see any behaviour change."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT FIRST 1 tabid, tabname FROM systables")
|
||||
row = cur.fetchone()
|
||||
assert type(row) is tuple
|
||||
with pytest.raises(TypeError):
|
||||
_ = row["tabname"]
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_row_factory_on_the_connection(conn_params: ConnParams) -> None:
|
||||
"""One line at connect time, then every cursor from it, which is what
|
||||
'without any additional coding' has to mean in practice."""
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 tabid, tabname FROM systables ORDER BY tabid"
|
||||
)
|
||||
row = cur.fetchone()
|
||||
assert row[0] == row["tabid"] == row.tabid
|
||||
assert row[1] == row["tabname"] == row.tabname
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_every_fetch_path_returns_rows(conn_params: ConnParams) -> None:
|
||||
"""fetchone, fetchmany, fetchall and iteration each return rows by a
|
||||
different route through the cursor."""
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
sql = "SELECT FIRST 4 tabid, tabname FROM systables ORDER BY tabid"
|
||||
cur = conn.cursor()
|
||||
|
||||
cur.execute(sql)
|
||||
assert cur.fetchone().tabname is not None
|
||||
|
||||
cur.execute(sql)
|
||||
assert all(r.tabname is not None for r in cur.fetchmany(2))
|
||||
|
||||
cur.execute(sql)
|
||||
assert all(r.tabname is not None for r in cur.fetchall())
|
||||
|
||||
cur.execute(sql)
|
||||
assert all(r.tabname is not None for r in cur)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_scrollable_cursor_returns_rows(conn_params: ConnParams) -> None:
|
||||
"""Scrollable cursors return each row straight from the wire rather
|
||||
than from the materialized list, so they are a separate path."""
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
cur = conn.cursor(scrollable=True)
|
||||
cur.execute("SELECT tabid, tabname FROM systables ORDER BY tabid")
|
||||
assert cur.fetch_first().tabname is not None
|
||||
assert cur.fetch_absolute(1).tabname is not None
|
||||
cur.close()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_per_cursor_override(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
named = conn.cursor()
|
||||
named.row_factory = informix_db.Row
|
||||
plain = conn.cursor()
|
||||
sql = "SELECT FIRST 1 tabid FROM systables"
|
||||
named.execute(sql)
|
||||
plain.execute(sql)
|
||||
assert isinstance(named.fetchone(), Row)
|
||||
assert type(plain.fetchone()) is tuple
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_informix_lowercases_so_the_obvious_name_works(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Informix folds unquoted identifiers, which is why the .lower() in
|
||||
the workaround people write by hand is a no-op."""
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_rows (Config_Key INT, Cnt INT)")
|
||||
cur.execute("INSERT INTO t_rows VALUES (1, 2)")
|
||||
cur.execute("SELECT Config_Key, Cnt FROM t_rows")
|
||||
row = cur.fetchone()
|
||||
assert row.config_key == 1
|
||||
assert row.cnt == 2
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_pool_forwards_the_factory(conn_params: ConnParams) -> None:
|
||||
pool = informix_db.create_pool(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
autocommit=True,
|
||||
row_factory=informix_db.Row,
|
||||
min_size=1,
|
||||
max_size=2,
|
||||
)
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone().tabid is not None
|
||||
finally:
|
||||
pool.close()
|
||||
|
||||
|
||||
def test_zero_column_row() -> None:
|
||||
row = make_row_class(())(())
|
||||
assert row == ()
|
||||
assert row.keys() == ()
|
||||
assert repr(row) == "Row()"
|
||||
|
||||
|
||||
def test_wide_row() -> None:
|
||||
"""500 columns: the name map is a dict, so this should be flat, but
|
||||
it is the shape most likely to expose an off-by-one."""
|
||||
fields = tuple(f"c{i}" for i in range(500))
|
||||
row = make_row_class(fields)(tuple(range(500)))
|
||||
assert row["c499"] == row.c499 == row[499] == 499
|
||||
assert row["c0"] == row[0] == 0
|
||||
|
||||
|
||||
def test_names_that_cannot_be_attributes_are_subscript_only() -> None:
|
||||
row = _row(("col with space", "1leading_digit", ""), (1, 2, 3))
|
||||
assert row["col with space"] == 1
|
||||
assert row["1leading_digit"] == 2
|
||||
assert row[""] == 3
|
||||
|
||||
|
||||
def test_unicode_column_name() -> None:
|
||||
row = _row(("café", "x"), (1, 2))
|
||||
assert row["café"] == 1
|
||||
assert row.café == 1
|
||||
|
||||
|
||||
def test_dunder_named_column_is_subscript_only() -> None:
|
||||
"""A dunder cannot be shadowed without breaking the object protocol,
|
||||
so the column is reachable by subscript and the attribute keeps its
|
||||
ordinary meaning. No real schema should notice."""
|
||||
row = _row(("__class__", "ok"), (1, 2))
|
||||
assert row["__class__"] == 1
|
||||
assert row.__class__.__name__ == "Row"
|
||||
|
||||
|
||||
def test_row_outlives_eviction_of_its_class_from_the_cache() -> None:
|
||||
"""The class cache is bounded. A row already handed to the caller
|
||||
holds its class alive, so eviction must not affect it."""
|
||||
row = make_row_class(("z1", "z2"))((9, 8))
|
||||
for i in range(300):
|
||||
make_row_class((f"evict{i}", "b"))
|
||||
assert (row.z1, row["z2"], row[0]) == (9, 8, 9)
|
||||
|
||||
|
||||
def test_class_cache_is_bounded() -> None:
|
||||
for i in range(400):
|
||||
make_row_class((f"bounded{i}", "b"))
|
||||
assert make_row_class.cache_info().currsize <= 256
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Row must not change a single value
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_WIDE_DDL = """CREATE TABLE t_rowdiff (
|
||||
c_int INT, c_small SMALLINT, c_big BIGINT, c_int8 INT8, c_serial SERIAL,
|
||||
c_float FLOAT, c_smallfloat SMALLFLOAT, c_dec DECIMAL(16,4),
|
||||
c_decu DECIMAL(16), c_money MONEY(12,2), c_char CHAR(10),
|
||||
c_vchar VARCHAR(40), c_nchar NCHAR(8), c_lvar LVARCHAR(200), c_date DATE,
|
||||
c_dt DATETIME YEAR TO FRACTION(5), c_ivl INTERVAL YEAR TO MONTH,
|
||||
c_bool BOOLEAN, c_set SET(INT NOT NULL), c_tail INT)"""
|
||||
|
||||
_WIDE_COLS = (
|
||||
"c_int,c_small,c_big,c_int8,c_serial,c_float,c_smallfloat,c_dec,c_decu,"
|
||||
"c_money,c_char,c_vchar,c_nchar,c_lvar,c_date,c_dt,c_ivl,c_bool,c_set,"
|
||||
"c_tail"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_rows_are_value_identical_to_tuples(conn_params: ConnParams) -> None:
|
||||
"""The strongest statement available: across every awkward type, with
|
||||
a fully populated row and a fully NULL one, wrapping must change
|
||||
nothing. If it does, Row is not a presentation layer, it is a bug."""
|
||||
sql = f"SELECT {_WIDE_COLS} FROM t_rowdiff ORDER BY c_tail"
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_rowdiff")
|
||||
cur.execute(_WIDE_DDL)
|
||||
try:
|
||||
cur.execute(
|
||||
"INSERT INTO t_rowdiff VALUES (1,2,3,4,0,1.5,2.5,12.34,99,"
|
||||
"9.99,'ch','vc','nc','lv',TODAY,CURRENT,"
|
||||
"INTERVAL(1-2) YEAR TO MONTH,'t',SET{1,2},77)"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_rowdiff VALUES (NULL,NULL,NULL,NULL,0,NULL,"
|
||||
"NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,"
|
||||
"NULL,NULL,88)"
|
||||
)
|
||||
cur.execute(sql)
|
||||
plain = cur.fetchall()
|
||||
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as named_conn:
|
||||
named = named_conn.cursor()
|
||||
named.execute(sql)
|
||||
rows = named.fetchall()
|
||||
|
||||
assert rows == plain, "wrapping changed a value"
|
||||
names = _WIDE_COLS.split(",")
|
||||
first = rows[0]
|
||||
for i, name in enumerate(names):
|
||||
assert first[i] == first[name] == getattr(first, name), name
|
||||
assert rows[1]["c_int"] is None
|
||||
assert rows[1].c_tail == 88
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_rowdiff")
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_threads_share_one_class_per_shape(conn_params: ConnParams) -> None:
|
||||
"""The cache is process-wide and the pool hands connections to many
|
||||
threads, so two threads running the same query must land on the same
|
||||
class rather than racing to build competing ones."""
|
||||
import threading
|
||||
|
||||
seen: list[type] = []
|
||||
errors: list[Exception] = []
|
||||
|
||||
def worker() -> None:
|
||||
try:
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
cur = conn.cursor()
|
||||
for _ in range(5):
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 tabid, tabname FROM systables"
|
||||
)
|
||||
row = cur.fetchone()
|
||||
assert row.tabname == row["tabname"] == row[1]
|
||||
seen.append(type(row))
|
||||
except Exception as exc:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker) for _ in range(6)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(60)
|
||||
assert not errors, errors[:2]
|
||||
assert len({id(c) for c in seen}) == 1, "same shape built more than once"
|
||||
172
tests/test_socket_reads.py
Normal file
172
tests/test_socket_reads.py
Normal file
@ -0,0 +1,172 @@
|
||||
"""Two readers, one stream, and a length field taken on trust.
|
||||
|
||||
``IfxSocket`` owns a read-ahead buffer that ``BufferedSocketReader``
|
||||
fills and drains. But ``Connection._drain_to_eot``, ``_raise_sq_err``
|
||||
and the login path bypass that reader and call ``IfxSocket.read_exact``
|
||||
directly, which recv'd from the socket without ever looking at the
|
||||
buffer. Bytes sitting in the buffer would simply be skipped, and skipped
|
||||
bytes in a length-framed protocol don't announce themselves — the next
|
||||
read lands mid-field and every read after it is wrong.
|
||||
|
||||
Nothing triggers it today. The server sends one response per request, so
|
||||
recv returns exactly that response and the buffered reader consumes all
|
||||
of it before control returns to a direct read. That is a property of the
|
||||
traffic, not of the code, and the buffer is connection-scoped precisely
|
||||
so read-ahead *can* cross response boundaries — pipelined executemany
|
||||
already puts several responses in flight. A latent desync waiting on a
|
||||
timing change is not a good thing to leave in a wire protocol.
|
||||
|
||||
Separately, ``fill_recv_buf`` took its byte count on trust, and that
|
||||
count is almost always a length field straight off the wire. A corrupt
|
||||
or desynced stream turned into an allocation of whatever the field
|
||||
happened to say: a garbage ``0x7FFFFFFF`` reads as a 2 GB request, and
|
||||
the fill loop sits in recv until the read timeout while the buffer
|
||||
grows. The limit turns that into an error that names the number, which
|
||||
is the actual diagnostic — a length that absurd means framing was
|
||||
already lost upstream.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from informix_db._protocol import BufferedSocketReader, ProtocolError
|
||||
from informix_db._socket import MAX_READ_BYTES, IfxSocket
|
||||
|
||||
|
||||
class _FakeSocket:
|
||||
"""Stands in for the raw socket. Records what recv actually asked for."""
|
||||
|
||||
def __init__(self, data: bytes = b"") -> None:
|
||||
self.data = data
|
||||
self.pos = 0
|
||||
self.recv_calls: list[int] = []
|
||||
|
||||
def recv(self, n: int) -> bytes:
|
||||
self.recv_calls.append(n)
|
||||
chunk = self.data[self.pos : self.pos + n]
|
||||
self.pos += len(chunk)
|
||||
return chunk
|
||||
|
||||
def close(self) -> None:
|
||||
# The EOF path force-closes; a stand-in has to survive that.
|
||||
pass
|
||||
|
||||
|
||||
def _socket_with(buffered: bytes, on_wire: bytes = b"") -> IfxSocket:
|
||||
"""An IfxSocket with ``buffered`` already read ahead into _recv_buf."""
|
||||
sock = IfxSocket.__new__(IfxSocket)
|
||||
sock._sock = _FakeSocket(on_wire)
|
||||
sock._recv_buf = bytearray(buffered)
|
||||
sock._recv_pos = 0
|
||||
sock._recv_size = 65536
|
||||
sock._read_timeout = None
|
||||
return sock
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# read_exact must not step over the buffer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_read_exact_consumes_the_buffer_first() -> None:
|
||||
sock = _socket_with(b"BUFFERED", on_wire=b"SOCKET")
|
||||
assert sock.read_exact(8) == b"BUFFERED"
|
||||
assert sock._sock.recv_calls == [], "must not touch the socket at all"
|
||||
assert sock._recv_pos == 8
|
||||
|
||||
|
||||
def test_read_exact_spans_buffer_then_socket() -> None:
|
||||
"""The interesting case: a read that starts in the buffer and
|
||||
finishes on the wire. Getting this wrong reorders the stream."""
|
||||
sock = _socket_with(b"HEAD", on_wire=b"TAIL")
|
||||
assert sock.read_exact(8) == b"HEADTAIL"
|
||||
assert sock._sock.recv_calls == [4], "only the shortfall comes from recv"
|
||||
|
||||
|
||||
def test_read_exact_respects_a_partly_consumed_buffer() -> None:
|
||||
sock = _socket_with(b"XXABCD")
|
||||
sock._recv_pos = 2 # first two bytes already decoded
|
||||
assert sock.read_exact(4) == b"ABCD"
|
||||
assert sock._sock.recv_calls == []
|
||||
|
||||
|
||||
def test_read_exact_of_zero_is_empty() -> None:
|
||||
sock = _socket_with(b"DATA")
|
||||
assert sock.read_exact(0) == b""
|
||||
assert sock.read_exact(-5) == b"", "a negative count must not rewind"
|
||||
assert sock._recv_pos == 0
|
||||
|
||||
|
||||
def test_short_read_error_reports_the_original_request() -> None:
|
||||
"""The message counts bytes; taking some from the buffer must not make
|
||||
it lie about how many were asked for."""
|
||||
from informix_db.exceptions import OperationalError
|
||||
|
||||
sock = _socket_with(b"AB", on_wire=b"") # 2 buffered, nothing on the wire
|
||||
with pytest.raises(OperationalError, match="wanted 10 bytes"):
|
||||
sock.read_exact(10)
|
||||
|
||||
|
||||
def test_buffered_reader_and_direct_read_agree_on_one_stream() -> None:
|
||||
"""End to end: a BufferedSocketReader over-reads, then a direct
|
||||
read_exact picks up exactly where it left off."""
|
||||
sock = _socket_with(b"", on_wire=b"\x00\x2aREST-OF-THE-STREAM")
|
||||
reader = BufferedSocketReader(sock)
|
||||
assert reader.read_short() == 42
|
||||
assert len(sock._recv_buf) - sock._recv_pos > 0, (
|
||||
"precondition: the reader must have over-read for this to mean "
|
||||
"anything"
|
||||
)
|
||||
assert sock.read_exact(18) == b"REST-OF-THE-STREAM"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# fill_recv_buf must not believe an arbitrary length
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_absurd_length_is_refused_not_allocated() -> None:
|
||||
sock = _socket_with(b"", on_wire=b"")
|
||||
with pytest.raises(ProtocolError, match="refusing to read"):
|
||||
sock.fill_recv_buf(MAX_READ_BYTES + 1)
|
||||
assert sock._sock.recv_calls == [], "must refuse before any recv"
|
||||
|
||||
|
||||
def test_refusal_names_the_knob() -> None:
|
||||
"""The error has to be actionable in both directions: framing is lost,
|
||||
or the value genuinely is that big and the limit needs raising."""
|
||||
sock = _socket_with(b"", on_wire=b"")
|
||||
with pytest.raises(ProtocolError) as exc:
|
||||
sock.fill_recv_buf(2**31 - 1)
|
||||
message = str(exc.value)
|
||||
assert "2147483647" in message
|
||||
assert "IFX_MAX_READ_BYTES" in message
|
||||
|
||||
|
||||
def test_a_normal_length_is_unaffected() -> None:
|
||||
sock = _socket_with(b"", on_wire=b"x" * 100)
|
||||
sock.fill_recv_buf(100)
|
||||
assert len(sock._recv_buf) - sock._recv_pos >= 100
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# skip
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_buffered_skip_does_not_rewind_on_a_negative_count() -> None:
|
||||
"""The base reader's skip delegates to read_exact, which guards. This
|
||||
one advances the cursor arithmetically, so an unguarded negative count
|
||||
re-decodes bytes already consumed as if they were the next field."""
|
||||
sock = _socket_with(b"ABCDEFGH")
|
||||
sock._recv_pos = 4
|
||||
BufferedSocketReader(sock).skip(-4)
|
||||
assert sock._recv_pos == 4, "skip must never move the cursor backwards"
|
||||
|
||||
|
||||
def test_buffered_skip_advances_normally() -> None:
|
||||
sock = _socket_with(b"ABCDEFGH")
|
||||
reader = BufferedSocketReader(sock)
|
||||
reader.skip(4)
|
||||
assert reader.read_exact(4) == b"EFGH"
|
||||
216
tests/test_statement_classification.py
Normal file
216
tests/test_statement_classification.py
Normal file
@ -0,0 +1,216 @@
|
||||
"""Deciding whether a statement needs a cursor, by asking rather than guessing.
|
||||
|
||||
The driver chose between "open a cursor and fetch" and "execute and
|
||||
release" by checking whether the first word of the SQL was ``SELECT``.
|
||||
That gets five ordinary forms wrong — a leading comment in any of the
|
||||
three Informix flavours, a parenthesized select, a parenthesized UNION,
|
||||
and a CTE. All five are perfectly good queries, and all five failed with
|
||||
``-260 Cursor name already in use``, an error that describes neither the
|
||||
cause nor anything the caller did. It says "cursor" because the driver
|
||||
sent SQ_EXECUTE where the server was waiting to open one.
|
||||
|
||||
The server had been telling us the answer the whole time.
|
||||
``statement_type`` is the first field of the DESCRIBE response, and
|
||||
``parse_describe`` has always parsed it into the metadata dict, where
|
||||
nothing read it. Every SELECT form above reports 2.
|
||||
|
||||
The comment that justified the heuristic said ``nfields`` couldn't
|
||||
distinguish these cases, because ``INSERT INTO t VALUES (?)`` also
|
||||
describes a column. That much was true — and it argued for the wrong
|
||||
conclusion, because ``statement_type`` distinguishes them exactly. That
|
||||
INSERT reports 6.
|
||||
|
||||
The predicate is now JDBC's ``IfxSqli.isResultSet``: type 2, or type 56
|
||||
(``EXECUTE PROCEDURE``/``FUNCTION``) with at least one column, since a
|
||||
routine may or may not return rows. That last clause fixes
|
||||
``EXECUTE FUNCTION`` as a side effect — it used to run down the DML path
|
||||
and discard its return value.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _produces_result_set
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The predicate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("statement_type", "ncolumns", "expected", "why"),
|
||||
[
|
||||
(2, 1, True, "SELECT"),
|
||||
(2, 0, True, "SELECT describing no columns is still a SELECT"),
|
||||
(6, 1, False, "INSERT ... VALUES (?) describes a column but is DML"),
|
||||
(6, 0, False, "INSERT with literals"),
|
||||
(32, 0, False, "DELETE"),
|
||||
(33, 0, False, "UPDATE"),
|
||||
(45, 0, False, "CREATE"),
|
||||
(56, 0, False, "EXECUTE PROCEDURE returning nothing"),
|
||||
(56, 2, True, "EXECUTE FUNCTION returning rows"),
|
||||
(0, 0, False, "unknown type defaults to the non-cursor path"),
|
||||
],
|
||||
)
|
||||
def test_result_set_predicate(
|
||||
statement_type: int, ncolumns: int, expected: bool, why: str
|
||||
) -> None:
|
||||
assert _produces_result_set(statement_type, ncolumns) is expected, why
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
)
|
||||
|
||||
|
||||
_ONE = "SELECT FIRST 1 tabid FROM systables"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.parametrize(
|
||||
("label", "sql"),
|
||||
[
|
||||
pytest.param("plain", _ONE, id="plain"),
|
||||
pytest.param("lowercase", _ONE.lower(), id="lowercase"),
|
||||
pytest.param("leading-whitespace", f" \n\t {_ONE}", id="whitespace"),
|
||||
pytest.param("line-comment", f"-- pick one\n{_ONE}", id="line-comment"),
|
||||
pytest.param("block-comment", f"/* pick one */ {_ONE}", id="block-comment"),
|
||||
pytest.param("brace-comment", f"{{ pick one }} {_ONE}", id="brace-comment"),
|
||||
pytest.param(
|
||||
"cte",
|
||||
"WITH c AS (SELECT tabid FROM systables) "
|
||||
"SELECT FIRST 1 tabid FROM c",
|
||||
id="cte",
|
||||
),
|
||||
pytest.param("parenthesized", f"({_ONE})", id="parenthesized"),
|
||||
pytest.param(
|
||||
"union", f"{_ONE} UNION SELECT 99 FROM systables", id="union"
|
||||
),
|
||||
pytest.param(
|
||||
"parenthesized-union",
|
||||
f"({_ONE}) UNION (SELECT 99 FROM systables)",
|
||||
id="paren-union",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_every_select_form_opens_a_cursor(
|
||||
conn_params: ConnParams, label: str, sql: str
|
||||
) -> None:
|
||||
"""The five non-``plain`` forms below the whitespace case all failed
|
||||
with -260 under the first-word heuristic."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
try:
|
||||
cur.execute(sql)
|
||||
except informix_db.ProgrammingError as exc:
|
||||
# Informix 12.10 has no CTEs and rejects WITH at offset 1.
|
||||
# A syntax error is the server declining the grammar, which
|
||||
# is a different thing from the driver routing it wrongly —
|
||||
# that produced -260, not -201.
|
||||
if getattr(exc, "sqlcode", None) == -201:
|
||||
pytest.skip(f"server does not support this syntax: {label}")
|
||||
raise
|
||||
rows = cur.fetchall()
|
||||
assert rows, f"{label}: expected rows, got none"
|
||||
assert cur.description is not None
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_parameterized_insert_is_not_mistaken_for_a_query(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The case the old comment worried about, and the reason it kept the
|
||||
heuristic: this DESCRIBEs a column. ``statement_type`` says 6."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_cls (k INT)")
|
||||
cur.execute("INSERT INTO t_cls VALUES (?)", (7,))
|
||||
assert cur.rowcount == 1
|
||||
cur.execute("SELECT k FROM t_cls")
|
||||
assert cur.fetchall() == [(7,)]
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_execute_function_returns_its_value(conn_params: ConnParams) -> None:
|
||||
"""Type 56 with columns. Under the first-word heuristic this ran down
|
||||
the DML path and the return value was discarded."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP FUNCTION ifxdrv_dbl")
|
||||
cur.execute(
|
||||
"CREATE FUNCTION ifxdrv_dbl(n INT) RETURNING INT; "
|
||||
"RETURN n * 2; END FUNCTION"
|
||||
)
|
||||
try:
|
||||
cur.execute("EXECUTE FUNCTION ifxdrv_dbl(21)")
|
||||
assert cur.fetchall() == [(42,)]
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP FUNCTION ifxdrv_dbl")
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_dml_still_takes_the_non_cursor_path(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_cls2 (k INT)")
|
||||
cur.execute("INSERT INTO t_cls2 VALUES (1)")
|
||||
cur.execute("UPDATE t_cls2 SET k = 2")
|
||||
assert cur.rowcount == 1
|
||||
cur.execute("DELETE FROM t_cls2")
|
||||
assert cur.rowcount == 1
|
||||
assert cur.description is None
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_executemany_refuses_a_query_the_first_word_missed(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The pre-flight check shares the heuristic's blind spots, so a
|
||||
comment-prefixed SELECT reaches PREPARE. The post-DESCRIBE check
|
||||
catches it, and the connection stays usable — the refusal releases
|
||||
the statement. (A leading comment rather than a CTE, so this also
|
||||
runs on 12.10, which has no CTEs.)"""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with pytest.raises(informix_db.NotSupportedError):
|
||||
cur.executemany(
|
||||
"/* batched? no */ SELECT FIRST 1 tabid FROM systables "
|
||||
"WHERE tabid <> ?",
|
||||
[(1,), (2,)],
|
||||
)
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None, "refusal leaked the statement"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_executemany_still_refuses_a_plain_select(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with pytest.raises(informix_db.NotSupportedError):
|
||||
cur.executemany(
|
||||
"SELECT FIRST 1 tabid FROM systables WHERE tabid <> ?",
|
||||
[(1,), (2,)],
|
||||
)
|
||||
189
tests/test_statement_release.py
Normal file
189
tests/test_statement_release.py
Normal file
@ -0,0 +1,189 @@
|
||||
"""Every door out of a statement has to release it on the way through.
|
||||
|
||||
A statement that fails is still allocated server-side. Skipping the
|
||||
RELEASE bricks the connection: the next PREPARE collides with the leaked
|
||||
one, and every subsequent call returns a nonsense error whose offset
|
||||
points back at the *failed* SQL rather than the new statement.
|
||||
|
||||
There are six exits from the execute paths, and the guard was added to
|
||||
them one at a time, each after a user hit it:
|
||||
|
||||
1. ``_execute_dml`` drain
|
||||
2. ``_execute_dml_with_params`` build
|
||||
3. ``_execute_dml_with_params`` drain
|
||||
4. ``executemany`` pipeline build/send
|
||||
5. ``_execute_select_with_params`` build
|
||||
6. ``_execute_select`` fetch loop
|
||||
|
||||
Two were still open, and both are ordinary to reach:
|
||||
|
||||
* **The parameterized-SELECT bind drain.** The build was guarded and the
|
||||
drain was not. Passing a string where the column is an INT gets a
|
||||
clean encode, so the rejection comes from the *server* (-1213, -415)
|
||||
during the bind drain — past the guard. A wrong-typed parameter is
|
||||
about as common as application mistakes get.
|
||||
|
||||
* **The scrollable-cursor open.** No guard at all, and the worst place
|
||||
to lack one: the GC-time finalizer is armed on the line *after* the
|
||||
drain, so a failure there left the statement allocated with no
|
||||
fallback of any kind.
|
||||
|
||||
The guard is now one helper rather than six hand-rolled copies, which
|
||||
also fixed a defect in the copies that had a cursor to close: CLOSE and
|
||||
RELEASE shared a single ``contextlib.suppress`` block, so a CLOSE that
|
||||
raised skipped the RELEASE — losing the half that actually matters.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _RELEASE_PDU, Cursor
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The helper — no server needed
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeConn:
|
||||
def __init__(self) -> None:
|
||||
self.sent: list[bytes] = []
|
||||
|
||||
def _send_pdu(self, pdu: bytes) -> None:
|
||||
self.sent.append(pdu)
|
||||
|
||||
|
||||
class _FakeCursor:
|
||||
"""Duck-types the four attributes ``_release_after_failure`` touches."""
|
||||
|
||||
def __init__(self, *, close_raises: bool = False) -> None:
|
||||
self._conn = _FakeConn()
|
||||
self._close_raises = close_raises
|
||||
|
||||
def _build_close_pdu(self) -> bytes:
|
||||
return b"CLOSE"
|
||||
|
||||
def _build_release_pdu(self) -> bytes:
|
||||
return _RELEASE_PDU
|
||||
|
||||
def _drain_to_eot(self) -> None:
|
||||
if self._close_raises and self._conn.sent[-1] == b"CLOSE":
|
||||
raise OSError("wire went away mid-close")
|
||||
|
||||
|
||||
def test_release_is_sent_even_when_close_fails() -> None:
|
||||
"""The regression in the hand-rolled copies. CLOSE and RELEASE shared
|
||||
one suppress block, so a failing CLOSE swallowed the RELEASE — and a
|
||||
lost cursor handle is a nuisance while a lost statement breaks the
|
||||
next call."""
|
||||
cur = _FakeCursor(close_raises=True)
|
||||
Cursor._release_after_failure(cur, close_cursor=True)
|
||||
assert _RELEASE_PDU in cur._conn.sent, (
|
||||
"a CLOSE that raises must not prevent the RELEASE"
|
||||
)
|
||||
|
||||
|
||||
def test_close_is_skipped_when_no_cursor_was_opened() -> None:
|
||||
cur = _FakeCursor()
|
||||
Cursor._release_after_failure(cur)
|
||||
assert cur._conn.sent == [_RELEASE_PDU]
|
||||
|
||||
|
||||
def test_cleanup_never_propagates() -> None:
|
||||
"""Cleanup runs inside an ``except``. If it raises, it replaces the
|
||||
real SQL error with a secondary failure from the cleanup path, which
|
||||
is strictly worse for the caller."""
|
||||
|
||||
class _Hostile(_FakeCursor):
|
||||
def _drain_to_eot(self) -> None:
|
||||
raise OSError("connection reset")
|
||||
|
||||
Cursor._release_after_failure(_Hostile(), close_cursor=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
# Bounded: a leaked statement used to manifest as a hang.
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.parametrize(
|
||||
("sql", "params"),
|
||||
[
|
||||
pytest.param(
|
||||
"SELECT tabid FROM systables WHERE tabid = ?",
|
||||
("not-an-int",),
|
||||
id="string-for-int",
|
||||
),
|
||||
pytest.param(
|
||||
"SELECT ?::INT FROM systables WHERE tabid = 1",
|
||||
(2**40,),
|
||||
id="out-of-range-int",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_server_rejected_bind_releases_statement(
|
||||
conn_params: ConnParams, sql: str, params: tuple
|
||||
) -> None:
|
||||
"""A parameterized SELECT whose bind the *server* rejects. The value
|
||||
encodes cleanly, so this lands past the build guard and in the drain
|
||||
that had none. Repeated because a leak only shows on the call after
|
||||
it — the first failure looks fine on its own."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
for i in range(4):
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute(sql, params)
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None, f"broken after {i + 1} binds"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_failed_scroll_open_releases_statement(conn_params: ConnParams) -> None:
|
||||
"""FOR UPDATE prepares cleanly and fails at OPEN with -526, which is
|
||||
exactly the branch that had no guard."""
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
for i in range(4):
|
||||
with pytest.raises(informix_db.Error):
|
||||
scroll.execute("SELECT tabid FROM systables FOR UPDATE")
|
||||
assert not scroll._server_cursor_open, (
|
||||
"a failed open must not leave the cursor marked live"
|
||||
)
|
||||
other = conn.cursor()
|
||||
other.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert other.fetchone() is not None, f"broken after {i + 1} opens"
|
||||
other.close()
|
||||
scroll.close()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_scrollable_cursor_still_works_after_a_failed_open(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The same cursor object must be reusable — a failed open is an
|
||||
ordinary error, not a terminal state for the cursor."""
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
with pytest.raises(informix_db.Error):
|
||||
scroll.execute("SELECT tabid FROM systables FOR UPDATE")
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
assert scroll.fetch_first() is not None
|
||||
scroll.close()
|
||||
355
tests/test_statement_slot.py
Normal file
355
tests/test_statement_slot.py
Normal file
@ -0,0 +1,355 @@
|
||||
"""One statement per session, and what happens when we forget that.
|
||||
|
||||
SQLI gives a session a single statement slot. ``SQ_CLOSE``,
|
||||
``SQ_RELEASE`` and ``SQ_SFETCH`` all act on whatever statement is
|
||||
current — none of them names one. Ordinary use never notices, because a
|
||||
non-scrollable cursor materializes its rows and releases the statement
|
||||
before returning, so the slot is free again by the time anyone looks.
|
||||
|
||||
A scrollable cursor is the exception: it holds the slot open on purpose.
|
||||
Two things then went wrong, and neither said so.
|
||||
|
||||
**Another statement on the same connection.** The server returns ``-285``
|
||||
for the new statement and *also* destroys the scrollable cursor — its
|
||||
next fetch comes back ``-267`` "the transaction has been rolled back,
|
||||
all locks released". Two unattributable errors from code that reads as
|
||||
completely ordinary: iterate a large result set, run a lookup partway
|
||||
through. It is now a ``ProgrammingError`` that says what happened.
|
||||
|
||||
**The deferred-cleanup queue drained at the wrong moment.** A cursor
|
||||
finalizer that can't get the wire lock hands its CLOSE/RELEASE to a
|
||||
queue for the next operation to flush. That queue was flushed before
|
||||
*every* PDU. But a finalizer enqueues precisely because another thread
|
||||
holds the lock, i.e. is mid-statement — so the flush landed inside that
|
||||
thread's own statement and released it. The victim saw ``-208`` when it
|
||||
happened before the first fetch, ``-267`` between fetch batches. The
|
||||
flush now happens only at a statement boundary, where the queued CLOSE
|
||||
addresses the orphan it was meant for.
|
||||
|
||||
A stale queue entry was fatal too: the finalizer enqueues, the cursor
|
||||
then gets closed properly, and the leftover CLOSE draws ``-267`` — an
|
||||
``OperationalError``, which is in ``WIRE_ERRORS``, which force-closed a
|
||||
perfectly healthy connection. Server-reported errors are now told apart
|
||||
from wire failures by the presence of a ``sqlcode``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _CLOSE_PDU, _RELEASE_PDU
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The conflict check — bookkeeping, no server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeScroll:
|
||||
def __init__(self, open_: bool = True) -> None:
|
||||
self._server_cursor_open = open_
|
||||
|
||||
|
||||
def _conflict_check(conn: informix_db.Connection, requester: object) -> None:
|
||||
conn._check_scroll_cursor_conflict(requester)
|
||||
|
||||
|
||||
def test_conflict_check_is_silent_with_no_scroll_cursor() -> None:
|
||||
conn = informix_db.Connection.__new__(informix_db.Connection)
|
||||
conn._open_scroll_cursor = None
|
||||
_conflict_check(conn, object())
|
||||
|
||||
|
||||
def test_conflict_check_forgets_a_collected_cursor() -> None:
|
||||
"""The ref is weak so an abandoned scrollable cursor can still be
|
||||
finalized. A dead ref means the slot is free."""
|
||||
import weakref
|
||||
|
||||
conn = informix_db.Connection.__new__(informix_db.Connection)
|
||||
victim = _FakeScroll()
|
||||
conn._open_scroll_cursor = weakref.ref(victim)
|
||||
del victim
|
||||
_conflict_check(conn, object())
|
||||
assert conn._open_scroll_cursor is None
|
||||
|
||||
|
||||
def test_conflict_check_lets_the_owner_through() -> None:
|
||||
"""Re-executing the *same* scrollable cursor is allowed — it closes
|
||||
its own server-side cursor first."""
|
||||
import weakref
|
||||
|
||||
conn = informix_db.Connection.__new__(informix_db.Connection)
|
||||
owner = _FakeScroll()
|
||||
conn._open_scroll_cursor = weakref.ref(owner)
|
||||
_conflict_check(conn, owner)
|
||||
|
||||
|
||||
def test_conflict_check_forgets_a_closed_cursor() -> None:
|
||||
import weakref
|
||||
|
||||
conn = informix_db.Connection.__new__(informix_db.Connection)
|
||||
done = _FakeScroll(open_=False)
|
||||
conn._open_scroll_cursor = weakref.ref(done)
|
||||
_conflict_check(conn, object())
|
||||
assert conn._open_scroll_cursor is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_second_statement_is_refused_while_scroll_cursor_is_open(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Used to be -285 for the new statement plus -267 for the scrollable
|
||||
cursor. Now it's one error that names the cause, and the scrollable
|
||||
cursor is untouched."""
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
first = scroll.fetch_first()
|
||||
assert first is not None
|
||||
|
||||
other = conn.cursor()
|
||||
with pytest.raises(informix_db.ProgrammingError, match="scrollable"):
|
||||
other.execute("SELECT COUNT(*) FROM systables")
|
||||
|
||||
assert scroll.fetch_absolute(1) is not None, (
|
||||
"the refused statement must not have disturbed the cursor"
|
||||
)
|
||||
scroll.close()
|
||||
other.execute("SELECT COUNT(*) FROM systables")
|
||||
assert other.fetchone() is not None, "slot must free on close"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_executemany_is_refused_too(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_slot (k INT)")
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
scroll.fetch_first()
|
||||
with pytest.raises(informix_db.ProgrammingError, match="scrollable"):
|
||||
cur.executemany("INSERT INTO t_slot VALUES (?)", [(1,), (2,)])
|
||||
scroll.close()
|
||||
cur.executemany("INSERT INTO t_slot VALUES (?)", [(1,), (2,)])
|
||||
cur.execute("SELECT COUNT(*) FROM t_slot")
|
||||
assert cur.fetchone() == (2,)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_scroll_cursor_can_be_re_executed(conn_params: ConnParams) -> None:
|
||||
"""The owner is the one caller allowed past the conflict check, and
|
||||
that is only safe because it closes its own server-side cursor
|
||||
first. Without that it collides with itself."""
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
for _ in range(4):
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
assert scroll.fetch_first() is not None
|
||||
scroll.close()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_abandoning_a_scroll_cursor_frees_the_slot(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Dropping the last reference must let the connection be used again
|
||||
— the finalizer closes the cursor and the weak ref goes dead."""
|
||||
import gc
|
||||
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
scroll.fetch_first()
|
||||
del scroll
|
||||
gc.collect()
|
||||
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT COUNT(*) FROM systables")
|
||||
assert cur.fetchone() is not None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deferred cleanup
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_queued_cleanup_does_not_land_inside_a_running_statement(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Exactly what a cross-thread finalizer does: it lost the wire lock,
|
||||
so it queued its CLOSE/RELEASE while another thread was mid-statement.
|
||||
The flush used to happen before that thread's very next PDU — its own
|
||||
CURNAME/NFETCH — releasing the statement out from under it."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_defer (k INT)")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_defer VALUES (?)", [(i,) for i in range(50)]
|
||||
)
|
||||
|
||||
original = cur._read_describe_response
|
||||
|
||||
def enqueue_between_prepare_and_fetch() -> None:
|
||||
result = original()
|
||||
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
|
||||
return result
|
||||
|
||||
cur._read_describe_response = enqueue_between_prepare_and_fetch
|
||||
try:
|
||||
cur.execute("SELECT k FROM t_defer ORDER BY k")
|
||||
assert len(cur.fetchall()) == 50, (
|
||||
"the queued cleanup released our own statement"
|
||||
)
|
||||
finally:
|
||||
cur._read_describe_response = original
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_stale_queued_cleanup_does_not_kill_the_connection(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""A queue entry goes stale whenever the cursor gets closed properly
|
||||
between enqueue and flush. The server answers the leftover CLOSE with
|
||||
-267, which is an OperationalError, which is in WIRE_ERRORS — so a
|
||||
stale entry used to force-close a healthy connection."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
cur.fetchall()
|
||||
|
||||
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None, "stale cleanup killed the connection"
|
||||
assert not conn.closed
|
||||
assert conn._pending_cleanup == [], "queue should have drained"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_connection_closes_cleanly_with_a_scroll_cursor_open(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
conn = _connect(conn_params)
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
scroll.fetch_first()
|
||||
with contextlib.suppress(Exception):
|
||||
conn.close()
|
||||
assert conn.closed
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The wire lock's blind spot
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_wire_lock_reports_its_own_thread() -> None:
|
||||
"""The whole point. ``RLock.acquire(blocking=False)`` returns True for
|
||||
the owning thread, so the finalizer's probe could not tell "nobody is
|
||||
using the wire" from "I am, right now, mid-statement"."""
|
||||
from informix_db.connections import _WireLock
|
||||
|
||||
lock = _WireLock()
|
||||
assert not lock.held_by_current_thread
|
||||
with lock:
|
||||
assert lock.held_by_current_thread
|
||||
assert lock.acquire(blocking=False), "must still be reentrant"
|
||||
lock.release()
|
||||
assert lock.held_by_current_thread, "still held at depth 1"
|
||||
assert not lock.held_by_current_thread
|
||||
|
||||
|
||||
def test_wire_lock_is_not_held_by_other_threads() -> None:
|
||||
import threading
|
||||
|
||||
from informix_db.connections import _WireLock
|
||||
|
||||
lock = _WireLock()
|
||||
seen: list[bool] = []
|
||||
entered = threading.Event()
|
||||
done = threading.Event()
|
||||
|
||||
def holder() -> None:
|
||||
with lock:
|
||||
entered.set()
|
||||
done.wait(5)
|
||||
|
||||
t = threading.Thread(target=holder)
|
||||
t.start()
|
||||
entered.wait(5)
|
||||
seen.append(lock.held_by_current_thread)
|
||||
done.set()
|
||||
t.join(5)
|
||||
assert seen == [False], "another thread's hold must not read as ours"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_finalizer_defers_when_gc_runs_on_the_locking_thread(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""GC fires on whatever thread allocated. When that is the thread
|
||||
holding the wire lock, the finalizer must write nothing — it used to
|
||||
acquire the RLock reentrantly and send CLOSE/RELEASE into the running
|
||||
statement, killing it with -208."""
|
||||
import gc
|
||||
|
||||
with _connect(conn_params) as conn:
|
||||
gc.disable()
|
||||
try:
|
||||
victim = conn.cursor(scrollable=True)
|
||||
victim.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
victim.fetch_first()
|
||||
# A reference cycle, so collection waits for gc rather than
|
||||
# happening at the drop. Cycles are ordinary in Python.
|
||||
cycle = [victim]
|
||||
cycle.append(cycle)
|
||||
del victim, cycle
|
||||
|
||||
writes: list[bytes] = []
|
||||
original_write = conn._sock.write_all
|
||||
conn._sock.write_all = lambda b: (
|
||||
writes.append(b),
|
||||
original_write(b),
|
||||
)[1]
|
||||
try:
|
||||
with conn._wire_lock: # stand in for "mid-statement"
|
||||
gc.collect()
|
||||
assert writes == [], (
|
||||
"finalizer wrote to the wire while a statement "
|
||||
"owned it"
|
||||
)
|
||||
assert conn._pending_cleanup, (
|
||||
"cleanup should have been deferred, not dropped"
|
||||
)
|
||||
finally:
|
||||
conn._sock.write_all = original_write
|
||||
finally:
|
||||
gc.enable()
|
||||
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT COUNT(*) FROM systables")
|
||||
assert cur.fetchone() is not None
|
||||
333
tests/test_tls_traffic.py
Normal file
333
tests/test_tls_traffic.py
Normal file
@ -0,0 +1,333 @@
|
||||
"""End-to-end SQLI traffic over a real TLS socket.
|
||||
|
||||
``tests/test_tls.py`` covers the handshake. This covers what happens
|
||||
*after* it: every byte of real SQLI traffic crossing genuine TLS records,
|
||||
through the same codecs, reader, and cursor machinery the plain-socket
|
||||
suite exercises.
|
||||
|
||||
That distinction matters because ``SSLSocket.recv`` is not
|
||||
``socket.recv``. It returns at most one TLS record's worth of plaintext
|
||||
however much you ask for, it can return fewer bytes than are available,
|
||||
and plaintext buffered inside the SSL object is invisible to the OS. The
|
||||
Phase 39 buffered reader asks for up to 64 KB per call and loops until
|
||||
satisfied — that loop is the thing which has to be right, and nothing in
|
||||
the plain-socket suite puts the same pressure on it.
|
||||
|
||||
**Scope.** A TLS-terminating proxy in front of the plain SQLI listener
|
||||
supplies the TLS half. This tests the driver's TLS path, which is the
|
||||
half we own. It does *not* test IBM's server-side TLS listener: Informix
|
||||
15 wants a PKCS#12 keystore whose stash the developer-edition image
|
||||
rejects (``GSK_ERROR_BAD_KEYFILE_PASSWORD``), and that side is IBM's
|
||||
code. Anything below ``ssl.wrap_socket`` is identical either way.
|
||||
|
||||
Skipped when ``openssl`` isn't on PATH — the proxy needs a certificate.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import datetime
|
||||
import decimal
|
||||
import select
|
||||
import shutil
|
||||
import socket
|
||||
import ssl
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TLS-terminating proxy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _TlsProxy:
|
||||
"""Accepts TLS, relays plaintext to the real Informix listener."""
|
||||
|
||||
def __init__(self, backend: tuple[str, int]) -> None:
|
||||
self.backend = backend
|
||||
self.tmpdir = tempfile.mkdtemp(prefix="ifx-tls-test-")
|
||||
self.cert = str(Path(self.tmpdir) / "cert.pem")
|
||||
key = str(Path(self.tmpdir) / "key.pem")
|
||||
subprocess.run(
|
||||
["openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes",
|
||||
"-keyout", key, "-out", self.cert, "-days", "1",
|
||||
"-subj", "/CN=127.0.0.1"],
|
||||
check=True, capture_output=True,
|
||||
)
|
||||
self._ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
self._ctx.load_cert_chain(self.cert, key)
|
||||
self._sock = socket.socket()
|
||||
self._sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
self._sock.bind(("127.0.0.1", 0))
|
||||
self._sock.listen(64)
|
||||
self.port: int = self._sock.getsockname()[1]
|
||||
self._stop = threading.Event()
|
||||
threading.Thread(target=self._serve, daemon=True).start()
|
||||
|
||||
def _serve(self) -> None:
|
||||
self._sock.settimeout(0.5)
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
raw, _ = self._sock.accept()
|
||||
except (TimeoutError, OSError):
|
||||
continue
|
||||
threading.Thread(
|
||||
target=self._handle, args=(raw,), daemon=True
|
||||
).start()
|
||||
|
||||
def _handle(self, raw: socket.socket) -> None:
|
||||
try:
|
||||
client = self._ctx.wrap_socket(raw, server_side=True)
|
||||
except (ssl.SSLError, OSError):
|
||||
with contextlib.suppress(OSError):
|
||||
raw.close()
|
||||
return
|
||||
try:
|
||||
upstream = socket.create_connection(self.backend, timeout=20)
|
||||
except OSError:
|
||||
with contextlib.suppress(OSError):
|
||||
client.close()
|
||||
return
|
||||
try:
|
||||
self._pump(client, upstream)
|
||||
finally:
|
||||
for s in (client, upstream):
|
||||
with contextlib.suppress(OSError):
|
||||
s.close()
|
||||
|
||||
@staticmethod
|
||||
def _pump(a: socket.socket, b: socket.socket) -> None:
|
||||
# Drain the SSL object's own buffer before consulting select():
|
||||
# select only sees the OS socket, so already-decrypted bytes
|
||||
# sitting inside the SSL object would stall the relay.
|
||||
socks = [a, b]
|
||||
while True:
|
||||
pending = [s for s in socks
|
||||
if isinstance(s, ssl.SSLSocket) and s.pending()]
|
||||
ready = pending or select.select(socks, [], [], 1.0)[0]
|
||||
for s in ready:
|
||||
other = b if s is a else a
|
||||
try:
|
||||
data = s.recv(65536)
|
||||
except (ssl.SSLError, OSError):
|
||||
return
|
||||
if not data:
|
||||
return
|
||||
try:
|
||||
other.sendall(data)
|
||||
except OSError:
|
||||
return
|
||||
|
||||
def close(self) -> None:
|
||||
self._stop.set()
|
||||
with contextlib.suppress(OSError):
|
||||
self._sock.close()
|
||||
shutil.rmtree(self.tmpdir, ignore_errors=True)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def tls_proxy(conn_params: ConnParams):
|
||||
if shutil.which("openssl") is None:
|
||||
pytest.skip("openssl not on PATH; needed to generate a test cert")
|
||||
proxy = _TlsProxy((conn_params.host, conn_params.port))
|
||||
try:
|
||||
yield proxy
|
||||
finally:
|
||||
proxy.close()
|
||||
|
||||
|
||||
def _client_ctx(proxy: _TlsProxy) -> ssl.SSLContext:
|
||||
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
|
||||
ctx.load_verify_locations(proxy.cert)
|
||||
ctx.check_hostname = False # self-signed CN=127.0.0.1
|
||||
return ctx
|
||||
|
||||
|
||||
def _connect(proxy: _TlsProxy, conn_params: ConnParams, **kw):
|
||||
return informix_db.connect(
|
||||
host="127.0.0.1",
|
||||
port=proxy.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=20.0,
|
||||
read_timeout=45.0,
|
||||
tls=_client_ctx(proxy),
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Traffic
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_query_over_tls(tls_proxy, conn_params: ConnParams) -> None:
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT FIRST 3 tabname FROM systables ORDER BY tabid")
|
||||
assert len(cur.fetchall()) == 3
|
||||
assert conn.server_version, "server_version empty over TLS"
|
||||
|
||||
|
||||
def test_type_round_trip_over_tls(tls_proxy, conn_params: ConnParams) -> None:
|
||||
"""The types that gave us framing bugs, every byte through TLS."""
|
||||
ts = datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
|
||||
row = (
|
||||
2001, "PackageRoot", None, "/content/package", 77,
|
||||
decimal.Decimal("1234567890123456"), True, ts, "nch",
|
||||
)
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_tls_types ("
|
||||
" a INT8 NOT NULL, k LVARCHAR(512), d LVARCHAR(512),"
|
||||
" v LVARCHAR(1024), n INT8, dec16 DECIMAL(16), b BOOLEAN,"
|
||||
" t DATETIME YEAR TO FRACTION(5), c NCHAR(6))"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_tls_types VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", row
|
||||
)
|
||||
cur.execute("SELECT a, k, d, v, n, dec16, b, t, c FROM t_tls_types")
|
||||
assert cur.fetchone() == row
|
||||
|
||||
|
||||
@pytest.mark.parametrize("size", [1, 4096, 16383, 16384, 16385, 32000])
|
||||
def test_payload_spans_tls_record_boundary(
|
||||
tls_proxy, conn_params: ConnParams, size: int
|
||||
) -> None:
|
||||
"""A TLS record holds ~16 KB, so these straddle the boundary where a
|
||||
single ``recv`` stops being enough."""
|
||||
payload = "x" * size
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_tls_big (k INT, v LVARCHAR(32000))")
|
||||
cur.execute("INSERT INTO t_tls_big VALUES (?, ?)", (1, payload))
|
||||
cur.execute("SELECT v, k FROM t_tls_big")
|
||||
assert cur.fetchone() == (payload, 1)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [500, 5000])
|
||||
def test_bulk_fetch_over_tls(
|
||||
tls_proxy, conn_params: ConnParams, n: int
|
||||
) -> None:
|
||||
"""Total bytes far beyond both one TLS record and the reader's 64 KB
|
||||
recv budget, so the top-up loop runs many times."""
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_tls_bulk (k INT, v VARCHAR(240))")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_tls_bulk VALUES (?, ?)",
|
||||
[(i, f"row{i}-" + "y" * 200) for i in range(n)],
|
||||
)
|
||||
cur.execute("SELECT k, v FROM t_tls_bulk ORDER BY k")
|
||||
rows = cur.fetchall()
|
||||
assert len(rows) == n
|
||||
assert rows[0][0] == 0
|
||||
assert rows[-1][0] == n - 1
|
||||
|
||||
|
||||
def test_error_recovery_over_tls(tls_proxy, conn_params: ConnParams) -> None:
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_tls_dup (k INT PRIMARY KEY)")
|
||||
cur.execute("INSERT INTO t_tls_dup VALUES (1)")
|
||||
for _ in range(4):
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("INSERT INTO t_tls_dup VALUES (1)")
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("SELECT * FROM t_tls_no_such_table_xyz")
|
||||
cur.execute("SELECT k FROM t_tls_dup")
|
||||
assert cur.fetchall() == [(1,)]
|
||||
|
||||
|
||||
def test_concurrent_tls_connections(tls_proxy, conn_params: ConnParams) -> None:
|
||||
"""Separate TLS sessions must not cross data."""
|
||||
failures: list[str] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def worker(tid: int) -> None:
|
||||
tag = f"t{tid}-{'z' * 12}"
|
||||
try:
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
for r in range(6):
|
||||
token = tid * 1000 + r
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 ?::INT, ?::VARCHAR(24) FROM systables",
|
||||
(token, tag),
|
||||
)
|
||||
if cur.fetchone() != (token, tag):
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: crossed data")
|
||||
return
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(i,)) for i in range(4)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
assert not failures, failures
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Negative cases — misuse must fail cleanly, never hang or downgrade
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_tls_client_against_plaintext_port_fails(
|
||||
tls_proxy, conn_params: ConnParams
|
||||
) -> None:
|
||||
with pytest.raises(informix_db.Error):
|
||||
informix_db.connect(
|
||||
host=conn_params.host, port=conn_params.port,
|
||||
user=conn_params.user, password=conn_params.password,
|
||||
database=conn_params.database, server=conn_params.server,
|
||||
connect_timeout=10.0, read_timeout=10.0,
|
||||
tls=_client_ctx(tls_proxy),
|
||||
)
|
||||
|
||||
|
||||
def test_plaintext_client_against_tls_port_fails(
|
||||
tls_proxy, conn_params: ConnParams
|
||||
) -> None:
|
||||
"""Must raise rather than hang — a stalled handshake is the failure
|
||||
mode that looks like a dead application."""
|
||||
with pytest.raises(informix_db.Error):
|
||||
informix_db.connect(
|
||||
host="127.0.0.1", port=tls_proxy.port,
|
||||
user=conn_params.user, password=conn_params.password,
|
||||
database=conn_params.database, server=conn_params.server,
|
||||
connect_timeout=10.0, read_timeout=10.0,
|
||||
)
|
||||
|
||||
|
||||
def test_verification_rejects_self_signed(
|
||||
tls_proxy, conn_params: ConnParams
|
||||
) -> None:
|
||||
"""`tls=True` disables verification by design; a caller-supplied
|
||||
verifying context must still reject an untrusted cert."""
|
||||
strict = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
|
||||
strict.check_hostname = True
|
||||
strict.verify_mode = ssl.CERT_REQUIRED
|
||||
with pytest.raises(informix_db.Error):
|
||||
informix_db.connect(
|
||||
host="127.0.0.1", port=tls_proxy.port,
|
||||
user=conn_params.user, password=conn_params.password,
|
||||
database=conn_params.database, server=conn_params.server,
|
||||
connect_timeout=10.0, read_timeout=10.0, tls=strict,
|
||||
)
|
||||
191
tests/test_transaction_state.py
Normal file
191
tests/test_transaction_state.py
Normal file
@ -0,0 +1,191 @@
|
||||
"""Transaction control run as SQL, and the flag that didn't notice.
|
||||
|
||||
``Connection._in_transaction`` decides whether ``commit()`` and
|
||||
``rollback()`` send anything at all, and the pool reads it to decide
|
||||
whether a returned connection needs cleaning up. It was maintained
|
||||
solely by the driver's own implicit ``SQ_BEGIN``, so a caller who wrote
|
||||
``cursor.execute("BEGIN WORK")`` — an entirely reasonable thing to
|
||||
write — walked straight past it.
|
||||
|
||||
With autocommit on, nothing stopped that statement reaching the server.
|
||||
A transaction opened, the flag stayed False, and ``rollback()`` returned
|
||||
successfully having sent nothing. The rows it was asked to discard were
|
||||
still there. The connection then went back to the pool holding an open
|
||||
transaction and its locks, because the pool's cleanup is guarded by the
|
||||
same flag.
|
||||
|
||||
With autocommit off it failed instead, and for a sillier reason: the
|
||||
driver's implicit ``SQ_BEGIN`` fired first, so the caller's ``BEGIN
|
||||
WORK`` got ``-535``, "already in transaction". The driver and the user
|
||||
competing to open the same transaction, and the user losing.
|
||||
|
||||
The server labels these statements: type 34 for BEGIN, 35 for COMMIT, 36
|
||||
for ROLLBACK, with and without the ``WORK`` keyword. JDBC reads the same
|
||||
three values off the describe and calls ``setTxBeginState`` /
|
||||
``setTxEndState``. It also calls ``initiateTransaction`` *after* the
|
||||
describe rather than before, which is what makes the skip possible —
|
||||
until the describe lands you can't know the statement is transaction
|
||||
control.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _TX_CONTROL_TYPES
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
def test_transaction_control_types_are_what_the_server_says(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""Pin the three constants against a live server rather than trusting
|
||||
the decompiled source. Both spellings must map to the same type."""
|
||||
with _connect(logged_db_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
seen = {}
|
||||
for sql in ("BEGIN WORK", "COMMIT WORK", "ROLLBACK WORK",
|
||||
"BEGIN", "COMMIT", "ROLLBACK"):
|
||||
with conn._wire_lock:
|
||||
conn._send_pdu(
|
||||
cur._build_prepare_pdu(sql, num_qmarks=0),
|
||||
statement_boundary=True,
|
||||
)
|
||||
cur._read_describe_response()
|
||||
cur._release_after_failure()
|
||||
seen[sql] = cur._statement_type
|
||||
assert seen["BEGIN WORK"] == seen["BEGIN"] == 34
|
||||
assert seen["COMMIT WORK"] == seen["COMMIT"] == 35
|
||||
assert seen["ROLLBACK WORK"] == seen["ROLLBACK"] == 36
|
||||
assert set(seen.values()) == _TX_CONTROL_TYPES
|
||||
|
||||
|
||||
def test_rollback_after_sql_begin_actually_rolls_back(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""The data-loss case. rollback() reported success and sent nothing,
|
||||
so the row it was asked to discard survived."""
|
||||
with _connect(logged_db_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate")
|
||||
cur.execute("CREATE TABLE t_txstate (k INT)")
|
||||
try:
|
||||
cur.execute("BEGIN WORK")
|
||||
assert conn._in_transaction, "SQL BEGIN must set the flag"
|
||||
cur.execute("INSERT INTO t_txstate VALUES (1)")
|
||||
conn.rollback()
|
||||
cur.execute("SELECT COUNT(*) FROM t_txstate")
|
||||
assert cur.fetchone() == (0,), "rollback() was a silent no-op"
|
||||
assert not conn._in_transaction
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate")
|
||||
|
||||
|
||||
def test_sql_commit_clears_the_flag(logged_db_params: ConnParams) -> None:
|
||||
"""The mirror image: with the flag stuck True after a SQL COMMIT, the
|
||||
next rollback() would send SQ_RBWORK with no transaction open and
|
||||
draw -255."""
|
||||
with _connect(logged_db_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate2")
|
||||
cur.execute("CREATE TABLE t_txstate2 (k INT)")
|
||||
try:
|
||||
cur.execute("BEGIN WORK")
|
||||
cur.execute("INSERT INTO t_txstate2 VALUES (1)")
|
||||
cur.execute("COMMIT WORK")
|
||||
assert not conn._in_transaction, "SQL COMMIT must clear the flag"
|
||||
conn.rollback() # must be a no-op, not a -255
|
||||
cur.execute("SELECT COUNT(*) FROM t_txstate2")
|
||||
assert cur.fetchone() == (1,), "the committed row must survive"
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate2")
|
||||
|
||||
|
||||
def test_sql_begin_does_not_collide_with_the_implicit_one(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""Non-autocommit. The driver's implicit SQ_BEGIN used to fire first
|
||||
and the caller's BEGIN WORK then got -535."""
|
||||
with _connect(logged_db_params, autocommit=False) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate3")
|
||||
conn.commit()
|
||||
cur.execute("CREATE TABLE t_txstate3 (k INT)")
|
||||
conn.commit()
|
||||
try:
|
||||
cur.execute("BEGIN WORK")
|
||||
cur.execute("INSERT INTO t_txstate3 VALUES (1)")
|
||||
conn.rollback()
|
||||
cur.execute("SELECT COUNT(*) FROM t_txstate3")
|
||||
assert cur.fetchone() == (0,)
|
||||
conn.commit()
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate3")
|
||||
conn.commit()
|
||||
|
||||
|
||||
def test_ordinary_dml_still_opens_a_transaction(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""_ensure_transaction moved from before the PREPARE to after the
|
||||
describe. It still has to fire for everything that isn't transaction
|
||||
control, or non-autocommit DML runs outside a transaction."""
|
||||
with _connect(logged_db_params, autocommit=False) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate4")
|
||||
conn.commit()
|
||||
cur.execute("CREATE TABLE t_txstate4 (k INT)")
|
||||
conn.commit()
|
||||
try:
|
||||
assert not conn._in_transaction
|
||||
cur.execute("INSERT INTO t_txstate4 VALUES (1)")
|
||||
assert conn._in_transaction, "DML must open a transaction"
|
||||
conn.rollback()
|
||||
cur.execute("SELECT COUNT(*) FROM t_txstate4")
|
||||
assert cur.fetchone() == (0,)
|
||||
conn.commit()
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate4")
|
||||
conn.commit()
|
||||
|
||||
|
||||
def test_a_failed_transaction_statement_does_not_move_the_flag(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""The sync runs on the success path only. A COMMIT that the server
|
||||
rejects has not ended anything."""
|
||||
with _connect(logged_db_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
assert not conn._in_transaction
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("COMMIT WORK") # -255, nothing to commit
|
||||
assert not conn._in_transaction
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None
|
||||
342
tests/test_type_matrix.py
Normal file
342
tests/test_type_matrix.py
Normal file
@ -0,0 +1,342 @@
|
||||
"""Type-matrix regression tests, and the fuzzer that found them.
|
||||
|
||||
Six framing bugs reached users before this file existed. Every one shared
|
||||
a shape: the value under test decoded fine on its own, and quietly
|
||||
corrupted whatever came *after* it. They were missed not because branches
|
||||
went unexercised but because the fixtures never supplied data that took
|
||||
them — the LVARCHAR fixture was ``'lv value'``, eight characters, even,
|
||||
never NULL, so neither of its two broken branches ever ran.
|
||||
|
||||
Three principles follow, and every test here applies them:
|
||||
|
||||
1. **Always put a column after the value under test.** A trailing column
|
||||
can be mis-sized with no visible effect. The sentinel is the detector.
|
||||
2. **Vary the data, not just the type.** Odd vs even length, empty vs
|
||||
NULL, min/max, negative, unscaled vs scaled.
|
||||
3. **Use an oracle our codecs don't share.** A Python round-trip cannot
|
||||
catch a symmetric encode/decode bug — our DATETIME encoder wrote zeros
|
||||
and our decoder read them back in perfect agreement. Asking the server
|
||||
to render the stored value as text breaks that symmetry.
|
||||
|
||||
The three bugs in the first section were found by the fuzzer at the
|
||||
bottom of this file, not by a person.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import decimal
|
||||
import random
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
SENTINEL = 424242
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
# Bounded so a wire desync fails the test instead of hanging it.
|
||||
read_timeout=20.0,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. BOOLEAN could not be used as a bind parameter at all
|
||||
# ---------------------------------------------------------------------------
|
||||
# _encode_bool emitted type code 45, which the server does not accept as a
|
||||
# bind type: it simply stopped responding. Any execute() with a bool
|
||||
# parameter hung until the read timeout, or forever without one. Type 41
|
||||
# (how BOOLEAN is *described* in results) hangs identically — descriptor
|
||||
# type and bind type are not interchangeable. Binding 't'/'f' as CHAR and
|
||||
# letting the server cast is what works.
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [True, False, None])
|
||||
def test_boolean_bind_round_trip(
|
||||
conn_params: ConnParams, value: bool | None
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_bool_bind (b BOOLEAN, s INT)")
|
||||
cur.execute("INSERT INTO t_bool_bind VALUES (?, ?)", (value, SENTINEL))
|
||||
cur.execute("SELECT b, s FROM t_bool_bind")
|
||||
assert cur.fetchone() == (value, SENTINEL)
|
||||
|
||||
|
||||
def test_boolean_bind_in_where_clause(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_bool_where (b BOOLEAN, k INT)")
|
||||
cur.execute("INSERT INTO t_bool_where VALUES (?, ?)", (True, 1))
|
||||
cur.execute("INSERT INTO t_bool_where VALUES (?, ?)", (False, 2))
|
||||
cur.execute("SELECT k FROM t_bool_where WHERE b = ?", (True,))
|
||||
assert cur.fetchall() == [(1,)]
|
||||
cur.execute("SELECT k FROM t_bool_where WHERE b = ?", (False,))
|
||||
assert cur.fetchall() == [(2,)]
|
||||
|
||||
|
||||
def test_boolean_bind_stores_real_boolean(conn_params: ConnParams) -> None:
|
||||
"""Independent oracle: the server must agree it stored a BOOLEAN, not
|
||||
the string we transported it as."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_bool_oracle (b BOOLEAN)")
|
||||
cur.execute("INSERT INTO t_bool_oracle VALUES (?)", (True,))
|
||||
cur.execute("SELECT b::CHAR(1) FROM t_bool_oracle")
|
||||
assert cur.fetchone() == ("t",)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Unscaled DECIMAL was read one byte short
|
||||
# ---------------------------------------------------------------------------
|
||||
# Width is ((precision) + (scale & 1) + 3) // 2, per IfxColumnInfo's
|
||||
# adjustedColumnLength. We omitted the `scale & 1` term, which only
|
||||
# matters when precision is even and scale is odd — and an unscaled
|
||||
# DECIMAL(p) reports scale 255. DECIMAL(16) is 10 bytes, not 9.
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("ddl", "value"),
|
||||
[
|
||||
("DECIMAL(16)", decimal.Decimal("1234567890123456")),
|
||||
("DECIMAL(16)", decimal.Decimal("-1")),
|
||||
("DECIMAL(16)", None),
|
||||
("DECIMAL(4)", decimal.Decimal("1234")),
|
||||
("DECIMAL(20)", decimal.Decimal("12345678901234567890")),
|
||||
("DECIMAL(8,2)", decimal.Decimal("12345.67")),
|
||||
("DECIMAL(10,4)", decimal.Decimal("123456.7890")),
|
||||
("DECIMAL(5,0)", decimal.Decimal("12345")),
|
||||
("DECIMAL(1,0)", decimal.Decimal("7")),
|
||||
("DECIMAL(32,10)", decimal.Decimal("1234567890.0123456789")),
|
||||
("MONEY(10,2)", decimal.Decimal("1234.56")),
|
||||
("MONEY(10,2)", decimal.Decimal("-1234.56")),
|
||||
],
|
||||
)
|
||||
def test_decimal_width_does_not_shift_next_column(
|
||||
conn_params: ConnParams, ddl: str, value: decimal.Decimal | None
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(f"CREATE TEMP TABLE t_dec_w (d {ddl}, s INT)")
|
||||
cur.execute("INSERT INTO t_dec_w VALUES (?, ?)", (value, SENTINEL))
|
||||
cur.execute("SELECT d, s FROM t_dec_w")
|
||||
got = cur.fetchone()
|
||||
assert got[1] == SENTINEL, f"{ddl} mis-sized; sentinel came back {got[1]}"
|
||||
if value is None:
|
||||
assert got[0] is None
|
||||
else:
|
||||
assert got[0] == value
|
||||
|
||||
|
||||
def test_packed_width_matches_the_reference_formula() -> None:
|
||||
"""Unit-level guard on the formula itself, including the term that was
|
||||
missing. Values verified against the wire."""
|
||||
from informix_db._resultset import _packed_width
|
||||
|
||||
assert _packed_width((8 << 8) | 2) == 5 # DECIMAL(8,2)
|
||||
assert _packed_width((16 << 8) | 255) == 10 # DECIMAL(16), unscaled
|
||||
assert _packed_width((10 << 8) | 4) == 6 # DECIMAL(10,4)
|
||||
assert _packed_width((5 << 8) | 0) == 4 # DECIMAL(5,0)
|
||||
assert _packed_width((32 << 8) | 10) == 17 # DECIMAL(32,10)
|
||||
assert _packed_width((1 << 8) | 0) == 2 # DECIMAL(1,0)
|
||||
assert _packed_width((14 << 8) | 10) == 8 # DATETIME YEAR TO SECOND
|
||||
assert _packed_width((19 << 8) | 15) == 11 # DATETIME .. FRACTION(5)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. NULL CHAR / NCHAR came back as an empty string
|
||||
# ---------------------------------------------------------------------------
|
||||
# The NULL marker is a leading 0x00; an empty CHAR is all spaces. They are
|
||||
# distinguishable on the wire, so conflating them made `IS NULL` disagree
|
||||
# with what the driver returned.
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ddl", ["CHAR(6)", "NCHAR(6)"])
|
||||
def test_null_char_is_none_not_empty_string(
|
||||
conn_params: ConnParams, ddl: str
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(f"CREATE TEMP TABLE t_char_null (k INT, c {ddl})")
|
||||
cur.execute("INSERT INTO t_char_null VALUES (?, ?)", (1, None))
|
||||
cur.execute("INSERT INTO t_char_null VALUES (?, ?)", (2, ""))
|
||||
cur.execute("INSERT INTO t_char_null VALUES (?, ?)", (3, "ab"))
|
||||
cur.execute("SELECT k, c FROM t_char_null ORDER BY k")
|
||||
assert cur.fetchall() == [(1, None), (2, ""), (3, "ab")]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ddl", ["CHAR(6)", "NCHAR(6)"])
|
||||
def test_null_char_agrees_with_is_null(
|
||||
conn_params: ConnParams, ddl: str
|
||||
) -> None:
|
||||
"""The server's own view is the oracle: what it calls NULL, we must
|
||||
return as None."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(f"CREATE TEMP TABLE t_char_isnull (k INT, c {ddl})")
|
||||
cur.execute("INSERT INTO t_char_isnull VALUES (?, ?)", (1, None))
|
||||
cur.execute("INSERT INTO t_char_isnull VALUES (?, ?)", (2, ""))
|
||||
cur.execute("SELECT k FROM t_char_isnull WHERE c IS NULL")
|
||||
server_nulls = {r[0] for r in cur.fetchall()}
|
||||
cur.execute("SELECT k, c FROM t_char_isnull ORDER BY k")
|
||||
driver_nulls = {k for k, c in cur.fetchall() if c is None}
|
||||
assert driver_nulls == server_nulls
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. The fuzzer itself, with fixed seeds so it is reproducible in CI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_ODD = "abcdefghijk" # 11
|
||||
_EVEN = "abcdefghij" # 10
|
||||
|
||||
# (name, DDL, edge values). Values are chosen to flip framing branches.
|
||||
_CORPUS: list[tuple[str, str, list]] = [
|
||||
("smallint", "SMALLINT", [0, 1, -1, 32767, -32767, None]),
|
||||
("int", "INT", [0, -1, 2147483647, None]),
|
||||
("int8", "INT8", [0, 1, -1, 123456789012, -123456789012, None]),
|
||||
("bigint", "BIGINT", [0, -1, 2**62, None]),
|
||||
("float", "FLOAT", [0.0, 3.141592653589793, -1.5, None]),
|
||||
("smallfloat", "SMALLFLOAT", [0.0, 2.5, None]),
|
||||
("dec_scaled", "DECIMAL(8,2)", [
|
||||
decimal.Decimal("12345.67"), decimal.Decimal("-98.76"), None,
|
||||
]),
|
||||
("dec_float", "DECIMAL(16)", [
|
||||
decimal.Decimal("1234567890123456"), decimal.Decimal("-1"), None,
|
||||
]),
|
||||
("money", "MONEY(10,2)", [decimal.Decimal("1234.56"), None]),
|
||||
("char", "CHAR(12)", ["", "a", _ODD, _EVEN, None]),
|
||||
("varchar", "VARCHAR(64)", ["", "a", _ODD, _EVEN, None]),
|
||||
("nchar", "NCHAR(12)", ["", "a", _ODD, None]),
|
||||
("nvarchar", "NVARCHAR(64)", ["", "a", _ODD, None]),
|
||||
("lvarchar", "LVARCHAR(1024)", [
|
||||
"", "a", "ab", _ODD, _EVEN, "x" * 255, "y" * 256, None,
|
||||
]),
|
||||
("date", "DATE", [datetime.date(1899, 12, 31), datetime.date(2026, 8, 31), None]),
|
||||
("dt_sec", "DATETIME YEAR TO SECOND", [
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15), None,
|
||||
]),
|
||||
("dt_frac", "DATETIME YEAR TO FRACTION(5)", [
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15),
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
|
||||
None,
|
||||
]),
|
||||
("interval", "INTERVAL DAY(5) TO SECOND", [
|
||||
datetime.timedelta(days=10, hours=4, minutes=30, seconds=15),
|
||||
datetime.timedelta(0),
|
||||
None,
|
||||
]),
|
||||
("boolean", "BOOLEAN", [True, False, None]),
|
||||
]
|
||||
|
||||
_BY_NAME = {c[0]: c for c in _CORPUS}
|
||||
|
||||
|
||||
def _check_row(cur, names: list[str], values: list) -> None:
|
||||
"""Create, insert, then read back from several projections. The
|
||||
sentinel column trails everything so any mis-sizing surfaces."""
|
||||
cols = [f"c{i}" for i in range(len(names))]
|
||||
ddl = ", ".join(f"{c} {_BY_NAME[n][1]}" for c, n in zip(cols, names, strict=True))
|
||||
cur.execute(f"CREATE TEMP TABLE t_fuzz ({ddl}, sentinel INT)")
|
||||
try:
|
||||
marks = ", ".join(["?"] * (len(cols) + 1))
|
||||
cur.execute(f"INSERT INTO t_fuzz VALUES ({marks})", (*values, SENTINEL))
|
||||
|
||||
expected = dict(zip(cols, values, strict=True))
|
||||
expected["sentinel"] = SENTINEL
|
||||
every = [*cols, "sentinel"]
|
||||
|
||||
cur.execute(f"SELECT {', '.join(every)} FROM t_fuzz")
|
||||
assert dict(zip(every, cur.fetchone(), strict=True)) == expected, (
|
||||
f"straight projection: {list(zip(names, values, strict=True))}"
|
||||
)
|
||||
|
||||
for shift in range(1, min(len(every), 4)):
|
||||
order = every[shift:] + every[:shift]
|
||||
cur.execute(f"SELECT {', '.join(order)} FROM t_fuzz")
|
||||
got = dict(zip(order, cur.fetchone(), strict=True))
|
||||
assert got == expected, (
|
||||
f"rotation {shift}: {list(zip(names, values, strict=True))}"
|
||||
)
|
||||
|
||||
cur.execute("SELECT * FROM t_fuzz")
|
||||
got_names = [d[0] for d in cur.description]
|
||||
assert dict(zip(got_names, cur.fetchone(), strict=True)) == expected, (
|
||||
f"SELECT *: {list(zip(names, values, strict=True))}"
|
||||
)
|
||||
finally:
|
||||
cur.execute("DROP TABLE t_fuzz")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("type_name", [c[0] for c in _CORPUS])
|
||||
def test_every_edge_value_with_a_trailing_sentinel(
|
||||
conn_params: ConnParams, type_name: str
|
||||
) -> None:
|
||||
"""Exhaustive over the corpus: each type, each edge value, always with
|
||||
a column behind it."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
for value in _BY_NAME[type_name][2]:
|
||||
_check_row(cur, [type_name], [value])
|
||||
|
||||
|
||||
@pytest.mark.parametrize("seed", [1, 2, 3])
|
||||
def test_random_mixed_rows(conn_params: ConnParams, seed: int) -> None:
|
||||
"""Random multi-type rows. Seeded so a failure is reproducible; the
|
||||
point is combinations no one thought to write by hand."""
|
||||
rng = random.Random(seed)
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
for _ in range(25):
|
||||
names = [rng.choice(_CORPUS)[0] for _ in range(5)]
|
||||
values = [rng.choice(_BY_NAME[n][2]) for n in names]
|
||||
_check_row(cur, names, values)
|
||||
|
||||
|
||||
def test_server_agrees_with_what_we_bound(conn_params: ConnParams) -> None:
|
||||
"""The independent oracle. Compare against the server's own rendering
|
||||
rather than our own decoder, so a symmetric encode/decode bug cannot
|
||||
hide — which is exactly how DATETIME lost its fractions."""
|
||||
checks: list[tuple[str, object, str]] = [
|
||||
("INT8", 123456789012, "123456789012"),
|
||||
("INT8", -123456789012, "-123456789012"),
|
||||
("BIGINT", 2**62, str(2**62)),
|
||||
("VARCHAR(64)", "PackageRoot", "PackageRoot"),
|
||||
("LVARCHAR(64)", "PackageRoot", "PackageRoot"),
|
||||
("NVARCHAR(64)", "odd", "odd"),
|
||||
("BOOLEAN", True, "t"),
|
||||
("BOOLEAN", False, "f"),
|
||||
("DATETIME YEAR TO FRACTION(5)",
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
|
||||
"2026-08-31 12:30:15.12000"),
|
||||
("DATETIME YEAR TO SECOND",
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15),
|
||||
"2026-08-31 12:30:15"),
|
||||
("DATE", datetime.date(2026, 8, 31), "2026-08-31"),
|
||||
]
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
for ddl, value, server_repr in checks:
|
||||
cur.execute(f"CREATE TEMP TABLE t_oracle (v {ddl})")
|
||||
try:
|
||||
cur.execute("INSERT INTO t_oracle VALUES (?)", (value,))
|
||||
cur.execute("SELECT v::LVARCHAR FROM t_oracle")
|
||||
(text,) = cur.fetchone()
|
||||
assert text.strip() == server_repr, (
|
||||
f"{ddl}: bound {value!r}, server stored {text.strip()!r}, "
|
||||
f"expected {server_repr!r}"
|
||||
)
|
||||
finally:
|
||||
cur.execute("DROP TABLE t_oracle")
|
||||
Loading…
x
Reference in New Issue
Block a user