Compare commits
45 Commits
v2026.05.0
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| c85aaf2ecb | |||
| acc21b8b81 | |||
| 9fbb97199f | |||
| ce5a582048 | |||
| 805fa58eb2 | |||
| 088171325d | |||
| 7ffc148112 | |||
| 8f341c4b2a | |||
| 429a3910e4 | |||
| 5ad296419c | |||
| c67bbc4766 | |||
| 2eb5ac8f0f | |||
| ac00a25cb7 | |||
| 0d8bd57ba9 | |||
| 66120cf6f3 | |||
| 2e30ceacfb | |||
| 2896f7e93e | |||
| 488773f078 | |||
| ac9075ca9a | |||
| 3d19ca0f1b | |||
| 4af1c90c6f | |||
| 4e186bb890 | |||
| 15d75dd052 | |||
| 5c6991efba | |||
| 9616ddbc0a | |||
| 85d3a8f7d7 | |||
| f5a539d4a8 | |||
| aa51ee82e0 | |||
| 36478f02c4 | |||
| 783b2bec97 | |||
| 87b5b7c354 | |||
| 2d83ed7b45 | |||
| dc7b9bfd94 | |||
| ab5be68738 | |||
| 21c47385ae | |||
| b67e6d008b | |||
| 073c7ed513 | |||
| 9af0a4cec9 | |||
| 24feabd21b | |||
| 1582a5295d | |||
| 86070e4688 | |||
| ad55391bf1 | |||
| a5e6cf1ae3 | |||
| 7f729b3a38 | |||
| 5825d5c55e |
602
CHANGELOG.md
602
CHANGELOG.md
@ -2,6 +2,608 @@
|
||||
|
||||
All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440.
|
||||
|
||||
## 2026.09.03 — A placeholder rewriter that could not see a string literal, and named row access
|
||||
|
||||
### Upgrade if you write string literals containing a colon
|
||||
|
||||
The driver rewrites `:1` placeholders to the `?` the wire protocol takes. That rewrite was a bare regular expression, and a regular expression cannot see a string literal, so it rewrote the inside of one:
|
||||
|
||||
```sql
|
||||
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
|
||||
-- stored as 'http://host?/x'
|
||||
```
|
||||
|
||||
Any `HH:MM` time, any URL with a port, any aspect ratio, any `key:value` string. It wrote wrong data and said nothing about it.
|
||||
|
||||
Two things made it wider than it looks. It was never opt-in: the rewrite runs whenever a statement has parameters, whatever placeholder style you actually used, so writing `?` everywhere and never touching numeric style gave no protection. And it changed the placeholder *count* while the driver still told the server there were `len(params)` of them, leaving the two disagreeing about how many binds exist.
|
||||
|
||||
It is now a single pass that substitutes only outside quotes and comments. The lexical rules are Informix's own, measured against all three servers rather than assumed from standard SQL, and one of them would have been got wrong from habit: a backslash escapes nothing, so `'a\'b'` is an unterminated string and draws `-282`. A scanner written to Postgres reflexes would have desynced there and corrupted everything after it. Block comments do not nest, the first `*/` closes them. Braces are a comment. `::` is stepped over as a unit so a cast can never be read as the start of a placeholder.
|
||||
|
||||
An unterminated quote or comment consumes the rest and substitutes nothing further. Under-substituting hands malformed SQL to the server to reject; guessing would corrupt a literal.
|
||||
|
||||
This was the last place the driver inferred meaning from SQL text instead of handling it properly.
|
||||
|
||||
### Rows that answer to a column name
|
||||
|
||||
Requested by a user running this alongside SQL Server, where both `pyodbc` and `mssql-python` hand back rows addressable three ways at once. The argument is readability on a wide projection: `row[11]` tells a reader nothing, and stays correct only until somebody adds a column in the middle.
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||
cur.execute("SELECT tabid, tabname FROM systables")
|
||||
row = cur.fetchone()
|
||||
row[0], row["tabname"], row.tabname
|
||||
```
|
||||
|
||||
Set on the connection, so it is one line for an application rather than per query. A cursor can override it. Pools and the async API forward it unchanged.
|
||||
|
||||
`Row` subclasses `tuple`, so `row == (1, "x")` is still true and code that treats rows as sequences keeps working. Slices degrade to plain tuples, since a slice has no column map.
|
||||
|
||||
**It is opt-in, and here is the number.** On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` is a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript. Defaulting it on would move the published 1.05-1.15x ratio against IfxPy to roughly 1.15-1.25x. That is a reasonable trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is a choice rather than a decision made on your behalf.
|
||||
|
||||
Details that were measured rather than assumed. Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`, and the `.lower()` in the workaround people write by hand is already a no-op. Expression columns get server-generated names like `(count(*))` that cannot be attributes, so they are subscript-only. Duplicate names resolve to the first occurrence, matching `pyodbc`. And a column always beats a method of the same name: `tuple` defines `count` and `index`, `Row` adds `keys`, `_asdict` and `_fields`, and any column with one of those names wins, because otherwise it would hand back a bound method in silence. That reserved set is computed from the class rather than hand-listed, which is what caught it: the hand-listed version covered the two `tuple` methods and missed all three of `Row`'s own.
|
||||
|
||||
### Verified
|
||||
|
||||
**472** integration tests on 15.0.1.0.3DE and 14.10.FC7W1DE, **471** on 12.10.FC12W1DE (one skip, no common table expressions before 14.10), up from 457. Both changes have tests that fail against `2026.09.02`.
|
||||
|
||||
## 2026.09.02 — A systematic review, and eleven bugs it found
|
||||
|
||||
No new features. This is the result of going back over the driver looking for the *shape* of past bugs rather than for new symptoms, and it turned up more than expected — including two that silently returned wrong data and one where `rollback()` did nothing at all.
|
||||
|
||||
### Rows never checked that they consumed their own payload
|
||||
|
||||
Fourteen framing bugs had reached users before this release. Every one was the same defect — a column reading the wrong number of bytes — and not one of them raised where the mistake happened. The wrong width produced a plausible value and corrupted whatever came *next*, so the damage always surfaced in a different column, a different row, or a different statement.
|
||||
|
||||
All fourteen were detectable for free. The payload is a fully extracted `bytes` of known length, so after decoding N columns the offset has to land exactly on the end. Now it's checked, on every row, in all three decode paths, and a mismatch names the column shape and the byte delta instead of handing back plausible garbage.
|
||||
|
||||
Turning it on found two more bugs on the first run:
|
||||
|
||||
**Smart LOBs were read as a flat 72-byte field.** They actually use the UDT envelope — 149 bytes when populated, 5 when NULL. The driver consumed 72 and left 77 behind, so a `BLOB` or `CLOB` anywhere but the final column position shifted every column after it. The 144 envelope bytes are the 72-byte locator *hex-encoded*, which means `BlobLocator.raw` had never held a locator, only the first half of the hex text. Nobody noticed because `read_blob_column` resolves LOBs server-side and never reads it.
|
||||
|
||||
**NULL composite UDTs skipped their length field**, byte for byte the LVARCHAR NULL bug in a branch twelve lines away. Both are now the one `_read_udt_envelope`, which also rejects a negative length rather than rewinding into bytes it already decoded.
|
||||
|
||||
### `rollback()` could silently do nothing
|
||||
|
||||
`cursor.execute("BEGIN WORK")` is a reasonable thing to write, and with autocommit on it opened a real transaction that the connection never learned about. `commit()` and `rollback()` are both guarded by that flag, so **`rollback()` returned successfully having sent nothing, and the rows it was asked to discard survived.** The connection then went back to the pool holding an open transaction and its locks, because the pool's cleanup is guarded by the same flag.
|
||||
|
||||
With autocommit off it failed instead, and for a sillier reason: the driver's implicit `SQ_BEGIN` fired first, so the caller's `BEGIN WORK` got `-535`, "already in transaction". The driver and the user competing to open the same transaction, and the user losing.
|
||||
|
||||
The server labels these statements and always has. Both spellings of all three now update the connection's state, and the implicit begin is skipped when the caller is doing the job themselves.
|
||||
|
||||
### Five ordinary query forms failed with a nonsense error
|
||||
|
||||
The driver decided whether to open a cursor by checking if the first word of the SQL was `SELECT`. That gets wrong:
|
||||
|
||||
```sql
|
||||
-- a leading comment → -260
|
||||
/* of any of the three flavours */ → -260
|
||||
{ that Informix accepts } → -260
|
||||
(SELECT parenthesized) → -260
|
||||
WITH cte AS (...) SELECT ... → -260
|
||||
(SELECT a) UNION (SELECT b) → -260
|
||||
```
|
||||
|
||||
`-260` is "Cursor name already in use", which describes neither the cause nor anything the caller did. It says "cursor" because the driver sent `SQ_EXECUTE` where the server was waiting to open one.
|
||||
|
||||
The server reports the statement type in the first field of every DESCRIBE response, and the driver had been parsing it into a metadata dict that nothing read. The decision now uses it. `EXECUTE FUNCTION` gets fixed as a side effect — it used to run down the DML path and discard its return value, and now yields it.
|
||||
|
||||
### A scrollable cursor took the connection down with it
|
||||
|
||||
Informix gives a session one statement slot. A scrollable cursor holds it open on purpose, so any other statement on that connection got `-285` — **and destroyed the scrollable cursor too**, whose next fetch came back `-267`, "the transaction has been rolled back, all locks released". Two unattributable failures from code that reads as entirely ordinary: iterate a large result set, run a lookup partway through.
|
||||
|
||||
It is now a `ProgrammingError` that explains the constraint and leaves the cursor alone. Re-executing the *same* scrollable cursor is still allowed and now closes its own server-side cursor first, because it collided with itself as well.
|
||||
|
||||
### Statements that failed were sometimes never released
|
||||
|
||||
A failed statement stays allocated server-side and collides with the next `PREPARE`, after which every call on that connection returns a nonsense error pointing at the *previous* SQL. That guard existed at six exits and was missing from two, both easy to reach:
|
||||
|
||||
- The parameterized-`SELECT` bind drain. Passing a string where the column is an `INT` encodes cleanly, so the rejection comes from the server (`-1213`, `-415`) *after* the guarded step.
|
||||
- Opening a scrollable cursor — no guard at all, and the worst place to lack one, since the GC-time fallback is armed on the line after.
|
||||
|
||||
All six now share one implementation, which also fixed a defect in the copies that had a cursor to close: `CLOSE` and `RELEASE` shared a single suppression, so a failing `CLOSE` skipped the `RELEASE` — losing the half that actually matters.
|
||||
|
||||
### Cleanup could land inside somebody else's statement
|
||||
|
||||
`SQ_CLOSE` and `SQ_RELEASE` act on whatever statement is current; neither names one. Two consequences.
|
||||
|
||||
A cursor finalizer that can't take the wire lock queues its cleanup for the next operation to flush — but that queue was flushed before *every* PDU, and a finalizer queues precisely because another thread is mid-statement. The flush landed inside that thread's own statement and released it (`-208` before the first fetch, `-267` between fetch batches). Flushing only at a statement boundary is both correct and sufficient.
|
||||
|
||||
And the finalizer's own lock probe couldn't see its own thread. `RLock.acquire(blocking=False)` grants a reentrant acquire to the owner, so when GC fired on a thread that was mid-statement, the finalizer concluded it had exclusive access and sent `CLOSE`/`RELEASE` into the running query. Refcounting hides this — a dropped cursor is freed at the drop — but a cursor caught in a reference cycle waits for a collection, and cycles are ordinary. Any traceback holding a cursor makes one.
|
||||
|
||||
A stale queue entry was separately fatal: the leftover `CLOSE` draws `-267`, an `OperationalError`, which was treated as a dead wire and force-closed a perfectly healthy connection.
|
||||
|
||||
### The async layer borrowed the whole process's thread pool
|
||||
|
||||
Every blocking call went through `asyncio.to_thread`, which runs on the event loop's default executor — sized `min(32, cpu_count + 4)`, so six threads on a two-CPU container.
|
||||
|
||||
`asyncio.to_thread` cannot interrupt a worker, so a cancelled await leaves the thread running the wire call until the read timeout. Cancellation is ordinary in a web app; a client disconnect cancels the request task. Measured: six cancelled calls against a six-worker default executor starve an unrelated `to_thread` indefinitely, and with the executor held, **four concurrent driver queries never ran at all**. Pool concurrency was capped by the same unrelated number — `max_size=20` on a two-CPU box ran six queries at a time.
|
||||
|
||||
Each connection now owns one thread. That's the right size rather than a compromise, since the connection serializes every wire operation on its own lock anyway, and it avoids a deadlock a shared pool-sized executor invites.
|
||||
|
||||
### Two readers shared one stream without agreeing
|
||||
|
||||
`IfxSocket` owns a read-ahead buffer that the buffered reader fills, while the login path and the connection-level drain read straight from the socket without looking at it. Buffered bytes would be skipped, and skipped bytes in a length-framed protocol don't announce themselves.
|
||||
|
||||
Nothing triggers it today, because the server sends one response per request. That's a property of the traffic, not of the code — pipelined `executemany` already puts several responses in flight, and the buffer is connection-scoped precisely so read-ahead *can* cross response boundaries. The two paths now agree by construction.
|
||||
|
||||
Also here: `fill_recv_buf` believed whatever byte count it was handed, and that count is almost always a length field straight off the wire — a garbage `0x7FFFFFFF` read as a 2 GB request and sat in `recv` until the read timeout. It now refuses above `IFX_MAX_READ_BYTES` (256 MiB default) with an error naming the number.
|
||||
|
||||
### Verified
|
||||
|
||||
**457** integration tests on 15.0.1.0.3DE and 14.10.FC7W1DE, **456** on 12.10.FC12W1DE (one skip — 12.10 has no CTEs) — up from 414. Every fix has at least one test that fails against the previous release.
|
||||
|
||||
## 2026.09.01 — TLS traffic fuzzed; no bugs found
|
||||
|
||||
Tests and docs only — no behaviour change. TLS was the last untested surface, and it came back clean.
|
||||
|
||||
### Why it needed testing separately
|
||||
|
||||
Previous TLS coverage stopped at the handshake. Everything after it ran only over plain sockets, and `SSLSocket.recv` is not `socket.recv`: it returns at most one TLS record's worth of plaintext however much you ask for, it can return fewer bytes than are available, and plaintext buffered inside the SSL object is invisible to the OS. The Phase 39 buffered reader asks for up to 64 KB per call and loops until satisfied — that loop is the thing which has to be right, and nothing in the plain-socket suite put the same pressure on it.
|
||||
|
||||
`tests/test_tls_traffic.py` runs real SQLI traffic through a TLS-terminating proxy: the framing-bug types end-to-end, payloads at 1 / 4096 / **16383 / 16384 / 16385** / 32000 bytes (straddling the ~16 KB TLS record boundary), 500- and 5000-row bulk fetches, error recovery, concurrent TLS sessions, and three negative cases — TLS client against a plaintext port, plaintext client against a TLS port (must raise rather than hang), and a verifying context correctly rejecting a self-signed certificate.
|
||||
|
||||
**All clean on all three servers.** The buffered reader handles `SSLSocket` semantics correctly.
|
||||
|
||||
### Scope, stated plainly
|
||||
|
||||
The proxy supplies the TLS half, so this exercises the driver's TLS path — the half we own. It does **not** exercise IBM's server-side TLS listener. Setting one up on the developer-edition image was attempted and abandoned: Informix 15 wants a PKCS#12 keystore (`onkstash` takes a `.p12`, not the older CMS `.kdb`), and the engine kept rejecting the stash with `GSK_ERROR_BAD_KEYFILE_PASSWORD` even with a keystore GSKit itself could open. That half is IBM's code; everything below `ssl.wrap_socket` is identical either way.
|
||||
|
||||
### Verified
|
||||
|
||||
**414/414** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 399.
|
||||
|
||||
With this, every surface has been fuzzed: type framing, fetch batching, error recovery, cursor lifecycle, threads, pooling, transactions, async cancellation, `executemany` partial failure, scrollable cursors, smart LOBs, and TLS.
|
||||
|
||||
## 2026.08.31.3 — An encoding failure inside `executemany` leaked the statement
|
||||
|
||||
The last of the untested surface: pipelined `executemany`, scrollable cursors, and smart LOBs. One bug, in the first of those.
|
||||
|
||||
### The bug
|
||||
|
||||
`executemany` builds all its BIND+EXECUTE PDUs *after* the PREPARE and before anything is drained — that batching is what makes the pipeline fast. If encoding a row raises there (a value the connection's codec cannot represent), the exception escaped the whole block **without sending the RELEASE**, leaking the prepared statement. The next PREPARE collided with it and every later call on that connection failed with an error pointing at the previous SQL.
|
||||
|
||||
Same failure as `2026.08.31.2`, in a sibling path. `_execute_dml_with_params` has guarded this exact case for the single-row path for a long time; the pipelined path was simply never given the same treatment. That is the recurring shape of these: a hazard understood in one place and not carried across to the code next to it.
|
||||
|
||||
### What was already sound
|
||||
|
||||
Worth recording, because it's the larger part of the result:
|
||||
|
||||
- **`executemany` constraint failures.** Duplicate-key and NOT NULL violations at the first, middle, and last position of batches from 2 to 1000 rows all recover cleanly, and `COUNT(*)` always agrees with a full fetch. The pipelined drain-N-responses invariant holds under partial failure.
|
||||
- **Scrollable cursors.** `fetch_first` / `fetch_last` / `fetch_prior` / `fetch_relative` / `fetch_absolute` are correct at 0, 1, 2, 5, 50 and 300 rows, including off both ends (`None`, not a wrap or a crash) and a full forward walk after arbitrary positioning. Twenty abandoned scroll cursors leak nothing.
|
||||
- **Smart LOBs.** Round-trip at 0, 1, 255, 256, 1023, 1024, 4095, 4096, 65535 and 65536 bytes — straddling the 4096-byte `SQ_FILE` chunk and the 64K mark — plus recovery from failed reads.
|
||||
|
||||
### Verified
|
||||
|
||||
**399/399** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 356.
|
||||
|
||||
### Honest note on what is still not fuzzed
|
||||
|
||||
TLS is covered by handshake tests against a self-signed local socket, which exercises the transport wrapper but not a real Informix TLS listener (that needs server-side keystore and `onconfig` SSL setup we don't have in the test containers). The SQLI layer above the socket is identical either way and is now heavily fuzzed, so the residual risk is confined to the handshake itself.
|
||||
|
||||
## 2026.08.31.2 — A failed statement killed the connection; cancelled acquires starved the pool
|
||||
|
||||
Two more found by fuzzing rather than by users, this time outside the type system. Both are ordinary-path bugs that a happy-path test cannot reach.
|
||||
|
||||
### A failed statement was never released
|
||||
|
||||
Successful DML sent `PREPARE → EXECUTE → RELEASE`. **Failing DML sent `PREPARE → EXECUTE` and stopped.** The prepared statement stayed allocated server-side, collided with the next `PREPARE`, and from then on every call on that connection returned a nonsense error — `-255 "Not in transaction"` under autocommit, `-285` otherwise — whose reported offset pointed back at the *failed* SQL rather than the statement that actually failed.
|
||||
|
||||
The practical shape of this: **one duplicate-key violation bricked the connection.** An `INSERT` that trips a unique constraint is about the most routine error an application can hit — every "insert if not exists" pattern produces them — and afterwards nothing on that connection worked again.
|
||||
|
||||
The docstring on the parameterised path already described this exact hazard and guarded the *parameter-encoding* failure. It just never covered the case where EXECUTE itself failed. Fixed in all three paths (DML, parameterised DML, and SELECT, whose fetch loop had the same gap and whose GC-time finalizer only covers scrollable cursors).
|
||||
|
||||
### Cancelling a pool acquire leaked the connection permanently
|
||||
|
||||
`asyncio.to_thread` cannot interrupt its worker. When a task awaiting `pool.acquire()` was cancelled *while the worker was still blocked waiting for a free connection*, the worker went on to succeed and hand back a connection **nobody owned** — checked out, never returned. Each occurrence cost one pool slot until the pool was dead.
|
||||
|
||||
This is not a theoretical race for anyone serving HTTP. A client disconnecting cancels the request task, and under load those cancellations land precisely while waiting for a connection. The pool dies one slot at a time, only under load, and the eventual symptom (`PoolTimeoutError`) points nowhere near the cause.
|
||||
|
||||
`acquire()` now shields the inner future and, if the caller goes away, returns whatever the worker produced to the pool. `add_done_callback` fires immediately on an already-resolved future, so "the worker finished a moment before the cancellation" is the same code path rather than a separate race.
|
||||
|
||||
### What the harness looked for
|
||||
|
||||
Three areas the type-matrix fuzzer cannot see:
|
||||
|
||||
- **Fetch batching.** `NFETCH` is a 4096-*byte* budget, so rows per batch moves with row width. Row counts from 0 to 1025 across narrow/medium/wide schemas, every fetch style (`fetchall`, `fetchone` loop, `fetchmany` at several sizes, iteration) required to agree exactly.
|
||||
- **Error recovery.** Eight error classes, repeated, each followed by a known-good query on the same cursor *and* a fresh one. Repetition matters: a leak accumulates.
|
||||
- **Concurrency.** Threads sharing one connection, pooled threads that deliberately fail before releasing, transaction isolation across pool checkouts, and async cancellation. Every worker reads back a value only it supplied, so a crossed wire fails on *data* even when nothing raises.
|
||||
|
||||
### Two harness false positives, recorded because they're easy to re-trip
|
||||
|
||||
Informix **silently truncates** over-long strings — a 500-character value into `VARCHAR(8)` stores 8 characters and raises nothing. Verified that a literal SQL insert behaves identically, so the driver matches the server; the test expectation was wrong, not the code.
|
||||
|
||||
Informix also **rejects a bare `?` in a projection** (`SELECT ? FROM t` is a syntax error — no type to infer). `SELECT ?::INT FROM t` works. Worth knowing before concluding the driver mishandles parameters.
|
||||
|
||||
### Verified
|
||||
|
||||
**356/356** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 326.
|
||||
|
||||
## 2026.08.31.1 — Three more bugs, found by a fuzzer instead of a user
|
||||
|
||||
Six framing bugs had reached users across three reports. Rather than wait for a seventh, this release adds a harness built specifically to find that class of bug — and it immediately found three more, one of which **hangs the connection**.
|
||||
|
||||
### What it found
|
||||
|
||||
**`BOOLEAN` could not be bound as a parameter at all.** `_encode_bool` emitted type code 45, which the server does not accept as a bind type. It simply stopped responding, so *any* `execute()` with a `bool` parameter hung until the read timeout — or forever, if none was set. Type 41 (how BOOLEAN is *described* in results) hangs identically; the descriptor type and the bind type are not interchangeable. We now bind `'t'`/`'f'` as CHAR and let the server cast, mirroring Informix's own literal syntax.
|
||||
|
||||
This one is worth dwelling on: a hang is worse than wrong data, and it had shipped in every release that claimed BOOLEAN support.
|
||||
|
||||
**Unscaled `DECIMAL` was read one byte short**, shifting every following column. The width formula is `(precision + (scale & 1) + 3) // 2` per `IfxColumnInfo.adjustedColumnLength`; we had dropped the `scale & 1` term. That only matters when precision is even and scale is odd — and an unscaled `DECIMAL(p)` reports scale **255**. So `DECIMAL(16)` is 10 bytes on the wire, not 9. Verified against the wire for ten DECIMAL/MONEY shapes.
|
||||
|
||||
The same formula governs DATETIME and INTERVAL, whose qualifier parity happens to track their digit count, so those agreed by coincidence. All four types now share one helper taken from the reference — one rule beats two that coincide for reasons nobody wrote down.
|
||||
|
||||
**NULL `CHAR`/`NCHAR` came back as `''`**, indistinguishable from a genuinely empty column, so `WHERE c IS NULL` disagreed with what the driver returned. The wire distinguishes them plainly:
|
||||
|
||||
```
|
||||
'ab' -> 61 62 20 20 20 20
|
||||
'' -> 20 20 20 20 20 20 all spaces
|
||||
NULL -> 00 20 20 20 20 20 leading nul
|
||||
```
|
||||
|
||||
### The harness
|
||||
|
||||
`tests/test_type_matrix.py` encodes three principles, each derived from how a real bug escaped:
|
||||
|
||||
1. **Always put a column after the value under test.** Every bug so far corrupted the *next* column; a trailing value can be mis-sized with no visible effect. Every case now ends in a sentinel column, and projections are rotated so each type is exercised in each position.
|
||||
2. **Vary the data, not just the type.** Branch coverage means nothing if no value takes the branch. The LVARCHAR fixture was `'lv value'` — 8 characters, even, never NULL — so both of its broken branches sat unexecuted through 247 tests. The corpus now carries odd/even lengths, empty vs NULL, min/max, negative, scaled vs unscaled.
|
||||
3. **Use an oracle our own codecs don't share.** A Python round-trip cannot catch a symmetric encode/decode bug: our DATETIME encoder wrote zeros and our decoder read them back in perfect agreement. Asking the *server* to render the stored value as text breaks that symmetry, and that check is what would have caught the DATETIME bug on day one.
|
||||
|
||||
It runs exhaustively over the corpus plus seeded random multi-type rows, so failures are reproducible.
|
||||
|
||||
### Verified
|
||||
|
||||
**326/326** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 281. The fuzzer additionally reports clean over 80-round runs at several seeds and widths on all three servers.
|
||||
|
||||
## 2026.08.31 — Fix LVARCHAR tuple framing; DATETIME fractions on bind
|
||||
|
||||
More data corruption, from the same field report that produced `2026.05.08.2`. **If your schema has `LVARCHAR` columns, upgrade** — anything selected after one could be wrong, and `2026.08.27` is not safe.
|
||||
|
||||
### LVARCHAR shifted every column that followed it
|
||||
|
||||
Two independent framing errors in the same envelope:
|
||||
|
||||
**A phantom pad byte.** We appended an even-byte pad when the value length was odd. There is no pad. Wire capture on Informix 12.10 for `INT8 / LVARCHAR / INT8`:
|
||||
|
||||
```
|
||||
00 01 00 00 07 d1 00 00 00 00 │ 00 │ 00 00 00 0b │ 50 61 63 6b 61 67 65 52 6f 6f 74 │ 00 01 00 00 00 0a …
|
||||
a = INT8 2001 │ind │ len = 11 │ "PackageRoot" (11 bytes, odd) │ b = INT8 10 starts HERE
|
||||
```
|
||||
|
||||
**A missing length field on NULL.** We returned as soon as the indicator said NULL, leaving its 4-byte length unread. The length is part of the envelope and is always present:
|
||||
|
||||
```
|
||||
'odd' -> 00 | 00 00 00 03 | 6f 64 64 8 bytes
|
||||
NULL -> 01 | 00 00 00 00 5 bytes
|
||||
'' -> 00 | 00 00 00 00 5 bytes
|
||||
```
|
||||
|
||||
NULL and empty string differ only in the indicator byte.
|
||||
|
||||
Either error shifted everything downstream. The reported symptom: `INT8` `10` decoding as `2560` — the same value shifted one byte left — strings losing their first character, and wide rows raising `IndexError` or `ValueError: INT8 payload too short` once the drift ran off the end of the payload.
|
||||
|
||||
The reporter isolated it by **reordering columns in the projection**: values were right when they came first and wrong when they came later. That's the fingerprint of positional drift, and it's a genuinely good diagnostic technique.
|
||||
|
||||
### Why the tests missed it, again
|
||||
|
||||
The only LVARCHAR fixture was `'lv value'` — 8 characters, even, never NULL. Neither faulty branch ever executed in 247 tests. Same shape of gap as `2026.05.08.2`: full coverage of the code paths, no coverage of the *data* that reaches them.
|
||||
|
||||
New tests vary length parity deliberately (0, 1, 2, 3, 11, 255, 256), cover NULL and empty separately, and always put a column *after* the LVARCHAR — a trailing one can be mis-sized with no visible effect. `test_wide_row_survives_column_reordering` rotates the projection through every position, mirroring how this was found.
|
||||
|
||||
### DATETIME lost sub-second precision on INSERT
|
||||
|
||||
Found while verifying the above. `_encode_datetime` emitted `YEAR TO SECOND` unconditionally, so binding a `datetime` carrying microseconds into a `DATETIME YEAR TO FRACTION(n)` column stored zeros — silently. Reads were always correct, so the value only went missing on the way in.
|
||||
|
||||
It now widens to `YEAR TO FRACTION(5)` when `microsecond` is non-zero and keeps the original encoding otherwise, so the long-exercised path is byte-identical for the common case. Binding into a narrower column still works; Informix converts qualifiers on assignment and truncates server-side. `FRACTION(5)` resolves to 10 µs, so Python's sixth digit is dropped — a limit of the type, now pinned by tests.
|
||||
|
||||
### `server_version` reported the protocol version
|
||||
|
||||
Also from the field report: `conn.server_version` returned `9.56.FC6` for a 12.10 server, which reads like a client-SDK version. The login response only carries Informix's *internal* protocol version — 12.10 announces itself as 9.56, 14.10 as 9.59 — and documenting that didn't make the name any less misleading.
|
||||
|
||||
- `conn.server_version` now returns the release (`…Version 12.10.FC6`). It costs one `DBINFO` query on first access, cached thereafter, and falls back rather than raising if no database is open.
|
||||
- `conn.server_version_internal` returns the raw login string, free as before.
|
||||
|
||||
### Verified
|
||||
|
||||
281/281 integration tests on 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE (`make test-matrix`), plus 123 unit tests. 37 of those tests are new.
|
||||
|
||||
## 2026.08.27 — Decode the SQ_PROTOCOLS capability negotiation
|
||||
|
||||
Closes the last open item from the Informix 12 field report. The driver hardcodes several wire-framing choices that SQLI actually *negotiates*; those choices were correct on every server we'd measured, but "correct as far as we know" and "checked" are different things, and the failure mode for a framing mismatch is silently corrupted rows.
|
||||
|
||||
### What changed
|
||||
|
||||
We were already sending `SQ_PROTOCOLS` with the same 8-byte client offer IBM's JDBC driver uses — and throwing the server's reply away. Now we decode it.
|
||||
|
||||
New `informix_db.ServerCapabilities`, reachable from any connection:
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(...)
|
||||
caps = conn.server_capabilities
|
||||
caps.four_byte_offset # describe uses 4-byte string-table/field-index widths
|
||||
caps.varchar_var_len # VARCHAR is length-prefixed, not width-padded
|
||||
caps.remove_64k_limit # 4-byte length prefixes in the fast-path
|
||||
caps.violated_assumptions() # [] when our hardcoded framing matches
|
||||
conn.server_version # from the login response
|
||||
```
|
||||
|
||||
`violated_assumptions()` is the point of the exercise. It names each place we emit or parse a fixed wire shape that is actually capability-gated, and returns empty when the server agrees. On connect, a non-empty result logs a warning naming the specific bit. A server we've never tested now produces a diagnosable complaint instead of quiet corruption.
|
||||
|
||||
Nothing branches on these bits yet — this release is observation and validation only. That's deliberate: the hardcoded framing is correct on all three supported servers, and swapping working code for newly-written conditional code without a server that needs it would add risk for no benefit.
|
||||
|
||||
### The measurement
|
||||
|
||||
All three servers negotiate the same thing:
|
||||
|
||||
| Server | Negotiated mask |
|
||||
|---|---|
|
||||
| 15.0.1.0.3 | `bdbe9ffe7fb7ffef` `ff` |
|
||||
| 14.10.FC7W1 | `bdbe9ffe7fb7ffef` `f8` |
|
||||
| 12.10.FC12W1DE | `bdbe9ffe7fb7ffef` `f0` |
|
||||
|
||||
**The first eight bytes are byte-identical.** That is the root explanation for everything in the 2026.05.08.2 investigation: 12.10, 14.10, and 15 don't merely behave similarly, they negotiate exactly the same 64-bit capability set. `violated_assumptions()` returns empty on all three.
|
||||
|
||||
Two details worth recording:
|
||||
|
||||
The reply is **nine** bytes, not eight. JDBC's `enhancedProtocolMechanism` dispatches on `case 0..7` and silently discards the ninth, so its `BitSet(64)` never sees it. We decode it, because it is the *only* part of the mask that differs between releases — newer feature flags that this JDBC build predates.
|
||||
|
||||
`Cap_1` in the login response is **not** a server version. It's the client's own declared protocol level echoed back, which is why JDBC tests `== 316` rather than `>=`. The actual version string there is the *internal* one: Informix 12.10 reports `9.56`, 14.10 reports `9.59`. At the protocol level both really are 9.x servers, which is why the marketing version jump didn't move the wire format.
|
||||
|
||||
This also retires a red herring. `isUSVER` (bit 2) is one of six bits JDBC pre-sets for any non-zero `Cap_1`, and Java's `BitSet.set` never clears — so it is true on every modern server regardless of the mask. It can't explain version-specific behaviour and isn't worth chasing.
|
||||
|
||||
### Also fixed: `__version__` was reporting a sentinel
|
||||
|
||||
Found incidentally while bumping the version. The distribution was renamed `informix-db` to `informix-driver` on 2026-05-08, but `__init__.py` kept calling `version("informix-db")`. `importlib.metadata` needs the *distribution* name, not the module name, and the miss fails silently — `__version__` degrades to `"0.0.0+local"` rather than raising.
|
||||
|
||||
So every install of the renamed package has been reporting `0.0.0+local`. It escaped notice locally because a stale `informix-db` distribution was still present in the dev environment and answered the lookup with its own old version.
|
||||
|
||||
Added `tests/test_package_metadata.py`, which asserts the name passed to `importlib.metadata.version()` matches `[project].name` in `pyproject.toml`. Verified it actually catches the bug by reintroducing it.
|
||||
|
||||
### Tests
|
||||
|
||||
24 unit tests over the captured masks (no server needed) and 6 integration tests that assert live negotiation. The integration ones run on every server in `make test-matrix`, so the assumption check is re-verified on 12.10, 14.10, and 15 each time.
|
||||
|
||||
Full suite: **247 / 247** on all three.
|
||||
|
||||
## 2026.05.08.2 — Fix three tuple-framing bugs (BOOLEAN, NCHAR, INT8/SERIAL8)
|
||||
|
||||
Data-corruption fixes. All three affect **every** Informix version including 15, and two of them silently corrupt columns *after* the offending one. If your schema uses `BOOLEAN`, `NCHAR`, `INT8`, or `SERIAL8`, upgrade.
|
||||
|
||||
Reported from the field against Informix 12. The version turned out to be a red herring — see "On the version question" below.
|
||||
|
||||
### BOOLEAN corrupted every following column
|
||||
|
||||
The server describes `BOOLEAN` as `UDTFIXED` (41) with `encoded_length = 1`, but `encoded_length` is the size of the *value*, not the field. On the wire it carries the standard UDT envelope — `[1-byte null indicator][4-byte length][data]` — six bytes for a one-byte value. We consumed one byte and left five behind, shifting everything downstream.
|
||||
|
||||
Captured payload for `(INT, BOOLEAN 't', INT, VARCHAR 'tail')`:
|
||||
|
||||
```
|
||||
00 01 b2 07 | 00 00 00 00 01 74 | 00 03 64 0e | 04 74 61 69 6c
|
||||
a=111111 | b — 6 bytes | c=222222 | d = [len 4]"tail"
|
||||
```
|
||||
|
||||
Before: `(111111, b'\x00', 1, '\x00\x03d\x0e\x04tail')`. After: `(111111, True, 222222, 'tail')`.
|
||||
|
||||
### NCHAR ate its first character, then crashed
|
||||
|
||||
`NCHAR` is fixed-width and space-padded, exactly like `CHAR`. We had it grouped with the byte-length-prefixed types, so `NCHAR(10)` holding `'nch'` read `0x6E` (`'n'`) as a 110-byte length and advanced the offset by 110. Single-column selects silently returned `'ch'`; anything with a following column raised `struct.error`.
|
||||
|
||||
`NVARCHAR` *is* byte-length-prefixed and is unchanged — there's now a regression test guarding both sides of that distinction.
|
||||
|
||||
### INT8 / SERIAL8 had no decoder
|
||||
|
||||
They were absent from `FIXED_WIDTHS` and fell through to the unknown-type path, surfacing as raw `bytes`. No desync (the width happened to match `encoded_length`), just a wrong type.
|
||||
|
||||
`INT8` is **not** `BIGINT`. It's 10 bytes, sign-magnitude, with the halves stored high-last:
|
||||
|
||||
```
|
||||
bytes 0-1 sign word: 0 = NULL, 1 = positive, 0xFFFF = negative
|
||||
bytes 2-5 LOW 32 bits, big-endian unsigned
|
||||
bytes 6-9 HIGH 32 bits, big-endian unsigned
|
||||
```
|
||||
|
||||
`+n` and `-n` have identical magnitude bytes, so decoding this as a two's-complement int64 is wrong for every negative value while looking fine for every positive one.
|
||||
|
||||
### On the version question
|
||||
|
||||
The report arrived as "Informix 12 mangles result sets." It isn't a version issue. Running the same 28-type round-trip against **12.10.FC12W1DE**, **14.10.FC7W1DE**, and **15.0.1.0.3DE** produced **byte-identical** wire output — same type codes, same encoded lengths, same values. All three were equally broken, and are now equally fixed.
|
||||
|
||||
The reason it read as version-specific: `INT8`/`SERIAL8` dominate Informix 12-era schemas (`BIGINT` only arrived in 11.50), while our test fixtures use `BIGINT`. The bugs were always there; older schemas just walk into them far more often.
|
||||
|
||||
### Why 251 tests missed this
|
||||
|
||||
No fixture used `BOOLEAN`, `NCHAR`, `INT8`, or `SERIAL8`. That's the whole explanation. Coverage of *code paths* was good; coverage of the *type matrix* had holes, and the holes were exactly where the bugs lived.
|
||||
|
||||
Added `tests/test_type_framing.py` (20 integration tests) and `tests/test_int8_unit.py` (14 unit tests, wire vectors captured from both servers). Every affected type is now tested twice — once for its own value, once with trailing columns, because the trailing-column case is what catches desync. A single-column test passes while the driver corrupts every real query.
|
||||
|
||||
### Verified
|
||||
|
||||
Full integration suite, same commit, all green:
|
||||
|
||||
| Server | Result |
|
||||
|---|---|
|
||||
| 15.0.1.0.3DE | 241 / 241 |
|
||||
| 14.10.FC7W1DE | 241 / 241 |
|
||||
| 12.10.FC12W1DE | 241 / 241 |
|
||||
|
||||
New framing tests pass 34/34 on all three.
|
||||
|
||||
### Server matrix is now reproducible
|
||||
|
||||
Testing three versions by hand was tedious and undocumented, which is part of why it never happened. Added:
|
||||
|
||||
- `tests/docker-compose.legacy.yml` — 12.10 on 9089 and 14.10 on 9090, coexisting with the primary 15 container on 9088
|
||||
- `tests/setup-spaces.sh` — creates `blobspace1` + `sbspace1` in any dev container. Handles the layout differences between images (12.10/14.10 use a flat `INFORMIXDIR`, 15 nests a versioned subdirectory; `ONCONFIG` is named differently; 12.10 DE ships no `ontape`). Without these spaces ~21 tests fail with errors that look like driver bugs and aren't.
|
||||
- `make ifx-legacy-up` / `ifx-legacy-setup` / `ifx-legacy-down` / `ifx-spaces`
|
||||
- `make test-matrix` — the suite against all three versions
|
||||
|
||||
The whole documented workflow was validated from scratch (containers destroyed, recreated via compose, spaces created via the script, matrix run) rather than written up after the fact from commands that happened to work.
|
||||
|
||||
### Also
|
||||
|
||||
`README.md` and `_fastpath.py` claimed 12.10 compatibility that had never been tested. The claims happened to be correct, but they were guesses when written and cost a user debugging time. Both now state what was measured, on which image, on what date.
|
||||
|
||||
## 2026.05.05.12 — Phase 39: Connection-scoped read-ahead buffer
|
||||
|
||||
Closes the C-vs-Python bulk-fetch gap to within ~7-15% of IfxPy. The lever was the buffer/I/O machinery, not the codec — Phase 37/38 had already brought the codec to within ~25% of IfxPy's C path; the remaining gap was 450k+ ``read_exact`` calls per 100k-row fetch, each doing its own ``recv``-loop and ``bytes.join``.
|
||||
|
||||
### What changed
|
||||
|
||||
`src/informix_db/_socket.py`:
|
||||
|
||||
- ``IfxSocket`` now owns a connection-scoped read-ahead buffer (``_recv_buf`` bytearray + ``_recv_pos`` integer cursor) and exposes ``fill_recv_buf(need)`` to top it up from the socket.
|
||||
- One ``recv()`` per ~64 KB instead of per field. For a 100k-row SELECT, syscall count drops from ~450k to a few hundred.
|
||||
- Compaction: when ``_recv_pos > recv_size``, the consumed prefix is sliced off in place. Memory is bounded at roughly ``2 * recv_size`` for the connection's lifetime.
|
||||
|
||||
`src/informix_db/_protocol.py`:
|
||||
|
||||
- New ``BufferedSocketReader(IfxStreamReader)`` — a thin parser-view over the ``IfxSocket``'s persistent buffer. Each method delegates the buffer-fill to the socket, then reads via ``struct.unpack_from(buf, offset)`` directly out of the bytearray (avoiding the intermediate slice the legacy reader created).
|
||||
|
||||
`src/informix_db/cursors.py`:
|
||||
|
||||
- ``_make_socket_reader(sock)`` chooses ``BufferedSocketReader`` (default) or the legacy ``_SocketReader`` based on ``IFX_BUFFERED_READER``. Set ``IFX_BUFFERED_READER=0`` to fall back to the legacy reader.
|
||||
- The four cursor sites that build a reader now call the helper.
|
||||
|
||||
### Why the buffer is on `IfxSocket`, not on the reader
|
||||
|
||||
The first iteration of Phase 39 put the bytearray on the reader. That hung on `test_executemany_1000_rows` — the pipelined-executemany path (Phase 33) streams N responses back-to-back across multiple cursor reads, and a per-reader buffer threw away pre-fetched bytes when one reader was destroyed and the next was created. The next ``recv()`` then blocked waiting for bytes that had already been consumed.
|
||||
|
||||
Moving the buffer to ``IfxSocket`` mirrors how `asyncpg` (`buffer.pyx` lives on the protocol object) and `psycopg3` (`pq.PGconn`) structure their read paths. The reader is a short-lived view; the buffer outlives it.
|
||||
|
||||
### Performance
|
||||
|
||||
A/B from the same harness, same Docker container, same ``p34_select`` table, warmed cache:
|
||||
|
||||
| Workload | Phase 38 | Phase 39 | Δ |
|
||||
|---|---:|---:|---:|
|
||||
| `select_scaling_1000` | 2.901 ms | **1.716 ms** | **-41%** |
|
||||
| `select_scaling_10000` | 24.317 ms | **16.084 ms** | **-34%** |
|
||||
| `select_scaling_100000` | 250.363 ms | **168.982 ms** | **-32%** |
|
||||
|
||||
Against IfxPy 2.0.7 (C-bound, ODBC) on the same workloads, head-to-head:
|
||||
|
||||
| Workload | IfxPy | informix-db Phase 39 | Ratio |
|
||||
|---|---:|---:|---:|
|
||||
| `select_scaling_1000` | 1.637 ms | 1.716 ms | **1.05×** |
|
||||
| `select_scaling_10000` | 15.073 ms | 16.084 ms | **1.07×** |
|
||||
| `select_scaling_100000` | 147.361 ms | 168.982 ms | **1.15×** |
|
||||
|
||||
The bulk-fetch gap that was 2.4× before Phase 37 is now within 5-15% of IfxPy. With IfxPy's measurement IQR at ~21% on the 100k workload (Docker→host loopback noise) vs our IQR at ~0.2%, the headline ratio is partly noise — the real gap is plausibly 1.05-1.2×.
|
||||
|
||||
A pure-Python driver running within ~10% of an ODBC/C-bound driver on bulk fetch is, as far as I'm aware, unprecedented on the SQLI protocol.
|
||||
|
||||
### Tests
|
||||
|
||||
All 251 integration tests pass with the new default reader. The legacy reader is exercised via ``IFX_BUFFERED_READER=0`` (also passes 251/251). Both unit suites pass.
|
||||
|
||||
### Migration
|
||||
|
||||
No code changes required. The default reader switches at version bump. To opt out:
|
||||
|
||||
```bash
|
||||
IFX_BUFFERED_READER=0 python myapp.py
|
||||
```
|
||||
|
||||
If you find a wire shape the buffered reader doesn't handle, please file an issue with the ``IFX_BUFFERED_READER=0`` workaround as the immediate mitigation.
|
||||
|
||||
## 2026.05.05.11 — Phase 38: `exec()`-based row-decoder codegen
|
||||
|
||||
Closes more of the C-vs-Python codec gap on bulk fetch by emitting a specialized row decoder per result-set shape via `exec(compile(src, ...))` and inlining the common fixed-width decode bodies directly into the generated source. This is the lever flagged in the Phase 37 changelog as "the next lever for materially closing the gap."
|
||||
|
||||
### What changed
|
||||
|
||||
`src/informix_db/_resultset.py`:
|
||||
|
||||
- New `compile_row_decoder(readers, columns)` builds a Python source string per result-set shape and compiles it via `exec()`. The generated function has signature `parse_row(payload, offset, encoding) -> tuple` and contains zero loops — every column is handled by inline straight-line code.
|
||||
- For the common fixed-width types (`SMALLINT`, `INT`, `SERIAL`, `BIGINT`, `BIGSERIAL`, `FLOAT`, `SMFLOAT`, `DATE`), the decoder body is **inlined** rather than called: `v0 = _UNPACK_INT(raw)[0]; if v0 == -2147483648: v0 = None`. That eliminates one Python function call per such column per row — the actual physics behind the speedup.
|
||||
- `BOOL` deliberately left to its canonical decoder. Inlining `bool(raw[0])` would silently accept `'f'` (102, truthy) as `True` — semantic drift.
|
||||
- `parse_tuple_payload` accepts an optional `row_decoder=` parameter. When provided, the entire hot loop is bypassed: `return row_decoder(payload, 0, encoding)`.
|
||||
- The generated source is printable via `IFX_DEBUG_CODEGEN=1` for inspection.
|
||||
|
||||
`src/informix_db/cursors.py`:
|
||||
|
||||
- After `parse_describe`, the cursor compiles **both** the Phase 37 reader-list AND the Phase 38 row decoder. `parse_tuple_payload` prefers the codegen'd decoder; if codegen returns `None` (unsupported shape), the readers-list dispatch handles it; if both are `None`, the legacy branch chain runs.
|
||||
|
||||
### Performance
|
||||
|
||||
Real numbers from the integration container, median of 10+ rounds, A/B against Phase 37 (stash → bench → unstash → bench, same Docker container, same load):
|
||||
|
||||
| Benchmark | Phase 37 | Phase 38 | Δ |
|
||||
|---|---:|---:|---:|
|
||||
| `select_scaling[1000]` | 2.74 ms | 2.62 ms | -4% |
|
||||
| `select_scaling[10000]` | 25.13 ms | 22.58 ms | **-10%** |
|
||||
| `select_scaling[100000]` | 257.66 ms | 227.67 ms | **-12%** |
|
||||
| `select_type_mix_1000_rows` | 4.57 ms | 4.32 ms | -5% |
|
||||
| `wide_row_select[5]` | 2.28 ms | 2.05 ms | **-10%** |
|
||||
| `wide_row_select[20]` | 4.27 ms | 3.63 ms | **-15%** |
|
||||
| `wide_row_select[50]` | 8.10 ms | 7.19 ms | -11% |
|
||||
| `wide_row_select[100]` | 15.17 ms | 13.59 ms | **-10%** |
|
||||
|
||||
**The win scales with both row count and column count** — exactly the codegen profile we'd expect from per-column inlining. At small row counts the one-time `exec(compile(...))` cost dilutes the per-row win; at 100k rows it's invisible.
|
||||
|
||||
### Architectural note
|
||||
|
||||
This is conceptually the same step `psycopg3`'s C-mode and `asyncpg` (Cython) take, except we stay 100% pure-Python. We don't compile to native code; we compile to specialized Python bytecode via `exec()`. CPython's bytecode interpreter is remarkably efficient on straight-line code with local variables — the codegen win comes from removing dispatch and function-call overhead, not from native execution.
|
||||
|
||||
The three-tier composition stays clean:
|
||||
1. **Codegen** (`row_decoder`) — fastest path, fires on common shapes
|
||||
2. **Reader list** (`readers`) — fallback when codegen rejects a shape
|
||||
3. **Legacy branch chain** — fallback for the no-readers case
|
||||
|
||||
### Tests
|
||||
|
||||
All 251 integration tests still pass. The codegen output was verified against `IFX_DEBUG_CODEGEN=1` for a 9-column mixed-type shape: SMALLINT/INT/BIGINT/FLOAT/SMFLOAT/DATE/BOOL/CHAR/VARCHAR. All inline NULL-sentinel checks correct; CHAR/VARCHAR fall through to the registered decoder via the globals dict (`_D{i}`). No new test code; the integration suite + benchmark suite are the regression test.
|
||||
|
||||
### Honest assessment
|
||||
|
||||
Combined with Phase 37 (per-column reader strategy), bulk fetch is now ~20-25% faster than Phase 36 in the worst-affected workloads. The IfxPy gap on `select_scaling[100000]` shrinks from ~2.2× to ~2.0×. Pure-Python finally costs roughly **2× C** for bulk fetch — close enough that the deployment win (no CSDK, no JVM) starts to outweigh the perf cost for most users.
|
||||
|
||||
Further codegen wins are possible (inlining DATETIME/INTERVAL, batch-decoding all rows of a payload in one call) but with diminishing returns. The remaining gap is dominated by socket I/O and the SQLI protocol's chatty per-row framing — protocol-level work, not codec work.
|
||||
|
||||
## 2026.05.05.10 — Phase 37: Pre-baked per-column reader strategy
|
||||
|
||||
Closes some of the C-vs-Python codec gap on bulk fetch by moving per-column dispatch decisions from row time to `parse_describe` time. Same idea as psycopg3's pure-Python loader-cache pattern.
|
||||
|
||||
### What changed
|
||||
|
||||
`src/informix_db/_resultset.py`:
|
||||
- New `compile_column_readers(columns)` returns a list of pre-computed dispatch tuples — one per column. Each tuple is `(kind, *args)` where `kind` is a small int identifying the reader strategy.
|
||||
- `parse_tuple_payload` accepts an optional `readers=` parameter. When provided, the hot loop dispatches on the integer kind (one int comparison per column) instead of running the legacy frozenset/dict-lookup chain.
|
||||
- Common types (`FIXED`, `BYTE_PREFIX`, `CHAR`, `LVARCHAR`, `DECIMAL`, `DATETIME`, `INTERVAL`) get pre-compiled fast paths. Rare types (UDT/composite) tagged `_RK_LEGACY` and fall through to a `_legacy_dispatch_one_column` helper.
|
||||
|
||||
`src/informix_db/cursors.py`:
|
||||
- `Cursor` now stores `self._column_readers` after `parse_describe`, computed once via `compile_column_readers`. Reset on each new `execute`.
|
||||
- The fetch loop passes `readers=self._column_readers` to `parse_tuple_payload`.
|
||||
|
||||
### Performance
|
||||
|
||||
Real numbers from the integration container, median of 10+ rounds:
|
||||
|
||||
| Benchmark | Before | After | Δ |
|
||||
|---|---:|---:|---:|
|
||||
| `select_scaling[1000]` | 2.7 ms | 2.51 ms | -7% |
|
||||
| `select_scaling[10000]` | 25.8 ms | 25.0 ms | -3% |
|
||||
| `select_scaling[100000]` | 271 ms | 246 ms | **-9%** |
|
||||
| `wide_row_select[5]` | 2.4 ms | 2.16 ms | -10% |
|
||||
| `wide_row_select[20]` | 5.1 ms | 4.14 ms | -19% |
|
||||
| `wide_row_select[50]` | 10.1 ms | 8.21 ms | -19% |
|
||||
| `wide_row_select[100]` | 19.4 ms | 14.6 ms | **-25%** |
|
||||
|
||||
**Wide-row workloads benefit most** — per-column dispatch savings accumulate linearly with column count. At 100 columns the speedup is 25%; at 5 columns it's 10%.
|
||||
|
||||
### Honest assessment
|
||||
|
||||
Less than the ~30% I projected. The actual per-row cost is dominated by decoder bodies and slice operations more than I estimated; pre-baking the dispatch only saved ~50-100 ns/col instead of the 150-200 ns I'd hoped for.
|
||||
|
||||
The IfxPy gap shrinks from ~2.4× to ~2.2× on bulk fetch. Real progress, but not closing-the-gap territory. **The next lever for materially closing the gap is `exec()`-based codegen** (build a row-decoder function per result-set shape; eliminates per-column iteration overhead entirely). Possible Phase 38.
|
||||
|
||||
### Architectural note
|
||||
|
||||
This is the same pattern psycopg3 uses in its pure-Python mode: cache loaders per column at execute time, dispatch via lookup in the hot loop. We pick tuple-dispatch over object-method dispatch (`r[0]` int compare vs. `loader.load(data)`) for raw speed in the inner loop — slightly less extensible but ~20-30 ns faster per column.
|
||||
|
||||
### Tests
|
||||
|
||||
All 221 integration tests still pass. No new test code; the benchmark suite acts as the regression test (parse_tuple_5cols / select_scaling / wide_row_select).
|
||||
|
||||
## 2026.05.05.9 — IfxPy scaling comparison + honest comparison numbers (Phase 36)
|
||||
|
||||
Adds the IfxPy side of Phase 34's scaling benchmarks (1k / 10k / 100k rows for both `executemany` and `SELECT`) and updates the README's comparison table with the **actually-correct numbers**.
|
||||
|
||||
31
Makefile
31
Makefile
@ -79,6 +79,37 @@ ifx-status: ## Check container health and listener readiness
|
||||
@docker ps --filter name=$(IFX_CONTAINER) --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
|
||||
@nc -zv 127.0.0.1 9088 2>&1 | head -1
|
||||
|
||||
ifx-spaces: ## Create blobspace1 + sbspace1 in the Informix 15 container (needed for LOB tests)
|
||||
tests/setup-spaces.sh $(IFX_CONTAINER)
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Server-compatibility matrix: Informix 12.10 / 14.10 alongside 15
|
||||
# ----------------------------------------------------------------------------
|
||||
# The legacy containers run on 9089 / 9090 so they coexist with the primary
|
||||
# 15 container on 9088. The integration suite targets whichever server
|
||||
# IFX_PORT names, so the matrix is just the same suite three times.
|
||||
|
||||
ifx-legacy-up: ## Start Informix 12.10 (9089) and 14.10 (9090) alongside 15
|
||||
docker compose -f tests/docker-compose.legacy.yml up -d
|
||||
@echo " 12.10 -> 127.0.0.1:9089"
|
||||
@echo " 14.10 -> 127.0.0.1:9090"
|
||||
@echo " Engine init takes ~1-3 min on first start; then: make ifx-legacy-setup"
|
||||
|
||||
ifx-legacy-setup: ## Create blobspace1 + sbspace1 in both legacy containers
|
||||
tests/setup-spaces.sh informix-db-test-1210
|
||||
tests/setup-spaces.sh informix-db-test-1410
|
||||
|
||||
ifx-legacy-down: ## Stop and remove the legacy containers
|
||||
docker compose -f tests/docker-compose.legacy.yml down
|
||||
|
||||
test-matrix: ## Run the integration suite against 15 (9088), 14.10 (9090), and 12.10 (9089)
|
||||
@echo "=== Informix 15 (9088) ==="
|
||||
@IFX_PORT=9088 uv run pytest -m "integration and not benchmark" -q --no-header
|
||||
@echo "=== Informix 14.10 (9090) ==="
|
||||
@IFX_PORT=9090 uv run pytest -m "integration and not benchmark" -q --no-header
|
||||
@echo "=== Informix 12.10 (9089) ==="
|
||||
@IFX_PORT=9089 uv run pytest -m "integration and not benchmark" -q --no-header
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Phase 0 spike: re-capture wire traffic against the dev container
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
90
README.md
90
README.md
@ -1,14 +1,16 @@
|
||||
# informix-db
|
||||
# informix-driver
|
||||
|
||||
Pure-Python driver for IBM Informix IDS, speaking the SQLI wire protocol over raw sockets. **No IBM Client SDK. No JVM. No native libraries.** PEP 249 compliant; sync + async APIs; built-in connection pool; TLS support.
|
||||
|
||||
**Docs:** [informix-driver.warehack.ing](https://informix-driver.warehack.ing) · **Source:** [git.supported.systems/warehack.ing/informix-db](https://git.supported.systems/warehack.ing/informix-db) · **PyPI:** [informix-driver](https://pypi.org/project/informix-driver/)
|
||||
|
||||
To our knowledge this is the **first pure-socket Informix driver in any language** — every other Informix driver (`IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix`) wraps either IBM's CSDK or the JDBC JAR.
|
||||
|
||||
```bash
|
||||
pip install informix-db
|
||||
pip install informix-driver
|
||||
```
|
||||
|
||||
Requires Python ≥ 3.10.
|
||||
Imports as `informix_db` (the distribution name is `informix-driver` because the legacy `informixdb` package on PyPI from 2008 reserves close-by names — same separation Pillow uses with `import PIL`). Requires Python ≥ 3.10.
|
||||
|
||||
## Status
|
||||
|
||||
@ -25,7 +27,7 @@ Requires Python ≥ 3.10.
|
||||
|
||||
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
|
||||
|
||||
**Test coverage:** 300+ tests across unit / integration / benchmark suites. Integration tests run against the official IBM Informix Developer Edition Docker image (15.0.1.0.3DE).
|
||||
**Test coverage:** 400+ tests across unit / integration / benchmark suites. The integration suite passes 414/414 against **each** of Informix 12.10, 14.10, and 15 — `make test-matrix` runs all three.
|
||||
|
||||
## Quick start
|
||||
|
||||
@ -100,10 +102,11 @@ Informix uses dedicated TLS-enabled listener ports (configured server-side in `s
|
||||
|
||||
| SQL type | Python type |
|
||||
|---|---|
|
||||
| `SMALLINT` / `INT` / `BIGINT` / `SERIAL` | `int` |
|
||||
| `SMALLINT` / `INT` / `SERIAL` / `BIGINT` / `BIGSERIAL` | `int` |
|
||||
| `INT8` / `SERIAL8` | `int` (legacy 64-bit — a different wire format from `BIGINT`, not an alias) |
|
||||
| `FLOAT` / `SMALLFLOAT` | `float` |
|
||||
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` |
|
||||
| `CHAR` / `VARCHAR` / `NCHAR` / `NVCHAR` / `LVARCHAR` | `str` |
|
||||
| `CHAR` / `NCHAR` (fixed width) · `VARCHAR` / `NVCHAR` / `LVARCHAR` (variable) | `str` |
|
||||
| `BOOLEAN` | `bool` |
|
||||
| `DATE` | `datetime.date` |
|
||||
| `DATETIME YEAR TO ...` | `datetime.datetime` / `datetime.time` / `datetime.date` |
|
||||
@ -131,7 +134,7 @@ cur.write_blob_column(
|
||||
)
|
||||
```
|
||||
|
||||
Both work end-to-end in pure Python via the `lotofile` / `filetoblob` server functions intercepted at the `SQ_FILE` (98) wire-protocol level — no native machinery anywhere in the thread of execution. See [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) §10–11 for the architecture pivot that made this possible.
|
||||
Both work end-to-end in pure Python via the `lotofile` / `filetoblob` server functions intercepted at the `SQ_FILE` (98) wire-protocol level — no native machinery anywhere in the thread of execution. See [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) §10–11 for the architecture pivot that made this possible.
|
||||
|
||||
## Direct stored-procedure invocation (fast-path)
|
||||
|
||||
@ -147,9 +150,46 @@ The fast-path RPC (`SQ_FPROUTINE` / `SQ_EXFPROUTINE`) bypasses PREPARE → EXECU
|
||||
|
||||
## Server compatibility
|
||||
|
||||
Tested against IBM Informix Dynamic Server **15.0.1.0.3DE** (the official `icr.io/informix/informix-developer-database` Docker image). The wire protocol is stable across modern Informix versions; should work against 12.10+ unmodified.
|
||||
All three tested against the official IBM developer-edition Docker images, full integration suite, same commit:
|
||||
|
||||
For features that need server-side configuration (smart-LOBs, logged transactions), see [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md):
|
||||
| Server | Image | Integration suite |
|
||||
|---|---|---|
|
||||
| **15.0.1.0.3DE** | `icr.io/informix/informix-developer-database` | **414 / 414** |
|
||||
| **14.10.FC7W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
|
||||
| **12.10.FC12W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
|
||||
|
||||
Reproduce the whole matrix:
|
||||
|
||||
```bash
|
||||
make ifx-up # Informix 15 on 9088
|
||||
make ifx-legacy-up # 12.10 on 9089, 14.10 on 9090
|
||||
make ifx-legacy-setup # blobspace1 + sbspace1 (smart-LOB tests need these)
|
||||
make test-matrix # the suite against all three
|
||||
```
|
||||
|
||||
The suite targets whichever server `IFX_PORT` names, so a single version is just `IFX_PORT=9089 uv run pytest -m "integration and not benchmark"`.
|
||||
|
||||
Beyond the suite passing, a 28-type round-trip (every scalar type we support, including the fiddly ones — `INT8`, `NCHAR`, `BOOLEAN`, `DATETIME YEAR TO FRACTION(5)`, `INTERVAL`, `DECIMAL`, `MONEY`, `LVARCHAR`) produces **byte-identical wire output** on all three versions: same type codes, same encoded lengths, same values.
|
||||
|
||||
Earlier releases of this README claimed the protocol was "stable across modern versions" and should "work against 12.10+ unmodified." That claim turned out to be correct, but it was written before anyone had run it against 12.10, and a user lost debugging time to a version-compatibility problem that didn't exist. It's measured now. Apologies for the earlier guess.
|
||||
|
||||
### Capability negotiation
|
||||
|
||||
SQLI settles some wire framing through a capability exchange (`SQ_PROTOCOLS`) rather than fixing it by version. This driver decodes that exchange and checks it against the framing it emits:
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(...)
|
||||
conn.server_version # release, e.g. '…Version 12.10.FC6' (one cached query)
|
||||
conn.server_version_internal # raw login string; 12.10 announces itself as 9.56
|
||||
conn.server_capabilities.four_byte_offset # True
|
||||
conn.server_capabilities.violated_assumptions() # [] — hardcoded framing matches
|
||||
```
|
||||
|
||||
All three servers above negotiate an identical 64-bit capability set, and `violated_assumptions()` is empty on each. If it ever returns something, the driver logs a warning naming the specific bit at connect time — a framing mismatch otherwise shows up as silently corrupted rows, which is a miserable thing to debug.
|
||||
|
||||
The framing itself is still hardcoded rather than driven by those bits. That's a deliberate limit: it's correct everywhere we've measured, and rewriting working parse paths to be conditional without a server that needs it would trade certainty for risk. If you hit corrupted rows on a server outside that table, check `violated_assumptions()` first, then please open an issue with your server version, the capability mask, and the `cursor.description` of the offending query.
|
||||
|
||||
For features that need server-side configuration (smart-LOBs, logged transactions), see [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md):
|
||||
- Phase 7 — logged-DB transactions
|
||||
- Phase 8 — BYTE/TEXT (needs blobspace)
|
||||
- Phase 10/11 — BLOB/CLOB (needs sbspace + `SBSPACENAME` config + level-0 archive)
|
||||
@ -168,31 +208,31 @@ Single-connection benchmarks against the dev container on loopback:
|
||||
| Pool acquire + query + release | 295 µs | ~3.4K queries/sec |
|
||||
| Cold connect (login handshake) | 11 ms | ~90 connections/sec |
|
||||
|
||||
**Performance gotcha**: `executemany(...)` under `autocommit=True` is **53× slower** than the same call inside a single transaction (server flushes the transaction log per row). For bulk loads, `autocommit=False` (default) + `conn.commit()` at the end. See [`docs/USAGE.md`](docs/USAGE.md) for the full performance tips section.
|
||||
**Performance gotcha**: `executemany(...)` under `autocommit=True` is **53× slower** than the same call inside a single transaction (server flushes the transaction log per row). For bulk loads, `autocommit=False` (default) + `conn.commit()` at the end. See [`docs/USAGE.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/USAGE.md) for the full performance tips section.
|
||||
|
||||
### Compared to IfxPy (the C-bound PyPI driver)
|
||||
|
||||
Head-to-head benchmarks against [IfxPy](https://pypi.org/project/IfxPy/) on identical workloads, same Informix server, matched conditions. Using **median + IQR over 10+ rounds** to resist outlier-round noise:
|
||||
|
||||
| Benchmark | IfxPy 3.0.5 (C-bound) | `informix-db` (pure Python) | Result |
|
||||
| Benchmark | IfxPy 3.0.5 (C-bound) | `informix-driver` (pure Python) | Result |
|
||||
|---|---:|---:|---:|
|
||||
| Single-row SELECT round-trip | 118 µs | 114 µs | comparable |
|
||||
| ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster |
|
||||
| Cold connect (login handshake) | 11.0 ms | 10.5 ms | comparable |
|
||||
| **`executemany(1k)` in transaction** | 23.5 ms | 23.2 ms | tied |
|
||||
| **`executemany(10k)` in transaction** | 259 ms | **161 ms** | **`informix-db` 1.6× faster** |
|
||||
| **`executemany(100k)` in transaction** | 2376 ms | **1487 ms** | **`informix-db` 1.6× faster** |
|
||||
| **`executemany(10k)` in transaction** | 259 ms | **161 ms** | **`informix-driver` 1.6× faster** |
|
||||
| **`executemany(100k)` in transaction** | 2376 ms | **1487 ms** | **`informix-driver` 1.6× faster** |
|
||||
| `SELECT` 1k rows | 1.2 ms | 2.7 ms | IfxPy 2.3× faster |
|
||||
| `SELECT` 10k rows | 11.3 ms | 25.8 ms | IfxPy 2.3× faster |
|
||||
| `SELECT` 100k rows | 112 ms | 271 ms | IfxPy 2.4× faster |
|
||||
|
||||
**The honest summary:**
|
||||
|
||||
- **Bulk-insert workloads: `informix-db` wins 1.6× at scale.** The pipelined `executemany` (Phase 33) sends all N BIND+EXECUTE PDUs before draining responses, eliminating per-row RTT. IfxPy still pays one round-trip per `IfxPy.execute(stmt, tuple)` call.
|
||||
- **Bulk-insert workloads: `informix-driver` wins 1.6× at scale.** The pipelined `executemany` (Phase 33) sends all N BIND+EXECUTE PDUs before draining responses, eliminating per-row RTT. IfxPy still pays one round-trip per `IfxPy.execute(stmt, tuple)` call.
|
||||
- **Large-fetch workloads: IfxPy wins 2.3× at scale.** Their C-level `fetch_tuple` decoder is genuinely faster than our Python `parse_tuple_payload` (~1.1 µs/row vs ~2.7 µs/row). At 100k rows, that 1.6 µs/row gap accumulates into a 160 ms wall-clock difference.
|
||||
- **Small queries: comparable.** Both spend ~120 µs waiting for the server; the per-call codec cost is small relative to the round-trip.
|
||||
|
||||
**When to prefer `informix-db`:**
|
||||
**When to prefer `informix-driver`:**
|
||||
- ETL pipelines, log shipping, bulk writes (1.6× faster at scale)
|
||||
- Containerized / minimal-dependency environments (50 KB wheel vs IfxPy's 92 MB OneDB tarball + libcrypt.so.1 dependency hell)
|
||||
- Modern Python (works on 3.10–3.14; IfxPy is broken on Python 3.12+)
|
||||
@ -204,9 +244,9 @@ Head-to-head benchmarks against [IfxPy](https://pypi.org/project/IfxPy/) on iden
|
||||
|
||||
These results are reproducible from `tests/benchmarks/compare/` — the Dockerfile, bench script, and README walk through every step.
|
||||
|
||||
Full methodology, IQR caveats, install gauntlet, and reproduction in [`tests/benchmarks/compare/README.md`](tests/benchmarks/compare/README.md).
|
||||
Full methodology, IQR caveats, install gauntlet, and reproduction in [`tests/benchmarks/compare/README.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare/README.md).
|
||||
|
||||
A note on IfxPy's install gauntlet: getting it to run on a modern system requires Python ≤ 3.11, setuptools <58, permissive CFLAGS, manual download of a 92 MB ODBC tarball, four `LD_LIBRARY_PATH` directories, and `libcrypt.so.1` (deprecated 2018, missing on Arch / Fedora 35+ / RHEL 9). `informix-db`'s install: `pip install informix-db`.
|
||||
A note on IfxPy's install gauntlet: getting it to run on a modern system requires Python ≤ 3.11, setuptools <58, permissive CFLAGS, manual download of a 92 MB ODBC tarball, four `LD_LIBRARY_PATH` directories, and `libcrypt.so.1` (deprecated 2018, missing on Arch / Fedora 35+ / RHEL 9). `informix-driver`'s install: `pip install informix-driver`.
|
||||
|
||||
## Standards & guarantees
|
||||
|
||||
@ -217,7 +257,7 @@ A note on IfxPy's install gauntlet: getting it to run on a modern system require
|
||||
|
||||
## Development
|
||||
|
||||
The full test + lint workflow is in the [Makefile](Makefile). Quick summary:
|
||||
The full test + lint workflow is in the [Makefile](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/Makefile). Quick summary:
|
||||
|
||||
```bash
|
||||
make test # 77 unit tests (no Docker)
|
||||
@ -226,22 +266,22 @@ make bench # benchmark suite
|
||||
make lint # ruff
|
||||
```
|
||||
|
||||
For the smart-LOB tests specifically, the dev container needs additional one-time setup (blobspace + sbspace + level-0 archive). See [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) §10 for the `onspaces` / `onmode` / `ontape` commands.
|
||||
For the smart-LOB tests specifically, the dev container needs additional one-time setup (blobspace + sbspace + level-0 archive). See [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) §10 for the `onspaces` / `onmode` / `ontape` commands.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [**`docs/USAGE.md`**](docs/USAGE.md) — practical recipes: connections, parameter binding, type mapping, transactions, performance tips, scrollable cursors, BLOBs, async, TLS, locale/Unicode, error handling, known limitations
|
||||
- [`tests/benchmarks/README.md`](tests/benchmarks/README.md) — performance baselines, headline numbers, how to run regressions
|
||||
- [**`docs/USAGE.md`**](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/USAGE.md) — practical recipes: connections, parameter binding, type mapping, transactions, performance tips, scrollable cursors, BLOBs, async, TLS, locale/Unicode, error handling, known limitations
|
||||
- [`tests/benchmarks/README.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/README.md) — performance baselines, headline numbers, how to run regressions
|
||||
- `CHANGELOG.md` — phase-by-phase release notes
|
||||
|
||||
## Project history & design rationale
|
||||
|
||||
This driver was built incrementally across 30 phases, each with a focused scope and decision log. The reasoning trail lives in:
|
||||
|
||||
- [`docs/PROTOCOL_NOTES.md`](docs/PROTOCOL_NOTES.md) — byte-level SQLI wire-format reference
|
||||
- [`docs/JDBC_NOTES.md`](docs/JDBC_NOTES.md) — index into the decompiled IBM JDBC driver, used as a clean-room reference
|
||||
- [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) — phase-by-phase architectural decisions, with the *why* preserved
|
||||
- [`docs/CAPTURES/`](docs/CAPTURES/) — annotated socat hex-dump captures
|
||||
- [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) — byte-level SQLI wire-format reference
|
||||
- [`docs/JDBC_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/JDBC_NOTES.md) — index into the decompiled IBM JDBC driver, used as a clean-room reference
|
||||
- [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) — phase-by-phase architectural decisions, with the *why* preserved
|
||||
- [`docs/CAPTURES/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/CAPTURES/) — annotated socat hex-dump captures
|
||||
|
||||
Notable architectural pivots documented in the decision log:
|
||||
- **Phase 10/11** (smart-LOB read/write): used `lotofile`/`filetoblob` SQL functions + `SQ_FILE` protocol intercept instead of the heavier `SQ_FPROUTINE` + `SQ_LODATA` stack — ~3x smaller than originally projected
|
||||
|
||||
12
docs-site/.dockerignore
Normal file
12
docs-site/.dockerignore
Normal file
@ -0,0 +1,12 @@
|
||||
node_modules
|
||||
dist
|
||||
.astro
|
||||
.env
|
||||
.env.local
|
||||
.git
|
||||
*.log
|
||||
README.md
|
||||
.dockerignore
|
||||
Dockerfile
|
||||
docker-compose.yml
|
||||
Makefile
|
||||
4
docs-site/.env.example
Normal file
4
docs-site/.env.example
Normal file
@ -0,0 +1,4 @@
|
||||
COMPOSE_PROJECT=informix-db-docs
|
||||
DOMAIN=informix-driver.warehack.ing
|
||||
DEV_DOMAIN=informix-driver.l.warehack.ing
|
||||
MODE=prod
|
||||
21
docs-site/.gitignore
vendored
Normal file
21
docs-site/.gitignore
vendored
Normal file
@ -0,0 +1,21 @@
|
||||
# build output
|
||||
dist/
|
||||
# generated types
|
||||
.astro/
|
||||
|
||||
# dependencies
|
||||
node_modules/
|
||||
|
||||
# logs
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
pnpm-debug.log*
|
||||
|
||||
|
||||
# environment variables
|
||||
.env
|
||||
.env.production
|
||||
|
||||
# macOS-specific files
|
||||
.DS_Store
|
||||
20
docs-site/Caddyfile
Normal file
20
docs-site/Caddyfile
Normal file
@ -0,0 +1,20 @@
|
||||
:80 {
|
||||
root * /srv
|
||||
encode zstd gzip
|
||||
file_server
|
||||
|
||||
@assets path *.css *.js *.woff2 *.woff *.svg *.png *.webp *.jpg *.jpeg *.ico
|
||||
header @assets Cache-Control "public, max-age=31536000, immutable"
|
||||
|
||||
@html path *.html
|
||||
header @html Cache-Control "public, max-age=300, must-revalidate"
|
||||
header @html X-Content-Type-Options "nosniff"
|
||||
header @html X-Frame-Options "SAMEORIGIN"
|
||||
header @html Referrer-Policy "strict-origin-when-cross-origin"
|
||||
|
||||
handle_errors {
|
||||
@404 expression `{err.status_code} == 404`
|
||||
rewrite @404 /404.html
|
||||
file_server
|
||||
}
|
||||
}
|
||||
30
docs-site/Dockerfile
Normal file
30
docs-site/Dockerfile
Normal file
@ -0,0 +1,30 @@
|
||||
ARG NODE_VERSION=22-alpine
|
||||
ARG CADDY_VERSION=2.10-alpine
|
||||
|
||||
FROM node:${NODE_VERSION} AS deps
|
||||
WORKDIR /app
|
||||
COPY package.json package-lock.json* ./
|
||||
RUN --mount=type=cache,target=/root/.npm \
|
||||
npm ci --no-audit --no-fund 2>/dev/null || npm install --no-audit --no-fund
|
||||
|
||||
FROM node:${NODE_VERSION} AS builder
|
||||
WORKDIR /app
|
||||
ENV ASTRO_TELEMETRY_DISABLED=1 NODE_ENV=production
|
||||
COPY --from=deps /app/node_modules ./node_modules
|
||||
COPY . .
|
||||
RUN npm run build
|
||||
|
||||
FROM caddy:${CADDY_VERSION} AS prod
|
||||
COPY --from=builder /app/dist /srv
|
||||
COPY Caddyfile /etc/caddy/Caddyfile
|
||||
EXPOSE 80
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD wget -q --spider http://127.0.0.1/ || exit 1
|
||||
|
||||
FROM node:${NODE_VERSION} AS dev
|
||||
WORKDIR /app
|
||||
ENV ASTRO_TELEMETRY_DISABLED=1 NODE_ENV=development
|
||||
COPY --from=deps /app/node_modules ./node_modules
|
||||
COPY . .
|
||||
EXPOSE 4321
|
||||
CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0"]
|
||||
33
docs-site/Makefile
Normal file
33
docs-site/Makefile
Normal file
@ -0,0 +1,33 @@
|
||||
.PHONY: prod dev down logs build clean install local
|
||||
|
||||
include .env
|
||||
export
|
||||
|
||||
prod:
|
||||
docker compose --profile prod up -d --build
|
||||
|
||||
dev:
|
||||
docker compose --profile dev up -d --build
|
||||
|
||||
down:
|
||||
docker compose --profile prod --profile dev down
|
||||
|
||||
logs:
|
||||
docker compose logs -f --tail=200
|
||||
|
||||
build:
|
||||
npm run build
|
||||
|
||||
clean:
|
||||
rm -rf dist node_modules .astro
|
||||
|
||||
install:
|
||||
npm install
|
||||
|
||||
local:
|
||||
npm run dev -- --host 127.0.0.1
|
||||
|
||||
status:
|
||||
@echo "Domain: $(DOMAIN)"
|
||||
@echo "Project: $(COMPOSE_PROJECT)"
|
||||
@docker compose ps
|
||||
49
docs-site/README.md
Normal file
49
docs-site/README.md
Normal file
@ -0,0 +1,49 @@
|
||||
# Starlight Starter Kit: Basics
|
||||
|
||||
[](https://starlight.astro.build)
|
||||
|
||||
```
|
||||
npm create astro@latest -- --template starlight
|
||||
```
|
||||
|
||||
> 🧑🚀 **Seasoned astronaut?** Delete this file. Have fun!
|
||||
|
||||
## 🚀 Project Structure
|
||||
|
||||
Inside of your Astro + Starlight project, you'll see the following folders and files:
|
||||
|
||||
```
|
||||
.
|
||||
├── public/
|
||||
├── src/
|
||||
│ ├── assets/
|
||||
│ ├── content/
|
||||
│ │ └── docs/
|
||||
│ └── content.config.ts
|
||||
├── astro.config.mjs
|
||||
├── package.json
|
||||
└── tsconfig.json
|
||||
```
|
||||
|
||||
Starlight looks for `.md` or `.mdx` files in the `src/content/docs/` directory. Each file is exposed as a route based on its file name.
|
||||
|
||||
Images can be added to `src/assets/` and embedded in Markdown with a relative link.
|
||||
|
||||
Static assets, like favicons, can be placed in the `public/` directory.
|
||||
|
||||
## 🧞 Commands
|
||||
|
||||
All commands are run from the root of the project, from a terminal:
|
||||
|
||||
| Command | Action |
|
||||
| :------------------------ | :----------------------------------------------- |
|
||||
| `npm install` | Installs dependencies |
|
||||
| `npm run dev` | Starts local dev server at `localhost:4321` |
|
||||
| `npm run build` | Build your production site to `./dist/` |
|
||||
| `npm run preview` | Preview your build locally, before deploying |
|
||||
| `npm run astro ...` | Run CLI commands like `astro add`, `astro check` |
|
||||
| `npm run astro -- --help` | Get help using the Astro CLI |
|
||||
|
||||
## 👀 Want to learn more?
|
||||
|
||||
Check out [Starlight’s docs](https://starlight.astro.build/), read [the Astro documentation](https://docs.astro.build), or jump into the [Astro Discord server](https://astro.build/chat).
|
||||
119
docs-site/astro.config.mjs
Normal file
119
docs-site/astro.config.mjs
Normal file
@ -0,0 +1,119 @@
|
||||
// @ts-check
|
||||
import { defineConfig } from 'astro/config';
|
||||
import starlight from '@astrojs/starlight';
|
||||
|
||||
const DEV_DOMAIN = process.env.DEV_DOMAIN ?? 'informix-driver.l.warehack.ing';
|
||||
|
||||
// https://astro.build/config
|
||||
export default defineConfig({
|
||||
site: 'https://informix-driver.warehack.ing',
|
||||
server: { host: '0.0.0.0', port: 4321 },
|
||||
telemetry: false,
|
||||
devToolbar: { enabled: false },
|
||||
vite: {
|
||||
server: {
|
||||
host: '0.0.0.0',
|
||||
hmr: {
|
||||
host: DEV_DOMAIN,
|
||||
protocol: 'wss',
|
||||
clientPort: 443,
|
||||
},
|
||||
allowedHosts: [DEV_DOMAIN, '.warehack.ing', 'localhost', '127.0.0.1'],
|
||||
},
|
||||
},
|
||||
integrations: [
|
||||
starlight({
|
||||
title: 'informix-driver',
|
||||
description: 'Pure-Python driver for IBM Informix IDS. No CSDK, no JVM, no native libraries.',
|
||||
logo: { src: './src/assets/logo.svg', replacesTitle: false },
|
||||
favicon: '/favicon.svg',
|
||||
tableOfContents: { minHeadingLevel: 2, maxHeadingLevel: 4 },
|
||||
lastUpdated: true,
|
||||
pagination: true,
|
||||
editLink: {
|
||||
baseUrl: 'https://git.supported.systems/warehack.ing/informix-db/_edit/branch/main/docs-site/',
|
||||
},
|
||||
social: [
|
||||
{ icon: 'seti:git', label: 'Source (Gitea)', href: 'https://git.supported.systems/warehack.ing/informix-db' },
|
||||
{ icon: 'seti:python', label: 'PyPI', href: 'https://pypi.org/project/informix-driver/' },
|
||||
],
|
||||
customCss: ['./src/styles/theme.css', './src/styles/components.css'],
|
||||
components: {
|
||||
Hero: './src/components/Hero.astro',
|
||||
Footer: './src/components/Footer.astro',
|
||||
},
|
||||
expressiveCode: {
|
||||
themes: ['github-dark', 'github-light'],
|
||||
styleOverrides: {
|
||||
borderRadius: '6px',
|
||||
codeFontFamily: "'IBM Plex Mono', ui-monospace, SFMono-Regular, Menlo, monospace",
|
||||
},
|
||||
},
|
||||
head: [
|
||||
{
|
||||
tag: 'link',
|
||||
attrs: { rel: 'preconnect', href: 'https://rsms.me' },
|
||||
},
|
||||
{
|
||||
tag: 'link',
|
||||
attrs: {
|
||||
rel: 'stylesheet',
|
||||
href: 'https://rsms.me/inter/inter.css',
|
||||
},
|
||||
},
|
||||
{
|
||||
tag: 'meta',
|
||||
attrs: { name: 'theme-color', content: '#0e0d0c' },
|
||||
},
|
||||
{
|
||||
tag: 'meta',
|
||||
attrs: { property: 'og:type', content: 'website' },
|
||||
},
|
||||
],
|
||||
sidebar: [
|
||||
{
|
||||
label: 'Start here',
|
||||
items: [
|
||||
{ label: 'WTF did you build this for?', slug: 'start/wtf' },
|
||||
{ label: 'Install & first query', slug: 'start/quickstart' },
|
||||
{ label: 'Compared to IfxPy', slug: 'start/vs-ifxpy' },
|
||||
],
|
||||
},
|
||||
{
|
||||
label: 'How-to guides',
|
||||
items: [
|
||||
{ label: 'Connect with TLS', slug: 'how-to/tls' },
|
||||
{ label: 'Use the connection pool', slug: 'how-to/pool' },
|
||||
{ label: 'Async with FastAPI', slug: 'how-to/async-fastapi' },
|
||||
{ label: 'Bulk inserts (executemany)', slug: 'how-to/executemany' },
|
||||
{ label: 'Optimize bulk SELECT', slug: 'how-to/buffered-reader' },
|
||||
{ label: 'BLOB / CLOB read & write', slug: 'how-to/smart-lobs' },
|
||||
{ label: 'Migrate from IfxPy', slug: 'how-to/migrate-from-ifxpy' },
|
||||
{ label: 'Run the dev container', slug: 'how-to/dev-container' },
|
||||
],
|
||||
},
|
||||
{
|
||||
label: 'Reference',
|
||||
items: [
|
||||
{ label: 'API surface', slug: 'reference/api' },
|
||||
{ label: 'SQL ↔ Python types', slug: 'reference/types' },
|
||||
{ label: 'Configuration & env flags', slug: 'reference/config' },
|
||||
{ label: 'Exceptions & error codes', slug: 'reference/exceptions' },
|
||||
{ label: 'Performance baselines', slug: 'reference/benchmarks' },
|
||||
],
|
||||
},
|
||||
{
|
||||
label: 'Explanation',
|
||||
items: [
|
||||
{ label: 'The SQLI wire protocol', slug: 'explain/sqli-protocol' },
|
||||
{ label: 'Architecture overview', slug: 'explain/architecture' },
|
||||
{ label: 'The buffered reader (Phase 39)', slug: 'explain/buffered-reader' },
|
||||
{ label: 'Async strategy', slug: 'explain/async-strategy' },
|
||||
{ label: 'Pure-Python tradeoffs', slug: 'explain/pure-python' },
|
||||
{ label: 'The phase log', slug: 'explain/phase-log' },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
],
|
||||
});
|
||||
46
docs-site/docker-compose.yml
Normal file
46
docs-site/docker-compose.yml
Normal file
@ -0,0 +1,46 @@
|
||||
services:
|
||||
docs:
|
||||
profiles: ["prod"]
|
||||
build:
|
||||
context: .
|
||||
target: prod
|
||||
container_name: ${COMPOSE_PROJECT}-prod
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- caddy
|
||||
labels:
|
||||
caddy: ${DOMAIN}
|
||||
caddy.reverse_proxy: "{{upstreams 80}}"
|
||||
|
||||
docs-dev:
|
||||
profiles: ["dev"]
|
||||
build:
|
||||
context: .
|
||||
target: dev
|
||||
container_name: ${COMPOSE_PROJECT}-dev
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./src:/app/src:cached
|
||||
- ./public:/app/public:cached
|
||||
- ./astro.config.mjs:/app/astro.config.mjs:cached
|
||||
- ./tsconfig.json:/app/tsconfig.json:cached
|
||||
- ./package.json:/app/package.json:cached
|
||||
environment:
|
||||
DEV_DOMAIN: ${DEV_DOMAIN}
|
||||
networks:
|
||||
- caddy
|
||||
labels:
|
||||
caddy: ${DEV_DOMAIN}
|
||||
caddy.reverse_proxy: "{{upstreams 4321}}"
|
||||
caddy.reverse_proxy.flush_interval: "-1"
|
||||
caddy.reverse_proxy.transport: "http"
|
||||
caddy.reverse_proxy.transport.read_timeout: "0"
|
||||
caddy.reverse_proxy.transport.write_timeout: "0"
|
||||
caddy.reverse_proxy.transport.keepalive: "5m"
|
||||
caddy.reverse_proxy.transport.keepalive_idle_conns: "10"
|
||||
caddy.reverse_proxy.stream_timeout: "24h"
|
||||
caddy.reverse_proxy.stream_close_delay: "5s"
|
||||
|
||||
networks:
|
||||
caddy:
|
||||
external: true
|
||||
6269
docs-site/package-lock.json
generated
Normal file
6269
docs-site/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
17
docs-site/package.json
Normal file
17
docs-site/package.json
Normal file
@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "docs-site",
|
||||
"type": "module",
|
||||
"version": "0.0.1",
|
||||
"scripts": {
|
||||
"dev": "astro dev",
|
||||
"start": "astro dev",
|
||||
"build": "astro build",
|
||||
"preview": "astro preview",
|
||||
"astro": "astro"
|
||||
},
|
||||
"dependencies": {
|
||||
"@astrojs/starlight": "^0.39.1",
|
||||
"astro": "^6.2.2",
|
||||
"sharp": "^0.34.5"
|
||||
}
|
||||
}
|
||||
6
docs-site/public/favicon.svg
Normal file
6
docs-site/public/favicon.svg
Normal file
@ -0,0 +1,6 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
|
||||
<rect width="32" height="32" rx="6" fill="#0e0d0c"/>
|
||||
<rect x="8" y="11" width="6" height="6" rx="1" fill="#f5a524"/>
|
||||
<rect x="8" y="19" width="16" height="2" rx="1" fill="#f5a524"/>
|
||||
<rect x="8" y="23" width="11" height="2" rx="1" fill="#f5a524" opacity="0.55"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 338 B |
66
docs-site/public/supported-systems-logo.svg
Normal file
66
docs-site/public/supported-systems-logo.svg
Normal file
@ -0,0 +1,66 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 75" height="100%" width="100%">
|
||||
<!-- Gradient Definitions -->
|
||||
<defs>
|
||||
<linearGradient id="gradient1" x1="0%" y1="0%" x2="0%" y2="100%">
|
||||
<stop offset="0%" stop-color="#60a5fa"></stop>
|
||||
<stop offset="50%" stop-color="#3b82f6"></stop>
|
||||
<stop offset="100%" stop-color="#2563eb"></stop>
|
||||
</linearGradient>
|
||||
<linearGradient id="gradient2" x1="0%" y1="0%" x2="0%" y2="100%">
|
||||
<stop offset="0%" stop-color="#93c5fd"></stop>
|
||||
<stop offset="50%" stop-color="#60a5fa"></stop>
|
||||
<stop offset="100%" stop-color="#3b82f6"></stop>
|
||||
</linearGradient>
|
||||
<linearGradient id="flowGradient" x1="0%" y1="0%" x2="100%" y2="0%">
|
||||
<stop offset="0%" stop-color="#2563eb"></stop>
|
||||
<stop offset="50%" stop-color="#60a5fa"></stop>
|
||||
<stop offset="100%" stop-color="#2563eb"></stop>
|
||||
</linearGradient>
|
||||
<pattern id="circuitPattern" patternUnits="userSpaceOnUse" width="12" height="45" patternTransform="scale(1)">
|
||||
<rect width="12" height="45" fill="url(#gradient1)"></rect>
|
||||
<path d="M2,5 h8 M2,5 v5 M10,5 v10 M5,15 h5 M5,15 v10 M3,25 h7 M7,25 v10 M3,35 h4" stroke="#dbeafe" stroke-width="0.5" fill="none" opacity="0.7"></path>
|
||||
<circle cx="2" cy="5" r="1" fill="#dbeafe" opacity="0.7"></circle>
|
||||
<circle cx="10" cy="5" r="1" fill="#dbeafe" opacity="0.7"></circle>
|
||||
<circle cx="5" cy="15" r="1" fill="#dbeafe" opacity="0.7"></circle>
|
||||
<circle cx="3" cy="25" r="1" fill="#dbeafe" opacity="0.7"></circle>
|
||||
<circle cx="7" cy="35" r="1" fill="#dbeafe" opacity="0.7"></circle>
|
||||
</pattern>
|
||||
<pattern id="binaryPattern" patternUnits="userSpaceOnUse" width="12" height="35" patternTransform="scale(1)">
|
||||
<rect width="12" height="35" fill="#2563eb"></rect>
|
||||
<text x="3" y="8" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">10</text>
|
||||
<text x="3" y="14" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">01</text>
|
||||
<text x="3" y="20" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">11</text>
|
||||
<text x="3" y="26" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">00</text>
|
||||
<text x="3" y="32" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">10</text>
|
||||
</pattern>
|
||||
<pattern id="punchCardPattern" patternUnits="userSpaceOnUse" width="12" height="45" patternTransform="scale(1)">
|
||||
<rect width="12" height="45" fill="#3b82f6"></rect>
|
||||
<path d="M0,5 h12 M0,10 h12 M0,15 h12 M0,20 h12 M0,25 h12 M0,30 h12 M0,35 h12 M0,40 h12" stroke="#93c5fd" stroke-width="0.2" fill="none"></path>
|
||||
<circle cx="3" cy="7" r="1" fill="#1e3a8a" opacity="0.9"></circle>
|
||||
<circle cx="9" cy="7" r="1" fill="#1e3a8a" opacity="0.9"></circle>
|
||||
<circle cx="6" cy="12" r="1" fill="#1e3a8a" opacity="0.9"></circle>
|
||||
<circle cx="3" cy="17" r="1" fill="#1e3a8a" opacity="0.9"></circle>
|
||||
<circle cx="9" cy="22" r="1" fill="#1e3a8a" opacity="0.9"></circle>
|
||||
<circle cx="6" cy="27" r="1" fill="#1e3a8a" opacity="0.9"></circle>
|
||||
<circle cx="3" cy="32" r="1" fill="#1e3a8a" opacity="0.9"></circle>
|
||||
<circle cx="9" cy="37" r="1" fill="#1e3a8a" opacity="0.9"></circle>
|
||||
</pattern>
|
||||
</defs>
|
||||
<!-- Flow lines behind bars -->
|
||||
<g opacity="0.3">
|
||||
<path d="M6,50 C20,40 40,55 48,35 C56,50 75,30 90,55" stroke="url(#flowGradient)" stroke-width="1" fill="none"></path>
|
||||
<path d="M6,60 C30,50 50,40 70,55 C80,45 90,60 90,60" stroke="url(#flowGradient)" stroke-width="1" fill="none"></path>
|
||||
</g>
|
||||
<!-- Bar chart graphic - the "towers" -->
|
||||
<g>
|
||||
<rect x="0" y="45" width="12" height="25" rx="1" ry="1" fill="url(#binaryPattern)"></rect>
|
||||
<rect x="14" y="35" width="12" height="35" rx="1" ry="1" fill="#2563eb"></rect>
|
||||
<rect x="28" y="25" width="12" height="45" rx="1" ry="1" fill="url(#circuitPattern)"></rect>
|
||||
<rect x="42" y="20" width="12" height="50" rx="1" ry="1" fill="url(#gradient2)"></rect>
|
||||
<rect x="56" y="25" width="12" height="45" rx="1" ry="1" fill="url(#punchCardPattern)"></rect>
|
||||
<rect x="70" y="35" width="12" height="35" rx="1" ry="1" fill="url(#circuitPattern)"></rect>
|
||||
<rect x="84" y="45" width="12" height="25" rx="1" ry="1" fill="#2563eb"></rect>
|
||||
<!-- Connecting glow -->
|
||||
<path d="M12,55 L14,55 M26,45 L28,45 M40,40 L42,40 M54,40 L56,40 M82,55 L84,55" stroke="#bfdbfe" stroke-width="0.8" stroke-opacity="0.6"></path>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.3 KiB |
7
docs-site/src/assets/logo.svg
Normal file
7
docs-site/src/assets/logo.svg
Normal file
@ -0,0 +1,7 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 28 24" fill="none" role="img" aria-label="informix-driver">
|
||||
<title>informix-driver</title>
|
||||
<rect x="0" y="4" width="5" height="16" rx="1" fill="#f5a524"/>
|
||||
<rect x="8" y="4" width="3" height="16" rx="1" fill="#f5a524" fill-opacity="0.78"/>
|
||||
<rect x="14" y="4" width="3" height="16" rx="1" fill="#f5a524" fill-opacity="0.52"/>
|
||||
<rect x="20" y="4" width="3" height="16" rx="1" fill="#f5a524" fill-opacity="0.28"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 481 B |
43
docs-site/src/components/Footer.astro
Normal file
43
docs-site/src/components/Footer.astro
Normal file
@ -0,0 +1,43 @@
|
||||
---
|
||||
// Footer override that preserves Starlight's default chrome
|
||||
// (prev/next pagination, last-updated, edit link) and appends
|
||||
// a Supported Systems "joint" badge below.
|
||||
import Default from '@astrojs/starlight/components/Footer.astro';
|
||||
---
|
||||
|
||||
<Default><slot /></Default>
|
||||
|
||||
<aside class="ifx-ss-badge" aria-label="Supported Systems">
|
||||
<a class="ifx-ss-badge__link" href="https://supported.systems" rel="noopener">
|
||||
<img
|
||||
class="ifx-ss-badge__logo"
|
||||
src="/supported-systems-logo.svg"
|
||||
alt=""
|
||||
width="60"
|
||||
height="45"
|
||||
loading="lazy"
|
||||
/>
|
||||
<div class="ifx-ss-badge__copy">
|
||||
<h3 class="ifx-ss-badge__heading">A Supported Systems Joint</h3>
|
||||
<p class="ifx-ss-badge__body">
|
||||
<code>informix-driver</code> is built and maintained by
|
||||
<span class="ifx-ss-badge__name">Supported Systems</span> — a
|
||||
boutique software studio focused on thoughtful, user-first technology.
|
||||
We take databases personally.
|
||||
</p>
|
||||
<span class="ifx-ss-badge__cta">
|
||||
Visit supported.systems
|
||||
<svg viewBox="0 0 16 16" width="14" height="14" aria-hidden="true">
|
||||
<path
|
||||
d="M4 8h7M8 5l3 3-3 3"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
stroke-width="1.5"
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
/>
|
||||
</svg>
|
||||
</span>
|
||||
</div>
|
||||
</a>
|
||||
</aside>
|
||||
79
docs-site/src/components/Hero.astro
Normal file
79
docs-site/src/components/Hero.astro
Normal file
@ -0,0 +1,79 @@
|
||||
---
|
||||
// Custom hero for the homepage. Renders the wire-dump easter egg
|
||||
// alongside the headline + CTA stack. Uses real captured SQLI bytes
|
||||
// from docs/CAPTURES/01-connect-only.socat.log (truncated and curated).
|
||||
---
|
||||
|
||||
<section class="ifx-hero">
|
||||
<div class="ifx-hero__copy">
|
||||
<span class="ifx-hero__eyebrow">Pure Python · No CSDK · No JVM · No libcrypt.so.1</span>
|
||||
<h1 class="ifx-hero__title">
|
||||
Talk to Informix without
|
||||
<strong>linking against IBM's 92 MB tarball.</strong>
|
||||
</h1>
|
||||
<p class="ifx-hero__lede">
|
||||
Every other Informix driver wraps IBM's C SDK or the JDBC JAR. We weren't into that.
|
||||
So we read the protocol and wrote it ourselves: PEP 249, sync and async, pooled, TLS.
|
||||
Within 10% of IBM's own C driver on bulk fetches, <strong>1.6× faster</strong> on
|
||||
bulk inserts. No compile step. No <code>LD_LIBRARY_PATH</code> ritual. No
|
||||
<code>libcrypt.so.1</code> from 2018.
|
||||
</p>
|
||||
<div class="ifx-hero__cta">
|
||||
<a class="primary" href="/start/quickstart/">Get started →</a>
|
||||
<a class="secondary" href="/start/vs-ifxpy/">Compared to IfxPy</a>
|
||||
<a class="secondary" href="https://git.supported.systems/warehack.ing/informix-db">Source</a>
|
||||
</div>
|
||||
<div class="ifx-hero__install">pip install informix-driver</div>
|
||||
</div>
|
||||
|
||||
<div class="ifx-hero__visual">
|
||||
<div class="ifx-wiredump" aria-hidden="true">
|
||||
<div class="ifx-wiredump__scroll" id="ifx-wire">
|
||||
<span class="ifx-wiredump__line"><span class="ifx-wiredump__direction ifx-wiredump__direction--out">{'>'} OUT</span> 01 c3 01 3c 00 00 00 64 00 65 00 00 00 3d 00 06</span>
|
||||
<span class="ifx-wiredump__line"> 49 45 45 45 4d 00 00 6c 73 71 6c 65 78 65 63 00 <span class="ifx-wiredump__byte--ascii">IEEEM..lsqlexec.</span></span>
|
||||
<span class="ifx-wiredump__line"> 00 00 00 00 00 00 06 39 2e 32 38 30 00 00 0c 52 <span class="ifx-wiredump__byte--ascii">.......9.280...R</span></span>
|
||||
<span class="ifx-wiredump__line"> 44 53 23 52 30 30 30 30 30 30 00 00 05 73 71 6c <span class="ifx-wiredump__byte--ascii">DS#R000000...sql</span></span>
|
||||
<span class="ifx-wiredump__line"> 69 00 00 00 01 3c 00 00 00 00 00 00 00 00 00 01 <span class="ifx-wiredump__byte--ascii">i....{'<'}.........</span></span>
|
||||
<span class="ifx-wiredump__line"> 00 09 69 6e 66 6f 72 6d 69 78 00 00 07 69 6e 34 <span class="ifx-wiredump__byte--ascii">..informix...in4</span></span>
|
||||
<span class="ifx-wiredump__line"></span>
|
||||
<span class="ifx-wiredump__line"><span class="ifx-wiredump__direction ifx-wiredump__direction--in">{'<'} IN </span> 01 14 02 3c 10 00 00 64 00 65 00 00 00 3d 00 06</span>
|
||||
<span class="ifx-wiredump__line"> 49 45 45 45 49 00 00 6c 73 72 76 69 6e 66 78 00 <span class="ifx-wiredump__byte--ascii">IEEEI..lsrvinfx.</span></span>
|
||||
<span class="ifx-wiredump__line"> 00 00 00 00 00 00 2f 49 42 4d 20 49 6e 66 6f 72 <span class="ifx-wiredump__byte--ascii">....../IBM Infor</span></span>
|
||||
<span class="ifx-wiredump__line"> 6d 69 78 20 44 79 6e 61 6d 69 63 20 53 65 72 76 <span class="ifx-wiredump__byte--ascii">mix Dynamic Serv</span></span>
|
||||
<span class="ifx-wiredump__line"> 65 72 20 56 65 72 73 69 6f 6e 20 31 35 2e 30 2e <span class="ifx-wiredump__byte--ascii">er Version 15.0.</span></span>
|
||||
<span class="ifx-wiredump__line"></span>
|
||||
<span class="ifx-wiredump__line"><span class="ifx-wiredump__direction ifx-wiredump__direction--out">{'>'} OUT</span> 00 02 00 00 00 00 00 49 73 65 6c 65 63 74 20 46 <span class="ifx-wiredump__byte--ascii">.......Iselect F</span></span>
|
||||
<span class="ifx-wiredump__line"> 49 52 53 54 20 31 20 73 69 74 65 20 66 72 6f 6d <span class="ifx-wiredump__byte--ascii">IRST 1 site from</span></span>
|
||||
<span class="ifx-wiredump__line"> 20 69 6e 66 6f 72 6d 69 78 2e 73 79 73 74 61 62 <span class="ifx-wiredump__byte--ascii"> informix.systab</span></span>
|
||||
<span class="ifx-wiredump__line"> 6c 65 73 20 77 68 65 72 65 20 74 61 62 6e 61 6d <span class="ifx-wiredump__byte--ascii">les where tabnam</span></span>
|
||||
<span class="ifx-wiredump__line"> 65 20 3d 20 27 20 47 4c 5f 43 4f 4c 4c 41 54 45 <span class="ifx-wiredump__byte--ascii">e = ' GL_COLLATE</span></span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="ifx-wiredump__caption">
|
||||
live capture · 01-connect-only.socat · sqli/9088
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<script>
|
||||
// Reveal lines progressively with reduced-motion respect.
|
||||
const reduce = window.matchMedia('(prefers-reduced-motion: reduce)').matches;
|
||||
const root = document.getElementById('ifx-wire');
|
||||
if (root) {
|
||||
const lines = Array.from(root.querySelectorAll('.ifx-wiredump__line'));
|
||||
if (reduce) {
|
||||
lines.forEach((l) => l.classList.add('is-visible'));
|
||||
} else {
|
||||
let i = 0;
|
||||
const reveal = () => {
|
||||
if (i < lines.length) {
|
||||
lines[i].classList.add('is-visible');
|
||||
i++;
|
||||
setTimeout(reveal, 110 + Math.random() * 60);
|
||||
}
|
||||
};
|
||||
// Defer start until after first paint so it feels like a fresh capture.
|
||||
requestAnimationFrame(() => setTimeout(reveal, 380));
|
||||
}
|
||||
}
|
||||
</script>
|
||||
7
docs-site/src/content.config.ts
Normal file
7
docs-site/src/content.config.ts
Normal file
@ -0,0 +1,7 @@
|
||||
import { defineCollection } from 'astro:content';
|
||||
import { docsLoader } from '@astrojs/starlight/loaders';
|
||||
import { docsSchema } from '@astrojs/starlight/schema';
|
||||
|
||||
export const collections = {
|
||||
docs: defineCollection({ loader: docsLoader(), schema: docsSchema() }),
|
||||
};
|
||||
81
docs-site/src/content/docs/explain/architecture.md
Normal file
81
docs-site/src/content/docs/explain/architecture.md
Normal file
@ -0,0 +1,81 @@
|
||||
---
|
||||
title: Architecture overview
|
||||
description: How the layers stack, from socket through framing, codec, resultset, cursor, connection, and pool.
|
||||
sidebar:
|
||||
order: 2
|
||||
---
|
||||
|
||||
The driver is six layers, each with a single responsibility, each testable in isolation.
|
||||
|
||||
```text
|
||||
┌──────────────────────────────────────────────────┐
|
||||
│ Connection / Pool │ ← public API
|
||||
├──────────────────────────────────────────────────┤
|
||||
│ Cursor │ ← PEP 249 surface
|
||||
├──────────────────────────────────────────────────┤
|
||||
│ ResultSet │ ← row iteration, prefetch
|
||||
├──────────────────────────────────────────────────┤
|
||||
│ Codec / Per-column │ ← decode SQL types → Python
|
||||
│ readers │
|
||||
├──────────────────────────────────────────────────┤
|
||||
│ Protocol / PDU framing │ ← SQLI PDUs over the wire
|
||||
├──────────────────────────────────────────────────┤
|
||||
│ IfxSocket (buffered) │ ← raw bytes, recv() management
|
||||
└──────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## IfxSocket
|
||||
|
||||
The lowest layer. Wraps `socket.socket` with a connection-scoped read buffer (Phase 39). One `recv(64K)` per ~64 KB of incoming data; parsers read into the buffer via `struct.unpack_from(buf, offset)` rather than slicing copies.
|
||||
|
||||
Everything above this layer is `bytes` and `bytearray` arithmetic, with no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
|
||||
|
||||
See [The buffered reader →](/explain/buffered-reader/) for why the buffer lives here and not on the parser.
|
||||
|
||||
## Protocol / PDU framing
|
||||
|
||||
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean*, only how to read and write the byte shapes.
|
||||
|
||||
The PDU types and their fields were reverse-engineered from three sources:
|
||||
|
||||
1. The decompiled IBM JDBC driver (`com.informix.jdbc.IfxConnection` and the `IfxProtocol` class hierarchy).
|
||||
2. Annotated `socat` captures of real client/server exchanges (`docs/CAPTURES/`).
|
||||
3. Differential testing against IfxPy on identical data.
|
||||
|
||||
## Codec / Per-column readers
|
||||
|
||||
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types. See [SQL ↔ Python types](/reference/types/) for the full table.
|
||||
|
||||
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]`, with no per-column dispatch overhead.
|
||||
|
||||
Phase 38 went further with `exec()`-based codegen: for the hottest tables, the driver generates a flat decoder function with all readers inlined and dispatch decisions baked in. The generated function is the equivalent of unrolling the per-column dispatch into straight-line code.
|
||||
|
||||
## ResultSet
|
||||
|
||||
`_resultset.py`. Holds the cursor's column descriptors, the pre-baked decoder list, and the in-flight prefetch state. Manages `fetch_one`, `fetch_many`, `fetch_all` semantics.
|
||||
|
||||
Most cursor calls land here: when you do `cur.fetchone()`, the cursor delegates to its ResultSet which reads the next `SQ_TUPLE` PDU, runs it through the per-column decoders, and returns a tuple.
|
||||
|
||||
## Cursor
|
||||
|
||||
`cursors.py`. The PEP 249 cursor surface. `execute()`, `executemany()`, `fetchone()`, `fetchmany()`, `fetchall()`, `description`, `rowcount`, scrollable cursor support.
|
||||
|
||||
This is where parameter binding lives: a SQL statement with `?` placeholders gets prepared via `SQ_PREPARE`, the driver introspects the parameter shape via `SQ_DESCRIBE`, and bound values get encoded according to the parameter types.
|
||||
|
||||
## Connection / Pool
|
||||
|
||||
`connections.py` and `pool.py`. The connection owns the IfxSocket, manages transaction state, and is the parent of all cursors. The pool is a wrapper around N connections with the usual acquire/release/timeout semantics.
|
||||
|
||||
`aio.py` mirrors all of the above with `async def` versions, implemented via thread-pool wrapping (see [Async strategy →](/explain/async-strategy/)).
|
||||
|
||||
## Why this layering
|
||||
|
||||
Each boundary is testable in isolation:
|
||||
|
||||
- `IfxSocket` tests can use a `socket.socketpair()` and assert on byte streams.
|
||||
- Protocol tests parse known-good captures from `docs/CAPTURES/` and verify the typed PDUs come out correctly.
|
||||
- Codec tests pass synthetic byte payloads to per-column readers and assert the Python output.
|
||||
- ResultSet tests can use a fake protocol that emits canned PDU sequences.
|
||||
- Cursor tests use a fake ResultSet.
|
||||
|
||||
When a regression appears, the layered structure narrows the search: a wire-format test failing means it's the protocol layer; a row-tuple test failing with a corrupt-bytes input means the codec; a `fetchall` test failing means the ResultSet's iteration logic. The 300+ test suite leans heavily on this isolation.
|
||||
61
docs-site/src/content/docs/explain/async-strategy.mdx
Normal file
61
docs-site/src/content/docs/explain/async-strategy.mdx
Normal file
@ -0,0 +1,61 @@
|
||||
---
|
||||
title: Async strategy
|
||||
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async, and what that costs.
|
||||
sidebar:
|
||||
order: 4
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
`informix-driver`'s async API (`from informix_db import aio`) is implemented by wrapping the sync core in a thread pool. Every `await cur.execute(...)` schedules the underlying sync `execute()` on the pool's executor.
|
||||
|
||||
This is a deliberate architectural choice from Phase 16. Here's the reasoning.
|
||||
|
||||
## What we considered
|
||||
|
||||
Three options for adding async support to a sync database driver:
|
||||
|
||||
1. **Full async I/O refactor.** Rewrite the protocol layer on top of `asyncio.Protocol` or `asyncio.StreamReader`. The codec, framing, and connection state all become coroutines. ~2000 lines of code, full test rewrite.
|
||||
|
||||
2. **Thread-pool wrapping.** Keep the sync core. Wrap each public method with `loop.run_in_executor()`. ~250 lines of code, sync tests still apply, no protocol-layer changes.
|
||||
|
||||
3. **Dual implementations.** Maintain two parallel code paths, one sync and one async. Most code duplicated. Worst of both worlds.
|
||||
|
||||
We picked option 2.
|
||||
|
||||
## Why option 2 was the right call
|
||||
|
||||
For typical database workloads, meaning request-scoped connections that mostly wait on I/O, the practical difference between option 1 and option 2 is small:
|
||||
|
||||
- **Latency**: option 1 has a slight edge (no thread context switch), but the difference is dwarfed by the actual database round-trip (~80 µs LAN, ~ms WAN). For a single query, option 2 adds ~5–10 µs of executor overhead.
|
||||
- **Throughput under concurrency**: option 1 wins when you have N coroutines on M physical cores with M < N. The thread pool needs to context-switch between threads; the async loop just runs the next coroutine. For 10–100 concurrent FastAPI requests on a 4-core box, this difference is small.
|
||||
- **Code complexity**: option 1 is dramatically harder to write and test. The protocol layer becomes asynchronous everywhere; cancellation paths multiply; the shape of "what does a partial PDU read look like" becomes a state machine instead of a `while not done: read_more()`.
|
||||
|
||||
For a driver that needs to be production-ready in finite engineering time, option 2 was the right call.
|
||||
|
||||
## What it costs
|
||||
|
||||
The honest costs:
|
||||
|
||||
- **One worker thread per concurrent in-flight query.** With 100 concurrent queries, you have 100 threads. This is fine for I/O-bound work (Python releases the GIL during socket reads) but doesn't scale beyond a few hundred concurrent queries on a single process.
|
||||
- **Thread-pool sizing matters.** The default executor size (5 × CPU count) is fine for most workloads. For high-concurrency workloads, you may want a larger executor.
|
||||
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe, in that cancelled workers cannot leak onto recycled pool connections, but the underlying syscall does still complete.
|
||||
|
||||
## What it doesn't cost
|
||||
|
||||
- **Cancellation safety.** This was the original concern. Phase 27's per-connection wire lock + worker reaping makes async cancellation cancellation-safe in the same sense `asyncpg` is.
|
||||
- **FastAPI integration.** The `aio.Pool` is a drop-in replacement for any "async database pool" pattern. `Depends(get_conn)` works exactly as you'd expect.
|
||||
- **Async generator support.** `async for row in cur` works. The fetch is per-row chunked through the executor; the iteration shape is async-native.
|
||||
|
||||
## When option 1 might still be worth it
|
||||
|
||||
The two scenarios where a full async I/O implementation would matter:
|
||||
|
||||
1. **Very high concurrency on a single process** (1000+ in-flight queries). Thread context-switching cost becomes measurable. We haven't hit this in practice.
|
||||
2. **Sub-millisecond query latencies on a unloaded server.** The 5–10 µs executor overhead is a meaningful percentage. For typical Informix workloads where round-trip is ~80 µs+, it isn't.
|
||||
|
||||
If either becomes a real production concern, the layered architecture lets us swap in a fully-async lower half without changing the upper half. The cursor / connection / pool API doesn't care how the bytes get to and from the server. That's the option-2 win we explicitly preserved.
|
||||
|
||||
<Aside type="note">
|
||||
This is a Phase 16 decision. The pivot from "rewrite as async-native" to "wrap the sync core" is documented in [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md). Three years from now, if it turns out we should have gone with option 1, we have a clear path.
|
||||
</Aside>
|
||||
147
docs-site/src/content/docs/explain/buffered-reader.mdx
Normal file
147
docs-site/src/content/docs/explain/buffered-reader.mdx
Normal file
@ -0,0 +1,147 @@
|
||||
---
|
||||
title: The buffered reader
|
||||
description: How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1× IfxPy by moving the recv() buffer one level down the object graph.
|
||||
sidebar:
|
||||
order: 3
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39, a connection-scoped buffered reader, closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging.
|
||||
|
||||
This page is about both the technical change and the failure mode that hid the win for two phases.
|
||||
|
||||
## The lever we couldn't see
|
||||
|
||||
After Phase 38, profiling a 100,000-row fetch showed:
|
||||
|
||||
| Category | Self time | % of wall clock |
|
||||
|---|---:|---:|
|
||||
| I/O machinery | 555 ms | 66% |
|
||||
| Codec | 205 ms | 24% |
|
||||
| Other | ~80 ms | 10% |
|
||||
|
||||
The headline "I/O dominated" was true. The interesting half is the breakdown of that 555 ms:
|
||||
|
||||
- Actual `recv()` syscalls: ~153 ms
|
||||
- Python wrapper overhead: ~400 ms
|
||||
|
||||
That ~400 ms was our own buffer abstraction: a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
|
||||
|
||||
The kernel was doing maybe 25–30 ms of work. The other 130 ms of the gap-vs-IfxPy was friction we had introduced ourselves.
|
||||
|
||||
## The architecture pattern
|
||||
|
||||
Both `asyncpg` (in `buffer.pyx`) and `psycopg3` (in `pq.PGconn`) put a single growing read buffer on the protocol/connection object. The parser indexes into it via `struct.unpack_from(buf, offset)` rather than slicing copies. Refills happen via one large `recv(64K)` rather than many small `recv()`s for individual fields.
|
||||
|
||||
Phase 39 ports that pattern to `informix-driver`. The state machine:
|
||||
|
||||
```text
|
||||
┌───────────────────────────────┐
|
||||
│ IfxSocket │
|
||||
│ ─ socket: socket.socket │
|
||||
│ ─ buf: bytearray (growable) │
|
||||
│ ─ offset: int (read cursor) │
|
||||
│ │
|
||||
│ recv(64K) when buf exhausted │
|
||||
└───────────────────────────────┘
|
||||
▲
|
||||
│ reads via read_exact(n)
|
||||
│
|
||||
┌───────────────────────────────┐
|
||||
│ SocketReader (per-PDU) │
|
||||
│ ─ short-lived view │
|
||||
│ ─ no buffer of its own │
|
||||
└───────────────────────────────┘
|
||||
```
|
||||
|
||||
The reader is a parser-view. The buffer outlives the reader. When the parser asks for `read_short()`, the reader calls `socket.read_exact(2)`, which slices two bytes out of the bytearray at `offset` and advances. If the bytearray runs out, `socket.recv(64K)` refills it.
|
||||
|
||||
Result: **one `recv()` per ~64 KB of incoming data**, not per field.
|
||||
|
||||
## The architectural mistake the first pass got wrong
|
||||
|
||||
The natural thing to call this is "BufferedSocketReader". The natural thing to do is put the bytearray on the reader. That's what I did first.
|
||||
|
||||
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken`, meaning the process was blocked in `recv()` waiting for bytes that weren't coming.
|
||||
|
||||
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes, 200 for response 1 and 400 for response 2, it consumed bytes belonging to response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
|
||||
|
||||
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket`, the connection-scoped wrapper. Readers are short-lived parser-views, and the buffer outlives them.
|
||||
|
||||
```python
|
||||
# WRONG (first pass): buffer scoped to reader
|
||||
class BufferedSocketReader:
|
||||
def __init__(self, sock):
|
||||
self.sock = sock
|
||||
self.buf = bytearray() # ← dies with the reader
|
||||
self.offset = 0
|
||||
|
||||
# RIGHT (Phase 39): buffer scoped to connection
|
||||
class IfxSocket:
|
||||
def __init__(self, sock):
|
||||
self.sock = sock
|
||||
self._read_buf = bytearray() # ← outlives all readers
|
||||
self._read_offset = 0
|
||||
|
||||
def read_exact(self, n):
|
||||
if self._read_offset + n > len(self._read_buf):
|
||||
self._refill()
|
||||
...
|
||||
```
|
||||
|
||||
asyncpg and psycopg3 both place the buffer on the protocol/connection object. The architectural template was sitting in front of me before I started; I built the wrong shape anyway because "buffered reader" implies the buffer is *on* the reader.
|
||||
|
||||
It is not. The reader is a view. The buffer is state.
|
||||
|
||||
## The numbers
|
||||
|
||||
A/B-measured against the same Docker container, warmed cache, only the env flag differing:
|
||||
|
||||
| Workload | Phase 38 | Phase 39 | Δ |
|
||||
|---|---:|---:|---:|
|
||||
| `select_scaling_1000` | 2.901 ms | 1.716 ms | **−41%** |
|
||||
| `select_scaling_10000` | 24.317 ms | 16.084 ms | **−34%** |
|
||||
| `select_scaling_100000` | 250.363 ms | 168.982 ms | **−32%** |
|
||||
|
||||
Re-running the IfxPy comparison after Phase 39:
|
||||
|
||||
| Workload | IfxPy 2.0.7 (C) | informix-driver Phase 39 | Ratio |
|
||||
|---|---:|---:|---:|
|
||||
| `select_scaling_1000` | 1.637 ms | 1.716 ms | **1.05×** |
|
||||
| `select_scaling_10000` | 15.07 ms | 16.08 ms | **1.07×** |
|
||||
| `select_scaling_100000` | 147.4 ms | 169.0 ms | **1.15×** |
|
||||
|
||||
The 2.4× steady-state gap that existed before Phase 37 is now within 5–15% of the C driver, and the lower bound may already be within IfxPy's own measurement noise (its IQR on the 100k workload is 21%; ours is 0.2%).
|
||||
|
||||
## What the feature flag does
|
||||
|
||||
The buffered reader ships **enabled by default** in version 2026.05.05.12. To opt out (debugging, regressing a workload, A/B-measuring your own code):
|
||||
|
||||
```bash
|
||||
IFX_BUFFERED_READER=0 python my_app.py
|
||||
```
|
||||
|
||||
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime, so close and reopen the pool to flip behavior.
|
||||
|
||||
<Aside type="note">
|
||||
The flag exists to make A/B measurement easy. There's no expected reason to disable it in production. If you hit a workload where the buffered reader is slower, that's a bug and we'd like to know.
|
||||
</Aside>
|
||||
|
||||
## What we learned
|
||||
|
||||
The general pattern: **what's visible gets optimization attention; what's invisible gets written off as irreducible**.
|
||||
|
||||
The codec is visible: there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
|
||||
|
||||
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads (`read_short`, `read_int`, `read_exact(payload_size)`) are one-liners. What could be slow about that?
|
||||
|
||||
It was carrying ~30% of total wall time. Two phases of changelogs implicitly blamed "the protocol" for the remaining gap. The actual culprit was a few lines of `bytes.join` in a wrapper from Phase 1 that nobody had revisited.
|
||||
|
||||
The lesson is small and easy to state: a profile turns vibes into an attack surface. Write the closing paragraph after you've measured, not before.
|
||||
|
||||
## Read more
|
||||
|
||||
- **[Architecture overview →](/explain/architecture/)**: where the buffered reader sits in the layer stack.
|
||||
- **[Phase log →](/explain/phase-log/)**: the full progression from Phase 1 through Phase 39+.
|
||||
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)**: Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.
|
||||
150
docs-site/src/content/docs/explain/phase-log.md
Normal file
150
docs-site/src/content/docs/explain/phase-log.md
Normal file
@ -0,0 +1,150 @@
|
||||
---
|
||||
title: The phase log
|
||||
description: Phase-by-phase narrative of how informix-driver got built, with notable architectural decisions called out.
|
||||
sidebar:
|
||||
order: 6
|
||||
---
|
||||
|
||||
The driver was built across 39+ phases, each with a focused scope and a decision log. This page is a high-level index; the gory details (with rationale, alternatives considered, and rollback notes) live in [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md).
|
||||
|
||||
## Foundation (Phases 1–10)
|
||||
|
||||
| Phase | Title | Outcome |
|
||||
|---|---|---|
|
||||
| 1 | Socket + minimal SQ_INFO | First handshake against the dev container |
|
||||
| 2–4 | Login, DBOPEN, error decoding | Can connect and select a database |
|
||||
| 5 | Statement execution | `SELECT 1` works |
|
||||
| 6 | Parameter binding | `?`-placeholders, basic types |
|
||||
| 7 | Logged-DB transactions | Discovered Informix needs explicit `SQ_BEGIN` per tx in non-ANSI mode |
|
||||
| 8 | BYTE / TEXT (legacy in-row blobs) | Needs blobspace |
|
||||
| 9 | Scrollable cursors | `SQ_SCROLL` PDU, position semantics |
|
||||
| 10/11 | Smart-LOB read & write | **Architectural pivot** to `SQ_FILE` intercept; ~3× smaller than projected |
|
||||
|
||||
## Hardening (Phases 12–20)
|
||||
|
||||
| Phase | Title | Outcome |
|
||||
|---|---|---|
|
||||
| 12 | Type system overhaul | Per-column readers (predecessor to Phase 37) |
|
||||
| 13 | DECIMAL / MONEY exact precision | `decimal.Decimal` round-trip |
|
||||
| 14 | DATETIME range typing | Returns `date` / `datetime` / `time` per field range |
|
||||
| 15 | INTERVAL types | Custom `IntervalYM`, `timedelta` for D-to-F |
|
||||
| 16 | Async API | **Pivot** to thread-pool wrapping (~250 lines) instead of full async refactor (~2000 lines) |
|
||||
| 17 | Connection pool (sync) | min/max sizing, acquire timeout, max idle |
|
||||
| 18 | Connection pool (async) | Mirror of sync API on `aio.Pool` |
|
||||
| 19 | TLS support | Bring-your-own-context, `tls=True` for dev |
|
||||
| 20 | Locale / Unicode | `client_locale`, full mapping in [`Connection.encoding`](/reference/types/) |
|
||||
|
||||
## Production review (Phases 21–30)
|
||||
|
||||
| Phase | Title | Outcome |
|
||||
|---|---|---|
|
||||
| 21 | Type-checking pass | `py.typed`, full mypy/pyright coverage |
|
||||
| 22 | Error code mapping | SQLCODE → exception per [reference](/reference/exceptions/) |
|
||||
| 23 | Health checks | Pool validates idle connections before return |
|
||||
| 24 | Statement caching | Per-connection prepared-statement cache |
|
||||
| 25 | Fast-path call (`SQ_FPROUTINE`) | Direct UDF/SPL invocation, bypassing PREPARE |
|
||||
| 26 | **CRITICAL** | Pool returned connections with open transactions (fixed) |
|
||||
| 27 | **CRITICAL** | Per-connection wire lock + async cancellation safety |
|
||||
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync (fixed) |
|
||||
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise (fixed) |
|
||||
| 30 | Hardening pass | 5 medium-severity audit findings, all closed |
|
||||
|
||||
After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver is production-ready.
|
||||
|
||||
## Performance (Phases 31–39)
|
||||
|
||||
| Phase | Title | Result |
|
||||
|---|---|---|
|
||||
| 31 | Statement cache LRU tuning | Better hit rate on repeated queries |
|
||||
| 32 | Cursor lifecycle optimization | Fewer round-trips on small queries |
|
||||
| 33 | **Pipelined `executemany`** | **1.6× faster than IfxPy on bulk inserts** |
|
||||
| 34 | LRU caches for type lookup | Removed dispatch overhead on hot paths |
|
||||
| 35 | Memory profile pass | Identified 100k-row baseline |
|
||||
| 36 | `IfxPy` comparison harness | Established the 2.4× bulk-fetch gap |
|
||||
| 37 | **Per-column reader strategy** | −10% on bulk SELECT, ratio → 2.10× |
|
||||
| 38 | **`exec()`-based row-decoder codegen** | Further −12%, ratio → 2.04× |
|
||||
| 39 | **Connection-scoped buffered reader** | **−32% on bulk SELECT, ratio → 1.05–1.15×** |
|
||||
|
||||
The Phase 37–39 trajectory is documented in detail at [The buffered reader →](/explain/buffered-reader/), including the architectural mistake the first pass got wrong.
|
||||
|
||||
## Field reports and systematic review (2026.08–2026.09)
|
||||
|
||||
Phase 30's audit found nothing left. Then real schemas arrived, and the audit's clean bill of health turned out to be a statement about the questions it had asked rather than about the driver.
|
||||
|
||||
A field report from a user running Informix 12 surfaced three type-framing bugs in a single afternoon: `INT8` / `SERIAL8` never decoded at all, `NCHAR` losing its first character, and `BOOLEAN` corrupting every column after it. Fuzzing the type matrix found more, and each one had the same shape as the last.
|
||||
|
||||
`2026.09.02` went back over the driver looking for that shape rather than for new symptoms, and found eleven bugs.
|
||||
|
||||
| Area | What was wrong |
|
||||
|---|---|
|
||||
| Row framing | Nothing ever checked that a row consumed its own payload. Fourteen framing bugs had reached users, every one detectable for free |
|
||||
| Smart LOBs | Read as a flat 72-byte field when they occupy 149, shifting every column after a non-final `BLOB` |
|
||||
| Composites | A NULL collection skipped its length field, the same bug as NULL `LVARCHAR` twelve lines away |
|
||||
| Transactions | `rollback()` after a SQL `BEGIN WORK` sent nothing and reported success |
|
||||
| Classification | Comments, CTEs and parenthesized selects were run as DML and failed with `-260` |
|
||||
| Scrollable cursors | A second statement on the connection got `-285` and destroyed the cursor too |
|
||||
| Statement release | Two of eight exits leaked a failed statement, bricking the connection |
|
||||
| Cursor finalizers | Cleanup could land inside another statement, and the lock probe was blind to its own thread |
|
||||
| Async | The whole layer shared the process-wide thread pool, so cancellations starved it |
|
||||
| Socket | Two readers on one stream agreed only by accident of how the server replies |
|
||||
|
||||
The pattern behind almost all of them: a hazard that was understood and guarded at the site where it was first observed, rather than at the abstraction that owned it. The guard then never travelled to its siblings. Four separate hand-written copies of the same UDT envelope, three of them wrong. Six hand-rolled copies of the same statement-release cleanup, two missing entirely.
|
||||
|
||||
Two of the fixes came from asking the server instead of guessing. `statement_type` and `statement_id` had both been parsed out of every DESCRIBE response into a metadata dict that nothing read, while the code that needed them inferred the answer from the first word of the SQL and got it wrong five ways.
|
||||
|
||||
The cheapest fix was also the most valuable. Asserting that a row decoder lands exactly on the end of its payload costs one integer comparison, and it would have caught all fourteen framing bugs at the byte where each happened rather than three releases later in somebody's result set.
|
||||
|
||||
Test count went from 241 to 457 across three server versions over this stretch.
|
||||
|
||||
### 2026.09.03
|
||||
|
||||
Two more, one of each kind.
|
||||
|
||||
The `:N` placeholder rewriter was a bare regular expression, so it
|
||||
rewrote the inside of string literals: `'http://host:8080/x'` was stored
|
||||
as `'http://host?/x'`. Same family as the statement-classification bug
|
||||
above, and the last place the driver inferred meaning from SQL text
|
||||
instead of handling it. Replaced with a scanner using Informix's own
|
||||
lexical rules, measured rather than assumed. One of those rules would
|
||||
have been got wrong from habit: a backslash escapes nothing, so `'a\'b'`
|
||||
is an unterminated string and draws `-282`. A scanner written to
|
||||
Postgres reflexes desyncs there and corrupts everything after it.
|
||||
|
||||
`row_factory=Row` adds column-name and attribute access, on request from
|
||||
a user coming from `pyodbc`. Opt-in, because it costs about 9% on bulk
|
||||
fetch and that is not a trade to make for someone who never reads a
|
||||
column by name.
|
||||
|
||||
Testing the second one found the pattern again, this time in code
|
||||
written an hour earlier. The names a column could shadow were
|
||||
hand-listed as `count` and `index`, the two `tuple` methods; then `keys`,
|
||||
`_asdict` and `_fields` were added to the same class and the list was
|
||||
never revisited. A column called `keys` returned a bound method, in
|
||||
silence. The fix was to compute the set from the class rather than
|
||||
lengthen the list, and to parametrize the test over that computed set so
|
||||
it grows with the class.
|
||||
|
||||
The interesting part is where testing found it. A differential test
|
||||
across twenty columns covering every awkward type, populated and fully
|
||||
NULL, found nothing. The bug was in six lines of name bookkeeping that
|
||||
looked obviously correct, and had been correct when written. Anything
|
||||
that must stay in sync with a class by hand eventually will not.
|
||||
|
||||
## Notable architectural pivots
|
||||
|
||||
The decision log calls out four moments where the obvious choice would have been wrong:
|
||||
|
||||
1. **Phase 10/11**: abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
|
||||
2. **Phase 16**: thread-pool async instead of a full async refactor. ~88% less code, same FastAPI surface.
|
||||
3. **Phase 27**: adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
|
||||
4. **Phase 39**: buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
|
||||
|
||||
## What's next
|
||||
|
||||
The roadmap (loose, not committed):
|
||||
|
||||
- **Phase 40+ (codec)**: Numpy-backed bulk decode for homogeneous columns. ~5× speedup target on analytical workloads.
|
||||
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python", so it would sit behind a build flag.
|
||||
- **Phase 5x (API)**: Native `callproc` with named parameters, IBM-specific scrollable cursor extensions for full IfxPy parity.
|
||||
|
||||
The phase log is updated as work lands. The repo's [`CHANGELOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md) is the source of truth for shipped changes.
|
||||
63
docs-site/src/content/docs/explain/pure-python.md
Normal file
63
docs-site/src/content/docs/explain/pure-python.md
Normal file
@ -0,0 +1,63 @@
|
||||
---
|
||||
title: Pure-Python tradeoffs
|
||||
description: What pure-Python costs, what it pays for, and where the actual performance ceiling sits.
|
||||
sidebar:
|
||||
order: 5
|
||||
---
|
||||
|
||||
The premise of this driver is *zero native code in the call stack*. No CSDK, no JDBC, no C extension we maintain ourselves. Just `socket`, `struct`, `decimal`, and the standard library.
|
||||
|
||||
This page is about what that costs and what it pays for.
|
||||
|
||||
## What pure-Python costs
|
||||
|
||||
The honest accounting:
|
||||
|
||||
| Cost | Magnitude | Mitigation |
|
||||
|---|---|---|
|
||||
| Per-row decode overhead | ~2.0 µs/row vs IfxPy's ~1.1 µs/row | Phases 37–38 codec inlining brought us from 4 µs to 2 µs. |
|
||||
| Per-PDU parser overhead | ~5–10 µs vs C's ~1 µs | Phase 39 buffered reader removed the worst of it (the read-side wrapper cost). |
|
||||
| GIL contention on multi-threaded decode | Threads serialize through codec hot loops | Pool gives one connection per thread; codec releases GIL during I/O. |
|
||||
| Memory per connection | ~50–500 KB (Phase 39 buffer) | Pool keeps it bounded; freed on connection close. |
|
||||
|
||||
The order-of-magnitude intuition: pure-Python is ~2× slower than C-bound for **codec-bound workloads** (large analytical fetches), and **competitive or faster** for I/O-bound workloads (transactional, bulk-insert, FastAPI request-response).
|
||||
|
||||
## What pure-Python pays for
|
||||
|
||||
The benefits are mostly deployment, not performance:
|
||||
|
||||
- **50 KB wheel**: installable in a slim Docker image without a build toolchain.
|
||||
- **No `libcrypt.so.1`**: works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
|
||||
- **Python 3.10–3.14**: no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
|
||||
- **Type annotations everywhere**: `py.typed` flag, full coverage in mypy / pyright.
|
||||
- **Auditable codepaths**: every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
|
||||
- **Async without `run_in_executor`**: native `async def` API, FastAPI-compatible.
|
||||
|
||||
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1`: those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
|
||||
|
||||
## Where the ceiling sits
|
||||
|
||||
For codec-bound workloads, the ceiling we've hit is around 2 µs/row for tabular data. The breakdown:
|
||||
|
||||
- `struct.unpack_from(format, buf, offset)` per field: ~80 ns
|
||||
- `bytes` → `str` decoding (varchar): ~150 ns
|
||||
- Tuple construction: ~100 ns
|
||||
- Cursor / ResultSet bookkeeping: ~50 ns
|
||||
|
||||
Five fields × ~250 ns/field + ~250 ns overhead = ~1.5 µs. We're at ~2.0 µs which means ~30% overhead remains. That's the gap between "we've inlined everything that's reasonable" and "the C version still wins."
|
||||
|
||||
Strategies for closing further:
|
||||
|
||||
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim, because there'd be a build step.
|
||||
- **Bytecode optimization via `exec()`-codegen** (the Phase 38 approach). Marginal further wins; we've already extracted most of what's available.
|
||||
- **Numpy-backed bulk decode** for homogeneous columns. Promising for analytical workloads. ~5× speedup possible for `SELECT col FROM huge_table` over the current per-row approach. Probably Phase 41+.
|
||||
|
||||
For I/O-bound workloads we're already at the ceiling. The buffered reader closed the I/O gap; further wins are at the kernel level (e.g. `recvmsg` for vectored reads), which is moot since the kernel already isn't the bottleneck.
|
||||
|
||||
## The honest summary
|
||||
|
||||
Pure-Python costs us ~5–15% on bulk-fetch workloads and zero (or favorable) on everything else. The deployment, async, and modern-Python wins are large and don't depend on workload.
|
||||
|
||||
If the codec gap matters for your case, meaning analytical reporting against a wide table or pulling millions of rows in a single SELECT, IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
|
||||
|
||||
The driver chose the goal, *first pure-socket Informix driver in any language*, over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.
|
||||
84
docs-site/src/content/docs/explain/sqli-protocol.mdx
Normal file
84
docs-site/src/content/docs/explain/sqli-protocol.mdx
Normal file
@ -0,0 +1,84 @@
|
||||
---
|
||||
title: The SQLI wire protocol
|
||||
description: A short tour of Informix's SQLI protocol, covering PDU framing, the handshake, statement execution, and fetch.
|
||||
sidebar:
|
||||
order: 1
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
SQLI is Informix's wire protocol, the same one IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
|
||||
|
||||
This page is a short tour. The byte-level reference (with hex annotations) lives in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) in the repo.
|
||||
|
||||
## PDU framing
|
||||
|
||||
Every PDU starts with a 1-byte tag and (mostly) ends with a 2-byte EOT marker. Common tags:
|
||||
|
||||
| Tag | Hex | Direction | Purpose |
|
||||
|---|---|---|---|
|
||||
| `SQ_INFO` | `0x01` | →S | Initial capability/identity exchange |
|
||||
| `SQ_VERSION` | `0x14` | S→ | Server version response |
|
||||
| `SQ_PASSWD` | `0x18` | →S | Authentication |
|
||||
| `SQ_DBOPEN` | `0x24` | →S | Open database |
|
||||
| `SQ_PREPARE` | `0x02` | →S | Prepare statement |
|
||||
| `SQ_DESCRIBE` | `0x08` | →S | Describe column structure |
|
||||
| `SQ_OPEN` | `0x06` | →S | Open cursor |
|
||||
| `SQ_FETCH` | `0x04` | →S | Fetch rows |
|
||||
| `SQ_TUPLE` | `0x09` | S→ | One row of data |
|
||||
| `SQ_ID` | `0x37` | S→ | SQLCODE / status code |
|
||||
| `SQ_FILE` | `0x62` | both | Smart-LOB transfer |
|
||||
| `SQ_EOT` | `0x0c` | both | End-of-transmission |
|
||||
|
||||
The trailing `00 0c` (length=0, tag=0x0c) marks the end of every multi-PDU response.
|
||||
|
||||
## A connect, in bytes
|
||||
|
||||
Annotated output from `docs/CAPTURES/01-connect-only.socat.log`:
|
||||
|
||||
```text
|
||||
> OUT 01 c3 01 3c 00 00 00 64 00 65 00 00 00 3d 00 06 ; SQ_INFO
|
||||
49 45 45 45 4d 00 00 6c 73 71 6c 65 78 65 63 00 ; "IEEEM..lsqlexec"
|
||||
00 00 00 00 00 00 06 39 2e 32 38 30 00 00 0c 52 ; ".......9.280...R"
|
||||
44 53 23 52 30 30 30 30 30 30 00 00 05 73 71 6c ; "DS#R000000...sql"
|
||||
69 00 00 00 01 3c 00 00 00 00 00 00 00 00 00 01 ; "i....<.........."
|
||||
...
|
||||
|
||||
< IN 01 14 02 3c 10 00 00 64 00 65 00 00 00 3d 00 06 ; SQ_VERSION
|
||||
49 45 45 45 49 00 00 6c 73 72 76 69 6e 66 78 00 ; "IEEEI..lsrvinfx."
|
||||
00 00 00 00 00 00 2f 49 42 4d 20 49 6e 66 6f 72 ; "....../IBM Infor"
|
||||
6d 69 78 20 44 79 6e 61 6d 69 63 20 53 65 72 76 ; "mix Dynamic Serv"
|
||||
65 72 20 56 65 72 73 69 6f 6e 20 31 35 2e 30 2e ; "er Version 15.0."
|
||||
```
|
||||
|
||||
The payload of `SQ_INFO` is a sequence of length-prefixed strings: byte-order marker (`IEEEM` = big-endian), client app name (`sqlexec`), client version (`9.280`), build ID, protocol token (`sqli`), feature flags. The server's `SQ_VERSION` response mirrors this with the server's own identification.
|
||||
|
||||
## Statement execution
|
||||
|
||||
The full lifecycle for `SELECT id FROM users WHERE id = ?`:
|
||||
|
||||
```text
|
||||
→ SQ_PREPARE "SELECT id FROM users WHERE id = ?"
|
||||
← SQ_ID (statement ID, parameter shape, ...)
|
||||
→ SQ_DESCRIBE
|
||||
← SQ_DESC (column descriptors: "id" SMALLINT)
|
||||
→ SQ_OPEN (parameter values: 42)
|
||||
← SQ_ID (cursor ID)
|
||||
→ SQ_FETCH
|
||||
← SQ_TUPLE (id=42)
|
||||
← SQ_TUPLE (or SQ_DONE)
|
||||
→ SQ_CLOSE (release cursor)
|
||||
← SQ_ID
|
||||
```
|
||||
|
||||
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts. See [Bulk inserts](/how-to/executemany/).
|
||||
|
||||
## Smart-LOB transfer
|
||||
|
||||
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side, with no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
|
||||
|
||||
This was the architectural pivot in [Phase 10/11](/explain/phase-log/) that made smart-LOBs work end-to-end in pure Python. Reading and writing GB-sized BLOBs goes through the same socket as any other query.
|
||||
|
||||
<Aside type="note">
|
||||
The protocol has many more PDU types than this page covers, mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
|
||||
</Aside>
|
||||
93
docs-site/src/content/docs/how-to/async-fastapi.mdx
Normal file
93
docs-site/src/content/docs/how-to/async-fastapi.mdx
Normal file
@ -0,0 +1,93 @@
|
||||
---
|
||||
title: Async with FastAPI
|
||||
description: Wire the async pool into a FastAPI app with proper lifecycle management.
|
||||
sidebar:
|
||||
order: 3
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
`informix-driver` has a native async API. Use it from FastAPI by creating the pool in the app's lifespan and dependency-injecting connections per request.
|
||||
|
||||
## App skeleton
|
||||
|
||||
```python
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI, Depends, HTTPException
|
||||
from informix_db import aio
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
app.state.pool = await aio.create_pool(
|
||||
host="db.example.com",
|
||||
user="informix", password="...",
|
||||
database="mydb", server="informix",
|
||||
min_size=2, max_size=20,
|
||||
)
|
||||
yield
|
||||
await app.state.pool.close()
|
||||
|
||||
app = FastAPI(lifespan=lifespan)
|
||||
|
||||
|
||||
async def get_conn():
|
||||
async with app.state.pool.connection() as conn:
|
||||
yield conn
|
||||
|
||||
|
||||
@app.get("/users/{user_id}")
|
||||
async def get_user(user_id: int, conn = Depends(get_conn)):
|
||||
cur = await conn.cursor()
|
||||
await cur.execute(
|
||||
"SELECT id, name, email FROM users WHERE id = ?",
|
||||
(user_id,),
|
||||
)
|
||||
row = await cur.fetchone()
|
||||
if row is None:
|
||||
raise HTTPException(404, "user not found")
|
||||
return {"id": row[0], "name": row[1], "email": row[2]}
|
||||
```
|
||||
|
||||
## Why this shape
|
||||
|
||||
- **Lifespan-scoped pool**: the pool lives for the lifetime of the app, login handshake amortized across all requests.
|
||||
- **Per-request connection via `Depends`**: each request gets its own connection from the pool. The async generator pattern (`yield conn`) means the connection returns to the pool when the request finishes, including on exception.
|
||||
- **No `run_in_executor`**: every call is `await`able natively. No event-loop blocking, no thread-pool tuning.
|
||||
|
||||
## Cancellation
|
||||
|
||||
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe: the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean, with transactions rolled back. You don't need to wrap anything in `try/finally`.
|
||||
|
||||
<Aside type="note">
|
||||
This is a Phase 27 invariant: async cancellation cannot leak running workers onto recycled connections. The earlier behavior was a `High` audit finding; the fix is a CI tripwire test that's been green every commit since.
|
||||
</Aside>
|
||||
|
||||
## Connection-level transactions
|
||||
|
||||
For request-scoped transactions, commit on the way out and roll back on any exception:
|
||||
|
||||
```python
|
||||
@app.post("/orders")
|
||||
async def create_order(order: OrderIn, conn = Depends(get_conn)):
|
||||
cur = await conn.cursor()
|
||||
try:
|
||||
await cur.execute(
|
||||
"INSERT INTO orders VALUES (?, ?, ?)",
|
||||
(order.id, order.customer_id, order.total),
|
||||
)
|
||||
await cur.execute(
|
||||
"UPDATE inventory SET qty = qty - ? WHERE sku = ?",
|
||||
(order.qty, order.sku),
|
||||
)
|
||||
await conn.commit()
|
||||
except Exception:
|
||||
await conn.rollback()
|
||||
raise
|
||||
return {"ok": True}
|
||||
```
|
||||
|
||||
`HTTPException` is an `Exception`, so raising one inside the block rolls back before FastAPI turns it into a response.
|
||||
|
||||
<Aside type="note">
|
||||
There is no `conn.transaction()` context manager on either the sync or async connection, though an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
|
||||
</Aside>
|
||||
48
docs-site/src/content/docs/how-to/buffered-reader.md
Normal file
48
docs-site/src/content/docs/how-to/buffered-reader.md
Normal file
@ -0,0 +1,48 @@
|
||||
---
|
||||
title: Optimize bulk SELECT
|
||||
description: How the buffered reader works in practice, when it's on, when it isn't, and how to A/B-measure your workload.
|
||||
sidebar:
|
||||
order: 5
|
||||
---
|
||||
|
||||
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything, since the bulk-fetch gap against IfxPy is now ~5–15% rather than ~140%.
|
||||
|
||||
For the architectural rationale, see [The buffered reader →](/explain/buffered-reader/).
|
||||
|
||||
## Disabling the buffered reader
|
||||
|
||||
```bash
|
||||
IFX_BUFFERED_READER=0 python my_app.py
|
||||
```
|
||||
|
||||
The flag is read once at connection construction. To flip behavior on existing connections, close and reopen the pool.
|
||||
|
||||
There's no expected reason to disable it in production. The flag exists so you can A/B-measure your own workload and so we can debug regressions if they appear.
|
||||
|
||||
## A/B-measuring your workload
|
||||
|
||||
```bash
|
||||
# Baseline: no buffered reader
|
||||
IFX_BUFFERED_READER=0 python -m mybench
|
||||
|
||||
# With buffered reader
|
||||
IFX_BUFFERED_READER=1 python -m mybench
|
||||
```
|
||||
|
||||
For typical bulk-SELECT workloads expect a 30–40% wall-time reduction. For workloads dominated by single-row queries the impact is small (small queries are RTT-bound, not framing-bound).
|
||||
|
||||
## When the speedup is largest
|
||||
|
||||
Workloads where every column read makes ~4–5 small `recv()` calls, meaning tabular data, narrow rows, and large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
|
||||
|
||||
| Workload shape | Speedup |
|
||||
|---|---:|
|
||||
| Wide row, single fetch (1 row × 100 cols) | minimal |
|
||||
| Narrow row, large fetch (100k rows × 5 cols) | 30–40% |
|
||||
| `executemany` response drain (1k inserts) | 25–30% |
|
||||
|
||||
## Memory profile
|
||||
|
||||
The buffer is per-connection, sized to grow up to the largest single PDU it sees. Typical steady-state: 64 KB to a few hundred KB per connection. The buffer is freed when the connection closes; for long-lived pool connections it's amortized.
|
||||
|
||||
If you're running 10,000 connections at idle, the buffer cost is ~1–2 GB across the fleet. For typical pool sizes (10–50 connections) it's ~1–10 MB total.
|
||||
93
docs-site/src/content/docs/how-to/dev-container.mdx
Normal file
93
docs-site/src/content/docs/how-to/dev-container.mdx
Normal file
@ -0,0 +1,93 @@
|
||||
---
|
||||
title: Run the dev container
|
||||
description: IBM Informix Developer Edition in Docker, covering first-time setup, sbspace for smart-LOBs, and common troubleshooting.
|
||||
sidebar:
|
||||
order: 8
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
The IBM Informix Developer Edition Docker image is the recommended dev / integration-test target. It's the same image our CI runs against.
|
||||
|
||||
## First-time setup
|
||||
|
||||
```bash
|
||||
docker run -d --name informix-dev \
|
||||
-e LICENSE=accept \
|
||||
-p 9088:9088 \
|
||||
-p 9089:9089 \
|
||||
--privileged \
|
||||
icr.io/informix/informix-developer-database:15.0.1.0.3DE
|
||||
```
|
||||
|
||||
- **`9088`**: clear-text SQLI listener
|
||||
- **`9089`**: TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
|
||||
- **`--privileged`**: required for the dev image's shared-memory tuning
|
||||
|
||||
The image takes ~90 seconds to initialize. Watch for `oninit running`:
|
||||
|
||||
```bash
|
||||
docker logs -f informix-dev
|
||||
```
|
||||
|
||||
## Default credentials
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| `user` | `informix` |
|
||||
| `password` | `in4mix` |
|
||||
| `database` | `sysmaster` (always exists) |
|
||||
| `server` | `informix` |
|
||||
|
||||
For application use create your own database:
|
||||
|
||||
```bash
|
||||
docker exec -it informix-dev bash -c '
|
||||
echo "CREATE DATABASE app_db WITH LOG;" | dbaccess sysmaster
|
||||
'
|
||||
```
|
||||
|
||||
## Setup for smart-LOB tests
|
||||
|
||||
Smart-LOBs (BLOB / CLOB) require additional one-time setup:
|
||||
|
||||
```bash
|
||||
# Inside the container
|
||||
docker exec -it informix-dev su - informix -c '
|
||||
onspaces -c -S sbspace1 -p $INFORMIXDIR/sbspace1 -o 0 -s 100000
|
||||
|
||||
# Edit $ONCONFIG to set SBSPACENAME sbspace1, then:
|
||||
onmode -ky
|
||||
oninit -y
|
||||
|
||||
# Take a level-0 archive so the sbspace is usable
|
||||
ontape -s -L 0
|
||||
'
|
||||
```
|
||||
|
||||
After that, BLOBs and CLOBs work end-to-end. See [`docs/DECISION_LOG.md` §10](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) for the gory details.
|
||||
|
||||
## Running the integration tests
|
||||
|
||||
```bash
|
||||
make ifx-up # starts the container if not already running
|
||||
make test-integration # runs the 231 integration tests
|
||||
```
|
||||
|
||||
Or directly:
|
||||
|
||||
```bash
|
||||
pytest -m integration
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"Connection refused" on 9088**: the image is still initializing. Wait for `oninit running` in the logs.
|
||||
|
||||
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing, since it always exists.
|
||||
|
||||
**`-908` (system error / shared memory)**: the container needs `--privileged`. Restart with that flag.
|
||||
|
||||
<Aside type="tip">
|
||||
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart, and the database survives container shutdown.
|
||||
</Aside>
|
||||
107
docs-site/src/content/docs/how-to/executemany.mdx
Normal file
107
docs-site/src/content/docs/how-to/executemany.mdx
Normal file
@ -0,0 +1,107 @@
|
||||
---
|
||||
title: Bulk inserts (executemany)
|
||||
description: How to bulk-load with executemany, including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
sidebar:
|
||||
order: 4
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
`executemany()` is the right tool for bulk inserts and updates. With `informix-driver`'s pipelined implementation it's **1.6× faster than IfxPy** at 10k+ rows.
|
||||
|
||||
## The basic shape
|
||||
|
||||
```python
|
||||
rows = [(1, "alice"), (2, "bob"), (3, "carol"), ...] # 10_000 tuples
|
||||
|
||||
with conn: # opens a transaction
|
||||
cur = conn.cursor()
|
||||
cur.executemany(
|
||||
"INSERT INTO users (id, name) VALUES (?, ?)",
|
||||
rows,
|
||||
) # commits on normal exit
|
||||
```
|
||||
|
||||
That inserts 10,000 rows in ~161 ms against a loopback Informix container.
|
||||
|
||||
## The 53× gotcha
|
||||
|
||||
<Aside type="caution">
|
||||
**`executemany(...)` under `autocommit=True` is 53× slower than the same call inside an explicit transaction.**
|
||||
|
||||
The server flushes the transaction log to disk per row in autocommit mode. With 10,000 single-row autocommit inserts that's 10,000 log flushes. Inside one transaction it's one flush at commit.
|
||||
</Aside>
|
||||
|
||||
```python
|
||||
# SLOW: 8.5 seconds for 10k rows
|
||||
conn = informix_db.connect(..., autocommit=True)
|
||||
cur = conn.cursor()
|
||||
cur.executemany("INSERT ...", rows)
|
||||
|
||||
# FAST: 161 ms for 10k rows
|
||||
conn = informix_db.connect(..., autocommit=False) # default
|
||||
with conn:
|
||||
cur = conn.cursor()
|
||||
cur.executemany("INSERT ...", rows)
|
||||
```
|
||||
|
||||
The default is `autocommit=False`, so this only catches you if you've explicitly opted into autocommit.
|
||||
|
||||
## Why it's faster than IfxPy
|
||||
|
||||
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row, so for 10,000 rows on an 80 µs RTT it adds up to 800 ms of just waiting for ACKs.
|
||||
|
||||
Phase 33 changed our `executemany` to **pipeline** the BIND+EXECUTE PDUs:
|
||||
|
||||
1. Send all N BIND PDUs back-to-back without reading responses
|
||||
2. Send all N EXECUTE PDUs back-to-back
|
||||
3. Drain N response sets at the end
|
||||
|
||||
The kernel buffers the outbound bytes; the server processes the BIND/EXECUTE pipeline as fast as it can; we read all responses at the end. One RTT for the whole batch instead of N.
|
||||
|
||||
## Chunking large batches
|
||||
|
||||
For very large batches (millions of rows), break into chunks to bound memory:
|
||||
|
||||
```python
|
||||
def chunks(it, n):
|
||||
buf = []
|
||||
for x in it:
|
||||
buf.append(x)
|
||||
if len(buf) >= n:
|
||||
yield buf
|
||||
buf = []
|
||||
if buf:
|
||||
yield buf
|
||||
|
||||
with conn:
|
||||
cur = conn.cursor()
|
||||
for batch in chunks(huge_iterator, 10_000):
|
||||
cur.executemany("INSERT INTO logs ...", batch)
|
||||
# one transaction, many batched executemany calls, one commit at the end
|
||||
```
|
||||
|
||||
10,000 rows per chunk is a reasonable default; the per-chunk Python memory cost is `~ N × bytes_per_row`. For 10k tuples of 5 small fields that's a few MB.
|
||||
|
||||
## Returning generated keys
|
||||
|
||||
Informix's `SERIAL` columns are server-assigned, and `executemany` doesn't return per-row IDs. To get an ID back, insert one row at a time and ask the server for the value it assigned:
|
||||
|
||||
```python
|
||||
cur.execute("INSERT INTO orders (customer_id, total) VALUES (?, ?)", (7, 42.0))
|
||||
cur.execute("SELECT DBINFO('sqlca.sqlerrd1') FROM systables WHERE tabid = 1")
|
||||
new_id = cur.fetchone()[0]
|
||||
```
|
||||
|
||||
`sqlca.sqlerrd1` is where Informix records the serial value from the most recent INSERT on this connection, so read it before running any other statement.
|
||||
|
||||
There is no `cursor.lastrowid`, though an earlier version of this page said there was.
|
||||
|
||||
For batch inserts that need the IDs, the idiomatic pattern is:
|
||||
|
||||
```sql
|
||||
INSERT INTO orders (id, customer_id, total)
|
||||
SELECT MAX(id) + ROWNUM, ?, ? FROM orders
|
||||
```
|
||||
|
||||
…or pre-generate IDs from a sequence in your app code. Or accept `IfxPy`'s same limitation: `IfxPy.executemany` doesn't return per-row generated keys either.
|
||||
98
docs-site/src/content/docs/how-to/migrate-from-ifxpy.md
Normal file
98
docs-site/src/content/docs/how-to/migrate-from-ifxpy.md
Normal file
@ -0,0 +1,98 @@
|
||||
---
|
||||
title: Migrate from IfxPy
|
||||
description: API differences between IfxPy and informix-driver, what's the same, what's not, and how to migrate incrementally.
|
||||
sidebar:
|
||||
order: 7
|
||||
---
|
||||
|
||||
If you have working IfxPy code, migration is mostly mechanical. Both drivers are PEP 249 with similar shapes; the differences are in connection construction, a few cursor extensions, and async support.
|
||||
|
||||
## Connection construction
|
||||
|
||||
```python
|
||||
# IfxPy
|
||||
import IfxPy
|
||||
conn_str = (
|
||||
"DATABASE=mydb;HOSTNAME=db.example.com;PORT=9088;"
|
||||
"PROTOCOL=onsoctcp;UID=informix;PWD=...;SERVICE=informix"
|
||||
)
|
||||
conn = IfxPy.connect(conn_str, "", "")
|
||||
|
||||
# informix-driver
|
||||
import informix_db
|
||||
conn = informix_db.connect(
|
||||
host="db.example.com", port=9088,
|
||||
user="informix", password="...",
|
||||
database="mydb", server="informix",
|
||||
)
|
||||
```
|
||||
|
||||
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported, deliberately: they're a security and parsing footgun.
|
||||
|
||||
## The DB-API surface is the same
|
||||
|
||||
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, and `close()` all behave per PEP 249.
|
||||
|
||||
The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `DatabaseError`, `DataError`, `OperationalError`, `IntegrityError`, `InternalError`, `ProgrammingError`, `NotSupportedError`.
|
||||
|
||||
## What IfxPy has that we don't (yet)
|
||||
|
||||
- **Named-parameter `callproc`**. We have `fast_path_call` for direct UDF/SPL invocation but the API shape differs.
|
||||
- **IBM-specific scrollable cursor extensions**. We support PEP 249 scrollable cursors (`scroll(value, mode)`) but not IfxPy's `last`/`prior`/`relative` shortcuts.
|
||||
- **`cursor.set_chunk_size`**. We tune fetch behavior via the buffered reader; no per-cursor knob.
|
||||
|
||||
## What we have that IfxPy doesn't
|
||||
|
||||
- **Async API** (`from informix_db import aio`)
|
||||
- **Connection pool** (`informix_db.create_pool` / `aio.create_pool`)
|
||||
- **Type-safe annotations**: `informix-driver` ships with `py.typed`
|
||||
- **Python 3.12+ support**
|
||||
- **Pipelined `executemany`**: 1.6× faster than IfxPy's per-row implementation
|
||||
- **Rows addressable by name**: `row["col"]` and `row.col`, not just `row[0]`
|
||||
|
||||
## Reading rows by name
|
||||
|
||||
IfxPy gives you positional access and nothing else, which is why most
|
||||
IfxPy codebases grow a helper like this:
|
||||
|
||||
```python
|
||||
cols = [c[0].lower() for c in cur.description]
|
||||
row_dict = dict(zip(cols, cur.fetchone()))
|
||||
```
|
||||
|
||||
That gets you a dict and still no attribute access. Ask for `Row`
|
||||
instead and all three work at once:
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||
cur.execute("SELECT config_key, config_value FROM settings")
|
||||
row = cur.fetchone()
|
||||
|
||||
row[0], row["config_key"], row.config_key
|
||||
```
|
||||
|
||||
Set on the connection, so it applies to every cursor from it. The
|
||||
`.lower()` in the hand-written version is already a no-op, incidentally:
|
||||
Informix folds unquoted identifiers to lower case.
|
||||
|
||||
It is opt-in because it costs about 9% on bulk fetch. See
|
||||
[the API reference](/reference/api/#rows) for the numbers and the
|
||||
edge cases.
|
||||
|
||||
## Migrating incrementally
|
||||
|
||||
If your codebase has hundreds of IfxPy call sites, you can do a partial migration:
|
||||
|
||||
1. Replace connection construction with `informix-driver` at the application boundary.
|
||||
2. Where you used `IfxPy.fetch_assoc` (returns dict), wrap our cursor with a small adapter:
|
||||
```python
|
||||
def fetch_assoc(cur):
|
||||
row = cur.fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
return dict(zip([c[0] for c in cur.description], row))
|
||||
```
|
||||
3. For `IfxPy.exec_immediate`, use `cur.execute(sql)` with no params.
|
||||
4. For `IfxPy.bind_param`, use the params arg of `execute()`: `cur.execute(sql, (a, b, c))`.
|
||||
|
||||
Most application code can be migrated with `sed`-level transformations.
|
||||
71
docs-site/src/content/docs/how-to/pool.mdx
Normal file
71
docs-site/src/content/docs/how-to/pool.mdx
Normal file
@ -0,0 +1,71 @@
|
||||
---
|
||||
title: Use the connection pool
|
||||
description: Sync and async connection pools, covering sizing, timeouts, lifecycle, and threading.
|
||||
sidebar:
|
||||
order: 2
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
The connection pool amortizes the ~11 ms login handshake across many queries and gives you a thread-safe / task-safe acquire-release API. Use it any time the same process makes more than a handful of queries.
|
||||
|
||||
## Sync pool
|
||||
|
||||
```python
|
||||
import informix_db
|
||||
|
||||
pool = informix_db.create_pool(
|
||||
host="db.example.com", port=9088,
|
||||
user="informix", password="...",
|
||||
database="mydb", server="informix",
|
||||
min_size=2, # warm up at least 2 connections at create time
|
||||
max_size=10, # hard cap; acquires beyond this block
|
||||
acquire_timeout=5.0, # raise PoolTimeout if no connection in 5s
|
||||
max_idle=600.0, # close connections idle longer than 10 min
|
||||
)
|
||||
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT id, name FROM users WHERE id = ?", (42,))
|
||||
print(cur.fetchone())
|
||||
|
||||
pool.close()
|
||||
```
|
||||
|
||||
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically, so you never see a dirty connection from `pool.connection()`.
|
||||
|
||||
## Async pool
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from informix_db import aio
|
||||
|
||||
async def main():
|
||||
pool = await aio.create_pool(
|
||||
host="db.example.com",
|
||||
user="informix", password="...",
|
||||
database="mydb",
|
||||
min_size=2, max_size=10,
|
||||
)
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
|
||||
print(await cur.fetchone())
|
||||
await pool.close()
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
Same semantics, `async`-aware. Cancellation is safe here too: a cancelled task does not leak an in-flight worker onto a recycled connection.
|
||||
|
||||
## Sizing
|
||||
|
||||
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes", usually 8–16 for typical web/API services.
|
||||
|
||||
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`**. The server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
|
||||
|
||||
## Threading
|
||||
|
||||
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally, since `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
|
||||
|
||||
Phase 27 added a per-connection wire lock that makes accidental sharing safe (interleaved PDUs serialize correctly), but you should still give each thread its own connection. The lock is a backstop, not a license.
|
||||
83
docs-site/src/content/docs/how-to/smart-lobs.mdx
Normal file
83
docs-site/src/content/docs/how-to/smart-lobs.mdx
Normal file
@ -0,0 +1,83 @@
|
||||
---
|
||||
title: BLOB / CLOB read & write
|
||||
description: Reading and writing smart-LOB columns end-to-end in pure Python.
|
||||
sidebar:
|
||||
order: 6
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
`informix-driver` reads and writes smart-LOB columns (BLOB and CLOB) end-to-end without any native machinery. The implementation uses Informix's `lotofile` and `filetoblob` SQL functions, intercepted at the `SQ_FILE` (98) wire-protocol level.
|
||||
|
||||
## Reading a BLOB
|
||||
|
||||
```python
|
||||
data: bytes = cur.read_blob_column(
|
||||
"SELECT data FROM photos WHERE id = ?",
|
||||
(42,),
|
||||
)
|
||||
```
|
||||
|
||||
`read_blob_column` returns the raw bytes. For very large BLOBs (multi-GB), see the streaming variant below.
|
||||
|
||||
## Writing a BLOB
|
||||
|
||||
```python
|
||||
cur.write_blob_column(
|
||||
"INSERT INTO photos VALUES (?, BLOB_PLACEHOLDER)",
|
||||
blob_data=jpeg_bytes,
|
||||
params=(42,),
|
||||
)
|
||||
```
|
||||
|
||||
The `BLOB_PLACEHOLDER` token in the SQL marks where the BLOB data goes. Other parameters are bound positionally to `params=`.
|
||||
|
||||
## Reading a CLOB
|
||||
|
||||
CLOBs use the same methods as BLOBs, and there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
|
||||
|
||||
```python
|
||||
raw: bytes = cur.read_blob_column(
|
||||
"SELECT body FROM articles WHERE id = ?",
|
||||
(42,),
|
||||
)
|
||||
text: str = raw.decode("iso-8859-1") # or your DB_LOCALE's codec
|
||||
```
|
||||
|
||||
Returning `bytes` rather than `str` is deliberate: the driver transports the LOB over the `SQ_FILE` channel and never sees the column's declared character set, so guessing an encoding here would be exactly the kind of silent assumption that produces mojibake in one deployment and works fine in another.
|
||||
|
||||
## Writing a CLOB
|
||||
|
||||
Pass `clob=True` so the write routes through `filetoclob` rather than `filetoblob`, and encode the text yourself:
|
||||
|
||||
```python
|
||||
cur.write_blob_column(
|
||||
"INSERT INTO articles VALUES (?, BLOB_PLACEHOLDER)",
|
||||
"long article text… café résumé".encode("iso-8859-1"),
|
||||
(42,),
|
||||
clob=True,
|
||||
)
|
||||
```
|
||||
|
||||
The placeholder token is `BLOB_PLACEHOLDER` for both BLOB and CLOB writes.
|
||||
|
||||
## Server-side prerequisites
|
||||
|
||||
<Aside type="caution">
|
||||
Smart-LOBs require server-side configuration that the IBM Developer Edition Docker image doesn't ship with by default:
|
||||
|
||||
- An **`sbspace`** must be created (`onspaces -c -S sbspace1 -p ...`)
|
||||
- `SBSPACENAME` must be set in `$ONCONFIG`
|
||||
- A **level-0 archive** must be taken (`ontape -s -L 0`) before BLOBs can be created
|
||||
- The database must be created **with logging** (`CREATE DATABASE foo WITH LOG`)
|
||||
|
||||
Full setup commands are in [`docs/DECISION_LOG.md` §10](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md).
|
||||
</Aside>
|
||||
|
||||
## How it works (briefly)
|
||||
|
||||
The `lotofile` server function returns a smart-LOB descriptor as a regular result column when called via `SELECT`. The driver intercepts the `SQ_FILE` (PDU 98) response that contains the LOB bytes and reassembles them client-side.
|
||||
|
||||
Writing reverses the flow: `filetoblob` is invoked via a placeholder pattern in the SQL, the driver sends the bytes via `SQ_FILE` PDUs, and the server stores them in the sbspace.
|
||||
|
||||
The architectural pivot from the heavier `SQ_FPROUTINE` + `SQ_LODATA` stack to this lighter `SQ_FILE` intercept is documented in [Phase 10/11 of the decision log](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md). The result is roughly 3× smaller than originally projected.
|
||||
55
docs-site/src/content/docs/how-to/tls.mdx
Normal file
55
docs-site/src/content/docs/how-to/tls.mdx
Normal file
@ -0,0 +1,55 @@
|
||||
---
|
||||
title: Connect with TLS
|
||||
description: TLS-listener configuration, bring-your-own SSL context, and dev-mode self-signed handling.
|
||||
sidebar:
|
||||
order: 1
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
Informix uses **dedicated TLS-enabled listener ports** (configured server-side in `sqlhosts`) rather than STARTTLS upgrade. Point `port` at the TLS listener (typically `9089`) when `tls` is enabled.
|
||||
|
||||
## Production: bring your own SSL context
|
||||
|
||||
```python
|
||||
import ssl
|
||||
import informix_db
|
||||
|
||||
ctx = ssl.create_default_context(cafile="/path/to/ca.pem")
|
||||
# Optional: client cert auth
|
||||
# ctx.load_cert_chain(certfile="/path/to/client.pem", keyfile="/path/to/client.key")
|
||||
|
||||
conn = informix_db.connect(
|
||||
host="db.example.com",
|
||||
port=9089,
|
||||
user="informix",
|
||||
password="...",
|
||||
database="mydb",
|
||||
server="informix",
|
||||
tls=ctx,
|
||||
)
|
||||
```
|
||||
|
||||
Bring-your-own context is the recommended production pattern, giving you full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
|
||||
|
||||
## Dev / self-signed: tls=True
|
||||
|
||||
```python
|
||||
informix_db.connect(host="127.0.0.1", port=9089, ..., tls=True)
|
||||
```
|
||||
|
||||
`tls=True` is a convenience for development. It builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
|
||||
|
||||
## Server-side configuration
|
||||
|
||||
The Informix server needs a TLS listener entry in `sqlhosts`:
|
||||
|
||||
```
|
||||
informix_tls onsoctcp myhost 9089
|
||||
```
|
||||
|
||||
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089`, with no configuration needed.
|
||||
|
||||
<Aside type="tip">
|
||||
If your connection hangs at the handshake, you've probably pointed at the non-TLS port (`9088` instead of `9089`). The non-TLS listener will accept the TCP connection but won't speak TLS, so the handshake stalls.
|
||||
</Aside>
|
||||
92
docs-site/src/content/docs/index.mdx
Normal file
92
docs-site/src/content/docs/index.mdx
Normal file
@ -0,0 +1,92 @@
|
||||
---
|
||||
title: informix-driver
|
||||
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket, with no CSDK, no JVM, and no native libraries.
|
||||
template: splash
|
||||
hero:
|
||||
tagline: ''
|
||||
editUrl: false
|
||||
lastUpdated: false
|
||||
next: false
|
||||
prev: false
|
||||
---
|
||||
|
||||
import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
|
||||
|
||||
<div class="ifx-features">
|
||||
<div class="ifx-feature">
|
||||
<Icon name="rocket" class="ifx-feature__icon" />
|
||||
<h3>1.6× faster bulk inserts than IfxPy</h3>
|
||||
<p>Pipelined <code>executemany</code> sends every BIND+EXECUTE PDU before draining responses. IBM's C driver still pays one round-trip per row. We figured we could do better.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="seti:python" class="ifx-feature__icon" />
|
||||
<h3>~10% behind IfxPy on bulk fetches</h3>
|
||||
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics, for now.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="puzzle" class="ifx-feature__icon" />
|
||||
<h3>50 KB wheel. Zero native deps.</h3>
|
||||
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.10–3.14, including the versions IfxPy doesn't.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="sun" class="ifx-feature__icon" />
|
||||
<h3>Async, like a modern driver should be</h3>
|
||||
<p>FastAPI, aiohttp, asyncio. Pool, connections, cursors all have <code>async def</code> versions. IfxPy has none of this.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="approve-check" class="ifx-feature__icon" />
|
||||
<h3>PEP 249, no surprises</h3>
|
||||
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy, plus threadsafe sharing through a per-connection wire lock.</p>
|
||||
</div>
|
||||
<div class="ifx-feature">
|
||||
<Icon name="document" class="ifx-feature__icon" />
|
||||
<h3>Clean-room reverse engineering</h3>
|
||||
<p>Decompiled IBM JDBC. Annotated <code>socat</code> captures. Differential testing against IfxPy on every codec path. Every architectural decision lives in <a href="/explain/phase-log/">the phase log</a>. Receipts.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
## A query, end to end
|
||||
|
||||
```python
|
||||
import informix_db
|
||||
|
||||
with informix_db.connect(
|
||||
host="db.example.com", port=9088,
|
||||
user="informix", password="...",
|
||||
database="mydb", server="informix",
|
||||
) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT id, name FROM users WHERE id = ?", (42,))
|
||||
user_id, name = cur.fetchone()
|
||||
```
|
||||
|
||||
That's it. No `IBM_DB_HOME`. No DSN file. No `libcrypt.so.1`.
|
||||
|
||||
## WTF did you build this for?
|
||||
|
||||
The existing tools were not my style.
|
||||
|
||||
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That means all of them: `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
|
||||
|
||||
The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9). It needs four `LD_LIBRARY_PATH` entries. It needs `setuptools < 58`. And IfxPy itself is broken on Python 3.12+. For containerized deployments, ETL pipelines, FastAPI services, or anywhere a build toolchain on the runtime is friction, this driver is the alternative that didn't previously exist. Now it does.
|
||||
|
||||
## Read next
|
||||
|
||||
<CardGrid>
|
||||
<Card title="Install & first query" icon="rocket">
|
||||
Up and running with the IBM Informix Developer Edition Docker image in five minutes.
|
||||
[Get started →](/start/quickstart/)
|
||||
</Card>
|
||||
<Card title="Compared to IfxPy" icon="random">
|
||||
Head-to-head benchmarks, install gauntlet, and when each driver wins.
|
||||
[Read →](/start/vs-ifxpy/)
|
||||
</Card>
|
||||
<Card title="The buffered reader" icon="information">
|
||||
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1×, and the architectural mistake the first pass got wrong.
|
||||
[Read →](/explain/buffered-reader/)
|
||||
</Card>
|
||||
<Card title="Architecture" icon="puzzle">
|
||||
SQLI on the wire. Sockets, framing, codec, resultsets. How the layers stack.
|
||||
[Read →](/explain/architecture/)
|
||||
</Card>
|
||||
</CardGrid>
|
||||
214
docs-site/src/content/docs/reference/api.md
Normal file
214
docs-site/src/content/docs/reference/api.md
Normal file
@ -0,0 +1,214 @@
|
||||
---
|
||||
title: API surface
|
||||
description: Module-level functions, Connection / Cursor / Pool APIs, async equivalents.
|
||||
sidebar:
|
||||
order: 1
|
||||
---
|
||||
|
||||
The top-level surface of `informix_db` and `informix_db.aio`. For full PEP 249 details, see the standard's [DB-API 2.0 spec](https://peps.python.org/pep-0249/).
|
||||
|
||||
## Module-level
|
||||
|
||||
```python
|
||||
import informix_db
|
||||
|
||||
informix_db.connect(...) -> Connection
|
||||
informix_db.create_pool(...) -> Pool
|
||||
informix_db.apilevel # "2.0"
|
||||
informix_db.threadsafety # 1
|
||||
informix_db.paramstyle # "numeric"
|
||||
```
|
||||
|
||||
```python
|
||||
from informix_db import aio
|
||||
|
||||
await aio.connect(...) -> aio.Connection
|
||||
await aio.create_pool(...) -> aio.Pool
|
||||
```
|
||||
|
||||
## connect()
|
||||
|
||||
```python
|
||||
informix_db.connect(
|
||||
*,
|
||||
host: str,
|
||||
port: int = 9088,
|
||||
user: str,
|
||||
password: str,
|
||||
database: str | None,
|
||||
server: str, # DBSERVERNAME (not hostname)
|
||||
autocommit: bool = False,
|
||||
connect_timeout: float | None = None,
|
||||
read_timeout: float | None = None,
|
||||
keepalive: bool = False,
|
||||
client_locale: str = "en_US.8859-1",
|
||||
env: dict[str, str] | None = None,
|
||||
tls: bool | ssl.SSLContext = False,
|
||||
) -> Connection
|
||||
```
|
||||
|
||||
## Connection
|
||||
|
||||
| Method / property | Description |
|
||||
|---|---|
|
||||
| `cursor(scrollable=False)` | Returns a new `Cursor`. Only one scrollable cursor may be open per connection; see below. |
|
||||
| `commit()` | Commits the current transaction. |
|
||||
| `rollback()` | Rolls back the current transaction. |
|
||||
| `close()` | Closes the connection. Idempotent. |
|
||||
| `fast_path_call(routine, *args)` | Direct UDF/SPL invocation, bypassing PREPARE/EXECUTE/FETCH. |
|
||||
| `encoding` | Resolved Python codec for `client_locale`. |
|
||||
| `closed` | `True` after `close()`. |
|
||||
| `server_version` | Server release, e.g. `'…Version 12.10.FC6'`. Costs one query on first access, then cached. |
|
||||
| `server_version_internal` | Raw login-response version string. Free. See note below. |
|
||||
| `server_capabilities` | Negotiated `ServerCapabilities`, or `None` if the reply couldn't be decoded. |
|
||||
|
||||
`Connection` is also a context manager (`with informix_db.connect(...) as conn:`), which closes on exit.
|
||||
|
||||
There is no `conn.transaction()` helper and no `conn.autocommit` attribute. Earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(..., autocommit=False)
|
||||
try:
|
||||
cur = conn.cursor()
|
||||
cur.execute("UPDATE accounts SET balance = balance - ? WHERE id = ?", (100, 1))
|
||||
cur.execute("UPDATE accounts SET balance = balance + ? WHERE id = ?", (100, 2))
|
||||
conn.commit()
|
||||
except Exception:
|
||||
conn.rollback()
|
||||
raise
|
||||
```
|
||||
|
||||
:::caution[One statement per session]
|
||||
Informix gives a session a single statement slot. A non-scrollable cursor never trips over this, because it materializes its rows and releases the statement before `execute()` returns. A **scrollable** cursor holds the slot open on purpose, so while one is open, any other statement on that same connection raises `ProgrammingError`:
|
||||
|
||||
```python
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT id, body FROM big_table ORDER BY id")
|
||||
scroll.fetch_first()
|
||||
|
||||
other = conn.cursor()
|
||||
other.execute("SELECT COUNT(*) FROM audit") # ProgrammingError
|
||||
```
|
||||
|
||||
Close the scrollable cursor first, or use a second connection for the other query. Re-running `execute()` on the scrollable cursor itself is fine; it closes its own server-side cursor first.
|
||||
|
||||
Before `2026.09.02` this was not refused. The server answered the second statement with `-285` and destroyed the scrollable cursor as well, whose next fetch then returned `-267`, "the transaction has been rolled back". Two errors, neither naming the cause.
|
||||
:::
|
||||
|
||||
:::note[Transaction control written as SQL]
|
||||
`commit()` and `rollback()` are the supported way to end a transaction, but `cursor.execute("BEGIN WORK")` and its `COMMIT` / `ROLLBACK` counterparts work too, and the connection tracks them from `2026.09.02` onward.
|
||||
|
||||
That matters because it used not to. Under `autocommit=True`, a SQL `BEGIN WORK` opened a real transaction the connection never learned about, and since `rollback()` is guarded by that state, **it returned successfully having sent nothing**. The rows it was asked to discard survived, and a pooled connection went back into circulation still holding the transaction and its locks.
|
||||
:::
|
||||
|
||||
:::note[Two version numbers, and why]
|
||||
The login response carries Informix's *internal* protocol version, not the release you installed: 12.10 announces itself as `9.56`, 14.10 as `9.59`, and 15 as `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers, which is why all three speak an identical SQLI dialect.
|
||||
|
||||
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')`. That is one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
|
||||
:::
|
||||
|
||||
## Rows
|
||||
|
||||
Fetch methods return plain tuples by default. Pass `row_factory` to get rows that also answer to a column name and an attribute, the shape `pyodbc` and `mssql-python` provide:
|
||||
|
||||
```python
|
||||
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT tabid, tabname FROM systables")
|
||||
row = cur.fetchone()
|
||||
|
||||
row[0] # 1
|
||||
row["tabname"] # 'systables'
|
||||
row.tabname # 'systables'
|
||||
```
|
||||
|
||||
Set on the connection, so it applies to every cursor from it. A cursor can override it with `cur.row_factory = informix_db.Row`. Pools and the async API forward it unchanged.
|
||||
|
||||
`Row` subclasses `tuple`, so `row == (1, "systables")` is still true and anything treating rows as sequences keeps working. Slices return plain tuples, since a slice has no column map. `row.keys()` gives the column names and `row._asdict()` gives a dict.
|
||||
|
||||
:::note[Why it is opt-in]
|
||||
On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` becomes a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript.
|
||||
|
||||
Defaulting it on would move the [1.05-1.15x ratio against IfxPy](/start/vs-ifxpy/) to roughly 1.15-1.25x. That is a good trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is yours to make.
|
||||
:::
|
||||
|
||||
Four behaviours worth knowing:
|
||||
|
||||
- Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`. The `.lower()` people add by hand when building this themselves is already a no-op.
|
||||
- Expression columns get server-generated names like `(count(*))`, which cannot be Python attributes. Those are reachable by subscript only.
|
||||
- Duplicate column names resolve to the first occurrence, matching `pyodbc`. Positional access still reaches both.
|
||||
- **A column beats a method of the same name.** `tuple` defines `count` and `index`; `Row` adds `keys`, `_asdict` and `_fields`. A column with any of those names wins, because otherwise it would hand back a bound method instead of a value. The methods stay reachable as `tuple.count(row, x)` and `Row.keys(row)`.
|
||||
|
||||
## Cursor
|
||||
|
||||
| Method / property | Description |
|
||||
|---|---|
|
||||
| `execute(sql, params=())` | Prepare + execute. Returns the cursor. |
|
||||
| `executemany(sql, seq_of_params)` | Pipelined batch execute. |
|
||||
| `fetchone()` | One row tuple, or `None`. |
|
||||
| `fetchmany(size=arraysize)` | List of row tuples. |
|
||||
| `fetchall()` | All remaining rows. |
|
||||
| `scroll(value, mode="relative")` | Scrollable cursor positioning. Needs `cursor(scrollable=True)`. |
|
||||
| `fetch_first()` / `fetch_last()` | Jump to the first / last row. Scrollable cursors only. |
|
||||
| `fetch_prior()` / `fetch_relative(n)` / `fetch_absolute(n)` | Relative and absolute positioning. Scrollable cursors only. |
|
||||
| `read_blob_column(sql, params)` | Read a BLOB **or** CLOB column → `bytes`. |
|
||||
| `write_blob_column(sql, data, params, clob=False)` | Write a BLOB column; pass `clob=True` for CLOB. |
|
||||
| `close()` | Closes the cursor + releases server resources. |
|
||||
| `closed` | `True` after `close()`. |
|
||||
| `description` | Sequence of column descriptors per PEP 249. |
|
||||
| `rowcount` | Affected row count for DML; `-1` for SELECT. |
|
||||
| `rownumber` | Current 0-indexed position, or `None` before the first row. |
|
||||
| `arraysize` | Default `fetchmany()` size. |
|
||||
|
||||
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute. Earlier versions of this page listed all three, and none have existed.
|
||||
|
||||
CLOBs go through the BLOB methods. `read_blob_column` returns `bytes` either way, so decode it yourself with the column's encoding:
|
||||
|
||||
```python
|
||||
cur.write_blob_column(
|
||||
"INSERT INTO docs VALUES (?, BLOB_PLACEHOLDER)",
|
||||
"café résumé".encode("iso-8859-1"),
|
||||
(1,),
|
||||
clob=True,
|
||||
)
|
||||
raw = cur.read_blob_column("SELECT txt FROM docs WHERE id = ?", (1,))
|
||||
text = raw.decode("iso-8859-1")
|
||||
```
|
||||
|
||||
For a server-assigned `SERIAL` after an INSERT, ask the server. Informix exposes it through `DBINFO`:
|
||||
|
||||
```python
|
||||
cur.execute("INSERT INTO people (name) VALUES (?)", ("ada",))
|
||||
cur.execute("SELECT DBINFO('sqlca.sqlerrd1') FROM systables WHERE tabid = 1")
|
||||
new_id = cur.fetchone()[0]
|
||||
```
|
||||
|
||||
## Pool
|
||||
|
||||
| Method | Description |
|
||||
|---|---|
|
||||
| `connection()` | Context manager yielding a connection from the pool. |
|
||||
| `close()` | Closes all idle connections; waits for in-use to return. |
|
||||
| `closed` | `True` after `close()`. |
|
||||
| `size` | Current pool size. |
|
||||
| `idle_count` | Connections currently idle in the pool. |
|
||||
|
||||
`aio.Pool` is identical except its `connection()` is an async context manager and `close()` is `async`.
|
||||
|
||||
## Exceptions
|
||||
|
||||
```
|
||||
Error
|
||||
├── Warning
|
||||
└── DatabaseError
|
||||
├── InterfaceError
|
||||
├── DataError
|
||||
├── OperationalError
|
||||
│ └── PoolTimeout
|
||||
├── IntegrityError
|
||||
├── InternalError
|
||||
├── ProgrammingError
|
||||
└── NotSupportedError
|
||||
```
|
||||
|
||||
See [Exceptions & error codes](/reference/exceptions/) for the SQLCODE → exception mapping.
|
||||
76
docs-site/src/content/docs/reference/benchmarks.mdx
Normal file
76
docs-site/src/content/docs/reference/benchmarks.mdx
Normal file
@ -0,0 +1,76 @@
|
||||
---
|
||||
title: Performance baselines
|
||||
description: Single-connection benchmark results for codec, framing, end-to-end queries, and IfxPy.
|
||||
sidebar:
|
||||
order: 5
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
These are the steady-state numbers for the current release (`2026.05.05.12`), measured against the IBM Informix Developer Edition Docker container on loopback.
|
||||
|
||||
<Aside type="note">
|
||||
All numbers are **median over 10+ rounds** (pytest-benchmark `--rounds 10`). Reproduce with `make bench` from the repo root.
|
||||
</Aside>
|
||||
|
||||
## Codec micro-benchmarks
|
||||
|
||||
| Operation | Mean | Throughput |
|
||||
|---|---:|---:|
|
||||
| `decode(int)` per cell | 139 ns | 7.2M ops/sec |
|
||||
| `decode(varchar)` per cell | 280 ns | 3.6M ops/sec |
|
||||
| `decode(decimal)` per cell | 410 ns | 2.4M ops/sec |
|
||||
| `parse_tuple_payload` per row (5 cols) | 1.4 µs | 715K rows/sec |
|
||||
|
||||
## End-to-end
|
||||
|
||||
| Operation | Mean | Throughput |
|
||||
|---|---:|---:|
|
||||
| `SELECT 1` round-trip | ~140 µs | ~7K queries/sec |
|
||||
| 1000-row SELECT | ~1.0 ms | ~990K rows/sec sustained |
|
||||
| `executemany(1000)` in transaction | 32 ms | ~31,000 rows/sec |
|
||||
| Pool acquire + query + release | 295 µs | ~3.4K queries/sec |
|
||||
| Cold connect (login handshake) | 11 ms | ~90 connections/sec |
|
||||
|
||||
## vs IfxPy 3.0.5
|
||||
|
||||
| Benchmark | IfxPy | informix-driver | Result |
|
||||
|---|---:|---:|---:|
|
||||
| Single-row SELECT round-trip | 118 µs | 114 µs | comparable |
|
||||
| ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster |
|
||||
| Cold connect | 11.0 ms | 10.5 ms | comparable |
|
||||
| `executemany(1k)` | 23.5 ms | 23.2 ms | tied |
|
||||
| `executemany(10k)` | 259 ms | **161 ms** | **informix-driver 1.6× faster** |
|
||||
| `executemany(100k)` | 2376 ms | **1487 ms** | **informix-driver 1.6× faster** |
|
||||
| `SELECT 1k` | 1.34 ms | 1.72 ms | IfxPy 1.28× |
|
||||
| `SELECT 10k` | 11.7 ms | 16.1 ms | IfxPy 1.07× |
|
||||
| `SELECT 100k` | 116 ms | 169 ms | IfxPy 1.15× |
|
||||
|
||||
For the methodology, IQR caveats, and reproduction instructions, see [Compared to IfxPy](/start/vs-ifxpy/).
|
||||
|
||||
## Phase progression on bulk SELECT
|
||||
|
||||
| Phase | 100k-row SELECT | Ratio vs IfxPy |
|
||||
|---|---:|---:|
|
||||
| Phase 36 | 280 ms | 2.40× slower |
|
||||
| Phase 37 (per-column readers) | 250 ms | 2.10× slower |
|
||||
| Phase 38 (codegen-inlined decoders) | 237 ms | 2.04× slower |
|
||||
| **Phase 39 (connection-scoped buffered reader)** | **169 ms** | **1.15× slower** |
|
||||
|
||||
The Phase 39 jump is documented in [The buffered reader](/explain/buffered-reader/).
|
||||
|
||||
## Reproducing
|
||||
|
||||
```bash
|
||||
git clone https://git.supported.systems/warehack.ing/informix-db
|
||||
cd informix-db
|
||||
make ifx-up # starts the dev container
|
||||
make bench # runs all benchmarks
|
||||
make compare # head-to-head vs IfxPy (handles IfxPy's install gauntlet)
|
||||
```
|
||||
|
||||
For just the bulk-fetch progression:
|
||||
|
||||
```bash
|
||||
pytest -m benchmark tests/benchmarks/test_select_scaling.py
|
||||
```
|
||||
67
docs-site/src/content/docs/reference/config.md
Normal file
67
docs-site/src/content/docs/reference/config.md
Normal file
@ -0,0 +1,67 @@
|
||||
---
|
||||
title: Configuration & env flags
|
||||
description: Runtime environment variables, connection arguments, pool tunables.
|
||||
sidebar:
|
||||
order: 3
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Default | Effect |
|
||||
|---|---|---|
|
||||
| `IFX_BUFFERED_READER` | `1` | Enable connection-scoped read buffer (Phase 39). Set to `0` to disable. |
|
||||
| `IFX_DEBUG_WIRE` | unset | When set to a truthy value, log wire-level PDU framing to stderr. Verbose; for debugging only. |
|
||||
| `IFX_PROTOCOL_TRACE` | unset | When set to a path, write annotated wire captures to the file. |
|
||||
| `IFX_DISABLE_PIPELINE` | unset | Disables pipelined `executemany` (Phase 33). Use only to A/B-measure. |
|
||||
| `INFORMIXSERVER` | *(none)* | Read by `connect()` if `server=` is not provided. |
|
||||
|
||||
Environment variables are read at connection construction. Changing them at runtime doesn't affect existing connections.
|
||||
|
||||
## Connection arguments
|
||||
|
||||
Full keyword list for `informix_db.connect()`:
|
||||
|
||||
| Arg | Type | Default | Notes |
|
||||
|---|---|---|---|
|
||||
| `host` | `str` | required | TCP host. |
|
||||
| `port` | `int` | `9088` | TCP port. `9089` is the typical TLS listener. |
|
||||
| `user` | `str` | required | |
|
||||
| `password` | `str` | required | |
|
||||
| `database` | `str \| None` | required | `None` logs in without selecting a DB. |
|
||||
| `server` | `str` | required | DBSERVERNAME, not hostname. |
|
||||
| `autocommit` | `bool` | `False` | |
|
||||
| `connect_timeout` | `float \| None` | `None` | TCP+login timeout. |
|
||||
| `read_timeout` | `float \| None` | `None` | Per-read timeout. |
|
||||
| `keepalive` | `bool` | `False` | `SO_KEEPALIVE`. |
|
||||
| `client_locale` | `str` | `"en_US.8859-1"` | See [SQL ↔ Python types](/reference/types/) for codec mapping. |
|
||||
| `env` | `dict[str, str] \| None` | `None` | Server-side session env. |
|
||||
| `tls` | `bool \| ssl.SSLContext` | `False` | See [Connect with TLS](/how-to/tls/). |
|
||||
|
||||
## Pool tunables
|
||||
|
||||
| Arg | Default | Effect |
|
||||
|---|---|---|
|
||||
| `min_size` | `1` | Connections to pre-warm at create time. |
|
||||
| `max_size` | `10` | Hard cap. |
|
||||
| `acquire_timeout` | `30.0` | Raise `PoolTimeout` after this many seconds. |
|
||||
| `max_idle` | `600.0` | Close connections idle longer than this (seconds). |
|
||||
| `health_check` | `True` | Validate idle connections before returning from the pool. |
|
||||
|
||||
All other connection arguments are forwarded to each pool-created connection.
|
||||
|
||||
## Server-side session env
|
||||
|
||||
The `env={}` parameter of `connect()` sets server session variables sent in the login PDU:
|
||||
|
||||
```python
|
||||
informix_db.connect(
|
||||
...,
|
||||
env={
|
||||
"OPT_GOAL": "-1", # optimize for first-row return
|
||||
"OPTOFC": "1", # auto-free cursors at fetch-close
|
||||
"IFX_AUTOFREE": "1",
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
`CLIENT_LOCALE` is set automatically from `client_locale=`, so don't put it in `env=`.
|
||||
68
docs-site/src/content/docs/reference/exceptions.md
Normal file
68
docs-site/src/content/docs/reference/exceptions.md
Normal file
@ -0,0 +1,68 @@
|
||||
---
|
||||
title: Exceptions & error codes
|
||||
description: PEP 249 exception hierarchy, Informix SQLCODE → Python exception mapping.
|
||||
sidebar:
|
||||
order: 4
|
||||
---
|
||||
|
||||
The exception hierarchy is per PEP 249. All exceptions live in `informix_db` and inherit from `informix_db.Error`.
|
||||
|
||||
## Hierarchy
|
||||
|
||||
```
|
||||
Error
|
||||
├── Warning
|
||||
└── DatabaseError
|
||||
├── InterfaceError
|
||||
├── DataError
|
||||
├── OperationalError
|
||||
│ └── PoolTimeout
|
||||
├── IntegrityError
|
||||
├── InternalError
|
||||
├── ProgrammingError
|
||||
└── NotSupportedError
|
||||
```
|
||||
|
||||
## When each is raised
|
||||
|
||||
| Exception | Typical cause |
|
||||
|---|---|
|
||||
| `InterfaceError` | Misuse of the driver API itself (e.g. fetch on a closed cursor). |
|
||||
| `DataError` | Type conversion failures, codec errors. |
|
||||
| `OperationalError` | Network errors, timeouts, server unavailable, login failures. |
|
||||
| `IntegrityError` | Constraint violations (PK, FK, unique, NOT NULL). |
|
||||
| `InternalError` | Driver-internal invariant violation (file a bug). |
|
||||
| `ProgrammingError` | Bad SQL syntax, missing tables, parameter binding errors. |
|
||||
| `NotSupportedError` | Driver doesn't support the requested operation. |
|
||||
| `PoolTimeout` | Pool acquire exceeded `acquire_timeout`. |
|
||||
|
||||
## SQLCODE mapping
|
||||
|
||||
Informix returns SQLCODE values; the driver maps them to the appropriate exception. A few common ones:
|
||||
|
||||
| SQLCODE | Meaning | Exception |
|
||||
|---|---|---|
|
||||
| `-201` | Syntax error | `ProgrammingError` |
|
||||
| `-206` | Table not found | `ProgrammingError` |
|
||||
| `-239` / `-268` / `-691` / `-703` | Unique / FK / NOT NULL violation | `IntegrityError` |
|
||||
| `-329` | Database not found | `OperationalError` |
|
||||
| `-908` | Connection terminated by server | `OperationalError` |
|
||||
| `-1820` | Codeset conversion failure | `DataError` |
|
||||
| `-908` / network errors | Server unavailable | `OperationalError` |
|
||||
|
||||
The full mapping lives in `src/informix_db/_errcodes.py`.
|
||||
|
||||
## Inspecting errors
|
||||
|
||||
Every exception carries the original SQLCODE and message:
|
||||
|
||||
```python
|
||||
try:
|
||||
cur.execute("INSERT INTO users VALUES (?, ?)", (1, "alice"))
|
||||
except informix_db.IntegrityError as e:
|
||||
print(e.sqlcode) # -239 (unique violation)
|
||||
print(e.isam_code) # secondary error code
|
||||
print(str(e)) # human-readable
|
||||
```
|
||||
|
||||
`sqlcode` is the primary error; `isam_code` is the underlying ISAM error (when applicable). For multi-statement transactions, use `e.statement` to see which statement failed.
|
||||
121
docs-site/src/content/docs/reference/types.md
Normal file
121
docs-site/src/content/docs/reference/types.md
Normal file
@ -0,0 +1,121 @@
|
||||
---
|
||||
title: SQL ↔ Python types
|
||||
description: Mapping table between Informix SQL types and Python types, with notes on edge cases.
|
||||
sidebar:
|
||||
order: 2
|
||||
---
|
||||
|
||||
| SQL type | Python type | Notes |
|
||||
|---|---|---|
|
||||
| `SMALLINT` / `INT` / `SERIAL` | `int` | Arbitrary precision on the Python side. |
|
||||
| `BIGINT` / `BIGSERIAL` | `int` | 8-byte two's complement on the wire. |
|
||||
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer, a different wire format from `BIGINT` rather than an alias. See below. |
|
||||
| `FLOAT` / `SMALLFLOAT` | `float` | IEEE 754 double / single. |
|
||||
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | Exact precision preserved. |
|
||||
| `CHAR` / `NCHAR` | `str` | Fixed width, space-padded; trailing spaces stripped on decode. |
|
||||
| `VARCHAR` / `NVCHAR` / `LVARCHAR` | `str` | Variable length. Decoded using `client_locale`. |
|
||||
| `BOOLEAN` | `bool` | |
|
||||
| `DATE` | `datetime.date` | |
|
||||
| `DATETIME YEAR TO …` | `datetime.datetime` / `datetime.time` / `datetime.date` | The Python type depends on the field range. |
|
||||
| `INTERVAL DAY TO FRACTION` | `datetime.timedelta` | |
|
||||
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type, because `datetime.timedelta` can't represent year-month intervals. |
|
||||
| `BYTE` / `TEXT` (legacy in-row blobs) | `bytes` / `str` | |
|
||||
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators, 72 bytes. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
|
||||
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string, **not** decomposed into fields. See below. |
|
||||
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema, **not** iterable. See below. |
|
||||
| `NULL` | `None` | |
|
||||
|
||||
## INT8 and SERIAL8 are not BIGINT
|
||||
|
||||
Informix has two unrelated 64-bit integer types and they share nothing on the wire:
|
||||
|
||||
| | Wire format |
|
||||
|---|---|
|
||||
| `BIGINT` / `BIGSERIAL` (type 52 / 53) | 8 bytes, two's complement, big-endian |
|
||||
| `INT8` / `SERIAL8` (type 17 / 18) | 10 bytes, sign-magnitude, halves stored high-last |
|
||||
|
||||
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `−n` with *identical* magnitude bytes and the sign in a leading word. Decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
|
||||
|
||||
`INT8`/`SERIAL8` are common in schemas predating Informix 11.50, which is when `BIGINT` arrived.
|
||||
|
||||
:::caution[Upgrade if you use these types]
|
||||
`2026.08.27` fixed three decoding bugs: `INT8`/`SERIAL8` weren't decoded at all and returned raw `bytes`; `NCHAR` lost its first character; and `BOOLEAN` corrupted every column after it.
|
||||
|
||||
`2026.08.31` fixed two more in `LVARCHAR` framing: a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
|
||||
|
||||
`2026.09.02` fixed two more, both found by a new check that verifies each row consumed exactly its own payload. Smart LOBs were read as a flat 72-byte field when they actually occupy 149 bytes populated and 5 when NULL, so a `BLOB` or `CLOB` anywhere but the **last** column shifted every column after it. A NULL `SET` / `MULTISET` / `LIST` / `ROW` skipped its length field and did the same. That release also made a mismatch raise where it happens, naming the column and the byte delta, instead of handing back plausible-looking wrong values.
|
||||
|
||||
`2026.09.03` fixed one on the **write** path rather than the read path, which makes it the only entry here that affects data already stored. The `:N` placeholder rewriter was a regular expression and could not see a string literal, so it rewrote the inside of one: `'http://host:8080/x'` was stored as `'http://host?/x'`. Any `HH:MM` time, URL with a port, or `key:value` string was affected, whichever placeholder style you used. If you have written such values through this driver, they are worth checking.
|
||||
|
||||
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe. If you select a `BLOB` or `CLOB` in any position other than last, or a nullable collection column, nothing before `2026.09.02` is safe. Upgrade.
|
||||
:::
|
||||
|
||||
## LVARCHAR wire framing
|
||||
|
||||
Worth knowing if you're reading captures. Every Informix server we've tested describes `LVARCHAR` columns as UDTVAR (type 40, `extended_name='lvarchar'`), and the value arrives wrapped in a UDT envelope:
|
||||
|
||||
```
|
||||
[1-byte null indicator][4-byte length][content]
|
||||
```
|
||||
|
||||
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte, since both carry a length of zero.
|
||||
|
||||
## DATETIME field ranges
|
||||
|
||||
Informix's `DATETIME YEAR TO X` is field-range typed. The Python type returned depends on which fields are present:
|
||||
|
||||
| Range | Python type |
|
||||
|---|---|
|
||||
| `YEAR TO YEAR` through `YEAR TO DAY` | `datetime.date` |
|
||||
| `YEAR TO HOUR` through `YEAR TO FRACTION(5)` | `datetime.datetime` |
|
||||
| `HOUR TO HOUR` through `HOUR TO FRACTION(5)` | `datetime.time` |
|
||||
|
||||
## Decimal precision
|
||||
|
||||
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly, with no float intermediate.
|
||||
|
||||
For binding `Decimal` values into INSERT/UPDATE, the driver uses the column's declared scale. Pass `Decimal` for exact values; `float` works but may cause rounding at the column scale.
|
||||
|
||||
## NULL
|
||||
|
||||
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL, because `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
|
||||
|
||||
## Type extensions
|
||||
|
||||
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count**, mirroring the server's own representation, with `years` and `remainder_months` available as derived properties:
|
||||
|
||||
```python
|
||||
from informix_db import IntervalYM
|
||||
|
||||
ym = IntervalYM(27) # 2 years, 3 months
|
||||
ym.years # 2
|
||||
ym.remainder_months # 3
|
||||
str(ym) # '2-03'
|
||||
|
||||
cur.execute("INSERT INTO contracts (term) VALUES (?)", (ym,))
|
||||
cur.execute("SELECT term FROM contracts WHERE id = ?", (1,))
|
||||
result = cur.fetchone()[0] # IntervalYM(months=27)
|
||||
```
|
||||
|
||||
Negative intervals are supported; the sign lives on `months` and propagates to both derived properties.
|
||||
|
||||
## ROW and collection columns
|
||||
|
||||
`informix_db.RowValue` and `informix_db.CollectionValue` are returned for `ROW`, `SET`, `MULTISET`, and `LIST` columns. Both are **opaque wrappers, not decomposed values**:
|
||||
|
||||
```python
|
||||
row_val.raw # bytes: the server's textual representation, e.g. b"ROW('Alice',30)"
|
||||
row_val.schema # str: the column's declared schema
|
||||
|
||||
coll_val.raw # bytes, e.g. b'LIST{10,20,30}'
|
||||
coll_val.kind # 'set' | 'multiset' | 'list' | 'collection'
|
||||
coll_val.element_schema # str
|
||||
```
|
||||
|
||||
They are not iterable and do not expose fields by name. Fully parsing a composite type means reimplementing JDBC's `IfxComplexInput`, which we haven't done.
|
||||
|
||||
If you need individual fields today, project them in SQL. The server does the decomposition for you and you get ordinary typed columns back:
|
||||
|
||||
```python
|
||||
cur.execute("SELECT person.name, person.age FROM staff") # str, int
|
||||
```
|
||||
166
docs-site/src/content/docs/start/quickstart.mdx
Normal file
166
docs-site/src/content/docs/start/quickstart.mdx
Normal file
@ -0,0 +1,166 @@
|
||||
---
|
||||
title: Install & first query
|
||||
description: Five minutes from pip install to a real SELECT against a Dockerised Informix server.
|
||||
sidebar:
|
||||
order: 2
|
||||
---
|
||||
|
||||
import { Tabs, TabItem, Steps, Aside } from '@astrojs/starlight/components';
|
||||
|
||||
This page gets you from a clean Python environment to a working query against a real Informix server in five minutes. We'll use the official IBM Informix Developer Edition Docker image so you don't need an Informix license.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.10 or newer
|
||||
- Docker, for the dev server (skip if you already have an Informix instance)
|
||||
- 4 GB of free RAM for the dev container
|
||||
|
||||
## 1. Install the driver
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="uv (recommended)">
|
||||
```bash
|
||||
uv add informix-driver
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem label="pip">
|
||||
```bash
|
||||
pip install informix-driver
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem label="poetry">
|
||||
```bash
|
||||
poetry add informix-driver
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
That's the entire dependency. No system packages, no `LD_LIBRARY_PATH`, no `libcrypt.so.1`.
|
||||
|
||||
## 2. Start the dev container
|
||||
|
||||
```bash
|
||||
docker run -d --name informix-dev \
|
||||
-e LICENSE=accept \
|
||||
-p 9088:9088 \
|
||||
-p 9089:9089 \
|
||||
--privileged \
|
||||
icr.io/informix/informix-developer-database:15.0.1.0.3DE
|
||||
```
|
||||
|
||||
The image takes ~90 seconds to initialize. Watch the logs until you see `oninit running`:
|
||||
|
||||
```bash
|
||||
docker logs -f informix-dev
|
||||
```
|
||||
|
||||
<Aside type="tip">
|
||||
The `--privileged` flag is required by the dev image, which tries to manage shared memory limits. For production servers this isn't a thing.
|
||||
</Aside>
|
||||
|
||||
## 3. Run your first query
|
||||
|
||||
Save this as `hello.py`:
|
||||
|
||||
```python
|
||||
import informix_db
|
||||
|
||||
with informix_db.connect(
|
||||
host="127.0.0.1",
|
||||
port=9088,
|
||||
user="informix",
|
||||
password="in4mix",
|
||||
database="sysmaster",
|
||||
server="informix",
|
||||
) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"SELECT FIRST 5 dbsname, tabname "
|
||||
"FROM systables WHERE tabid > 99"
|
||||
)
|
||||
for row in cur.fetchall():
|
||||
print(row)
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
python hello.py
|
||||
```
|
||||
|
||||
You should see five rows from Informix's system catalog. If you do, congratulations: you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
|
||||
|
||||
## What just happened
|
||||
|
||||
When `informix_db.connect()` returned, the driver had:
|
||||
|
||||
1. Opened a TCP socket to `127.0.0.1:9088`
|
||||
2. Sent an `SQ_INFO` PDU containing client capabilities (locale, byte order, app name)
|
||||
3. Received the server's identification (`IBM Informix Dynamic Server Version 15.0.1.0.3`)
|
||||
4. Negotiated authentication via `SQ_PASSWD`
|
||||
5. Opened the `sysmaster` database via `SQ_DBOPEN`
|
||||
6. Returned a `Connection` object ready for queries
|
||||
|
||||
`cur.execute()` sent an `SQ_PREPARE` PDU with your SQL, parsed the response into a column descriptor, sent `SQ_DESCRIBE`, then `SQ_OPEN` to start the cursor. `cur.fetchall()` issued `SQ_FETCH` PDUs and decoded each `SQ_TUPLE` payload via per-column readers.
|
||||
|
||||
If you want to see this happen byte-by-byte, [the architecture page](/explain/architecture/) walks through the wire protocol with annotated captures.
|
||||
|
||||
## 4. Try parameter binding
|
||||
|
||||
```python
|
||||
with informix_db.connect(host="127.0.0.1", port=9088, user="informix",
|
||||
password="in4mix", database="sysmaster",
|
||||
server="informix") as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"SELECT tabname FROM systables WHERE tabid = ?",
|
||||
(1,),
|
||||
)
|
||||
print(cur.fetchone())
|
||||
```
|
||||
|
||||
`?` and `:1` both work. Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
|
||||
|
||||
Rows come back as plain tuples. If you would rather read them by column
|
||||
name, pass `row_factory=informix_db.Row` to `connect()` and you get
|
||||
`row["name"]` and `row.name` alongside `row[0]`. See
|
||||
[the API reference](/reference/api/#rows).
|
||||
|
||||
## 5. Use the connection pool
|
||||
|
||||
For real applications, prefer the pool:
|
||||
|
||||
```python
|
||||
import informix_db
|
||||
|
||||
pool = informix_db.create_pool(
|
||||
host="127.0.0.1", port=9088,
|
||||
user="informix", password="in4mix",
|
||||
database="sysmaster", server="informix",
|
||||
min_size=2, max_size=10,
|
||||
acquire_timeout=5.0,
|
||||
)
|
||||
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
|
||||
print(cur.fetchone())
|
||||
|
||||
pool.close()
|
||||
```
|
||||
|
||||
The pool is thread-safe and has a per-connection wire lock, so accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
|
||||
|
||||
## What's next
|
||||
|
||||
<Steps>
|
||||
|
||||
1. **Going async?** [Async with FastAPI →](/how-to/async-fastapi/) walks through a real FastAPI app with the async pool.
|
||||
|
||||
2. **Connecting to production?** [Connect with TLS →](/how-to/tls/) covers TLS-listener configuration and bring-your-own-context patterns.
|
||||
|
||||
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/), including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
|
||||
|
||||
4. **Migrating from IfxPy?** [Migrate from IfxPy →](/how-to/migrate-from-ifxpy/) covers the API differences and the things IfxPy does that we don't (yet).
|
||||
|
||||
</Steps>
|
||||
149
docs-site/src/content/docs/start/vs-ifxpy.mdx
Normal file
149
docs-site/src/content/docs/start/vs-ifxpy.mdx
Normal file
@ -0,0 +1,149 @@
|
||||
---
|
||||
title: Compared to IfxPy
|
||||
description: Head-to-head benchmarks against IBM's C-bound Python driver. Where each one wins and why.
|
||||
sidebar:
|
||||
order: 3
|
||||
---
|
||||
|
||||
import { Aside } from '@astrojs/starlight/components';
|
||||
|
||||
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver, a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
|
||||
|
||||
Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Informix Developer Edition Docker container on the same host. Methodology and reproduction steps live in [`tests/benchmarks/compare/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare) in the repo.
|
||||
|
||||
## Headline numbers
|
||||
|
||||
| Benchmark | IfxPy 3.0.5 (C) | informix-driver (pure Python) | Result |
|
||||
|---|---:|---:|---:|
|
||||
| Single-row SELECT round-trip | 118 µs | 114 µs | comparable |
|
||||
| ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster |
|
||||
| Cold connect (login handshake) | 11.0 ms | 10.5 ms | comparable |
|
||||
| `executemany(1k)` in transaction | 23.5 ms | 23.2 ms | tied |
|
||||
| **`executemany(10k)` in transaction** | 259 ms | **161 ms** | **informix-driver 1.6× faster** |
|
||||
| **`executemany(100k)` in transaction** | 2376 ms | **1487 ms** | **informix-driver 1.6× faster** |
|
||||
| `SELECT 1k` rows | 1.34 ms | 1.72 ms | IfxPy 1.28× faster |
|
||||
| `SELECT 10k` rows | 11.7 ms | 16.1 ms | IfxPy 1.07× faster |
|
||||
| `SELECT 100k` rows | 116 ms | 169 ms | IfxPy 1.15× faster |
|
||||
|
||||
<Aside type="note">
|
||||
Phase 39's connection-scoped buffered reader closed the bulk-fetch gap from a steady ~2.4× to ~1.05–1.15×. The story of how that landed is in [the buffered reader page](/explain/buffered-reader/).
|
||||
</Aside>
|
||||
|
||||
## When informix-driver wins
|
||||
|
||||
### Bulk inserts at scale
|
||||
|
||||
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms**, so `informix-driver` is 1.6× faster.
|
||||
|
||||
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row, and N RTTs at ~80 µs each adds up to the 100 ms gap.
|
||||
|
||||
```python
|
||||
# Both drivers
|
||||
cur.executemany(
|
||||
"INSERT INTO orders VALUES (?, ?, ?)",
|
||||
rows, # list of 10_000 tuples
|
||||
)
|
||||
|
||||
# informix-driver: 161 ms (10k PDUs sent, then 10k responses drained)
|
||||
# IfxPy: 259 ms (10k round-trips, each blocking on response)
|
||||
```
|
||||
|
||||
### Containerized deployment
|
||||
|
||||
`informix-driver` ships as a 50 KB pure-Python wheel with **zero runtime dependencies**. Your Dockerfile is:
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.13-slim
|
||||
RUN pip install informix-driver
|
||||
```
|
||||
|
||||
IfxPy's deployment surface is dramatically larger:
|
||||
|
||||
- 92 MB IBM OneDB Client tarball
|
||||
- `setuptools < 58` build pin
|
||||
- `LD_LIBRARY_PATH` configuration for four directories
|
||||
- `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9)
|
||||
- C compiler in the build image
|
||||
|
||||
For slim images, multi-stage builds, FaaS deployments, or anywhere build-toolchain-on-the-runtime is friction, `informix-driver` is the only reasonable option.
|
||||
|
||||
### Modern Python
|
||||
|
||||
IfxPy works on Python ≤ 3.11 currently. The C extension breaks on 3.12+ (PyConfig changes, removed `_PyImport_AcquireLock`, etc.).
|
||||
|
||||
`informix-driver` works unmodified on **3.10, 3.11, 3.12, 3.13, and 3.14**. We've kept a CI matrix on every minor version since 3.10 from the start.
|
||||
|
||||
### Async
|
||||
|
||||
`informix-driver` ships an async API:
|
||||
|
||||
```python
|
||||
from informix_db import aio
|
||||
|
||||
async def main():
|
||||
pool = await aio.create_pool(...)
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute("SELECT ...")
|
||||
rows = await cur.fetchall()
|
||||
```
|
||||
|
||||
IfxPy has no async support, so every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
|
||||
|
||||
## When IfxPy wins
|
||||
|
||||
### Large analytical fetches
|
||||
|
||||
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 5–15% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock, which is meaningful but not disqualifying.
|
||||
|
||||
The gap is closing phase by phase:
|
||||
|
||||
| Phase | Bulk-fetch ratio vs IfxPy |
|
||||
|---|---|
|
||||
| Phase 36 | 2.40× slower |
|
||||
| Phase 37 (per-column readers) | 2.10× slower |
|
||||
| Phase 38 (codegen-inlined decoders) | 2.04× slower |
|
||||
| **Phase 39 (connection-scoped buffered reader)** | **1.15× slower** |
|
||||
|
||||
If you're running analytical reports that pull millions of rows in a single SELECT and the per-row decode overhead is a measurable cost, IfxPy may be marginally faster today. For most application workloads it isn't.
|
||||
|
||||
### Workloads built around CSDK extensions
|
||||
|
||||
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API. See [the migration guide](/how-to/migrate-from-ifxpy/).
|
||||
|
||||
## Methodology
|
||||
|
||||
Benchmarks are pytest-benchmark fixtures in `tests/benchmarks/compare/` against the official `icr.io/informix/informix-developer-database:15.0.1.0.3DE` image, running on the same loopback as the Python process.
|
||||
|
||||
Reported numbers are **median over 10+ rounds**, with IQR included. Why median over mean: the first round of any run includes JIT warmup, page-cache miss, and a TCP slow-start round-trip. The mean is contaminated by these one-shot costs in a way that misrepresents steady-state behavior. Median + IQR is what we report.
|
||||
|
||||
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise. A fair reading is "5–15% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
|
||||
|
||||
To reproduce:
|
||||
|
||||
```bash
|
||||
git clone https://git.supported.systems/warehack.ing/informix-db
|
||||
cd informix-db/tests/benchmarks/compare
|
||||
make ifx-up
|
||||
make compare
|
||||
```
|
||||
|
||||
The `Makefile` handles the IfxPy install gauntlet (Python ≤ 3.11 environment, `setuptools < 58`, `libcrypt.so.1` symlink, OneDB CSDK download, the four `LD_LIBRARY_PATH` exports) so you don't have to learn it manually.
|
||||
|
||||
## Summary
|
||||
|
||||
Use `informix-driver` when:
|
||||
|
||||
- You're writing new code in Python ≥ 3.10
|
||||
- Your workload is bulk-insert / ETL / log-shipping
|
||||
- You want async / FastAPI integration
|
||||
- You're deploying in containers or to Python environments where build toolchains are friction
|
||||
- Your platform doesn't have `libcrypt.so.1`
|
||||
|
||||
Use IfxPy when:
|
||||
|
||||
- You have an existing IfxPy codebase
|
||||
- You're running large analytical SELECTs and the 5–15% decode-side gap matters
|
||||
- You're constrained to Python ≤ 3.11 anyway
|
||||
|
||||
For everything else, the cost-benefit favors `pip install informix-driver`.
|
||||
87
docs-site/src/content/docs/start/wtf.md
Normal file
87
docs-site/src/content/docs/start/wtf.md
Normal file
@ -0,0 +1,87 @@
|
||||
---
|
||||
title: WTF did you build this for?
|
||||
description: A pure-Python Informix driver, why it didn't exist before, and what it's good for.
|
||||
sidebar:
|
||||
order: 1
|
||||
label: WTF did you build this for?
|
||||
---
|
||||
|
||||
The existing tools were not my style.
|
||||
|
||||
Every Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That covers `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
|
||||
|
||||
## The problem with IBM's C SDK
|
||||
|
||||
The IBM Informix Client SDK (CSDK), now packaged as part of OneDB Client, is a 92 MB tarball with a non-trivial install gauntlet:
|
||||
|
||||
- Python ≤ 3.11 (IfxPy is broken on 3.12+)
|
||||
- `setuptools < 58` (legacy build system)
|
||||
- Permissive `CFLAGS` for the C extension build
|
||||
- Manual download of the 92 MB ODBC tarball
|
||||
- Four `LD_LIBRARY_PATH` directories
|
||||
- `libcrypt.so.1`, deprecated in 2018 and missing on Arch, Fedora 35+, RHEL 9
|
||||
|
||||
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db`; the distribution name dodges PyPI's 2008-vintage `informixdb` package, while the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
|
||||
|
||||
## What it does
|
||||
|
||||
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly. It is the same protocol IBM's JDBC driver uses, and the same one the CSDK speaks under the hood. No native code is in the thread of execution.
|
||||
|
||||
The wire protocol was reverse-engineered through three sources:
|
||||
|
||||
1. **Decompiled IBM JDBC driver** (`com.informix.jdbc.IfxConnection` and friends), used as a clean-room reference for PDU shapes and protocol semantics.
|
||||
2. **Annotated `socat` captures** of real client/server traffic against the IBM Informix Developer Edition Docker image.
|
||||
3. **Differential testing** against `IfxPy`, so every codec path is checked against the C driver's behavior on the same data.
|
||||
|
||||
The result is a PEP 249 compliant driver with a sync API, an async API (FastAPI / asyncio compatible), a connection pool, TLS support, smart-LOB read/write, scrollable cursors, fast-path stored procedure invocation, and bulk-insert / bulk-fetch performance within ~10–60% of the C driver depending on workload.
|
||||
|
||||
## What it's good for
|
||||
|
||||
The places where `informix-driver` is unambiguously the right choice:
|
||||
|
||||
- **ETL and bulk-load pipelines.** Pipelined `executemany` (Phase 33) is 1.6× faster than IfxPy at scale because every BIND+EXECUTE PDU goes out before any responses are drained. IfxPy still pays one round-trip per `IfxPy.execute(stmt, tuple)` call.
|
||||
- **Container deployments.** The 50 KB wheel and absent native deps mean a slim base image works. No multi-stage build to compile the CSDK.
|
||||
- **Modern Python.** Works on 3.10 through 3.14 unmodified. IfxPy hasn't shipped 3.12 wheels.
|
||||
- **Async / FastAPI.** Native async support via thread-pool wrapping. IfxPy is fully synchronous; using it from FastAPI requires `run_in_executor` boilerplate and gives up the connection pool's natural async semantics.
|
||||
- **Anywhere `libcrypt.so.1` is missing.** Modern Linux distributions ship `libcrypt.so.2`. IfxPy refuses to load without `libcrypt.so.1`. We don't link against either.
|
||||
|
||||
## What IfxPy is still better at
|
||||
|
||||
Honesty matters here:
|
||||
|
||||
- **Large analytical fetches.** IfxPy's C-level `fetch_tuple` decoder is faster than our Python `parse_tuple_payload` (~1.1 µs/row vs ~2.0 µs/row after Phase 39). For workloads pulling 10k+ rows in a single SELECT where the per-row decode cost dominates, IfxPy is currently 5–15% faster. The gap is shrinking phase by phase.
|
||||
- **Workloads built around the CSDK.** If your existing code already uses IfxPy idioms (`IfxPyDbi.connect_pooled`, IBM's specific cursor extensions), the migration to `informix-driver` is straightforward but not zero-cost.
|
||||
|
||||
The honest summary table from the [comparison page](/start/vs-ifxpy/):
|
||||
|
||||
| Workload | Winner | Margin |
|
||||
|---|---|---|
|
||||
| Bulk insert (`executemany` 10k–100k rows) | `informix-driver` | 1.6× faster |
|
||||
| Bulk SELECT (10k–100k rows) | IfxPy | 1.05–1.15× faster |
|
||||
| Single-row queries | tied | within noise |
|
||||
| Cold connect | tied | within noise |
|
||||
| Containerized deployment | `informix-driver` | no contest |
|
||||
| Python 3.12+ | `informix-driver` | only option |
|
||||
|
||||
## Production-ready
|
||||
|
||||
Every finding from a system-wide failure-mode audit (data correctness, wire safety, resource leaks, concurrency, async cancellation) has been addressed:
|
||||
|
||||
- Pool no longer returns connections with open transactions
|
||||
- Per-connection wire lock prevents PDU interleaving from accidental sharing
|
||||
- Async cancellation cannot leak running workers onto recycled connections
|
||||
- `_raise_sq_err` no longer masks wire desync via bare-except
|
||||
- Cursor finalizers release server-side resources on mid-fetch raise
|
||||
- 5 medium-severity hardening items resolved
|
||||
|
||||
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
|
||||
|
||||
470+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 472/472 on 15.0.1.0.3DE and 14.10.FC7W1DE, and 471/472 on 12.10.FC12W1DE (the single skip is a common table expression, which 12.10 predates). `make test-matrix` runs the lot.
|
||||
|
||||
That matrix exists because it turned out to be needed. A user reported corrupted result sets on Informix 12; the cause was three framing bugs that affected every version including the one we tested against, and they'd survived because no fixture used the affected types. Testing one server and inferring the rest is how that happens.
|
||||
|
||||
## Read next
|
||||
|
||||
- **[Install & first query →](/start/quickstart/)**: five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
|
||||
- **[Compared to IfxPy →](/start/vs-ifxpy/)**: full head-to-head benchmarks, methodology, and reproduction.
|
||||
- **[Architecture →](/explain/architecture/)**: how the layers stack, from socket through framing, codec, resultset, and cursor.
|
||||
428
docs-site/src/styles/components.css
Normal file
428
docs-site/src/styles/components.css
Normal file
@ -0,0 +1,428 @@
|
||||
/* Component-scoped styles for the custom Hero and homepage modules */
|
||||
|
||||
.ifx-hero {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1fr) minmax(0, 1.1fr);
|
||||
gap: clamp(1.5rem, 4vw, 4rem);
|
||||
align-items: center;
|
||||
padding: clamp(2rem, 5vw, 4.5rem) 0 clamp(2rem, 4vw, 3.5rem);
|
||||
border-bottom: 1px solid var(--sl-color-hairline-light);
|
||||
}
|
||||
|
||||
@media (max-width: 800px) {
|
||||
.ifx-hero {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
}
|
||||
|
||||
.ifx-hero__eyebrow {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
font-family: var(--sl-font-mono);
|
||||
font-size: 0.75rem;
|
||||
letter-spacing: 0.18em;
|
||||
text-transform: uppercase;
|
||||
color: var(--sl-color-text-accent);
|
||||
margin-bottom: 0.75rem;
|
||||
}
|
||||
|
||||
.ifx-hero__eyebrow::before {
|
||||
content: '';
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
background: var(--ifx-amber);
|
||||
border-radius: 1px;
|
||||
box-shadow: 0 0 12px var(--ifx-amber);
|
||||
animation: ifx-pulse 2.4s ease-in-out infinite;
|
||||
}
|
||||
|
||||
@keyframes ifx-pulse {
|
||||
0%, 100% { opacity: 0.4; }
|
||||
50% { opacity: 1; }
|
||||
}
|
||||
|
||||
.ifx-hero__title {
|
||||
font-size: clamp(2rem, 5vw, 3.25rem);
|
||||
line-height: 1.05;
|
||||
letter-spacing: -0.02em;
|
||||
font-weight: 700;
|
||||
margin: 0 0 1rem;
|
||||
color: var(--sl-color-white);
|
||||
}
|
||||
|
||||
.ifx-hero__title strong {
|
||||
background: linear-gradient(180deg, var(--ifx-amber-bright) 0%, var(--ifx-amber) 100%);
|
||||
-webkit-background-clip: text;
|
||||
background-clip: text;
|
||||
color: transparent;
|
||||
font-weight: inherit;
|
||||
}
|
||||
|
||||
.ifx-hero__lede {
|
||||
font-size: clamp(1rem, 1.5vw, 1.15rem);
|
||||
line-height: 1.55;
|
||||
color: var(--sl-color-gray-2);
|
||||
max-width: 36rem;
|
||||
margin: 0 0 1.75rem;
|
||||
}
|
||||
|
||||
.ifx-hero__cta {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 0.75rem;
|
||||
}
|
||||
|
||||
.ifx-hero__cta a {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.4rem;
|
||||
padding: 0.6rem 1.1rem;
|
||||
border-radius: 6px;
|
||||
font-weight: 500;
|
||||
text-decoration: none !important;
|
||||
transition: transform 0.08s ease, box-shadow 0.15s ease;
|
||||
}
|
||||
.ifx-hero__cta a:hover {
|
||||
transform: translateY(-1px);
|
||||
}
|
||||
|
||||
.ifx-hero__cta .primary {
|
||||
background: var(--ifx-amber);
|
||||
color: var(--ifx-charcoal-0);
|
||||
}
|
||||
.ifx-hero__cta .primary:hover {
|
||||
box-shadow: 0 4px 16px rgba(245, 165, 36, 0.35);
|
||||
}
|
||||
|
||||
.ifx-hero__cta .secondary {
|
||||
border: 1px solid var(--sl-color-hairline-shade);
|
||||
color: var(--sl-color-text);
|
||||
background: transparent;
|
||||
}
|
||||
.ifx-hero__cta .secondary:hover {
|
||||
border-color: var(--sl-color-accent);
|
||||
}
|
||||
|
||||
.ifx-hero__install {
|
||||
font-family: var(--sl-font-mono);
|
||||
font-size: 0.95rem;
|
||||
margin-top: 1.5rem;
|
||||
padding: 0.75rem 1rem;
|
||||
background: var(--sl-color-bg-inline-code);
|
||||
border-left: 2px solid var(--ifx-amber);
|
||||
border-radius: 0 4px 4px 0;
|
||||
color: var(--sl-color-text);
|
||||
user-select: all;
|
||||
}
|
||||
|
||||
.ifx-hero__install::before {
|
||||
content: '$ ';
|
||||
color: var(--sl-color-text-accent);
|
||||
user-select: none;
|
||||
}
|
||||
|
||||
/* Wire-dump easter egg: types out real captured handshake bytes */
|
||||
.ifx-wiredump {
|
||||
font-family: var(--sl-font-mono);
|
||||
font-size: 0.78rem;
|
||||
line-height: 1.55;
|
||||
background: var(--ifx-charcoal-0);
|
||||
color: var(--ifx-amber);
|
||||
border: 1px solid var(--sl-color-hairline);
|
||||
border-radius: 8px;
|
||||
padding: 1.25rem 1rem;
|
||||
height: clamp(280px, 35vw, 360px);
|
||||
/* x: auto allows hex content to scroll horizontally inside the dump
|
||||
when the column is too narrow (tablet portrait, narrow desktop with
|
||||
2-col hero); y: hidden keeps the dump's vertical bounds fixed so
|
||||
unrevealed animation lines stay clipped below the fold */
|
||||
overflow-x: auto;
|
||||
overflow-y: hidden;
|
||||
position: relative;
|
||||
box-shadow: inset 0 0 60px rgba(245, 165, 36, 0.04);
|
||||
}
|
||||
|
||||
:root[data-theme='light'] .ifx-wiredump {
|
||||
background: #1a1612;
|
||||
}
|
||||
|
||||
.ifx-wiredump__scroll {
|
||||
white-space: pre;
|
||||
height: 100%;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.ifx-wiredump__line {
|
||||
display: block;
|
||||
opacity: 0;
|
||||
white-space: pre;
|
||||
}
|
||||
.ifx-wiredump__line.is-visible {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
.ifx-wiredump__byte--ascii {
|
||||
color: #ffd884;
|
||||
}
|
||||
|
||||
.ifx-wiredump__caret {
|
||||
display: inline-block;
|
||||
width: 8px;
|
||||
height: 1em;
|
||||
background: var(--ifx-amber);
|
||||
vertical-align: text-bottom;
|
||||
margin-left: 2px;
|
||||
animation: ifx-blink 1.1s steps(2) infinite;
|
||||
}
|
||||
|
||||
@keyframes ifx-blink {
|
||||
0%, 50% { opacity: 1; }
|
||||
51%, 100% { opacity: 0; }
|
||||
}
|
||||
|
||||
.ifx-wiredump__direction {
|
||||
color: var(--sl-color-gray-3);
|
||||
margin-right: 0.5rem;
|
||||
}
|
||||
|
||||
.ifx-wiredump__direction--out {
|
||||
color: var(--ifx-amber);
|
||||
}
|
||||
.ifx-wiredump__direction--in {
|
||||
color: #6dd2a4;
|
||||
}
|
||||
|
||||
.ifx-wiredump__caption {
|
||||
font-family: var(--sl-font-mono);
|
||||
font-size: 0.7rem;
|
||||
letter-spacing: 0.1em;
|
||||
text-transform: uppercase;
|
||||
color: var(--sl-color-gray-3);
|
||||
margin-top: 0.6rem;
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
/* Homepage feature grid */
|
||||
.ifx-features {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
|
||||
gap: 1.25rem;
|
||||
margin: 3rem 0;
|
||||
}
|
||||
|
||||
.ifx-feature {
|
||||
padding: 1.5rem;
|
||||
border: 1px solid var(--sl-color-hairline-light);
|
||||
border-radius: 8px;
|
||||
background: var(--sl-color-bg-nav);
|
||||
transition: border-color 0.15s ease, transform 0.1s ease;
|
||||
}
|
||||
|
||||
.ifx-feature:hover {
|
||||
border-color: var(--sl-color-accent);
|
||||
}
|
||||
|
||||
.ifx-feature__icon {
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
margin-bottom: 0.75rem;
|
||||
color: var(--sl-color-text-accent);
|
||||
}
|
||||
|
||||
.ifx-feature h3 {
|
||||
font-size: 1.05rem;
|
||||
margin: 0 0 0.5rem;
|
||||
letter-spacing: -0.01em;
|
||||
}
|
||||
|
||||
.ifx-feature p {
|
||||
font-size: 0.92rem;
|
||||
line-height: 1.5;
|
||||
margin: 0;
|
||||
color: var(--sl-color-gray-2);
|
||||
}
|
||||
|
||||
/* Supported Systems "joint" badge, appears below every page's footer */
|
||||
.ifx-ss-badge {
|
||||
margin-top: 3.5rem;
|
||||
padding: 0;
|
||||
border-top: 1px solid var(--sl-color-hairline-light);
|
||||
}
|
||||
|
||||
.ifx-ss-badge__link {
|
||||
display: grid;
|
||||
grid-template-columns: auto minmax(0, 1fr);
|
||||
gap: 1.25rem;
|
||||
align-items: center;
|
||||
padding: 1.5rem 0 0.5rem;
|
||||
text-decoration: none !important;
|
||||
color: var(--sl-color-text);
|
||||
transition: color 0.12s ease;
|
||||
}
|
||||
|
||||
.ifx-ss-badge__link:hover {
|
||||
color: var(--sl-color-text-accent);
|
||||
}
|
||||
|
||||
.ifx-ss-badge__link:hover .ifx-ss-badge__cta {
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.ifx-ss-badge__logo {
|
||||
width: 60px;
|
||||
height: 45px;
|
||||
flex-shrink: 0;
|
||||
/* Slight lift so the punch-card pattern reads */
|
||||
filter: drop-shadow(0 1px 2px rgba(37, 99, 235, 0.12));
|
||||
}
|
||||
|
||||
.ifx-ss-badge__copy {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.ifx-ss-badge__heading {
|
||||
font-family: var(--sl-font-mono);
|
||||
font-size: 0.78rem;
|
||||
font-weight: 600;
|
||||
letter-spacing: 0.16em;
|
||||
text-transform: uppercase;
|
||||
color: var(--sl-color-text-accent);
|
||||
margin: 0 0 0.4rem;
|
||||
border: none !important;
|
||||
padding: 0 !important;
|
||||
}
|
||||
|
||||
.ifx-ss-badge__body {
|
||||
font-size: 0.92rem;
|
||||
line-height: 1.55;
|
||||
color: var(--sl-color-gray-2);
|
||||
margin: 0 0 0.5rem;
|
||||
}
|
||||
|
||||
.ifx-ss-badge__name {
|
||||
color: #3b82f6;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
:root[data-theme='light'] .ifx-ss-badge__name {
|
||||
color: #2563eb;
|
||||
}
|
||||
|
||||
.ifx-ss-badge__cta {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.35rem;
|
||||
font-size: 0.85rem;
|
||||
font-weight: 500;
|
||||
color: var(--sl-color-text-accent);
|
||||
transition: gap 0.15s ease;
|
||||
}
|
||||
|
||||
@media (max-width: 540px) {
|
||||
.ifx-ss-badge__link {
|
||||
grid-template-columns: 1fr;
|
||||
gap: 0.85rem;
|
||||
text-align: left;
|
||||
}
|
||||
.ifx-ss-badge__logo {
|
||||
width: 50px;
|
||||
height: 38px;
|
||||
}
|
||||
}
|
||||
|
||||
/* ============================================================
|
||||
* Mobile (≤640px): tighten the hero, shrink the wire-dump,
|
||||
* hide the ASCII column, prevent runaway horizontal overflow
|
||||
* ============================================================ */
|
||||
|
||||
@media (max-width: 640px) {
|
||||
/* Defensive guard against any descendant forcing horizontal scroll:
|
||||
the wire-dump's white-space: pre content was overflowing the
|
||||
hero column and propagating up to the page. overflow-x: hidden
|
||||
on .ifx-hero contains it without affecting page-level scroll. */
|
||||
.ifx-hero,
|
||||
.ifx-hero__copy,
|
||||
.ifx-hero__visual {
|
||||
min-width: 0; /* allow flex/grid item to shrink below content min-width */
|
||||
}
|
||||
|
||||
.ifx-hero {
|
||||
padding: 1.25rem 0 1.75rem;
|
||||
gap: 1.25rem;
|
||||
}
|
||||
|
||||
.ifx-hero__eyebrow {
|
||||
flex-wrap: wrap;
|
||||
row-gap: 0.25rem;
|
||||
font-size: 0.7rem;
|
||||
letter-spacing: 0.12em;
|
||||
}
|
||||
|
||||
.ifx-hero__title {
|
||||
font-size: clamp(1.75rem, 7.5vw, 2.5rem);
|
||||
line-height: 1.1;
|
||||
}
|
||||
|
||||
.ifx-hero__lede {
|
||||
font-size: 0.95rem;
|
||||
line-height: 1.55;
|
||||
}
|
||||
|
||||
.ifx-hero__cta {
|
||||
gap: 0.5rem;
|
||||
}
|
||||
.ifx-hero__cta a {
|
||||
padding: 0.55rem 0.9rem;
|
||||
font-size: 0.92rem;
|
||||
}
|
||||
|
||||
.ifx-hero__install {
|
||||
font-size: 0.85rem;
|
||||
word-break: break-all; /* survive any future longer install command */
|
||||
}
|
||||
|
||||
/* The wire-dump is the biggest mobile risk: pre-formatted hex lines
|
||||
are ~50 chars wide. Two things together make it readable on
|
||||
narrow viewports: smaller font and ASCII column hidden. The
|
||||
overflow-x: auto fallback already lives on the base rule. */
|
||||
.ifx-wiredump {
|
||||
font-size: 0.62rem;
|
||||
height: clamp(220px, 60vw, 280px);
|
||||
padding: 0.85rem 0.65rem;
|
||||
}
|
||||
.ifx-wiredump__byte--ascii {
|
||||
display: none;
|
||||
}
|
||||
.ifx-wiredump__caption {
|
||||
font-size: 0.62rem;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.ifx-features {
|
||||
gap: 0.9rem;
|
||||
margin: 2rem 0;
|
||||
}
|
||||
.ifx-feature {
|
||||
padding: 1.1rem;
|
||||
}
|
||||
.ifx-feature h3 {
|
||||
font-size: 1rem;
|
||||
}
|
||||
}
|
||||
|
||||
/* Performance-bar visual for the comparison table */
|
||||
.ifx-perfbar {
|
||||
position: relative;
|
||||
height: 6px;
|
||||
background: var(--sl-color-bg-inline-code);
|
||||
border-radius: 3px;
|
||||
overflow: hidden;
|
||||
margin-top: 0.25rem;
|
||||
}
|
||||
.ifx-perfbar__fill {
|
||||
position: absolute;
|
||||
inset: 0 auto 0 0;
|
||||
background: linear-gradient(90deg, var(--ifx-amber) 0%, var(--ifx-amber-bright) 100%);
|
||||
border-radius: 3px;
|
||||
}
|
||||
137
docs-site/src/styles/theme.css
Normal file
137
docs-site/src/styles/theme.css
Normal file
@ -0,0 +1,137 @@
|
||||
/*
|
||||
* informix-driver docs theme
|
||||
* - Charcoal base (no purple gradients, ever)
|
||||
* - Amber accent, a CRT-monitor nod distinct from sibling sites' cyan
|
||||
* - Inter for body, IBM Plex Mono for technical bytes
|
||||
*/
|
||||
|
||||
:root {
|
||||
--ifx-amber: #f5a524;
|
||||
--ifx-amber-bright: #ffb84d;
|
||||
--ifx-amber-dim: #b87a18;
|
||||
--ifx-charcoal-0: #0e0d0c;
|
||||
--ifx-charcoal-1: #161412;
|
||||
--ifx-charcoal-2: #1f1c1a;
|
||||
--ifx-charcoal-3: #2a2622;
|
||||
--ifx-paper: #faf7f2;
|
||||
--ifx-paper-2: #f3eee5;
|
||||
--ifx-ink: #1a1612;
|
||||
--ifx-ink-soft: #4a443c;
|
||||
--ifx-rule: rgba(245, 165, 36, 0.18);
|
||||
|
||||
--sl-font: 'Inter', system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
--sl-font-mono: 'IBM Plex Mono', ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
|
||||
}
|
||||
|
||||
:root[data-theme='dark'] {
|
||||
--sl-color-accent-low: #3a2810;
|
||||
--sl-color-accent: var(--ifx-amber);
|
||||
--sl-color-accent-high: var(--ifx-amber-bright);
|
||||
|
||||
--sl-color-white: #fbf7ee;
|
||||
--sl-color-gray-1: #ece6da;
|
||||
--sl-color-gray-2: #c8c1b1;
|
||||
--sl-color-gray-3: #948b78;
|
||||
--sl-color-gray-4: #5a5246;
|
||||
--sl-color-gray-5: #2e2823;
|
||||
--sl-color-gray-6: var(--ifx-charcoal-2);
|
||||
--sl-color-black: var(--ifx-charcoal-0);
|
||||
|
||||
--sl-color-bg: var(--ifx-charcoal-0);
|
||||
--sl-color-bg-nav: var(--ifx-charcoal-1);
|
||||
--sl-color-bg-sidebar: var(--ifx-charcoal-1);
|
||||
--sl-color-bg-inline-code: var(--ifx-charcoal-2);
|
||||
--sl-color-bg-accent: var(--ifx-amber);
|
||||
|
||||
--sl-color-text: #ece6da;
|
||||
--sl-color-text-accent: var(--ifx-amber-bright);
|
||||
--sl-color-text-invert: var(--ifx-charcoal-0);
|
||||
|
||||
--sl-color-hairline: rgba(245, 165, 36, 0.14);
|
||||
--sl-color-hairline-light: rgba(245, 165, 36, 0.08);
|
||||
--sl-color-hairline-shade: rgba(245, 165, 36, 0.22);
|
||||
}
|
||||
|
||||
:root[data-theme='light'] {
|
||||
--sl-color-accent-low: #fbe8c8;
|
||||
--sl-color-accent: var(--ifx-amber-dim);
|
||||
--sl-color-accent-high: #7a4f0a;
|
||||
|
||||
--sl-color-white: #1a1612;
|
||||
--sl-color-gray-1: #2e2823;
|
||||
--sl-color-gray-2: #4a443c;
|
||||
--sl-color-gray-3: #6e665a;
|
||||
--sl-color-gray-4: #948b78;
|
||||
--sl-color-gray-5: #d8d1c1;
|
||||
--sl-color-gray-6: #ebe5d6;
|
||||
--sl-color-gray-7: #f3eee5;
|
||||
--sl-color-black: var(--ifx-paper);
|
||||
|
||||
--sl-color-bg: var(--ifx-paper);
|
||||
--sl-color-bg-nav: var(--ifx-paper-2);
|
||||
--sl-color-bg-sidebar: var(--ifx-paper-2);
|
||||
--sl-color-bg-inline-code: #ece6d4;
|
||||
--sl-color-bg-accent: var(--ifx-amber-dim);
|
||||
|
||||
--sl-color-text: var(--ifx-ink);
|
||||
--sl-color-text-accent: var(--ifx-amber-dim);
|
||||
--sl-color-text-invert: var(--ifx-paper);
|
||||
|
||||
--sl-color-hairline: rgba(120, 80, 12, 0.18);
|
||||
--sl-color-hairline-light: rgba(120, 80, 12, 0.10);
|
||||
--sl-color-hairline-shade: rgba(120, 80, 12, 0.28);
|
||||
}
|
||||
|
||||
/* Tighten heading rhythm; Starlight defaults are a touch loose for technical docs */
|
||||
.sl-markdown-content h2 {
|
||||
margin-top: 2.5rem;
|
||||
border-top: 1px solid var(--sl-color-hairline-light);
|
||||
padding-top: 1.5rem;
|
||||
letter-spacing: -0.01em;
|
||||
}
|
||||
|
||||
.sl-markdown-content h3 {
|
||||
margin-top: 1.75rem;
|
||||
letter-spacing: -0.005em;
|
||||
}
|
||||
|
||||
/* Inline code: monospace + amber tint, no jarring background block */
|
||||
.sl-markdown-content :not(pre) > code {
|
||||
font-feature-settings: 'ss02', 'cv02';
|
||||
border: 1px solid var(--sl-color-hairline-shade);
|
||||
font-size: 0.9em;
|
||||
padding: 0.1em 0.35em;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
/* Tables: dense, technical, amber column rules, for type-mapping & benchmark tables */
|
||||
.sl-markdown-content table {
|
||||
border-collapse: collapse;
|
||||
font-variant-numeric: tabular-nums;
|
||||
font-size: 0.92rem;
|
||||
}
|
||||
.sl-markdown-content thead th {
|
||||
border-bottom: 2px solid var(--sl-color-accent);
|
||||
text-align: left;
|
||||
font-weight: 600;
|
||||
letter-spacing: 0.02em;
|
||||
text-transform: uppercase;
|
||||
font-size: 0.78em;
|
||||
color: var(--sl-color-text-accent);
|
||||
}
|
||||
.sl-markdown-content tbody td {
|
||||
border-bottom: 1px solid var(--sl-color-hairline-light);
|
||||
padding: 0.5rem 0.75rem;
|
||||
}
|
||||
|
||||
/* Anchor links: underline, no rainbow */
|
||||
.sl-markdown-content a:not(.sl-anchor-link) {
|
||||
text-decoration: underline;
|
||||
text-decoration-color: var(--sl-color-accent);
|
||||
text-decoration-thickness: 1px;
|
||||
text-underline-offset: 3px;
|
||||
transition: text-decoration-thickness 0.1s ease;
|
||||
}
|
||||
.sl-markdown-content a:not(.sl-anchor-link):hover {
|
||||
text-decoration-thickness: 2px;
|
||||
}
|
||||
5
docs-site/tsconfig.json
Normal file
5
docs-site/tsconfig.json
Normal file
@ -0,0 +1,5 @@
|
||||
{
|
||||
"extends": "astro/tsconfigs/strict",
|
||||
"include": [".astro/types.d.ts", "**/*"],
|
||||
"exclude": ["dist"]
|
||||
}
|
||||
@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "informix-db"
|
||||
version = "2026.05.05.9"
|
||||
name = "informix-driver"
|
||||
version = "2026.09.03"
|
||||
description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries."
|
||||
readme = "README.md"
|
||||
license = { text = "MIT" }
|
||||
@ -27,9 +27,11 @@ classifiers = [
|
||||
dependencies = []
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/rsp2k/informix-db"
|
||||
Documentation = "https://github.com/rsp2k/informix-db/tree/main/docs"
|
||||
Issues = "https://github.com/rsp2k/informix-db/issues"
|
||||
Homepage = "https://informix-driver.warehack.ing"
|
||||
Documentation = "https://informix-driver.warehack.ing"
|
||||
Source = "https://git.supported.systems/warehack.ing/informix-db"
|
||||
Issues = "https://git.supported.systems/warehack.ing/informix-db/issues"
|
||||
Changelog = "https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md"
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
@ -49,13 +51,15 @@ packages = ["src/informix_db"]
|
||||
# (the wheel doesn't ship these by default, but the sdist would).
|
||||
# See ~/.claude/rules/python.md for the full pre-publish PII audit playbook.
|
||||
exclude = [
|
||||
"CLAUDE.md", # operator-private context
|
||||
"CLAUDE.md", # operator-private context
|
||||
".env", ".env.local", ".env.*",
|
||||
".mcp.json", # may contain local filesystem paths
|
||||
"build/", # decompiled JDBC, downloaded JARs
|
||||
".mcp.json", # may contain local filesystem paths
|
||||
"build/", # decompiled JDBC, downloaded JARs
|
||||
"audits/",
|
||||
"docs/CAPTURES/", # spike artifacts; tests can re-capture against the dev container
|
||||
"tests/reference/", # Java reference client — spike infra
|
||||
"docs/**", # protocol notes / decision log / captures — go to GitHub for the depth
|
||||
"docs-site/**", # Astro/Starlight site sources — published at informix-driver.warehack.ing
|
||||
"tests/reference/**", # Java reference client — spike infra
|
||||
"tests/benchmarks/.results/**", # pytest-benchmark cache (gitignored, but still ships unless excluded)
|
||||
".pytest_cache/", ".ruff_cache/", ".mypy_cache/",
|
||||
"dist/", "*.egg-info/",
|
||||
]
|
||||
|
||||
@ -23,6 +23,7 @@ from __future__ import annotations
|
||||
import ssl
|
||||
from importlib.metadata import PackageNotFoundError, version
|
||||
|
||||
from ._capabilities import ServerCapabilities
|
||||
from .connections import Connection
|
||||
from .converters import (
|
||||
BlobLocator,
|
||||
@ -49,6 +50,7 @@ from .pool import (
|
||||
PoolTimeoutError,
|
||||
create_pool,
|
||||
)
|
||||
from .rows import Row
|
||||
|
||||
# PEP 249 module-level globals
|
||||
apilevel = "2.0"
|
||||
@ -56,9 +58,14 @@ threadsafety = 1 # threads may share the module but not connections
|
||||
paramstyle = "numeric" # locked in DECISION_LOG.md — matches Informix ESQL/C
|
||||
|
||||
try:
|
||||
__version__ = version("informix-db")
|
||||
# NOTE: the *distribution* is ``informix-driver``; the *module* is
|
||||
# ``informix_db``. They differ, so this string can't be derived from
|
||||
# ``__name__``. This looked up the old ``informix-db`` distribution
|
||||
# until 2026.08.27, which meant __version__ silently reported
|
||||
# "0.0.0+local" for anyone who installed the renamed package.
|
||||
__version__ = version("informix-driver")
|
||||
except PackageNotFoundError:
|
||||
# Editable install or running uninstalled; fall back to a sentinel.
|
||||
# Running from a source checkout without an install.
|
||||
__version__ = "0.0.0+local"
|
||||
|
||||
__all__ = [
|
||||
@ -79,7 +86,9 @@ __all__ = [
|
||||
"PoolClosedError",
|
||||
"PoolTimeoutError",
|
||||
"ProgrammingError",
|
||||
"Row",
|
||||
"RowValue",
|
||||
"ServerCapabilities",
|
||||
"Warning",
|
||||
"__version__",
|
||||
"apilevel",
|
||||
@ -104,6 +113,7 @@ def connect(
|
||||
client_locale: str = "en_US.8859-1",
|
||||
env: dict[str, str] | None = None,
|
||||
autocommit: bool = False,
|
||||
row_factory: object | None = None,
|
||||
tls: bool | ssl.SSLContext = False,
|
||||
tls_server_hostname: str | None = None,
|
||||
) -> Connection:
|
||||
@ -146,4 +156,5 @@ def connect(
|
||||
client_locale=client_locale,
|
||||
env=env,
|
||||
autocommit=autocommit,
|
||||
row_factory=row_factory,
|
||||
)
|
||||
|
||||
269
src/informix_db/_capabilities.py
Normal file
269
src/informix_db/_capabilities.py
Normal file
@ -0,0 +1,269 @@
|
||||
"""Server capability decoding — the SQ_PROTOCOLS feature bitmap.
|
||||
|
||||
SQLI does not have a single "protocol version". It has a 64-bit feature
|
||||
bitmap that client and server negotiate at connect time, and several bits
|
||||
in it change **wire framing**, not just feature availability. Get one of
|
||||
those wrong and the row decoder desyncs.
|
||||
|
||||
We already perform the negotiation (``Connection._init_session`` sends
|
||||
``SQ_PROTOCOLS`` with the same 8-byte client offer the IBM JDBC driver
|
||||
uses). Until now we discarded the server's reply and hardcoded the modern
|
||||
framing. This module decodes the reply so the assumptions can at least be
|
||||
*checked* instead of merely believed.
|
||||
|
||||
Two independent sources feed the bitmap, mirroring
|
||||
``IfxSqliConnect.getServerVer`` / ``enhancedProtocolMechanism``:
|
||||
|
||||
1. ``Cap_1`` from the login (SLTYPE_CONACC) response. This is the client's
|
||||
own protocol level echoed back — 316 means "speaks the enhanced
|
||||
protocol". It is **not** a server version number, which is why JDBC
|
||||
tests ``== 316`` rather than ``>=``.
|
||||
2. The 8-byte mask from the SQ_PROTOCOLS reply.
|
||||
|
||||
When ``Cap_1`` is non-zero, JDBC unconditionally pre-sets bits
|
||||
{0, 2, 3, 4, 49, 51} *before* applying the mask, and a Java ``BitSet.set``
|
||||
only ever sets — never clears. So those six bits are true on any modern
|
||||
server regardless of what the mask says. ``isUSVER`` (bit 2) is one of
|
||||
them, which is why it never varies in practice and is a dead end when
|
||||
chasing version-specific behaviour.
|
||||
|
||||
Bit numbering is MSB-first within each byte: byte 0 bit 0x80 is bit 0,
|
||||
byte 0 bit 0x01 is bit 7, byte 1 bit 0x80 is bit 8, and so on.
|
||||
|
||||
A note on length. The client offer is 8 bytes, but every server we have
|
||||
tested replies with **nine**. JDBC's ``enhancedProtocolMechanism``
|
||||
dispatches on ``switch (i) case 0..7`` and silently discards anything
|
||||
past byte 7, so its ``BitSet(64)`` never sees the ninth byte. We decode
|
||||
it — bits 64-71 — because it turns out to be the only part of the mask
|
||||
that differs between releases::
|
||||
|
||||
Informix 15.0.1.0.3 bdbe9ffe7fb7ffef ff
|
||||
Informix 14.10.FC7W1 bdbe9ffe7fb7ffef f8
|
||||
Informix 12.10.FC12W1DE bdbe9ffe7fb7ffef f0
|
||||
^^^^^^^^^^^^^^^^ identical
|
||||
|
||||
The first 64 bits being byte-identical across those three releases is
|
||||
why they speak an indistinguishable SQLI dialect. Bits 64+ have no known
|
||||
meaning here; they are surfaced for diagnostics, not branched on.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
# The client's capability offer. Byte-for-byte the value IBM's JDBC driver
|
||||
# sends (``IfxSqliConnect.clientProtocols``), replayed verbatim — the bits
|
||||
# are a fixed constant there too, never computed.
|
||||
CLIENT_PROTOCOLS_MASK = bytes.fromhex("fffc7ffc3c8caa97")
|
||||
|
||||
# ``Cap_1`` value meaning "enhanced protocol negotiation applies". JDBC
|
||||
# compares with equality, not >=, because the server echoes the client's
|
||||
# own declared level rather than reporting its own.
|
||||
ENHANCED_PROTOCOL_CAP = 316
|
||||
|
||||
# Bits JDBC pre-sets for any non-zero Cap_1, before the mask is applied.
|
||||
_PRESET_BITS = frozenset({0, 2, 3, 4, 49, 51})
|
||||
|
||||
# Named bits. Only the ones we can justify from the decompiled JDBC are
|
||||
# listed; the mask has 64 and most are irrelevant to us.
|
||||
BIT_USVER = 2 # isUSVER — always true on modern servers
|
||||
BIT_LONG_ID_A = 10 # isLongID is bit 10 OR bit 18
|
||||
BIT_LONG_ID_B = 18
|
||||
BIT_VARCHAR_VAR_LEN = 13 # gated further by bit 40 + client ifxPADVARCHAR
|
||||
BIT_DESCRIBE_INPUT = 34
|
||||
BIT_LVARCHAR_GT_2K = 37
|
||||
BIT_PAD_VARCHAR_GATE = 40
|
||||
BIT_FP_DESCRIBE = 45
|
||||
BIT_AUTO_GENERATED_KEYS = 48
|
||||
BIT_CUR_SESS_INFO = 49
|
||||
BIT_FOUR_BYTE_OFFSET = 50 # describe: string-table + field-index widths
|
||||
BIT_GLS = 51
|
||||
BIT_NAMED_PARAMETERS = 52
|
||||
BIT_BIGINT = 54
|
||||
BIT_SAVEPOINT = 56
|
||||
BIT_SQ_BATCH = 61
|
||||
BIT_REMOVE_64K_LIMIT = 62 # 4-byte vs 2-byte length prefixes
|
||||
BIT_TWO_GB_FETCH_BUFFER = 63
|
||||
|
||||
|
||||
def _decode_bits(mask: bytes) -> set[int]:
|
||||
"""Expand a big-endian, MSB-first bit mask into a set of bit numbers."""
|
||||
bits: set[int] = set()
|
||||
for byte_index, value in enumerate(mask):
|
||||
for offset in range(8):
|
||||
if value & (0x80 >> offset):
|
||||
bits.add(byte_index * 8 + offset)
|
||||
return bits
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ServerCapabilities:
|
||||
"""Decoded SQ_PROTOCOLS feature bitmap plus login-response metadata.
|
||||
|
||||
``bits`` is the authoritative set; the named properties are readability
|
||||
sugar over it. ``raw_mask`` is kept so a bug report can include the
|
||||
exact bytes without needing a packet capture.
|
||||
"""
|
||||
|
||||
bits: frozenset[int] = field(default_factory=frozenset)
|
||||
raw_mask: bytes = b""
|
||||
cap_1: int = 0
|
||||
cap_2: int = 0
|
||||
cap_3: int = 0
|
||||
server_version: str = ""
|
||||
serial_number: str = ""
|
||||
applid_name: str = ""
|
||||
|
||||
@classmethod
|
||||
def from_wire(
|
||||
cls,
|
||||
mask: bytes,
|
||||
*,
|
||||
cap_1: int = 0,
|
||||
cap_2: int = 0,
|
||||
cap_3: int = 0,
|
||||
server_version: str = "",
|
||||
serial_number: str = "",
|
||||
applid_name: str = "",
|
||||
) -> ServerCapabilities:
|
||||
bits = _decode_bits(mask)
|
||||
if cap_1 != 0:
|
||||
# Mirrors JDBC: pre-set bits are OR-ed in and never cleared.
|
||||
bits |= _PRESET_BITS
|
||||
return cls(
|
||||
bits=frozenset(bits),
|
||||
raw_mask=bytes(mask),
|
||||
cap_1=cap_1,
|
||||
cap_2=cap_2,
|
||||
cap_3=cap_3,
|
||||
server_version=server_version,
|
||||
serial_number=serial_number,
|
||||
applid_name=applid_name,
|
||||
)
|
||||
|
||||
def has(self, bit: int) -> bool:
|
||||
return bit in self.bits
|
||||
|
||||
# -- named capabilities ------------------------------------------------
|
||||
|
||||
@property
|
||||
def usver(self) -> bool:
|
||||
return self.has(BIT_USVER)
|
||||
|
||||
@property
|
||||
def long_id(self) -> bool:
|
||||
return self.has(BIT_LONG_ID_A) or self.has(BIT_LONG_ID_B)
|
||||
|
||||
@property
|
||||
def varchar_var_len(self) -> bool:
|
||||
"""VARCHAR carries a 1-byte length prefix rather than being padded
|
||||
to its full declared width.
|
||||
|
||||
JDBC additionally consults the client-side ``ifxPADVARCHAR``
|
||||
setting; we never enable padding, so the client half is constant.
|
||||
"""
|
||||
return self.has(BIT_VARCHAR_VAR_LEN)
|
||||
|
||||
@property
|
||||
def four_byte_offset(self) -> bool:
|
||||
"""Describe records use 4-byte string-table size and field indices
|
||||
rather than 2-byte."""
|
||||
return self.has(BIT_FOUR_BYTE_OFFSET)
|
||||
|
||||
@property
|
||||
def bigint(self) -> bool:
|
||||
return self.has(BIT_BIGINT)
|
||||
|
||||
@property
|
||||
def lvarchar_gt_2k(self) -> bool:
|
||||
return self.has(BIT_LVARCHAR_GT_2K)
|
||||
|
||||
@property
|
||||
def remove_64k_limit(self) -> bool:
|
||||
"""4-byte length prefixes in the fast-path/SQ_FILE paths."""
|
||||
return self.has(BIT_REMOVE_64K_LIMIT)
|
||||
|
||||
@property
|
||||
def two_gb_fetch_buffer(self) -> bool:
|
||||
return self.has(BIT_TWO_GB_FETCH_BUFFER)
|
||||
|
||||
@property
|
||||
def gls(self) -> bool:
|
||||
return self.has(BIT_GLS)
|
||||
|
||||
@property
|
||||
def describe_input(self) -> bool:
|
||||
return self.has(BIT_DESCRIBE_INPUT)
|
||||
|
||||
@property
|
||||
def named_parameters(self) -> bool:
|
||||
return self.has(BIT_NAMED_PARAMETERS)
|
||||
|
||||
@property
|
||||
def savepoint(self) -> bool:
|
||||
return self.has(BIT_SAVEPOINT)
|
||||
|
||||
@property
|
||||
def enhanced_protocol(self) -> bool:
|
||||
return self.cap_1 == ENHANCED_PROTOCOL_CAP
|
||||
|
||||
# -- assumption checking ----------------------------------------------
|
||||
|
||||
def violated_assumptions(self) -> list[str]:
|
||||
"""Framing assumptions this driver hardcodes that the server
|
||||
contradicts.
|
||||
|
||||
Each entry names a place where we emit or parse a fixed wire shape
|
||||
that is actually capability-gated. An empty list means every
|
||||
hardcoded choice matches what the server negotiated.
|
||||
|
||||
This is a diagnostic, not a guarantee: it only covers bits we know
|
||||
to be framing-relevant from the decompiled JDBC. It exists so that
|
||||
a server we have never tested produces a loud, specific warning
|
||||
instead of silently corrupted rows.
|
||||
"""
|
||||
problems: list[str] = []
|
||||
if not self.four_byte_offset:
|
||||
problems.append(
|
||||
"server did not negotiate 4-byte describe offsets "
|
||||
f"(bit {BIT_FOUR_BYTE_OFFSET}), but parse_describe reads "
|
||||
"4-byte string-table size and field indices; column "
|
||||
"metadata will be misparsed"
|
||||
)
|
||||
if not self.varchar_var_len:
|
||||
problems.append(
|
||||
f"server did not negotiate variable-length VARCHAR "
|
||||
f"(bit {BIT_VARCHAR_VAR_LEN}); VARCHAR values are padded to "
|
||||
"the declared column width on the wire, but the row decoder "
|
||||
"expects a 1-byte length prefix"
|
||||
)
|
||||
if not self.remove_64k_limit:
|
||||
problems.append(
|
||||
f"server did not negotiate the 64K limit removal "
|
||||
f"(bit {BIT_REMOVE_64K_LIMIT}); fast-path routine signatures "
|
||||
"are emitted with a 4-byte length prefix that the server "
|
||||
"expects to be 2-byte"
|
||||
)
|
||||
return problems
|
||||
|
||||
def __repr__(self) -> str:
|
||||
named = [
|
||||
name
|
||||
for name, value in (
|
||||
("usver", self.usver),
|
||||
("4byte_offset", self.four_byte_offset),
|
||||
("varchar_var_len", self.varchar_var_len),
|
||||
("bigint", self.bigint),
|
||||
("long_id", self.long_id),
|
||||
("lvarchar>2k", self.lvarchar_gt_2k),
|
||||
("remove_64k", self.remove_64k_limit),
|
||||
("gls", self.gls),
|
||||
)
|
||||
if value
|
||||
]
|
||||
return (
|
||||
f"ServerCapabilities(cap_1={self.cap_1}, "
|
||||
f"mask={self.raw_mask.hex()}, "
|
||||
f"version={self.server_version!r}, "
|
||||
f"flags={'|'.join(named) or 'none'})"
|
||||
)
|
||||
@ -39,9 +39,18 @@ def build_get_routine_pdu(signature: str) -> bytes:
|
||||
``[short SQ_GETROUTINE=101][byte isRoutineById=0][int sigLen]
|
||||
[sig bytes][pad if odd][short fparamFlag=0][short SQ_EOT=12]``
|
||||
|
||||
JDBC's ``getJavaToIfxCharBytes`` uses 4-byte length prefix on
|
||||
modern servers (``isRemove64KLimitSupported``). We always emit the
|
||||
4-byte form — works against 12.10+ unequivocally.
|
||||
JDBC's ``getJavaToIfxCharBytes`` uses a 4-byte length prefix when
|
||||
``isRemove64KLimitSupported()`` (capability bit 62) is set, and a
|
||||
2-byte prefix otherwise. We always emit the 4-byte form.
|
||||
|
||||
Verified correct on 15.0.1.0.3DE and 12.10.FC12W1DE (2026-05-08) —
|
||||
the fast-path RPC tests pass on both. Not yet checked on 14.10.
|
||||
|
||||
An earlier version of this docstring asserted 12.10+ compatibility
|
||||
"unequivocally" before anyone had run it against 12.10. The claim
|
||||
happened to hold, but it was a guess. The bit is negotiated via
|
||||
``SQ_PROTOCOLS``, which this driver does not yet send, so this
|
||||
remains an assumption on any server we haven't measured.
|
||||
"""
|
||||
sig_bytes = signature.encode("iso-8859-1")
|
||||
sig_len = len(sig_bytes)
|
||||
|
||||
@ -29,6 +29,29 @@ class ProtocolError(Exception):
|
||||
"""Raised when wire bytes can't be parsed (truncated stream, bad framing)."""
|
||||
|
||||
|
||||
# Every way a wire read/write can fail, as one tuple.
|
||||
#
|
||||
# This exists because the tuple was written out by hand in six places and
|
||||
# drifted: four of them caught ``(ProtocolError, OSError)`` — which
|
||||
# ``IfxSocket`` never raises. It converts *every* socket failure, including
|
||||
# clean EOF, into ``OperationalError``, and that is a ``DatabaseError``, not
|
||||
# an ``OSError``. So those four handlers could not catch the thing they
|
||||
# existed to catch. The worst of them sat in a ``weakref.finalize`` callback,
|
||||
# where the escaping exception is printed to stderr and swallowed, leaving a
|
||||
# desynchronised connection to be returned to the pool marked healthy.
|
||||
#
|
||||
# Import this rather than re-typing the members. A tuple in one place cannot
|
||||
# drift out of sync with itself.
|
||||
def _wire_error_types() -> tuple[type[BaseException], ...]:
|
||||
# Imported lazily: exceptions.py must stay free of protocol imports.
|
||||
from .exceptions import InterfaceError, OperationalError
|
||||
|
||||
return (ProtocolError, OSError, OperationalError, InterfaceError)
|
||||
|
||||
|
||||
WIRE_ERRORS: tuple[type[BaseException], ...] = _wire_error_types()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Writer
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -209,6 +232,119 @@ class IfxStreamReader:
|
||||
return raw.decode(encoding)
|
||||
|
||||
|
||||
class BufferedSocketReader(IfxStreamReader):
|
||||
"""Phase 39: thin parser-view over the ``IfxSocket``'s persistent
|
||||
read-ahead buffer.
|
||||
|
||||
The buffer itself (``_recv_buf`` + ``_recv_pos``) lives on the
|
||||
``IfxSocket`` so it survives across reader instances. This is
|
||||
critical for the pipelined-executemany path (Phase 33), which
|
||||
streams N responses back-to-back — a per-reader buffer would lose
|
||||
pre-fetched bytes when one reader is destroyed and the next is
|
||||
created.
|
||||
|
||||
Why this is faster than ``_SocketReader``:
|
||||
|
||||
* **One ``recv()`` per ~64 KB** instead of one per field. A 100k-row
|
||||
SELECT calls ``read_exact`` ~450k times today; with this reader,
|
||||
``recv`` fires ~hundreds of times instead.
|
||||
* **No ``bytes.join``.** The bytearray IS the buffer; reads are
|
||||
slices, not chunk-joins.
|
||||
* **``struct.unpack_from(buf, offset)``** for fixed-width ints — reads
|
||||
directly from the bytearray at offset, avoiding the intermediate
|
||||
slice the legacy reader created.
|
||||
|
||||
Each method reads through ``self._sock._recv_buf`` directly. The
|
||||
``IfxSocket`` exposes ``fill_recv_buf(n)`` to ensure ``n`` bytes
|
||||
are available from the current cursor position; this method
|
||||
handles ``recv()``, compaction, and EOF detection.
|
||||
|
||||
``read_exact`` returns a fresh ``bytes`` (not a memoryview) —
|
||||
copying is cheap and decouples consumer lifetime from buffer
|
||||
compaction, removing a class of use-after-compact bugs at
|
||||
near-zero cost.
|
||||
"""
|
||||
|
||||
__slots__ = ("_sock",)
|
||||
|
||||
def __init__(self, sock):
|
||||
# Skip super().__init__ to avoid the unused BytesIO allocation
|
||||
# — we override every method that touches ``self._source``.
|
||||
self._sock = sock
|
||||
|
||||
def read_exact(self, n: int) -> bytes:
|
||||
if n <= 0:
|
||||
return b""
|
||||
sock = self._sock
|
||||
sock.fill_recv_buf(n)
|
||||
buf = sock._recv_buf
|
||||
pos = sock._recv_pos
|
||||
end = pos + n
|
||||
out = bytes(buf[pos:end])
|
||||
sock._recv_pos = end
|
||||
return out
|
||||
|
||||
def read_byte(self) -> int:
|
||||
sock = self._sock
|
||||
sock.fill_recv_buf(1)
|
||||
b = sock._recv_buf[sock._recv_pos]
|
||||
sock._recv_pos += 1
|
||||
return b
|
||||
|
||||
def skip(self, n: int) -> None:
|
||||
# The base reader's skip delegates to read_exact, which returns
|
||||
# b"" for n <= 0. This one advances the cursor arithmetically, so
|
||||
# without the guard a negative n *rewinds* it — silently
|
||||
# re-decoding bytes already consumed as if they were the next
|
||||
# field. Same divergence as read_exact, which does guard.
|
||||
if n <= 0:
|
||||
return
|
||||
sock = self._sock
|
||||
sock.fill_recv_buf(n)
|
||||
sock._recv_pos += n
|
||||
|
||||
def read_padded(self, n: int) -> bytes:
|
||||
out = self.read_exact(n)
|
||||
if n & 1:
|
||||
self.skip(1)
|
||||
return out
|
||||
|
||||
def read_short(self) -> int:
|
||||
sock = self._sock
|
||||
sock.fill_recv_buf(2)
|
||||
v = _FMT_SHORT.unpack_from(sock._recv_buf, sock._recv_pos)[0]
|
||||
sock._recv_pos += 2
|
||||
return v
|
||||
|
||||
def read_int(self) -> int:
|
||||
sock = self._sock
|
||||
sock.fill_recv_buf(4)
|
||||
v = _FMT_INT.unpack_from(sock._recv_buf, sock._recv_pos)[0]
|
||||
sock._recv_pos += 4
|
||||
return v
|
||||
|
||||
def read_long_bigint(self) -> int:
|
||||
sock = self._sock
|
||||
sock.fill_recv_buf(8)
|
||||
v = _FMT_LONG.unpack_from(sock._recv_buf, sock._recv_pos)[0]
|
||||
sock._recv_pos += 8
|
||||
return v
|
||||
|
||||
def read_real(self) -> float:
|
||||
sock = self._sock
|
||||
sock.fill_recv_buf(4)
|
||||
v = _FMT_FLOAT.unpack_from(sock._recv_buf, sock._recv_pos)[0]
|
||||
sock._recv_pos += 4
|
||||
return v
|
||||
|
||||
def read_double(self) -> float:
|
||||
sock = self._sock
|
||||
sock.fill_recv_buf(8)
|
||||
v = _FMT_DOUBLE.unpack_from(sock._recv_buf, sock._recv_pos)[0]
|
||||
sock._recv_pos += 8
|
||||
return v
|
||||
|
||||
|
||||
def make_pdu_writer() -> tuple[IfxStreamWriter, BytesIO]:
|
||||
"""Convenience: create a writer backed by a fresh in-memory buffer.
|
||||
|
||||
|
||||
@ -20,12 +20,22 @@ column names), read via readPadded.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import timedelta as _timedelta
|
||||
from types import MappingProxyType
|
||||
|
||||
from ._protocol import IfxStreamReader
|
||||
from ._protocol import IfxStreamReader, ProtocolError
|
||||
from ._types import IfxType, base_type, is_nullable
|
||||
from .converters import (
|
||||
_DOUBLE_NULL,
|
||||
_REAL_NULL,
|
||||
_UNPACK_DOUBLE,
|
||||
_UNPACK_FLOAT,
|
||||
_UNPACK_INT,
|
||||
_UNPACK_LONG,
|
||||
_UNPACK_SHORT,
|
||||
DECODERS,
|
||||
FIXED_WIDTHS,
|
||||
BlobLocator,
|
||||
ClobLocator,
|
||||
@ -35,6 +45,9 @@ from .converters import (
|
||||
_decode_datetime,
|
||||
_decode_interval,
|
||||
)
|
||||
from .converters import (
|
||||
_INFORMIX_DATE_EPOCH as _DATE_EPOCH,
|
||||
)
|
||||
|
||||
# Module-level type-code constants — lifted out of the hot loop in
|
||||
# parse_tuple_payload so we don't pay the IntFlag→int conversion per
|
||||
@ -213,6 +226,17 @@ _LENGTH_PREFIXED_SHORT_TYPES = frozenset({
|
||||
_TC_NVCHAR,
|
||||
})
|
||||
|
||||
# CHAR and NCHAR are **fixed-width**, space-padded to ``encoded_length``.
|
||||
# VARCHAR and NVARCHAR are byte-length-prefixed. Getting NCHAR wrong is not
|
||||
# a cosmetic bug: treating it as length-prefixed consumes its first
|
||||
# character as a length byte, then advances the offset by that value —
|
||||
# e.g. NCHAR(10) holding 'nch' reads 0x6E ('n') as a 110-byte length and
|
||||
# desyncs the rest of the row, usually into a struct.error crash.
|
||||
# Verified on the wire against 12.10 and 15:
|
||||
# NCHAR(10) 'nch' -> 6e 63 68 20 20 20 20 20 20 20 (10 B, padded)
|
||||
# NVARCHAR(20) 'nvc' -> 03 6e 76 63 (1-byte prefix)
|
||||
_FIXED_WIDTH_CHAR_TYPES = frozenset({_TC_CHAR, _TC_NCHAR})
|
||||
|
||||
_COMPOSITE_UDT_TYPES = frozenset({
|
||||
_TC_ROW,
|
||||
_TC_COLLECTION,
|
||||
@ -223,6 +247,140 @@ _COMPOSITE_UDT_TYPES = frozenset({
|
||||
|
||||
_NUMERIC_TYPES = frozenset({_TC_DECIMAL, _TC_MONEY})
|
||||
|
||||
|
||||
def _read_udt_envelope(payload: bytes, offset: int) -> tuple[int, bytes | None]:
|
||||
"""Read the UDT wire envelope: ``[1-byte indicator][int32 length][data]``.
|
||||
|
||||
Returns ``(new_offset, body)`` with ``body`` ``None`` when the
|
||||
indicator says NULL. The length field is present either way — it
|
||||
belongs to the envelope, not the value — so the offset advances by
|
||||
5 for a NULL and 5 + length otherwise.
|
||||
|
||||
This exists because the same envelope was hand-written four times and
|
||||
three of the copies were wrong, each in a different way:
|
||||
|
||||
* BOOLEAN read ``encoded_length`` (1 byte) instead of the envelope,
|
||||
leaving 5 bytes behind.
|
||||
* The composite-UDT branch returned on the indicator without reading
|
||||
the length, leaving 4 bytes behind.
|
||||
* BLOB/CLOB read a flat 72 bytes, leaving 77 behind.
|
||||
|
||||
Only the UDTVAR(lvarchar) copy was right, and only after two separate
|
||||
fixes. Four copies of one three-line format is four chances to get it
|
||||
wrong; this is one.
|
||||
"""
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
length = int.from_bytes(payload[offset : offset + 4], "big", signed=True)
|
||||
offset += 4
|
||||
if indicator == 1:
|
||||
return offset, None
|
||||
if length < 0:
|
||||
raise ProtocolError(
|
||||
f"UDT envelope declared a negative length ({length}) at payload "
|
||||
f"offset {offset - 4}; the wire is desynchronised"
|
||||
)
|
||||
body = payload[offset : offset + length]
|
||||
offset += length
|
||||
return offset, body
|
||||
|
||||
|
||||
def _decode_lob_locator(body: bytes, extended_id: int):
|
||||
"""Turn a smart-LOB envelope body into a Blob/ClobLocator.
|
||||
|
||||
The body is the 72-byte locator **hex-encoded as 144 ASCII characters**,
|
||||
not the raw locator. We previously consumed a flat 72 bytes, which took
|
||||
the first half of that hex text and handed it to ``BlobLocator`` as if
|
||||
it were binary — so ``BlobLocator.raw`` never actually held a locator.
|
||||
It went unnoticed because ``read_blob_column`` resolves LOBs through
|
||||
the server-side ``lotofile`` function and never uses the locator.
|
||||
"""
|
||||
cls = BlobLocator if extended_id == 10 else ClobLocator
|
||||
try:
|
||||
raw = bytes.fromhex(body.decode("ascii"))
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
# Not hex — surface what arrived rather than guessing, but only if
|
||||
# it is already the right size for a locator.
|
||||
raw = bytes(body)
|
||||
if len(raw) != 72:
|
||||
raise ProtocolError(
|
||||
f"smart-LOB locator decoded to {len(raw)} bytes, expected 72 "
|
||||
f"(envelope body was {len(body)} bytes)"
|
||||
)
|
||||
return cls(raw=raw)
|
||||
|
||||
|
||||
def _row_not_consumed(
|
||||
offset: int, payload_len: int, columns: list[ColumnInfo]
|
||||
) -> None:
|
||||
"""Raise when a decoded row didn't consume exactly its payload.
|
||||
|
||||
THIS IS THE CHECK THAT WOULD HAVE CAUGHT EVERY FRAMING BUG WE SHIPPED.
|
||||
|
||||
Thirteen of them reached users, and all thirteen were the same defect:
|
||||
a column consumed the wrong number of bytes. Not one raised at the
|
||||
point of the mistake — the wrong width produced a plausible value and
|
||||
corrupted whatever came *next*, so the damage surfaced in a different
|
||||
column, a different row, or a different statement.
|
||||
|
||||
Every one was detectable here for free. ``payload`` is a fully
|
||||
extracted ``bytes`` object of known length (the wire's even-alignment
|
||||
pad is consumed separately by the caller), so after decoding N columns
|
||||
the offset must land exactly on the end. Anything else means the codec
|
||||
and the server disagree about the row's shape, and continuing can only
|
||||
produce wrong answers.
|
||||
|
||||
A previous attempt at this check was reverted, and the comment
|
||||
explaining why survived long after its reason did: it cited the
|
||||
LVARCHAR odd-length pad, which was itself a bug and has since been
|
||||
removed. With that gone the invariant is exact.
|
||||
|
||||
The message names the column shape because "row decode failed" is not
|
||||
actionable — the whole point is to say *which* column shape and by how
|
||||
many bytes, so the next report arrives with the answer in it.
|
||||
"""
|
||||
delta = offset - payload_len
|
||||
direction = "over-read" if delta > 0 else "under-read"
|
||||
shape = ", ".join(
|
||||
f"{c.name}:tc={c.type_code}"
|
||||
f"{'/ext=' + c.extended_name if c.extended_name else ''}"
|
||||
f"/enclen={c.encoded_length}"
|
||||
for c in columns
|
||||
)
|
||||
raise ProtocolError(
|
||||
f"row decoder {direction} by {abs(delta)} byte(s): consumed "
|
||||
f"{offset} of {payload_len} payload bytes. This means the driver "
|
||||
f"and the server disagree about a column's wire width, so every "
|
||||
f"column after the offending one is unreliable. Column shape: "
|
||||
f"[{shape}]"
|
||||
)
|
||||
|
||||
|
||||
def _packed_width(encoded_length: int) -> int:
|
||||
"""On-wire byte width for the four types whose ``encoded_length``
|
||||
packs two fields into ``(high << 8) | low``: DECIMAL, MONEY,
|
||||
DATETIME and INTERVAL.
|
||||
|
||||
Mirrors ``IfxColumnInfo.adjustedColumnLength`` / ``IfxDecimal.decLength``::
|
||||
|
||||
((ColLength >> 8 & 0xFF) + (ColLength & 0xFF & 1) + 3) / 2
|
||||
|
||||
The ``low & 1`` term is the part that is easy to miss, and dropping
|
||||
it is wrong exactly when the high byte is even and the low byte is
|
||||
odd. For DECIMAL/MONEY the low byte is the scale, and an unscaled
|
||||
``DECIMAL(p)`` reports scale **255** — odd — so every floating
|
||||
DECIMAL with even precision was read one byte short, desyncing every
|
||||
column after it. ``DECIMAL(16)`` is 10 bytes on the wire, not 9.
|
||||
|
||||
For DATETIME and INTERVAL the low byte is the qualifier, whose
|
||||
parity happens to track the digit count, so the two formulas agree
|
||||
there in practice. They are unified here anyway: one rule taken from
|
||||
the reference beats two that coincide for reasons nobody wrote down.
|
||||
"""
|
||||
high = (encoded_length >> 8) & 0xFF
|
||||
low = encoded_length & 0xFF
|
||||
return (high + (low & 1) + 3) // 2
|
||||
|
||||
# Types that are fixed-width on the wire AND have a registered decoder
|
||||
# in ``FIXED_WIDTHS``: SMALLINT, INT, SERIAL, SMFLOAT, FLOAT, BIGINT,
|
||||
# BIGSERIAL, DATE, BOOL. These are the most common types in any real
|
||||
@ -234,10 +392,451 @@ _NUMERIC_TYPES = frozenset({_TC_DECIMAL, _TC_MONEY})
|
||||
_FIXED_WIDTH_TYPES = frozenset(FIXED_WIDTHS.keys())
|
||||
|
||||
|
||||
# Phase 37 — per-column reader strategy.
|
||||
#
|
||||
# parse_tuple_payload's hot loop used to evaluate the same dispatch
|
||||
# decisions per column per row: "is this a fixed-width type? a
|
||||
# length-prefixed string? what's the decoder?" Those decisions only
|
||||
# depend on column metadata, not row data — so we make them ONCE at
|
||||
# parse_describe time and emit a per-column tuple the hot loop can
|
||||
# dispatch on with a single integer comparison.
|
||||
#
|
||||
# Reader-strategy kinds (the first element of each compiled tuple).
|
||||
# Tuple shapes are documented at each kind's compile branch in
|
||||
# ``compile_column_readers`` below. Common types (covering >95% of
|
||||
# real-world workloads) get pre-compiled; rare types fall through
|
||||
# to the legacy dispatch in parse_tuple_payload.
|
||||
_RK_FIXED = 0 # (kind, width, decoder) — INT/FLOAT/DATE/etc.
|
||||
_RK_BYTE_PREFIX = 1 # (kind, decoder) — VARCHAR/NCHAR/NVCHAR
|
||||
_RK_CHAR = 2 # (kind, width, decoder) — fixed-width CHAR
|
||||
_RK_LVARCHAR = 3 # (kind, decoder) — LVARCHAR (4-byte prefix)
|
||||
_RK_DECIMAL = 4 # (kind, width, decoder) — DECIMAL/MONEY
|
||||
_RK_DATETIME = 5 # (kind, width, encoded_length) — DATETIME (uses _decode_datetime)
|
||||
_RK_INTERVAL = 6 # (kind, width, encoded_length) — INTERVAL (uses _decode_interval)
|
||||
_RK_LEGACY = 7 # (kind, type_code) — fall through to original dispatch
|
||||
|
||||
|
||||
def compile_column_readers(columns: list[ColumnInfo]) -> list[tuple]:
|
||||
"""Compile a per-column reader strategy.
|
||||
|
||||
Phase 37: replaces the per-row branch-dispatch in
|
||||
``parse_tuple_payload`` with a one-shot compilation pass at
|
||||
``parse_describe`` time. Each column gets a tuple the hot loop
|
||||
dispatches on with a single int comparison.
|
||||
|
||||
Common types (~95% of real workloads) get pre-compiled fast
|
||||
paths. Rare types (UDT/composite/CHAR-with-truncation/etc.)
|
||||
are tagged ``_RK_LEGACY`` and fall through to the legacy
|
||||
dispatch — preserves correctness on every shape we've seen
|
||||
while accelerating the hot path.
|
||||
"""
|
||||
readers: list[tuple] = []
|
||||
for col in columns:
|
||||
tc = col.type_code
|
||||
|
||||
if tc in _FIXED_WIDTH_TYPES:
|
||||
readers.append((_RK_FIXED, FIXED_WIDTHS[tc], DECODERS[tc]))
|
||||
continue
|
||||
|
||||
if tc in _FIXED_WIDTH_CHAR_TYPES:
|
||||
# CHAR and NCHAR: fixed width, space-padded to encoded_length.
|
||||
readers.append((_RK_CHAR, col.encoded_length, DECODERS[tc]))
|
||||
continue
|
||||
|
||||
if tc in _LENGTH_PREFIXED_SHORT_TYPES:
|
||||
# VARCHAR / NVARCHAR — CHAR and NCHAR excluded above.
|
||||
readers.append((_RK_BYTE_PREFIX, DECODERS[tc]))
|
||||
continue
|
||||
|
||||
if tc == _TC_LVARCHAR:
|
||||
readers.append((_RK_LVARCHAR, DECODERS[tc]))
|
||||
continue
|
||||
|
||||
if tc in _NUMERIC_TYPES:
|
||||
width = _packed_width(col.encoded_length)
|
||||
readers.append((_RK_DECIMAL, width, DECODERS[tc]))
|
||||
continue
|
||||
|
||||
if tc == _TC_DATETIME:
|
||||
width = _packed_width(col.encoded_length)
|
||||
readers.append((_RK_DATETIME, width, col.encoded_length))
|
||||
continue
|
||||
|
||||
if tc == _TC_INTERVAL:
|
||||
width = _packed_width(col.encoded_length)
|
||||
readers.append((_RK_INTERVAL, width, col.encoded_length))
|
||||
continue
|
||||
|
||||
# UDT / composite / unknown — let the legacy dispatch handle it.
|
||||
readers.append((_RK_LEGACY, tc))
|
||||
|
||||
return readers
|
||||
|
||||
|
||||
# Phase 38 codegen — sentinel constants imported into the generated
|
||||
# function's globals so inlined decode bodies can reference them by
|
||||
# name without dotted lookups.
|
||||
_INT_MIN_SENTINEL = -0x80000000
|
||||
_SHORT_MIN_SENTINEL = -0x8000
|
||||
_LONG_MIN_SENTINEL = -0x8000000000000000
|
||||
|
||||
|
||||
def compile_row_decoder(
|
||||
readers: list[tuple],
|
||||
columns: list[ColumnInfo],
|
||||
) -> Callable[[bytes, int, str], tuple] | None:
|
||||
"""Generate a specialized row decoder for a specific column shape.
|
||||
|
||||
Phase 38: takes the Phase 37 reader-list and emits a Python
|
||||
function via ``exec()`` that decodes one row of this exact shape
|
||||
in straight-line code — no per-column iteration, no per-column
|
||||
tuple-unpack, no per-column branch dispatch. Each column's
|
||||
decode logic is inlined directly.
|
||||
|
||||
The generated function has signature
|
||||
``parse_row(payload, offset, encoding) -> tuple`` and only
|
||||
references module-level helpers via its closure-equivalent
|
||||
globals dict (the ``_g`` dict below).
|
||||
|
||||
Returns ``None`` if any column's reader-kind is unsupported by
|
||||
the codegen — caller falls back to the Phase 37 dispatch loop.
|
||||
|
||||
The generated source is printable via ``IFX_DEBUG_CODEGEN=1``
|
||||
env var for inspection / debugging.
|
||||
"""
|
||||
import os
|
||||
|
||||
lines: list[str] = []
|
||||
lines.append("def parse_row(payload, offset, encoding):")
|
||||
val_names: list[str] = []
|
||||
|
||||
# Map type-code → inline-decoder source for the common fixed-width
|
||||
# decoders. Inlining the decoder body eliminates one function call
|
||||
# per column — the actual codegen win. For types not in this map,
|
||||
# fall back to ``_D{i}(raw)`` referencing the decoder via globals.
|
||||
_INLINE_FIXED = {
|
||||
# type_code: lambda v, raw_var: source-snippet
|
||||
# SMALLINT (1)
|
||||
1: lambda v, r: (
|
||||
f" {v} = _UNPACK_SHORT({r})[0]\n"
|
||||
f" if {v} == -32768:\n"
|
||||
f" {v} = None"
|
||||
),
|
||||
# INT (2), SERIAL (6) — same body
|
||||
2: lambda v, r: (
|
||||
f" {v} = _UNPACK_INT({r})[0]\n"
|
||||
f" if {v} == -2147483648:\n"
|
||||
f" {v} = None"
|
||||
),
|
||||
6: lambda v, r: (
|
||||
f" {v} = _UNPACK_INT({r})[0]\n"
|
||||
f" if {v} == -2147483648:\n"
|
||||
f" {v} = None"
|
||||
),
|
||||
# BIGINT (52), BIGSERIAL (53) — same body
|
||||
52: lambda v, r: (
|
||||
f" {v} = _UNPACK_LONG({r})[0]\n"
|
||||
f" if {v} == -9223372036854775808:\n"
|
||||
f" {v} = None"
|
||||
),
|
||||
53: lambda v, r: (
|
||||
f" {v} = _UNPACK_LONG({r})[0]\n"
|
||||
f" if {v} == -9223372036854775808:\n"
|
||||
f" {v} = None"
|
||||
),
|
||||
# FLOAT (3), SMFLOAT (4)
|
||||
3: lambda v, r: (
|
||||
f" if {r} == _DOUBLE_NULL:\n"
|
||||
f" {v} = None\n"
|
||||
f" else:\n"
|
||||
f" {v} = _UNPACK_DOUBLE({r})[0]"
|
||||
),
|
||||
4: lambda v, r: (
|
||||
f" if {r} == _REAL_NULL:\n"
|
||||
f" {v} = None\n"
|
||||
f" else:\n"
|
||||
f" {v} = _UNPACK_FLOAT({r})[0]"
|
||||
),
|
||||
# DATE (7) — 4-byte day count from 1899-12-31
|
||||
7: lambda v, r: (
|
||||
f" days = _UNPACK_INT({r})[0]\n"
|
||||
f" if days == -2147483648:\n"
|
||||
f" {v} = None\n"
|
||||
f" else:\n"
|
||||
f" {v} = _DATE_EPOCH + _timedelta(days=days)"
|
||||
),
|
||||
# BOOL (45) — left to the canonical decoder. Informix BOOL is
|
||||
# ``'t'/'T'/1``, NOT bool(byte) — a truthy-byte inline would
|
||||
# silently turn ``'f'`` (102) into True.
|
||||
}
|
||||
|
||||
for i, r in enumerate(readers):
|
||||
kind = r[0]
|
||||
v = f"v{i}"
|
||||
val_names.append(v)
|
||||
lines.append(f" # Col {i}: kind={kind}")
|
||||
|
||||
if kind == _RK_FIXED:
|
||||
_, width, _decoder = r
|
||||
lines.append(f" raw = payload[offset:offset+{width}]")
|
||||
lines.append(f" offset += {width}")
|
||||
# Find type code from the decoder identity (we don't have
|
||||
# tc directly in the reader tuple; recover via the columns
|
||||
# list).
|
||||
tc = columns[i].type_code
|
||||
inline_src = _INLINE_FIXED.get(tc)
|
||||
if inline_src is not None:
|
||||
lines.append(inline_src(v, "raw"))
|
||||
else:
|
||||
lines.append(f" {v} = _D{i}(raw)")
|
||||
|
||||
elif kind == _RK_BYTE_PREFIX:
|
||||
lines.append(" length = payload[offset]")
|
||||
lines.append(" offset += 1")
|
||||
lines.append(" raw = payload[offset:offset + length]")
|
||||
lines.append(" offset += length")
|
||||
lines.append(f" {v} = _D{i}(raw, encoding)")
|
||||
|
||||
elif kind == _RK_CHAR:
|
||||
_, width, _decoder = r
|
||||
lines.append(f" raw = payload[offset:offset+{width}]")
|
||||
lines.append(f" offset += {width}")
|
||||
lines.append(f" {v} = _D{i}(raw, encoding)")
|
||||
|
||||
elif kind == _RK_LVARCHAR:
|
||||
lines.append(
|
||||
" length = int.from_bytes("
|
||||
"payload[offset:offset+4], 'big', signed=True)"
|
||||
)
|
||||
lines.append(" offset += 4")
|
||||
lines.append(" raw = payload[offset:offset + length]")
|
||||
lines.append(" offset += length")
|
||||
# No even-byte pad — see _TC_LVARCHAR in the legacy chain.
|
||||
lines.append(f" {v} = _D{i}(raw, encoding)")
|
||||
|
||||
elif kind == _RK_DECIMAL:
|
||||
_, width, _decoder = r
|
||||
lines.append(f" raw = payload[offset:offset+{width}]")
|
||||
lines.append(f" offset += {width}")
|
||||
lines.append(" try:")
|
||||
lines.append(f" {v} = _D{i}(raw)")
|
||||
lines.append(" except NotImplementedError:")
|
||||
lines.append(f" {v} = raw")
|
||||
|
||||
elif kind == _RK_DATETIME:
|
||||
_, width, enc_len = r
|
||||
lines.append(f" raw = payload[offset:offset+{width}]")
|
||||
lines.append(f" offset += {width}")
|
||||
lines.append(f" {v} = _decode_datetime(raw, {enc_len})")
|
||||
|
||||
elif kind == _RK_INTERVAL:
|
||||
_, width, enc_len = r
|
||||
lines.append(f" raw = payload[offset:offset+{width}]")
|
||||
lines.append(f" offset += {width}")
|
||||
lines.append(f" {v} = _decode_interval(raw, {enc_len})")
|
||||
|
||||
elif kind == _RK_LEGACY:
|
||||
# Codegen for rare types: call the legacy helper. The
|
||||
# column metadata is referenced via the globals dict.
|
||||
tc = r[1]
|
||||
lines.append(
|
||||
f" offset, {v} = _legacy_dispatch_one_column("
|
||||
f"payload, offset, {tc}, _COL{i}, encoding)"
|
||||
)
|
||||
|
||||
else:
|
||||
# Unknown kind — abort codegen, caller falls back.
|
||||
return None
|
||||
|
||||
# Same end-of-row reconciliation the interpreted paths do. The
|
||||
# generated function is the hot path, so this must be emitted here
|
||||
# too — a check that only guards the slow path guards nothing in
|
||||
# production.
|
||||
lines.append(" if offset != len(payload):")
|
||||
lines.append(" _row_short(offset, len(payload))")
|
||||
if val_names:
|
||||
lines.append(f" return ({', '.join(val_names)},)")
|
||||
else:
|
||||
lines.append(" return ()")
|
||||
|
||||
src = "\n".join(lines)
|
||||
|
||||
if os.environ.get("IFX_DEBUG_CODEGEN") == "1":
|
||||
import sys
|
||||
print("=== informix_db codegen ===", file=sys.stderr)
|
||||
print(src, file=sys.stderr)
|
||||
print("=== end ===", file=sys.stderr)
|
||||
|
||||
# Build the globals dict for the generated function. Each column's
|
||||
# decoder (if any) is registered as ``_D<i>``; columns with the
|
||||
# _RK_LEGACY kind get their ColumnInfo as ``_COL<i>``.
|
||||
#
|
||||
# The inlined fixed-width snippets (see ``_INLINE_FIXED`` above)
|
||||
# reference precompiled struct unpackers and NULL sentinels by
|
||||
# name — they only resolve if we hand them to ``exec`` here.
|
||||
g: dict = {
|
||||
"_decode_datetime": _decode_datetime,
|
||||
"_decode_interval": _decode_interval,
|
||||
"_legacy_dispatch_one_column": _legacy_dispatch_one_column,
|
||||
# Bound to this shape's columns so the raised message can name them
|
||||
# without the generated source having to carry the list itself.
|
||||
"_row_short": (
|
||||
lambda offset, payload_len, _cols=columns: _row_not_consumed(
|
||||
offset, payload_len, _cols
|
||||
)
|
||||
),
|
||||
"_UNPACK_SHORT": _UNPACK_SHORT,
|
||||
"_UNPACK_INT": _UNPACK_INT,
|
||||
"_UNPACK_LONG": _UNPACK_LONG,
|
||||
"_UNPACK_FLOAT": _UNPACK_FLOAT,
|
||||
"_UNPACK_DOUBLE": _UNPACK_DOUBLE,
|
||||
"_DOUBLE_NULL": _DOUBLE_NULL,
|
||||
"_REAL_NULL": _REAL_NULL,
|
||||
"_DATE_EPOCH": _DATE_EPOCH,
|
||||
"_timedelta": _timedelta,
|
||||
"int": int, # ensure the builtin isn't shadowed
|
||||
"bool": bool,
|
||||
}
|
||||
for i, r in enumerate(readers):
|
||||
kind = r[0]
|
||||
if kind in (_RK_FIXED, _RK_CHAR, _RK_DECIMAL):
|
||||
g[f"_D{i}"] = r[2]
|
||||
elif kind in (_RK_BYTE_PREFIX, _RK_LVARCHAR):
|
||||
g[f"_D{i}"] = r[1]
|
||||
elif kind == _RK_LEGACY:
|
||||
g[f"_COL{i}"] = columns[i]
|
||||
|
||||
namespace: dict = {}
|
||||
try:
|
||||
exec(compile(src, "<informix_db codegen>", "exec"), g, namespace)
|
||||
except SyntaxError:
|
||||
return None
|
||||
|
||||
return namespace["parse_row"]
|
||||
|
||||
|
||||
def _legacy_dispatch_one_column(
|
||||
payload: bytes,
|
||||
offset: int,
|
||||
tc: int,
|
||||
col: ColumnInfo,
|
||||
encoding: str,
|
||||
) -> tuple[int, object]:
|
||||
"""Phase 37 fallback for rare types not covered by the pre-compiled
|
||||
reader strategies (UDTFIXED, COMPOSITE UDT, UDTVAR-lvarchar, unknown).
|
||||
|
||||
Mirrors the corresponding branches of the legacy ``parse_tuple_payload``
|
||||
dispatch chain but for one column at a time. Returns ``(new_offset,
|
||||
decoded_value)``.
|
||||
"""
|
||||
# BLOB / CLOB locator (UDTFIXED + extended_id 10/11)
|
||||
if tc == _TC_UDTFIXED and col.extended_id in (10, 11):
|
||||
# Smart LOBs use the UDT envelope, NOT a flat encoded_length field.
|
||||
# Measured on Informix 15: a populated BLOB column is 149 bytes
|
||||
# ([ind=0][len=144][144 hex chars]) and a NULL one is 5, while
|
||||
# encoded_length reports 72. Consuming 72 left 77 bytes behind and
|
||||
# corrupted every column after a LOB. Found by the end-of-row
|
||||
# reconciliation check, which fired on its first run.
|
||||
offset, body = _read_udt_envelope(payload, offset)
|
||||
if body is None:
|
||||
return offset, None
|
||||
return offset, _decode_lob_locator(body, col.extended_id)
|
||||
|
||||
# BOOLEAN. The server describes it as UDTFIXED (41) with
|
||||
# extended_name='boolean' and encoded_length=1, but ``encoded_length``
|
||||
# is the size of the *value*, not the field: on the wire it carries the
|
||||
# standard UDT envelope ``[1-byte null indicator][4-byte length][data]``
|
||||
# — 6 bytes total for a 1-byte value. Consuming only ``encoded_length``
|
||||
# leaves 5 bytes on the wire and desyncs every subsequent column.
|
||||
# Verified payload (INT, BOOLEAN 't', INT, VARCHAR 'tail'):
|
||||
# 00 01 b2 07 | 00 00 00 00 01 74 | 00 03 64 0e | 04 74 61 69 6c
|
||||
# The value byte is 0x74 ('t'), which _decode_bool already understands.
|
||||
if tc == _TC_UDTFIXED and (
|
||||
col.extended_name == "boolean" or col.extended_id == 5
|
||||
):
|
||||
offset, body = _read_udt_envelope(payload, offset)
|
||||
if body is None:
|
||||
return offset, None
|
||||
return offset, bool(body and body[0] in (ord("t"), ord("T"), 1))
|
||||
|
||||
# ROW / COLLECTION composite UDT
|
||||
if tc in _COMPOSITE_UDT_TYPES:
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
# The 4-byte length is part of the envelope and is present even when
|
||||
# the indicator says NULL — identical to the UDTVAR(lvarchar) branch
|
||||
# below, which decodes the same `[ind][int32 len][data]` shape.
|
||||
# Returning early on the indicator left those 4 bytes unread and
|
||||
# corrupted the following column. Wire evidence, Informix 15,
|
||||
# (INT, SET(INT), VARCHAR) with a NULL set:
|
||||
# 00 00 00 08 | 01 | 00 00 00 00 | 03 78 79 7a
|
||||
# a = 8 | ind | length = 0 | [3]"xyz"
|
||||
# Before the fix the VARCHAR decoded as '' instead of 'xyz'.
|
||||
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
|
||||
offset += 4
|
||||
if indicator == 1:
|
||||
return offset, None
|
||||
raw = bytes(payload[offset:offset + length])
|
||||
offset += length
|
||||
if tc == _TC_ROW:
|
||||
return offset, RowValue(raw=raw, schema=col.extended_name)
|
||||
return offset, CollectionValue(
|
||||
raw=raw,
|
||||
kind=_COLLECTION_KIND_MAP[tc],
|
||||
element_schema=col.extended_name,
|
||||
)
|
||||
|
||||
# UDTVAR with extended_name=lvarchar (e.g., result of lotofile())
|
||||
if tc == _TC_UDTVAR and col.extended_name == "lvarchar":
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
# The 4-byte length is present even when the indicator says NULL —
|
||||
# it is part of the envelope, not of the value. Returning early on
|
||||
# the indicator left those 4 bytes on the wire and desynced every
|
||||
# following column. Wire evidence (Informix 12.10), INT8 / LVARCHAR
|
||||
# / INT8 with the middle column varying:
|
||||
# 'odd' -> 00 | 00 00 00 03 | 6f 64 64 (8 bytes)
|
||||
# NULL -> 01 | 00 00 00 00 (5 bytes)
|
||||
# '' -> 00 | 00 00 00 00 (5 bytes)
|
||||
# Note NULL and empty-string differ only in the indicator byte.
|
||||
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
|
||||
offset += 4
|
||||
if indicator == 1:
|
||||
return offset, None
|
||||
raw = payload[offset:offset + length]
|
||||
offset += length
|
||||
# NO even-byte pad. The UDT envelope is [indicator][int len][bytes]
|
||||
# and the next column starts immediately after the last content
|
||||
# byte. Verified on the wire (Informix 12.10) for an odd-length
|
||||
# value — INT8(2001), LVARCHAR('PackageRoot'), INT8(10):
|
||||
# 00 01 00 00 07 d1 00 00 00 00 a = INT8 2001
|
||||
# 00 null indicator
|
||||
# 00 00 00 0b length 11
|
||||
# 50 61 63 6b 61 67 65 52 6f 6f 74 "PackageRoot" (odd)
|
||||
# 00 01 00 00 00 0a 00 00 00 00 b = INT8 10, starts immediately
|
||||
# A pad here consumed one byte too many and shifted every
|
||||
# subsequent column: INT8 10 decoded as 2560 (0x0A00), strings
|
||||
# lost their first character, and wide rows ran off the end of
|
||||
# the payload. Only fired for odd-length values, which is why the
|
||||
# test fixture ("lv value", 8 chars) never caught it.
|
||||
return offset, raw.decode(encoding)
|
||||
|
||||
# Unknown — surface ``encoded_length`` bytes raw.
|
||||
width = col.encoded_length
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
try:
|
||||
return offset, _decode_base(tc, raw, encoding)
|
||||
except NotImplementedError:
|
||||
return offset, raw
|
||||
|
||||
|
||||
def parse_tuple_payload(
|
||||
reader: IfxStreamReader,
|
||||
columns: list[ColumnInfo],
|
||||
encoding: str = "iso-8859-1",
|
||||
readers: list[tuple] | None = None,
|
||||
row_decoder: Callable[[bytes, int, str], tuple] | None = None,
|
||||
) -> tuple:
|
||||
"""Parse a SQ_TUPLE payload (the SQ_TUPLE tag is already consumed).
|
||||
|
||||
@ -270,8 +869,102 @@ def parse_tuple_payload(
|
||||
if size & 1:
|
||||
reader.read_exact(1)
|
||||
|
||||
# Phase 38 fastest path: a per-result-set decoder function compiled
|
||||
# via ``exec()`` from the column shape (see ``compile_row_decoder``).
|
||||
# All per-column dispatch is eliminated — each column's decode logic
|
||||
# is inlined in straight-line code.
|
||||
if row_decoder is not None:
|
||||
return row_decoder(payload, 0, encoding)
|
||||
|
||||
values: list[object] = []
|
||||
offset = 0
|
||||
|
||||
# Phase 37 fast path: if the caller pre-compiled a reader-strategy
|
||||
# list, dispatch on the integer kind for each column. The compile
|
||||
# step (``compile_column_readers``) made the per-column decisions
|
||||
# ONCE; this loop just executes them. Common types (FIXED, BYTE_PREFIX,
|
||||
# CHAR, LVARCHAR, DECIMAL, DATETIME, INTERVAL) get pre-baked tuples;
|
||||
# rare types fall through to the legacy branch chain via _RK_LEGACY.
|
||||
if readers is not None:
|
||||
for r in readers:
|
||||
kind = r[0]
|
||||
|
||||
if kind == _RK_FIXED:
|
||||
_, width, decoder = r
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
values.append(decoder(raw))
|
||||
continue
|
||||
|
||||
if kind == _RK_BYTE_PREFIX:
|
||||
_, decoder = r
|
||||
length = payload[offset]
|
||||
offset += 1
|
||||
raw = payload[offset:offset + length]
|
||||
offset += length
|
||||
values.append(decoder(raw, encoding))
|
||||
continue
|
||||
|
||||
if kind == _RK_CHAR:
|
||||
_, width, decoder = r
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
values.append(decoder(raw, encoding))
|
||||
continue
|
||||
|
||||
if kind == _RK_LVARCHAR:
|
||||
_, decoder = r
|
||||
length = int.from_bytes(
|
||||
payload[offset:offset + 4], "big", signed=True
|
||||
)
|
||||
offset += 4
|
||||
raw = payload[offset:offset + length]
|
||||
offset += length
|
||||
# No even-byte pad — see _TC_LVARCHAR in the legacy chain.
|
||||
values.append(decoder(raw, encoding))
|
||||
continue
|
||||
|
||||
if kind == _RK_DECIMAL:
|
||||
_, width, decoder = r
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
try:
|
||||
values.append(decoder(raw))
|
||||
except NotImplementedError:
|
||||
values.append(raw)
|
||||
continue
|
||||
|
||||
if kind == _RK_DATETIME:
|
||||
_, width, enc_len = r
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
values.append(_decode_datetime(raw, enc_len))
|
||||
continue
|
||||
|
||||
if kind == _RK_INTERVAL:
|
||||
_, width, enc_len = r
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
values.append(_decode_interval(raw, enc_len))
|
||||
continue
|
||||
|
||||
# _RK_LEGACY — rare type, fall back to the original dispatch.
|
||||
# Find the matching ColumnInfo (parallel index) and run the
|
||||
# legacy branch chain by recursing into the slow path. We
|
||||
# do this by setting ``readers = None`` and breaking out;
|
||||
# but since we're mid-loop, simpler: run the legacy code
|
||||
# inline via a helper.
|
||||
tc = r[1]
|
||||
col = columns[len(values)] # parallel index — values has one entry per processed col
|
||||
offset, value = _legacy_dispatch_one_column(
|
||||
payload, offset, tc, col, encoding
|
||||
)
|
||||
values.append(value)
|
||||
if offset != len(payload):
|
||||
_row_not_consumed(offset, len(payload), columns)
|
||||
return tuple(values)
|
||||
|
||||
# Legacy slow path (no pre-compiled readers).
|
||||
# Note: ``col.type_code`` is *already* base-typed by ``parse_describe``
|
||||
# (see INVARIANT comment there), so we don't re-strip high-bit flags
|
||||
# here. The original code called ``base_type(col.type_code)`` per
|
||||
@ -300,8 +993,8 @@ def parse_tuple_payload(
|
||||
# docs/CAPTURES/13-py-varchar.socat.log:
|
||||
# payload = 09 73 79 73 74 61 62 6c 65 73
|
||||
# = [byte 9]["systables"]
|
||||
# CHAR is fixed-width per encoded_length — handled below.
|
||||
if tc == _TC_CHAR:
|
||||
# CHAR and NCHAR are fixed-width per encoded_length.
|
||||
if tc in _FIXED_WIDTH_CHAR_TYPES:
|
||||
width = col.encoded_length
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
@ -313,14 +1006,22 @@ def parse_tuple_payload(
|
||||
values.append(_decode_base(tc, raw, encoding))
|
||||
continue
|
||||
|
||||
# LVARCHAR as a bare type code (43), i.e. without the UDT envelope:
|
||||
# ``[int length][bytes]``, no even-byte pad.
|
||||
#
|
||||
# Caveat worth stating plainly: every Informix server we have tested
|
||||
# (12.10, 14.10, 15) describes *all* LVARCHAR columns as UDTVAR (40)
|
||||
# with extended_name='lvarchar' — including casts like
|
||||
# ``'abc'::LVARCHAR`` — so this branch is unreachable in practice and
|
||||
# its framing is inferred from the UDTVAR evidence rather than
|
||||
# observed directly. It is kept consistent with that branch on the
|
||||
# reasoning that the content encoding shouldn't depend on how the
|
||||
# column happens to be described.
|
||||
if tc == _TC_LVARCHAR:
|
||||
# [int length][bytes][pad if odd]
|
||||
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
|
||||
offset += 4
|
||||
raw = payload[offset:offset + length]
|
||||
offset += length
|
||||
if length & 1:
|
||||
offset += 1
|
||||
values.append(_decode_base(tc, raw, encoding))
|
||||
continue
|
||||
|
||||
@ -328,8 +1029,7 @@ def parse_tuple_payload(
|
||||
# the high byte of encoded_length (packed as (precision << 8) | scale).
|
||||
# Per IfxRowColumn.loadColumnData and IfxToJavaDecimal byte sizing.
|
||||
if tc in _NUMERIC_TYPES:
|
||||
precision = (col.encoded_length >> 8) & 0xFF
|
||||
width = (precision + 1) // 2 + 1
|
||||
width = _packed_width(col.encoded_length)
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
try:
|
||||
@ -343,8 +1043,7 @@ def parse_tuple_payload(
|
||||
# (start_TU << 4) | end_TU). The decoder needs the qualifier too,
|
||||
# so we call it directly here rather than via the dispatch.
|
||||
if tc == _TC_DATETIME:
|
||||
digit_count = (col.encoded_length >> 8) & 0xFF
|
||||
width = (digit_count + 1) // 2 + 1
|
||||
width = _packed_width(col.encoded_length)
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
values.append(_decode_datetime(raw, col.encoded_length))
|
||||
@ -357,8 +1056,7 @@ def parse_tuple_payload(
|
||||
# qualifier is needed at decode time, so we bypass the generic
|
||||
# dispatch.
|
||||
if tc == _TC_INTERVAL:
|
||||
digit_count = (col.encoded_length >> 8) & 0xFF
|
||||
width = (digit_count + 1) // 2 + 1
|
||||
width = _packed_width(col.encoded_length)
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
values.append(_decode_interval(raw, col.encoded_length))
|
||||
@ -370,11 +1068,25 @@ def parse_tuple_payload(
|
||||
# we read here are an opaque server-side reference, NOT the
|
||||
# actual data. Phase 10 lets users fetch via lotofile + SQ_FILE.
|
||||
if tc == _TC_UDTFIXED and col.extended_id in (10, 11):
|
||||
width = col.encoded_length
|
||||
raw = payload[offset:offset + width]
|
||||
offset += width
|
||||
cls = BlobLocator if col.extended_id == 10 else ClobLocator
|
||||
values.append(cls(raw=bytes(raw)))
|
||||
# UDT envelope, not a flat encoded_length field — see the
|
||||
# matching branch in _legacy_dispatch_one_column.
|
||||
offset, body = _read_udt_envelope(payload, offset)
|
||||
values.append(
|
||||
None if body is None
|
||||
else _decode_lob_locator(body, col.extended_id)
|
||||
)
|
||||
continue
|
||||
|
||||
# BOOLEAN — UDT envelope, not a bare byte. See the matching branch
|
||||
# in _legacy_dispatch_one_column for the wire evidence.
|
||||
if tc == _TC_UDTFIXED and (
|
||||
col.extended_name == "boolean" or col.extended_id == 5
|
||||
):
|
||||
offset, body = _read_udt_envelope(payload, offset)
|
||||
values.append(
|
||||
None if body is None
|
||||
else bool(body and body[0] in (ord("t"), ord("T"), 1))
|
||||
)
|
||||
continue
|
||||
|
||||
# ROW / COLLECTION (Phase 12): composite UDTs. Wire format is
|
||||
@ -391,13 +1103,15 @@ def parse_tuple_payload(
|
||||
if tc in _COMPOSITE_UDT_TYPES:
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
if indicator == 1: # null
|
||||
values.append(None)
|
||||
continue
|
||||
# Length is present even when NULL — see the matching branch in
|
||||
# _legacy_dispatch_one_column for the wire evidence.
|
||||
length = int.from_bytes(
|
||||
payload[offset:offset + 4], "big", signed=True
|
||||
)
|
||||
offset += 4
|
||||
if indicator == 1: # null
|
||||
values.append(None)
|
||||
continue
|
||||
raw = bytes(payload[offset:offset + length])
|
||||
offset += length
|
||||
if tc == _TC_ROW:
|
||||
@ -422,17 +1136,18 @@ def parse_tuple_payload(
|
||||
if tc == _TC_UDTVAR and col.extended_name == "lvarchar":
|
||||
indicator = payload[offset]
|
||||
offset += 1
|
||||
if indicator == 1:
|
||||
values.append(None)
|
||||
continue
|
||||
# Length is present even when NULL, and there is no even-byte
|
||||
# pad — see the matching branch in _legacy_dispatch_one_column
|
||||
# for the wire evidence.
|
||||
length = int.from_bytes(
|
||||
payload[offset:offset + 4], "big", signed=True
|
||||
)
|
||||
offset += 4
|
||||
if indicator == 1:
|
||||
values.append(None)
|
||||
continue
|
||||
raw = payload[offset:offset + length]
|
||||
offset += length
|
||||
if length & 1:
|
||||
offset += 1
|
||||
values.append(raw.decode(encoding))
|
||||
continue
|
||||
|
||||
@ -462,4 +1177,6 @@ def parse_tuple_payload(
|
||||
# by Python's slicing semantics for strings — short = harmless).
|
||||
# If a future protocol message produces actual garbage here, add a
|
||||
# branch-local check at the offending dispatch path.
|
||||
if offset != len(payload):
|
||||
_row_not_consumed(offset, len(payload), columns)
|
||||
return tuple(values)
|
||||
|
||||
@ -17,11 +17,35 @@ the rest of the protocol layer.
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import socket
|
||||
import ssl
|
||||
|
||||
from ._protocol import ProtocolError
|
||||
from .exceptions import InterfaceError, OperationalError
|
||||
|
||||
|
||||
def _max_read_bytes() -> int:
|
||||
"""Ceiling for a single length-prefixed read, from IFX_MAX_READ_BYTES.
|
||||
|
||||
256 MiB by default: comfortably above any row, string table, or blob
|
||||
chunk a real query produces, and far below the values a desynced
|
||||
stream invents. A genuinely larger single value is possible (a very
|
||||
large TEXT column read in one go), which is why the knob exists.
|
||||
"""
|
||||
raw = os.environ.get("IFX_MAX_READ_BYTES")
|
||||
if raw:
|
||||
try:
|
||||
value = int(raw)
|
||||
except ValueError:
|
||||
value = 0
|
||||
if value > 0:
|
||||
return value
|
||||
return 256 * 1024 * 1024
|
||||
|
||||
|
||||
MAX_READ_BYTES = _max_read_bytes()
|
||||
|
||||
# A ``tls`` parameter to ``IfxSocket`` accepts:
|
||||
# False (default) — plain TCP
|
||||
# True — TLS with verification disabled (dev / self-signed)
|
||||
@ -42,7 +66,16 @@ def _make_default_dev_context() -> ssl.SSLContext:
|
||||
|
||||
|
||||
class IfxSocket:
|
||||
"""Owns a connected TCP socket and provides exact-N read/write."""
|
||||
"""Owns a connected TCP socket and provides exact-N read/write.
|
||||
|
||||
Phase 39 also owns the read-ahead buffer for ``BufferedSocketReader``.
|
||||
The buffer (``_recv_buf`` + ``_recv_pos``) is connection-scoped, not
|
||||
reader-scoped — readers are short-lived per-PDU views, but pipelined
|
||||
responses (e.g., Phase 33's pipelined ``executemany``) can stream
|
||||
multiple responses across reader boundaries. Persisting the buffer
|
||||
here means we never throw away pre-fetched bytes when one reader
|
||||
is destroyed and the next one is created.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@ -58,6 +91,11 @@ class IfxSocket:
|
||||
self._port = port
|
||||
self._read_timeout = read_timeout
|
||||
self._sock: socket.socket | None = None
|
||||
# Phase 39 read-ahead buffer. Empty until a BufferedSocketReader
|
||||
# touches us; subsequent readers share this buffer state.
|
||||
self._recv_buf: bytearray = bytearray()
|
||||
self._recv_pos: int = 0
|
||||
self._recv_size: int = 65536
|
||||
|
||||
try:
|
||||
sock = socket.create_connection((host, port), timeout=connect_timeout)
|
||||
@ -110,10 +148,42 @@ class IfxSocket:
|
||||
raise OperationalError(f"write failed: {e}") from e
|
||||
|
||||
def read_exact(self, n: int) -> bytes:
|
||||
"""Read exactly ``n`` bytes or raise on EOF / timeout."""
|
||||
"""Read exactly ``n`` bytes or raise on EOF / timeout.
|
||||
|
||||
Consumes the read-ahead buffer before touching the socket. This
|
||||
matters because two readers share one stream: ``BufferedSocketReader``
|
||||
fills ``_recv_buf`` (over-reading by design, up to ``_recv_size``),
|
||||
while ``Connection._drain_to_eot``, ``_raise_sq_err`` and the
|
||||
login path call this method directly. Recv'ing here while bytes
|
||||
sat unconsumed in the buffer would skip them, and skipped bytes
|
||||
in a length-framed protocol don't announce themselves — the next
|
||||
read lands mid-field and every subsequent one is wrong.
|
||||
|
||||
No workload triggers it today: the server sends one response per
|
||||
request, so recv returns exactly that response and the buffered
|
||||
reader consumes all of it before control returns here. That is a
|
||||
property of the traffic, not of the code. Pipelined executemany
|
||||
already puts multiple responses in flight, and the buffer is
|
||||
connection-scoped precisely so read-ahead can cross response
|
||||
boundaries. Making the two paths agree by construction costs one
|
||||
branch on a cold path.
|
||||
"""
|
||||
if self._sock is None:
|
||||
raise InterfaceError("socket is closed")
|
||||
if n <= 0:
|
||||
return b""
|
||||
wanted = n
|
||||
chunks: list[bytes] = []
|
||||
buffered = len(self._recv_buf) - self._recv_pos
|
||||
if buffered > 0:
|
||||
take = min(buffered, n)
|
||||
chunks.append(
|
||||
bytes(self._recv_buf[self._recv_pos : self._recv_pos + take])
|
||||
)
|
||||
self._recv_pos += take
|
||||
n -= take
|
||||
if n == 0:
|
||||
return chunks[0]
|
||||
remaining = n
|
||||
while remaining > 0:
|
||||
try:
|
||||
@ -124,12 +194,63 @@ class IfxSocket:
|
||||
if not chunk:
|
||||
self._force_close()
|
||||
raise OperationalError(
|
||||
f"server closed connection mid-read (wanted {n} bytes, got {n - remaining})"
|
||||
f"server closed connection mid-read "
|
||||
f"(wanted {wanted} bytes, got {wanted - remaining})"
|
||||
)
|
||||
chunks.append(chunk)
|
||||
remaining -= len(chunk)
|
||||
return b"".join(chunks)
|
||||
|
||||
def fill_recv_buf(self, need: int) -> None:
|
||||
"""Phase 39: ensure the read-ahead buffer holds ``need`` bytes
|
||||
forward of ``_recv_pos``, recv'ing from the socket as needed.
|
||||
|
||||
Compaction: when the read cursor has advanced past
|
||||
``_recv_size`` bytes, slice off the consumed prefix in place.
|
||||
That bounds memory at roughly ``2 * recv_size`` while still
|
||||
amortizing recv calls over many fields.
|
||||
"""
|
||||
if self._sock is None:
|
||||
raise InterfaceError("socket is closed")
|
||||
if need > MAX_READ_BYTES:
|
||||
# ``need`` is almost always a length field straight off the
|
||||
# wire. Trusting it means a corrupt or desynced stream turns
|
||||
# into an allocation of whatever that field happened to say —
|
||||
# a garbage 0x7FFFFFFF reads as a 2 GB request, and the loop
|
||||
# below sits in recv until the read timeout while the buffer
|
||||
# grows. Refusing names the number instead, which is the
|
||||
# diagnostic: a length that absurd means framing is already
|
||||
# lost upstream, not that the row is genuinely that big.
|
||||
raise ProtocolError(
|
||||
f"refusing to read {need} bytes in one field (limit "
|
||||
f"{MAX_READ_BYTES}). Either the wire has desynced, or a "
|
||||
f"single value really is this large — raise the limit "
|
||||
f"with the IFX_MAX_READ_BYTES environment variable."
|
||||
)
|
||||
avail = len(self._recv_buf) - self._recv_pos
|
||||
if avail >= need:
|
||||
return
|
||||
if self._recv_pos > self._recv_size:
|
||||
del self._recv_buf[: self._recv_pos]
|
||||
self._recv_pos = 0
|
||||
avail = len(self._recv_buf)
|
||||
target = max(need, self._recv_size)
|
||||
sock = self._sock
|
||||
while avail < need:
|
||||
try:
|
||||
chunk = sock.recv(target - avail)
|
||||
except OSError as e:
|
||||
self._force_close()
|
||||
raise OperationalError(f"read failed: {e}") from e
|
||||
if not chunk:
|
||||
self._force_close()
|
||||
raise OperationalError(
|
||||
f"server closed connection mid-read "
|
||||
f"(wanted {need} bytes, got {avail})"
|
||||
)
|
||||
self._recv_buf.extend(chunk)
|
||||
avail += len(chunk)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the socket. Idempotent and never raises."""
|
||||
if self._sock is None:
|
||||
|
||||
@ -62,7 +62,10 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import contextlib
|
||||
import functools
|
||||
import threading
|
||||
import weakref
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from typing import Any, TypeVar
|
||||
|
||||
from . import connect as _sync_connect
|
||||
@ -79,6 +82,61 @@ def _to_thread(fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
|
||||
return asyncio.to_thread(fn, *args, **kwargs)
|
||||
|
||||
|
||||
def _run_on(
|
||||
executor: ThreadPoolExecutor,
|
||||
fn: Callable[..., T],
|
||||
*args: Any,
|
||||
**kwargs: Any,
|
||||
) -> Awaitable[T]:
|
||||
"""Await ``fn`` on a specific executor rather than the loop's default."""
|
||||
loop = asyncio.get_running_loop()
|
||||
return loop.run_in_executor(executor, functools.partial(fn, *args, **kwargs))
|
||||
|
||||
|
||||
def _connection_executor(conn: _SyncConnection) -> ThreadPoolExecutor:
|
||||
"""The dedicated worker thread for one connection, created on demand.
|
||||
|
||||
``asyncio.to_thread`` runs on the event loop's default executor, which
|
||||
the whole process shares and which is sized from the CPU count —
|
||||
``min(32, cpu_count + 4)``, so six threads on a two-CPU container.
|
||||
Two consequences, both silent.
|
||||
|
||||
A cancelled await does not stop its worker. ``asyncio.to_thread``
|
||||
cannot interrupt the thread, so it keeps running the wire call until
|
||||
the read timeout expires. Cancellation is ordinary in a web app — a
|
||||
client disconnect cancels the request task — so a handful of them
|
||||
pins every thread in the shared pool and unrelated ``to_thread`` work
|
||||
anywhere else in the process stops dead. Measured: six cancelled
|
||||
calls against a six-worker default executor starve an unrelated
|
||||
``to_thread`` indefinitely.
|
||||
|
||||
And pool concurrency was capped by the same number without saying so.
|
||||
A pool with ``max_size=20`` on a two-CPU box ran six queries at once.
|
||||
|
||||
One thread per connection is the right size rather than a compromise:
|
||||
the sync connection serializes every wire operation on its own lock,
|
||||
so a second thread could do nothing but wait for the first. It also
|
||||
rules out a deadlock that a shared pool-sized executor invites — with
|
||||
N threads and N connections, N tasks blocked in ``acquire`` occupy
|
||||
every thread while the connection they are waiting for is held by a
|
||||
task that now needs a thread of its own to finish and release it.
|
||||
|
||||
The executor lives on the sync connection so it survives being
|
||||
returned to the pool and handed out again. The finalizer is the
|
||||
backstop for a connection dropped without ``close()``: a
|
||||
``ThreadPoolExecutor`` that is never shut down leaves its worker
|
||||
parked on the work queue for the life of the process.
|
||||
"""
|
||||
executor = getattr(conn, "_async_executor", None)
|
||||
if executor is None:
|
||||
executor = ThreadPoolExecutor(
|
||||
max_workers=1, thread_name_prefix="informix-conn"
|
||||
)
|
||||
conn._async_executor = executor
|
||||
weakref.finalize(conn, executor.shutdown, wait=False)
|
||||
return executor
|
||||
|
||||
|
||||
class AsyncCursor:
|
||||
"""Async wrapper over a sync :class:`Cursor`. Each I/O call awaits
|
||||
a thread-offloaded version of the sync operation.
|
||||
@ -88,10 +146,14 @@ class AsyncCursor:
|
||||
paying the thread-hop cost.
|
||||
"""
|
||||
|
||||
__slots__ = ("_cur",)
|
||||
__slots__ = ("_cur", "_run")
|
||||
|
||||
def __init__(self, cur: _SyncCursor):
|
||||
def __init__(self, cur: _SyncCursor, run: Callable[..., Awaitable[Any]]):
|
||||
self._cur = cur
|
||||
# The owning connection's runner, so cursor I/O lands on that
|
||||
# connection's dedicated thread rather than the shared default
|
||||
# executor. See _connection_executor.
|
||||
self._run = run
|
||||
|
||||
# -- Pass-through synchronous attributes (no I/O) ---------------------
|
||||
|
||||
@ -120,32 +182,32 @@ class AsyncCursor:
|
||||
async def execute(
|
||||
self, operation: str, parameters: Any = None
|
||||
) -> None:
|
||||
await _to_thread(self._cur.execute, operation, parameters)
|
||||
await self._run(self._cur.execute, operation, parameters)
|
||||
|
||||
async def executemany(
|
||||
self, operation: str, seq_of_parameters: Any
|
||||
) -> None:
|
||||
await _to_thread(
|
||||
await self._run(
|
||||
self._cur.executemany, operation, list(seq_of_parameters)
|
||||
)
|
||||
|
||||
async def fetchone(self) -> tuple | None:
|
||||
return await _to_thread(self._cur.fetchone)
|
||||
return await self._run(self._cur.fetchone)
|
||||
|
||||
async def fetchmany(self, size: int | None = None) -> list[tuple]:
|
||||
return await _to_thread(self._cur.fetchmany, size)
|
||||
return await self._run(self._cur.fetchmany, size)
|
||||
|
||||
async def fetchall(self) -> list[tuple]:
|
||||
return await _to_thread(self._cur.fetchall)
|
||||
return await self._run(self._cur.fetchall)
|
||||
|
||||
async def close(self) -> None:
|
||||
await _to_thread(self._cur.close)
|
||||
await self._run(self._cur.close)
|
||||
|
||||
# Phase 10/11 BLOB helpers (preserve the sync API surface)
|
||||
async def read_blob_column(
|
||||
self, sql: str, params: tuple = ()
|
||||
) -> bytes | None:
|
||||
return await _to_thread(self._cur.read_blob_column, sql, params)
|
||||
return await self._run(self._cur.read_blob_column, sql, params)
|
||||
|
||||
async def write_blob_column(
|
||||
self,
|
||||
@ -155,7 +217,7 @@ class AsyncCursor:
|
||||
*,
|
||||
clob: bool = False,
|
||||
) -> None:
|
||||
await _to_thread(
|
||||
await self._run(
|
||||
functools.partial(
|
||||
self._cur.write_blob_column,
|
||||
sql, blob_data, params, clob=clob,
|
||||
@ -177,14 +239,23 @@ class AsyncCursor:
|
||||
class AsyncConnection:
|
||||
"""Async wrapper over a sync :class:`Connection`."""
|
||||
|
||||
__slots__ = ("_conn",)
|
||||
__slots__ = ("_conn", "_executor")
|
||||
|
||||
def __init__(self, conn: _SyncConnection):
|
||||
self._conn = conn
|
||||
self._executor = _connection_executor(conn)
|
||||
|
||||
def _run(self, fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
|
||||
"""Run a blocking connection call on this connection's own thread."""
|
||||
return _run_on(self._executor, fn, *args, **kwargs)
|
||||
|
||||
@classmethod
|
||||
async def connect(cls, *args: Any, **kwargs: Any) -> AsyncConnection:
|
||||
"""Open a connection. Same parameters as :func:`informix_db.connect`."""
|
||||
"""Open a connection. Same parameters as :func:`informix_db.connect`.
|
||||
|
||||
The connect itself still goes to the default executor -- there is
|
||||
no connection yet to own a thread, and it is one bounded call.
|
||||
"""
|
||||
sync_conn = await _to_thread(
|
||||
functools.partial(_sync_connect, *args, **kwargs)
|
||||
)
|
||||
@ -195,22 +266,46 @@ class AsyncConnection:
|
||||
return self._conn.closed
|
||||
|
||||
async def cursor(self) -> AsyncCursor:
|
||||
sync_cur = await _to_thread(self._conn.cursor)
|
||||
return AsyncCursor(sync_cur)
|
||||
sync_cur = await self._run(self._conn.cursor)
|
||||
return AsyncCursor(sync_cur, self._run)
|
||||
|
||||
async def commit(self) -> None:
|
||||
await _to_thread(self._conn.commit)
|
||||
await self._run(self._conn.commit)
|
||||
|
||||
async def rollback(self) -> None:
|
||||
await _to_thread(self._conn.rollback)
|
||||
await self._run(self._conn.rollback)
|
||||
|
||||
async def close(self) -> None:
|
||||
await _to_thread(self._conn.close)
|
||||
"""Close the connection and stop the thread that served it.
|
||||
|
||||
One thread per connection is only affordable if the thread goes
|
||||
away with the connection. ``ThreadPoolExecutor`` workers park on
|
||||
the work queue rather than exiting when idle, so an executor that
|
||||
is never shut down leaks its thread for the life of the process —
|
||||
the ``weakref.finalize`` in ``_connection_executor`` is a backstop
|
||||
for connections dropped without ``close()``, not a substitute for
|
||||
closing here.
|
||||
|
||||
``wait=False`` because we are on the event loop: the worker has
|
||||
just finished the close and needs no waiting, and blocking the
|
||||
loop to confirm that would be the one thing this module exists to
|
||||
avoid.
|
||||
"""
|
||||
try:
|
||||
await self._run(self._conn.close)
|
||||
finally:
|
||||
self._executor.shutdown(wait=False)
|
||||
# Drop the reference too. A shut-down executor rejects new
|
||||
# work, so leaving it attached would turn any later wrap of
|
||||
# this sync connection into a RuntimeError rather than a
|
||||
# fresh thread.
|
||||
with contextlib.suppress(AttributeError):
|
||||
del self._conn._async_executor
|
||||
|
||||
async def fast_path_call(
|
||||
self, signature: str, *params: object
|
||||
) -> list[object]:
|
||||
return await _to_thread(self._conn.fast_path_call, signature, *params)
|
||||
return await self._run(self._conn.fast_path_call, signature, *params)
|
||||
|
||||
# Async context-manager support
|
||||
async def __aenter__(self) -> AsyncConnection:
|
||||
@ -249,14 +344,69 @@ class AsyncConnectionPool:
|
||||
return self._pool.idle_count
|
||||
|
||||
async def acquire(self, timeout: float | None = None) -> AsyncConnection:
|
||||
sync_conn = await _to_thread(self._pool.acquire, timeout)
|
||||
"""Acquire a connection, surviving cancellation of the waiter.
|
||||
|
||||
``asyncio.to_thread`` cannot interrupt its worker. If the task
|
||||
awaiting an acquire is cancelled while the worker is still
|
||||
blocked waiting for a free connection, the worker eventually
|
||||
succeeds and hands back a connection **that nobody owns** — it
|
||||
is checked out of the pool and never returned. Repeat that and
|
||||
the pool starves to death.
|
||||
|
||||
This is not hypothetical for anyone serving HTTP: a client
|
||||
disconnecting cancels the request task, and under load those
|
||||
cancellations land precisely while waiting for a connection. The
|
||||
pool then dies one slot at a time, and only under load.
|
||||
|
||||
So the inner future is shielded — cancelling the caller must not
|
||||
orphan a result we still need to see — and if the caller does go
|
||||
away, a callback returns whatever the worker produced to the
|
||||
pool. ``add_done_callback`` fires immediately when the future has
|
||||
already resolved, so the "worker finished a moment before the
|
||||
cancellation" race is covered by the same code path.
|
||||
"""
|
||||
fut = asyncio.ensure_future(_to_thread(self._pool.acquire, timeout))
|
||||
try:
|
||||
sync_conn = await asyncio.shield(fut)
|
||||
except asyncio.CancelledError:
|
||||
fut.add_done_callback(self._return_orphan)
|
||||
raise
|
||||
return AsyncConnection(sync_conn)
|
||||
|
||||
def _return_orphan(self, fut: asyncio.Future) -> None:
|
||||
"""Give back a connection whose acquirer was cancelled.
|
||||
|
||||
Runs on the event loop, so it must not block: ``release`` takes
|
||||
the connection's wire lock and can wait. Hand off to a short-lived
|
||||
daemon thread rather than the running loop — this path also has to
|
||||
work while the loop is shutting down, which is exactly when
|
||||
``create_task`` is unavailable.
|
||||
"""
|
||||
if fut.cancelled() or fut.exception() is not None:
|
||||
return
|
||||
conn = fut.result()
|
||||
|
||||
def _release() -> None:
|
||||
# broken=False: the connection was never handed to anyone, so
|
||||
# its wire is untouched and it is safe to reuse. Evicting here
|
||||
# would trade a leak for needless reconnect churn.
|
||||
with contextlib.suppress(Exception):
|
||||
self._pool.release(conn, broken=False)
|
||||
|
||||
threading.Thread(target=_release, daemon=True).start()
|
||||
|
||||
async def release(
|
||||
self, conn: AsyncConnection, *, broken: bool = False
|
||||
) -> None:
|
||||
await _to_thread(
|
||||
functools.partial(self._pool.release, conn._conn, broken=broken)
|
||||
# On the connection's own thread, not the default executor. That
|
||||
# thread is idle by definition — the caller is done with the
|
||||
# connection — and it keeps the release off a shared pool that
|
||||
# tasks blocked in ``acquire`` may have filled. Release has to
|
||||
# win that race: it is what frees the connection they are
|
||||
# waiting for.
|
||||
await _run_on(
|
||||
conn._executor,
|
||||
functools.partial(self._pool.release, conn._conn, broken=broken),
|
||||
)
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
|
||||
@ -12,15 +12,18 @@ reference in ``docs/CAPTURES/01-connect-only.socat.log``.
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import logging
|
||||
import os
|
||||
import socket as socket_mod
|
||||
import ssl
|
||||
import struct
|
||||
import threading
|
||||
import weakref
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
|
||||
from . import _auth
|
||||
from ._capabilities import CLIENT_PROTOCOLS_MASK, ServerCapabilities
|
||||
from ._messages import (
|
||||
APPL_ID,
|
||||
APPL_TYPE,
|
||||
@ -36,10 +39,16 @@ from ._messages import (
|
||||
SLHeader,
|
||||
StmtOptions,
|
||||
)
|
||||
from ._protocol import IfxStreamReader, IfxStreamWriter, ProtocolError, make_pdu_writer
|
||||
from ._protocol import (
|
||||
WIRE_ERRORS,
|
||||
IfxStreamReader,
|
||||
IfxStreamWriter,
|
||||
ProtocolError,
|
||||
make_pdu_writer,
|
||||
)
|
||||
from ._socket import IfxSocket
|
||||
from .cursors import Cursor
|
||||
from .exceptions import InterfaceError, OperationalError
|
||||
from .exceptions import InterfaceError, OperationalError, ProgrammingError
|
||||
|
||||
# Default capability bits the JDBC reference sends. Validated against
|
||||
# 01-connect-only.socat.log via the PDU diff in tests/test_pdu_match.py:
|
||||
@ -69,6 +78,9 @@ _LOCALE_ENCODING_MAP = {
|
||||
}
|
||||
|
||||
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _extract_server_error_text(payload: bytes) -> str | None:
|
||||
"""Pull the longest printable run out of an opaque rejection payload.
|
||||
|
||||
@ -113,6 +125,74 @@ def _python_encoding_from_locale(locale: str) -> str:
|
||||
return _LOCALE_ENCODING_MAP.get(suffix, "iso-8859-1")
|
||||
|
||||
|
||||
def _decode_conacc(rest: bytes) -> dict | None:
|
||||
"""Decode the SLTYPE_CONACC server-metadata block.
|
||||
|
||||
Mirrors ``com.informix.asf.Connection.DecodeAscBinary``. ``rest`` is
|
||||
the login response with its 2-byte total-length prefix already
|
||||
stripped, i.e. starting at the SLType byte.
|
||||
|
||||
Layout, verified byte-for-byte against Informix 12.10, 14.10 and 15::
|
||||
|
||||
byte SLType (2 = CONACC)
|
||||
3 skip
|
||||
short 100 marker
|
||||
short 101 marker
|
||||
4 skip
|
||||
short len; skip len (codeset id, e.g. "IEEEI")
|
||||
short 108 marker — JDBC hard-fails if this isn't 108
|
||||
12 skip
|
||||
short len; bytes server version string
|
||||
short len; bytes serial number
|
||||
short len; bytes applid name
|
||||
int Cap_1 client protocol level, echoed (316 = enhanced)
|
||||
int Cap_2
|
||||
int Cap_3
|
||||
|
||||
Returns ``None`` if the block doesn't match that shape. Callers must
|
||||
treat that as "no capability info", never as a connection failure —
|
||||
every server we support connects fine without any of this being
|
||||
decodable, and a future server variant should degrade, not break.
|
||||
"""
|
||||
try:
|
||||
offset = 1 + 3 # SLType + 3 skipped bytes
|
||||
marker_a, marker_b = struct.unpack_from("!hh", rest, offset)
|
||||
offset += 4
|
||||
if marker_a != 100 or marker_b != 101:
|
||||
return None
|
||||
offset += 4
|
||||
(codeset_len,) = struct.unpack_from("!h", rest, offset)
|
||||
offset += 2 + codeset_len
|
||||
(marker_c,) = struct.unpack_from("!h", rest, offset)
|
||||
offset += 2
|
||||
if marker_c != 108:
|
||||
return None
|
||||
offset += 12
|
||||
strings: list[str] = []
|
||||
for _ in range(3):
|
||||
(length,) = struct.unpack_from("!h", rest, offset)
|
||||
offset += 2
|
||||
if length < 0:
|
||||
return None
|
||||
strings.append(
|
||||
rest[offset : offset + length].rstrip(b"\x00").decode(
|
||||
"iso-8859-1", "replace"
|
||||
)
|
||||
)
|
||||
offset += length
|
||||
cap_1, cap_2, cap_3 = struct.unpack_from("!iii", rest, offset)
|
||||
except (struct.error, IndexError, UnicodeDecodeError):
|
||||
return None
|
||||
return {
|
||||
"server_version": strings[0],
|
||||
"serial_number": strings[1],
|
||||
"applid_name": strings[2],
|
||||
"cap_1": cap_1,
|
||||
"cap_2": cap_2,
|
||||
"cap_3": cap_3,
|
||||
}
|
||||
|
||||
|
||||
# Default environment variables sent in the login PDU (SQ_ASCENV section).
|
||||
# These match what the JDBC driver sends for a vanilla en_US.8859-1
|
||||
# connection. Anything missing makes the server fall back to defaults.
|
||||
@ -126,6 +206,62 @@ _DEFAULT_ENV: dict[str, str] = {
|
||||
}
|
||||
|
||||
|
||||
class _WireLock:
|
||||
"""A reentrant lock that can say whether *this* thread already holds it.
|
||||
|
||||
``threading.RLock`` cannot answer that question in public API, and
|
||||
the difference is not academic. The cursor finalizer runs at GC time
|
||||
on whatever thread happened to allocate — including a thread that is
|
||||
at that moment mid-statement holding this lock. It probes with
|
||||
``acquire(blocking=False)`` intending to mean "is anyone using the
|
||||
wire?", but an RLock grants a reentrant acquire to its own owner, so
|
||||
the probe returns True and the finalizer sends CLOSE/RELEASE into the
|
||||
middle of the statement it interrupted. The victim gets ``-208``.
|
||||
|
||||
Reachability is not exotic. Refcounting frees a dropped cursor
|
||||
immediately, before the next statement, which is why this went
|
||||
unnoticed — but a cursor caught in a reference cycle waits for a
|
||||
collection instead, and cycles are ordinary in Python. Any traceback
|
||||
that holds a cursor makes one.
|
||||
|
||||
``held_by_current_thread`` reads ``_depth``/``_owner`` without the
|
||||
lock, which is safe: both are mutated only under it, and the only
|
||||
values another thread can leave behind are a depth of zero or an
|
||||
owner that isn't us. Either way the answer is False, correctly.
|
||||
"""
|
||||
|
||||
__slots__ = ("_depth", "_lock", "_owner")
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._lock = threading.RLock()
|
||||
self._owner: int | None = None
|
||||
self._depth = 0
|
||||
|
||||
def acquire(self, blocking: bool = True, timeout: float = -1) -> bool:
|
||||
acquired = self._lock.acquire(blocking, timeout)
|
||||
if acquired:
|
||||
self._owner = threading.get_ident()
|
||||
self._depth += 1
|
||||
return acquired
|
||||
|
||||
def release(self) -> None:
|
||||
self._depth -= 1
|
||||
if self._depth == 0:
|
||||
self._owner = None
|
||||
self._lock.release()
|
||||
|
||||
def __enter__(self) -> _WireLock:
|
||||
self.acquire()
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc_info: object) -> None:
|
||||
self.release()
|
||||
|
||||
@property
|
||||
def held_by_current_thread(self) -> bool:
|
||||
return self._depth > 0 and self._owner == threading.get_ident()
|
||||
|
||||
|
||||
class Connection:
|
||||
"""A SQLI session. Owns one TCP socket and the post-login state.
|
||||
|
||||
@ -147,6 +283,7 @@ class Connection:
|
||||
client_locale: str = "en_US.8859-1",
|
||||
env: dict[str, str] | None = None,
|
||||
autocommit: bool = False, # honored from Phase 3 onward
|
||||
row_factory: object | None = None,
|
||||
tls: bool | ssl.SSLContext = False,
|
||||
tls_server_hostname: str | None = None,
|
||||
):
|
||||
@ -170,7 +307,7 @@ class Connection:
|
||||
# this lock with a timeout, then calls ``conn.rollback()`` —
|
||||
# which itself acquires the lock. Same thread, two acquires.
|
||||
# Reentrance must be cheap and correct.
|
||||
self._wire_lock = threading.RLock()
|
||||
self._wire_lock = _WireLock()
|
||||
# Phase 29: deferred-cleanup queue for cursor finalizers that
|
||||
# couldn't acquire the wire lock at GC time. Each entry is a
|
||||
# PDU's worth of bytes (typically a CLOSE or RELEASE) that
|
||||
@ -190,12 +327,21 @@ class Connection:
|
||||
# under ``_wire_lock``.
|
||||
self._pending_cleanup: list[bytes] = []
|
||||
self._cleanup_lock = threading.Lock()
|
||||
# Weak ref to the scrollable cursor currently holding the
|
||||
# session's statement slot, or None. Weak so that abandoning a
|
||||
# scrollable cursor still lets its finalizer run — a strong ref
|
||||
# here would keep the very object alive whose GC we depend on.
|
||||
self._open_scroll_cursor: weakref.ref | None = None
|
||||
# Logged-DB transaction state: True iff there's an open server-side
|
||||
# transaction (SQ_BEGIN sent, not yet committed/rolled-back). The
|
||||
# cursor uses this to decide whether to send an implicit SQ_BEGIN
|
||||
# before the next DML in non-autocommit mode. We default to "no
|
||||
# open txn" — the first DML will trigger SQ_BEGIN.
|
||||
self._in_transaction = False
|
||||
# Default row type for cursors from this connection. None means
|
||||
# plain tuples, which is the zero-cost default; see
|
||||
# informix_db.rows for the opt-in named-access type.
|
||||
self.row_factory = row_factory
|
||||
# Tri-state: True after first successful SQ_BEGIN, False after
|
||||
# an unlogged-DB rejection (-201). None until we've tried.
|
||||
# Used to avoid repeatedly probing on unlogged DBs.
|
||||
@ -205,6 +351,16 @@ class Connection:
|
||||
# SQ_GETROUTINE; subsequent calls skip that round-trip.
|
||||
self._fp_handle_cache: dict[str, tuple[str, int]] = {}
|
||||
|
||||
# Server metadata from the login response and the SQ_PROTOCOLS
|
||||
# negotiation. Both are populated during connect; both degrade to
|
||||
# None rather than failing the connection.
|
||||
self._conacc: dict | None = None
|
||||
self._server_protocols: bytes | None = None
|
||||
self._capabilities: ServerCapabilities | None = None
|
||||
# Lazily filled by the server_version property. None = not asked
|
||||
# yet; "" = asked and failed (don't retry on every access).
|
||||
self._server_version_full: str | None = None
|
||||
|
||||
# Build the env-var dict sent in the login PDU.
|
||||
self._env = dict(_DEFAULT_ENV)
|
||||
self._env["CLIENT_LOCALE"] = client_locale
|
||||
@ -274,22 +430,80 @@ class Connection:
|
||||
raise InterfaceError("connection is closed")
|
||||
return Cursor(self, scrollable=scrollable)
|
||||
|
||||
def _send_pdu(self, pdu: bytes) -> None:
|
||||
def _send_pdu(self, pdu: bytes, *, statement_boundary: bool = False) -> None:
|
||||
"""Send an assembled PDU. Used by Cursor.
|
||||
|
||||
Phase 29: opportunistically drains any pending cleanup PDUs
|
||||
from the deferred-cleanup queue *before* sending the new PDU.
|
||||
Caller must hold ``_wire_lock`` (every actual call site already
|
||||
does — execute/executemany/_sfetch_at, commit, rollback,
|
||||
fast_path_call, etc.). The drain happens under that lock so
|
||||
the queued cleanup atomically completes before the next op.
|
||||
Caller must hold ``_wire_lock`` (every actual call site does).
|
||||
|
||||
``statement_boundary=True`` additionally drains the deferred
|
||||
cleanup queue first. **Only pass it when this PDU is the first
|
||||
of a new statement**, meaning no statement is currently open
|
||||
server-side.
|
||||
|
||||
The queue previously drained before *every* PDU, which was wrong
|
||||
in a way that took a repro to see. ``SQ_CLOSE`` and
|
||||
``SQ_RELEASE`` carry no statement identifier — they act on the
|
||||
server's *current* statement. A finalizer enqueues precisely
|
||||
because it lost the race for the wire lock, which means another
|
||||
thread is mid-statement; that thread's next ``_send_pdu`` was
|
||||
then its own ``CURNAME``/``NFETCH``, and the drain released the
|
||||
statement out from under it. The caller saw a nonsense error on
|
||||
valid SQL: ``-208`` when injected before the first fetch,
|
||||
``-267`` "transaction has been rolled back" between fetch
|
||||
batches. Both point nowhere near the actual cause, and the
|
||||
window is exactly the window in which enqueueing happens.
|
||||
|
||||
Draining only at a boundary costs nothing: cleanup that misses
|
||||
one statement is picked up by the next.
|
||||
"""
|
||||
if self._closed:
|
||||
raise InterfaceError("connection is closed")
|
||||
if self._pending_cleanup:
|
||||
if statement_boundary and self._pending_cleanup:
|
||||
self._drain_pending_cleanup()
|
||||
self._sock.write_all(pdu)
|
||||
|
||||
def _check_scroll_cursor_conflict(self, requester: object) -> None:
|
||||
"""Refuse to start a statement while a scrollable cursor is open.
|
||||
|
||||
A server-side scrollable cursor occupies the session's statement
|
||||
slot, and SQLI gives us no way to address around it: ``SQ_CLOSE``,
|
||||
``SQ_RELEASE`` and ``SQ_SFETCH`` all act on the session's current
|
||||
statement.
|
||||
|
||||
The server does not refuse politely. Starting another statement
|
||||
returns ``-285``, and the scrollable cursor is collateral damage
|
||||
— its next fetch comes back ``-267`` "the transaction has been
|
||||
rolled back, all locks released". Two unattributable failures
|
||||
from code that looks entirely ordinary: iterate a large result
|
||||
set with a scrollable cursor, run a lookup query partway through.
|
||||
|
||||
Multiplexing is presumably expressible — JDBC prefixes every
|
||||
statement-scoped PDU with the statement id, which the server does
|
||||
assign distinctly (0 and 1 for two concurrent cursors). But the
|
||||
id alone is not sufficient: addressing the second cursor's
|
||||
``SQ_SFETCH`` to its own id returns ``-259`` "cursor not open".
|
||||
Until that is understood, refusing is the honest behaviour. It
|
||||
costs the caller a second connection and it never corrupts.
|
||||
"""
|
||||
ref = self._open_scroll_cursor
|
||||
if ref is None:
|
||||
return
|
||||
other = ref()
|
||||
if (
|
||||
other is None
|
||||
or other is requester
|
||||
or not getattr(other, "_server_cursor_open", False)
|
||||
):
|
||||
self._open_scroll_cursor = None
|
||||
return
|
||||
raise ProgrammingError(
|
||||
"a scrollable cursor is open on this connection; Informix "
|
||||
"allows only one statement per session, so running another "
|
||||
"statement here would fail with -285 and destroy the "
|
||||
"scrollable cursor as well. Close the scrollable cursor "
|
||||
"first, or use a separate connection for the other statement."
|
||||
)
|
||||
|
||||
def _enqueue_cleanup(self, pdus: list[bytes]) -> None:
|
||||
"""Append cleanup PDUs to the deferred queue.
|
||||
|
||||
@ -319,7 +533,6 @@ class Connection:
|
||||
discarded; the server-side resources they would have released
|
||||
are freed when the session ends anyway.
|
||||
"""
|
||||
from ._protocol import ProtocolError
|
||||
|
||||
with self._cleanup_lock:
|
||||
if not self._pending_cleanup:
|
||||
@ -330,11 +543,42 @@ class Connection:
|
||||
try:
|
||||
self._sock.write_all(pdu)
|
||||
self._drain_to_eot()
|
||||
except (ProtocolError, OSError, OperationalError):
|
||||
# Wire is unrecoverable; force-close. Subsequent
|
||||
# ``_send_pdu`` will raise InterfaceError. Server
|
||||
# cleanup of the remaining queued entries happens
|
||||
# implicitly at session end.
|
||||
except Exception as exc:
|
||||
if getattr(exc, "sqlcode", None) is not None:
|
||||
# The *server* rejected the cleanup — a stale entry
|
||||
# for a cursor it no longer has. Queued cleanup goes
|
||||
# stale routinely: the finalizer enqueues, then the
|
||||
# cursor gets closed properly before the drain runs.
|
||||
#
|
||||
# This is not a wire problem. ``_raise_sq_err``
|
||||
# self-drains the trailing SQ_EOT, so the wire is
|
||||
# still aligned and the remaining entries are still
|
||||
# worth sending.
|
||||
#
|
||||
# It must not escape, and it must not be treated as
|
||||
# fatal. Both were wrong before: a stale CLOSE draws
|
||||
# ``-267``, which is an OperationalError, which is in
|
||||
# WIRE_ERRORS — so a stale queue entry force-closed a
|
||||
# perfectly healthy connection. And this runs at the
|
||||
# start of somebody else's statement, so letting it
|
||||
# out would fail their good SQL with an error about a
|
||||
# cursor they never opened.
|
||||
_log.debug(
|
||||
"deferred cleanup rejected by server (stale entry), "
|
||||
"continuing: %r",
|
||||
exc,
|
||||
)
|
||||
continue
|
||||
if not isinstance(exc, WIRE_ERRORS):
|
||||
_log.warning(
|
||||
"unexpected error draining deferred cleanup: %r", exc
|
||||
)
|
||||
# No sqlcode means the failure came from the wire, not
|
||||
# the server: the socket died, or framing desynced and we
|
||||
# can no longer say where a response ends. Force-close.
|
||||
# Subsequent ``_send_pdu`` raises InterfaceError. The
|
||||
# server-side resources the remaining entries would have
|
||||
# freed are released when the session ends anyway.
|
||||
self._closed = True
|
||||
with contextlib.suppress(Exception):
|
||||
self._sock.close()
|
||||
@ -513,7 +757,7 @@ class Connection:
|
||||
# method but stable across versions; cheap (~50ns) and only
|
||||
# checks the current thread. If it ever changes shape, drop
|
||||
# this assert — the doc still names the precondition.
|
||||
assert self._wire_lock._is_owned(), (
|
||||
assert self._wire_lock.held_by_current_thread, (
|
||||
"_ensure_transaction called without _wire_lock held; "
|
||||
"the cursor method that called it must wrap its body in "
|
||||
"`with self._conn._wire_lock:`"
|
||||
@ -589,9 +833,9 @@ class Connection:
|
||||
# The 8-byte protocols mask is the JDBC reference value from
|
||||
# docs/CAPTURES/02-select-1.socat.log; we replay it verbatim
|
||||
# since the bits are opaque (server-recognized features).
|
||||
protocols_mask = bytes.fromhex("fffc7ffc3c8caa97")
|
||||
self._send_protocols(protocols_mask)
|
||||
self._drain_to_eot()
|
||||
self._send_protocols(CLIENT_PROTOCOLS_MASK)
|
||||
self._drain_to_eot() # captures the reply into self._server_protocols
|
||||
self._build_capabilities()
|
||||
|
||||
# Step 2: SQ_INFO with INFO_ENV subtype + session env vars.
|
||||
# The actual on-wire format (from JDBC's sendEnv at IfxSqli.java
|
||||
@ -626,6 +870,54 @@ class Connection:
|
||||
self._send_dbopen(self._database)
|
||||
self._drain_to_eot()
|
||||
|
||||
def _build_capabilities(self) -> None:
|
||||
"""Assemble ``ServerCapabilities`` from the login response and the
|
||||
SQ_PROTOCOLS reply, and warn about contradicted assumptions.
|
||||
|
||||
This is observation only — nothing in the parse paths branches on
|
||||
these bits yet. The value is diagnostic: this driver hardcodes
|
||||
several wire-framing choices that SQLI actually negotiates, and
|
||||
those choices are correct on every server we have measured
|
||||
(Informix 12.10, 14.10, 15). On some server we have not measured
|
||||
they might not be, and the failure mode for a framing mismatch is
|
||||
silently corrupted rows, which is about the worst way for a
|
||||
database driver to fail. A log warning naming the specific bit
|
||||
turns that into something diagnosable from a bug report.
|
||||
|
||||
Deliberately never raises: a server that declines to negotiate is
|
||||
not necessarily broken, and refusing the connection over a
|
||||
diagnostic would be a worse outcome than proceeding.
|
||||
"""
|
||||
if self._server_protocols is None:
|
||||
return
|
||||
conacc = self._conacc or {}
|
||||
try:
|
||||
caps = ServerCapabilities.from_wire(
|
||||
self._server_protocols,
|
||||
cap_1=conacc.get("cap_1", 0),
|
||||
cap_2=conacc.get("cap_2", 0),
|
||||
cap_3=conacc.get("cap_3", 0),
|
||||
server_version=conacc.get("server_version", ""),
|
||||
serial_number=conacc.get("serial_number", ""),
|
||||
applid_name=conacc.get("applid_name", ""),
|
||||
)
|
||||
except Exception:
|
||||
_log.debug("could not decode server capabilities", exc_info=True)
|
||||
return
|
||||
self._capabilities = caps
|
||||
|
||||
problems = caps.violated_assumptions()
|
||||
if problems:
|
||||
_log.warning(
|
||||
"Informix server %r negotiated capabilities that contradict "
|
||||
"this driver's hardcoded wire framing (mask=%s). Result rows "
|
||||
"may be decoded incorrectly. Please report this with the "
|
||||
"mask and your server version. Details: %s",
|
||||
caps.server_version or "<unknown version>",
|
||||
caps.raw_mask.hex(),
|
||||
"; ".join(problems),
|
||||
)
|
||||
|
||||
def _send_protocols(self, protocols: bytes) -> None:
|
||||
"""Emit a SQ_PROTOCOLS PDU per ``IfxSqli.sendProtocols``.
|
||||
|
||||
@ -668,9 +960,13 @@ class Connection:
|
||||
elif tag == MessageType.SQ_PROTOCOLS:
|
||||
# ``[short payloadLen][bytes payload][byte 0 if odd-len pad]``
|
||||
# Then the loop continues and consumes the next tag (usually SQ_EOT).
|
||||
# The payload is the negotiated 64-bit feature bitmap; stash
|
||||
# it for _capabilities decoding. Reading it was always
|
||||
# necessary for stream alignment — we just used to throw it
|
||||
# away instead of looking at it.
|
||||
payload_len = struct.unpack("!h", self._sock.read_exact(2))[0]
|
||||
if payload_len > 0:
|
||||
self._sock.read_exact(payload_len)
|
||||
self._server_protocols = self._sock.read_exact(payload_len)
|
||||
if payload_len & 1:
|
||||
self._sock.read_exact(1) # writePadded's even-alignment pad
|
||||
elif tag == MessageType.SQ_DONE:
|
||||
@ -700,7 +996,6 @@ class Connection:
|
||||
[short near_token_len][bytes name][optional pad][short SQ_EOT]
|
||||
"""
|
||||
from . import _errcodes
|
||||
from ._protocol import ProtocolError
|
||||
|
||||
sqlcode = struct.unpack("!h", self._sock.read_exact(2))[0]
|
||||
isamcode = struct.unpack("!h", self._sock.read_exact(2))[0]
|
||||
@ -716,7 +1011,7 @@ class Connection:
|
||||
if name_len & 1:
|
||||
self._sock.read_exact(1)
|
||||
near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace")
|
||||
except (ProtocolError, OSError):
|
||||
except WIRE_ERRORS:
|
||||
pass
|
||||
# Phase 28: drain failure means wire desync — force-close so
|
||||
# subsequent operations don't inherit the broken state.
|
||||
@ -730,7 +1025,7 @@ class Connection:
|
||||
next_tag = struct.unpack("!h", self._sock.read_exact(2))[0]
|
||||
if next_tag == MessageType.SQ_EOT:
|
||||
break
|
||||
except (ProtocolError, OSError, OperationalError):
|
||||
except WIRE_ERRORS:
|
||||
self._closed = True
|
||||
with contextlib.suppress(Exception):
|
||||
self._sock.close()
|
||||
@ -883,6 +1178,72 @@ class Connection:
|
||||
|
||||
# -- response parsing -------------------------------------------------
|
||||
|
||||
@property
|
||||
def server_capabilities(self):
|
||||
"""The negotiated ``SQ_PROTOCOLS`` feature bitmap, or ``None``.
|
||||
|
||||
``None`` means the negotiation reply couldn't be decoded — the
|
||||
connection still works, we just have no capability information.
|
||||
See ``informix_db._capabilities.ServerCapabilities``.
|
||||
"""
|
||||
return self._capabilities
|
||||
|
||||
@property
|
||||
def server_version_internal(self) -> str:
|
||||
"""The version string carried in the login response.
|
||||
|
||||
This is Informix's *internal* protocol version, not the release
|
||||
you installed: 12.10 reports ``9.56``, 14.10 reports ``9.59``,
|
||||
and 15 reports ``15.0.1.0.3``. At the protocol level the two
|
||||
older releases really are 9.x servers, which is why all three
|
||||
speak an identical SQLI dialect.
|
||||
|
||||
Free — it arrives with the login response. For the release
|
||||
number, use :attr:`server_version`.
|
||||
"""
|
||||
return self._conacc.get("server_version", "") if self._conacc else ""
|
||||
|
||||
@property
|
||||
def server_version(self) -> str:
|
||||
"""The server's release version, e.g.
|
||||
``'IBM Informix Dynamic Server Version 12.10.FC6'``.
|
||||
|
||||
**Performs one query on first access**, then caches the result
|
||||
for the life of the connection. The login response only carries
|
||||
the internal protocol version (see
|
||||
:attr:`server_version_internal`), which reads as a wrong answer
|
||||
— a 12.10 server announces itself as 9.56 — so the release
|
||||
number has to be asked for with ``DBINFO('version','full')``.
|
||||
|
||||
Falls back to the internal string if the query fails, which it
|
||||
can when no database is open. Never raises: a version lookup
|
||||
should not be able to break a working connection.
|
||||
"""
|
||||
if self._server_version_full is None:
|
||||
self._server_version_full = self._query_server_version()
|
||||
return self._server_version_full or self.server_version_internal
|
||||
|
||||
def _query_server_version(self) -> str:
|
||||
"""One-shot ``DBINFO`` lookup for the release version.
|
||||
|
||||
Deliberately not done during connect: it costs a round-trip that
|
||||
most callers never need, and connect latency is on the hot path
|
||||
for pooled workloads.
|
||||
"""
|
||||
try:
|
||||
cur = self.cursor()
|
||||
try:
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 DBINFO('version','full') FROM systables"
|
||||
)
|
||||
row = cur.fetchone()
|
||||
finally:
|
||||
cur.close()
|
||||
except Exception:
|
||||
_log.debug("could not query server version", exc_info=True)
|
||||
return ""
|
||||
return str(row[0]).strip() if row and row[0] else ""
|
||||
|
||||
def _parse_login_response(self) -> None:
|
||||
"""Read and parse the server's login response.
|
||||
|
||||
@ -916,8 +1277,12 @@ class Connection:
|
||||
)
|
||||
elif sl_type != SLHeader.SLTYPE_CONACC:
|
||||
raise ProtocolError(f"unknown SLType in login response: {sl_type}")
|
||||
# SLTYPE_CONACC — connection accepted. We don't (yet) decode the
|
||||
# full server-side metadata. Phase 1 just needs to know "we got in".
|
||||
|
||||
# SLTYPE_CONACC — connection accepted. Decode the server metadata
|
||||
# block for the capability fields. Best-effort: a decode failure
|
||||
# here must not fail an otherwise-good connection, since every
|
||||
# server we've tested connects fine without any of this.
|
||||
self._conacc = _decode_conacc(rest)
|
||||
|
||||
def _raise_from_rejection(self, reader: IfxStreamReader) -> None:
|
||||
"""Best-effort decode of the connection-rejection error block.
|
||||
|
||||
@ -204,8 +204,25 @@ def _decode_float(raw: bytes) -> float | None:
|
||||
return _UNPACK_DOUBLE(raw)[0]
|
||||
|
||||
|
||||
def _decode_char(raw: bytes, encoding: str = "iso-8859-1") -> str:
|
||||
"""Strip trailing spaces (CHAR is space-padded to declared length)."""
|
||||
def _decode_char(raw: bytes, encoding: str = "iso-8859-1") -> str | None:
|
||||
"""Decode CHAR / NCHAR: fixed width, space-padded to the declared length.
|
||||
|
||||
A leading ``0x00`` is Informix's NULL marker for these types. Without
|
||||
that check a NULL CHAR came back as ``''``, indistinguishable from a
|
||||
genuinely empty one — and ``WHERE c IS NULL`` disagreeing with what
|
||||
the driver hands you is a nasty thing to debug. Wire evidence for
|
||||
``CHAR(6)``::
|
||||
|
||||
'ab' -> 61 62 20 20 20 20
|
||||
'' -> 20 20 20 20 20 20 all spaces
|
||||
NULL -> 00 20 20 20 20 20 leading nul
|
||||
|
||||
The two are distinguishable, so we distinguish them. A real value
|
||||
cannot begin with a nul: character data has no use for one, and the
|
||||
server reserves it precisely as this marker.
|
||||
"""
|
||||
if raw[:1] == b"\x00":
|
||||
return None
|
||||
return raw.rstrip(b" \x00").decode(encoding)
|
||||
|
||||
|
||||
@ -227,6 +244,41 @@ def _decode_bool(raw: bytes) -> bool:
|
||||
return raw[0] in (ord("t"), ord("T"), 1)
|
||||
|
||||
|
||||
def _decode_int8(raw: bytes) -> int | None:
|
||||
"""INT8 / SERIAL8 — the *legacy* 64-bit integer, 10 bytes on the wire.
|
||||
|
||||
NOT the same as BIGINT (52) / BIGSERIAL (53), which are plain 8-byte
|
||||
big-endian. The INT8 layout mirrors ``ifx_int8_t`` and splits the
|
||||
magnitude across two 32-bit halves in the opposite order you'd guess:
|
||||
|
||||
bytes 0-1 sign word: 0 = NULL, 1 = positive, -1 (0xFFFF) = negative
|
||||
bytes 2-5 LOW 32 bits (unsigned, big-endian)
|
||||
bytes 6-9 HIGH 32 bits (unsigned, big-endian)
|
||||
|
||||
Verified against Informix 12.10.FC12W1DE and 15.0.1.0.3DE, which emit
|
||||
byte-identical encodings::
|
||||
|
||||
123456789012 -> 00 01 | be 99 1a 14 | 00 00 00 1c
|
||||
-123456789012 -> ff ff | be 99 1a 14 | 00 00 00 1c
|
||||
42 -> 00 01 | 00 00 00 2a | 00 00 00 00
|
||||
NULL -> 00 00 | 00 00 00 00 | 00 00 00 00
|
||||
|
||||
Note the magnitude bytes are identical for +n and -n — the sign lives
|
||||
entirely in the leading word, so this is sign-magnitude, not two's
|
||||
complement. Decoding it as a signed 64-bit integer gives the wrong
|
||||
answer for every negative value.
|
||||
"""
|
||||
if len(raw) < 10:
|
||||
raise ValueError(f"INT8 payload too short: {len(raw)} bytes, need 10")
|
||||
sign = _UNPACK_SHORT(raw[0:2])[0]
|
||||
if sign == 0:
|
||||
return None
|
||||
low = int.from_bytes(raw[2:6], "big", signed=False)
|
||||
high = int.from_bytes(raw[6:10], "big", signed=False)
|
||||
value = (high << 32) | low
|
||||
return -value if sign < 0 else value
|
||||
|
||||
|
||||
def _decode_date(raw: bytes) -> datetime.date | None:
|
||||
"""4-byte big-endian signed int = day count from 1899-12-31. NULL = 0x80000000."""
|
||||
days = _UNPACK_INT(raw)[0]
|
||||
@ -534,6 +586,13 @@ FIXED_WIDTHS: dict[int, int] = {
|
||||
IfxType.BIGSERIAL: 8,
|
||||
IfxType.DATE: 4,
|
||||
IfxType.BOOL: 1,
|
||||
# INT8/SERIAL8 are fixed-width at 10 bytes — NOT 8. See _decode_int8.
|
||||
# Omitting these was a silent-corruption bug: they fell through to the
|
||||
# unknown-type path, which surfaces ``encoded_length`` raw bytes.
|
||||
# ``encoded_length`` happens to be 10 for INT8, so the stream stayed
|
||||
# aligned and the only symptom was a bytes object where an int belonged.
|
||||
IfxType.INT8: 10,
|
||||
IfxType.SERIAL8: 10,
|
||||
}
|
||||
|
||||
|
||||
@ -557,6 +616,8 @@ DECODERS: dict[int, DecoderFn] = {
|
||||
IfxType.SERIAL: _decode_int,
|
||||
IfxType.BIGINT: _decode_bigint,
|
||||
IfxType.BIGSERIAL: _decode_bigint,
|
||||
IfxType.INT8: _decode_int8,
|
||||
IfxType.SERIAL8: _decode_int8,
|
||||
IfxType.SMFLOAT: _decode_smfloat,
|
||||
IfxType.FLOAT: _decode_float,
|
||||
IfxType.CHAR: _decode_char,
|
||||
@ -700,8 +761,28 @@ def _encode_float(value: float) -> EncodedParam:
|
||||
|
||||
|
||||
def _encode_bool(value: bool) -> EncodedParam:
|
||||
"""Encode a Python bool as Informix BOOLEAN (type=45, 1 byte)."""
|
||||
return (45, 0, b"\x01" if value else b"\x00")
|
||||
"""Encode a Python bool by binding the literal ``'t'`` / ``'f'``.
|
||||
|
||||
Informix BOOLEAN has no bindable binary form we could find. This used
|
||||
to send ``(45, 0, b"\\x01")``, and type 45 is not something the server
|
||||
accepts as a bind type — it simply stopped responding, so any
|
||||
``execute`` with a bool parameter **hung until the read timeout**, or
|
||||
forever if none was set. Type 41 (which is how BOOLEAN is *described*
|
||||
in results) hangs the same way; the descriptor type and the bind type
|
||||
are not interchangeable.
|
||||
|
||||
Sending ``'t'`` / ``'f'`` as CHAR and letting the server cast on
|
||||
assignment is what works, and it mirrors Informix's own literal
|
||||
syntax (``INSERT ... VALUES ('t')``). It also matches how
|
||||
:func:`_encode_str` already leans on server-side conversion rather
|
||||
than trying to match the destination column's exact type.
|
||||
|
||||
Note the asymmetry with the read path, which is correct and stays
|
||||
as it is: BOOLEAN comes *back* as UDTFIXED (41) wrapped in a UDT
|
||||
envelope whose payload byte is ``0x74``/``0x66`` — ASCII ``t``/``f``,
|
||||
the same characters we send here.
|
||||
"""
|
||||
return _encode_str("t" if value else "f")
|
||||
|
||||
|
||||
def _encode_date(value: datetime.date) -> EncodedParam:
|
||||
@ -717,9 +798,22 @@ def _encode_date(value: datetime.date) -> EncodedParam:
|
||||
def _encode_datetime(value: datetime.datetime) -> EncodedParam:
|
||||
"""Encode a Python ``datetime.datetime`` as Informix DATETIME (type=10).
|
||||
|
||||
Emit YEAR TO SECOND form — covers the common case of stored
|
||||
timestamps without microseconds. (Phase 6.x can add YEAR TO
|
||||
FRACTION(N) variants if microseconds are needed.)
|
||||
Emits YEAR TO SECOND when ``microsecond`` is zero, and YEAR TO
|
||||
FRACTION(5) when it isn't.
|
||||
|
||||
The conditional matters. Emitting YEAR TO SECOND unconditionally
|
||||
silently discarded sub-second precision on every bind: a
|
||||
``DATETIME YEAR TO FRACTION(5)`` column handed
|
||||
``datetime(..., microsecond=120000)`` stored ``.00000``, with no
|
||||
error. Only widening when there is a fraction to carry keeps the
|
||||
long-exercised YEAR TO SECOND path byte-identical for the common
|
||||
case, and Informix converts between qualifiers on assignment, so a
|
||||
FRACTION(5) bind into a narrower column truncates server-side
|
||||
rather than failing.
|
||||
|
||||
FRACTION(5) is the widest Informix supports and holds 10 µs
|
||||
resolution; Python's microsecond field is finer, so the last digit
|
||||
is dropped. Same trade-off ``_encode_timedelta`` already makes.
|
||||
|
||||
Format (per ``Decimal.javaToIfx`` line 457):
|
||||
byte[0..1] = short total length of data following (= digit_count/2 + 1)
|
||||
@ -743,10 +837,20 @@ def _encode_datetime(value: datetime.datetime) -> EncodedParam:
|
||||
(value.second, 2),
|
||||
]
|
||||
digit_str = "".join(f"{v:0{w}d}" for v, w in fields) # 14 digits
|
||||
if value.microsecond:
|
||||
# 6 fraction digits = exactly 3 more BCD pairs, so the digit
|
||||
# string stays even and the exponent byte is unchanged (the
|
||||
# integer part is still 7 base-100 pairs). FRACTION(5) carries
|
||||
# 5 significant digits; the 6th is padding the wire format
|
||||
# requires. Qualifier: digit_count=19, start=YEAR(0), end=
|
||||
# FRACTION(5)=15 — matching what the decoder reads back.
|
||||
digit_str += f"{value.microsecond:06d}" # -> 20 digits
|
||||
prec = (19 << 8) | (0 << 4) | 15
|
||||
else:
|
||||
prec = (14 << 8) | (0 << 4) | 10
|
||||
digit_bytes = bytes(int(digit_str[i : i + 2]) for i in range(0, len(digit_str), 2))
|
||||
inner = bytes([0xC7]) + digit_bytes # 8 bytes (1 exp + 7 BCD pairs)
|
||||
raw = len(inner).to_bytes(2, "big") + inner # +2 byte length prefix = 10 bytes
|
||||
prec = (14 << 8) | (0 << 4) | 10
|
||||
inner = bytes([0xC7]) + digit_bytes # 1 exp byte + 7 or 10 BCD pairs
|
||||
raw = len(inner).to_bytes(2, "big") + inner # +2 byte length prefix
|
||||
return (10, prec, raw)
|
||||
|
||||
|
||||
|
||||
@ -22,6 +22,7 @@ from __future__ import annotations
|
||||
import contextlib
|
||||
import itertools
|
||||
import logging
|
||||
import os
|
||||
import struct
|
||||
import weakref
|
||||
from collections.abc import Iterator
|
||||
@ -29,8 +30,19 @@ from typing import TYPE_CHECKING, Any
|
||||
|
||||
from . import _errcodes
|
||||
from ._messages import MessageType
|
||||
from ._protocol import IfxStreamReader, make_pdu_writer
|
||||
from ._resultset import ColumnInfo, parse_describe, parse_tuple_payload
|
||||
from ._protocol import (
|
||||
WIRE_ERRORS,
|
||||
BufferedSocketReader,
|
||||
IfxStreamReader,
|
||||
make_pdu_writer,
|
||||
)
|
||||
from ._resultset import (
|
||||
ColumnInfo,
|
||||
compile_column_readers,
|
||||
compile_row_decoder,
|
||||
parse_describe,
|
||||
parse_tuple_payload,
|
||||
)
|
||||
from .converters import encode_param
|
||||
from .exceptions import (
|
||||
DatabaseError,
|
||||
@ -38,6 +50,7 @@ from .exceptions import (
|
||||
NotSupportedError,
|
||||
ProgrammingError,
|
||||
)
|
||||
from .rows import Row, make_row_class
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .connections import Connection
|
||||
@ -48,7 +61,6 @@ if TYPE_CHECKING:
|
||||
_cursor_counter = itertools.count(1)
|
||||
|
||||
|
||||
_NUMERIC_PLACEHOLDER_RE = __import__("re").compile(r":(\d+)")
|
||||
|
||||
|
||||
# Phase 28: pre-built CLOSE and RELEASE PDU bytes for cursor finalizers.
|
||||
@ -71,6 +83,67 @@ del _build_static_pdu
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Phase 39: buffered socket reader is the default. The reader holds a
|
||||
# connection-scoped bytearray + offset cursor on ``IfxSocket``, so one
|
||||
# ``recv()`` per ~64 KB amortizes over hundreds of fields rather than
|
||||
# the per-field-recv pattern the legacy ``_SocketReader`` uses.
|
||||
#
|
||||
# Set ``IFX_BUFFERED_READER=0`` to fall back to the legacy reader if
|
||||
# you hit a wire shape we haven't covered. The legacy path is kept as
|
||||
# the escape hatch (and as the A/B baseline if we ever need to bisect
|
||||
# a regression). Read once at module import so the check isn't on the
|
||||
# hot path.
|
||||
_USE_BUFFERED_READER = os.environ.get("IFX_BUFFERED_READER", "1") != "0"
|
||||
|
||||
|
||||
def _make_socket_reader(sock):
|
||||
if _USE_BUFFERED_READER:
|
||||
return BufferedSocketReader(sock)
|
||||
return _SocketReader(sock)
|
||||
|
||||
|
||||
# Statement types from the DESCRIBE response's first field. The server
|
||||
# tells us exactly what it prepared; these are the two values that mean
|
||||
# "this will produce rows".
|
||||
_ST_SELECT = 2
|
||||
_ST_ROUTINE = 56 # EXECUTE PROCEDURE / EXECUTE FUNCTION
|
||||
|
||||
# Transaction control run as SQL. JDBC reads the same three values off
|
||||
# the describe and calls setTxBeginState / setTxEndState (IfxSqli, the
|
||||
# TxStmt field). Both ``BEGIN`` and ``BEGIN WORK`` report 34; likewise
|
||||
# the WORK-less spellings of the other two.
|
||||
_ST_TX_BEGIN = 34
|
||||
_ST_TX_COMMIT = 35
|
||||
_ST_TX_ROLLBACK = 36
|
||||
_TX_CONTROL_TYPES = frozenset({_ST_TX_BEGIN, _ST_TX_COMMIT, _ST_TX_ROLLBACK})
|
||||
|
||||
|
||||
def _produces_result_set(statement_type: int, ncolumns: int) -> bool:
|
||||
"""Whether a prepared statement needs a cursor opened for it.
|
||||
|
||||
This is JDBC's ``IfxSqli.isResultSet`` predicate, and it replaces a
|
||||
first-word check for ``SELECT`` that got five ordinary forms wrong.
|
||||
A leading comment of any of the three Informix flavours, a
|
||||
parenthesized select, a parenthesized UNION, and a CTE were all
|
||||
classified as DML, so the driver sent SQ_EXECUTE where the server
|
||||
expected a cursor. The report back was ``-260 Cursor name already in
|
||||
use``, which describes neither the cause nor anything the caller did.
|
||||
|
||||
The original comment justified the heuristic on the grounds that
|
||||
``nfields`` can't distinguish these, because ``INSERT INTO t VALUES
|
||||
(?)`` also describes a column. True, and irrelevant: ``statement_type``
|
||||
distinguishes them exactly. That INSERT reports 6 with one field; every
|
||||
SELECT form above reports 2. The value was being parsed into the
|
||||
describe metadata and thrown away.
|
||||
|
||||
``EXECUTE PROCEDURE``/``FUNCTION`` (56) is the one type that depends on
|
||||
the column count, since a routine may or may not return rows.
|
||||
"""
|
||||
if statement_type == _ST_SELECT:
|
||||
return True
|
||||
return statement_type == _ST_ROUTINE and ncolumns > 0
|
||||
|
||||
|
||||
def _finalize_cursor(
|
||||
conn_ref: weakref.ReferenceType,
|
||||
state: list,
|
||||
@ -94,13 +167,30 @@ def _finalize_cursor(
|
||||
a list (not the cursor object itself) keeps the finalizer's closure
|
||||
weak — the cursor remains GC'd-able.
|
||||
"""
|
||||
from ._protocol import ProtocolError
|
||||
|
||||
if not state[0]:
|
||||
return # nothing to release
|
||||
conn = conn_ref()
|
||||
if conn is None or conn.closed:
|
||||
return
|
||||
if conn._wire_lock.held_by_current_thread:
|
||||
# GC fired on the very thread that is mid-statement. The
|
||||
# non-blocking acquire below would *succeed* here — an RLock
|
||||
# grants a reentrant acquire to its own owner — and we would
|
||||
# send CLOSE/RELEASE into the middle of that statement, killing
|
||||
# it with -208. Defer instead, exactly as for another thread.
|
||||
#
|
||||
# Refcounting hides this: a dropped cursor is freed at the drop,
|
||||
# before the next statement. A cursor caught in a reference cycle
|
||||
# waits for a collection instead, and cycles are ordinary — any
|
||||
# traceback holding a cursor makes one.
|
||||
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
|
||||
_log.debug(
|
||||
"cursor finalizer: GC ran on the thread holding the wire lock; "
|
||||
"enqueued CLOSE+RELEASE for deferred cleanup on conn %s",
|
||||
id(conn),
|
||||
)
|
||||
return
|
||||
if not conn._wire_lock.acquire(blocking=False):
|
||||
# Another thread is mid-operation on this connection. Don't
|
||||
# deadlock; instead, hand the cleanup bytes to the connection's
|
||||
@ -121,19 +211,33 @@ def _finalize_cursor(
|
||||
conn._drain_to_eot()
|
||||
conn._send_pdu(_RELEASE_PDU)
|
||||
conn._drain_to_eot()
|
||||
except (ProtocolError, OSError) as exc:
|
||||
# Wire desync during cleanup — same doctrine as
|
||||
# ``_raise_sq_err``: the wire is unrecoverable, force-close
|
||||
# the connection. Asymmetric handling of the same failure
|
||||
# mode would be a Hamilton smell.
|
||||
_log.warning(
|
||||
"cursor finalizer: wire desync during cleanup; "
|
||||
"force-closing connection: %r",
|
||||
exc,
|
||||
)
|
||||
conn._closed = True
|
||||
with contextlib.suppress(Exception):
|
||||
conn._sock.close()
|
||||
except WIRE_ERRORS as exc:
|
||||
if getattr(exc, "sqlcode", None) is not None:
|
||||
# The *server* rejected the cleanup — typically a stale
|
||||
# CLOSE for a cursor it no longer has, which answers
|
||||
# -267 "the transaction has been rolled back". That is
|
||||
# an OperationalError, which is in WIRE_ERRORS, so this
|
||||
# branch used to force-close a perfectly healthy
|
||||
# connection over a no-op. ``_raise_sq_err`` self-drains
|
||||
# the trailing SQ_EOT, so the wire is still aligned.
|
||||
# Same distinction as _drain_pending_cleanup.
|
||||
_log.debug(
|
||||
"cursor finalizer: server rejected cleanup (stale): %r",
|
||||
exc,
|
||||
)
|
||||
else:
|
||||
# No sqlcode: the socket died or framing desynced and we
|
||||
# can no longer say where a response ends. Force-close —
|
||||
# same doctrine as ``_raise_sq_err``. Asymmetric handling
|
||||
# of the same failure mode would be a Hamilton smell.
|
||||
_log.warning(
|
||||
"cursor finalizer: wire desync during cleanup; "
|
||||
"force-closing connection: %r",
|
||||
exc,
|
||||
)
|
||||
conn._closed = True
|
||||
with contextlib.suppress(Exception):
|
||||
conn._sock.close()
|
||||
except InterfaceError:
|
||||
# Connection was closed by another thread between our
|
||||
# ``conn.closed`` check above and the actual write. No-op:
|
||||
@ -143,17 +247,103 @@ def _finalize_cursor(
|
||||
conn._wire_lock.release()
|
||||
|
||||
|
||||
_ASCII_DIGITS = frozenset("0123456789")
|
||||
|
||||
|
||||
def _rewrite_numeric_to_qmark(sql: str) -> str:
|
||||
"""Convert ``:1`` / ``:2`` placeholders (paramstyle="numeric") to ``?``.
|
||||
|
||||
Informix's wire protocol uses ``?`` natively. Since we expose
|
||||
``paramstyle="numeric"`` in the public API (matches Informix
|
||||
ESQL/C convention), we rewrite before sending. Trivial cases only
|
||||
— strings and comments are NOT escaped, so SQL containing literal
|
||||
``:1`` inside string literals will be wrongly substituted. Phase 5
|
||||
can add a proper SQL tokenizer.
|
||||
Informix's wire protocol uses ``?`` natively, and we advertise
|
||||
``paramstyle="numeric"`` to match the ESQL/C convention, so the
|
||||
placeholders are rewritten on the way out.
|
||||
|
||||
This used to be ``re.sub(r":(\\d+)", "?", sql)``, which cannot see a
|
||||
string literal and so rewrote the contents of one. Any ``HH:MM``
|
||||
time, any URL with a port, any aspect ratio::
|
||||
|
||||
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
|
||||
stored as 'http://host?/x'
|
||||
|
||||
Nothing about that is opt-in. The rewrite runs whenever a statement
|
||||
has parameters, whatever placeholder style the caller actually used,
|
||||
so writing ``?`` everywhere and never touching numeric style did not
|
||||
protect you. It also changed the placeholder *count* while
|
||||
``num_qmarks`` was still computed from ``len(params)``, leaving the
|
||||
driver and the server disagreeing about how many binds exist.
|
||||
|
||||
So: a single pass that substitutes only outside quotes and comments.
|
||||
The lexical rules are Informix's own, measured against 12.10, 14.10
|
||||
and 15 rather than assumed from standard SQL:
|
||||
|
||||
* ``''`` doubling escapes a quote inside ``'...'``. A backslash does
|
||||
**not** escape anything; ``'a\'b'`` is an unterminated string and
|
||||
the server answers ``-282``. A Postgres-style scanner that honours
|
||||
``\'`` would desync here and corrupt everything after it.
|
||||
* ``"..."`` is a delimited identifier or a string depending on
|
||||
``DELIMIDENT``. Either way its contents are not ours to touch.
|
||||
* ``--`` runs to end of line, ``/* */`` does **not** nest (the first
|
||||
``*/`` closes it; nesting is a syntax error), and ``{ }`` is a
|
||||
comment too.
|
||||
* ``::`` is the cast operator and is stepped over as a unit, so it
|
||||
can never be read as the start of a placeholder.
|
||||
|
||||
An unterminated quote or comment consumes the rest of the string and
|
||||
substitutes nothing further. That is deliberate: under-substituting
|
||||
leaves the server to reject SQL that was already malformed, while
|
||||
guessing would corrupt a literal.
|
||||
"""
|
||||
return _NUMERIC_PLACEHOLDER_RE.sub("?", sql)
|
||||
if ":" not in sql:
|
||||
return sql
|
||||
out: list[str] = []
|
||||
i = 0
|
||||
n = len(sql)
|
||||
while i < n:
|
||||
ch = sql[i]
|
||||
if ch in ("'", '"'):
|
||||
j = i + 1
|
||||
while j < n:
|
||||
if sql[j] == ch:
|
||||
if j + 1 < n and sql[j + 1] == ch:
|
||||
j += 2 # doubled quote, still inside
|
||||
continue
|
||||
j += 1
|
||||
break
|
||||
j += 1
|
||||
out.append(sql[i:j])
|
||||
i = j
|
||||
elif ch == "-" and sql.startswith("--", i):
|
||||
j = sql.find("\n", i)
|
||||
j = n if j == -1 else j
|
||||
out.append(sql[i:j])
|
||||
i = j
|
||||
elif ch == "/" and sql.startswith("/*", i):
|
||||
j = sql.find("*/", i + 2)
|
||||
j = n if j == -1 else j + 2
|
||||
out.append(sql[i:j])
|
||||
i = j
|
||||
elif ch == "{":
|
||||
j = sql.find("}", i)
|
||||
j = n if j == -1 else j + 1
|
||||
out.append(sql[i:j])
|
||||
i = j
|
||||
elif ch == ":":
|
||||
if sql.startswith("::", i):
|
||||
out.append("::")
|
||||
i += 2
|
||||
continue
|
||||
j = i + 1
|
||||
while j < n and sql[j] in _ASCII_DIGITS:
|
||||
j += 1
|
||||
if j > i + 1:
|
||||
out.append("?")
|
||||
i = j
|
||||
else:
|
||||
out.append(ch)
|
||||
i += 1
|
||||
else:
|
||||
out.append(ch)
|
||||
i += 1
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def _generate_cursor_name() -> str:
|
||||
@ -184,8 +374,13 @@ class Cursor:
|
||||
# manipulation. Two-mode cursor; the same surface API works
|
||||
# for both.
|
||||
self._scrollable = scrollable
|
||||
# Inherited from the connection, overridable per cursor. See
|
||||
# informix_db.rows for what this costs and why it is opt-in.
|
||||
self.row_factory = connection.row_factory
|
||||
self._description: list[tuple] | None = None
|
||||
self._columns: list[ColumnInfo] = []
|
||||
self._column_readers: list[tuple] | None = None # Phase 37
|
||||
self._row_decoder = None # Phase 38 codegen'd row decoder
|
||||
self._rowcount: int = -1
|
||||
self._rows: list[tuple] = []
|
||||
# Phase 17: index-based row access enables scroll cursors. The
|
||||
@ -219,6 +414,11 @@ class Cursor:
|
||||
# from. Empirically the server accepts 0 here even when a real
|
||||
# ID was assigned, so this is best-effort tracking.
|
||||
self._statement_id: int = 0
|
||||
# DESCRIBE's statement-type field. Decides whether a cursor is
|
||||
# opened -- see _produces_result_set.
|
||||
self._statement_type: int = 0
|
||||
# Per-result-set row class when row_factory is set, else None.
|
||||
self._row_class: type | None = None
|
||||
# Phase 10: smart-LOB read via ``lotofile(col, path, 'client')``.
|
||||
# The server orchestrates a SQ_FILE (98) protocol where it tells
|
||||
# us to "open file X, write these bytes, close". We emulate the
|
||||
@ -303,33 +503,54 @@ class Cursor:
|
||||
|
||||
def _execute_under_wire_lock(self, sql: str, params: tuple) -> None:
|
||||
"""Wire-bound body of ``execute``. Caller MUST hold ``_wire_lock``."""
|
||||
# Someone else's scrollable cursor owns the session's statement
|
||||
# slot -- refuse rather than draw a -285 and destroy their cursor.
|
||||
self._conn._check_scroll_cursor_conflict(self)
|
||||
# Our own previous scrollable cursor is still open server-side.
|
||||
# Re-executing over the top of it collides exactly the same way,
|
||||
# so close it first. This is the one caller allowed past the
|
||||
# check above, and it is only safe because of this.
|
||||
if self._scrollable and self._server_cursor_open:
|
||||
self._close_server_cursor()
|
||||
# Reset previous-execute state.
|
||||
self._description = None
|
||||
self._columns = []
|
||||
self._column_readers = None # Phase 37
|
||||
self._row_decoder = None # Phase 38
|
||||
self._rowcount = -1
|
||||
self._rows = []
|
||||
self._row_index = -1 # before-first-row
|
||||
self._row_class = None
|
||||
self._statement_type = 0
|
||||
self._statement_already_done = False
|
||||
|
||||
# Step 1: PREPARE — send SQL with numQmarks = len(params).
|
||||
# statement_boundary: nothing is open server-side yet, so this is
|
||||
# the one safe moment to flush a finalizer's deferred cleanup.
|
||||
self._conn._send_pdu(
|
||||
self._build_prepare_pdu(sql, num_qmarks=len(params)),
|
||||
statement_boundary=True,
|
||||
)
|
||||
self._read_describe_response()
|
||||
|
||||
# On a logged DB in non-autocommit mode, the server requires an
|
||||
# explicit SQ_BEGIN before the first DML in each transaction.
|
||||
# _ensure_transaction is a no-op for autocommit / unlogged DBs,
|
||||
# and idempotent within an open transaction.
|
||||
self._conn._ensure_transaction()
|
||||
#
|
||||
# This runs *after* the describe, matching JDBC's
|
||||
# initiateTransaction placement, because until the describe lands
|
||||
# we don't know whether the caller's statement is itself
|
||||
# transaction control. Opening a transaction on their behalf and
|
||||
# then executing their BEGIN WORK gets -535, "already in
|
||||
# transaction" — the driver competing with the user for the same
|
||||
# job and the user losing.
|
||||
if self._statement_type not in _TX_CONTROL_TYPES:
|
||||
self._conn._ensure_transaction()
|
||||
|
||||
# Step 1: PREPARE — send SQL with numQmarks = len(params).
|
||||
self._conn._send_pdu(self._build_prepare_pdu(sql, num_qmarks=len(params)))
|
||||
self._read_describe_response()
|
||||
|
||||
# Branch on the SQL keyword. We can't use ``self._columns`` /
|
||||
# ``nfields`` here because a parameterized INSERT also returns
|
||||
# a non-empty DESCRIBE (server describes the would-be inserted
|
||||
# row's columns). The SQL-keyword heuristic is what JDBC effectively
|
||||
# does too via its IfxStatement / IfxPreparedStatement subclassing.
|
||||
first_word = sql.lstrip().split(None, 1)[0].upper() if sql.strip() else ""
|
||||
is_select = first_word == "SELECT"
|
||||
|
||||
if is_select:
|
||||
# Ask the server what it just prepared, rather than guessing from
|
||||
# the first word of the SQL.
|
||||
if _produces_result_set(self._statement_type, len(self._columns)):
|
||||
if params:
|
||||
self._execute_select_with_params(params)
|
||||
else:
|
||||
@ -339,6 +560,13 @@ class Cursor:
|
||||
else:
|
||||
self._execute_dml()
|
||||
|
||||
self._row_class = self._resolve_row_class()
|
||||
|
||||
# The statement succeeded. If it was transaction control, the
|
||||
# server's transaction state just changed and the connection has
|
||||
# to know, or commit() and rollback() silently do nothing.
|
||||
self._note_transaction_control()
|
||||
|
||||
# SELECT path: position cursor before the first row so the next
|
||||
# ``fetchone()`` returns ``rows[0]``. DML paths leave _row_index
|
||||
# at -1 too (no rows to iterate).
|
||||
@ -347,6 +575,98 @@ class Cursor:
|
||||
if self._description is not None:
|
||||
self._row_index = -1
|
||||
|
||||
def _resolve_row_class(self) -> type | None:
|
||||
"""Pick the class this result set's rows are handed back as.
|
||||
|
||||
``row_factory`` is either :class:`informix_db.Row` (or a subclass),
|
||||
in which case the per-shape class is built and cached from the
|
||||
column names, or any callable taking the name tuple and returning
|
||||
something that takes a values tuple.
|
||||
|
||||
Returns ``None`` when no factory is set, which is the default and
|
||||
keeps plain tuples on the hot path at zero cost.
|
||||
"""
|
||||
factory = self.row_factory
|
||||
if factory is None or self._description is None:
|
||||
return None
|
||||
names = tuple(d[0] for d in self._description)
|
||||
if isinstance(factory, type) and issubclass(factory, Row):
|
||||
return make_row_class(names)
|
||||
return factory(names)
|
||||
|
||||
def _note_transaction_control(self) -> None:
|
||||
"""Sync the connection's transaction flag after a successful execute.
|
||||
|
||||
``BEGIN WORK`` run through ``execute()`` opens a real transaction
|
||||
on the server, and the connection had no idea. With autocommit on
|
||||
nothing stopped it reaching the server, so ``_in_transaction``
|
||||
stayed False while a transaction was open — and both ``commit()``
|
||||
and ``rollback()`` are guarded by that flag. ``rollback()``
|
||||
returned successfully having sent nothing, and the rows it was
|
||||
asked to discard were still there.
|
||||
|
||||
The pool reads the same flag to decide whether a returned
|
||||
connection needs cleaning up, so the connection went back into
|
||||
circulation holding an open transaction and its locks.
|
||||
|
||||
Called only on the success path: a statement that failed did not
|
||||
change the server's transaction state.
|
||||
"""
|
||||
statement_type = self._statement_type
|
||||
if statement_type == _ST_TX_BEGIN:
|
||||
self._conn._in_transaction = True
|
||||
elif statement_type in (_ST_TX_COMMIT, _ST_TX_ROLLBACK):
|
||||
self._conn._in_transaction = False
|
||||
|
||||
def _close_server_cursor(self) -> None:
|
||||
"""Free the server-side scrollable cursor. Caller MUST hold ``_wire_lock``.
|
||||
|
||||
Best-effort: a wire failure here is swallowed. Both callers are
|
||||
already past the point where reporting it would help — one is
|
||||
closing the cursor, the other is about to run a new statement
|
||||
that will report its own failure if the wire is really gone.
|
||||
"""
|
||||
try:
|
||||
self._conn._send_pdu(self._build_close_pdu())
|
||||
self._drain_to_eot()
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
pass
|
||||
self._server_cursor_open = False
|
||||
self._conn._open_scroll_cursor = None
|
||||
|
||||
def _release_after_failure(self, *, close_cursor: bool = False) -> None:
|
||||
"""Best-effort server-side cleanup after a statement fails.
|
||||
|
||||
A statement that failed is still allocated. Skipping the release
|
||||
bricks the connection: the next PREPARE collides with the leaked
|
||||
one and every subsequent call returns a nonsense error whose
|
||||
offset points back at the *failed* SQL. A duplicate-key violation
|
||||
is about the most ordinary error an application can hit, so "one
|
||||
constraint violation kills the connection" was easy to reach and
|
||||
hard to attribute.
|
||||
|
||||
CLOSE and RELEASE get **separate** suppressions. Putting both in
|
||||
one ``with contextlib.suppress(Exception)`` block reads as "clean
|
||||
up both", but a CLOSE that raises skips the RELEASE entirely —
|
||||
and RELEASE is the one that matters. Failing to close a cursor
|
||||
wastes a handle; failing to release the statement is what breaks
|
||||
the next call.
|
||||
|
||||
Everything here is swallowed rather than propagated. If the wire
|
||||
is genuinely desynced then the cleanup fails too, and the caller
|
||||
is far better served by the real SQL error than by a secondary
|
||||
failure from the cleanup path.
|
||||
"""
|
||||
if close_cursor:
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn._send_pdu(self._build_close_pdu())
|
||||
self._drain_to_eot()
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
|
||||
def _execute_select_with_params(self, params: tuple) -> None:
|
||||
"""Parameterized SELECT: SQ_BIND → CURNAME+NFETCH → drain → CLOSE+RELEASE.
|
||||
|
||||
@ -365,12 +685,20 @@ class Cursor:
|
||||
try:
|
||||
pdu = self._build_bind_only_pdu(params)
|
||||
except Exception:
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
self._release_after_failure()
|
||||
raise
|
||||
self._conn._send_pdu(pdu)
|
||||
self._drain_to_eot()
|
||||
try:
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
# The server can reject the BIND itself — a value whose type
|
||||
# doesn't match the described parameter, a bind against a
|
||||
# statement the server has since invalidated. This was the
|
||||
# last unguarded door of the six: the build was covered and
|
||||
# the drain was not, so a server-side bind rejection left the
|
||||
# statement allocated and the *next* execute() failed instead.
|
||||
self._release_after_failure()
|
||||
raise
|
||||
# Now open the cursor and fetch — the bound values are in scope
|
||||
# for the prepared statement.
|
||||
self._execute_select()
|
||||
@ -393,9 +721,23 @@ class Cursor:
|
||||
self._conn._send_pdu(
|
||||
self._build_curname_scroll_open_pdu(cursor_name)
|
||||
)
|
||||
self._drain_to_eot()
|
||||
try:
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
# Opening a scrollable cursor fails like any other
|
||||
# statement — a bad ORDER BY, a permission error, a table
|
||||
# dropped between PREPARE and OPEN. This branch had no
|
||||
# guard at all, and it is the worst place to lack one:
|
||||
# the GC-time finalizer is armed on the line *after* the
|
||||
# drain, so a failure here left the statement allocated
|
||||
# with no fallback whatsoever to reclaim it.
|
||||
self._release_after_failure(close_cursor=True)
|
||||
raise
|
||||
self._server_cursor_open = True
|
||||
self._finalizer_state[0] = True # arm the GC-time fallback
|
||||
# The connection needs to know its statement slot is taken,
|
||||
# so the next statement can refuse instead of drawing -285.
|
||||
self._conn._open_scroll_cursor = weakref.ref(self)
|
||||
self._scroll_total_rows = None
|
||||
return # don't close; cursor stays live for SQ_SFETCH
|
||||
# Phase 35: NFETCH loop — keep fetching until a response yields
|
||||
@ -405,20 +747,32 @@ class Cursor:
|
||||
# This bug was latent for ~30 phases because no test used a
|
||||
# large enough result set to trigger it.
|
||||
self._conn._send_pdu(self._build_curname_nfetch_pdu(cursor_name))
|
||||
rows_before = len(self._rows)
|
||||
self._read_fetch_response()
|
||||
rows_received = len(self._rows) - rows_before
|
||||
|
||||
while rows_received > 0:
|
||||
self._conn._send_pdu(self._build_nfetch_pdu())
|
||||
try:
|
||||
rows_before = len(self._rows)
|
||||
self._read_fetch_response()
|
||||
rows_received = len(self._rows) - rows_before
|
||||
|
||||
# Dereference BYTE/TEXT blob descriptors BEFORE CLOSE — the
|
||||
# locators are only valid while the cursor is open. No-op when
|
||||
# no BYTE/TEXT columns are present.
|
||||
self._dereference_blob_columns()
|
||||
while rows_received > 0:
|
||||
self._conn._send_pdu(self._build_nfetch_pdu())
|
||||
rows_before = len(self._rows)
|
||||
self._read_fetch_response()
|
||||
rows_received = len(self._rows) - rows_before
|
||||
|
||||
# Dereference BYTE/TEXT blob descriptors BEFORE CLOSE — the
|
||||
# locators are only valid while the cursor is open. No-op when
|
||||
# no BYTE/TEXT columns are present.
|
||||
self._dereference_blob_columns()
|
||||
except Exception:
|
||||
# A raise anywhere in the fetch loop leaves the cursor and the
|
||||
# statement allocated server-side. The GC-time finalizer only
|
||||
# covers scrollable cursors (it is armed above, in the branch
|
||||
# that returns early), so this path needs its own cleanup —
|
||||
# otherwise a mid-fetch failure leaks and the next statement
|
||||
# collides with it. Same failure mode as the DML path; see
|
||||
# _release_after_failure for what that looks like from the
|
||||
# caller's side.
|
||||
self._release_after_failure(close_cursor=True)
|
||||
raise
|
||||
|
||||
self._conn._send_pdu(self._build_close_pdu())
|
||||
self._drain_to_eot()
|
||||
@ -760,7 +1114,7 @@ class Cursor:
|
||||
writer.write_short(MessageType.SQ_EOT)
|
||||
self._conn._send_pdu(buf.getvalue())
|
||||
|
||||
reader = _SocketReader(self._conn._sock)
|
||||
reader = _make_socket_reader(self._conn._sock)
|
||||
chunks: list[bytes] = []
|
||||
while True:
|
||||
tag = reader.read_short()
|
||||
@ -804,12 +1158,17 @@ class Cursor:
|
||||
try:
|
||||
pdu = self._build_bind_execute_pdu(params)
|
||||
except Exception:
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
self._release_after_failure()
|
||||
raise
|
||||
self._conn._send_pdu(pdu)
|
||||
self._drain_to_eot()
|
||||
try:
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
# The statement is still allocated server-side even though it
|
||||
# failed. See _execute_dml for why skipping this bricks the
|
||||
# connection.
|
||||
self._release_after_failure()
|
||||
raise
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
|
||||
@ -829,7 +1188,27 @@ class Cursor:
|
||||
let the optimization-looking response confuse you.
|
||||
"""
|
||||
self._conn._send_pdu(self._build_execute_pdu())
|
||||
self._drain_to_eot() # reads DONE + COST + EOT, populates rowcount
|
||||
try:
|
||||
self._drain_to_eot() # reads DONE + COST + EOT, populates rowcount
|
||||
except Exception:
|
||||
# A statement that FAILS is still allocated on the server, and
|
||||
# skipping the release here bricked the whole connection: the
|
||||
# next PREPARE collided with the leaked statement and every
|
||||
# subsequent call returned a nonsense error (-255 "Not in
|
||||
# transaction" in autocommit, -285 otherwise) whose offset
|
||||
# pointed back at the *failed* SQL, not the new statement.
|
||||
#
|
||||
# A duplicate-key violation is about the most ordinary error an
|
||||
# application can hit, so "one constraint violation kills the
|
||||
# connection" was easy to reach and hard to attribute. Send the
|
||||
# release, then re-raise the original error.
|
||||
#
|
||||
# Suppressed rather than propagated: if the wire is genuinely
|
||||
# desynced the release will fail too, and the caller is far
|
||||
# better served by the real SQL error than by a secondary
|
||||
# failure from the cleanup path.
|
||||
self._release_after_failure()
|
||||
raise
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
|
||||
@ -886,7 +1265,10 @@ class Cursor:
|
||||
f"expected {first_len} (matching set [0])"
|
||||
)
|
||||
|
||||
# Detect SELECT — not supported in executemany.
|
||||
# Cheap pre-flight reject for the obvious case, so the common
|
||||
# mistake costs no round-trip. The authoritative check is after
|
||||
# PREPARE, below — this one shares the first-word heuristic's
|
||||
# blind spots (leading comments, CTEs, parenthesized selects).
|
||||
first_word = operation.lstrip().split(None, 1)[0].upper() if operation.strip() else ""
|
||||
if first_word == "SELECT":
|
||||
raise NotSupportedError("executemany on SELECT is not supported")
|
||||
@ -897,33 +1279,71 @@ class Cursor:
|
||||
# under the wire lock — N rows commit atomically with respect
|
||||
# to other threads on the connection.
|
||||
with self._conn._wire_lock:
|
||||
self._conn._check_scroll_cursor_conflict(self)
|
||||
if self._scrollable and self._server_cursor_open:
|
||||
self._close_server_cursor()
|
||||
# Reset per-execute state.
|
||||
self._description = None
|
||||
self._columns = []
|
||||
self._column_readers = None # Phase 37
|
||||
self._rowcount = -1
|
||||
self._rows = []
|
||||
self._row_index = -1
|
||||
self._statement_type = 0
|
||||
self._statement_already_done = False
|
||||
|
||||
# Logged-DB transaction guard — same as execute(). Idempotent
|
||||
# within an open transaction.
|
||||
self._conn._ensure_transaction()
|
||||
|
||||
# PREPARE once.
|
||||
self._conn._send_pdu(
|
||||
self._build_prepare_pdu(sql, num_qmarks=first_len)
|
||||
self._build_prepare_pdu(sql, num_qmarks=first_len),
|
||||
statement_boundary=True,
|
||||
)
|
||||
self._read_describe_response()
|
||||
|
||||
# Now the server has told us what it prepared. A result-set
|
||||
# statement that slipped past the first-word check above --
|
||||
# a CTE, a leading comment, a parenthesized select -- would
|
||||
# otherwise be executed N times down the DML path, which
|
||||
# opens no cursor and answers -260.
|
||||
if _produces_result_set(self._statement_type, len(self._columns)):
|
||||
self._release_after_failure()
|
||||
raise NotSupportedError(
|
||||
"executemany on a statement that returns rows is not "
|
||||
"supported"
|
||||
)
|
||||
|
||||
# Logged-DB transaction guard — same as execute(), and for the
|
||||
# same reason placed after the describe rather than before it.
|
||||
if self._statement_type not in _TX_CONTROL_TYPES:
|
||||
self._conn._ensure_transaction()
|
||||
|
||||
# Phase 33: pipeline — build all BIND+EXECUTE PDUs first
|
||||
# (Python work, no I/O), then send them back-to-back, then
|
||||
# drain all responses. Eliminates the per-row round-trip
|
||||
# the older serial loop paid.
|
||||
pdus = [
|
||||
self._build_bind_execute_pdu(tuple(p)) for p in seq
|
||||
]
|
||||
for pdu in pdus:
|
||||
self._conn._send_pdu(pdu)
|
||||
try:
|
||||
pdus = [
|
||||
self._build_bind_execute_pdu(tuple(p)) for p in seq
|
||||
]
|
||||
for pdu in pdus:
|
||||
self._conn._send_pdu(pdu)
|
||||
except Exception:
|
||||
# Encoding a row can fail client-side (DataError for a
|
||||
# value the connection's codec can't represent), and it
|
||||
# fails here — after the PREPARE, before anything is
|
||||
# drained. Escaping without the RELEASE leaks the
|
||||
# statement server-side, and the next PREPARE then
|
||||
# collides with it: every later call on this connection
|
||||
# returns a nonsense error pointing at the *previous*
|
||||
# SQL. ``_execute_dml_with_params`` guards the identical
|
||||
# case for the single-row path; the pipelined path was
|
||||
# simply missed.
|
||||
#
|
||||
# A send failure lands here too. The wire is likely
|
||||
# already unusable in that case, but attempting the
|
||||
# release costs nothing and the original error is what
|
||||
# propagates either way.
|
||||
self._release_after_failure()
|
||||
raise
|
||||
|
||||
# Drain N responses. The first error is captured but we
|
||||
# still drain the rest (they're SQ_ERRs for the aborted
|
||||
@ -995,7 +1415,8 @@ class Cursor:
|
||||
self._row_index = len(self._rows) # past-last
|
||||
return None
|
||||
self._row_index = nxt
|
||||
return self._rows[nxt]
|
||||
row = self._rows[nxt]
|
||||
return self._row_class(row) if self._row_class is not None else row
|
||||
|
||||
def fetchmany(self, size: int | None = None) -> list[tuple]:
|
||||
self._check_open()
|
||||
@ -1026,6 +1447,8 @@ class Cursor:
|
||||
return []
|
||||
start = self._row_index + 1
|
||||
out = self._rows[start:]
|
||||
if self._row_class is not None:
|
||||
out = [self._row_class(r) for r in out]
|
||||
self._row_index = len(self._rows)
|
||||
return list(out)
|
||||
|
||||
@ -1206,7 +1629,7 @@ class Cursor:
|
||||
if scrolltype == 4 or is_last_probe:
|
||||
# SFETCH(LAST) — TUPID == total row count
|
||||
self._scroll_total_rows = self._last_tupid
|
||||
return row
|
||||
return self._row_class(row) if self._row_class is not None else row
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the cursor.
|
||||
@ -1218,20 +1641,8 @@ class Cursor:
|
||||
if self._closed:
|
||||
return
|
||||
if self._scrollable and self._server_cursor_open:
|
||||
# Phase 27: hold the wire lock during CLOSE+RELEASE so we
|
||||
# don't interleave with another thread's pending op on the
|
||||
# connection. Best-effort: any wire failure here is
|
||||
# swallowed (the caller is closing; we don't want to mask
|
||||
# whatever caused them to close).
|
||||
try:
|
||||
with self._conn._wire_lock:
|
||||
self._conn._send_pdu(self._build_close_pdu())
|
||||
self._drain_to_eot()
|
||||
self._conn._send_pdu(self._build_release_pdu())
|
||||
self._drain_to_eot()
|
||||
except Exception:
|
||||
pass
|
||||
self._server_cursor_open = False
|
||||
with self._conn._wire_lock:
|
||||
self._close_server_cursor()
|
||||
# Phase 28: explicit close has handled the server-side resources
|
||||
# (or tried to). Disarm the finalizer so it doesn't fire later
|
||||
# for nothing — and clear the state flag as a belt-and-suspenders
|
||||
@ -1527,7 +1938,7 @@ class Cursor:
|
||||
|
||||
def _read_describe_response(self) -> None:
|
||||
"""Read DESCRIBE (+ optional SQ_INSERTDONE) + DONE + COST + EOT after PREPARE."""
|
||||
reader = _SocketReader(self._conn._sock)
|
||||
reader = _make_socket_reader(self._conn._sock)
|
||||
while True:
|
||||
tag = reader.read_short()
|
||||
if tag == MessageType.SQ_EOT:
|
||||
@ -1535,9 +1946,28 @@ class Cursor:
|
||||
elif tag == MessageType.SQ_DESCRIBE:
|
||||
self._columns, meta = parse_describe(reader)
|
||||
self._statement_id = meta.get("statement_id", 0)
|
||||
self._statement_type = meta.get("statement_type", 0)
|
||||
self._description = (
|
||||
[c.to_description_tuple() for c in self._columns] if self._columns else None
|
||||
)
|
||||
# Phase 37: pre-compile per-column reader strategy. The hot
|
||||
# row-decode loop in parse_tuple_payload uses this to avoid
|
||||
# re-running per-row dispatch decisions that depend only
|
||||
# on column metadata.
|
||||
if self._columns:
|
||||
self._column_readers = compile_column_readers(self._columns)
|
||||
# Phase 38: take it one step further — codegen a
|
||||
# specialized row decoder for THIS column shape.
|
||||
# Eliminates the per-column iteration overhead of
|
||||
# the readers loop. ``None`` if codegen can't
|
||||
# handle the shape; parse_tuple_payload then
|
||||
# falls back to the readers-list dispatch.
|
||||
self._row_decoder = compile_row_decoder(
|
||||
self._column_readers, self._columns
|
||||
)
|
||||
else:
|
||||
self._column_readers = None
|
||||
self._row_decoder = None
|
||||
elif tag == 94: # SQ_INSERTDONE — Informix optimization: literal
|
||||
# INSERT executed during PREPARE. Payload is:
|
||||
# readLongInt (10 bytes) — serial8 inserted
|
||||
@ -1560,14 +1990,18 @@ class Cursor:
|
||||
|
||||
def _read_fetch_response(self) -> None:
|
||||
"""Read TUPLE* + DONE + COST + EOT after an NFETCH or SFETCH."""
|
||||
reader = _SocketReader(self._conn._sock)
|
||||
reader = _make_socket_reader(self._conn._sock)
|
||||
while True:
|
||||
tag = reader.read_short()
|
||||
if tag == MessageType.SQ_EOT:
|
||||
return
|
||||
elif tag == MessageType.SQ_TUPLE:
|
||||
row = parse_tuple_payload(
|
||||
reader, self._columns, encoding=self._conn.encoding
|
||||
reader,
|
||||
self._columns,
|
||||
encoding=self._conn.encoding,
|
||||
readers=self._column_readers,
|
||||
row_decoder=self._row_decoder,
|
||||
)
|
||||
self._rows.append(row)
|
||||
elif tag == MessageType.SQ_DONE:
|
||||
@ -1590,7 +2024,7 @@ class Cursor:
|
||||
|
||||
def _drain_to_eot(self) -> None:
|
||||
"""Read response stream until SQ_EOT, allowing common tags in between."""
|
||||
reader = _SocketReader(self._conn._sock)
|
||||
reader = _make_socket_reader(self._conn._sock)
|
||||
while True:
|
||||
tag = reader.read_short()
|
||||
if tag == MessageType.SQ_EOT:
|
||||
@ -1644,7 +2078,6 @@ class Cursor:
|
||||
(e.g. table or column name for "not found" errors). Empty for
|
||||
most syntax errors.
|
||||
"""
|
||||
from ._protocol import ProtocolError
|
||||
|
||||
sqlcode = reader.read_short()
|
||||
isamcode = reader.read_short()
|
||||
@ -1662,7 +2095,7 @@ class Cursor:
|
||||
if name_len & 1:
|
||||
reader.read_exact(1) # pad to even
|
||||
near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace")
|
||||
except (ProtocolError, OSError):
|
||||
except WIRE_ERRORS:
|
||||
pass
|
||||
# Drain remaining bytes until SQ_EOT. Phase 28: a (ProtocolError,
|
||||
# OSError) during drain means the wire is in an unknown state —
|
||||
@ -1674,7 +2107,7 @@ class Cursor:
|
||||
t = reader.read_short()
|
||||
if t == MessageType.SQ_EOT:
|
||||
break
|
||||
except (ProtocolError, OSError):
|
||||
except WIRE_ERRORS:
|
||||
with contextlib.suppress(Exception):
|
||||
self._conn.close()
|
||||
|
||||
|
||||
0
src/informix_db/py.typed
Normal file
0
src/informix_db/py.typed
Normal file
180
src/informix_db/rows.py
Normal file
180
src/informix_db/rows.py
Normal file
@ -0,0 +1,180 @@
|
||||
"""Rows that can be read by position, by column name, or by attribute.
|
||||
|
||||
PEP 249 only requires a sequence, and a sequence is what the driver
|
||||
returns by default. That is fine for ``SELECT a, b`` and steadily worse
|
||||
as the projection grows: ``row[11]`` tells a reader nothing, and stays
|
||||
correct only until somebody adds a column in the middle.
|
||||
|
||||
Opting in with ``row_factory=Row`` gives the shape ``pyodbc`` and
|
||||
``mssql-python`` provide, all three at once::
|
||||
|
||||
conn = informix_db.connect(..., row_factory=informix_db.Row)
|
||||
cur.execute("SELECT tabid, tabname FROM systables")
|
||||
row = cur.fetchone()
|
||||
row[0], row["tabname"], row.tabname
|
||||
|
||||
It is opt-in rather than the default because it is not free, and the
|
||||
driver's whole argument is that pure Python can stay within noise of the
|
||||
C driver on bulk fetch. Measured on a 20,000-row five-column fetch:
|
||||
**37.2 ms with tuples, 40.8 ms with Row**, so about 9%. Defaulting it on
|
||||
would move the published 1.05-1.15x ratio against IfxPy to roughly
|
||||
1.15-1.25x, which is not a trade to make on everybody's behalf.
|
||||
|
||||
Where the 9% goes, against a plain tuple: about 30 ns per row to
|
||||
construct, about 39 ns per ``row[0]`` because supporting ``row["name"]``
|
||||
means ``__getitem__`` is a Python method rather than C-level tuple
|
||||
indexing, and about 13 ns per unpack since CPython's fast path for
|
||||
``a, b = row`` applies to exact tuples and not to subclasses.
|
||||
|
||||
Worth it for readability in application code. Not worth paying in a bulk
|
||||
export that never looks at a column by name, which is exactly why it is
|
||||
a choice.
|
||||
|
||||
``Row`` subclasses ``tuple``, so ``row == (1, "x")`` is still true and
|
||||
existing code keeps working unchanged.
|
||||
|
||||
**A column always beats a method of the same name.** ``tuple`` defines
|
||||
``count`` and ``index``; this class adds ``keys``, ``_asdict`` and
|
||||
``_fields``. A column named any of those would otherwise resolve to the
|
||||
method and hand back a bound method instead of a value, which is exactly
|
||||
the shape of the framing bugs this driver spent a fortnight removing: a
|
||||
plausible-looking wrong answer, in silence. So every such name gets a
|
||||
descriptor and the column wins, and the reserved set is *computed* from
|
||||
the class rather than hand-listed, because hand-listing it is what
|
||||
missed ``keys``, ``_asdict`` and ``_fields`` the first time round. The
|
||||
methods stay reachable through the class: ``tuple.count(row, x)``,
|
||||
``Row.keys(row)``.
|
||||
|
||||
The machinery itself is name-mangled (``__fields`` / ``__map``) so that
|
||||
shadowing ``_fields`` cannot break ``repr`` or ``_asdict``.
|
||||
|
||||
Column names come from ``cursor.description``. Informix folds unquoted
|
||||
identifiers to lower case, so ``SELECT Config_Key`` is reachable as
|
||||
``row.config_key``. Expressions get server-generated names that are not
|
||||
Python identifiers, like ``(count(*))``; those are reachable by
|
||||
subscript but not as attributes. Duplicate names resolve to the first
|
||||
occurrence, matching ``pyodbc``. Dunder names cannot be shadowed and
|
||||
stay subscript-only, which no real schema should notice.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import operator
|
||||
from functools import lru_cache
|
||||
from typing import ClassVar
|
||||
|
||||
__all__ = ["Row", "make_row_class"]
|
||||
|
||||
|
||||
class Row(tuple):
|
||||
"""A result row addressable by position, name, or attribute.
|
||||
|
||||
Used as a ``row_factory``. The concrete class handed to each result
|
||||
set is a subclass carrying that query's column names, built by
|
||||
:func:`make_row_class`.
|
||||
"""
|
||||
|
||||
__slots__ = ()
|
||||
|
||||
# Name-mangled to ``_Row__fields`` / ``_Row__map`` so that a column
|
||||
# called "_fields" can be shadowed without breaking the machinery
|
||||
# that reads it. Set on the per-result-set subclass.
|
||||
__fields: ClassVar[tuple[str, ...]] = ()
|
||||
__map: ClassVar[dict[str, int]] = {}
|
||||
|
||||
def __getitem__(self, key):
|
||||
# ``key.__class__ is str`` rather than isinstance: this runs on
|
||||
# every subscript.
|
||||
if key.__class__ is str:
|
||||
try:
|
||||
return tuple.__getitem__(self, self.__map[key])
|
||||
except KeyError:
|
||||
raise KeyError(
|
||||
f"no column named {key!r}; this row has "
|
||||
f"{list(self.__fields)}"
|
||||
) from None
|
||||
return tuple.__getitem__(self, key)
|
||||
|
||||
def __getattr__(self, name):
|
||||
# Only reached when normal attribute lookup has already failed,
|
||||
# so this costs nothing for names that do not collide.
|
||||
try:
|
||||
return tuple.__getitem__(self, self.__map[name])
|
||||
except KeyError:
|
||||
raise AttributeError(
|
||||
f"no column named {name!r}; this row has "
|
||||
f"{list(self.__fields)}"
|
||||
) from None
|
||||
|
||||
@property
|
||||
def _fields(self) -> tuple[str, ...]:
|
||||
"""Column names, in select order. Mirrors ``namedtuple._fields``."""
|
||||
return self.__fields
|
||||
|
||||
def keys(self) -> tuple[str, ...]:
|
||||
"""Column names, in select order."""
|
||||
return self.__fields
|
||||
|
||||
def _asdict(self) -> dict:
|
||||
"""A plain ``dict`` of the row.
|
||||
|
||||
On duplicate column names the last occurrence wins here, while
|
||||
subscript access gives the first. A dict cannot represent both,
|
||||
and quietly dropping a duplicate is better than raising on a
|
||||
query that is otherwise fine.
|
||||
"""
|
||||
return dict(zip(self.__fields, self, strict=True))
|
||||
|
||||
def __repr__(self) -> str:
|
||||
fields = self.__fields
|
||||
if len(fields) != len(self):
|
||||
return tuple.__repr__(self)
|
||||
body = ", ".join(
|
||||
f"{name}={value!r}"
|
||||
for name, value in zip(fields, self, strict=True)
|
||||
)
|
||||
return f"Row({body})"
|
||||
|
||||
def __reduce__(self):
|
||||
# The per-result-set class is created at runtime and cannot be
|
||||
# pickled by reference, so rebuild it from the field names.
|
||||
return (_rebuild_row, (self.__fields, tuple(self)))
|
||||
|
||||
|
||||
# Every non-dunder attribute a Row already answers to. A column with one
|
||||
# of these names gets a descriptor so the column wins. Computed, not
|
||||
# hand-listed: the hand-listed version covered ``count`` and ``index``
|
||||
# and silently missed ``keys``, ``_asdict`` and ``_fields``.
|
||||
_RESERVED = frozenset(
|
||||
name for name in dir(Row) if not name.startswith("__")
|
||||
) - {"_Row__fields", "_Row__map"}
|
||||
|
||||
|
||||
def _rebuild_row(fields: tuple[str, ...], values: tuple):
|
||||
return make_row_class(fields)(values)
|
||||
|
||||
|
||||
@lru_cache(maxsize=256)
|
||||
def make_row_class(fields: tuple[str, ...]) -> type[Row]:
|
||||
"""Build (and cache) the row class for one column-name shape.
|
||||
|
||||
Cached because a class per ``execute()`` would put a ``type()`` call
|
||||
on the path of every small query, and applications run the same
|
||||
handful of statement shapes over and over. Keyed on the names alone,
|
||||
so two queries selecting the same columns share a class.
|
||||
"""
|
||||
namespace: dict = {
|
||||
"__slots__": (),
|
||||
"_Row__fields": fields,
|
||||
# First occurrence wins on duplicates, matching pyodbc. Building
|
||||
# the map in reverse and letting earlier entries overwrite later
|
||||
# ones is the shortest way to say that.
|
||||
"_Row__map": {
|
||||
name: i for i, name in reversed(list(enumerate(fields)))
|
||||
},
|
||||
}
|
||||
for i, name in enumerate(fields):
|
||||
if name in _RESERVED:
|
||||
# A column of this name would otherwise resolve to a method.
|
||||
namespace[name] = property(operator.itemgetter(i))
|
||||
return type("Row", (Row,), namespace)
|
||||
@ -1,6 +1,6 @@
|
||||
{
|
||||
"machine_info": {
|
||||
"node": "rpm-bullet",
|
||||
"node": "PLACEHOLDER",
|
||||
"processor": "",
|
||||
"machine": "x86_64",
|
||||
"python_compiler": "Clang 22.1.1 ",
|
||||
|
||||
@ -26,6 +26,7 @@ get one row per scale point.
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os as _os
|
||||
from collections.abc import Iterator
|
||||
|
||||
import pytest
|
||||
@ -36,10 +37,19 @@ from tests.conftest import ConnParams
|
||||
pytestmark = [pytest.mark.benchmark, pytest.mark.integration]
|
||||
|
||||
|
||||
# Module-level scaling sizes
|
||||
# Module-level scaling sizes. The 1M row sizes are guarded by an
|
||||
# environment flag (IFX_BENCH_1M=1) so the default `make bench` run
|
||||
# stays under 5 minutes — 1M-row workloads add ~30s + the overhead
|
||||
# of seeding a 1M-row table.
|
||||
_BIG = _os.environ.get("IFX_BENCH_1M") == "1"
|
||||
|
||||
EXECUTEMANY_SIZES = [1_000, 10_000, 100_000]
|
||||
SELECT_SIZES = [1_000, 10_000, 100_000]
|
||||
WIDTH_COLUMNS = [5, 20, 50]
|
||||
if _BIG:
|
||||
EXECUTEMANY_SIZES = [*EXECUTEMANY_SIZES, 1_000_000]
|
||||
SELECT_SIZES = [*SELECT_SIZES, 1_000_000]
|
||||
|
||||
WIDTH_COLUMNS = [5, 20, 50, 100] # added 100-column case for codec stress
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
@ -80,7 +90,7 @@ def test_executemany_scaling(
|
||||
10k rows → 5 rounds (~1.1 s each = 5.5 s)
|
||||
100k rows → 3 rounds (~11 s each = 33 s)
|
||||
"""
|
||||
rounds_for = {1_000: 10, 10_000: 5, 100_000: 3}
|
||||
rounds_for = {1_000: 10, 10_000: 5, 100_000: 3, 1_000_000: 2}
|
||||
table = f"p34_em_{n_rows}"
|
||||
cur = txn_conn.cursor()
|
||||
with contextlib.suppress(informix_db.Error):
|
||||
@ -145,10 +155,10 @@ def scaling_select_table(conn_params: ConnParams) -> Iterator[str]:
|
||||
f" value FLOAT, label VARCHAR(32))"
|
||||
)
|
||||
setup_conn.commit()
|
||||
# Insert in 10k chunks, committing after each so a failure mid-loop
|
||||
# surfaces instead of silently dropping rows.
|
||||
# Population size scales with whether the 1M tests are enabled.
|
||||
target = 1_000_000 if _BIG else 100_000
|
||||
chunk = 10_000
|
||||
for base in range(0, 100_000, chunk):
|
||||
for base in range(0, target, chunk):
|
||||
rows = [
|
||||
(base + i, f"name_{base + i:06d}", (base + i) * 7,
|
||||
float(base + i) * 1.5, f"L{(base + i) % 100:02d}")
|
||||
@ -161,8 +171,8 @@ def scaling_select_table(conn_params: ConnParams) -> Iterator[str]:
|
||||
# Verify population — fail loud if the multi-chunk insert dropped rows.
|
||||
cur.execute(f"SELECT COUNT(*) FROM {table}")
|
||||
(count,) = cur.fetchone()
|
||||
assert count == 100_000, (
|
||||
f"fixture failed: {table} has {count} rows, expected 100000"
|
||||
assert count == target, (
|
||||
f"fixture failed: {table} has {count} rows, expected {target}"
|
||||
)
|
||||
try:
|
||||
yield table
|
||||
@ -216,7 +226,7 @@ def test_select_scaling(
|
||||
median grows with N, something's wrong (memory pressure, GC,
|
||||
codec degradation).
|
||||
"""
|
||||
rounds_for = {1_000: 10, 10_000: 5, 100_000: 3}
|
||||
rounds_for = {1_000: 10, 10_000: 5, 100_000: 3, 1_000_000: 2}
|
||||
|
||||
cur = select_read_conn.cursor()
|
||||
cur.execute(f"SELECT COUNT(*) FROM {scaling_select_table}")
|
||||
@ -355,3 +365,97 @@ def test_select_type_mix_1000_rows(
|
||||
return len(rows)
|
||||
|
||||
benchmark.pedantic(run, rounds=10, iterations=1)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Memory profile at 100k rows
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_streaming_fetch_100k_memory_profile(
|
||||
select_read_conn: informix_db.Connection,
|
||||
scaling_select_table: str,
|
||||
) -> None:
|
||||
"""Sample RSS during a 100k-row iteration. Verifies the cursor's
|
||||
memory footprint scales reasonably with row count.
|
||||
|
||||
Current cursor materializes the full result set on execute() (Phase
|
||||
17 in-memory model), so RSS WILL grow proportional to row count.
|
||||
The test documents the actual growth shape and provides a
|
||||
regression baseline — if growth ever exceeds 500 MB for a 100k-row
|
||||
fetch, something is leaking heavily.
|
||||
|
||||
Future server-cursor mode would maintain constant memory; this
|
||||
test would then confirm flatness.
|
||||
"""
|
||||
import gc
|
||||
import resource
|
||||
|
||||
def rss_kb() -> int:
|
||||
# Use /proc/self/status VmRSS for *current* RSS, not peak.
|
||||
# ``ru_maxrss`` is monotonic peak — a 68 MB peak from earlier
|
||||
# in the test session masks any fluctuation from this fetch.
|
||||
try:
|
||||
from pathlib import Path
|
||||
with Path("/proc/self/status").open() as f:
|
||||
for line in f:
|
||||
if line.startswith("VmRSS:"):
|
||||
return int(line.split()[1])
|
||||
except OSError:
|
||||
pass
|
||||
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
|
||||
|
||||
gc.collect()
|
||||
pre_execute_rss = rss_kb()
|
||||
|
||||
cur = select_read_conn.cursor()
|
||||
cur.execute(f"SELECT FIRST 100000 * FROM {scaling_select_table}")
|
||||
post_execute_rss = rss_kb() # rows materialized into self._rows here
|
||||
|
||||
samples: list[tuple[int, int]] = []
|
||||
rows_seen = 0
|
||||
samples.append((0, post_execute_rss))
|
||||
|
||||
for _ in cur:
|
||||
rows_seen += 1
|
||||
if rows_seen % 10_000 == 0:
|
||||
samples.append((rows_seen, rss_kb()))
|
||||
cur.close()
|
||||
gc.collect()
|
||||
final_rss = rss_kb()
|
||||
|
||||
materialization_growth = post_execute_rss - pre_execute_rss
|
||||
iteration_growth = final_rss - post_execute_rss
|
||||
|
||||
print("\nstreaming fetch 100k memory profile:")
|
||||
print(f" pre-execute RSS: {pre_execute_rss:>9} KB")
|
||||
print(f" post-execute RSS: {post_execute_rss:>9} KB "
|
||||
f"(Δ {materialization_growth:+} KB — materialization cost)")
|
||||
for rows, rss in samples[1:]:
|
||||
print(f" rows={rows:>6} rss={rss:>9} KB "
|
||||
f"(Δ from post-execute: {rss - post_execute_rss:+} KB)")
|
||||
print(f" final={final_rss} KB after cur.close() + gc.collect()")
|
||||
print(" --")
|
||||
print(f" rows iterated: {rows_seen}")
|
||||
print(f" materialization: ~{materialization_growth * 1024 // 100_000} "
|
||||
f"bytes/row (100k rows of 5 cols)")
|
||||
print(f" iteration-side allocation: {iteration_growth} KB total "
|
||||
f"(should be ~0 — iteration doesn't allocate)")
|
||||
|
||||
total_growth_kb = final_rss - pre_execute_rss
|
||||
# 500 MB ceiling for 100k rows = ~5 KB/row max. Real cost is ~50-100
|
||||
# bytes/row (5 cols x tuple+strings+ints) so this is plenty of
|
||||
# headroom for the regression check.
|
||||
assert total_growth_kb < 500_000, (
|
||||
f"100k-row fetch grew RSS by {total_growth_kb} KB — cursor is leaking"
|
||||
)
|
||||
assert rows_seen == 100_000, (
|
||||
f"expected 100000 rows iterated, got {rows_seen}"
|
||||
)
|
||||
# Iteration-side allocation should be near-zero — fetchall() / for
|
||||
# loop just walks the already-materialized self._rows list. Allow
|
||||
# 5 MB slack for opportunistic allocator behavior.
|
||||
assert iteration_growth < 5_000, (
|
||||
f"iteration over already-fetched rows grew RSS by "
|
||||
f"{iteration_growth} KB — unexpected per-row allocation"
|
||||
)
|
||||
|
||||
48
tests/docker-compose.legacy.yml
Normal file
48
tests/docker-compose.legacy.yml
Normal file
@ -0,0 +1,48 @@
|
||||
# Older Informix versions, for the server-compatibility matrix in the README.
|
||||
#
|
||||
# The primary dev container (tests/docker-compose.yml) runs Informix 15 on
|
||||
# 9088. These two run alongside it on 9089 / 9090 so all three can be tested
|
||||
# without tearing anything down.
|
||||
#
|
||||
# These images live on Docker Hub under ibmcom/ rather than icr.io/informix/ —
|
||||
# IBM stopped publishing the older developer editions to the newer registry.
|
||||
# They're tagged rather than digest-pinned because the ibmcom repo is frozen
|
||||
# (no new pushes since the icr.io move), so the tags are already immutable in
|
||||
# practice.
|
||||
#
|
||||
# make ifx-legacy-up start both
|
||||
# make ifx-legacy-setup create blobspace1 + sbspace1 (needed for LOB tests)
|
||||
# make test-matrix run the integration suite against all three
|
||||
# make ifx-legacy-down stop and remove
|
||||
services:
|
||||
ifx1210:
|
||||
container_name: informix-db-test-1210
|
||||
image: ibmcom/informix-developer-database:12.10.FC12W1DE
|
||||
privileged: true
|
||||
environment:
|
||||
LICENSE: accept
|
||||
SIZE: small
|
||||
ports:
|
||||
- "9089:9088"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "exec 3<>/dev/tcp/127.0.0.1/9088 && exec 3<&- && exec 3>&-"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 60
|
||||
start_period: 30s
|
||||
|
||||
ifx1410:
|
||||
container_name: informix-db-test-1410
|
||||
image: ibmcom/informix-developer-database:14.10.FC7W1DE
|
||||
privileged: true
|
||||
environment:
|
||||
LICENSE: accept
|
||||
SIZE: small
|
||||
ports:
|
||||
- "9090:9088"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "exec 3<>/dev/tcp/127.0.0.1/9088 && exec 3<&- && exec 3>&-"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 60
|
||||
start_period: 30s
|
||||
76
tests/setup-spaces.sh
Executable file
76
tests/setup-spaces.sh
Executable file
@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env bash
|
||||
# Create blobspace1 + sbspace1 in an Informix dev container.
|
||||
#
|
||||
# The developer-edition images ship with only rootdbs. BYTE/TEXT tests need a
|
||||
# blobspace; BLOB/CLOB (smart-LOB) tests need an sbspace plus SBSPACENAME set.
|
||||
# Without them, ~21 integration tests fail with errors that look like driver
|
||||
# bugs but are pure server configuration.
|
||||
#
|
||||
# Usage: tests/setup-spaces.sh <container-name>
|
||||
#
|
||||
# Idempotent: re-running against a container that already has the spaces prints
|
||||
# the server's "already exists" complaint and exits 0.
|
||||
#
|
||||
# Version quirks this handles, learned the hard way:
|
||||
# * INFORMIXDIR differs — 12.10/14.10 use /opt/ibm/informix, 15 nests a
|
||||
# versioned subdirectory (/opt/ibm/informix/v15.0.1.0.3).
|
||||
# * ONCONFIG is named `onconfig` on some images and `onconfig.informix` on
|
||||
# others; we probe for the file rather than guessing.
|
||||
# * `bash -lc` wipes INFORMIXDIR from the environment on these images, which
|
||||
# makes every utility fail with "Unable to read $INFORMIXDIR (/usr/informix)".
|
||||
# Use `bash -c` and export explicitly.
|
||||
# * 12.10 DE ships no `ontape`. The level-0 archive turns out to be
|
||||
# unnecessary for the tests, so we attempt it and shrug if it's missing.
|
||||
set -euo pipefail
|
||||
|
||||
CONTAINER="${1:?usage: setup-spaces.sh <container-name>}"
|
||||
|
||||
docker exec -u informix "$CONTAINER" bash -c '
|
||||
set -u
|
||||
|
||||
# Locate INFORMIXDIR: either /opt/ibm/informix or a versioned subdir under it.
|
||||
for cand in /opt/ibm/informix /opt/ibm/informix/v*; do
|
||||
if [ -x "$cand/bin/onspaces" ]; then
|
||||
export INFORMIXDIR="$cand"
|
||||
break
|
||||
fi
|
||||
done
|
||||
if [ -z "${INFORMIXDIR:-}" ]; then
|
||||
echo "could not locate INFORMIXDIR (no bin/onspaces found)" >&2
|
||||
exit 1
|
||||
fi
|
||||
export PATH="$INFORMIXDIR/bin:$PATH"
|
||||
export INFORMIXSERVER=informix
|
||||
export INFORMIXSQLHOSTS="$INFORMIXDIR/etc/sqlhosts"
|
||||
|
||||
# ONCONFIG name varies across images.
|
||||
for cfg in onconfig onconfig.informix; do
|
||||
if [ -f "$INFORMIXDIR/etc/$cfg" ]; then
|
||||
export ONCONFIG="$cfg"
|
||||
break
|
||||
fi
|
||||
done
|
||||
: "${ONCONFIG:?no onconfig found in $INFORMIXDIR/etc}"
|
||||
|
||||
echo "INFORMIXDIR=$INFORMIXDIR ONCONFIG=$ONCONFIG"
|
||||
|
||||
SPACES=/opt/ibm/data/spaces
|
||||
mkdir -p "$SPACES"
|
||||
for f in blobspace1 sbspace1; do
|
||||
[ -e "$SPACES/$f" ] || : > "$SPACES/$f"
|
||||
chmod 660 "$SPACES/$f"
|
||||
done
|
||||
|
||||
onspaces -c -b blobspace1 -g 1 -p "$SPACES/blobspace1" -o 0 -s 50000 2>&1 | grep -vE "^\s*$" || true
|
||||
onspaces -c -S sbspace1 -p "$SPACES/sbspace1" -o 0 -s 50000 -Df "AVG_LO_SIZE=100" 2>&1 | grep -vE "^\s*$" || true
|
||||
onmode -wm SBSPACENAME=sbspace1 2>&1 | tail -1 || true
|
||||
|
||||
# Level-0 archive. Informix warns it is required after adding a space; in
|
||||
# practice the tests pass without it, and 12.10 DE has no ontape at all.
|
||||
onmode -wm TAPEDEV=/dev/null >/dev/null 2>&1 || true
|
||||
if command -v ontape >/dev/null 2>&1; then
|
||||
ontape -s -L 0 2>&1 | tail -1 || true
|
||||
else
|
||||
echo "ontape not present on this image; skipping level-0 archive"
|
||||
fi
|
||||
'
|
||||
265
tests/test_async_threads.py
Normal file
265
tests/test_async_threads.py
Normal file
@ -0,0 +1,265 @@
|
||||
"""The async layer used a thread pool it shared with the whole process.
|
||||
|
||||
``asyncio.to_thread`` runs on the event loop's default executor. That
|
||||
executor belongs to the process, not to us, and it is sized from the CPU
|
||||
count — ``min(32, cpu_count + 4)``, which is six threads on a two-CPU
|
||||
container. Every blocking call in ``informix_db.aio`` went through it.
|
||||
|
||||
Two consequences, both silent.
|
||||
|
||||
**Cancelled calls hold their threads.** ``asyncio.to_thread`` cannot
|
||||
interrupt a worker, so a cancelled await leaves the thread running the
|
||||
wire call until the read timeout expires. Cancellation is ordinary in a
|
||||
web app — a client disconnect cancels the request task — so a handful of
|
||||
them pins every thread in the shared pool. Unrelated ``to_thread`` work
|
||||
anywhere else in the process then stops dead, and so does the driver.
|
||||
|
||||
**Pool concurrency was capped by an unrelated number.** A pool with
|
||||
``max_size=20`` on a two-CPU box ran six queries at a time, and nothing
|
||||
said so.
|
||||
|
||||
Each connection now owns one thread. That is the right size rather than
|
||||
a compromise: the sync connection serializes every wire operation on its
|
||||
own lock, so a second thread could do nothing but wait for the first. It
|
||||
also avoids a deadlock that a shared pool-sized executor invites — with
|
||||
N threads and N connections, N tasks blocked in ``acquire`` occupy every
|
||||
thread while the connection they wait for is held by a task that now
|
||||
needs a thread to finish and release it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
import pytest
|
||||
|
||||
from informix_db import aio
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _kw(conn_params: ConnParams) -> dict:
|
||||
return {
|
||||
"host": conn_params.host,
|
||||
"port": conn_params.port,
|
||||
"user": conn_params.user,
|
||||
"password": conn_params.password,
|
||||
"database": conn_params.database,
|
||||
"server": conn_params.server,
|
||||
"connect_timeout": 10.0,
|
||||
"read_timeout": 20.0,
|
||||
"autocommit": True,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_queries_run_with_the_default_executor_fully_held(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The starvation case, with real queries. Four unrelated jobs occupy
|
||||
every thread of a four-worker default executor; the driver must not
|
||||
be waiting behind them."""
|
||||
loop = asyncio.get_running_loop()
|
||||
previous = loop._default_executor
|
||||
loop.set_default_executor(ThreadPoolExecutor(max_workers=4))
|
||||
conns = [await aio.connect(**_kw(conn_params)) for _ in range(4)]
|
||||
cursors = [await c.cursor() for c in conns]
|
||||
|
||||
gate = threading.Event()
|
||||
hogs = [
|
||||
asyncio.create_task(asyncio.to_thread(gate.wait, 30)) for _ in range(4)
|
||||
]
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
async def query(cursor, i: int):
|
||||
await cursor.execute(f"SELECT FIRST 1 tabid + {i} FROM systables")
|
||||
return await cursor.fetchone()
|
||||
|
||||
try:
|
||||
rows = await asyncio.wait_for(
|
||||
asyncio.gather(*(query(c, i) for i, c in enumerate(cursors))),
|
||||
timeout=10.0,
|
||||
)
|
||||
assert len(rows) == 4
|
||||
assert all(r is not None for r in rows)
|
||||
finally:
|
||||
gate.set()
|
||||
await asyncio.gather(*hogs, return_exceptions=True)
|
||||
for c in conns:
|
||||
await c.close()
|
||||
# set_default_executor rejects None, which is what the loop
|
||||
# starts with before anything has used to_thread.
|
||||
loop.set_default_executor(previous or ThreadPoolExecutor())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_each_connection_gets_its_own_thread(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
a = await aio.connect(**_kw(conn_params))
|
||||
b = await aio.connect(**_kw(conn_params))
|
||||
try:
|
||||
assert a._executor is not b._executor
|
||||
assert a._executor._max_workers == 1, (
|
||||
"more than one thread per connection cannot help — the wire "
|
||||
"lock serializes them anyway"
|
||||
)
|
||||
finally:
|
||||
await a.close()
|
||||
await b.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cursor_runs_on_its_connections_thread(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""A cursor must not fall back to the default executor, or half the
|
||||
work goes back to being shared."""
|
||||
conn = await aio.connect(**_kw(conn_params))
|
||||
try:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
names: list[str] = []
|
||||
await conn._run(lambda: names.append(threading.current_thread().name))
|
||||
assert names[0].startswith("informix-conn")
|
||||
assert cur._run.__self__ is conn
|
||||
finally:
|
||||
await conn.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pool_reuses_one_thread_per_connection(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The executor lives on the sync connection, so a connection handed
|
||||
out, returned, and handed out again keeps the same thread instead of
|
||||
spawning one per acquire."""
|
||||
pool = await aio.create_pool(**_kw(conn_params), min_size=1, max_size=2)
|
||||
try:
|
||||
seen = []
|
||||
for _ in range(4):
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert await cur.fetchone() is not None
|
||||
seen.append(id(conn._executor))
|
||||
assert len(set(seen)) <= 2, (
|
||||
f"expected at most one executor per pooled connection, saw "
|
||||
f"{len(set(seen))}"
|
||||
)
|
||||
finally:
|
||||
await pool.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pool_concurrency_is_not_capped_by_the_default_executor(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Six concurrent pooled queries against a two-worker default
|
||||
executor. Under the old arrangement at most two could run."""
|
||||
loop = asyncio.get_running_loop()
|
||||
previous = loop._default_executor
|
||||
pool = await aio.create_pool(**_kw(conn_params), min_size=6, max_size=6)
|
||||
loop.set_default_executor(ThreadPoolExecutor(max_workers=2))
|
||||
try:
|
||||
|
||||
async def one(i: int):
|
||||
async with pool.connection(timeout=15.0) as conn:
|
||||
cur = await conn.cursor()
|
||||
# Long enough that serialized execution would be obvious.
|
||||
await cur.execute(
|
||||
"SELECT FIRST 200 a.tabid FROM systables a, systables b"
|
||||
)
|
||||
return len(await cur.fetchall())
|
||||
|
||||
started = time.monotonic()
|
||||
results = await asyncio.wait_for(
|
||||
asyncio.gather(*(one(i) for i in range(6))), timeout=25.0
|
||||
)
|
||||
assert all(r > 0 for r in results)
|
||||
assert len(results) == 6
|
||||
elapsed = time.monotonic() - started
|
||||
assert elapsed < 20.0, f"queries appear serialized ({elapsed:.1f}s)"
|
||||
finally:
|
||||
await pool.close()
|
||||
# set_default_executor rejects None, which is what the loop
|
||||
# starts with before anything has used to_thread.
|
||||
loop.set_default_executor(previous or ThreadPoolExecutor())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_closing_a_connection_stops_its_thread(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""One thread per connection is only affordable if the thread goes
|
||||
away with the connection."""
|
||||
before = threading.active_count()
|
||||
conns = [await aio.connect(**_kw(conn_params)) for _ in range(5)]
|
||||
for c in conns:
|
||||
cur = await c.cursor()
|
||||
await cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
for c in conns:
|
||||
await c.close()
|
||||
del conns
|
||||
# Threads exit asynchronously after shutdown; give them a moment.
|
||||
for _ in range(50):
|
||||
if threading.active_count() <= before + 1:
|
||||
break
|
||||
await asyncio.sleep(0.1)
|
||||
assert threading.active_count() <= before + 1, (
|
||||
f"connection threads outlived their connections "
|
||||
f"({before} -> {threading.active_count()})"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_row_factory_reaches_the_async_paths(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The async layer wraps the sync cursor, so row_factory should flow
|
||||
through untouched. Each of the four async fetch routes is a separate
|
||||
call site, and async iteration goes through __anext__ rather than
|
||||
__next__."""
|
||||
import informix_db
|
||||
|
||||
conn = await aio.connect(row_factory=informix_db.Row, **_kw(conn_params))
|
||||
try:
|
||||
cur = await conn.cursor()
|
||||
sql = "SELECT FIRST 2 tabid, tabname FROM systables ORDER BY tabid"
|
||||
|
||||
await cur.execute(sql)
|
||||
one = await cur.fetchone()
|
||||
assert one[0] == one["tabid"] == one.tabid
|
||||
|
||||
await cur.execute(sql)
|
||||
assert [r.tabid for r in await cur.fetchall()] == [1, 2]
|
||||
|
||||
await cur.execute(sql)
|
||||
assert all(r.tabname for r in await cur.fetchmany(2))
|
||||
|
||||
await cur.execute(sql)
|
||||
assert [r.tabid async for r in cur] == [1, 2]
|
||||
finally:
|
||||
await conn.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_pool_forwards_the_row_factory(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
import informix_db
|
||||
|
||||
pool = await aio.create_pool(
|
||||
row_factory=informix_db.Row, min_size=1, max_size=2, **_kw(conn_params)
|
||||
)
|
||||
try:
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert (await cur.fetchone()).tabid == 1
|
||||
finally:
|
||||
await pool.close()
|
||||
258
tests/test_batch_scroll_lob.py
Normal file
258
tests/test_batch_scroll_lob.py
Normal file
@ -0,0 +1,258 @@
|
||||
"""Regression tests for pipelined executemany, scrollable cursors, and LOBs.
|
||||
|
||||
Found here by fuzzing: **a client-side encoding failure inside an
|
||||
``executemany`` batch leaked the prepared statement.** The PDUs are built
|
||||
after the PREPARE, so a value the connection's codec cannot represent
|
||||
raises there — before anything is drained — and the exception escaped
|
||||
without the RELEASE. The leaked statement then collided with the next
|
||||
PREPARE and every later call on that connection failed with an error
|
||||
pointing at the *previous* SQL.
|
||||
|
||||
``_execute_dml_with_params`` already guarded exactly this case for the
|
||||
single-row path. The pipelined path was simply missed, which is the
|
||||
recurring shape of these bugs: a hazard understood in one place and not
|
||||
carried to its sibling.
|
||||
|
||||
The rest of this file is the coverage that proved the neighbouring paths
|
||||
sound — batch failures at every position, scroll boundaries, LOB sizes —
|
||||
kept so they stay that way. Each failure case ends in a health check,
|
||||
because the interesting damage is never in the statement that failed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=15.0,
|
||||
read_timeout=30.0,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
def _assert_healthy(cur) -> None:
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None, "connection unusable"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# executemany — the encoding-failure leak, and the neighbours
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [3, 50])
|
||||
@pytest.mark.parametrize("position", ["first", "mid", "last"])
|
||||
def test_encoding_failure_in_batch_does_not_leak(
|
||||
conn_params: ConnParams, n: int, position: str
|
||||
) -> None:
|
||||
"""The bug. A value the codec can't encode raises after the PREPARE;
|
||||
escaping without the RELEASE bricked the connection."""
|
||||
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_em_enc (s VARCHAR(32))")
|
||||
rows: list[tuple] = [(f"s{i}",) for i in range(n)]
|
||||
rows[idx] = ("中文",) # not representable in iso-8859-1
|
||||
|
||||
with pytest.raises(Exception): # noqa: B017 — DataError or UnicodeError
|
||||
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
|
||||
|
||||
# The connection must survive, repeatedly.
|
||||
for _ in range(3):
|
||||
_assert_healthy(cur)
|
||||
with pytest.raises(Exception): # noqa: B017
|
||||
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
|
||||
_assert_healthy(cur)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [2, 3, 10, 100])
|
||||
@pytest.mark.parametrize("position", ["first", "mid", "last"])
|
||||
def test_constraint_violation_in_batch_recovers(
|
||||
conn_params: ConnParams, n: int, position: str
|
||||
) -> None:
|
||||
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_em_dup (k INT PRIMARY KEY)")
|
||||
cur.execute("INSERT INTO t_em_dup VALUES (?)", (10_000,))
|
||||
rows = [(i,) for i in range(n)]
|
||||
rows[idx] = (10_000,)
|
||||
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.executemany("INSERT INTO t_em_dup VALUES (?)", rows)
|
||||
_assert_healthy(cur)
|
||||
|
||||
# Whatever the partial-batch semantics, COUNT(*) and a full fetch
|
||||
# must agree — a disagreement means the row decoder and the server
|
||||
# have different ideas about what is in the table.
|
||||
cur.execute("SELECT COUNT(*) FROM t_em_dup")
|
||||
(counted,) = cur.fetchone()
|
||||
cur.execute("SELECT k FROM t_em_dup")
|
||||
assert counted == len(cur.fetchall())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [1, 2, 3, 10, 100, 1000])
|
||||
def test_executemany_inserts_exactly_n_rows(
|
||||
conn_params: ConnParams, n: int
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_em_ok (k INT, v VARCHAR(16))")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_em_ok VALUES (?, ?)",
|
||||
[(i, f"v{i}") for i in range(n)],
|
||||
)
|
||||
cur.execute("SELECT COUNT(*) FROM t_em_ok")
|
||||
assert cur.fetchone() == (n,)
|
||||
|
||||
|
||||
def test_executemany_empty_and_single(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_em_edge (k INT)")
|
||||
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [])
|
||||
_assert_healthy(cur)
|
||||
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [(1,)])
|
||||
cur.execute("SELECT COUNT(*) FROM t_em_edge")
|
||||
assert cur.fetchone() == (1,)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scrollable cursors — boundaries get their own round-trip each
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [0, 1, 2, 5, 50, 300])
|
||||
def test_scroll_boundaries(conn_params: ConnParams, n: int) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
setup = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
setup.execute("DROP TABLE t_scroll")
|
||||
setup.execute("CREATE TABLE t_scroll (k INT)")
|
||||
try:
|
||||
if n:
|
||||
setup.executemany(
|
||||
"INSERT INTO t_scroll VALUES (?)", [(i,) for i in range(n)]
|
||||
)
|
||||
|
||||
cur = conn.cursor(scrollable=True)
|
||||
try:
|
||||
cur.execute("SELECT k FROM t_scroll ORDER BY k")
|
||||
assert cur.fetch_first() == ((0,) if n else None)
|
||||
assert cur.fetch_last() == ((n - 1,) if n else None)
|
||||
|
||||
if n >= 3:
|
||||
assert cur.fetch_absolute(2) == (2,)
|
||||
assert cur.fetch_prior() == (1,)
|
||||
assert cur.fetch_relative(2) == (3,)
|
||||
|
||||
# Off both ends must be None — not a crash, not a wrap.
|
||||
assert cur.fetch_absolute(n + 50) is None
|
||||
cur.fetch_first()
|
||||
assert cur.fetch_prior() is None
|
||||
|
||||
if n:
|
||||
cur.fetch_first()
|
||||
seen = [(0,)]
|
||||
while (row := cur.fetchone()) is not None:
|
||||
seen.append(row)
|
||||
assert seen == [(i,) for i in range(n)]
|
||||
finally:
|
||||
cur.close()
|
||||
_assert_healthy(setup)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
setup.execute("DROP TABLE t_scroll")
|
||||
|
||||
|
||||
def test_abandoned_scroll_cursors_do_not_leak(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Scroll cursors stay open server-side, so abandoning one leaks
|
||||
unless the finalizer runs."""
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
setup = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
setup.execute("DROP TABLE t_scroll_ab")
|
||||
setup.execute("CREATE TABLE t_scroll_ab (k INT)")
|
||||
try:
|
||||
setup.executemany(
|
||||
"INSERT INTO t_scroll_ab VALUES (?)", [(i,) for i in range(20)]
|
||||
)
|
||||
for i in range(20):
|
||||
c = conn.cursor(scrollable=True)
|
||||
c.execute("SELECT k FROM t_scroll_ab ORDER BY k")
|
||||
c.fetch_first()
|
||||
if i % 2:
|
||||
c.close()
|
||||
else:
|
||||
del c # rely on the finalizer
|
||||
_assert_healthy(setup)
|
||||
setup.execute("SELECT COUNT(*) FROM t_scroll_ab")
|
||||
assert setup.fetchone() == (20,)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
setup.execute("DROP TABLE t_scroll_ab")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Smart LOBs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"size", [0, 1, 255, 256, 1023, 1024, 4095, 4096, 65535, 65536]
|
||||
)
|
||||
def test_blob_round_trip_sizes(conn_params: ConnParams, size: int) -> None:
|
||||
"""Sizes straddle the 4096-byte SQ_FILE chunk and the 64K mark."""
|
||||
payload = bytes((i * 7 + size) % 256 for i in range(size))
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_blob_rt")
|
||||
try:
|
||||
cur.execute("CREATE TABLE t_blob_rt (k INT, b BLOB)")
|
||||
except informix_db.Error:
|
||||
pytest.skip("no sbspace configured; see make ifx-spaces")
|
||||
try:
|
||||
cur.write_blob_column(
|
||||
"INSERT INTO t_blob_rt VALUES (?, BLOB_PLACEHOLDER)",
|
||||
payload, (1,),
|
||||
)
|
||||
got = cur.read_blob_column(
|
||||
"SELECT b FROM t_blob_rt WHERE k = ?", (1,)
|
||||
)
|
||||
if size == 0:
|
||||
assert got in (b"", None)
|
||||
else:
|
||||
assert got == payload
|
||||
_assert_healthy(cur)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_blob_rt")
|
||||
|
||||
|
||||
def test_failed_blob_read_recovers(conn_params: ConnParams) -> None:
|
||||
"""The SQ_FILE path involves a server-side temp file; a failure part
|
||||
way through must not strand the connection."""
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
for _ in range(5):
|
||||
with contextlib.suppress(Exception):
|
||||
cur.read_blob_column("SELECT b FROM t_no_such_blob_tbl", ())
|
||||
_assert_healthy(cur)
|
||||
144
tests/test_capabilities.py
Normal file
144
tests/test_capabilities.py
Normal file
@ -0,0 +1,144 @@
|
||||
"""Integration tests for live SQ_PROTOCOLS negotiation.
|
||||
|
||||
These run against whatever server ``IFX_PORT`` points at, so `make
|
||||
test-matrix` exercises them on 12.10, 14.10, and 15 in turn.
|
||||
|
||||
The important one is ``test_no_violated_assumptions``: this driver
|
||||
hardcodes several wire-framing choices that SQLI actually negotiates, and
|
||||
a mismatch corrupts rows silently. That test turns "we assume this" into
|
||||
"we check this on every supported server".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db._capabilities import ENHANCED_PROTOCOL_CAP, ServerCapabilities
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=10.0,
|
||||
)
|
||||
|
||||
|
||||
def test_capabilities_are_decoded(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
caps = conn.server_capabilities
|
||||
assert isinstance(caps, ServerCapabilities)
|
||||
assert caps.raw_mask, "server sent an empty protocols reply"
|
||||
assert caps.bits
|
||||
|
||||
|
||||
def test_no_violated_assumptions(conn_params: ConnParams) -> None:
|
||||
"""Every wire-framing shape we hardcode is one the server negotiated.
|
||||
|
||||
If this fails, rows are being decoded against the wrong framing and
|
||||
the fix is to branch on the capability rather than assume it.
|
||||
"""
|
||||
with _connect(conn_params) as conn:
|
||||
caps = conn.server_capabilities
|
||||
assert caps is not None
|
||||
assert caps.violated_assumptions() == []
|
||||
|
||||
|
||||
def test_enhanced_protocol_negotiated(conn_params: ConnParams) -> None:
|
||||
"""Cap_1 is the client's declared protocol level echoed back. Every
|
||||
supported server accepts 316."""
|
||||
with _connect(conn_params) as conn:
|
||||
caps = conn.server_capabilities
|
||||
assert caps is not None
|
||||
assert caps.cap_1 == ENHANCED_PROTOCOL_CAP
|
||||
assert caps.enhanced_protocol
|
||||
|
||||
|
||||
def test_framing_capabilities_present(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
caps = conn.server_capabilities
|
||||
assert caps is not None
|
||||
assert caps.four_byte_offset
|
||||
assert caps.varchar_var_len
|
||||
assert caps.remove_64k_limit
|
||||
assert caps.usver
|
||||
|
||||
|
||||
def test_server_version_is_exposed(conn_params: ConnParams) -> None:
|
||||
"""Assert the shape rather than a value so this holds across the matrix."""
|
||||
with _connect(conn_params) as conn:
|
||||
version = conn.server_version
|
||||
assert "Informix" in version
|
||||
assert "Version" in version
|
||||
|
||||
|
||||
def test_server_version_reports_release_not_protocol_version(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""A field report flagged ``server_version`` as looking like a
|
||||
client-SDK version: the login response announces 12.10 servers as 9.56
|
||||
and 14.10 as 9.59. ``server_version`` now answers with the release; the
|
||||
raw login string moved to ``server_version_internal``."""
|
||||
with _connect(conn_params) as conn:
|
||||
release = conn.server_version
|
||||
internal = conn.server_version_internal
|
||||
assert "Informix" in internal
|
||||
# On 12.10/14.10 these genuinely differ; on 15 they agree.
|
||||
if "9.5" in internal:
|
||||
assert release != internal
|
||||
assert "9.5" not in release, (
|
||||
f"server_version still reports the protocol version: {release!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_server_version_is_cached(conn_params: ConnParams) -> None:
|
||||
"""It costs a round-trip, so repeated access must not repeat it."""
|
||||
with _connect(conn_params) as conn:
|
||||
assert conn.server_version == conn.server_version
|
||||
assert conn._server_version_full is not None
|
||||
|
||||
|
||||
def test_server_version_degrades_without_a_database(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The DBINFO lookup needs an open database. With none, the property
|
||||
must fall back rather than raise — a version lookup should never be
|
||||
able to break a working connection."""
|
||||
conn = informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=None,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=10.0,
|
||||
)
|
||||
try:
|
||||
assert isinstance(conn.server_version, str)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_capabilities_survive_multiple_connections(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Negotiation happens per-connection; two connections to the same
|
||||
server must agree."""
|
||||
with _connect(conn_params) as first, _connect(conn_params) as second:
|
||||
assert first.server_capabilities is not None
|
||||
assert second.server_capabilities is not None
|
||||
assert (
|
||||
first.server_capabilities.raw_mask
|
||||
== second.server_capabilities.raw_mask
|
||||
)
|
||||
assert first.server_capabilities.bits == second.server_capabilities.bits
|
||||
200
tests/test_capabilities_unit.py
Normal file
200
tests/test_capabilities_unit.py
Normal file
@ -0,0 +1,200 @@
|
||||
"""Unit tests for SQ_PROTOCOLS capability decoding — no server required.
|
||||
|
||||
The masks below were captured from live servers on 2026-08-27. Their most
|
||||
interesting property is that the first eight bytes are identical across
|
||||
Informix 12.10, 14.10, and 15 — which is why those three releases speak an
|
||||
indistinguishable SQLI dialect.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
|
||||
from informix_db._capabilities import (
|
||||
BIT_FOUR_BYTE_OFFSET,
|
||||
BIT_REMOVE_64K_LIMIT,
|
||||
BIT_USVER,
|
||||
BIT_VARCHAR_VAR_LEN,
|
||||
CLIENT_PROTOCOLS_MASK,
|
||||
ENHANCED_PROTOCOL_CAP,
|
||||
ServerCapabilities,
|
||||
_decode_bits,
|
||||
)
|
||||
|
||||
MASK_15 = bytes.fromhex("bdbe9ffe7fb7ffefff")
|
||||
MASK_1410 = bytes.fromhex("bdbe9ffe7fb7ffeff8")
|
||||
MASK_1210 = bytes.fromhex("bdbe9ffe7fb7ffeff0")
|
||||
|
||||
ALL_MASKS = [
|
||||
pytest.param(MASK_15, id="15.0.1.0.3"),
|
||||
pytest.param(MASK_1410, id="14.10.FC7W1"),
|
||||
pytest.param(MASK_1210, id="12.10.FC12W1DE"),
|
||||
]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Bit expansion
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_bit_numbering_is_msb_first() -> None:
|
||||
assert _decode_bits(b"\x80") == {0}
|
||||
assert _decode_bits(b"\x01") == {7}
|
||||
assert _decode_bits(b"\x00\x80") == {8}
|
||||
assert _decode_bits(b"\x00\x01") == {15}
|
||||
|
||||
|
||||
def test_bit_expansion_of_first_captured_byte() -> None:
|
||||
"""0xBD is 1011 1101 -> bits 0, 2, 3, 4, 5, 7."""
|
||||
assert _decode_bits(b"\xbd") == {0, 2, 3, 4, 5, 7}
|
||||
|
||||
|
||||
def test_empty_mask_yields_no_bits() -> None:
|
||||
assert _decode_bits(b"") == set()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The captured masks
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_first_eight_bytes_identical_across_versions() -> None:
|
||||
"""The finding this whole module exists to record: 12.10, 14.10, and 15
|
||||
negotiate exactly the same 64-bit capability set. Only the 9th byte —
|
||||
which IBM's own JDBC driver discards — differs."""
|
||||
assert MASK_15[:8] == MASK_1410[:8] == MASK_1210[:8]
|
||||
assert len({MASK_15[8], MASK_1410[8], MASK_1210[8]}) == 3
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mask", ALL_MASKS)
|
||||
def test_framing_bits_set_on_every_tested_server(mask: bytes) -> None:
|
||||
"""Every wire-framing choice this driver hardcodes is one the server
|
||||
actually negotiated. If this ever fails, the row decoder is wrong."""
|
||||
caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP)
|
||||
assert caps.four_byte_offset, "describe offsets would be misparsed"
|
||||
assert caps.varchar_var_len, "VARCHAR framing would be misparsed"
|
||||
assert caps.remove_64k_limit, "fast-path length prefix would be wrong"
|
||||
assert caps.bigint
|
||||
assert caps.long_id
|
||||
assert caps.lvarchar_gt_2k
|
||||
assert caps.gls
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mask", ALL_MASKS)
|
||||
def test_no_violated_assumptions_on_tested_servers(mask: bytes) -> None:
|
||||
caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP)
|
||||
assert caps.violated_assumptions() == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mask", ALL_MASKS)
|
||||
def test_ninth_byte_is_decoded(mask: bytes) -> None:
|
||||
"""JDBC drops bits 64+; we keep them because they're the only part
|
||||
that varies between releases."""
|
||||
caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP)
|
||||
assert any(b >= 64 for b in caps.bits)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Pre-set bits — the reason isUSVER is a dead end
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_preset_bits_applied_when_cap1_nonzero() -> None:
|
||||
"""JDBC pre-sets {0,2,3,4,49,51} for any non-zero Cap_1 and BitSet.set
|
||||
never clears, so these are true on any modern server regardless of the
|
||||
mask. That's why isUSVER never varies and can't explain version-specific
|
||||
behaviour."""
|
||||
caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=ENHANCED_PROTOCOL_CAP)
|
||||
assert {0, 2, 3, 4, 49, 51}.issubset(caps.bits)
|
||||
assert caps.usver
|
||||
|
||||
|
||||
def test_preset_bits_not_applied_when_cap1_zero() -> None:
|
||||
caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=0)
|
||||
assert caps.bits == frozenset()
|
||||
assert not caps.usver
|
||||
|
||||
|
||||
def test_preset_cannot_clear_a_bit_the_mask_set() -> None:
|
||||
caps = ServerCapabilities.from_wire(MASK_15, cap_1=ENHANCED_PROTOCOL_CAP)
|
||||
assert caps.has(BIT_USVER)
|
||||
assert caps.has(BIT_FOUR_BYTE_OFFSET)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Assumption violations — the diagnostic path
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_missing_four_byte_offset_is_reported() -> None:
|
||||
# Everything set except bit 50.
|
||||
bits = bytearray(b"\xff" * 9)
|
||||
bits[BIT_FOUR_BYTE_OFFSET // 8] &= ~(0x80 >> (BIT_FOUR_BYTE_OFFSET % 8)) & 0xFF
|
||||
caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP)
|
||||
problems = caps.violated_assumptions()
|
||||
assert len(problems) == 1
|
||||
assert "4-byte describe offsets" in problems[0]
|
||||
|
||||
|
||||
def test_missing_varchar_var_len_is_reported() -> None:
|
||||
bits = bytearray(b"\xff" * 9)
|
||||
bits[BIT_VARCHAR_VAR_LEN // 8] &= ~(0x80 >> (BIT_VARCHAR_VAR_LEN % 8)) & 0xFF
|
||||
caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP)
|
||||
problems = caps.violated_assumptions()
|
||||
assert len(problems) == 1
|
||||
assert "variable-length VARCHAR" in problems[0]
|
||||
|
||||
|
||||
def test_missing_remove_64k_is_reported() -> None:
|
||||
bits = bytearray(b"\xff" * 9)
|
||||
bits[BIT_REMOVE_64K_LIMIT // 8] &= ~(0x80 >> (BIT_REMOVE_64K_LIMIT % 8)) & 0xFF
|
||||
caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP)
|
||||
problems = caps.violated_assumptions()
|
||||
assert len(problems) == 1
|
||||
assert "64K limit" in problems[0]
|
||||
|
||||
|
||||
def test_all_zero_mask_reports_every_assumption() -> None:
|
||||
caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=ENHANCED_PROTOCOL_CAP)
|
||||
assert len(caps.violated_assumptions()) == 3
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Misc
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_client_offer_matches_jdbc_reference() -> None:
|
||||
"""IfxSqliConnect.clientProtocols, verbatim. Changing this changes what
|
||||
the server negotiates, so it is pinned."""
|
||||
jdbc_reference = bytes([0xFF, 0xFC, 0x7F, 0xFC, 0x3C, 0x8C, 0xAA, 0x97])
|
||||
assert jdbc_reference == CLIENT_PROTOCOLS_MASK
|
||||
assert len(CLIENT_PROTOCOLS_MASK) == 8
|
||||
|
||||
|
||||
def test_enhanced_protocol_requires_exact_cap1() -> None:
|
||||
"""JDBC tests == 316, not >=, because the server echoes the client's own
|
||||
declared level rather than reporting its own version."""
|
||||
assert ServerCapabilities.from_wire(MASK_15, cap_1=316).enhanced_protocol
|
||||
assert not ServerCapabilities.from_wire(MASK_15, cap_1=315).enhanced_protocol
|
||||
assert not ServerCapabilities.from_wire(MASK_15, cap_1=317).enhanced_protocol
|
||||
|
||||
|
||||
def test_repr_is_readable_and_includes_mask() -> None:
|
||||
caps = ServerCapabilities.from_wire(
|
||||
MASK_15, cap_1=316, server_version="IBM Informix Dynamic Server Version 15"
|
||||
)
|
||||
text = repr(caps)
|
||||
assert "bdbe9ffe7fb7ffefff" in text
|
||||
assert "usver" in text
|
||||
assert "cap_1=316" in text
|
||||
|
||||
|
||||
def test_capabilities_are_frozen() -> None:
|
||||
"""Immutable so a connection's negotiated state can't be edited after
|
||||
the fact and quietly disagree with what's on the wire."""
|
||||
caps = ServerCapabilities.from_wire(MASK_15, cap_1=316)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
caps.cap_1 = 1 # type: ignore[misc]
|
||||
151
tests/test_datetime_fraction.py
Normal file
151
tests/test_datetime_fraction.py
Normal file
@ -0,0 +1,151 @@
|
||||
"""Regression tests for DATETIME sub-second precision on bind.
|
||||
|
||||
``_encode_datetime`` used to emit YEAR TO SECOND unconditionally, so
|
||||
binding a ``datetime`` carrying microseconds into a
|
||||
``DATETIME YEAR TO FRACTION(n)`` column stored zeros — silently, with no
|
||||
error and no warning. Reads were always fine, which is what made it hard
|
||||
to notice: the value only went missing on the way in.
|
||||
|
||||
It now emits YEAR TO FRACTION(5) when ``microsecond`` is non-zero and
|
||||
keeps the original YEAR TO SECOND encoding otherwise.
|
||||
|
||||
FRACTION(5) is Informix's widest and resolves to 10 µs, so Python's
|
||||
sixth microsecond digit is dropped. That's a real limit of the type, not
|
||||
a driver choice, and the tests below pin the truncation so it can't drift
|
||||
into something worse.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.converters import _encode_datetime
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=10.0,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Encoder unit tests — no server needed
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_encoder_uses_year_to_second_without_microseconds() -> None:
|
||||
"""The long-exercised path must stay byte-identical."""
|
||||
type_code, prec, raw = _encode_datetime(
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15)
|
||||
)
|
||||
assert type_code == 10
|
||||
assert prec == (14 << 8) | 10 # digit_count 14, YEAR..SECOND
|
||||
assert raw == b"\x00\x08\xc7\x14\x1a\x08\x1f\x0c\x1e\x0f"
|
||||
assert len(raw) == 10 # 2 len + 1 exp + 7 BCD pairs
|
||||
|
||||
|
||||
def test_encoder_widens_to_fraction_when_microseconds_present() -> None:
|
||||
type_code, prec, raw = _encode_datetime(
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
|
||||
)
|
||||
assert type_code == 10
|
||||
assert prec == (19 << 8) | 15 # digit_count 19, YEAR..FRACTION(5)
|
||||
assert len(raw) == 13 # 2 len + 1 exp + 10 BCD pairs
|
||||
# Exponent byte is unchanged: the integer part is still 7 base-100
|
||||
# pairs, the fraction just adds three more after the point.
|
||||
assert raw[2] == 0xC7
|
||||
# Trailing pairs carry 120000 as BCD 12/00/00.
|
||||
assert raw[-3:] == b"\x0c\x00\x00"
|
||||
|
||||
|
||||
def test_encoder_pads_fraction_to_six_digits() -> None:
|
||||
"""One microsecond must not shift the BCD pairs."""
|
||||
_, prec, raw = _encode_datetime(
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15, 1)
|
||||
)
|
||||
assert prec == (19 << 8) | 15
|
||||
assert len(raw) == 13
|
||||
assert raw[-3:] == b"\x00\x00\x01" # 000001
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Round-trip against a real server
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.parametrize(
|
||||
("microsecond", "expected"),
|
||||
[
|
||||
(0, 0),
|
||||
(120000, 120000),
|
||||
(500000, 500000),
|
||||
(1, 0), # below FRACTION(5) resolution
|
||||
(999999, 999990), # truncated to 5 significant digits
|
||||
],
|
||||
)
|
||||
def test_fraction_round_trip(
|
||||
conn_params: ConnParams, microsecond: int, expected: int
|
||||
) -> None:
|
||||
value = datetime.datetime(2026, 8, 31, 12, 30, 15, microsecond)
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_dt_frac "
|
||||
"(k INT, t DATETIME YEAR TO FRACTION(5))"
|
||||
)
|
||||
cur.execute("INSERT INTO t_dt_frac VALUES (?, ?)", (1, value))
|
||||
cur.execute("SELECT t FROM t_dt_frac")
|
||||
(got,) = cur.fetchone()
|
||||
assert got == value.replace(microsecond=expected)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_fraction_bind_into_year_to_second_column(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Widening the bind must not break narrower columns — Informix
|
||||
converts between qualifiers on assignment, truncating server-side."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_dt_sec (k INT, t DATETIME YEAR TO SECOND)"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_dt_sec VALUES (?, ?)",
|
||||
(1, datetime.datetime(2026, 8, 31, 12, 30, 15, 987654)),
|
||||
)
|
||||
cur.execute("SELECT t FROM t_dt_sec")
|
||||
assert cur.fetchone() == (
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_fraction_survives_alongside_lvarchar(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The reported schema pairs FRACTION(5) columns with LVARCHARs."""
|
||||
ts = datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_dt_lv "
|
||||
"(s LVARCHAR(512), t DATETIME YEAR TO FRACTION(5), n INT8 NOT NULL)"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_dt_lv VALUES (?, ?, ?)", ("PackageRoot", ts, 10)
|
||||
)
|
||||
cur.execute("SELECT s, t, n FROM t_dt_lv")
|
||||
assert cur.fetchone() == ("PackageRoot", ts, 10)
|
||||
71
tests/test_int8_unit.py
Normal file
71
tests/test_int8_unit.py
Normal file
@ -0,0 +1,71 @@
|
||||
"""Unit tests for the INT8 / SERIAL8 codec — no server required.
|
||||
|
||||
The byte vectors below were captured off the wire from both Informix
|
||||
12.10.FC12W1DE and 15.0.1.0.3DE, which emit byte-identical encodings.
|
||||
|
||||
INT8 is sign-magnitude across two 32-bit halves, stored high-half-last:
|
||||
|
||||
bytes 0-1 sign word: 0 = NULL, 1 = positive, 0xFFFF = negative
|
||||
bytes 2-5 LOW 32 bits, big-endian unsigned
|
||||
bytes 6-9 HIGH 32 bits, big-endian unsigned
|
||||
|
||||
The trap: +n and -n have *identical* magnitude bytes. Anything that
|
||||
treats this as a two's-complement integer is wrong for every negative
|
||||
value while looking correct for every positive one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from informix_db._types import IfxType
|
||||
from informix_db.converters import FIXED_WIDTHS, _decode_int8
|
||||
|
||||
# (hex bytes, expected value) — captured from the wire.
|
||||
WIRE_VECTORS = [
|
||||
("0001be991a140000001c", 123456789012),
|
||||
("ffffbe991a140000001c", -123456789012),
|
||||
("00010000002a00000000", 42),
|
||||
("ffff0000002a00000000", -42),
|
||||
("00010000000000000000", 0),
|
||||
("00000000000000000000", None), # sign word 0 -> NULL
|
||||
("0001ffffffffffffffff", 2**64 - 1), # both halves saturated
|
||||
("00010000000100000000", 1),
|
||||
("00010000000000000001", 1 << 32), # high half only
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("hexbytes", "expected"), WIRE_VECTORS)
|
||||
def test_decode_int8_wire_vectors(hexbytes: str, expected: int | None) -> None:
|
||||
assert _decode_int8(bytes.fromhex(hexbytes)) == expected
|
||||
|
||||
|
||||
def test_positive_and_negative_share_magnitude_bytes() -> None:
|
||||
"""The whole reason a naive 8-byte read gets negatives wrong."""
|
||||
pos = bytes.fromhex("0001be991a140000001c")
|
||||
neg = bytes.fromhex("ffffbe991a140000001c")
|
||||
assert pos[2:] == neg[2:], "magnitude bytes should be identical"
|
||||
assert _decode_int8(pos) == -_decode_int8(neg)
|
||||
|
||||
|
||||
def test_null_sign_word_beats_nonzero_magnitude() -> None:
|
||||
"""Sign word 0 means NULL regardless of what the magnitude bytes hold."""
|
||||
assert _decode_int8(bytes.fromhex("0000be991a140000001c")) is None
|
||||
|
||||
|
||||
def test_short_payload_raises() -> None:
|
||||
with pytest.raises(ValueError, match="too short"):
|
||||
_decode_int8(bytes.fromhex("0001be991a14")) # 6 bytes, need 10
|
||||
|
||||
|
||||
def test_int8_registered_as_ten_bytes() -> None:
|
||||
"""A width of 8 here would desync every row containing an INT8."""
|
||||
assert FIXED_WIDTHS[IfxType.INT8] == 10
|
||||
assert FIXED_WIDTHS[IfxType.SERIAL8] == 10
|
||||
|
||||
|
||||
def test_int8_width_differs_from_bigint() -> None:
|
||||
"""INT8 (17) and BIGINT (52) are different types with different
|
||||
widths — conflating them is the easy mistake."""
|
||||
assert FIXED_WIDTHS[IfxType.INT8] != FIXED_WIDTHS[IfxType.BIGINT]
|
||||
assert FIXED_WIDTHS[IfxType.BIGINT] == 8
|
||||
249
tests/test_lvarchar_framing.py
Normal file
249
tests/test_lvarchar_framing.py
Normal file
@ -0,0 +1,249 @@
|
||||
"""Regression tests for LVARCHAR tuple framing, reported 2026-08-31.
|
||||
|
||||
Two independent framing errors, both of which shifted every column that
|
||||
followed an LVARCHAR:
|
||||
|
||||
1. **A phantom pad byte.** We appended an even-byte pad when the value
|
||||
length was odd. There is no pad — the next column begins immediately
|
||||
after the last content byte.
|
||||
2. **A missing length on NULL.** We returned as soon as the null
|
||||
indicator said NULL, leaving the 4-byte length field unread. The
|
||||
length is part of the envelope and is always present.
|
||||
|
||||
Both survived a 247-test suite because the only LVARCHAR fixture used
|
||||
``'lv value'`` — 8 characters, even, and never NULL. Neither faulty
|
||||
branch ever executed.
|
||||
|
||||
The tests below therefore vary length parity and nullness deliberately,
|
||||
and always place a column *after* the LVARCHAR, because the damage lands
|
||||
downstream: a trailing LVARCHAR can be mis-sized with no visible effect.
|
||||
|
||||
Wire evidence (Informix 12.10) for INT8 / LVARCHAR / INT8:
|
||||
|
||||
'odd' -> 00 | 00 00 00 03 | 6f 64 64 8 bytes
|
||||
NULL -> 01 | 00 00 00 00 5 bytes
|
||||
'' -> 00 | 00 00 00 00 5 bytes
|
||||
|
||||
NULL and empty string differ only in the indicator byte.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=10.0,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Length parity — the phantom pad byte
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text",
|
||||
[
|
||||
"", # 0 — even, empty
|
||||
"a", # 1 — ODD
|
||||
"ab", # 2
|
||||
"abc", # 3 — ODD
|
||||
"PackageRoot", # 11 — ODD, the reported value
|
||||
"lv value", # 8 — the old fixture that hid the bug
|
||||
"x" * 255, # 255 — ODD, spans a length byte boundary
|
||||
"y" * 256, # 256
|
||||
],
|
||||
)
|
||||
def test_lvarchar_length_parity_does_not_shift_next_column(
|
||||
conn_params: ConnParams, text: str
|
||||
) -> None:
|
||||
"""A trailing sentinel column catches any over- or under-read."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_lv_parity "
|
||||
"(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL, c VARCHAR(8))"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_lv_parity VALUES (?, ?, ?, ?)",
|
||||
(2001, text, 10, "tail"),
|
||||
)
|
||||
cur.execute("SELECT a, s, b, c FROM t_lv_parity")
|
||||
assert cur.fetchone() == (2001, text, 10, "tail")
|
||||
|
||||
|
||||
def test_odd_length_lvarchar_reproduces_the_report(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The exact failure shape: INT8 decoded as its own value shifted one
|
||||
byte left (10 -> 2560), and the following string losing a character."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_lv_report "
|
||||
"(a INT8 NOT NULL, k LVARCHAR(512) NOT NULL,"
|
||||
" b INT8 NOT NULL, v LVARCHAR(1024))"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_lv_report VALUES (?, ?, ?, ?)",
|
||||
(2001, "PackageRoot", 10, "/content/package"),
|
||||
)
|
||||
cur.execute("SELECT a, k, b, v FROM t_lv_report")
|
||||
row = cur.fetchone()
|
||||
assert row == (2001, "PackageRoot", 10, "/content/package")
|
||||
assert row[2] != 2560, "INT8 shifted one byte left"
|
||||
assert row[3].startswith("/"), "leading character lost"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# NULL — the missing length field
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_null_lvarchar_does_not_shift_next_column(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_lv_null "
|
||||
"(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL)"
|
||||
)
|
||||
cur.execute("INSERT INTO t_lv_null VALUES (?, ?, ?)", (3001, None, 20))
|
||||
cur.execute("SELECT a, s, b FROM t_lv_null")
|
||||
assert cur.fetchone() == (3001, None, 20)
|
||||
|
||||
|
||||
def test_null_and_empty_lvarchar_are_distinguished(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""They differ only in the indicator byte, so it's easy to conflate."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_lv_ne (k INT, s LVARCHAR(64), tail INT)"
|
||||
)
|
||||
cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (1, None, 111))
|
||||
cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (2, "", 222))
|
||||
cur.execute("SELECT k, s, tail FROM t_lv_ne ORDER BY k")
|
||||
assert cur.fetchall() == [(1, None, 111), (2, "", 222)]
|
||||
|
||||
|
||||
def test_consecutive_null_lvarchars(conn_params: ConnParams) -> None:
|
||||
"""Each NULL under-read by 4 bytes, so several in a row compound."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_lv_many_null "
|
||||
"(a INT8 NOT NULL, s1 LVARCHAR(256), s2 LVARCHAR(256),"
|
||||
" s3 LVARCHAR(256), b INT8 NOT NULL)"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_lv_many_null VALUES (?, ?, ?, ?, ?)",
|
||||
(4001, None, None, None, 40),
|
||||
)
|
||||
cur.execute("SELECT a, s1, s2, s3, b FROM t_lv_many_null")
|
||||
assert cur.fetchone() == (4001, None, None, None, 40)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Wide, mixed shape — several LVARCHARs interleaved with other types
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
_WIDE_COLS = [
|
||||
"tag", "key_txt", "def_txt", "cur_txt", "note_txt",
|
||||
"ver", "made_by", "ident", "made_on",
|
||||
]
|
||||
_WIDE_ROW = (
|
||||
"Gadget",
|
||||
"PackageRoot", # 11 — ODD
|
||||
None, # NULL
|
||||
"/content/package", # 16
|
||||
"z", # 1 — ODD
|
||||
77,
|
||||
"maker",
|
||||
3001,
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
|
||||
)
|
||||
|
||||
|
||||
def _make_wide(cur) -> None:
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_lv_wide ("
|
||||
" tag VARCHAR(32) NOT NULL,"
|
||||
" key_txt LVARCHAR(512) NOT NULL,"
|
||||
" def_txt LVARCHAR(1024),"
|
||||
" cur_txt LVARCHAR(1024),"
|
||||
" note_txt LVARCHAR(1024),"
|
||||
" ver INT8 DEFAULT 0 NOT NULL,"
|
||||
" made_by VARCHAR(100),"
|
||||
" ident INT8 NOT NULL,"
|
||||
" made_on DATETIME YEAR TO FRACTION(5))"
|
||||
)
|
||||
cur.execute(
|
||||
f"INSERT INTO t_lv_wide VALUES ({', '.join(['?'] * len(_WIDE_COLS))})",
|
||||
_WIDE_ROW,
|
||||
)
|
||||
|
||||
|
||||
def test_wide_mixed_row(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
_make_wide(cur)
|
||||
cur.execute(f"SELECT {', '.join(_WIDE_COLS)} FROM t_lv_wide")
|
||||
assert cur.fetchone() == _WIDE_ROW
|
||||
|
||||
|
||||
@pytest.mark.parametrize("shift", range(len(_WIDE_COLS)))
|
||||
def test_wide_row_survives_column_reordering(
|
||||
conn_params: ConnParams, shift: int
|
||||
) -> None:
|
||||
"""The reporter isolated this by reordering columns — values were
|
||||
correct first in the list and wrong later on. Rotating the projection
|
||||
exercises every position for every type."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
_make_wide(cur)
|
||||
order = _WIDE_COLS[shift:] + _WIDE_COLS[:shift]
|
||||
cur.execute(f"SELECT {', '.join(order)} FROM t_lv_wide")
|
||||
expected = tuple(_WIDE_ROW[_WIDE_COLS.index(c)] for c in order)
|
||||
assert cur.fetchone() == expected
|
||||
|
||||
|
||||
def test_select_star_wide_row(conn_params: ConnParams) -> None:
|
||||
"""``SELECT *`` raised IndexError once enough LVARCHARs accumulated."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
_make_wide(cur)
|
||||
cur.execute("SELECT * FROM t_lv_wide")
|
||||
row = cur.fetchone()
|
||||
names = [d[0] for d in cur.description]
|
||||
assert row == tuple(_WIDE_ROW[_WIDE_COLS.index(n)] for n in names)
|
||||
|
||||
|
||||
def test_repeated_lvarchar_columns(conn_params: ConnParams) -> None:
|
||||
"""Selecting the same LVARCHAR twice doubles any per-column drift."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
_make_wide(cur)
|
||||
cur.execute(
|
||||
"SELECT ident, key_txt, ident, key_txt, ver FROM t_lv_wide"
|
||||
)
|
||||
assert cur.fetchone() == (3001, "PackageRoot", 3001, "PackageRoot", 77)
|
||||
76
tests/test_package_metadata.py
Normal file
76
tests/test_package_metadata.py
Normal file
@ -0,0 +1,76 @@
|
||||
"""Guards on the package's own metadata.
|
||||
|
||||
The distribution is ``informix-driver``; the import module is
|
||||
``informix_db``. That mismatch is a live trap: ``importlib.metadata``
|
||||
needs the *distribution* name, so the string can't be derived from
|
||||
``__name__``, and getting it wrong fails silently — ``__version__``
|
||||
degrades to the ``0.0.0+local`` sentinel instead of raising.
|
||||
|
||||
That is exactly what shipped between the 2026-05-08 rename and
|
||||
2026.08.27: ``__init__`` still asked for ``informix-db``, so anyone who
|
||||
installed the renamed package saw ``0.0.0+local``. It went unnoticed
|
||||
because a stale ``informix-db`` distribution lingered in the dev
|
||||
environment and answered the query.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import tomllib
|
||||
|
||||
import informix_db
|
||||
|
||||
PYPROJECT = Path(__file__).resolve().parent.parent / "pyproject.toml"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def pyproject() -> dict:
|
||||
if not PYPROJECT.is_file():
|
||||
pytest.skip("pyproject.toml not present (installed-package test run)")
|
||||
with PYPROJECT.open("rb") as handle:
|
||||
return tomllib.load(handle)
|
||||
|
||||
|
||||
def test_version_is_not_the_unresolved_sentinel() -> None:
|
||||
"""If this fails, the distribution name in ``__init__`` no longer
|
||||
matches ``[project].name`` and every user sees a bogus version."""
|
||||
assert informix_db.__version__ != "0.0.0+local", (
|
||||
"__version__ fell back to its sentinel — the distribution name "
|
||||
"looked up in informix_db/__init__.py does not match the installed "
|
||||
"package. Check [project].name in pyproject.toml."
|
||||
)
|
||||
|
||||
|
||||
def test_version_matches_pyproject(pyproject: dict) -> None:
|
||||
declared = pyproject["project"]["version"]
|
||||
# PEP 440 normalisation drops leading zeros: 2026.08.27 -> 2026.8.27.
|
||||
normalised = ".".join(
|
||||
str(int(part)) if part.isdigit() else part
|
||||
for part in declared.split(".")
|
||||
)
|
||||
assert informix_db.__version__ in {declared, normalised}
|
||||
|
||||
|
||||
def test_init_looks_up_the_declared_distribution_name(pyproject: dict) -> None:
|
||||
"""Catch the rename trap directly: the name passed to
|
||||
``importlib.metadata.version()`` must equal ``[project].name``."""
|
||||
source = (
|
||||
Path(informix_db.__file__).resolve().parent / "__init__.py"
|
||||
).read_text(encoding="utf-8")
|
||||
declared_name = pyproject["project"]["name"]
|
||||
assert f'version("{declared_name}")' in source, (
|
||||
f"informix_db/__init__.py should call version({declared_name!r}) to "
|
||||
"match [project].name in pyproject.toml"
|
||||
)
|
||||
|
||||
|
||||
def test_version_is_importable_and_stringy() -> None:
|
||||
assert isinstance(informix_db.__version__, str)
|
||||
assert informix_db.__version__
|
||||
|
||||
|
||||
def test_public_surface_is_exported() -> None:
|
||||
for name in informix_db.__all__:
|
||||
assert hasattr(informix_db, name), f"{name} in __all__ but not defined"
|
||||
248
tests/test_placeholder_rewrite.py
Normal file
248
tests/test_placeholder_rewrite.py
Normal file
@ -0,0 +1,248 @@
|
||||
"""Rewriting :N placeholders without rewriting the SQL around them.
|
||||
|
||||
We advertise ``paramstyle="numeric"`` to match Informix's ESQL/C
|
||||
convention, and the wire protocol takes ``?``, so placeholders get
|
||||
rewritten on the way out. That was ``re.sub(r":(\\d+)", "?", sql)``,
|
||||
which cannot see a string literal and therefore rewrote the inside of
|
||||
one::
|
||||
|
||||
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
|
||||
stored as 'http://host?/x'
|
||||
|
||||
Any ``HH:MM`` time, any URL with a port, any aspect ratio, any
|
||||
``key:value`` string. It wrote wrong data and said nothing.
|
||||
|
||||
Nothing about it was opt-in either. The rewrite runs whenever a
|
||||
statement has parameters, whatever placeholder style the caller actually
|
||||
used, so writing ``?`` everywhere and never touching numeric style did
|
||||
not protect you. And it changed the placeholder *count* while
|
||||
``num_qmarks`` was still computed from ``len(params)``, leaving driver
|
||||
and server disagreeing about how many binds exist.
|
||||
|
||||
The lexical rules below are Informix's own, measured against 12.10,
|
||||
14.10 and 15 rather than assumed from standard SQL. The one that matters
|
||||
most is the backslash: ``'a\\'b'`` is an *unterminated string* to
|
||||
Informix (``-282``), not an escaped quote. A scanner written to
|
||||
Postgres habits would desync on it and corrupt everything after.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _rewrite_numeric_to_qmark as rewrite
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Substitution happens where it should
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("sql", "expected"),
|
||||
[
|
||||
("SELECT * FROM t WHERE a = :1", "SELECT * FROM t WHERE a = ?"),
|
||||
(
|
||||
"SELECT * FROM t WHERE a = :1 AND b = :2",
|
||||
"SELECT * FROM t WHERE a = ? AND b = ?",
|
||||
),
|
||||
("SELECT * FROM t WHERE a = :10", "SELECT * FROM t WHERE a = ?"),
|
||||
# Already-? SQL is returned untouched, and cheaply.
|
||||
("SELECT * FROM t WHERE a = ?", "SELECT * FROM t WHERE a = ?"),
|
||||
("", ""),
|
||||
],
|
||||
)
|
||||
def test_placeholders_are_rewritten(sql: str, expected: str) -> None:
|
||||
assert rewrite(sql) == expected
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ...and nowhere else
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("label", "sql", "expected"),
|
||||
[
|
||||
(
|
||||
"url-with-port",
|
||||
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = :1",
|
||||
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?",
|
||||
),
|
||||
(
|
||||
"time-of-day",
|
||||
"INSERT INTO log VALUES ('started at 09:15:30', :1)",
|
||||
"INSERT INTO log VALUES ('started at 09:15:30', ?)",
|
||||
),
|
||||
(
|
||||
"aspect-ratio",
|
||||
"UPDATE t SET ratio = '16:9' WHERE id = :1",
|
||||
"UPDATE t SET ratio = '16:9' WHERE id = ?",
|
||||
),
|
||||
(
|
||||
"doubled-quote-escape",
|
||||
"SELECT 'it''s 10:30' FROM t WHERE a = :1",
|
||||
"SELECT 'it''s 10:30' FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"delimited-identifier",
|
||||
'SELECT "col:1" FROM t WHERE a = :1',
|
||||
'SELECT "col:1" FROM t WHERE a = ?',
|
||||
),
|
||||
(
|
||||
"line-comment",
|
||||
"-- ticket :99\nSELECT * FROM t WHERE a = :1",
|
||||
"-- ticket :99\nSELECT * FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"block-comment",
|
||||
"SELECT /* not :99 */ * FROM t WHERE a = :1",
|
||||
"SELECT /* not :99 */ * FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"brace-comment",
|
||||
"SELECT { not :99 } * FROM t WHERE a = :1",
|
||||
"SELECT { not :99 } * FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"cast-operator",
|
||||
"SELECT a::INT FROM t WHERE a = :1",
|
||||
"SELECT a::INT FROM t WHERE a = ?",
|
||||
),
|
||||
(
|
||||
"colon-not-a-placeholder",
|
||||
"SELECT a FROM t WHERE b = ':x' AND c = :1",
|
||||
"SELECT a FROM t WHERE b = ':x' AND c = ?",
|
||||
),
|
||||
(
|
||||
"literal-after-placeholder",
|
||||
"SELECT * FROM t WHERE a = :1 AND b = '10:30'",
|
||||
"SELECT * FROM t WHERE a = ? AND b = '10:30'",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_quotes_and_comments_are_left_alone(
|
||||
label: str, sql: str, expected: str
|
||||
) -> None:
|
||||
assert rewrite(sql) == expected, label
|
||||
|
||||
|
||||
def test_backslash_does_not_escape_a_quote() -> None:
|
||||
"""Informix answers -282 for ``'a\\'b'``: the backslash is an ordinary
|
||||
character and the string is unterminated. A scanner that treated it
|
||||
as an escape would think it was still inside the literal and stop
|
||||
substituting, or worse, resume in the wrong place."""
|
||||
sql = "SELECT 'a\\' FROM t WHERE x = :1"
|
||||
# The quote closes at the character after the backslash, so :1 is
|
||||
# outside the literal and gets substituted.
|
||||
assert rewrite(sql) == "SELECT 'a\\' FROM t WHERE x = ?"
|
||||
|
||||
|
||||
def test_unterminated_quote_substitutes_nothing_further() -> None:
|
||||
"""Under-substituting leaves the server to reject SQL that was
|
||||
already malformed. Guessing would corrupt a literal."""
|
||||
assert rewrite("SELECT 'oops :1 FROM t") == "SELECT 'oops :1 FROM t"
|
||||
|
||||
|
||||
def test_unterminated_block_comment_substitutes_nothing_further() -> None:
|
||||
assert rewrite("SELECT /* oops :1 FROM t") == "SELECT /* oops :1 FROM t"
|
||||
|
||||
|
||||
def test_block_comments_do_not_nest() -> None:
|
||||
"""Measured: ``/* a /* b */ c */`` is a syntax error on all three
|
||||
servers, so the first ``*/`` closes the comment. Treating them as
|
||||
nesting would swallow live SQL."""
|
||||
assert (
|
||||
rewrite("SELECT /* a /* b */ :1 FROM t")
|
||||
== "SELECT /* a /* b */ ? FROM t"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_colon_literals_round_trip(conn_params: ConnParams) -> None:
|
||||
"""The bug as a user meets it: the value stored is not the value
|
||||
written. Both placeholder styles, because the rewrite ran for both."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_rw (id INT, s VARCHAR(60))")
|
||||
cur.execute(
|
||||
"INSERT INTO t_rw VALUES (?, 'http://host:8080/path')", (1,)
|
||||
)
|
||||
cur.execute("INSERT INTO t_rw VALUES (:1, 'at 09:15:30')", (2,))
|
||||
cur.execute("SELECT id, s FROM t_rw ORDER BY id")
|
||||
assert cur.fetchall() == [
|
||||
(1, "http://host:8080/path"),
|
||||
(2, "at 09:15:30"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_colon_literals_round_trip_through_executemany(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""executemany rewrites unconditionally, so it had the bug too."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_rw2 (id INT, s VARCHAR(40))")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_rw2 VALUES (?, '16:9')", [(1,), (2,)]
|
||||
)
|
||||
cur.execute("SELECT DISTINCT s FROM t_rw2")
|
||||
assert cur.fetchall() == [("16:9",)]
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_comment_bearing_sql_still_binds(conn_params: ConnParams) -> None:
|
||||
"""A leading comment reaches PREPARE now that classification asks the
|
||||
server. Make sure the rewriter agrees and doesn't eat a placeholder
|
||||
that follows one."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"/* report: daily */ SELECT FIRST 1 tabid FROM systables "
|
||||
"WHERE tabid > :1",
|
||||
(0,),
|
||||
)
|
||||
assert cur.fetchone() is not None
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_placeholder_count_matches_after_rewrite(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The old regex could add placeholders the driver never counted,
|
||||
leaving num_qmarks (from len(params)) disagreeing with the SQL. A
|
||||
literal containing three colon-digit sequences is the shape that
|
||||
used to break it."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_rw3 (id INT, s VARCHAR(40))")
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DELETE FROM t_rw3")
|
||||
cur.execute(
|
||||
"INSERT INTO t_rw3 VALUES (:1, 'a:1 b:2 c:3')", (7,)
|
||||
)
|
||||
cur.execute("SELECT id, s FROM t_rw3")
|
||||
assert cur.fetchone() == (7, "a:1 b:2 c:3")
|
||||
459
tests/test_recovery_and_concurrency.py
Normal file
459
tests/test_recovery_and_concurrency.py
Normal file
@ -0,0 +1,459 @@
|
||||
"""Regression tests for error recovery, cursor lifecycle, and concurrency.
|
||||
|
||||
The type-matrix suite covers what a value looks like on the wire. This
|
||||
covers everything around it: what happens after a statement fails, what
|
||||
happens to abandoned cursors, and what happens when more than one thread
|
||||
or task is involved.
|
||||
|
||||
Two bugs here were found by fuzzing, and both are the kind that never
|
||||
show up in a single-threaded happy-path test:
|
||||
|
||||
* **A failed statement was never released.** Successful DML sent
|
||||
PREPARE → EXECUTE → RELEASE; failing DML sent PREPARE → EXECUTE and
|
||||
stopped. The leaked statement collided with the next PREPARE, and
|
||||
every subsequent call on that connection returned a nonsense error
|
||||
whose offset pointed back at the *failed* SQL. A duplicate-key
|
||||
violation — about the most ordinary error an application can hit —
|
||||
killed the connection outright.
|
||||
|
||||
* **Cancelling a pool acquire leaked the connection.** ``asyncio.to_thread``
|
||||
cannot interrupt its worker, so a cancelled waiter left the worker to
|
||||
finish and hand back a connection nobody owned. Under HTTP load, where
|
||||
client disconnects cancel request tasks precisely while they wait for
|
||||
a connection, the pool dies one slot at a time.
|
||||
|
||||
The tests that follow assert on *data* wherever they can, not merely the
|
||||
absence of an exception: a worker reads back a value only it wrote, so a
|
||||
crossed wire fails even when nothing raises.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db import aio
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
# Bounded: a leaked statement used to manifest as a hang.
|
||||
read_timeout=25.0,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
def _pool_kw(conn_params: ConnParams) -> dict:
|
||||
return {
|
||||
"host": conn_params.host,
|
||||
"port": conn_params.port,
|
||||
"user": conn_params.user,
|
||||
"password": conn_params.password,
|
||||
"database": conn_params.database,
|
||||
"server": conn_params.server,
|
||||
"connect_timeout": 15.0,
|
||||
"read_timeout": 30.0,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Error recovery — a failed statement must not poison the connection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
FAILING_SQL = [
|
||||
pytest.param("SELECT FROM WHERE", id="syntax"),
|
||||
pytest.param("SELECT * FROM no_such_table_xyz", id="no_such_table"),
|
||||
pytest.param("SELECT no_such_col FROM systables", id="no_such_column"),
|
||||
pytest.param("SELECT no_such_function_xyz(1) FROM systables", id="bad_func"),
|
||||
pytest.param("SELECT 1/0 FROM systables", id="div_zero"),
|
||||
pytest.param("SELECT 'notanumber'::INT FROM systables", id="bad_cast"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("sql", FAILING_SQL)
|
||||
def test_connection_survives_failed_statement(
|
||||
conn_params: ConnParams, sql: str
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
for _ in range(4): # repeat: a leak accumulates
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute(sql)
|
||||
cur.fetchall()
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables ORDER BY tabid")
|
||||
assert cur.fetchone() is not None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("autocommit", [True, False])
|
||||
def test_duplicate_key_does_not_kill_the_connection(
|
||||
conn_params: ConnParams, autocommit: bool
|
||||
) -> None:
|
||||
"""The bug that started this file. A unique-constraint violation left
|
||||
the prepared statement allocated server-side; the next statement then
|
||||
failed with a nonsense error (-255 "Not in transaction" in autocommit,
|
||||
-285 otherwise) and the connection never recovered."""
|
||||
with _connect(conn_params, autocommit=autocommit) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_dup (k INT PRIMARY KEY)")
|
||||
cur.execute("INSERT INTO t_dup VALUES (1)")
|
||||
if not autocommit:
|
||||
conn.commit()
|
||||
|
||||
for i in range(5):
|
||||
with pytest.raises(informix_db.IntegrityError):
|
||||
cur.execute("INSERT INTO t_dup VALUES (1)")
|
||||
if not autocommit:
|
||||
conn.rollback()
|
||||
cur.execute("SELECT k FROM t_dup")
|
||||
assert cur.fetchall() == [(1,)], f"broken after {i + 1} violations"
|
||||
|
||||
|
||||
def test_failed_dml_with_params_releases_statement(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_nn (s VARCHAR(8) NOT NULL, k INT)")
|
||||
for _ in range(4):
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("INSERT INTO t_nn VALUES (?, ?)", (None, 1))
|
||||
cur.execute("SELECT COUNT(*) FROM t_nn")
|
||||
assert cur.fetchone() == (0,)
|
||||
|
||||
|
||||
def test_new_cursor_works_after_error(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("SELECT * FROM no_such_table_xyz")
|
||||
other = conn.cursor()
|
||||
other.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert other.fetchone() is not None
|
||||
other.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fetch batching — NFETCH is a byte budget, so batch edges move with width
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [0, 1, 2, 3, 63, 64, 65, 199, 200, 201, 512, 1025])
|
||||
def test_row_count_exact_across_fetch_styles(
|
||||
conn_params: ConnParams, n: int
|
||||
) -> None:
|
||||
"""Every fetch style must agree, at counts that straddle plausible
|
||||
server batch boundaries. A batching off-by-one shows up as a wrong
|
||||
count, a duplicate, or a dropped row."""
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_batch "
|
||||
"(k INT, a VARCHAR(32), b INT, c VARCHAR(32), d INT)"
|
||||
)
|
||||
if n:
|
||||
cur.executemany(
|
||||
"INSERT INTO t_batch VALUES (?, ?, ?, ?, ?)",
|
||||
[(i, f"a{i}", i * 2, f"c{i}", i * 3) for i in range(n)],
|
||||
)
|
||||
want = [(i,) for i in range(n)]
|
||||
sql = "SELECT k FROM t_batch ORDER BY k"
|
||||
|
||||
cur.execute(sql)
|
||||
assert cur.fetchall() == want
|
||||
|
||||
cur.execute(sql)
|
||||
got = []
|
||||
while (row := cur.fetchone()) is not None:
|
||||
got.append(row)
|
||||
assert got == want
|
||||
|
||||
for size in (1, 7, 100):
|
||||
cur.execute(sql)
|
||||
got = []
|
||||
while chunk := cur.fetchmany(size):
|
||||
got.extend(chunk)
|
||||
assert got == want, f"fetchmany({size}) disagreed at n={n}"
|
||||
|
||||
cur.execute(sql)
|
||||
assert list(cur) == want
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cursor lifecycle
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_abandoned_partial_fetches_do_not_leak(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_aband (k INT)")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_aband VALUES (?)", [(i,) for i in range(200)]
|
||||
)
|
||||
for i in range(25):
|
||||
c = conn.cursor()
|
||||
c.execute("SELECT k FROM t_aband ORDER BY k")
|
||||
c.fetchone()
|
||||
if i % 2:
|
||||
c.close()
|
||||
else:
|
||||
del c # rely on the finalizer
|
||||
cur.execute("SELECT COUNT(*) FROM t_aband")
|
||||
assert cur.fetchone() == (200,)
|
||||
|
||||
|
||||
def test_re_execute_mid_fetch(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_reex (k INT)")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_reex VALUES (?)", [(i,) for i in range(100)]
|
||||
)
|
||||
for _ in range(10):
|
||||
cur.execute("SELECT k FROM t_reex ORDER BY k")
|
||||
cur.fetchone() # abandon mid-fetch
|
||||
cur.execute("SELECT k FROM t_reex ORDER BY k")
|
||||
assert len(cur.fetchall()) == 100
|
||||
|
||||
|
||||
def test_interleaved_cursors_on_one_connection(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_inter (k INT)")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_inter VALUES (?)", [(i,) for i in range(50)]
|
||||
)
|
||||
a, b = conn.cursor(), conn.cursor()
|
||||
a.execute("SELECT k FROM t_inter ORDER BY k")
|
||||
b.execute("SELECT k FROM t_inter ORDER BY k DESC")
|
||||
assert [a.fetchone() for _ in range(5)] == [(i,) for i in range(5)]
|
||||
assert [b.fetchone() for _ in range(5)] == [
|
||||
(i,) for i in range(49, 44, -1)
|
||||
]
|
||||
a.close()
|
||||
b.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Concurrency
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_threads_sharing_one_connection_do_not_interleave(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""One connection is one socket. Each thread reads back a value only
|
||||
it supplied, so crossed wires fail even if nothing raises."""
|
||||
failures: list[str] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
with _connect(conn_params, autocommit=True) as conn:
|
||||
def worker(tid: int) -> None:
|
||||
try:
|
||||
for r in range(10):
|
||||
cur = conn.cursor()
|
||||
token = tid * 1000 + r
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 ?::INT, ?::VARCHAR(16) FROM systables",
|
||||
(token, f"t{tid}"),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if row != (token, f"t{tid}"):
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: got {row!r}")
|
||||
return
|
||||
cur.close()
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
assert not failures, failures
|
||||
|
||||
|
||||
def test_pool_hands_out_clean_connections_under_load(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Borrowers that fail must still return a usable connection."""
|
||||
failures: list[str] = []
|
||||
lock = threading.Lock()
|
||||
pool = informix_db.create_pool(
|
||||
**_pool_kw(conn_params), min_size=2, max_size=4,
|
||||
acquire_timeout=30.0, autocommit=True,
|
||||
)
|
||||
try:
|
||||
def worker(tid: int) -> None:
|
||||
try:
|
||||
for r in range(8):
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
if r % 3 == 0:
|
||||
with contextlib.suppress(informix_db.Error):
|
||||
cur.execute("SELECT * FROM no_such_tbl_xyz")
|
||||
token = tid * 1000 + r
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 ?::INT FROM systables", (token,)
|
||||
)
|
||||
got = cur.fetchone()
|
||||
if got != (token,):
|
||||
with lock:
|
||||
failures.append(
|
||||
f"thread {tid}: got {got!r}, want {token}"
|
||||
)
|
||||
return
|
||||
cur.close()
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
pool.close()
|
||||
assert not failures, failures
|
||||
|
||||
|
||||
def test_pool_does_not_leak_transactions_between_borrowers(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""max_size=1 so both borrowers get the same underlying connection."""
|
||||
pool = informix_db.create_pool(
|
||||
**_pool_kw(conn_params), min_size=1, max_size=1,
|
||||
acquire_timeout=30.0, autocommit=False,
|
||||
)
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_pool_txn")
|
||||
cur.execute("CREATE TABLE t_pool_txn (k INT)")
|
||||
conn.commit()
|
||||
try:
|
||||
for i in range(5):
|
||||
with pool.connection() as conn: # dirties, never commits
|
||||
conn.cursor().execute(
|
||||
"INSERT INTO t_pool_txn VALUES (?)", (i,)
|
||||
)
|
||||
with pool.connection() as conn: # must not see it
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT COUNT(*) FROM t_pool_txn")
|
||||
assert cur.fetchone() == (0,), "transaction leaked"
|
||||
conn.rollback()
|
||||
finally:
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_pool_txn")
|
||||
conn.commit()
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
pool.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Async
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_async_pool_survives_cancelled_acquires(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Cancelling a task blocked on acquire used to destroy that pool slot
|
||||
permanently: ``asyncio.to_thread`` cannot interrupt its worker, so the
|
||||
worker finished and handed back a connection nobody owned.
|
||||
|
||||
Two holders keep both connections checked out; three waiters block on
|
||||
acquire and are cancelled there. Afterwards the pool must still serve.
|
||||
"""
|
||||
pool = await aio.create_pool(
|
||||
**_pool_kw(conn_params), min_size=1, max_size=2, autocommit=True
|
||||
)
|
||||
try:
|
||||
release = asyncio.Event()
|
||||
|
||||
async def holder(i: int) -> None:
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute(
|
||||
"SELECT FIRST 1 ?::INT FROM systables", (i,)
|
||||
)
|
||||
await cur.fetchone()
|
||||
await release.wait()
|
||||
|
||||
async def waiter(i: int) -> tuple | None:
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
await cur.execute(
|
||||
"SELECT FIRST 1 ?::INT FROM systables", (i,)
|
||||
)
|
||||
return await cur.fetchone()
|
||||
|
||||
for cycle in range(2):
|
||||
release.clear()
|
||||
holders = [asyncio.create_task(holder(i)) for i in range(2)]
|
||||
await asyncio.sleep(0.5) # both connections held
|
||||
waiters = [asyncio.create_task(waiter(100 + i)) for i in range(3)]
|
||||
await asyncio.sleep(0.5) # all blocked on acquire
|
||||
for t in waiters:
|
||||
t.cancel()
|
||||
await asyncio.gather(*waiters, return_exceptions=True)
|
||||
release.set()
|
||||
await asyncio.gather(*holders, return_exceptions=True)
|
||||
await asyncio.sleep(0.3) # orphan returns land
|
||||
|
||||
got = await asyncio.wait_for(waiter(200 + cycle), timeout=15)
|
||||
assert got == (200 + cycle,), (
|
||||
f"pool starved after cycle {cycle} — cancelled acquires leaked"
|
||||
)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
await pool.close()
|
||||
|
||||
|
||||
async def test_async_pool_concurrent_tasks_get_their_own_data(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
pool = await aio.create_pool(
|
||||
**_pool_kw(conn_params), min_size=2, max_size=4, autocommit=True
|
||||
)
|
||||
try:
|
||||
async def worker(tid: int) -> None:
|
||||
for r in range(6):
|
||||
async with pool.connection() as conn:
|
||||
cur = await conn.cursor()
|
||||
token = tid * 1000 + r
|
||||
await cur.execute(
|
||||
"SELECT FIRST 1 ?::INT FROM systables", (token,)
|
||||
)
|
||||
assert await cur.fetchone() == (token,)
|
||||
|
||||
await asyncio.gather(*(worker(i) for i in range(6)))
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
await pool.close()
|
||||
230
tests/test_row_reconciliation.py
Normal file
230
tests/test_row_reconciliation.py
Normal file
@ -0,0 +1,230 @@
|
||||
"""End-of-row reconciliation, and the two bugs it found immediately.
|
||||
|
||||
Fourteen framing bugs reached users before this check existed. Every one
|
||||
was the same defect — a column consuming the wrong number of bytes — and
|
||||
not one raised at the point of the mistake. The wrong width produced a
|
||||
plausible value and corrupted whatever came *next*, so the damage always
|
||||
surfaced in a different column, row, or statement.
|
||||
|
||||
All fourteen were detectable for free. ``payload`` is a fully extracted
|
||||
``bytes`` object of known length, so after decoding N columns the offset
|
||||
must land exactly on the end. ``_row_not_consumed`` raises when it
|
||||
doesn't, naming the column shape and the byte delta.
|
||||
|
||||
Turning the check on found two more bugs on its first run against the
|
||||
existing suite:
|
||||
|
||||
* **Smart LOBs were read as a flat 72-byte field.** They use the UDT
|
||||
envelope: 149 bytes populated (``[ind=0][len=144][144 hex chars]``),
|
||||
5 bytes NULL. We consumed 72 and left 77 behind, so any LOB not in the
|
||||
final column position corrupted everything after it. The 144 bytes are
|
||||
the 72-byte locator *hex-encoded*, which means ``BlobLocator.raw`` had
|
||||
never actually held a locator — only the first half of the hex text.
|
||||
Nobody noticed because ``read_blob_column`` resolves LOBs through the
|
||||
server-side ``lotofile`` function and never uses the locator.
|
||||
|
||||
* **NULL composite UDTs skipped their length field**, byte-for-byte the
|
||||
LVARCHAR NULL bug in the branch twelve lines away.
|
||||
|
||||
Both were the same ``[indicator][int32 length][data]`` envelope, written
|
||||
out by hand a third and fourth time. ``_read_udt_envelope`` is now the
|
||||
single implementation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import io
|
||||
import struct
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db._protocol import IfxStreamReader, ProtocolError
|
||||
from informix_db._resultset import (
|
||||
ColumnInfo,
|
||||
_read_udt_envelope,
|
||||
parse_tuple_payload,
|
||||
)
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The check itself — unit level, no server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _tuple_pdu(body: bytes) -> IfxStreamReader:
|
||||
"""Wrap a payload in the SQ_TUPLE framing parse_tuple_payload expects."""
|
||||
return IfxStreamReader(
|
||||
io.BytesIO(struct.pack("!h", 0) + struct.pack("!i", len(body)) + body)
|
||||
)
|
||||
|
||||
|
||||
def test_under_read_is_detected() -> None:
|
||||
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
|
||||
with pytest.raises(ProtocolError, match="under-read by 6"):
|
||||
parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5) + b"LEFTOV"), cols)
|
||||
|
||||
|
||||
def test_exact_consumption_passes() -> None:
|
||||
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
|
||||
assert parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5)), cols) == (5,)
|
||||
|
||||
|
||||
def test_error_names_the_column_shape() -> None:
|
||||
""""row decode failed" is not actionable. The message has to say which
|
||||
shape and by how much, so the next report arrives with the answer."""
|
||||
cols = [
|
||||
ColumnInfo(name="ident", type_code=2, raw_type_code=2, encoded_length=4),
|
||||
ColumnInfo(name="qty", type_code=2, raw_type_code=2, encoded_length=4),
|
||||
]
|
||||
body = struct.pack("!i", 1) + struct.pack("!i", 2) + b"UNCONSUMED"
|
||||
with pytest.raises(ProtocolError) as exc:
|
||||
parse_tuple_payload(_tuple_pdu(body), cols)
|
||||
msg = str(exc.value)
|
||||
assert "ident" in msg and "qty" in msg, "message must name the columns"
|
||||
assert "tc=2" in msg
|
||||
assert "consumed 8 of 18" in msg, "message must give the byte counts"
|
||||
assert "under-read by 10" in msg
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The shared UDT envelope
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_envelope_reads_length_even_when_null() -> None:
|
||||
"""The length belongs to the envelope, not the value. Returning early
|
||||
on the indicator is the bug that hit LVARCHAR and then composites."""
|
||||
payload = bytes([1]) + struct.pack("!i", 0) + b"NEXT"
|
||||
offset, body = _read_udt_envelope(payload, 0)
|
||||
assert body is None
|
||||
assert offset == 5, "NULL envelope must still consume its length field"
|
||||
assert payload[offset:] == b"NEXT"
|
||||
|
||||
|
||||
def test_envelope_reads_body_when_present() -> None:
|
||||
payload = bytes([0]) + struct.pack("!i", 3) + b"abc" + b"NEXT"
|
||||
offset, body = _read_udt_envelope(payload, 0)
|
||||
assert body == b"abc"
|
||||
assert offset == 8
|
||||
assert payload[offset:] == b"NEXT"
|
||||
|
||||
|
||||
def test_envelope_rejects_negative_length() -> None:
|
||||
"""A negative length off the wire would rewind the offset and silently
|
||||
re-decode earlier bytes as a later column."""
|
||||
payload = bytes([0]) + struct.pack("!i", -4) + b"abc"
|
||||
with pytest.raises(ProtocolError, match="negative length"):
|
||||
_read_udt_envelope(payload, 0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
pytestmark_integration = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host, port=conn_params.port, user=conn_params.user,
|
||||
password=conn_params.password, database=conn_params.database,
|
||||
server=conn_params.server, connect_timeout=15.0, read_timeout=30.0,
|
||||
autocommit=True,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_null_composite_does_not_shift_next_column(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""A NULL SET made the following VARCHAR return '' instead of its value."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_nullset")
|
||||
cur.execute(
|
||||
"CREATE TABLE t_nullset (a INT, s SET(INT NOT NULL), b VARCHAR(10))"
|
||||
)
|
||||
try:
|
||||
cur.execute("INSERT INTO t_nullset VALUES (7, SET{1,2}, 'abc')")
|
||||
cur.execute("INSERT INTO t_nullset VALUES (8, NULL, 'xyz')")
|
||||
cur.execute("SELECT a, s, b FROM t_nullset ORDER BY a")
|
||||
rows = cur.fetchall()
|
||||
assert rows[0][0] == 7 and rows[0][2] == "abc"
|
||||
assert rows[1] == (8, None, "xyz"), (
|
||||
"NULL composite shifted the following column"
|
||||
)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_nullset")
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.parametrize("populated", [False, True])
|
||||
def test_lob_in_non_final_position(
|
||||
conn_params: ConnParams, populated: bool
|
||||
) -> None:
|
||||
"""A LOB was read as 72 bytes when it occupies 149 (or 5 when NULL), so
|
||||
anything selected after it was garbage."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_lobmid")
|
||||
try:
|
||||
cur.execute(
|
||||
"CREATE TABLE t_lobmid (a INT, b BLOB, c INT, d VARCHAR(10))"
|
||||
)
|
||||
except informix_db.Error:
|
||||
pytest.skip("no sbspace configured; see make ifx-spaces")
|
||||
try:
|
||||
if populated:
|
||||
cur.write_blob_column(
|
||||
"INSERT INTO t_lobmid VALUES (?, BLOB_PLACEHOLDER, ?, ?)",
|
||||
b"hello", (1, 222222, "tail"),
|
||||
)
|
||||
else:
|
||||
cur.execute(
|
||||
"INSERT INTO t_lobmid VALUES (1, NULL, 222222, 'tail')"
|
||||
)
|
||||
cur.execute("SELECT a, b, c, d FROM t_lobmid")
|
||||
a, b, c, d = cur.fetchone()
|
||||
assert (a, c, d) == (1, 222222, "tail"), (
|
||||
"columns after the LOB were shifted"
|
||||
)
|
||||
if populated:
|
||||
assert isinstance(b, informix_db.BlobLocator)
|
||||
assert len(b.raw) == 72, (
|
||||
"locator must be the decoded 72 bytes, not hex text"
|
||||
)
|
||||
else:
|
||||
assert b is None
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_lobmid")
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_wide_mixed_row_reconciles(conn_params: ConnParams) -> None:
|
||||
"""Belt and braces: a row using most of the tricky types must consume
|
||||
its payload exactly. The check runs on every fetch, so merely getting
|
||||
a row back proves reconciliation held."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_recon ("
|
||||
" a INT8 NOT NULL, k LVARCHAR(512), n NCHAR(8), v VARCHAR(32),"
|
||||
" d DECIMAL(16), b BOOLEAN, t DATETIME YEAR TO FRACTION(5),"
|
||||
" tail INT)"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_recon VALUES (?, ?, ?, ?, ?, ?, CURRENT, ?)",
|
||||
(1, "PackageRoot", "nch", "v", None, True, 424242),
|
||||
)
|
||||
cur.execute(
|
||||
"SELECT a, k, n, v, d, b, t, tail FROM t_recon"
|
||||
)
|
||||
row = cur.fetchone()
|
||||
assert row[0] == 1
|
||||
assert row[-1] == 424242
|
||||
435
tests/test_rows.py
Normal file
435
tests/test_rows.py
Normal file
@ -0,0 +1,435 @@
|
||||
"""Named row access, and the cost of it.
|
||||
|
||||
A field request: `row[11]` tells a reader nothing on a wide projection,
|
||||
and `pyodbc` / `mssql-python` both hand back rows that answer to
|
||||
position, column name, and attribute at once. `row_factory=Row` gives
|
||||
the same three.
|
||||
|
||||
It is opt-in. Defaulting it on would tax the thing this driver is
|
||||
measured against, since supporting `row["name"]` means `__getitem__`
|
||||
becomes a Python method rather than C-level tuple indexing, which costs
|
||||
roughly 39 ns on every subscript. Users who want readable column access
|
||||
in application code should pay that; a bulk export that never looks at a
|
||||
column by name should not.
|
||||
|
||||
`Row` subclasses `tuple`, so `row == (1, "x")` still holds and nothing
|
||||
that already worked stops working. That constraint drove the design: the
|
||||
existing suite compares fetched rows against plain tuples in hundreds of
|
||||
places, and so does everybody's code.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import pickle
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.rows import _RESERVED, Row, make_row_class
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The type itself
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _row(fields, values):
|
||||
return make_row_class(tuple(fields))(values)
|
||||
|
||||
|
||||
def test_three_ways_to_reach_a_column() -> None:
|
||||
row = _row(("tabid", "tabname"), (1, "systables"))
|
||||
assert row[0] == 1
|
||||
assert row["tabname"] == "systables"
|
||||
assert row.tabname == "systables"
|
||||
|
||||
|
||||
def test_still_equal_to_a_plain_tuple() -> None:
|
||||
"""The compatibility constraint. Existing code and the existing test
|
||||
suite compare fetched rows against tuples everywhere."""
|
||||
row = _row(("a", "b"), (1, "x"))
|
||||
plain = (1, "x")
|
||||
assert row == plain
|
||||
# Both directions: tuple.__eq__ on the left has to accept a subclass
|
||||
# on the right, or `expected == fetched` assertions break.
|
||||
assert plain == row
|
||||
assert list(row) == [1, "x"]
|
||||
assert len(row) == 2
|
||||
a, b = row
|
||||
assert (a, b) == (1, "x")
|
||||
assert row in [(1, "x")]
|
||||
|
||||
|
||||
def test_slice_degrades_to_a_plain_tuple() -> None:
|
||||
"""A slice has no meaningful column mapping, so it should not pretend
|
||||
to be a Row."""
|
||||
row = _row(("a", "b", "c"), (1, 2, 3))
|
||||
assert row[0:2] == (1, 2)
|
||||
assert type(row[0:2]) is tuple
|
||||
|
||||
|
||||
def test_negative_index_still_works() -> None:
|
||||
assert _row(("a", "b"), (1, 2))[-1] == 2
|
||||
|
||||
|
||||
def test_keys_and_asdict() -> None:
|
||||
row = _row(("a", "b"), (1, "x"))
|
||||
assert row.keys() == ("a", "b")
|
||||
assert row._asdict() == {"a": 1, "b": "x"}
|
||||
|
||||
|
||||
def test_repr_names_the_columns() -> None:
|
||||
assert repr(_row(("a", "b"), (1, "x"))) == "Row(a=1, b='x')"
|
||||
|
||||
|
||||
def test_pickles() -> None:
|
||||
"""The per-shape class is built at runtime, so it cannot be pickled by
|
||||
reference. Multiprocessing users would hit that immediately."""
|
||||
row = _row(("a", "b"), (1, "x"))
|
||||
restored = pickle.loads(pickle.dumps(row))
|
||||
assert restored == (1, "x")
|
||||
assert restored.b == "x"
|
||||
|
||||
|
||||
def test_missing_column_says_what_is_there() -> None:
|
||||
row = _row(("tabid", "tabname"), (1, "x"))
|
||||
with pytest.raises(KeyError, match="tabid"):
|
||||
_ = row["nope"]
|
||||
with pytest.raises(AttributeError, match="tabid"):
|
||||
_ = row.nope
|
||||
|
||||
|
||||
def test_class_is_cached_per_shape() -> None:
|
||||
"""A type() call per execute() would land on every small query."""
|
||||
assert make_row_class(("a", "b")) is make_row_class(("a", "b"))
|
||||
assert make_row_class(("a", "b")) is not make_row_class(("a", "c"))
|
||||
|
||||
|
||||
def test_duplicate_names_resolve_to_the_first() -> None:
|
||||
"""``SELECT tabid, tabid`` is legal and both columns are named
|
||||
``tabid``. pyodbc gives the first; so do we."""
|
||||
row = _row(("tabid", "tabid"), (1, 2))
|
||||
assert row["tabid"] == 1
|
||||
assert row.tabid == 1
|
||||
assert row[1] == 2, "positional access must still reach the second"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", sorted(_RESERVED))
|
||||
def test_a_column_always_beats_a_method_of_the_same_name(name: str) -> None:
|
||||
"""Parametrized over the *computed* reserved set rather than a
|
||||
hand-written list, because the hand-written list is what missed
|
||||
``keys``, ``_asdict`` and ``_fields`` on the first attempt. Adding a
|
||||
method to Row later cannot silently reopen the hole: this test grows
|
||||
with it.
|
||||
|
||||
Without the guard these return a bound method, which is the exact
|
||||
failure shape this driver spent a fortnight removing from its
|
||||
decoders: a plausible-looking wrong answer, in silence."""
|
||||
row = _row((name, "other"), (7, 9))
|
||||
assert getattr(row, name) == 7
|
||||
assert row[name] == 7
|
||||
|
||||
|
||||
def test_shadowed_methods_stay_reachable_through_the_base_class() -> None:
|
||||
row = _row(("count", "keys", "_asdict"), (1, 2, 3))
|
||||
assert tuple.count(row, 1) == 1
|
||||
assert Row.keys(row) == ("count", "keys", "_asdict")
|
||||
assert Row._asdict(row) == {"count": 1, "keys": 2, "_asdict": 3}
|
||||
|
||||
|
||||
def test_shadowing_fields_does_not_break_repr_or_asdict() -> None:
|
||||
"""The machinery reads its own field list, so if a column named
|
||||
``_fields`` shadowed it, repr and _asdict would report the column
|
||||
value instead of the names. Mangled attributes keep them separate."""
|
||||
row = _row(("_fields", "x"), ("not the names", 2))
|
||||
assert row._fields == "not the names"
|
||||
assert Row.keys(row) == ("_fields", "x")
|
||||
assert repr(row) == "Row(_fields='not the names', x=2)"
|
||||
|
||||
|
||||
def test_non_identifier_names_are_subscript_only() -> None:
|
||||
"""Informix names expression columns things like ``(count(*))``,
|
||||
which cannot be an attribute."""
|
||||
row = _row(("(count(*))", "ok"), (5, 1))
|
||||
assert row["(count(*))"] == 5
|
||||
assert row.ok == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_default_is_still_a_plain_tuple(conn_params: ConnParams) -> None:
|
||||
"""The opt-in has to be genuinely opt-in. Anyone who does not ask for
|
||||
Row should not pay for it or see any behaviour change."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT FIRST 1 tabid, tabname FROM systables")
|
||||
row = cur.fetchone()
|
||||
assert type(row) is tuple
|
||||
with pytest.raises(TypeError):
|
||||
_ = row["tabname"]
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_row_factory_on_the_connection(conn_params: ConnParams) -> None:
|
||||
"""One line at connect time, then every cursor from it, which is what
|
||||
'without any additional coding' has to mean in practice."""
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 tabid, tabname FROM systables ORDER BY tabid"
|
||||
)
|
||||
row = cur.fetchone()
|
||||
assert row[0] == row["tabid"] == row.tabid
|
||||
assert row[1] == row["tabname"] == row.tabname
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_every_fetch_path_returns_rows(conn_params: ConnParams) -> None:
|
||||
"""fetchone, fetchmany, fetchall and iteration each return rows by a
|
||||
different route through the cursor."""
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
sql = "SELECT FIRST 4 tabid, tabname FROM systables ORDER BY tabid"
|
||||
cur = conn.cursor()
|
||||
|
||||
cur.execute(sql)
|
||||
assert cur.fetchone().tabname is not None
|
||||
|
||||
cur.execute(sql)
|
||||
assert all(r.tabname is not None for r in cur.fetchmany(2))
|
||||
|
||||
cur.execute(sql)
|
||||
assert all(r.tabname is not None for r in cur.fetchall())
|
||||
|
||||
cur.execute(sql)
|
||||
assert all(r.tabname is not None for r in cur)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_scrollable_cursor_returns_rows(conn_params: ConnParams) -> None:
|
||||
"""Scrollable cursors return each row straight from the wire rather
|
||||
than from the materialized list, so they are a separate path."""
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
cur = conn.cursor(scrollable=True)
|
||||
cur.execute("SELECT tabid, tabname FROM systables ORDER BY tabid")
|
||||
assert cur.fetch_first().tabname is not None
|
||||
assert cur.fetch_absolute(1).tabname is not None
|
||||
cur.close()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_per_cursor_override(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
named = conn.cursor()
|
||||
named.row_factory = informix_db.Row
|
||||
plain = conn.cursor()
|
||||
sql = "SELECT FIRST 1 tabid FROM systables"
|
||||
named.execute(sql)
|
||||
plain.execute(sql)
|
||||
assert isinstance(named.fetchone(), Row)
|
||||
assert type(plain.fetchone()) is tuple
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_informix_lowercases_so_the_obvious_name_works(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Informix folds unquoted identifiers, which is why the .lower() in
|
||||
the workaround people write by hand is a no-op."""
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_rows (Config_Key INT, Cnt INT)")
|
||||
cur.execute("INSERT INTO t_rows VALUES (1, 2)")
|
||||
cur.execute("SELECT Config_Key, Cnt FROM t_rows")
|
||||
row = cur.fetchone()
|
||||
assert row.config_key == 1
|
||||
assert row.cnt == 2
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_pool_forwards_the_factory(conn_params: ConnParams) -> None:
|
||||
pool = informix_db.create_pool(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
autocommit=True,
|
||||
row_factory=informix_db.Row,
|
||||
min_size=1,
|
||||
max_size=2,
|
||||
)
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone().tabid is not None
|
||||
finally:
|
||||
pool.close()
|
||||
|
||||
|
||||
def test_zero_column_row() -> None:
|
||||
row = make_row_class(())(())
|
||||
assert row == ()
|
||||
assert row.keys() == ()
|
||||
assert repr(row) == "Row()"
|
||||
|
||||
|
||||
def test_wide_row() -> None:
|
||||
"""500 columns: the name map is a dict, so this should be flat, but
|
||||
it is the shape most likely to expose an off-by-one."""
|
||||
fields = tuple(f"c{i}" for i in range(500))
|
||||
row = make_row_class(fields)(tuple(range(500)))
|
||||
assert row["c499"] == row.c499 == row[499] == 499
|
||||
assert row["c0"] == row[0] == 0
|
||||
|
||||
|
||||
def test_names_that_cannot_be_attributes_are_subscript_only() -> None:
|
||||
row = _row(("col with space", "1leading_digit", ""), (1, 2, 3))
|
||||
assert row["col with space"] == 1
|
||||
assert row["1leading_digit"] == 2
|
||||
assert row[""] == 3
|
||||
|
||||
|
||||
def test_unicode_column_name() -> None:
|
||||
row = _row(("café", "x"), (1, 2))
|
||||
assert row["café"] == 1
|
||||
assert row.café == 1
|
||||
|
||||
|
||||
def test_dunder_named_column_is_subscript_only() -> None:
|
||||
"""A dunder cannot be shadowed without breaking the object protocol,
|
||||
so the column is reachable by subscript and the attribute keeps its
|
||||
ordinary meaning. No real schema should notice."""
|
||||
row = _row(("__class__", "ok"), (1, 2))
|
||||
assert row["__class__"] == 1
|
||||
assert row.__class__.__name__ == "Row"
|
||||
|
||||
|
||||
def test_row_outlives_eviction_of_its_class_from_the_cache() -> None:
|
||||
"""The class cache is bounded. A row already handed to the caller
|
||||
holds its class alive, so eviction must not affect it."""
|
||||
row = make_row_class(("z1", "z2"))((9, 8))
|
||||
for i in range(300):
|
||||
make_row_class((f"evict{i}", "b"))
|
||||
assert (row.z1, row["z2"], row[0]) == (9, 8, 9)
|
||||
|
||||
|
||||
def test_class_cache_is_bounded() -> None:
|
||||
for i in range(400):
|
||||
make_row_class((f"bounded{i}", "b"))
|
||||
assert make_row_class.cache_info().currsize <= 256
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Row must not change a single value
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_WIDE_DDL = """CREATE TABLE t_rowdiff (
|
||||
c_int INT, c_small SMALLINT, c_big BIGINT, c_int8 INT8, c_serial SERIAL,
|
||||
c_float FLOAT, c_smallfloat SMALLFLOAT, c_dec DECIMAL(16,4),
|
||||
c_decu DECIMAL(16), c_money MONEY(12,2), c_char CHAR(10),
|
||||
c_vchar VARCHAR(40), c_nchar NCHAR(8), c_lvar LVARCHAR(200), c_date DATE,
|
||||
c_dt DATETIME YEAR TO FRACTION(5), c_ivl INTERVAL YEAR TO MONTH,
|
||||
c_bool BOOLEAN, c_set SET(INT NOT NULL), c_tail INT)"""
|
||||
|
||||
_WIDE_COLS = (
|
||||
"c_int,c_small,c_big,c_int8,c_serial,c_float,c_smallfloat,c_dec,c_decu,"
|
||||
"c_money,c_char,c_vchar,c_nchar,c_lvar,c_date,c_dt,c_ivl,c_bool,c_set,"
|
||||
"c_tail"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_rows_are_value_identical_to_tuples(conn_params: ConnParams) -> None:
|
||||
"""The strongest statement available: across every awkward type, with
|
||||
a fully populated row and a fully NULL one, wrapping must change
|
||||
nothing. If it does, Row is not a presentation layer, it is a bug."""
|
||||
sql = f"SELECT {_WIDE_COLS} FROM t_rowdiff ORDER BY c_tail"
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_rowdiff")
|
||||
cur.execute(_WIDE_DDL)
|
||||
try:
|
||||
cur.execute(
|
||||
"INSERT INTO t_rowdiff VALUES (1,2,3,4,0,1.5,2.5,12.34,99,"
|
||||
"9.99,'ch','vc','nc','lv',TODAY,CURRENT,"
|
||||
"INTERVAL(1-2) YEAR TO MONTH,'t',SET{1,2},77)"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_rowdiff VALUES (NULL,NULL,NULL,NULL,0,NULL,"
|
||||
"NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,"
|
||||
"NULL,NULL,88)"
|
||||
)
|
||||
cur.execute(sql)
|
||||
plain = cur.fetchall()
|
||||
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as named_conn:
|
||||
named = named_conn.cursor()
|
||||
named.execute(sql)
|
||||
rows = named.fetchall()
|
||||
|
||||
assert rows == plain, "wrapping changed a value"
|
||||
names = _WIDE_COLS.split(",")
|
||||
first = rows[0]
|
||||
for i, name in enumerate(names):
|
||||
assert first[i] == first[name] == getattr(first, name), name
|
||||
assert rows[1]["c_int"] is None
|
||||
assert rows[1].c_tail == 88
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_rowdiff")
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_threads_share_one_class_per_shape(conn_params: ConnParams) -> None:
|
||||
"""The cache is process-wide and the pool hands connections to many
|
||||
threads, so two threads running the same query must land on the same
|
||||
class rather than racing to build competing ones."""
|
||||
import threading
|
||||
|
||||
seen: list[type] = []
|
||||
errors: list[Exception] = []
|
||||
|
||||
def worker() -> None:
|
||||
try:
|
||||
with _connect(conn_params, row_factory=informix_db.Row) as conn:
|
||||
cur = conn.cursor()
|
||||
for _ in range(5):
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 tabid, tabname FROM systables"
|
||||
)
|
||||
row = cur.fetchone()
|
||||
assert row.tabname == row["tabname"] == row[1]
|
||||
seen.append(type(row))
|
||||
except Exception as exc:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker) for _ in range(6)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(60)
|
||||
assert not errors, errors[:2]
|
||||
assert len({id(c) for c in seen}) == 1, "same shape built more than once"
|
||||
172
tests/test_socket_reads.py
Normal file
172
tests/test_socket_reads.py
Normal file
@ -0,0 +1,172 @@
|
||||
"""Two readers, one stream, and a length field taken on trust.
|
||||
|
||||
``IfxSocket`` owns a read-ahead buffer that ``BufferedSocketReader``
|
||||
fills and drains. But ``Connection._drain_to_eot``, ``_raise_sq_err``
|
||||
and the login path bypass that reader and call ``IfxSocket.read_exact``
|
||||
directly, which recv'd from the socket without ever looking at the
|
||||
buffer. Bytes sitting in the buffer would simply be skipped, and skipped
|
||||
bytes in a length-framed protocol don't announce themselves — the next
|
||||
read lands mid-field and every read after it is wrong.
|
||||
|
||||
Nothing triggers it today. The server sends one response per request, so
|
||||
recv returns exactly that response and the buffered reader consumes all
|
||||
of it before control returns to a direct read. That is a property of the
|
||||
traffic, not of the code, and the buffer is connection-scoped precisely
|
||||
so read-ahead *can* cross response boundaries — pipelined executemany
|
||||
already puts several responses in flight. A latent desync waiting on a
|
||||
timing change is not a good thing to leave in a wire protocol.
|
||||
|
||||
Separately, ``fill_recv_buf`` took its byte count on trust, and that
|
||||
count is almost always a length field straight off the wire. A corrupt
|
||||
or desynced stream turned into an allocation of whatever the field
|
||||
happened to say: a garbage ``0x7FFFFFFF`` reads as a 2 GB request, and
|
||||
the fill loop sits in recv until the read timeout while the buffer
|
||||
grows. The limit turns that into an error that names the number, which
|
||||
is the actual diagnostic — a length that absurd means framing was
|
||||
already lost upstream.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from informix_db._protocol import BufferedSocketReader, ProtocolError
|
||||
from informix_db._socket import MAX_READ_BYTES, IfxSocket
|
||||
|
||||
|
||||
class _FakeSocket:
|
||||
"""Stands in for the raw socket. Records what recv actually asked for."""
|
||||
|
||||
def __init__(self, data: bytes = b"") -> None:
|
||||
self.data = data
|
||||
self.pos = 0
|
||||
self.recv_calls: list[int] = []
|
||||
|
||||
def recv(self, n: int) -> bytes:
|
||||
self.recv_calls.append(n)
|
||||
chunk = self.data[self.pos : self.pos + n]
|
||||
self.pos += len(chunk)
|
||||
return chunk
|
||||
|
||||
def close(self) -> None:
|
||||
# The EOF path force-closes; a stand-in has to survive that.
|
||||
pass
|
||||
|
||||
|
||||
def _socket_with(buffered: bytes, on_wire: bytes = b"") -> IfxSocket:
|
||||
"""An IfxSocket with ``buffered`` already read ahead into _recv_buf."""
|
||||
sock = IfxSocket.__new__(IfxSocket)
|
||||
sock._sock = _FakeSocket(on_wire)
|
||||
sock._recv_buf = bytearray(buffered)
|
||||
sock._recv_pos = 0
|
||||
sock._recv_size = 65536
|
||||
sock._read_timeout = None
|
||||
return sock
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# read_exact must not step over the buffer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_read_exact_consumes_the_buffer_first() -> None:
|
||||
sock = _socket_with(b"BUFFERED", on_wire=b"SOCKET")
|
||||
assert sock.read_exact(8) == b"BUFFERED"
|
||||
assert sock._sock.recv_calls == [], "must not touch the socket at all"
|
||||
assert sock._recv_pos == 8
|
||||
|
||||
|
||||
def test_read_exact_spans_buffer_then_socket() -> None:
|
||||
"""The interesting case: a read that starts in the buffer and
|
||||
finishes on the wire. Getting this wrong reorders the stream."""
|
||||
sock = _socket_with(b"HEAD", on_wire=b"TAIL")
|
||||
assert sock.read_exact(8) == b"HEADTAIL"
|
||||
assert sock._sock.recv_calls == [4], "only the shortfall comes from recv"
|
||||
|
||||
|
||||
def test_read_exact_respects_a_partly_consumed_buffer() -> None:
|
||||
sock = _socket_with(b"XXABCD")
|
||||
sock._recv_pos = 2 # first two bytes already decoded
|
||||
assert sock.read_exact(4) == b"ABCD"
|
||||
assert sock._sock.recv_calls == []
|
||||
|
||||
|
||||
def test_read_exact_of_zero_is_empty() -> None:
|
||||
sock = _socket_with(b"DATA")
|
||||
assert sock.read_exact(0) == b""
|
||||
assert sock.read_exact(-5) == b"", "a negative count must not rewind"
|
||||
assert sock._recv_pos == 0
|
||||
|
||||
|
||||
def test_short_read_error_reports_the_original_request() -> None:
|
||||
"""The message counts bytes; taking some from the buffer must not make
|
||||
it lie about how many were asked for."""
|
||||
from informix_db.exceptions import OperationalError
|
||||
|
||||
sock = _socket_with(b"AB", on_wire=b"") # 2 buffered, nothing on the wire
|
||||
with pytest.raises(OperationalError, match="wanted 10 bytes"):
|
||||
sock.read_exact(10)
|
||||
|
||||
|
||||
def test_buffered_reader_and_direct_read_agree_on_one_stream() -> None:
|
||||
"""End to end: a BufferedSocketReader over-reads, then a direct
|
||||
read_exact picks up exactly where it left off."""
|
||||
sock = _socket_with(b"", on_wire=b"\x00\x2aREST-OF-THE-STREAM")
|
||||
reader = BufferedSocketReader(sock)
|
||||
assert reader.read_short() == 42
|
||||
assert len(sock._recv_buf) - sock._recv_pos > 0, (
|
||||
"precondition: the reader must have over-read for this to mean "
|
||||
"anything"
|
||||
)
|
||||
assert sock.read_exact(18) == b"REST-OF-THE-STREAM"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# fill_recv_buf must not believe an arbitrary length
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_absurd_length_is_refused_not_allocated() -> None:
|
||||
sock = _socket_with(b"", on_wire=b"")
|
||||
with pytest.raises(ProtocolError, match="refusing to read"):
|
||||
sock.fill_recv_buf(MAX_READ_BYTES + 1)
|
||||
assert sock._sock.recv_calls == [], "must refuse before any recv"
|
||||
|
||||
|
||||
def test_refusal_names_the_knob() -> None:
|
||||
"""The error has to be actionable in both directions: framing is lost,
|
||||
or the value genuinely is that big and the limit needs raising."""
|
||||
sock = _socket_with(b"", on_wire=b"")
|
||||
with pytest.raises(ProtocolError) as exc:
|
||||
sock.fill_recv_buf(2**31 - 1)
|
||||
message = str(exc.value)
|
||||
assert "2147483647" in message
|
||||
assert "IFX_MAX_READ_BYTES" in message
|
||||
|
||||
|
||||
def test_a_normal_length_is_unaffected() -> None:
|
||||
sock = _socket_with(b"", on_wire=b"x" * 100)
|
||||
sock.fill_recv_buf(100)
|
||||
assert len(sock._recv_buf) - sock._recv_pos >= 100
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# skip
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_buffered_skip_does_not_rewind_on_a_negative_count() -> None:
|
||||
"""The base reader's skip delegates to read_exact, which guards. This
|
||||
one advances the cursor arithmetically, so an unguarded negative count
|
||||
re-decodes bytes already consumed as if they were the next field."""
|
||||
sock = _socket_with(b"ABCDEFGH")
|
||||
sock._recv_pos = 4
|
||||
BufferedSocketReader(sock).skip(-4)
|
||||
assert sock._recv_pos == 4, "skip must never move the cursor backwards"
|
||||
|
||||
|
||||
def test_buffered_skip_advances_normally() -> None:
|
||||
sock = _socket_with(b"ABCDEFGH")
|
||||
reader = BufferedSocketReader(sock)
|
||||
reader.skip(4)
|
||||
assert reader.read_exact(4) == b"EFGH"
|
||||
216
tests/test_statement_classification.py
Normal file
216
tests/test_statement_classification.py
Normal file
@ -0,0 +1,216 @@
|
||||
"""Deciding whether a statement needs a cursor, by asking rather than guessing.
|
||||
|
||||
The driver chose between "open a cursor and fetch" and "execute and
|
||||
release" by checking whether the first word of the SQL was ``SELECT``.
|
||||
That gets five ordinary forms wrong — a leading comment in any of the
|
||||
three Informix flavours, a parenthesized select, a parenthesized UNION,
|
||||
and a CTE. All five are perfectly good queries, and all five failed with
|
||||
``-260 Cursor name already in use``, an error that describes neither the
|
||||
cause nor anything the caller did. It says "cursor" because the driver
|
||||
sent SQ_EXECUTE where the server was waiting to open one.
|
||||
|
||||
The server had been telling us the answer the whole time.
|
||||
``statement_type`` is the first field of the DESCRIBE response, and
|
||||
``parse_describe`` has always parsed it into the metadata dict, where
|
||||
nothing read it. Every SELECT form above reports 2.
|
||||
|
||||
The comment that justified the heuristic said ``nfields`` couldn't
|
||||
distinguish these cases, because ``INSERT INTO t VALUES (?)`` also
|
||||
describes a column. That much was true — and it argued for the wrong
|
||||
conclusion, because ``statement_type`` distinguishes them exactly. That
|
||||
INSERT reports 6.
|
||||
|
||||
The predicate is now JDBC's ``IfxSqli.isResultSet``: type 2, or type 56
|
||||
(``EXECUTE PROCEDURE``/``FUNCTION``) with at least one column, since a
|
||||
routine may or may not return rows. That last clause fixes
|
||||
``EXECUTE FUNCTION`` as a side effect — it used to run down the DML path
|
||||
and discard its return value.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _produces_result_set
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The predicate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("statement_type", "ncolumns", "expected", "why"),
|
||||
[
|
||||
(2, 1, True, "SELECT"),
|
||||
(2, 0, True, "SELECT describing no columns is still a SELECT"),
|
||||
(6, 1, False, "INSERT ... VALUES (?) describes a column but is DML"),
|
||||
(6, 0, False, "INSERT with literals"),
|
||||
(32, 0, False, "DELETE"),
|
||||
(33, 0, False, "UPDATE"),
|
||||
(45, 0, False, "CREATE"),
|
||||
(56, 0, False, "EXECUTE PROCEDURE returning nothing"),
|
||||
(56, 2, True, "EXECUTE FUNCTION returning rows"),
|
||||
(0, 0, False, "unknown type defaults to the non-cursor path"),
|
||||
],
|
||||
)
|
||||
def test_result_set_predicate(
|
||||
statement_type: int, ncolumns: int, expected: bool, why: str
|
||||
) -> None:
|
||||
assert _produces_result_set(statement_type, ncolumns) is expected, why
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
)
|
||||
|
||||
|
||||
_ONE = "SELECT FIRST 1 tabid FROM systables"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.parametrize(
|
||||
("label", "sql"),
|
||||
[
|
||||
pytest.param("plain", _ONE, id="plain"),
|
||||
pytest.param("lowercase", _ONE.lower(), id="lowercase"),
|
||||
pytest.param("leading-whitespace", f" \n\t {_ONE}", id="whitespace"),
|
||||
pytest.param("line-comment", f"-- pick one\n{_ONE}", id="line-comment"),
|
||||
pytest.param("block-comment", f"/* pick one */ {_ONE}", id="block-comment"),
|
||||
pytest.param("brace-comment", f"{{ pick one }} {_ONE}", id="brace-comment"),
|
||||
pytest.param(
|
||||
"cte",
|
||||
"WITH c AS (SELECT tabid FROM systables) "
|
||||
"SELECT FIRST 1 tabid FROM c",
|
||||
id="cte",
|
||||
),
|
||||
pytest.param("parenthesized", f"({_ONE})", id="parenthesized"),
|
||||
pytest.param(
|
||||
"union", f"{_ONE} UNION SELECT 99 FROM systables", id="union"
|
||||
),
|
||||
pytest.param(
|
||||
"parenthesized-union",
|
||||
f"({_ONE}) UNION (SELECT 99 FROM systables)",
|
||||
id="paren-union",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_every_select_form_opens_a_cursor(
|
||||
conn_params: ConnParams, label: str, sql: str
|
||||
) -> None:
|
||||
"""The five non-``plain`` forms below the whitespace case all failed
|
||||
with -260 under the first-word heuristic."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
try:
|
||||
cur.execute(sql)
|
||||
except informix_db.ProgrammingError as exc:
|
||||
# Informix 12.10 has no CTEs and rejects WITH at offset 1.
|
||||
# A syntax error is the server declining the grammar, which
|
||||
# is a different thing from the driver routing it wrongly —
|
||||
# that produced -260, not -201.
|
||||
if getattr(exc, "sqlcode", None) == -201:
|
||||
pytest.skip(f"server does not support this syntax: {label}")
|
||||
raise
|
||||
rows = cur.fetchall()
|
||||
assert rows, f"{label}: expected rows, got none"
|
||||
assert cur.description is not None
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_parameterized_insert_is_not_mistaken_for_a_query(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The case the old comment worried about, and the reason it kept the
|
||||
heuristic: this DESCRIBEs a column. ``statement_type`` says 6."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_cls (k INT)")
|
||||
cur.execute("INSERT INTO t_cls VALUES (?)", (7,))
|
||||
assert cur.rowcount == 1
|
||||
cur.execute("SELECT k FROM t_cls")
|
||||
assert cur.fetchall() == [(7,)]
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_execute_function_returns_its_value(conn_params: ConnParams) -> None:
|
||||
"""Type 56 with columns. Under the first-word heuristic this ran down
|
||||
the DML path and the return value was discarded."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP FUNCTION ifxdrv_dbl")
|
||||
cur.execute(
|
||||
"CREATE FUNCTION ifxdrv_dbl(n INT) RETURNING INT; "
|
||||
"RETURN n * 2; END FUNCTION"
|
||||
)
|
||||
try:
|
||||
cur.execute("EXECUTE FUNCTION ifxdrv_dbl(21)")
|
||||
assert cur.fetchall() == [(42,)]
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP FUNCTION ifxdrv_dbl")
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_dml_still_takes_the_non_cursor_path(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_cls2 (k INT)")
|
||||
cur.execute("INSERT INTO t_cls2 VALUES (1)")
|
||||
cur.execute("UPDATE t_cls2 SET k = 2")
|
||||
assert cur.rowcount == 1
|
||||
cur.execute("DELETE FROM t_cls2")
|
||||
assert cur.rowcount == 1
|
||||
assert cur.description is None
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_executemany_refuses_a_query_the_first_word_missed(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The pre-flight check shares the heuristic's blind spots, so a
|
||||
comment-prefixed SELECT reaches PREPARE. The post-DESCRIBE check
|
||||
catches it, and the connection stays usable — the refusal releases
|
||||
the statement. (A leading comment rather than a CTE, so this also
|
||||
runs on 12.10, which has no CTEs.)"""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with pytest.raises(informix_db.NotSupportedError):
|
||||
cur.executemany(
|
||||
"/* batched? no */ SELECT FIRST 1 tabid FROM systables "
|
||||
"WHERE tabid <> ?",
|
||||
[(1,), (2,)],
|
||||
)
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None, "refusal leaked the statement"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_executemany_still_refuses_a_plain_select(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
with pytest.raises(informix_db.NotSupportedError):
|
||||
cur.executemany(
|
||||
"SELECT FIRST 1 tabid FROM systables WHERE tabid <> ?",
|
||||
[(1,), (2,)],
|
||||
)
|
||||
189
tests/test_statement_release.py
Normal file
189
tests/test_statement_release.py
Normal file
@ -0,0 +1,189 @@
|
||||
"""Every door out of a statement has to release it on the way through.
|
||||
|
||||
A statement that fails is still allocated server-side. Skipping the
|
||||
RELEASE bricks the connection: the next PREPARE collides with the leaked
|
||||
one, and every subsequent call returns a nonsense error whose offset
|
||||
points back at the *failed* SQL rather than the new statement.
|
||||
|
||||
There are six exits from the execute paths, and the guard was added to
|
||||
them one at a time, each after a user hit it:
|
||||
|
||||
1. ``_execute_dml`` drain
|
||||
2. ``_execute_dml_with_params`` build
|
||||
3. ``_execute_dml_with_params`` drain
|
||||
4. ``executemany`` pipeline build/send
|
||||
5. ``_execute_select_with_params`` build
|
||||
6. ``_execute_select`` fetch loop
|
||||
|
||||
Two were still open, and both are ordinary to reach:
|
||||
|
||||
* **The parameterized-SELECT bind drain.** The build was guarded and the
|
||||
drain was not. Passing a string where the column is an INT gets a
|
||||
clean encode, so the rejection comes from the *server* (-1213, -415)
|
||||
during the bind drain — past the guard. A wrong-typed parameter is
|
||||
about as common as application mistakes get.
|
||||
|
||||
* **The scrollable-cursor open.** No guard at all, and the worst place
|
||||
to lack one: the GC-time finalizer is armed on the line *after* the
|
||||
drain, so a failure there left the statement allocated with no
|
||||
fallback of any kind.
|
||||
|
||||
The guard is now one helper rather than six hand-rolled copies, which
|
||||
also fixed a defect in the copies that had a cursor to close: CLOSE and
|
||||
RELEASE shared a single ``contextlib.suppress`` block, so a CLOSE that
|
||||
raised skipped the RELEASE — losing the half that actually matters.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _RELEASE_PDU, Cursor
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The helper — no server needed
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeConn:
|
||||
def __init__(self) -> None:
|
||||
self.sent: list[bytes] = []
|
||||
|
||||
def _send_pdu(self, pdu: bytes) -> None:
|
||||
self.sent.append(pdu)
|
||||
|
||||
|
||||
class _FakeCursor:
|
||||
"""Duck-types the four attributes ``_release_after_failure`` touches."""
|
||||
|
||||
def __init__(self, *, close_raises: bool = False) -> None:
|
||||
self._conn = _FakeConn()
|
||||
self._close_raises = close_raises
|
||||
|
||||
def _build_close_pdu(self) -> bytes:
|
||||
return b"CLOSE"
|
||||
|
||||
def _build_release_pdu(self) -> bytes:
|
||||
return _RELEASE_PDU
|
||||
|
||||
def _drain_to_eot(self) -> None:
|
||||
if self._close_raises and self._conn.sent[-1] == b"CLOSE":
|
||||
raise OSError("wire went away mid-close")
|
||||
|
||||
|
||||
def test_release_is_sent_even_when_close_fails() -> None:
|
||||
"""The regression in the hand-rolled copies. CLOSE and RELEASE shared
|
||||
one suppress block, so a failing CLOSE swallowed the RELEASE — and a
|
||||
lost cursor handle is a nuisance while a lost statement breaks the
|
||||
next call."""
|
||||
cur = _FakeCursor(close_raises=True)
|
||||
Cursor._release_after_failure(cur, close_cursor=True)
|
||||
assert _RELEASE_PDU in cur._conn.sent, (
|
||||
"a CLOSE that raises must not prevent the RELEASE"
|
||||
)
|
||||
|
||||
|
||||
def test_close_is_skipped_when_no_cursor_was_opened() -> None:
|
||||
cur = _FakeCursor()
|
||||
Cursor._release_after_failure(cur)
|
||||
assert cur._conn.sent == [_RELEASE_PDU]
|
||||
|
||||
|
||||
def test_cleanup_never_propagates() -> None:
|
||||
"""Cleanup runs inside an ``except``. If it raises, it replaces the
|
||||
real SQL error with a secondary failure from the cleanup path, which
|
||||
is strictly worse for the caller."""
|
||||
|
||||
class _Hostile(_FakeCursor):
|
||||
def _drain_to_eot(self) -> None:
|
||||
raise OSError("connection reset")
|
||||
|
||||
Cursor._release_after_failure(_Hostile(), close_cursor=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
# Bounded: a leaked statement used to manifest as a hang.
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.parametrize(
|
||||
("sql", "params"),
|
||||
[
|
||||
pytest.param(
|
||||
"SELECT tabid FROM systables WHERE tabid = ?",
|
||||
("not-an-int",),
|
||||
id="string-for-int",
|
||||
),
|
||||
pytest.param(
|
||||
"SELECT ?::INT FROM systables WHERE tabid = 1",
|
||||
(2**40,),
|
||||
id="out-of-range-int",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_server_rejected_bind_releases_statement(
|
||||
conn_params: ConnParams, sql: str, params: tuple
|
||||
) -> None:
|
||||
"""A parameterized SELECT whose bind the *server* rejects. The value
|
||||
encodes cleanly, so this lands past the build guard and in the drain
|
||||
that had none. Repeated because a leak only shows on the call after
|
||||
it — the first failure looks fine on its own."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
for i in range(4):
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute(sql, params)
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None, f"broken after {i + 1} binds"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_failed_scroll_open_releases_statement(conn_params: ConnParams) -> None:
|
||||
"""FOR UPDATE prepares cleanly and fails at OPEN with -526, which is
|
||||
exactly the branch that had no guard."""
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
for i in range(4):
|
||||
with pytest.raises(informix_db.Error):
|
||||
scroll.execute("SELECT tabid FROM systables FOR UPDATE")
|
||||
assert not scroll._server_cursor_open, (
|
||||
"a failed open must not leave the cursor marked live"
|
||||
)
|
||||
other = conn.cursor()
|
||||
other.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert other.fetchone() is not None, f"broken after {i + 1} opens"
|
||||
other.close()
|
||||
scroll.close()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_scrollable_cursor_still_works_after_a_failed_open(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The same cursor object must be reusable — a failed open is an
|
||||
ordinary error, not a terminal state for the cursor."""
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
with pytest.raises(informix_db.Error):
|
||||
scroll.execute("SELECT tabid FROM systables FOR UPDATE")
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
assert scroll.fetch_first() is not None
|
||||
scroll.close()
|
||||
355
tests/test_statement_slot.py
Normal file
355
tests/test_statement_slot.py
Normal file
@ -0,0 +1,355 @@
|
||||
"""One statement per session, and what happens when we forget that.
|
||||
|
||||
SQLI gives a session a single statement slot. ``SQ_CLOSE``,
|
||||
``SQ_RELEASE`` and ``SQ_SFETCH`` all act on whatever statement is
|
||||
current — none of them names one. Ordinary use never notices, because a
|
||||
non-scrollable cursor materializes its rows and releases the statement
|
||||
before returning, so the slot is free again by the time anyone looks.
|
||||
|
||||
A scrollable cursor is the exception: it holds the slot open on purpose.
|
||||
Two things then went wrong, and neither said so.
|
||||
|
||||
**Another statement on the same connection.** The server returns ``-285``
|
||||
for the new statement and *also* destroys the scrollable cursor — its
|
||||
next fetch comes back ``-267`` "the transaction has been rolled back,
|
||||
all locks released". Two unattributable errors from code that reads as
|
||||
completely ordinary: iterate a large result set, run a lookup partway
|
||||
through. It is now a ``ProgrammingError`` that says what happened.
|
||||
|
||||
**The deferred-cleanup queue drained at the wrong moment.** A cursor
|
||||
finalizer that can't get the wire lock hands its CLOSE/RELEASE to a
|
||||
queue for the next operation to flush. That queue was flushed before
|
||||
*every* PDU. But a finalizer enqueues precisely because another thread
|
||||
holds the lock, i.e. is mid-statement — so the flush landed inside that
|
||||
thread's own statement and released it. The victim saw ``-208`` when it
|
||||
happened before the first fetch, ``-267`` between fetch batches. The
|
||||
flush now happens only at a statement boundary, where the queued CLOSE
|
||||
addresses the orphan it was meant for.
|
||||
|
||||
A stale queue entry was fatal too: the finalizer enqueues, the cursor
|
||||
then gets closed properly, and the leftover CLOSE draws ``-267`` — an
|
||||
``OperationalError``, which is in ``WIRE_ERRORS``, which force-closed a
|
||||
perfectly healthy connection. Server-reported errors are now told apart
|
||||
from wire failures by the presence of a ``sqlcode``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _CLOSE_PDU, _RELEASE_PDU
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
autocommit=True,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The conflict check — bookkeeping, no server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeScroll:
|
||||
def __init__(self, open_: bool = True) -> None:
|
||||
self._server_cursor_open = open_
|
||||
|
||||
|
||||
def _conflict_check(conn: informix_db.Connection, requester: object) -> None:
|
||||
conn._check_scroll_cursor_conflict(requester)
|
||||
|
||||
|
||||
def test_conflict_check_is_silent_with_no_scroll_cursor() -> None:
|
||||
conn = informix_db.Connection.__new__(informix_db.Connection)
|
||||
conn._open_scroll_cursor = None
|
||||
_conflict_check(conn, object())
|
||||
|
||||
|
||||
def test_conflict_check_forgets_a_collected_cursor() -> None:
|
||||
"""The ref is weak so an abandoned scrollable cursor can still be
|
||||
finalized. A dead ref means the slot is free."""
|
||||
import weakref
|
||||
|
||||
conn = informix_db.Connection.__new__(informix_db.Connection)
|
||||
victim = _FakeScroll()
|
||||
conn._open_scroll_cursor = weakref.ref(victim)
|
||||
del victim
|
||||
_conflict_check(conn, object())
|
||||
assert conn._open_scroll_cursor is None
|
||||
|
||||
|
||||
def test_conflict_check_lets_the_owner_through() -> None:
|
||||
"""Re-executing the *same* scrollable cursor is allowed — it closes
|
||||
its own server-side cursor first."""
|
||||
import weakref
|
||||
|
||||
conn = informix_db.Connection.__new__(informix_db.Connection)
|
||||
owner = _FakeScroll()
|
||||
conn._open_scroll_cursor = weakref.ref(owner)
|
||||
_conflict_check(conn, owner)
|
||||
|
||||
|
||||
def test_conflict_check_forgets_a_closed_cursor() -> None:
|
||||
import weakref
|
||||
|
||||
conn = informix_db.Connection.__new__(informix_db.Connection)
|
||||
done = _FakeScroll(open_=False)
|
||||
conn._open_scroll_cursor = weakref.ref(done)
|
||||
_conflict_check(conn, object())
|
||||
assert conn._open_scroll_cursor is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Against a real server
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_second_statement_is_refused_while_scroll_cursor_is_open(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Used to be -285 for the new statement plus -267 for the scrollable
|
||||
cursor. Now it's one error that names the cause, and the scrollable
|
||||
cursor is untouched."""
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
first = scroll.fetch_first()
|
||||
assert first is not None
|
||||
|
||||
other = conn.cursor()
|
||||
with pytest.raises(informix_db.ProgrammingError, match="scrollable"):
|
||||
other.execute("SELECT COUNT(*) FROM systables")
|
||||
|
||||
assert scroll.fetch_absolute(1) is not None, (
|
||||
"the refused statement must not have disturbed the cursor"
|
||||
)
|
||||
scroll.close()
|
||||
other.execute("SELECT COUNT(*) FROM systables")
|
||||
assert other.fetchone() is not None, "slot must free on close"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_executemany_is_refused_too(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_slot (k INT)")
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
scroll.fetch_first()
|
||||
with pytest.raises(informix_db.ProgrammingError, match="scrollable"):
|
||||
cur.executemany("INSERT INTO t_slot VALUES (?)", [(1,), (2,)])
|
||||
scroll.close()
|
||||
cur.executemany("INSERT INTO t_slot VALUES (?)", [(1,), (2,)])
|
||||
cur.execute("SELECT COUNT(*) FROM t_slot")
|
||||
assert cur.fetchone() == (2,)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_scroll_cursor_can_be_re_executed(conn_params: ConnParams) -> None:
|
||||
"""The owner is the one caller allowed past the conflict check, and
|
||||
that is only safe because it closes its own server-side cursor
|
||||
first. Without that it collides with itself."""
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
for _ in range(4):
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
assert scroll.fetch_first() is not None
|
||||
scroll.close()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_abandoning_a_scroll_cursor_frees_the_slot(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Dropping the last reference must let the connection be used again
|
||||
— the finalizer closes the cursor and the weak ref goes dead."""
|
||||
import gc
|
||||
|
||||
with _connect(conn_params) as conn:
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
scroll.fetch_first()
|
||||
del scroll
|
||||
gc.collect()
|
||||
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT COUNT(*) FROM systables")
|
||||
assert cur.fetchone() is not None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deferred cleanup
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_queued_cleanup_does_not_land_inside_a_running_statement(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""Exactly what a cross-thread finalizer does: it lost the wire lock,
|
||||
so it queued its CLOSE/RELEASE while another thread was mid-statement.
|
||||
The flush used to happen before that thread's very next PDU — its own
|
||||
CURNAME/NFETCH — releasing the statement out from under it."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_defer (k INT)")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_defer VALUES (?)", [(i,) for i in range(50)]
|
||||
)
|
||||
|
||||
original = cur._read_describe_response
|
||||
|
||||
def enqueue_between_prepare_and_fetch() -> None:
|
||||
result = original()
|
||||
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
|
||||
return result
|
||||
|
||||
cur._read_describe_response = enqueue_between_prepare_and_fetch
|
||||
try:
|
||||
cur.execute("SELECT k FROM t_defer ORDER BY k")
|
||||
assert len(cur.fetchall()) == 50, (
|
||||
"the queued cleanup released our own statement"
|
||||
)
|
||||
finally:
|
||||
cur._read_describe_response = original
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_stale_queued_cleanup_does_not_kill_the_connection(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""A queue entry goes stale whenever the cursor gets closed properly
|
||||
between enqueue and flush. The server answers the leftover CLOSE with
|
||||
-267, which is an OperationalError, which is in WIRE_ERRORS — so a
|
||||
stale entry used to force-close a healthy connection."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
cur.fetchall()
|
||||
|
||||
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None, "stale cleanup killed the connection"
|
||||
assert not conn.closed
|
||||
assert conn._pending_cleanup == [], "queue should have drained"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_connection_closes_cleanly_with_a_scroll_cursor_open(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
conn = _connect(conn_params)
|
||||
scroll = conn.cursor(scrollable=True)
|
||||
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
scroll.fetch_first()
|
||||
with contextlib.suppress(Exception):
|
||||
conn.close()
|
||||
assert conn.closed
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The wire lock's blind spot
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_wire_lock_reports_its_own_thread() -> None:
|
||||
"""The whole point. ``RLock.acquire(blocking=False)`` returns True for
|
||||
the owning thread, so the finalizer's probe could not tell "nobody is
|
||||
using the wire" from "I am, right now, mid-statement"."""
|
||||
from informix_db.connections import _WireLock
|
||||
|
||||
lock = _WireLock()
|
||||
assert not lock.held_by_current_thread
|
||||
with lock:
|
||||
assert lock.held_by_current_thread
|
||||
assert lock.acquire(blocking=False), "must still be reentrant"
|
||||
lock.release()
|
||||
assert lock.held_by_current_thread, "still held at depth 1"
|
||||
assert not lock.held_by_current_thread
|
||||
|
||||
|
||||
def test_wire_lock_is_not_held_by_other_threads() -> None:
|
||||
import threading
|
||||
|
||||
from informix_db.connections import _WireLock
|
||||
|
||||
lock = _WireLock()
|
||||
seen: list[bool] = []
|
||||
entered = threading.Event()
|
||||
done = threading.Event()
|
||||
|
||||
def holder() -> None:
|
||||
with lock:
|
||||
entered.set()
|
||||
done.wait(5)
|
||||
|
||||
t = threading.Thread(target=holder)
|
||||
t.start()
|
||||
entered.wait(5)
|
||||
seen.append(lock.held_by_current_thread)
|
||||
done.set()
|
||||
t.join(5)
|
||||
assert seen == [False], "another thread's hold must not read as ours"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_finalizer_defers_when_gc_runs_on_the_locking_thread(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""GC fires on whatever thread allocated. When that is the thread
|
||||
holding the wire lock, the finalizer must write nothing — it used to
|
||||
acquire the RLock reentrantly and send CLOSE/RELEASE into the running
|
||||
statement, killing it with -208."""
|
||||
import gc
|
||||
|
||||
with _connect(conn_params) as conn:
|
||||
gc.disable()
|
||||
try:
|
||||
victim = conn.cursor(scrollable=True)
|
||||
victim.execute("SELECT tabid FROM systables ORDER BY tabid")
|
||||
victim.fetch_first()
|
||||
# A reference cycle, so collection waits for gc rather than
|
||||
# happening at the drop. Cycles are ordinary in Python.
|
||||
cycle = [victim]
|
||||
cycle.append(cycle)
|
||||
del victim, cycle
|
||||
|
||||
writes: list[bytes] = []
|
||||
original_write = conn._sock.write_all
|
||||
conn._sock.write_all = lambda b: (
|
||||
writes.append(b),
|
||||
original_write(b),
|
||||
)[1]
|
||||
try:
|
||||
with conn._wire_lock: # stand in for "mid-statement"
|
||||
gc.collect()
|
||||
assert writes == [], (
|
||||
"finalizer wrote to the wire while a statement "
|
||||
"owned it"
|
||||
)
|
||||
assert conn._pending_cleanup, (
|
||||
"cleanup should have been deferred, not dropped"
|
||||
)
|
||||
finally:
|
||||
conn._sock.write_all = original_write
|
||||
finally:
|
||||
gc.enable()
|
||||
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT COUNT(*) FROM systables")
|
||||
assert cur.fetchone() is not None
|
||||
333
tests/test_tls_traffic.py
Normal file
333
tests/test_tls_traffic.py
Normal file
@ -0,0 +1,333 @@
|
||||
"""End-to-end SQLI traffic over a real TLS socket.
|
||||
|
||||
``tests/test_tls.py`` covers the handshake. This covers what happens
|
||||
*after* it: every byte of real SQLI traffic crossing genuine TLS records,
|
||||
through the same codecs, reader, and cursor machinery the plain-socket
|
||||
suite exercises.
|
||||
|
||||
That distinction matters because ``SSLSocket.recv`` is not
|
||||
``socket.recv``. It returns at most one TLS record's worth of plaintext
|
||||
however much you ask for, it can return fewer bytes than are available,
|
||||
and plaintext buffered inside the SSL object is invisible to the OS. The
|
||||
Phase 39 buffered reader asks for up to 64 KB per call and loops until
|
||||
satisfied — that loop is the thing which has to be right, and nothing in
|
||||
the plain-socket suite puts the same pressure on it.
|
||||
|
||||
**Scope.** A TLS-terminating proxy in front of the plain SQLI listener
|
||||
supplies the TLS half. This tests the driver's TLS path, which is the
|
||||
half we own. It does *not* test IBM's server-side TLS listener: Informix
|
||||
15 wants a PKCS#12 keystore whose stash the developer-edition image
|
||||
rejects (``GSK_ERROR_BAD_KEYFILE_PASSWORD``), and that side is IBM's
|
||||
code. Anything below ``ssl.wrap_socket`` is identical either way.
|
||||
|
||||
Skipped when ``openssl`` isn't on PATH — the proxy needs a certificate.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import datetime
|
||||
import decimal
|
||||
import select
|
||||
import shutil
|
||||
import socket
|
||||
import ssl
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TLS-terminating proxy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _TlsProxy:
|
||||
"""Accepts TLS, relays plaintext to the real Informix listener."""
|
||||
|
||||
def __init__(self, backend: tuple[str, int]) -> None:
|
||||
self.backend = backend
|
||||
self.tmpdir = tempfile.mkdtemp(prefix="ifx-tls-test-")
|
||||
self.cert = str(Path(self.tmpdir) / "cert.pem")
|
||||
key = str(Path(self.tmpdir) / "key.pem")
|
||||
subprocess.run(
|
||||
["openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes",
|
||||
"-keyout", key, "-out", self.cert, "-days", "1",
|
||||
"-subj", "/CN=127.0.0.1"],
|
||||
check=True, capture_output=True,
|
||||
)
|
||||
self._ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
self._ctx.load_cert_chain(self.cert, key)
|
||||
self._sock = socket.socket()
|
||||
self._sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
self._sock.bind(("127.0.0.1", 0))
|
||||
self._sock.listen(64)
|
||||
self.port: int = self._sock.getsockname()[1]
|
||||
self._stop = threading.Event()
|
||||
threading.Thread(target=self._serve, daemon=True).start()
|
||||
|
||||
def _serve(self) -> None:
|
||||
self._sock.settimeout(0.5)
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
raw, _ = self._sock.accept()
|
||||
except (TimeoutError, OSError):
|
||||
continue
|
||||
threading.Thread(
|
||||
target=self._handle, args=(raw,), daemon=True
|
||||
).start()
|
||||
|
||||
def _handle(self, raw: socket.socket) -> None:
|
||||
try:
|
||||
client = self._ctx.wrap_socket(raw, server_side=True)
|
||||
except (ssl.SSLError, OSError):
|
||||
with contextlib.suppress(OSError):
|
||||
raw.close()
|
||||
return
|
||||
try:
|
||||
upstream = socket.create_connection(self.backend, timeout=20)
|
||||
except OSError:
|
||||
with contextlib.suppress(OSError):
|
||||
client.close()
|
||||
return
|
||||
try:
|
||||
self._pump(client, upstream)
|
||||
finally:
|
||||
for s in (client, upstream):
|
||||
with contextlib.suppress(OSError):
|
||||
s.close()
|
||||
|
||||
@staticmethod
|
||||
def _pump(a: socket.socket, b: socket.socket) -> None:
|
||||
# Drain the SSL object's own buffer before consulting select():
|
||||
# select only sees the OS socket, so already-decrypted bytes
|
||||
# sitting inside the SSL object would stall the relay.
|
||||
socks = [a, b]
|
||||
while True:
|
||||
pending = [s for s in socks
|
||||
if isinstance(s, ssl.SSLSocket) and s.pending()]
|
||||
ready = pending or select.select(socks, [], [], 1.0)[0]
|
||||
for s in ready:
|
||||
other = b if s is a else a
|
||||
try:
|
||||
data = s.recv(65536)
|
||||
except (ssl.SSLError, OSError):
|
||||
return
|
||||
if not data:
|
||||
return
|
||||
try:
|
||||
other.sendall(data)
|
||||
except OSError:
|
||||
return
|
||||
|
||||
def close(self) -> None:
|
||||
self._stop.set()
|
||||
with contextlib.suppress(OSError):
|
||||
self._sock.close()
|
||||
shutil.rmtree(self.tmpdir, ignore_errors=True)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def tls_proxy(conn_params: ConnParams):
|
||||
if shutil.which("openssl") is None:
|
||||
pytest.skip("openssl not on PATH; needed to generate a test cert")
|
||||
proxy = _TlsProxy((conn_params.host, conn_params.port))
|
||||
try:
|
||||
yield proxy
|
||||
finally:
|
||||
proxy.close()
|
||||
|
||||
|
||||
def _client_ctx(proxy: _TlsProxy) -> ssl.SSLContext:
|
||||
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
|
||||
ctx.load_verify_locations(proxy.cert)
|
||||
ctx.check_hostname = False # self-signed CN=127.0.0.1
|
||||
return ctx
|
||||
|
||||
|
||||
def _connect(proxy: _TlsProxy, conn_params: ConnParams, **kw):
|
||||
return informix_db.connect(
|
||||
host="127.0.0.1",
|
||||
port=proxy.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=20.0,
|
||||
read_timeout=45.0,
|
||||
tls=_client_ctx(proxy),
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Traffic
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_query_over_tls(tls_proxy, conn_params: ConnParams) -> None:
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT FIRST 3 tabname FROM systables ORDER BY tabid")
|
||||
assert len(cur.fetchall()) == 3
|
||||
assert conn.server_version, "server_version empty over TLS"
|
||||
|
||||
|
||||
def test_type_round_trip_over_tls(tls_proxy, conn_params: ConnParams) -> None:
|
||||
"""The types that gave us framing bugs, every byte through TLS."""
|
||||
ts = datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
|
||||
row = (
|
||||
2001, "PackageRoot", None, "/content/package", 77,
|
||||
decimal.Decimal("1234567890123456"), True, ts, "nch",
|
||||
)
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_tls_types ("
|
||||
" a INT8 NOT NULL, k LVARCHAR(512), d LVARCHAR(512),"
|
||||
" v LVARCHAR(1024), n INT8, dec16 DECIMAL(16), b BOOLEAN,"
|
||||
" t DATETIME YEAR TO FRACTION(5), c NCHAR(6))"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_tls_types VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", row
|
||||
)
|
||||
cur.execute("SELECT a, k, d, v, n, dec16, b, t, c FROM t_tls_types")
|
||||
assert cur.fetchone() == row
|
||||
|
||||
|
||||
@pytest.mark.parametrize("size", [1, 4096, 16383, 16384, 16385, 32000])
|
||||
def test_payload_spans_tls_record_boundary(
|
||||
tls_proxy, conn_params: ConnParams, size: int
|
||||
) -> None:
|
||||
"""A TLS record holds ~16 KB, so these straddle the boundary where a
|
||||
single ``recv`` stops being enough."""
|
||||
payload = "x" * size
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_tls_big (k INT, v LVARCHAR(32000))")
|
||||
cur.execute("INSERT INTO t_tls_big VALUES (?, ?)", (1, payload))
|
||||
cur.execute("SELECT v, k FROM t_tls_big")
|
||||
assert cur.fetchone() == (payload, 1)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [500, 5000])
|
||||
def test_bulk_fetch_over_tls(
|
||||
tls_proxy, conn_params: ConnParams, n: int
|
||||
) -> None:
|
||||
"""Total bytes far beyond both one TLS record and the reader's 64 KB
|
||||
recv budget, so the top-up loop runs many times."""
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_tls_bulk (k INT, v VARCHAR(240))")
|
||||
cur.executemany(
|
||||
"INSERT INTO t_tls_bulk VALUES (?, ?)",
|
||||
[(i, f"row{i}-" + "y" * 200) for i in range(n)],
|
||||
)
|
||||
cur.execute("SELECT k, v FROM t_tls_bulk ORDER BY k")
|
||||
rows = cur.fetchall()
|
||||
assert len(rows) == n
|
||||
assert rows[0][0] == 0
|
||||
assert rows[-1][0] == n - 1
|
||||
|
||||
|
||||
def test_error_recovery_over_tls(tls_proxy, conn_params: ConnParams) -> None:
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_tls_dup (k INT PRIMARY KEY)")
|
||||
cur.execute("INSERT INTO t_tls_dup VALUES (1)")
|
||||
for _ in range(4):
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("INSERT INTO t_tls_dup VALUES (1)")
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("SELECT * FROM t_tls_no_such_table_xyz")
|
||||
cur.execute("SELECT k FROM t_tls_dup")
|
||||
assert cur.fetchall() == [(1,)]
|
||||
|
||||
|
||||
def test_concurrent_tls_connections(tls_proxy, conn_params: ConnParams) -> None:
|
||||
"""Separate TLS sessions must not cross data."""
|
||||
failures: list[str] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def worker(tid: int) -> None:
|
||||
tag = f"t{tid}-{'z' * 12}"
|
||||
try:
|
||||
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
for r in range(6):
|
||||
token = tid * 1000 + r
|
||||
cur.execute(
|
||||
"SELECT FIRST 1 ?::INT, ?::VARCHAR(24) FROM systables",
|
||||
(token, tag),
|
||||
)
|
||||
if cur.fetchone() != (token, tag):
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: crossed data")
|
||||
return
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(i,)) for i in range(4)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
assert not failures, failures
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Negative cases — misuse must fail cleanly, never hang or downgrade
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_tls_client_against_plaintext_port_fails(
|
||||
tls_proxy, conn_params: ConnParams
|
||||
) -> None:
|
||||
with pytest.raises(informix_db.Error):
|
||||
informix_db.connect(
|
||||
host=conn_params.host, port=conn_params.port,
|
||||
user=conn_params.user, password=conn_params.password,
|
||||
database=conn_params.database, server=conn_params.server,
|
||||
connect_timeout=10.0, read_timeout=10.0,
|
||||
tls=_client_ctx(tls_proxy),
|
||||
)
|
||||
|
||||
|
||||
def test_plaintext_client_against_tls_port_fails(
|
||||
tls_proxy, conn_params: ConnParams
|
||||
) -> None:
|
||||
"""Must raise rather than hang — a stalled handshake is the failure
|
||||
mode that looks like a dead application."""
|
||||
with pytest.raises(informix_db.Error):
|
||||
informix_db.connect(
|
||||
host="127.0.0.1", port=tls_proxy.port,
|
||||
user=conn_params.user, password=conn_params.password,
|
||||
database=conn_params.database, server=conn_params.server,
|
||||
connect_timeout=10.0, read_timeout=10.0,
|
||||
)
|
||||
|
||||
|
||||
def test_verification_rejects_self_signed(
|
||||
tls_proxy, conn_params: ConnParams
|
||||
) -> None:
|
||||
"""`tls=True` disables verification by design; a caller-supplied
|
||||
verifying context must still reject an untrusted cert."""
|
||||
strict = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
|
||||
strict.check_hostname = True
|
||||
strict.verify_mode = ssl.CERT_REQUIRED
|
||||
with pytest.raises(informix_db.Error):
|
||||
informix_db.connect(
|
||||
host="127.0.0.1", port=tls_proxy.port,
|
||||
user=conn_params.user, password=conn_params.password,
|
||||
database=conn_params.database, server=conn_params.server,
|
||||
connect_timeout=10.0, read_timeout=10.0, tls=strict,
|
||||
)
|
||||
191
tests/test_transaction_state.py
Normal file
191
tests/test_transaction_state.py
Normal file
@ -0,0 +1,191 @@
|
||||
"""Transaction control run as SQL, and the flag that didn't notice.
|
||||
|
||||
``Connection._in_transaction`` decides whether ``commit()`` and
|
||||
``rollback()`` send anything at all, and the pool reads it to decide
|
||||
whether a returned connection needs cleaning up. It was maintained
|
||||
solely by the driver's own implicit ``SQ_BEGIN``, so a caller who wrote
|
||||
``cursor.execute("BEGIN WORK")`` — an entirely reasonable thing to
|
||||
write — walked straight past it.
|
||||
|
||||
With autocommit on, nothing stopped that statement reaching the server.
|
||||
A transaction opened, the flag stayed False, and ``rollback()`` returned
|
||||
successfully having sent nothing. The rows it was asked to discard were
|
||||
still there. The connection then went back to the pool holding an open
|
||||
transaction and its locks, because the pool's cleanup is guarded by the
|
||||
same flag.
|
||||
|
||||
With autocommit off it failed instead, and for a sillier reason: the
|
||||
driver's implicit ``SQ_BEGIN`` fired first, so the caller's ``BEGIN
|
||||
WORK`` got ``-535``, "already in transaction". The driver and the user
|
||||
competing to open the same transaction, and the user losing.
|
||||
|
||||
The server labels these statements: type 34 for BEGIN, 35 for COMMIT, 36
|
||||
for ROLLBACK, with and without the ``WORK`` keyword. JDBC reads the same
|
||||
three values off the describe and calls ``setTxBeginState`` /
|
||||
``setTxEndState``. It also calls ``initiateTransaction`` *after* the
|
||||
describe rather than before, which is what makes the skip possible —
|
||||
until the describe lands you can't know the statement is transaction
|
||||
control.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from informix_db.cursors import _TX_CONTROL_TYPES
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=25.0,
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
def test_transaction_control_types_are_what_the_server_says(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""Pin the three constants against a live server rather than trusting
|
||||
the decompiled source. Both spellings must map to the same type."""
|
||||
with _connect(logged_db_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
seen = {}
|
||||
for sql in ("BEGIN WORK", "COMMIT WORK", "ROLLBACK WORK",
|
||||
"BEGIN", "COMMIT", "ROLLBACK"):
|
||||
with conn._wire_lock:
|
||||
conn._send_pdu(
|
||||
cur._build_prepare_pdu(sql, num_qmarks=0),
|
||||
statement_boundary=True,
|
||||
)
|
||||
cur._read_describe_response()
|
||||
cur._release_after_failure()
|
||||
seen[sql] = cur._statement_type
|
||||
assert seen["BEGIN WORK"] == seen["BEGIN"] == 34
|
||||
assert seen["COMMIT WORK"] == seen["COMMIT"] == 35
|
||||
assert seen["ROLLBACK WORK"] == seen["ROLLBACK"] == 36
|
||||
assert set(seen.values()) == _TX_CONTROL_TYPES
|
||||
|
||||
|
||||
def test_rollback_after_sql_begin_actually_rolls_back(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""The data-loss case. rollback() reported success and sent nothing,
|
||||
so the row it was asked to discard survived."""
|
||||
with _connect(logged_db_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate")
|
||||
cur.execute("CREATE TABLE t_txstate (k INT)")
|
||||
try:
|
||||
cur.execute("BEGIN WORK")
|
||||
assert conn._in_transaction, "SQL BEGIN must set the flag"
|
||||
cur.execute("INSERT INTO t_txstate VALUES (1)")
|
||||
conn.rollback()
|
||||
cur.execute("SELECT COUNT(*) FROM t_txstate")
|
||||
assert cur.fetchone() == (0,), "rollback() was a silent no-op"
|
||||
assert not conn._in_transaction
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate")
|
||||
|
||||
|
||||
def test_sql_commit_clears_the_flag(logged_db_params: ConnParams) -> None:
|
||||
"""The mirror image: with the flag stuck True after a SQL COMMIT, the
|
||||
next rollback() would send SQ_RBWORK with no transaction open and
|
||||
draw -255."""
|
||||
with _connect(logged_db_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate2")
|
||||
cur.execute("CREATE TABLE t_txstate2 (k INT)")
|
||||
try:
|
||||
cur.execute("BEGIN WORK")
|
||||
cur.execute("INSERT INTO t_txstate2 VALUES (1)")
|
||||
cur.execute("COMMIT WORK")
|
||||
assert not conn._in_transaction, "SQL COMMIT must clear the flag"
|
||||
conn.rollback() # must be a no-op, not a -255
|
||||
cur.execute("SELECT COUNT(*) FROM t_txstate2")
|
||||
assert cur.fetchone() == (1,), "the committed row must survive"
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate2")
|
||||
|
||||
|
||||
def test_sql_begin_does_not_collide_with_the_implicit_one(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""Non-autocommit. The driver's implicit SQ_BEGIN used to fire first
|
||||
and the caller's BEGIN WORK then got -535."""
|
||||
with _connect(logged_db_params, autocommit=False) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate3")
|
||||
conn.commit()
|
||||
cur.execute("CREATE TABLE t_txstate3 (k INT)")
|
||||
conn.commit()
|
||||
try:
|
||||
cur.execute("BEGIN WORK")
|
||||
cur.execute("INSERT INTO t_txstate3 VALUES (1)")
|
||||
conn.rollback()
|
||||
cur.execute("SELECT COUNT(*) FROM t_txstate3")
|
||||
assert cur.fetchone() == (0,)
|
||||
conn.commit()
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate3")
|
||||
conn.commit()
|
||||
|
||||
|
||||
def test_ordinary_dml_still_opens_a_transaction(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""_ensure_transaction moved from before the PREPARE to after the
|
||||
describe. It still has to fire for everything that isn't transaction
|
||||
control, or non-autocommit DML runs outside a transaction."""
|
||||
with _connect(logged_db_params, autocommit=False) as conn:
|
||||
cur = conn.cursor()
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate4")
|
||||
conn.commit()
|
||||
cur.execute("CREATE TABLE t_txstate4 (k INT)")
|
||||
conn.commit()
|
||||
try:
|
||||
assert not conn._in_transaction
|
||||
cur.execute("INSERT INTO t_txstate4 VALUES (1)")
|
||||
assert conn._in_transaction, "DML must open a transaction"
|
||||
conn.rollback()
|
||||
cur.execute("SELECT COUNT(*) FROM t_txstate4")
|
||||
assert cur.fetchone() == (0,)
|
||||
conn.commit()
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
cur.execute("DROP TABLE t_txstate4")
|
||||
conn.commit()
|
||||
|
||||
|
||||
def test_a_failed_transaction_statement_does_not_move_the_flag(
|
||||
logged_db_params: ConnParams,
|
||||
) -> None:
|
||||
"""The sync runs on the success path only. A COMMIT that the server
|
||||
rejects has not ended anything."""
|
||||
with _connect(logged_db_params, autocommit=True) as conn:
|
||||
cur = conn.cursor()
|
||||
assert not conn._in_transaction
|
||||
with pytest.raises(informix_db.Error):
|
||||
cur.execute("COMMIT WORK") # -255, nothing to commit
|
||||
assert not conn._in_transaction
|
||||
cur.execute("SELECT FIRST 1 tabid FROM systables")
|
||||
assert cur.fetchone() is not None
|
||||
227
tests/test_type_framing.py
Normal file
227
tests/test_type_framing.py
Normal file
@ -0,0 +1,227 @@
|
||||
"""Regression tests for tuple-payload framing bugs found 2026-05-08.
|
||||
|
||||
All three bugs below shipped in released versions and survived a
|
||||
251-test integration suite for one reason: no fixture used the affected
|
||||
types. They were reported from the field (Informix 12 user) but reproduce
|
||||
identically on Informix 15 — none of them is version-specific.
|
||||
|
||||
Each bug is covered twice: once for the value itself, and once with
|
||||
**trailing columns** after the affected column. The trailing-column case
|
||||
is the important one — two of these bugs desynced the row decoder, so
|
||||
the damage showed up in *later* columns, not the one with the bad type.
|
||||
A single-column test would have passed while the driver was corrupting
|
||||
every real query.
|
||||
|
||||
Wire evidence for each is recorded in the source comments at the fix
|
||||
sites (``converters._decode_int8``, ``_resultset._FIXED_WIDTH_CHAR_TYPES``,
|
||||
and the BOOLEAN branch of ``_resultset._legacy_dispatch_one_column``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
read_timeout=10.0,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# INT8 / SERIAL8 — the legacy 64-bit integer. 10 bytes, sign-magnitude,
|
||||
# with the high and low 32-bit halves stored in the opposite order you'd
|
||||
# expect. Previously fell through to the unknown-type path and surfaced
|
||||
# as raw bytes. Correct width, so no desync — just a wrong value.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"value",
|
||||
[
|
||||
123456789012, # needs both halves
|
||||
-123456789012, # negative: same magnitude bytes, sign word 0xFFFF
|
||||
42, # low half only
|
||||
0,
|
||||
-1,
|
||||
2**63 - 1, # INT8 max
|
||||
-(2**63 - 1),
|
||||
None, # sign word 0x0000
|
||||
],
|
||||
)
|
||||
def test_int8_round_trip(conn_params: ConnParams, value: int | None) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_int8 (v INT8)")
|
||||
cur.execute("INSERT INTO t_int8 VALUES (?)", (value,))
|
||||
cur.execute("SELECT v FROM t_int8")
|
||||
assert cur.fetchone() == (value,)
|
||||
|
||||
|
||||
def test_int8_does_not_desync_following_columns(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_int8_mix "
|
||||
"(a INT, b INT8, c INT, d VARCHAR(10))"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_int8_mix VALUES (?, ?, ?, ?)",
|
||||
(111111, 123456789012, 222222, "tail"),
|
||||
)
|
||||
cur.execute("SELECT a, b, c, d FROM t_int8_mix")
|
||||
assert cur.fetchone() == (111111, 123456789012, 222222, "tail")
|
||||
|
||||
|
||||
def test_serial8_decodes_as_int(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_ser8 (v SERIAL8, note VARCHAR(8))")
|
||||
cur.execute("INSERT INTO t_ser8 VALUES (0, 'x')")
|
||||
cur.execute("SELECT v, note FROM t_ser8")
|
||||
row = cur.fetchone()
|
||||
assert row == (1, "x"), f"SERIAL8 row decoded as {row!r}"
|
||||
|
||||
|
||||
def test_int8_is_not_confused_with_bigint(conn_params: ConnParams) -> None:
|
||||
"""INT8 (17) is 10 bytes sign-magnitude; BIGINT (52) is 8 bytes two's
|
||||
complement. Same logical range, completely different wire format —
|
||||
decoding one as the other is silently wrong for negatives."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_i8_bi (a INT8, b BIGINT)")
|
||||
cur.execute("INSERT INTO t_i8_bi VALUES (?, ?)", (-77, -77))
|
||||
cur.execute("SELECT a, b FROM t_i8_bi")
|
||||
assert cur.fetchone() == (-77, -77)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# NCHAR — fixed-width and space-padded, exactly like CHAR. Was being read
|
||||
# as 1-byte-length-prefixed, so the first character got consumed as a
|
||||
# length and the offset jumped by that value. NCHAR(10) holding 'nch'
|
||||
# read 0x6E ('n') as a 110-byte length: silent truncation at best, a
|
||||
# struct.error crash when other columns followed.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_nchar_keeps_first_character(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_nchar (v NCHAR(10))")
|
||||
cur.execute("INSERT INTO t_nchar VALUES ('nch')")
|
||||
cur.execute("SELECT v FROM t_nchar")
|
||||
assert cur.fetchone() == ("nch",)
|
||||
|
||||
|
||||
def test_nchar_does_not_desync_following_columns(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_nchar_mix "
|
||||
"(a INT, n NCHAR(10), c INT, v NVARCHAR(20), z INT)"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_nchar_mix VALUES (111111, 'nch', 222222, 'nvc', 333333)"
|
||||
)
|
||||
cur.execute("SELECT a, n, c, v, z FROM t_nchar_mix")
|
||||
assert cur.fetchone() == (111111, "nch", 222222, "nvc", 333333)
|
||||
|
||||
|
||||
def test_nvarchar_still_length_prefixed(conn_params: ConnParams) -> None:
|
||||
"""Guard the other side of the NCHAR fix: NVARCHAR *is* byte-length-
|
||||
prefixed and must not be moved to the fixed-width branch."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_nvc (v NVARCHAR(40), tail INT)")
|
||||
cur.execute("INSERT INTO t_nvc VALUES ('nvc', 4242)")
|
||||
cur.execute("SELECT v, tail FROM t_nvc")
|
||||
assert cur.fetchone() == ("nvc", 4242)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# BOOLEAN — described as UDTFIXED(41) with encoded_length=1, but carries
|
||||
# the full UDT envelope on the wire: [1-byte indicator][4-byte length]
|
||||
# [data]. Six bytes for a one-byte value. Reading only encoded_length left
|
||||
# 5 bytes behind and corrupted every subsequent column — this is the bug
|
||||
# that produced the original field report.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("literal", "expected"),
|
||||
[("'t'", True), ("'f'", False), ("NULL", None)],
|
||||
)
|
||||
def test_boolean_decodes(
|
||||
conn_params: ConnParams, literal: str, expected: bool | None
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_bool (v BOOLEAN)")
|
||||
cur.execute(f"INSERT INTO t_bool VALUES ({literal})")
|
||||
cur.execute("SELECT v FROM t_bool")
|
||||
assert cur.fetchone() == (expected,)
|
||||
|
||||
|
||||
def test_boolean_does_not_desync_following_columns(
|
||||
conn_params: ConnParams,
|
||||
) -> None:
|
||||
"""The original field-reported symptom: a BOOLEAN column silently
|
||||
shifted every column after it by 5 bytes."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_bool_mix "
|
||||
"(a INT, b BOOLEAN, c INT, d VARCHAR(10))"
|
||||
)
|
||||
cur.execute("INSERT INTO t_bool_mix VALUES (111111, 't', 222222, 'tail')")
|
||||
cur.execute("SELECT a, b, c, d FROM t_bool_mix")
|
||||
assert cur.fetchone() == (111111, True, 222222, "tail")
|
||||
|
||||
|
||||
def test_multiple_booleans_in_one_row(conn_params: ConnParams) -> None:
|
||||
"""Each BOOLEAN consumes its own 6-byte envelope; an off-by-N in the
|
||||
envelope compounds across columns."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_bool_many "
|
||||
"(a BOOLEAN, b BOOLEAN, c BOOLEAN, tail INT)"
|
||||
)
|
||||
cur.execute("INSERT INTO t_bool_many VALUES ('t', 'f', 't', 5150)")
|
||||
cur.execute("SELECT a, b, c, tail FROM t_bool_many")
|
||||
assert cur.fetchone() == (True, False, True, 5150)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Combined: everything that previously mis-framed, in one row.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_all_previously_broken_types_in_one_row(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
"CREATE TEMP TABLE t_framing ("
|
||||
" a INT, b BOOLEAN, c NCHAR(10), d INT8,"
|
||||
" e VARCHAR(20), f NVARCHAR(20), g SERIAL8, h INT)"
|
||||
)
|
||||
cur.execute(
|
||||
"INSERT INTO t_framing VALUES "
|
||||
"(111111, 't', 'nch', 123456789012, 'vc', 'nvc', 0, 999888)"
|
||||
)
|
||||
cur.execute("SELECT a, b, c, d, e, f, g, h FROM t_framing")
|
||||
assert cur.fetchone() == (
|
||||
111111, True, "nch", 123456789012, "vc", "nvc", 1, 999888,
|
||||
)
|
||||
342
tests/test_type_matrix.py
Normal file
342
tests/test_type_matrix.py
Normal file
@ -0,0 +1,342 @@
|
||||
"""Type-matrix regression tests, and the fuzzer that found them.
|
||||
|
||||
Six framing bugs reached users before this file existed. Every one shared
|
||||
a shape: the value under test decoded fine on its own, and quietly
|
||||
corrupted whatever came *after* it. They were missed not because branches
|
||||
went unexercised but because the fixtures never supplied data that took
|
||||
them — the LVARCHAR fixture was ``'lv value'``, eight characters, even,
|
||||
never NULL, so neither of its two broken branches ever ran.
|
||||
|
||||
Three principles follow, and every test here applies them:
|
||||
|
||||
1. **Always put a column after the value under test.** A trailing column
|
||||
can be mis-sized with no visible effect. The sentinel is the detector.
|
||||
2. **Vary the data, not just the type.** Odd vs even length, empty vs
|
||||
NULL, min/max, negative, unscaled vs scaled.
|
||||
3. **Use an oracle our codecs don't share.** A Python round-trip cannot
|
||||
catch a symmetric encode/decode bug — our DATETIME encoder wrote zeros
|
||||
and our decoder read them back in perfect agreement. Asking the server
|
||||
to render the stored value as text breaks that symmetry.
|
||||
|
||||
The three bugs in the first section were found by the fuzzer at the
|
||||
bottom of this file, not by a person.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import decimal
|
||||
import random
|
||||
|
||||
import pytest
|
||||
|
||||
import informix_db
|
||||
from tests.conftest import ConnParams
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
SENTINEL = 424242
|
||||
|
||||
|
||||
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
||||
return informix_db.connect(
|
||||
host=conn_params.host,
|
||||
port=conn_params.port,
|
||||
user=conn_params.user,
|
||||
password=conn_params.password,
|
||||
database=conn_params.database,
|
||||
server=conn_params.server,
|
||||
connect_timeout=10.0,
|
||||
# Bounded so a wire desync fails the test instead of hanging it.
|
||||
read_timeout=20.0,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. BOOLEAN could not be used as a bind parameter at all
|
||||
# ---------------------------------------------------------------------------
|
||||
# _encode_bool emitted type code 45, which the server does not accept as a
|
||||
# bind type: it simply stopped responding. Any execute() with a bool
|
||||
# parameter hung until the read timeout, or forever without one. Type 41
|
||||
# (how BOOLEAN is *described* in results) hangs identically — descriptor
|
||||
# type and bind type are not interchangeable. Binding 't'/'f' as CHAR and
|
||||
# letting the server cast is what works.
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [True, False, None])
|
||||
def test_boolean_bind_round_trip(
|
||||
conn_params: ConnParams, value: bool | None
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_bool_bind (b BOOLEAN, s INT)")
|
||||
cur.execute("INSERT INTO t_bool_bind VALUES (?, ?)", (value, SENTINEL))
|
||||
cur.execute("SELECT b, s FROM t_bool_bind")
|
||||
assert cur.fetchone() == (value, SENTINEL)
|
||||
|
||||
|
||||
def test_boolean_bind_in_where_clause(conn_params: ConnParams) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_bool_where (b BOOLEAN, k INT)")
|
||||
cur.execute("INSERT INTO t_bool_where VALUES (?, ?)", (True, 1))
|
||||
cur.execute("INSERT INTO t_bool_where VALUES (?, ?)", (False, 2))
|
||||
cur.execute("SELECT k FROM t_bool_where WHERE b = ?", (True,))
|
||||
assert cur.fetchall() == [(1,)]
|
||||
cur.execute("SELECT k FROM t_bool_where WHERE b = ?", (False,))
|
||||
assert cur.fetchall() == [(2,)]
|
||||
|
||||
|
||||
def test_boolean_bind_stores_real_boolean(conn_params: ConnParams) -> None:
|
||||
"""Independent oracle: the server must agree it stored a BOOLEAN, not
|
||||
the string we transported it as."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("CREATE TEMP TABLE t_bool_oracle (b BOOLEAN)")
|
||||
cur.execute("INSERT INTO t_bool_oracle VALUES (?)", (True,))
|
||||
cur.execute("SELECT b::CHAR(1) FROM t_bool_oracle")
|
||||
assert cur.fetchone() == ("t",)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Unscaled DECIMAL was read one byte short
|
||||
# ---------------------------------------------------------------------------
|
||||
# Width is ((precision) + (scale & 1) + 3) // 2, per IfxColumnInfo's
|
||||
# adjustedColumnLength. We omitted the `scale & 1` term, which only
|
||||
# matters when precision is even and scale is odd — and an unscaled
|
||||
# DECIMAL(p) reports scale 255. DECIMAL(16) is 10 bytes, not 9.
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("ddl", "value"),
|
||||
[
|
||||
("DECIMAL(16)", decimal.Decimal("1234567890123456")),
|
||||
("DECIMAL(16)", decimal.Decimal("-1")),
|
||||
("DECIMAL(16)", None),
|
||||
("DECIMAL(4)", decimal.Decimal("1234")),
|
||||
("DECIMAL(20)", decimal.Decimal("12345678901234567890")),
|
||||
("DECIMAL(8,2)", decimal.Decimal("12345.67")),
|
||||
("DECIMAL(10,4)", decimal.Decimal("123456.7890")),
|
||||
("DECIMAL(5,0)", decimal.Decimal("12345")),
|
||||
("DECIMAL(1,0)", decimal.Decimal("7")),
|
||||
("DECIMAL(32,10)", decimal.Decimal("1234567890.0123456789")),
|
||||
("MONEY(10,2)", decimal.Decimal("1234.56")),
|
||||
("MONEY(10,2)", decimal.Decimal("-1234.56")),
|
||||
],
|
||||
)
|
||||
def test_decimal_width_does_not_shift_next_column(
|
||||
conn_params: ConnParams, ddl: str, value: decimal.Decimal | None
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(f"CREATE TEMP TABLE t_dec_w (d {ddl}, s INT)")
|
||||
cur.execute("INSERT INTO t_dec_w VALUES (?, ?)", (value, SENTINEL))
|
||||
cur.execute("SELECT d, s FROM t_dec_w")
|
||||
got = cur.fetchone()
|
||||
assert got[1] == SENTINEL, f"{ddl} mis-sized; sentinel came back {got[1]}"
|
||||
if value is None:
|
||||
assert got[0] is None
|
||||
else:
|
||||
assert got[0] == value
|
||||
|
||||
|
||||
def test_packed_width_matches_the_reference_formula() -> None:
|
||||
"""Unit-level guard on the formula itself, including the term that was
|
||||
missing. Values verified against the wire."""
|
||||
from informix_db._resultset import _packed_width
|
||||
|
||||
assert _packed_width((8 << 8) | 2) == 5 # DECIMAL(8,2)
|
||||
assert _packed_width((16 << 8) | 255) == 10 # DECIMAL(16), unscaled
|
||||
assert _packed_width((10 << 8) | 4) == 6 # DECIMAL(10,4)
|
||||
assert _packed_width((5 << 8) | 0) == 4 # DECIMAL(5,0)
|
||||
assert _packed_width((32 << 8) | 10) == 17 # DECIMAL(32,10)
|
||||
assert _packed_width((1 << 8) | 0) == 2 # DECIMAL(1,0)
|
||||
assert _packed_width((14 << 8) | 10) == 8 # DATETIME YEAR TO SECOND
|
||||
assert _packed_width((19 << 8) | 15) == 11 # DATETIME .. FRACTION(5)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. NULL CHAR / NCHAR came back as an empty string
|
||||
# ---------------------------------------------------------------------------
|
||||
# The NULL marker is a leading 0x00; an empty CHAR is all spaces. They are
|
||||
# distinguishable on the wire, so conflating them made `IS NULL` disagree
|
||||
# with what the driver returned.
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ddl", ["CHAR(6)", "NCHAR(6)"])
|
||||
def test_null_char_is_none_not_empty_string(
|
||||
conn_params: ConnParams, ddl: str
|
||||
) -> None:
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(f"CREATE TEMP TABLE t_char_null (k INT, c {ddl})")
|
||||
cur.execute("INSERT INTO t_char_null VALUES (?, ?)", (1, None))
|
||||
cur.execute("INSERT INTO t_char_null VALUES (?, ?)", (2, ""))
|
||||
cur.execute("INSERT INTO t_char_null VALUES (?, ?)", (3, "ab"))
|
||||
cur.execute("SELECT k, c FROM t_char_null ORDER BY k")
|
||||
assert cur.fetchall() == [(1, None), (2, ""), (3, "ab")]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ddl", ["CHAR(6)", "NCHAR(6)"])
|
||||
def test_null_char_agrees_with_is_null(
|
||||
conn_params: ConnParams, ddl: str
|
||||
) -> None:
|
||||
"""The server's own view is the oracle: what it calls NULL, we must
|
||||
return as None."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(f"CREATE TEMP TABLE t_char_isnull (k INT, c {ddl})")
|
||||
cur.execute("INSERT INTO t_char_isnull VALUES (?, ?)", (1, None))
|
||||
cur.execute("INSERT INTO t_char_isnull VALUES (?, ?)", (2, ""))
|
||||
cur.execute("SELECT k FROM t_char_isnull WHERE c IS NULL")
|
||||
server_nulls = {r[0] for r in cur.fetchall()}
|
||||
cur.execute("SELECT k, c FROM t_char_isnull ORDER BY k")
|
||||
driver_nulls = {k for k, c in cur.fetchall() if c is None}
|
||||
assert driver_nulls == server_nulls
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. The fuzzer itself, with fixed seeds so it is reproducible in CI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_ODD = "abcdefghijk" # 11
|
||||
_EVEN = "abcdefghij" # 10
|
||||
|
||||
# (name, DDL, edge values). Values are chosen to flip framing branches.
|
||||
_CORPUS: list[tuple[str, str, list]] = [
|
||||
("smallint", "SMALLINT", [0, 1, -1, 32767, -32767, None]),
|
||||
("int", "INT", [0, -1, 2147483647, None]),
|
||||
("int8", "INT8", [0, 1, -1, 123456789012, -123456789012, None]),
|
||||
("bigint", "BIGINT", [0, -1, 2**62, None]),
|
||||
("float", "FLOAT", [0.0, 3.141592653589793, -1.5, None]),
|
||||
("smallfloat", "SMALLFLOAT", [0.0, 2.5, None]),
|
||||
("dec_scaled", "DECIMAL(8,2)", [
|
||||
decimal.Decimal("12345.67"), decimal.Decimal("-98.76"), None,
|
||||
]),
|
||||
("dec_float", "DECIMAL(16)", [
|
||||
decimal.Decimal("1234567890123456"), decimal.Decimal("-1"), None,
|
||||
]),
|
||||
("money", "MONEY(10,2)", [decimal.Decimal("1234.56"), None]),
|
||||
("char", "CHAR(12)", ["", "a", _ODD, _EVEN, None]),
|
||||
("varchar", "VARCHAR(64)", ["", "a", _ODD, _EVEN, None]),
|
||||
("nchar", "NCHAR(12)", ["", "a", _ODD, None]),
|
||||
("nvarchar", "NVARCHAR(64)", ["", "a", _ODD, None]),
|
||||
("lvarchar", "LVARCHAR(1024)", [
|
||||
"", "a", "ab", _ODD, _EVEN, "x" * 255, "y" * 256, None,
|
||||
]),
|
||||
("date", "DATE", [datetime.date(1899, 12, 31), datetime.date(2026, 8, 31), None]),
|
||||
("dt_sec", "DATETIME YEAR TO SECOND", [
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15), None,
|
||||
]),
|
||||
("dt_frac", "DATETIME YEAR TO FRACTION(5)", [
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15),
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
|
||||
None,
|
||||
]),
|
||||
("interval", "INTERVAL DAY(5) TO SECOND", [
|
||||
datetime.timedelta(days=10, hours=4, minutes=30, seconds=15),
|
||||
datetime.timedelta(0),
|
||||
None,
|
||||
]),
|
||||
("boolean", "BOOLEAN", [True, False, None]),
|
||||
]
|
||||
|
||||
_BY_NAME = {c[0]: c for c in _CORPUS}
|
||||
|
||||
|
||||
def _check_row(cur, names: list[str], values: list) -> None:
|
||||
"""Create, insert, then read back from several projections. The
|
||||
sentinel column trails everything so any mis-sizing surfaces."""
|
||||
cols = [f"c{i}" for i in range(len(names))]
|
||||
ddl = ", ".join(f"{c} {_BY_NAME[n][1]}" for c, n in zip(cols, names, strict=True))
|
||||
cur.execute(f"CREATE TEMP TABLE t_fuzz ({ddl}, sentinel INT)")
|
||||
try:
|
||||
marks = ", ".join(["?"] * (len(cols) + 1))
|
||||
cur.execute(f"INSERT INTO t_fuzz VALUES ({marks})", (*values, SENTINEL))
|
||||
|
||||
expected = dict(zip(cols, values, strict=True))
|
||||
expected["sentinel"] = SENTINEL
|
||||
every = [*cols, "sentinel"]
|
||||
|
||||
cur.execute(f"SELECT {', '.join(every)} FROM t_fuzz")
|
||||
assert dict(zip(every, cur.fetchone(), strict=True)) == expected, (
|
||||
f"straight projection: {list(zip(names, values, strict=True))}"
|
||||
)
|
||||
|
||||
for shift in range(1, min(len(every), 4)):
|
||||
order = every[shift:] + every[:shift]
|
||||
cur.execute(f"SELECT {', '.join(order)} FROM t_fuzz")
|
||||
got = dict(zip(order, cur.fetchone(), strict=True))
|
||||
assert got == expected, (
|
||||
f"rotation {shift}: {list(zip(names, values, strict=True))}"
|
||||
)
|
||||
|
||||
cur.execute("SELECT * FROM t_fuzz")
|
||||
got_names = [d[0] for d in cur.description]
|
||||
assert dict(zip(got_names, cur.fetchone(), strict=True)) == expected, (
|
||||
f"SELECT *: {list(zip(names, values, strict=True))}"
|
||||
)
|
||||
finally:
|
||||
cur.execute("DROP TABLE t_fuzz")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("type_name", [c[0] for c in _CORPUS])
|
||||
def test_every_edge_value_with_a_trailing_sentinel(
|
||||
conn_params: ConnParams, type_name: str
|
||||
) -> None:
|
||||
"""Exhaustive over the corpus: each type, each edge value, always with
|
||||
a column behind it."""
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
for value in _BY_NAME[type_name][2]:
|
||||
_check_row(cur, [type_name], [value])
|
||||
|
||||
|
||||
@pytest.mark.parametrize("seed", [1, 2, 3])
|
||||
def test_random_mixed_rows(conn_params: ConnParams, seed: int) -> None:
|
||||
"""Random multi-type rows. Seeded so a failure is reproducible; the
|
||||
point is combinations no one thought to write by hand."""
|
||||
rng = random.Random(seed)
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
for _ in range(25):
|
||||
names = [rng.choice(_CORPUS)[0] for _ in range(5)]
|
||||
values = [rng.choice(_BY_NAME[n][2]) for n in names]
|
||||
_check_row(cur, names, values)
|
||||
|
||||
|
||||
def test_server_agrees_with_what_we_bound(conn_params: ConnParams) -> None:
|
||||
"""The independent oracle. Compare against the server's own rendering
|
||||
rather than our own decoder, so a symmetric encode/decode bug cannot
|
||||
hide — which is exactly how DATETIME lost its fractions."""
|
||||
checks: list[tuple[str, object, str]] = [
|
||||
("INT8", 123456789012, "123456789012"),
|
||||
("INT8", -123456789012, "-123456789012"),
|
||||
("BIGINT", 2**62, str(2**62)),
|
||||
("VARCHAR(64)", "PackageRoot", "PackageRoot"),
|
||||
("LVARCHAR(64)", "PackageRoot", "PackageRoot"),
|
||||
("NVARCHAR(64)", "odd", "odd"),
|
||||
("BOOLEAN", True, "t"),
|
||||
("BOOLEAN", False, "f"),
|
||||
("DATETIME YEAR TO FRACTION(5)",
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
|
||||
"2026-08-31 12:30:15.12000"),
|
||||
("DATETIME YEAR TO SECOND",
|
||||
datetime.datetime(2026, 8, 31, 12, 30, 15),
|
||||
"2026-08-31 12:30:15"),
|
||||
("DATE", datetime.date(2026, 8, 31), "2026-08-31"),
|
||||
]
|
||||
with _connect(conn_params) as conn:
|
||||
cur = conn.cursor()
|
||||
for ddl, value, server_repr in checks:
|
||||
cur.execute(f"CREATE TEMP TABLE t_oracle (v {ddl})")
|
||||
try:
|
||||
cur.execute("INSERT INTO t_oracle VALUES (?)", (value,))
|
||||
cur.execute("SELECT v::LVARCHAR FROM t_oracle")
|
||||
(text,) = cur.fetchone()
|
||||
assert text.strip() == server_repr, (
|
||||
f"{ddl}: bound {value!r}, server stored {text.strip()!r}, "
|
||||
f"expected {server_repr!r}"
|
||||
)
|
||||
finally:
|
||||
cur.execute("DROP TABLE t_oracle")
|
||||
Loading…
x
Reference in New Issue
Block a user