warehack.ing/informix-db
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: warehack.ing:v2026.05.04
Branches Tags
warehack.ing:main
warehack.ing:v2026.09.03
warehack.ing:v2026.09.02
warehack.ing:v2026.09.01
warehack.ing:v2026.08.31.3
warehack.ing:v2026.08.31.2
warehack.ing:v2026.08.31.1
warehack.ing:v2026.08.31
warehack.ing:v2026.08.27
warehack.ing:v2026.05.05.10
warehack.ing:v2026.05.05.9
warehack.ing:v2026.05.05.8
warehack.ing:v2026.05.05.7
warehack.ing:v2026.05.05.6
warehack.ing:v2026.05.05.5
warehack.ing:v2026.05.05.4
warehack.ing:v2026.05.05.3
warehack.ing:v2026.05.05.2
warehack.ing:v2026.05.05.1
warehack.ing:v2026.05.05
warehack.ing:v2026.05.04.10
warehack.ing:v2026.05.04.9
warehack.ing:v2026.05.04.8
warehack.ing:v2026.05.04.7
warehack.ing:v2026.05.04.6
warehack.ing:v2026.05.04.5
warehack.ing:v2026.05.04.4
warehack.ing:v2026.05.04.3
warehack.ing:v2026.05.04.2
warehack.ing:v2026.05.04.1
warehack.ing:v2026.05.04
...
compare: warehack.ing:main
Branches Tags
warehack.ing:main
warehack.ing:v2026.09.03
warehack.ing:v2026.09.02
warehack.ing:v2026.09.01
warehack.ing:v2026.08.31.3
warehack.ing:v2026.08.31.2
warehack.ing:v2026.08.31.1
warehack.ing:v2026.08.31
warehack.ing:v2026.08.27
warehack.ing:v2026.05.05.10
warehack.ing:v2026.05.05.9
warehack.ing:v2026.05.05.8
warehack.ing:v2026.05.05.7
warehack.ing:v2026.05.05.6
warehack.ing:v2026.05.05.5
warehack.ing:v2026.05.05.4
warehack.ing:v2026.05.05.3
warehack.ing:v2026.05.05.2
warehack.ing:v2026.05.05.1
warehack.ing:v2026.05.05
warehack.ing:v2026.05.04.10
warehack.ing:v2026.05.04.9
warehack.ing:v2026.05.04.8
warehack.ing:v2026.05.04.7
warehack.ing:v2026.05.04.6
warehack.ing:v2026.05.04.5
warehack.ing:v2026.05.04.4
warehack.ing:v2026.05.04.3
warehack.ing:v2026.05.04.2
warehack.ing:v2026.05.04.1
warehack.ing:v2026.05.04

67 Commits
v2026.05.0 ... main

Author SHA1 Message Date
Ryan Malloy
c85aaf2ecb Docs: put named row access where someone leaving IfxPy will find it 
The API reference gained the Row section with the release, which is the
right home for the details and the wrong place to discover the feature
exists. Three gaps closed.

The IfxPy migration guide is the page for people in exactly the position
that prompted this: IfxPy is positional-only, so those codebases grow a
description/zip helper that yields a dict and still no attribute access.
That page now shows the helper it replaces alongside the one-liner, and
notes the .lower() in the hand-written version is already a no-op since
Informix folds unquoted identifiers.

Quickstart gains one sentence, next to the paramstyle note where someone
is already thinking about how rows come back.

The phase log records 2026.09.03, including where the testing actually
found the shadowed-names bug: not in the differential test across twenty
awkward types, which found nothing, but in six lines of name bookkeeping
that were correct when written and stopped being correct an hour later.

Verified live by content, and the #rows anchor the new cross-links point
at resolves.
 2026-09-03 16:06:34 -06:00
Ryan Malloy
acc21b8b81 Docs: row_factory reference, and the write-path corruption warning 
The API page documents Row: the three access styles, that it is set on
the connection, that it subclasses tuple so existing code is unaffected,
the measured 9% and why that makes it opt-in, and the four behaviours
worth knowing (lower-case folding, expression column names, duplicate
names, and a column beating a method of the same name).

The types page gains the placeholder-rewriter entry. It is the only one
in that list affecting the write path rather than the read path, so
anyone who has stored HH:MM times or URLs with ports through this driver
should check what actually landed.

Test counts updated to 472/471. Verified live by content.
 2026-09-03 16:00:53 -06:00
Ryan Malloy
9fbb97199f Release 2026.09.03 — placeholder rewriter, and named row access 
The rewriter that converts :N placeholders to ? was a bare regular
expression, so it rewrote the inside of string literals. Any HH:MM time,
any URL with a port, any key:value string was silently stored wrong. It
was not opt-in either: the rewrite runs whenever a statement has
parameters, whatever placeholder style the caller used. Replaced with a
scanner that substitutes only outside quotes and comments, using
Informix's own lexical rules measured against all three servers.

Row adds column-name and attribute access alongside the existing
positional access, on request from a user coming from pyodbc and
mssql-python. Opt-in at connect time, because it costs about 9% on bulk
fetch and that is not a trade to make for someone who never reads a
column by name.

472 tests on 15 and 14.10, 471 on 12.10.
 2026-09-03 15:59:26 -06:00
Ryan Malloy
ce5a582048 Testing Row found the same mistake I keep making 
I hand-listed the names a column could shadow: count and index, the two
tuple methods. Then I wrote keys(), _asdict() and _fields on the same
class and did not go back. A column called "keys" returned a bound
method instead of a value, silently, which is precisely the failure
shape this driver spent a fortnight removing from its decoders.

The fix is not a longer list. The reserved set is now computed from
dir(Row), so adding a method later cannot reopen the hole, and the test
is parametrized over that computed set so it grows with the class.

_fields and _map moved to name-mangled attributes. They are read by
repr() and _asdict(), so shadowing the public _fields with a column
would have made the machinery report the column value instead of the
column names. Mangling keeps the two apart, and there is a test that
shadows _fields and checks repr still works.

Also exercised, and all clean: value-identical output against the plain
tuple path across twenty columns covering every awkward type plus a
fully NULL row, which is the strongest available statement that Row is a
presentation layer and not a bug; zero-column and 500-column rows;
unicode, spaced, digit-leading, empty-string and dunder column names;
rows outliving eviction of their class from the bounded cache; the four
async fetch routes and the async pool; and twelve threads racing on the
class cache, which correctly share one class per shape.

472 tests on 15 and 14.10, 471 on 12.10.
 2026-09-03 15:51:46 -06:00
Ryan Malloy
805fa58eb2 Rows can answer to a column name, if you ask for it 
Field request from a user running this alongside SQL Server, where both
pyodbc and mssql-python hand back rows that take a position, a column
name, and an attribute. The argument is readability on wide
projections: row[11] tells a reader nothing, and stays correct only
until somebody adds a column in the middle.

row_factory=Row gives all three. It is set on the connection, so it is
one line for the whole application rather than per query, which is what
"without any additional coding" has to mean in practice. A cursor can
override it.

Opt-in, not default, and the number is why. Measured on a 20,000-row
five-column fetch: 37.2 ms with tuples, 40.8 ms with Row, about 9%.
Defaulting it on would move the published 1.05-1.15x ratio against IfxPy
to roughly 1.15-1.25x. That is not a trade to make on behalf of somebody
running a bulk export that never looks at a column by name.

Row subclasses tuple, so row == (1, "x") still holds. That constraint
shaped the design more than anything else: this suite compares fetched
rows against plain tuples in hundreds of places, and so does everybody's
code. Slices degrade to plain tuples, since a slice has no column map.

Details that had to be decided rather than assumed, all measured against
the servers: Informix folds unquoted identifiers to lower case, so the
.lower() in the workaround people write by hand is a no-op. Expression
columns get names like "(count(*))" that cannot be attributes, so they
are subscript-only. Duplicate names resolve to the first occurrence,
matching pyodbc. And tuple already defines count and index, so a column
with either name gets a descriptor and wins, because on a database row
that is plainly what the caller meant.

The per-shape class is cached, so a type() call does not land on every
small query, and rows pickle by rebuilding from their field names.
 2026-09-03 15:42:56 -06:00
Ryan Malloy
088171325d The placeholder rewriter could not see a string literal 
We advertise paramstyle="numeric" and the wire protocol takes ?, so :N
placeholders are rewritten on the way out. That was a bare
re.sub(r":(\d+)", "?", sql), which rewrote the inside of string literals:

    UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
                      stored as 'http://host?/x'

Any HH:MM time, any URL with a port, any aspect ratio, any key:value
string. It wrote wrong data and said nothing about it.

None of it was opt-in. The rewrite fires whenever a statement has
parameters, whatever placeholder style the caller actually used, so
writing ? everywhere and never touching numeric style did not protect
you. It also changed the placeholder count while num_qmarks was still
computed from len(params), leaving driver and server disagreeing about
how many binds exist.

Replaced with a single pass that substitutes only outside quotes and
comments. The lexical rules are Informix's own, measured against all
three servers rather than assumed from standard SQL, and one of them
would have been got wrong by habit: a backslash escapes nothing, so
'a\'b' is an unterminated string and draws -282. A scanner written to
Postgres reflexes would desync there and corrupt everything after it.
Block comments do not nest, the first */ closes them. Braces are a
comment. :: is stepped over as a unit so a cast can never be read as the
start of a placeholder.

An unterminated quote or comment consumes the rest and substitutes
nothing further. Under-substituting leaves the server to reject SQL that
was already malformed; guessing would corrupt a literal.

This was the last place the driver inferred meaning from SQL text
instead of handling it properly. Eleven of the new tests fail against
the previous commit.
 2026-09-03 12:45:58 -06:00
Ryan Malloy
7ffc148112 Docs: cover the 2026.09.02 review, and drop every em-dash 
Content. The types page now warns about the two framing bugs this
release fixed, since both silently corrupt data on versions before it:
a BLOB or CLOB in any position but last shifted every column after it,
and a NULL collection did the same. The API page documents two things
that are now user-visible: only one scrollable cursor may be open per
connection (previously the second statement drew -285 and destroyed the
cursor as collateral), and transaction control written as SQL is tracked
(previously rollback() after a SQL BEGIN WORK sent nothing and reported
success). The phase log gains a section on the review itself, including
the pattern behind almost all eleven bugs. Test counts updated to
457/456.

Style. 136 em-dashes across 22 content files, plus Hero.astro and two
stylesheets, rewritten rather than substituted -- swapping the character
for a comma leaves prose that reads like it lost an argument with a
linter. Bullets that used a dash to gloss a term now use a colon;
parenthetical asides became their own sentences or moved inside
brackets. The rendered HTML is clean.

Verified against the live site by content, not status code: these pages
return 200 for every path and render the 404 body, so a stale deploy
looks perfectly healthy.
 2026-09-02 10:36:44 -06:00
Ryan Malloy
8f341c4b2a Release 2026.09.02 — a systematic review, and eleven bugs it found 
No new features. The result of going back over the driver looking for
the shape of past bugs rather than for new symptoms.

Two returned wrong data silently: smart LOBs were read 77 bytes short,
shifting every column after them, and NULL composite UDTs skipped their
length field. One did nothing at all: rollback() after a SQL BEGIN WORK
returned successfully having sent nothing, leaving the rows it was asked
to discard in place and the connection back in the pool holding an open
transaction.

The rest: five ordinary query forms failing with -260 because
classification guessed from the first word; a scrollable cursor taking
the whole connection down with it; two unguarded exits leaking failed
statements; deferred cleanup landing inside somebody else's statement,
and a GC-time finalizer that couldn't tell it was already on the thread
holding the wire lock; the async layer borrowing the process-wide thread
pool; and two readers sharing one socket stream without agreeing.

457 integration tests on 15 and 14.10, 456 on 12.10 (one skip, no CTEs
before 14.10) -- up from 414. Every fix has at least one test that fails
against 2026.09.01.
 2026-09-02 10:28:27 -06:00
Ryan Malloy
429a3910e4 The async layer borrowed a thread pool belonging to the whole process 
Every blocking call in informix_db.aio went through asyncio.to_thread,
which runs on the event loop's default executor. That executor belongs
to the process, not to us, and it is sized from the CPU count --
min(32, cpu_count + 4), so six threads on a two-CPU container.

Cancelled calls hold their threads. asyncio.to_thread cannot interrupt a
worker, so a cancelled await leaves the thread running the wire call
until the read timeout. Cancellation is ordinary in a web app -- a
client disconnect cancels the request task -- so a handful of them pins
every thread in the shared pool. Measured: six cancelled calls against a
six-worker default executor starve an unrelated to_thread indefinitely,
and with the executor held, four concurrent driver queries never ran at
all.

Pool concurrency was capped by the same number without saying so. A pool
with max_size=20 on a two-CPU box ran six queries at a time.

Each connection now owns one thread. That is the right size rather than
a compromise -- the sync connection serializes every wire operation on
its own lock, so a second thread could do nothing but wait for the
first. It also avoids a deadlock a shared pool-sized executor invites:
with N threads and N connections, N tasks blocked in acquire occupy
every thread while the connection they are waiting for is held by a task
that now needs a thread to finish and release it.

The executor lives on the sync connection so it survives being returned
to the pool and handed out again, rather than being rebuilt per acquire.
release() runs on the connection's own thread, which is idle by
definition and keeps release off any pool that waiters may have filled
-- release has to win that race, since it frees what they are waiting
for. connect() and pool acquire stay on the default executor: there is
no connection yet to own a thread, and neither can deadlock against
query threads any more.

close() shuts the executor down, and a weakref finalizer is the backstop
for a connection dropped without it -- ThreadPoolExecutor workers park
on the work queue rather than exiting when idle, so an executor that is
never shut down leaks its thread for the life of the process. The
thread-count test caught that gap; it was missing from the first cut.
 2026-09-02 01:03:12 -06:00
Ryan Malloy
5ad296419c Two readers shared a stream without agreeing, and a length was taken on trust 
IfxSocket owns a read-ahead buffer that BufferedSocketReader fills and
drains. Connection._drain_to_eot, _raise_sq_err and the login path
bypass that reader and call IfxSocket.read_exact directly, which recv'd
from the socket without looking at the buffer. Bytes sitting in the
buffer were skipped, and skipped bytes in a length-framed protocol do
not announce themselves -- the next read lands mid-field and every read
after it is wrong.

Nothing triggers it today. The server sends one response per request, so
recv returns exactly that response and the buffered reader consumes all
of it before control returns to a direct read. That is a property of the
traffic, not of the code, and the buffer is connection-scoped precisely
so read-ahead can cross response boundaries -- pipelined executemany
already puts several responses in flight. read_exact now drains the
buffer first, which costs one branch on a cold path and makes the two
paths agree by construction rather than by luck.

fill_recv_buf believed whatever byte count it was handed, and that count
is almost always a length field straight off the wire. A garbage
0x7FFFFFFF reads as a 2 GB request and the fill loop sits in recv until
the read timeout while the buffer grows. It now refuses above
IFX_MAX_READ_BYTES (256 MiB default) with an error naming the number,
which is the actual diagnostic: a length that absurd means framing was
already lost upstream.

BufferedSocketReader.skip advanced the cursor arithmetically with no
guard, so a negative count rewound it and re-decoded consumed bytes as
the next field. The base reader's skip delegates to read_exact and does
guard; this one diverged.

Transaction control run as SQL desynced Connection._in_transaction,
which is what commit() and rollback() are guarded by and what the pool
reads to decide whether a returned connection needs cleaning up. With
autocommit on, cursor.execute("BEGIN WORK") opened a real transaction
while the flag stayed False, so rollback() returned successfully having
sent nothing and the rows it was asked to discard survived. The
connection then went back to the pool holding an open transaction and
its locks. With autocommit off it failed instead: the driver's implicit
SQ_BEGIN fired first and the caller's BEGIN WORK got -535.

The server labels these -- 34 BEGIN, 35 COMMIT, 36 ROLLBACK, with and
without the WORK keyword, measured on all three versions. JDBC reads the
same values off the describe and calls setTxBeginState/setTxEndState.
_ensure_transaction moves to after the describe, matching JDBC's
initiateTransaction placement, which is what makes it possible to skip
for transaction control at all -- until the describe lands you cannot
know that is what the statement is.
 2026-09-02 00:54:12 -06:00
Ryan Malloy
c67bbc4766 Statement classification guessed from the first word and got five forms wrong 
The driver chose between "open a cursor and fetch" and "execute and
release" by checking whether the first word of the SQL was SELECT. That
misses a leading comment in any of the three Informix flavours (--,
/* */, { }), a parenthesized select, a parenthesized UNION, and a CTE.
All five are perfectly ordinary queries. All five failed with -260,
"Cursor name already in use" -- an error that describes neither the
cause nor anything the caller did. It says "cursor" because the driver
sent SQ_EXECUTE where the server was waiting to open one.

The server had been answering this question all along. statement_type is
the first field of the DESCRIBE response and parse_describe has always
put it in the metadata dict, where nothing read it.

The comment defending the heuristic said nfields couldn't distinguish
these cases because INSERT INTO t VALUES (?) also describes a column.
That was true, and it argued for the wrong conclusion: statement_type
distinguishes them exactly. Measured against all three servers -- every
SELECT form above reports 2, that INSERT reports 6, UPDATE 33, DELETE
32, CREATE 45, EXECUTE PROCEDURE 56.

The predicate is now JDBC's IfxSqli.isResultSet: type 2, or type 56 with
at least one column, since a routine may or may not return rows. That
last clause fixes EXECUTE FUNCTION as a side effect -- it used to run
down the DML path and discard the return value. It now yields it.

executemany keeps the first-word check as a cheap pre-flight reject, and
gains an authoritative one after PREPARE for the forms the first word
misses. The refusal releases the statement, so the connection survives.

Informix 12.10 has no CTEs and rejects WITH with -201 at offset 1. That
is the server declining the grammar rather than the driver routing it
wrongly, and the test skips on it.
 2026-09-02 00:38:31 -06:00
Ryan Malloy
2eb5ac8f0f The finalizer's lock probe couldn't see its own thread 
The cursor finalizer runs at GC time on whatever thread happened to
allocate, so it must not touch the wire while a statement owns it. It
checked with _wire_lock.acquire(blocking=False), meaning to ask "is
anyone using the wire?" -- but an RLock grants a reentrant acquire to
its own owner. When GC fired on the thread that was mid-statement, the
probe returned True, the finalizer concluded it had exclusive access,
and it sent CLOSE/RELEASE into the middle of the statement it had just
interrupted. The victim got -208 on valid SQL.

Refcounting hid it. A dropped cursor is freed at the drop, before the
next statement runs. A cursor caught in a reference cycle waits for a
collection instead, and cycles are ordinary -- any traceback that holds
a cursor makes one. Reproduced by putting an abandoned scrollable cursor
in a cycle and collecting mid-statement.

_wire_lock is now a small wrapper that tracks owner and depth, so
held_by_current_thread answers the question the finalizer was actually
asking. Same-thread GC defers to the cleanup queue exactly as another
thread's would. This also retires the _is_owned() call in
_ensure_transaction, which was reaching into CPython private API for
the same information.

The finalizer's error handling had the stale-cleanup defect too: a
leftover CLOSE draws -267, an OperationalError, which is in WIRE_ERRORS,
so it force-closed a healthy connection over a no-op. Server-reported
errors are told apart from wire failures by their sqlcode, matching
_drain_pending_cleanup.
 2026-09-02 00:29:04 -06:00
Ryan Malloy
ac00a25cb7 A session has one statement slot and the driver acted as if it had many 
SQLI gives a session a single statement slot. SQ_CLOSE, SQ_RELEASE and
SQ_SFETCH act on whatever statement is current -- none of them names
one. Ordinary use never notices, because a non-scrollable cursor
materializes its rows and releases the statement before returning.

A scrollable cursor holds the slot open on purpose, and two things went
wrong there without saying so.

Another statement on the same connection got -285, and the scrollable
cursor was collateral damage: its next fetch came back -267, "the
transaction has been rolled back, all locks released". Two errors,
neither naming the cause, from code that reads as completely ordinary --
iterate a large result set, run a lookup partway through. It is now a
ProgrammingError that explains the constraint, and the scrollable cursor
is left alone. Re-executing the same scrollable cursor is the one caller
allowed past the check, and it now closes its own server-side cursor
first, because it collided with itself too.

The deferred-cleanup queue was flushed before every PDU. But a finalizer
enqueues precisely because it lost the wire lock, meaning another thread
is mid-statement -- so the flush landed inside that thread's own
statement and released it. The victim saw -208 when it landed before the
first fetch, -267 between fetch batches. Flushing only at a statement
boundary is both correct and sufficient: cleanup that misses one
statement is picked up by the next.

A stale queue entry was fatal on top of that. The finalizer enqueues,
the cursor is then closed properly, and the leftover CLOSE draws -267 --
an OperationalError, which is in WIRE_ERRORS, which force-closed a
healthy connection. Server-reported errors are now told apart from wire
failures by whether they carry a sqlcode.

Noted while tracing this: _build_close_pdu and _build_release_pdu write
SQ_ID followed by write_int(opcode), which frames correctly only because
the four bytes happen to land as [statement_id=0][opcode]. JDBC's
sendStatementID writes the real id there, and _read_describe_response
has been parsing it into _statement_id all along without ever using it.
Addressing CLOSE/RELEASE correctly is not enough for multiplexing on its
own -- a second cursor's SFETCH addressed to its own id returns -259 --
so that stays open rather than half-done.
 2026-09-02 00:24:09 -06:00
Ryan Malloy
0d8bd57ba9 The last two doors out of a statement didn't release it 
A statement that fails is still allocated server-side, and skipping the
RELEASE bricks the connection: the next PREPARE collides with the leaked
one and every later call returns a nonsense error whose offset points
back at the failed SQL. That guard has now been added six times, once
per exit, each after a user found the exit. Two were still open.

The parameterized-SELECT bind drain. The build was guarded and the drain
was not, and the drain is where the interesting failures land: passing a
string where the column is an INT encodes cleanly, so the rejection
comes from the server (-1213, -415) after the guard has been passed. A
wrong-typed parameter is about as common as application mistakes get.

The scrollable-cursor open, which had no guard at all. Worst place to
lack one -- the GC-time finalizer is armed on the line after the drain,
so a failure there left the statement allocated with no fallback of any
kind. SELECT ... FOR UPDATE reaches it: prepares clean, fails at OPEN
with -526.

All six now call one helper instead of hand-rolling the cleanup, which
also fixes a defect in the two copies that had a cursor to close. CLOSE
and RELEASE shared a single contextlib.suppress block, so a CLOSE that
raised skipped the RELEASE -- losing the half that matters, since a
leaked cursor handle is a nuisance and a leaked statement breaks the
next call. Each PDU gets its own suppression now.

All four new integration tests fail against the previous commit.
 2026-09-02 00:10:23 -06:00
Ryan Malloy
66120cf6f3 Rows never checked that they consumed their own payload 
Fourteen framing bugs shipped before this. Every one was a column
reading the wrong number of bytes, and not one of them raised at the
point of the mistake -- the wrong width produced a plausible value and
corrupted whatever came next, so the damage surfaced in a different
column, a different row, or a different statement entirely.

All fourteen were detectable for free. The payload is a fully extracted
bytes object of known length, so after decoding N columns the offset has
to land exactly on the end. It didn't, and nobody looked.

_row_not_consumed now raises when it doesn't, naming the column shape
and the byte delta, and it is wired into all three decode paths --
readers fast path, legacy chain, and the generated decoder.

Turning it on found two more bugs on the first run:

Smart LOBs were read as a flat 72-byte field. They use the UDT envelope:
149 bytes populated ([ind=0][len=144][144 hex chars]), 5 bytes NULL. We
consumed 72 and left 77 behind, so a BLOB or CLOB anywhere but the final
column position shifted every column after it. The 144 bytes are the
72-byte locator hex-encoded, which means BlobLocator.raw had never held
a locator, only the first half of the hex text -- unnoticed because
read_blob_column resolves through the server's lotofile and never reads
the locator.

NULL composite UDTs skipped their length field, byte for byte the
LVARCHAR NULL bug in a branch twelve lines away.

Both are the same [indicator][int32 length][data] envelope, hand-written
a third and fourth time. _read_udt_envelope is now the only copy, and it
rejects a negative length instead of rewinding the offset into bytes it
already decoded.
 2026-09-02 00:04:26 -06:00
Ryan Malloy
2e30ceacfb Regenerate uv.lock for 2026.09.01 2026-09-01 22:32:33 -06:00
Ryan Malloy
2896f7e93e TLS traffic fuzzed; no bugs found (2026.09.01) 
Tests and docs only, no behaviour change. TLS was the last untested
surface and it came back clean.

Previous coverage stopped at the handshake; everything after it ran only
over plain sockets. That gap mattered because SSLSocket.recv is not
socket.recv: it returns at most one TLS record's worth of plaintext
however much you ask for, can return fewer bytes than are available, and
plaintext buffered inside the SSL object is invisible to the OS. The
Phase 39 buffered reader asks for up to 64 KB per call and loops until
satisfied — that loop had never been pressured the same way.

tests/test_tls_traffic.py runs real SQLI through a TLS-terminating
proxy: the framing-bug types end to end, payloads at 1/4096/16383/16384/
16385/32000 bytes (straddling the ~16 KB record boundary), 500- and
5000-row bulk fetches, error recovery, concurrent TLS sessions, and
three negative cases — TLS client against a plaintext port, plaintext
client against a TLS port (must raise, not hang), and a verifying
context rejecting a self-signed cert.

All clean on all three servers. The buffered reader handles SSLSocket
semantics correctly.

Scope, stated plainly: the proxy supplies the TLS half, so this covers
the driver's TLS path — the half we own. It does NOT cover IBM's
server-side TLS listener. Setting one up on the developer-edition image
was attempted and abandoned: Informix 15 wants a PKCS#12 keystore
(onkstash takes a .p12, not the older CMS .kdb) and the engine kept
rejecting the stash with GSK_ERROR_BAD_KEYFILE_PASSWORD even with a
keystore GSKit itself could open. That half is IBM's code; everything
below ssl.wrap_socket is identical either way.

414/414 integration on 15, 14.10 and 12.10 (was 399).

Every surface is now fuzzed: type framing, fetch batching, error
recovery, cursor lifecycle, threads, pooling, transactions, async
cancellation, executemany partial failure, scrollable cursors, smart
LOBs, and TLS.
 2026-09-01 22:32:10 -06:00
Ryan Malloy
488773f078 Regenerate uv.lock for 2026.08.31.3 2026-08-31 19:13:32 -06:00
Ryan Malloy
ac9075ca9a An encoding failure inside executemany leaked the statement (2026.08.31.3) 
Last of the untested surface: pipelined executemany, scrollable cursors,
and smart LOBs. One bug, in the first.

executemany builds all its BIND+EXECUTE PDUs after the PREPARE and
before anything is drained — that batching is what makes the pipeline
fast. If encoding a row raises there (a value the connection's codec
cannot represent), the exception escaped without sending the RELEASE,
leaking the prepared statement. The next PREPARE collided with it and
every later call on the connection failed with an error pointing at the
previous SQL.

Same failure as 2026.08.31.2, in a sibling path.
_execute_dml_with_params has guarded this exact case for the single-row
path for a long time; the pipelined path was never given the same
treatment. That is the recurring shape of these bugs: a hazard
understood in one place and not carried to the code beside it.

What the fuzzer found sound, which is the larger part of the result:

  executemany constraint failures — duplicate-key and NOT NULL at
  first/mid/last of batches from 2 to 1000 rows all recover, and
  COUNT(*) always agrees with a full fetch, so the drain-N-responses
  invariant holds under partial failure.

  Scrollable cursors — fetch_first/last/prior/relative/absolute correct
  at 0, 1, 2, 5, 50, 300 rows, including off both ends (None, not a wrap
  or a crash) and a full forward walk after arbitrary positioning.
  Twenty abandoned scroll cursors leak nothing.

  Smart LOBs — round-trip at 0, 1, 255, 256, 1023, 1024, 4095, 4096,
  65535, 65536 bytes, straddling the 4096-byte SQ_FILE chunk and the
  64K mark, plus recovery from failed reads.

Still not fuzzed, stated plainly: TLS is handshake-tested against a
self-signed local socket, not a real Informix TLS listener (that needs
server-side keystore + onconfig SSL setup absent from the test
containers). The SQLI layer above the socket is identical either way.

399/399 integration on 15, 14.10 and 12.10 (was 356).
 2026-08-31 19:11:05 -06:00
Ryan Malloy
3d19ca0f1b Regenerate uv.lock for 2026.08.31.2 2026-08-31 17:17:30 -06:00
Ryan Malloy
4af1c90c6f Failed statements bricked the connection; cancelled acquires starved the 
pool (2026.08.31.2)

Two more found by fuzzing rather than by users, outside the type system.
Both are ordinary-path bugs a happy-path test cannot reach.

A failed statement was never released. Successful DML sent PREPARE ->
EXECUTE -> RELEASE; failing DML sent PREPARE -> EXECUTE and stopped. The
statement stayed allocated server-side, collided with the next PREPARE,
and every subsequent call on that connection returned a nonsense error
(-255 "Not in transaction" under autocommit, -285 otherwise) whose
offset pointed back at the FAILED sql rather than the new statement.

In practice: one duplicate-key violation bricked the connection. An
INSERT tripping a unique constraint is about the most routine error an
application can hit — every insert-if-not-exists pattern makes them —
and afterwards nothing on that connection worked.

The docstring on the parameterised path already described this hazard
and guarded the parameter-ENCODING failure; it never covered EXECUTE
itself failing. Fixed in all three paths (DML, parameterised DML, and
SELECT — whose fetch loop had the same gap, and whose GC-time finalizer
only covers scrollable cursors).

Cancelling a pool acquire leaked the connection permanently.
asyncio.to_thread cannot interrupt its worker, so a task cancelled while
blocked in pool.acquire() left the worker to finish and hand back a
connection nobody owned — checked out, never returned, one slot gone per
occurrence until the pool was dead.

Not theoretical for anything serving HTTP: a client disconnect cancels
the request task, and under load those cancellations land precisely
while waiting for a connection. The pool dies one slot at a time, only
under load, and PoolTimeoutError points nowhere near the cause.
acquire() now shields the inner future and returns whatever the worker
produced if the caller went away; add_done_callback fires immediately on
an already-resolved future, so the "worker finished just before the
cancel" race is the same code path rather than a second one.

The harness covered three things the type fuzzer cannot see: fetch
batching (NFETCH is a 4096-BYTE budget, so batch edges move with row
width — counts 0..1025 across three schema widths, every fetch style
required to agree), error recovery (eight error classes, repeated, each
followed by a known-good query on the same cursor and a fresh one), and
concurrency (threads sharing one connection, pooled threads that fail
before releasing, transaction isolation across checkouts, async
cancellation). Every worker reads back a value only it supplied, so a
crossed wire fails on data even when nothing raises.

Two harness false positives worth recording: Informix silently truncates
over-long strings (a literal SQL insert does the same, so the driver
matches the server), and it rejects a bare `?` in a projection — a cast
is required.

356/356 integration on 15, 14.10 and 12.10 (was 326).
 2026-08-31 17:13:43 -06:00
Ryan Malloy
4e186bb890 Regenerate uv.lock for 2026.08.31.1 2026-08-31 16:01:34 -06:00
Ryan Malloy
15d75dd052 Type-matrix fuzzer, and the three bugs it found (2026.08.31.1) 
Six framing bugs had reached users across three reports. Rather than
wait for a seventh, this adds a harness built to find that class of bug.
It immediately found three more, one of which hangs the connection.

BOOLEAN could not be bound as a parameter at all. _encode_bool emitted
type code 45, which the server does not accept as a bind type — it
simply stopped responding, so any execute() with a bool parameter hung
until the read timeout, or forever without one. Type 41 (how BOOLEAN is
*described* in results) hangs identically; descriptor type and bind type
are not interchangeable. Now binds 't'/'f' as CHAR and lets the server
cast, mirroring Informix's own literal syntax. A hang is worse than
wrong data, and this shipped in every release claiming BOOLEAN support.

Unscaled DECIMAL was read one byte short, shifting every following
column. Width is (precision + (scale & 1) + 3) // 2 per
IfxColumnInfo.adjustedColumnLength; we dropped the `scale & 1` term.
That only matters when precision is even and scale is odd — and an
unscaled DECIMAL(p) reports scale 255. DECIMAL(16) is 10 bytes, not 9.
Verified on the wire for ten DECIMAL/MONEY shapes. The same formula
governs DATETIME and INTERVAL, whose qualifier parity tracks their digit
count, so those agreed by coincidence; all four now share one helper
taken from the reference.

NULL CHAR/NCHAR came back as '', indistinguishable from an empty column,
so `WHERE c IS NULL` disagreed with what the driver returned. The wire
distinguishes them plainly — NULL is a leading 0x00, empty is all
spaces.

The harness (tests/test_type_matrix.py) encodes three principles, each
derived from how a real bug escaped:

  1. Always put a column after the value under test. Every bug so far
     corrupted the NEXT column; a trailing value can be mis-sized
     invisibly. Every case ends in a sentinel, and projections rotate.
  2. Vary the data, not just the type. Branch coverage is worthless if
     no value takes the branch — the LVARCHAR fixture was 'lv value',
     8 chars, even, never NULL, so both broken branches sat unexecuted
     through 247 tests.
  3. Use an oracle our codecs don't share. A Python round-trip cannot
     catch a symmetric encode/decode bug: our DATETIME encoder wrote
     zeros and our decoder read them back in perfect agreement. Asking
     the server to render the stored value breaks that symmetry.

Exhaustive over the corpus plus seeded random multi-type rows, so
failures reproduce.

326/326 integration on 15, 14.10 and 12.10 (was 281). Fuzzer reports
clean over 80-round runs at several seeds and widths on all three.
 2026-08-31 15:48:48 -06:00
Ryan Malloy
5c6991efba Regenerate uv.lock for 2026.08.31 2026-08-31 14:32:50 -06:00
Ryan Malloy
9616ddbc0a Fix LVARCHAR tuple framing; DATETIME fractions on bind (2026.08.31) 
More data corruption from the same field report as 2026.05.08.2. Anyone
with LVARCHAR columns should upgrade: 2026.08.27 is not safe for them.

Two independent errors in the LVARCHAR envelope, both shifting every
column selected after it.

1. A phantom pad byte. We appended an even-byte pad when the value
   length was odd. There is no pad. Wire capture (12.10), INT8 /
   LVARCHAR / INT8:
     00 01 00 00 07 d1 00 00 00 00   a = INT8 2001
     00                              null indicator
     00 00 00 0b                     length 11
     50 61 63 6b 61 67 65 52 6f 6f 74   "PackageRoot" (odd)
     00 01 00 00 00 0a 00 00 00 00   b = INT8 10, starts immediately

2. A missing length on NULL. We returned as soon as the indicator said
   NULL, leaving its 4-byte length unread. The length belongs to the
   envelope and is always present:
     'odd' -> 00 | 00 00 00 03 | 6f 64 64    8 bytes
     NULL  -> 01 | 00 00 00 00               5 bytes
     ''    -> 00 | 00 00 00 00               5 bytes
   NULL and empty string differ only in the indicator byte.

Reported symptom: INT8 10 decoding as 2560 (the same value shifted one
byte left), strings losing their first character, and IndexError or
"INT8 payload too short" once the drift ran off the payload. The
reporter isolated it by reordering columns in the projection — right
when first, wrong when later. That's the fingerprint of positional
drift and a genuinely good diagnostic.

Missed by 247 tests because the only LVARCHAR fixture was 'lv value':
8 characters, even, never NULL. Neither faulty branch ever ran. Same
gap shape as last time — code paths covered, the data reaching them
not. New tests vary parity (0,1,2,3,11,255,256), cover NULL and empty
separately, always place a column AFTER the LVARCHAR, and rotate the
projection through every position.

Separately, DATETIME lost sub-second precision on INSERT. The encoder
emitted YEAR TO SECOND unconditionally, so binding a datetime with
microseconds into a FRACTION(n) column stored zeros, silently. Reads
were always right, so it only went missing on the way in. Now widens
to FRACTION(5) when microsecond is non-zero and keeps the original
encoding otherwise, so the well-exercised path stays byte-identical.
Binding into a narrower column still truncates server-side.

Also: server_version reported 9.56 for a 12.10 server, which reads like
a client-SDK version. The login response only carries the internal
protocol version, and documenting that didn't make the name less
misleading. server_version now returns the release via one cached
DBINFO query; server_version_internal returns the raw login string.

281/281 integration on 15, 14.10 and 12.10; 123 unit tests.
 2026-08-31 14:26:17 -06:00
Ryan Malloy
85d3a8f7d7 Docs site: correct APIs that don't exist, document what does 
Auditing the docs site against the actual module surface — prompted by
the same field report as 2026.05.08.2 — turned up a partly fictional API.
Every claim below was checked by calling it, not by reading the source.

Documented but nonexistent, now removed:
  conn.transaction()        AttributeError (neither sync nor async)
  conn.autocommit           never existed; set via connect(autocommit=)
  cursor.lastrowid          never existed
  cursor.read_clob_column   never existed
  cursor.write_clob_column  never existed

A copy-pasteable example in reference/types.md raised TypeError:
IntervalYM takes a single total month count, not (years=, months=).

False descriptions corrected: RowValue was described as "named-tuple-like
with .name-accessible fields" and CollectionValue as "iterable" with
"indexed access". Neither is true — both are opaque wrappers over raw
bytes plus a schema string. The docs now say so and point at SQL
projection as the way to get fields today.

Replacements are the real idioms, taken from passing tests:
  CLOBs        -> write_blob_column(..., clob=True); read returns bytes
  SERIAL value -> SELECT DBINFO('sqlca.sqlerrd1')
  transactions -> commit()/rollback() in try/except

Also added: INT8/SERIAL8/BIGSERIAL to the type table (with why INT8 is
not BIGINT), server_capabilities + server_version, the scrollable-cursor
methods, and the note that server_version reports the internal protocol
version (12.10 says 9.56, 14.10 says 9.59).

Updated the test-count claim to 400+ across three server versions,
replacing "integration tests run against 15.0.1.0.3DE".

Added a checker pass over every conn.*/cursor.* reference in the docs;
the only remaining unresolved names are deliberate mentions of IfxPy
APIs we don't implement. Site builds clean.
 2026-08-27 09:29:02 -06:00
Ryan Malloy
f5a539d4a8 Add py.typed marker (PEP 561) 
pyproject has carried the "Typing :: Typed" classifier without shipping
the marker file, so type checkers ignored the inline annotations
entirely and the classifier was a false claim. The annotations were
always there; only the marker was missing.

Caught during the pre-publish audit.
 2026-08-27 02:51:23 -06:00
Ryan Malloy
aa51ee82e0 Regenerate uv.lock after the distribution rename 
The lockfile still carried the pre-rename name and a stale version
(informix-db 2026.5.5.9). Same class of miss as the __version__ bug:
the rename touched pyproject.toml but not everything that records the
distribution name.
 2026-08-27 02:49:36 -06:00
Ryan Malloy
36478f02c4 Decode the SQ_PROTOCOLS capability negotiation (2026.08.27) 
Closes the last open item from the Informix 12 field report. The driver
hardcodes several wire-framing choices that SQLI actually negotiates.
Those choices are correct on every server we've measured, but "correct
as far as we know" and "checked" are different things, and a framing
mismatch corrupts rows silently.

We were already sending SQ_PROTOCOLS with the same 8-byte client offer
IBM's JDBC driver uses, and discarding the reply. Now we decode it.

New informix_db.ServerCapabilities, reachable from any connection:

    conn.server_capabilities.four_byte_offset
    conn.server_capabilities.varchar_var_len
    conn.server_capabilities.violated_assumptions()   # [] when we agree
    conn.server_version

violated_assumptions() names each place we emit or parse a fixed wire
shape that is actually capability-gated. Non-empty at connect time logs
a warning naming the bit, so an untested server produces a diagnosable
complaint instead of quiet corruption.

Nothing branches on these bits yet — this is observation and validation
only. The hardcoded framing is correct on all three supported servers,
and rewriting working parse paths to be conditional without a server
that needs it trades certainty for risk.

The measurement, and why 12/14/15 are interchangeable:

  15.0.1.0.3       bdbe9ffe7fb7ffef ff
  14.10.FC7W1      bdbe9ffe7fb7ffef f8
  12.10.FC12W1DE   bdbe9ffe7fb7ffef f0
                   ^^^^^^^^^^^^^^^^ identical

The first 64 bits are byte-identical. Those releases don't merely behave
alike, they negotiate exactly the same capability set.

Two details worth recording. The reply is NINE bytes; JDBC's
enhancedProtocolMechanism switches on case 0..7 and drops the ninth, so
its BitSet(64) never sees it — yet that dropped byte is the only part
that differs between releases. And Cap_1 in the login response is not a
server version, it's the client's declared protocol level echoed back,
which is why JDBC tests == 316 rather than >=. The version string there
is the internal one: 12.10 reports 9.56, 14.10 reports 9.59. At the
protocol level both really are 9.x servers.

This also retires isUSVER as a red herring: it's one of six bits JDBC
pre-sets for any non-zero Cap_1, and Java's BitSet.set never clears, so
it is true on every modern server regardless of the mask.

Separately, fixed __version__. The distribution was renamed informix-db
-> informix-driver on 2026-05-08 but __init__ kept looking up the old
name. importlib.metadata needs the distribution name, not the module
name, and the miss fails silently — so every install of the renamed
package has reported "0.0.0+local". It escaped notice because a stale
informix-db distribution lingered in the dev venv and answered the
query. tests/test_package_metadata.py now pins the name against
[project].name; verified it catches the bug by reintroducing it.

247/247 integration on 15, 14.10, and 12.10. 120 unit tests.
 2026-08-27 02:18:08 -06:00
Ryan Malloy
783b2bec97 Verify Informix 14.10 and 12.10; make the version matrix reproducible 
Closes out the field report that prompted 2026.05.08.2. Full integration
suite, same commit, all green:

  Informix 15.0.1.0.3DE     241 / 241
  Informix 14.10.FC7W1DE    241 / 241
  Informix 12.10.FC12W1DE   241 / 241

Including smart-LOB. The 28-type round-trip produces byte-identical wire
output on all three — same type codes, same encoded lengths, same values.
There is no 12-vs-14-vs-15 wire difference for anything we support.

Testing three versions by hand was tedious and undocumented, which is
part of why it never happened. Now:

  make ifx-legacy-up      12.10 on 9089, 14.10 on 9090, alongside 15 on 9088
  make ifx-legacy-setup   blobspace1 + sbspace1 in both
  make test-matrix        the suite against all three

tests/setup-spaces.sh absorbs the per-image differences that made this
annoying: 12.10/14.10 use a flat INFORMIXDIR while 15 nests a versioned
subdirectory; ONCONFIG is named differently across images; `bash -lc`
wipes INFORMIXDIR and makes every utility fail with a misleading
"Unable to read $INFORMIXDIR (/usr/informix)"; and 12.10 DE ships no
ontape at all (the level-0 archive turns out to be unnecessary anyway).
Without blobspace1/sbspace1, ~21 tests fail with errors that look like
driver bugs and aren't.

The documented workflow was validated from scratch — containers removed,
recreated via compose, spaces created via the script, matrix run — rather
than written up afterward from whichever commands happened to work. The
README claim this replaces was wrong precisely because nobody did that.
 2026-08-27 00:36:35 -06:00
Ryan Malloy
87b5b7c354 Fix three tuple-framing bugs: BOOLEAN, NCHAR, INT8/SERIAL8 (2026.05.08.2) 
Data corruption affecting every Informix version including 15. Two of the
three silently corrupt columns AFTER the offending one, so the damage
shows up far from its cause.

BOOLEAN: described as UDTFIXED(41) with encoded_length=1, but carries the
full UDT envelope on the wire — [1-byte indicator][4-byte length][data],
six bytes for a one-byte value. We read one byte and left five behind.
Captured payload for (INT, BOOLEAN 't', INT, VARCHAR 'tail'):
  00 01 b2 07 | 00 00 00 00 01 74 | 00 03 64 0e | 04 74 61 69 6c
Before: (111111, b'\x00', 1, '\x00\x03d\x0e\x04tail')
After:  (111111, True, 222222, 'tail')

NCHAR: fixed-width and space-padded like CHAR, but we had it grouped with
the byte-length-prefixed types. NCHAR(10) holding 'nch' read 0x6E ('n') as
a 110-byte length. Silent truncation alone, struct.error with any column
following. NVARCHAR really is length-prefixed and is unchanged; there's a
test guarding both sides now.

INT8/SERIAL8: absent from FIXED_WIDTHS, fell through to the unknown-type
path and surfaced as raw bytes. INT8 is not BIGINT — 10 bytes,
sign-magnitude, halves stored high-last:
  bytes 0-1 sign word (0=NULL, 1=pos, 0xFFFF=neg)
  bytes 2-5 LOW 32 bits, bytes 6-9 HIGH 32 bits
+n and -n share magnitude bytes, so a two's-complement read is wrong for
every negative while looking correct for every positive.

Not a version bug. Reported against Informix 12, but the same 28-type
round-trip against 12.10.FC12W1DE and 15.0.1.0.3DE produced byte-identical
wire output. It read as version-specific only because INT8/SERIAL8 dominate
12-era schemas while our fixtures use BIGINT.

251 tests missed all three because no fixture used these types. Added
tests/test_type_framing.py (20 integration) and tests/test_int8_unit.py
(14 unit, wire vectors from both servers). Each type is tested twice —
alone, and with trailing columns, since only the latter catches desync.

Verified: 271/271 integration on 15; 241/241 non-smart-LOB on 12.10 (the
LOB tests need an sbspace that image lacks); 20/20 framing tests on both.

Also corrected README and _fastpath docstring, which asserted 12.10
compatibility that had never been tested. The claims held up, but they
were guesses when written and cost a user debugging time.
 2026-08-26 23:40:39 -06:00
Ryan Malloy
2d83ed7b45 Wire-dump: lift overflow-x: auto to base rule (tablet-portrait fix) 
Previously the horizontal-scroll fallback only applied at ≤640px; between
641px and 799px (tablet portrait, narrow desktop 2-col), the hero stayed
2-column so the wire-dump column could still be too narrow for the
~564px hex content, and overflow: hidden silently clipped the right side.

Lifting overflow-x: auto to all widths means: (a) any width where the
column is wider than the hex, content displays normally with no scroll;
(b) any width where the column is narrower, content becomes
horizontally-scrollable inside the dump. y stays hidden to keep the
typed-out animation's unrevealed lines clipped below the fold.
 2026-05-08 06:07:43 -06:00
Ryan Malloy
dc7b9bfd94 Mobile: fix hero overflow at ≤640px, shrink wire-dump, hide ASCII column 
Root cause: hero's grid items inherited min-width: auto, so the wire-dump's
white-space: pre hex lines (~564px wide content) forced the hero column to
that width, propagating up to the page and causing 193px of horizontal
overflow at narrow viewports.

Fix:
- min-width: 0 on .ifx-hero, .ifx-hero__copy, .ifx-hero__visual (lets
  grid items shrink below content's intrinsic min-width)
- overflow-x: auto on .ifx-wiredump (contains residual hex overflow
  inside the dump, not on the page)
- Font shrunk 0.78rem → 0.62rem on mobile, ASCII column hidden, padding
  tightened — readable hex without horizontal scroll inside the dump
- Eyebrow flex-wrap: wrap so the No-libcrypt suffix wraps cleanly
- Title floor 2rem → 1.75rem at 7.5vw for narrow screens
- Tightened CTA padding, install command word-break: break-all
 2026-05-08 06:01:47 -06:00
Ryan Malloy
ab5be68738 Hero CTA: 'GitHub' button text → 'Source' (link already pointed at Gitea) 2026-05-08 05:44:11 -06:00
Ryan Malloy
21c47385ae Prose rebrand: informix-db → informix-driver across docs site 
Sweep all backticked + bold + table-cell + heading mentions of the
project's brand to match the PyPI distribution name and the docs domain.
Path references (`cd informix-db`, `git.supported.systems/.../informix-db`)
stay — those reference the actual Gitea repo directory which we did NOT
rename. Same with `import informix_db` (Python module name, separate
from distribution brand).

Also flip GitHub references to Gitea throughout the docs site:
- `github.com/rsp2k/informix-db/blob/main/X` → Gitea `/src/branch/main/X`
- `github.com/rsp2k/informix-db/tree/main/X` → Gitea same path
- `github.com/rsp2k/informix-db` (plain) → Gitea
- Hero "GitHub" CTA button → Gitea source URL
- Social icon: `github` → `seti:git` (generic git icon, not octocat)

Net result: zero stale GitHub references, brand consistency matches what
users `pip install`.
 2026-05-08 05:43:07 -06:00
Ryan Malloy
b67e6d008b Homepage frontmatter title: informix-db → informix-driver (browser tab consistency) 2026-05-08 05:01:22 -06:00
Ryan Malloy
073c7ed513 Site header: informix-db → informix-driver 
Match what users see when they `pip install informix-driver`. Also fix
two broken social links: PyPI was pointing at the now-404 informix-db
project, GitHub link was pointing at github.com/rsp2k/informix-db
(repo doesn't exist there — source lives on Gitea). Rewired to:

- Source (Gitea): git.supported.systems/warehack.ing/informix-db
- PyPI: pypi.org/project/informix-driver
- editLink: Gitea's _edit/branch/main URL pattern

Logo SVG aria-label + title also updated for accessibility consistency.
 2026-05-08 05:00:36 -06:00
Ryan Malloy
9af0a4cec9 Rename docs domain: informix-db.warehack.ing → informix-driver.warehack.ing 
Brand-consistency with PyPI distribution name. Old domain stops serving
(no redirect — clean cutover per the rename decision).

- pyproject.toml urls: Homepage + Documentation → new domain
- README badge row + sdist exclude comment
- docs-site .env.example + astro.config.mjs site URL + DEV_DOMAIN default
- Version bumped to 2026.05.08.1 (PEP 440 post-release; PyPI 2026.5.8
  still resolvable but with stale URL — yank or leave per preference)
 2026-05-08 04:29:08 -06:00
Ryan Malloy
24feabd21b Rename PyPI distribution: informix-db → informix-driver 
PyPI rejected `informix-db` as too similar to legacy `informixdb` (v2.5,
2008). Renamed distribution to `informix-driver`. Import name stays
`informix_db` — same separation Pillow uses with `import PIL`.

Updated:
- pyproject.toml [project].name
- README pip-install command + brief explanation note
- docs-site quickstart, vs-ifxpy, wtf, Hero install commands

First PyPI release: pypi.org/project/informix-driver/2026.5.8/
 2026-05-08 04:17:57 -06:00
Ryan Malloy
1582a5295d Prepare 2026.05.08 for first PyPI publish 
- Bump version 2026.05.05.12 → 2026.05.08 (CalVer, publish date)
- Expand sdist excludes to **/glob patterns: docs/**, docs-site/**,
  tests/reference/**, tests/benchmarks/.results/** — the trailing-slash
  form was silently passing through subtree contents
- Sanitize tests/benchmarks/baseline.json hostname → PLACEHOLDER
- Rewrite README relative docs/, tests/, Makefile links to absolute
  Gitea URLs (git.supported.systems/warehack.ing/informix-db)
- pyproject urls: Homepage + Documentation → informix-db.warehack.ing,
  Source/Issues/Changelog → Gitea (warehack.ing org is now public)
 2026-05-08 04:13:09 -06:00
Ryan Malloy
86070e4688 Add docs-site: Astro + Starlight at informix-db.warehack.ing 
22 pages across Diataxis quadrants (start / how-to / reference / explain).
Custom amber-on-charcoal theme, wire-dump hero animation, Supported
Systems footer badge. caddy-docker-proxy deployment with prod + dev
profiles, Makefile with prod/dev/down/logs/local targets.
 2026-05-08 03:23:22 -06:00
Ryan Malloy
ad55391bf1 Phase 39: Connection-scoped read-ahead buffer (2026.05.05.12) 
Closes the bulk-fetch gap to within ~7-15% of IfxPy. Lever was the
buffer/I/O machinery, not the codec — Phase 37/38 had already brought
the codec close to IfxPy's C path; the remaining gap was ~450k
read_exact calls per 100k-row fetch, each doing its own recv-loop
and bytes.join.

Architecture: IfxSocket owns a connection-scoped bytearray + integer
offset cursor; BufferedSocketReader is a thin parser-view that delegates
buffer-fill to the socket. One recv() per ~64 KB instead of per field.
This is how asyncpg (buffer.pyx) and psycopg3 (pq.PGconn) structure
their read paths.

The buffer MUST be socket-scoped, not reader-scoped: the pipelined-
executemany path (Phase 33) streams N responses back-to-back across
multiple cursor reads, and a per-reader buffer would throw away
pre-fetched bytes when one reader is destroyed. (The first iteration
of this phase tried per-reader and hung on test_executemany_1000_rows.)

A/B vs Phase 38, same harness, warmed cache:
  select_scaling_1000     2.90 ->  1.72 ms  (-41%)
  select_scaling_10000   24.32 -> 16.08 ms  (-34%)
  select_scaling_100000 250.36 -> 168.98 ms (-32%)

Head-to-head vs IfxPy 2.0.7:
  select_scaling_1000     1.05x   (basically tied)
  select_scaling_10000    1.07x
  select_scaling_100000   1.15x

IQR collapsed 9x at 100k (3.6 ms -> 0.4 ms) — fewer recvs means fewer
scheduler/jitter pulses showing up in the measurement.

Default ON. Set IFX_BUFFERED_READER=0 to fall back to the legacy reader
(still tested in CI as the escape hatch). Both paths green: 251/251
integration tests pass on each.
 2026-05-08 01:40:54 -06:00
Ryan Malloy
a5e6cf1ae3 Phase 38: exec()-based row decoder codegen (2026.05.05.11) 
Generates a specialized row-decoder function per result-set shape via
exec(compile(src, ...)) and inlines the common fixed-width decode bodies
directly into the generated source — closing more of the C-vs-Python
codec gap on bulk fetch.

For SMALLINT/INT/SERIAL/BIGINT/BIGSERIAL/FLOAT/SMFLOAT/DATE the decode
body is inlined ("v0 = _UNPACK_INT(raw)[0]; if v0 == sentinel: v0 = None")
rather than called, eliminating one Python function call per such column
per row. BOOL deliberately left to its canonical decoder (Informix BOOL
is 't'/'T'/1, not bool(byte)).

Real A/B vs Phase 37 (median, integration container):
  select_scaling[100000]    257.66 -> 227.67 ms  (-12%)
  wide_row_select[20]         4.27 ->   3.63 ms  (-15%)
  select_scaling[10000]      25.13 ->  22.58 ms  (-10%)
  wide_row_select[100]       15.17 ->  13.59 ms  (-10%)

Win scales with row count and column count — exactly the codegen
profile expected from per-column inlining.

Generated source is printable via IFX_DEBUG_CODEGEN=1.

Three-tier composition: codegen -> reader-list -> legacy chain;
parse_tuple_payload prefers the codegen'd decoder, falls back to the
Phase 37 readers list, falls back to the legacy branch chain.

All 251 integration tests pass.
 2026-05-05 14:19:26 -06:00
Ryan Malloy
7f729b3a38 Phase 37: Pre-baked per-column reader strategy (2026.05.05.10) 
Closes some of the C-vs-Python codec gap on bulk fetch by moving
per-column dispatch decisions from row time to parse_describe time.
Same approach psycopg3 uses in its pure-Python mode (loader cache
per column).

What changed:

_resultset.py:
* New compile_column_readers(columns) builds a per-column dispatch
  tuple at parse_describe time. Each tuple is (kind, *args) where
  kind is a small int (FIXED/BYTE_PREFIX/CHAR/LVARCHAR/DECIMAL/
  DATETIME/INTERVAL/LEGACY).
* parse_tuple_payload accepts optional readers= parameter. Fast
  path uses int comparison + tuple unpack instead of the legacy
  frozenset/dict-lookup chain.
* _legacy_dispatch_one_column factored out to handle rare types
  (UDT/composite/UDTVAR) that fall through.

cursors.py:
* Cursor caches self._column_readers after parse_describe,
  computed once via compile_column_readers. Reset on new execute.
* Fetch loop passes readers=self._column_readers.

Performance (median of 10+ rounds):

  select_scaling[1000]:    2.7 ms -> 2.51 ms (-7%)
  select_scaling[10000]:  25.8 ms -> 25.0 ms (-3%)
  select_scaling[100000]: 271 ms  -> 246 ms (-9%)
  wide_row_select[5]:     2.4 ms  -> 2.16 ms (-10%)
  wide_row_select[20]:    5.1 ms  -> 4.14 ms (-19%)
  wide_row_select[50]:    10.1 ms -> 8.21 ms (-19%)
  wide_row_select[100]:   19.4 ms -> 14.6 ms (-25%)

Wide-row workloads benefit most - per-column dispatch savings
accumulate linearly with column count. At 100 cols, 25% speedup.

IfxPy gap shrinks from ~2.4x to ~2.2x on bulk fetch. Real progress
but not closing-the-gap. Next lever is exec()-based codegen
(per-result-set decoder function) - possible Phase 38.

221 integration tests still pass. Benchmark suite acts as regression
test.

Architectural note: chose tuple dispatch (r[0] int compare) over
object-method dispatch (loader.load(data)) for ~20-30 ns/col speed
advantage in the inner loop. Slightly less extensible than psycopg3's
class-based loaders but materially faster in pure Python.
 2026-05-05 13:50:40 -06:00
Ryan Malloy
5825d5c55e Extend scaling benches: 100-column case + 100k memory profile + 1M gating 
Adds three things to test_scaling_perf.py:

1. 100-column wide-row SELECT - codec stress test at extreme widths.
   1k rows x 100 cols = 19.4 ms (~194 us/row, ~1.94 us/column-decode).
   Per-column cost continues to drop with width thanks to loop
   amortization (5 cols: 480 ns/col -> 100 cols: 194 ns/col).

2. 100k-row memory profile - samples RSS pre-execute, post-execute
   (materialization cost), and during iteration. Real numbers:
     pre-execute:  45.8 MB
     post-execute: 71.2 MB  (+25.4 MB = ~259 bytes/row materialization)
     iteration:    0 KB extra (just walks the existing list)

   Documents the in-memory cursor's actual cost: 100k rows = 25 MB,
   1M rows = ~250 MB. Fair regression baseline (tripped at 500 MB).

3. 1M-row scaling gated behind IFX_BENCH_1M=1 env var. Default off
   because the dev container's rootdbs runs out of space. For
   production-sized servers users can opt in. The implementation
   is linear-extrapolation-correct (executemany 100k -> 1M = ~15s,
   SELECT 100k -> 1M = ~3s).

Note on the dev-container size limit: dev image's rootdbs is sized
for typical developer workloads, not stress testing. A 1M-row
INSERT exceeds the available pages and fails with -242 ISAM -113
(out of space). This is correct behavior - the limit is enforced
at the storage layer.

Switched RSS sampling from ru_maxrss (peak, monotonic) to
/proc/self/status VmRSS (current). Earlier runs showed flat
RSS because peak from earlier in the test session masked the
fluctuation.
 2026-05-05 13:10:32 -06:00
Ryan Malloy
270155d2de Phase 36: IfxPy scaling comparison + honest comparison numbers (2026.05.05.9) 
Extends the IfxPy comparison bench script with scaling workloads
(1k/10k/100k rows for both executemany and SELECT). Re-runs the
full comparison with consistent measurement methodology and updates
the README with the actually-correct numbers.

Earlier comparison runs reported informix-db winning all 5
benchmarks. Re-running select_bench_table_all with consistent
measurement gives 3.04 ms, not the 891 us I cited earlier - a
3.4x discrepancy attributable to noisy warmup + small-fixture
artifacts. The "we win everything" framing was wrong.

Corrected comparison reveals two clear stories:

Bulk-insert: pure-Python wins 1.6x at scale.
  executemany(10k):  IfxPy 259ms  -> us 161ms (1.6x faster)
  executemany(100k): IfxPy 2376ms -> us 1487ms (1.6x faster)
Reason: Phase 33's pipelining eliminates per-row RTT. IfxPy's
per-call API can't pipeline.

Large-fetch: IfxPy wins 2.3-2.4x at scale.
  SELECT 1k rows:   IfxPy 1.2ms  / us 2.7ms (IfxPy 2.3x)
  SELECT 10k rows:  IfxPy 11.3ms / us 25.8ms (IfxPy 2.3x)
  SELECT 100k rows: IfxPy 112ms  / us 271ms (IfxPy 2.4x)
Reason: C-level fetch_tuple at ~1.1us/row beats Python
parse_tuple_payload at ~2.7us/row. Real C-vs-Python codec gap
showing up at scale.

For everyday workloads (single SELECT in a request, INSERT a
handful of rows), drivers are within 5-25%. For workloads where
the gap widens, direction depends on what you're doing - bulk-
write favors us, bulk-read favors IfxPy.

README's "Compared to IfxPy" section rewritten with the corrected
numbers and an honest "when to prefer which" subsection.
tests/benchmarks/compare/README.md mirror updated.

Net narrative: a "faster at bulk-write, slower at bulk-read,
comparable elsewhere" comparison story is more honest and more
durable than a "we win everything" claim that would have collapsed
the first time a user ran their own benchmark.

Side note (lint): one ambiguous unicode `×` in cursors.py replaced
with `x`.

Phase 37 ticket: parse_tuple_payload is the bottleneck at scale.
Closing the 1.6 us/row gap to IfxPy would make us competitive on
bulk-fetch too. Possible approaches: Cython codec, deeper inlining,
per-column dispatch pre-bake.
 2026-05-05 12:44:52 -06:00
Ryan Malloy
8eb19f7534 Phase 34: Scaling benchmarks (1k/10k/100k rows; 5/20/50 cols) (2026.05.05.8) 
Adds tests/benchmarks/test_scaling_perf.py with parametrized
benchmarks across row-count, column-width, and type-mix axes.
Caught the NFETCH-loop bug (Phase 35) immediately on first run.

Headline numbers:

Bulk insert (executemany in transaction):
  1k rows:   23 ms (23 us/row)
  10k rows:  161 ms (16 us/row)
  100k rows: 1487 ms (15 us/row, ~67k rows/sec sustained)

SELECT (linear scaling, near-constant per-row):
  1k rows:   2.7 ms (2.7 us/row)
  10k rows:  25.8 ms (2.6 us/row)
  100k rows: 271 ms (2.7 us/row)

Wide-row SELECT (1k rows x N cols):
  5 cols:  2.4 ms
  20 cols: 5.1 ms
  50 cols: 10.1 ms

Type-mix SELECT (INT + VARCHAR + DECIMAL + DATE + FLOAT + SMALLINT):
  1000 rows: 4.7 ms (4.7 us/row, ~1.7x baseline)

Per-row codec cost is essentially constant from 1k to 100k rows
(2.7 us/row), proving parse_tuple_payload optimizations (Phases
23-25) hold at 100x scale with no GC-pause amplification or
memory-pressure degradation.

Per-row insert cost actually DECREASES with scale (23us at 1k to
15us at 100k) - Phase 33's pipelining amortizes prepare/release
overhead better at larger N.

10 new parametrized benchmarks. Total: 77 unit + 249 integration +
43 benchmark = 369 tests.
 2026-05-05 12:38:07 -06:00
Ryan Malloy
1282893412 Phase 35: CRITICAL fix - NFETCH loop for large result sets (2026.05.05.7) 
DATA-LOSS BUG: cursor.fetchall() on result sets larger than ~200 rows
was silently truncating to the first ~200 rows. The exact cap depended
on row width and the server's per-NFETCH buffer (4096 bytes default).

The bug:

_execute_select sent NFETCH twice and stopped:
  self._conn._send_pdu(self._build_curname_nfetch_pdu(cursor_name))
  self._read_fetch_response()
  self._conn._send_pdu(self._build_nfetch_pdu())  # comment: "DONE only"
  self._read_fetch_response()
  # then CLOSE+RELEASE — discarding remaining queued rows

The "second fetch returns DONE only" comment was wrong. For any
result set larger than the server's per-NFETCH batch, the second
fetch returns more tuples AND there are still tuples queued
server-side. The cursor closed and dropped them.

Latent for 30 phases because every existing test used either a small
result set (FIRST 10) or relied on row counts that fit naturally in
1-2 batches. Discovered by Phase 34's scaling benchmark when
SELECT FIRST 100000 from a 100k-row table returned 200 rows.

The fix: loop NFETCH until a response yields zero new tuples.

  self._conn._send_pdu(self._build_curname_nfetch_pdu(cursor_name))
  rows_before = len(self._rows)
  self._read_fetch_response()
  rows_received = len(self._rows) - rows_before
  while rows_received > 0:
      self._conn._send_pdu(self._build_nfetch_pdu())
      rows_before = len(self._rows)
      self._read_fetch_response()
      rows_received = len(self._rows) - rows_before

249 integration tests pass. The scaling benchmark suite (Phase 34,
shipping next) is the regression test going forward.

Workaround for users on older versions: use scrollable cursors
(cursor(scrollable=True)) which use the SQ_SFETCH protocol path
and don't have this bug.

If you've been using this driver for queries returning large result
sets, your queries may have been truncating silently. Re-run them
against 2026.05.05.7+ to verify your data.
 2026-05-05 12:37:22 -06:00
Ryan Malloy
362ecb3d63 Phase 33: Pipelined executemany - 2.85x faster bulk insert (2026.05.05.6) 
The serial-loop executemany paid one wire round-trip per row (~30us/
row on loopback). It was the one benchmark where IfxPy beat us in
the comparison work - 10% slower at executemany(1000) in txn.

Phase 33 pipelines the BIND+EXECUTE PDUs: build all N PDUs, send
them back-to-back, then drain all N responses. Eliminates per-row
RTT entirely.

Performance impact:
* executemany(1000) in txn:   31.3 ms -> 11.0 ms (2.85x faster)
* executemany(100) autocommit: 173 ms -> 154 ms (11% faster)
* executemany(1000) autocommit: 1740 ms -> 1590 ms (9% faster)

(Autocommit gets smaller wins because server-side log flushes
dominate - Phase 21.1's "autocommit cliff".)

IfxPy comparison flipped: us 10% slower -> us 2.05x faster on bulk
inserts. We now win all 5 head-to-head benchmarks against the C-bound
driver.

Margaret Hamilton review surfaced one CRITICAL concern (C1) - the
pipeline assumes Informix sends N responses for N pipelined PDUs
even when one fails. If the server cut the stream short, the drain
loop would deadlock on the next read.

Verified by 3 new integration tests in tests/test_executemany_pipeline.py:
* test_pipelined_executemany_mid_batch_constraint_violation (row 500/1000)
* test_pipelined_executemany_first_row_fails (row 0/100)
* test_pipelined_executemany_last_row_fails (row 99/100)

All confirm Informix sends N responses; wire stays aligned; connection
is usable after.

Plus 4 lower-priority fixes Hamilton recommended:
* H1: documented _raise_sq_err self-drains-SQ_EOT invariant + tripwire
* H2: docstring warning about O(N) lock duration; chunk for huge batches
* M1: prepend row-index to exception message rather than reformat
* M2: documented sendall-no-timeout caveat on hostile networks

77 unit + 239 integration + 33 benchmark = 349 tests; ruff clean.

Note: Phase 32 (Tier 1+2 benchmarks) was tagged without bumping
pyproject.toml's version string. .5 was git-tag-only; .6 is the next
published version increment.
 2026-05-05 12:26:15 -06:00
Ryan Malloy
01757415a5 Phase 32: Benchmark improvements (Tier 1 + Tier 2) 
Tier 1 — make existing benchmarks reliable:
* Bumped slow-bench rounds: cold_connect_disconnect 5->15, executemany
  series 3->10. Single-round outliers no longer dominate.
* Switched bench reporting to median + IQR. Mean was being moved by
  individual GC pauses / scheduler hiccups (IfxPy executemany IQR
  was 8.2 ms on a 28 ms median - 29% spread - mean was unreliable).
* Updated ifxpy_bench.py to also report median + IQR alongside mean
  for cross-comparable numbers.
* Makefile bench targets now show median, iqr, mean, stddev, ops, rounds.

The robust statistics flipped the comparison story:

  Old (mean, 3 rounds):   us 9% faster  / IfxPy 30% faster on 2 of 5
  New (median, 10+ rds):  us faster on 4 of 5 benchmarks

| Benchmark | IfxPy | informix-db | Δ |
|---|---|---|---|
| select_one_row             | 170us | 119us | us 30% faster |
| select_systables_first_10  | 186us | 142us | us 24% faster |
| select_bench_table_all 1k  | 980us | 832us | us 15% faster |
| executemany 1k in txn      | 28.3ms | 31.3ms | us 10% slower |
| cold_connect_disconnect    | 12.0ms | 10.7ms | us 11% faster |

Tier 2 — add benchmarks for claims we make but don't verify:

tests/benchmarks/test_observability_perf.py:
* test_streaming_fetch_memory_profile — RSS sampling during a
  cursor iteration. Documents memory growth shape; regression
  wall at 100 MB / 1k rows. Currently flat (in-memory cursor
  doesn't grow detectably for 278 rows).
* test_select_1_latency_percentiles — 1000-query distribution
  with p50/p90/p95/p99/max. Result: p99/p50 = 1.42x (tight tail).
  p50=108us, p99=153us.
* test_concurrent_pool_throughput[2,4,8] — N worker threads
  through pool, measures aggregate QPS + per-thread fairness.
  Plateaus at ~6K QPS (server-bound); per-thread latency scales
  ~linearly with N (server serialization expected).

README.md (project root): updated Compared-to-IfxPy table with
the median-based numbers + IQR awareness note.
tests/benchmarks/compare/README.md: added "Statistical robustness"
section explaining why median over mean for fair comparison.

236 integration tests pass; ruff clean.
 2026-05-05 12:01:11 -06:00
Ryan Malloy
a9e1f17bae Phase 31: Head-to-head benchmark vs IfxPy (the C-bound PyPI driver) 
Adds a paired benchmark of informix-db (pure Python) against IfxPy
3.0.5 (IBM's C-bound driver via OneDB ODBC) on identical workloads
against the same Informix dev container.

Headline result: pure Python is competitive — and faster on 2/5
benchmarks where wire round-trip dominates over codec/marshaling.

| Benchmark | IfxPy | informix-db | Result |
|---|---:|---:|---:|
| select_one_row (single-row latency) | 128 us | 116 us | us 9% faster |
| select_systables_first_10 | 126 us | 184 us | IfxPy 32% faster |
| select_bench_table_all (1k rows) | 969 us | 855 us | us 12% faster |
| executemany(1000) in txn | 21.5 ms | 30.8 ms | IfxPy 30% slower |
| cold_connect_disconnect | 11.0 ms | 10.9 ms | comparable |

Why the surprising wins: IfxPy's path is Python -> OneDB ODBC ->
libifdmr -> wire. Ours is Python -> wire. When wire round-trip
dominates (single-row, bulk fetch), the missing abstraction layer
makes us faster. When per-row marshaling dominates (executemany),
IfxPy's C-level execute(stmt, tuple) beats Python BIND-PDU build.

Files added under tests/benchmarks/compare/:
* Dockerfile.ifxpy — Ubuntu 20.04 base with IfxPy + OneDB drivers
* ifxpy_bench.py — IfxPy benchmark workloads matching test_*_perf.py
* README.md — methodology, results, install gauntlet, reproduction

The IfxPy install gauntlet itself is part of the comparison story:
modern Python 3.11 (not 3.13), setuptools <58, permissive CFLAGS,
manual download of 92MB OneDB ODBC tarball, four LD_LIBRARY_PATH
directories, libcrypt.so.1 (deprecated 2018, missing on Arch /
Fedora 35+ / RHEL 9). Versus our `pip install informix-db`.

README.md (project root): added "Compared to IfxPy" section under
Performance with the headline numbers and a pointer to the full
methodology.

.gitignore: keep Dockerfile/script/README under tests/benchmarks/
compare/, exclude the 92MB OneDB tarball and the local venv.
 2026-05-05 11:41:47 -06:00
Ryan Malloy
eb8d15d204 README + classifier polish for PyPI launch 
PyPI users landing on the README need to know quickly:
- What this is (already strong)
- Whether it's safe to use in production (was missing)
- Performance expectations (was missing)
- Python version requirement (was only in pyproject.toml metadata)

Updates:
* Added "Status" section with the Hamilton audit findings table -
  every critical/high/medium addressed, 0 remaining. Names the
  Hamilton-style review process explicitly as the credibility signal.
* Added Python ≥ 3.10 requirement under the install command.
* Added "Performance" section with single-connection benchmarks and
  the 53x autocommit-cliff gotcha (most important perf pitfall).
* Updated "Standards & guarantees" to mention Phase 27's wire lock
  alongside the PEP 249 Threadsafety=1 declaration - accurate context
  for sophisticated readers.
* Tightened "Development" to PyPI-appropriate brevity (short Makefile
  target list instead of full uv invocations).
* Updated stale phase count (22+ → 30) and test counts (69 → 77 unit,
  163 → 231 integration). Added "300+ tests" rough number in the
  Status section to reduce future staleness churn.
* Fixed typo: "no thread of native machinery" → "no native machinery
  anywhere in the thread of execution".
* Bumped pyproject.toml classifier from "Development Status :: 4 -
  Beta" to "5 - Production/Stable" - earned by the audit work.

No code changes.
 2026-05-05 11:06:49 -06:00
Ryan Malloy
0b13acb13d Phase 30: Final hardening pass (2026.05.05.4) 
Closes the last 3 medium-severity items from Hamilton's system-wide
audit. **0 critical, 0 high, 0 medium remaining.**

What changed:

pool.py:
* Pool acquire() growth path: restructured to remove _lock._is_owned()
  (CPython-private API) usage. Two explicit re-acquires (success path
  + exception path) replace the older try/finally + private check.

connections.py:
* _raise_from_rejection now extracts the server's human-readable
  error string from the rejection payload and surfaces it in the
  OperationalError. Wrong-password vs wrong-database now produce
  distinguishable errors. New helper _extract_server_error_text
  finds the longest printable-ASCII run (8-256 chars). Falls back
  to a hex preview when no string is found.
* _send_exit: broadened catch from (OperationalError, InterfaceError,
  OSError, ProtocolError) to bare Exception. Best-effort by
  definition; the socket FD is freed by close()'s finally clause via
  _socket.IfxSocket.close (idempotent, never-raising). Prevents
  unexpected errors from escaping close() and leaving partial state.

5 new unit tests in test_protocol.py for _extract_server_error_text:
finds-longest-run, picks-longest-of-multiple, too-short-returns-None,
empty-handled, caps-at-256.

77 unit + 231 integration + 28 benchmark = 336 tests; ruff clean.

Hamilton audit punch list final state: every actionable finding
addressed. No CRITICAL, no HIGH, no MEDIUM remaining.

  Pre-Phase-26: 2 critical, 3 high, 5 medium
  Post-Phase-30: 0 critical, 0 high, 0 medium - PRODUCTION READY
 2026-05-05 10:52:39 -06:00
Ryan Malloy
8e8b81fe8d Phase 29: Deferred-cleanup queue (2026.05.05.3) 
Closes the unbounded-leak gap on long-lived pooled connections that
Phase 28's cursor finalizer left as future work. When the finalizer
can't acquire the wire lock (cross-thread GC during another thread's
op), instead of leaking + logging, it enqueues the cleanup PDUs to a
per-connection deferred queue. The next normal operation drains the
queue under the wire lock, completing the cleanup atomically before
the new op.

What changed:

connections.py:
* Connection._pending_cleanup: list[bytes] + Connection._cleanup_lock
  (separate from _wire_lock - tiny critical section for list mutation
  only, allows enqueue without waiting for an in-flight wire op)
* _enqueue_cleanup(pdus): thread-safe append, callable from any
  thread (including finalizers without lock ownership)
* _drain_pending_cleanup(): pop-the-list + send-each-PDU. Caller
  must hold _wire_lock. Force-closes on wire desync (same doctrine
  as _raise_sq_err)
* _send_pdu opportunistically drains the queue before sending. Cost
  is one length-check when queue is empty (the common case)

cursors.py:
* _finalize_cursor enqueues [_CLOSE_PDU, _RELEASE_PDU] instead of
  leaking when the lock is busy. WARNING demoted to DEBUG since
  leak no longer accumulates.

Lock-order discipline: _cleanup_lock is held only for list extend/pop;
_wire_lock is held for the actual wire I/O. Never grab _cleanup_lock
while holding _wire_lock - the drain pops-and-clears under
_cleanup_lock, then iterates under _wire_lock (which caller holds).

Two new regression tests:
* test_enqueue_cleanup_drains_on_next_send_pdu - verifies queue
  mechanism end-to-end
* test_pending_cleanup_thread_safe_enqueue - 8x50 concurrent enqueues,
  no race-loss

72 unit + 231 integration + 28 benchmark = 331 tests; ruff clean.

Hamilton audit punch list status:
  0 critical, 0 high, 3 medium remaining (login errors, _send_exit
  cleanup, pool acquire re-entrance) - all Phase 30 scope.
 2026-05-05 10:47:49 -06:00
Ryan Malloy
fdb9ba32d5 Phase 28: Resource leak hardening (2026.05.05.2) 
Closes Hamilton audit High #4 (bare-except in error drain) and
High #5 (no cursor finalizers), plus 1 medium one-liner.

After Phases 26-28, 0 CRITICAL and 0 HIGH audit findings remain.
Driver is PRODUCTION READY.

What changed:

cursors.py:
* Cursor finalizers via weakref.finalize. Mid-fetch raises (or any
  GC without explicit close()) now release server-side resources
  (CLOSE + RELEASE PDUs). Pre-built static PDU bytes at module load
  so finalizer can run on any thread without allocating or calling
  cursor methods.
* Non-blocking lock acquire prevents cross-thread GC deadlock.
  WARNING log on lock-busy so leak accumulation is visible.
* state=[False] list pattern keeps finalizer closure weak. GIL
  dependency of atomic single-element mutation documented.
* _raise_sq_err near-token parse: (ProtocolError, OSError) only.
* _raise_sq_err drain: force-close connection on same exceptions
  (wire unrecoverable after desync).

connections.py:
* _raise_sq_err drain: same hardening as cursor version. Force-close
  on (ProtocolError, OSError, OperationalError) - the latter from
  _drain_to_eot raising on unknown tags. Documented inline.
* Added contextlib import for force-close suppression.

cursors.py write_blob_column:
* BLOB_PLACEHOLDER validation now requires EXACTLY ONE occurrence.
  Pre-Phase-28, str.replace silently substituted every occurrence -
  corrupting SQL containing the literal string in comments etc.
  Now raises ProgrammingError with workaround pointer.

_resultset.py:
* Investigated end-of-loop bounds check for parse_tuple_payload.
  Reverted: long-standing off-by-one in UDTVAR(lvarchar) trailing-
  pad logic produces benign over-reads (payload is a fully-extracted
  bytes object; over-reads return empty slices through unused
  branches). Real silent-corruption surfaces are length-prefix
  decoders, needing branch-local checks. Documented as deliberate
  non-fix.

Margaret Hamilton review surfaced two blocking conditions:

* Asymmetric failure handling: _raise_sq_err force-closed the
  connection on wire desync, but the cursor finalizer silently
  swallowed identical failures. "Same wire, same failure mode,
  same response" - finalizer now matches _raise_sq_err's discipline.

* Leak visibility: wire-lock-busy log was DEBUG. Promoted to WARNING
  so leak accumulation on pooled connections is visible.

Plus three documentation improvements (GIL dependency, OperationalError
in desync taxonomy, parse_tuple non-fix rationale).

One new regression test:
* test_write_blob_column_rejects_multiple_placeholders

72 unit + 229 integration + 28 benchmark = 329 tests; ruff clean.

Phase 29 ticket (Hamilton recommended): deferred-cleanup queue
drained at next _send_pdu, closes unbounded-leak gap on long-lived
pooled connections. Not blocking Phase 28.

Hamilton audit verdict:
  Pre-26:  2 critical, 3 high, 5 medium
  Post-28: 0 critical, 0 high, 4 medium
 2026-05-05 03:56:24 -06:00
Ryan Malloy
6afdbcabb3 Phase 27: Wire lock + async cancellation eviction (2026.05.05.1) 
Closes Hamilton audit Critical #2 (concurrency / wire lock) and
High #3 (async cancellation evicts cleanly). Phase 26 fixed what
gets returned to the pool; Phase 27 fixes what can interleave on
the wire while it's running.

What changed:

connections.py:
* Added Connection._wire_lock = threading.RLock(). Wrapped commit(),
  rollback(), fast_path_call() under the lock.
* _ensure_transaction documents the lock as a precondition AND
  asserts ownership at runtime (_wire_lock._is_owned()) so a future
  caller adding a third call site fails loudly.
* close() tries to acquire wire lock with 0.5s timeout before
  SQ_EXIT; skips polite exit and force-closes if busy.

cursors.py:
* execute() body extracted into _execute_under_wire_lock() and
  called under the lock.
* executemany() body wrapped inline.
* _sfetch_at() wrapped - covers all scrollable fetch_* methods
  that delegate to it.
* close() locks the CLOSE+RELEASE for scrollable cursors.

pool.py:
* release() acquires conn._wire_lock with 5s timeout before rollback.
  On timeout: log WARNING, evict connection. Constant
  _RELEASE_WIRE_LOCK_TIMEOUT for tunability.

aio.py:
* AsyncConnectionPool.connection() now catches CancelledError /
  TimeoutError separately and routes to broken=True. Combined with
  the wire lock, asyncio.wait_for around aio DB calls is now safe.
* Updated docstring; mirrored in docs/USAGE.md.

Margaret Hamilton review surfaced three actionable conditions, all
addressed before tagging:
* Cancellation test used contextlib.suppress - could pass without
  exercising the cancellation path on a fast runner. Switched to
  pytest.raises so the test fails if timeout doesn't fire.
* _ensure_transaction precondition documented but unchecked at
  runtime. Added assert self._wire_lock._is_owned() guard.
* Connection.close() was unsynchronized. Now tries 0.5s acquire
  before SQ_EXIT.

Two new regression tests in tests/test_pool.py:
* test_concurrent_threads_on_one_connection_dont_interleave_pdus
  (without lock: garbled results / hangs)
* test_async_wait_for_cancellation_evicts_connection
  (asserts pool size shrinks; cancellation actually fires)

72 unit + 228 integration + 28 benchmark = 328 tests; ruff clean.

Hamilton verdict: PRODUCTION READY WITH CAVEATS (was) -> CAVEATS
NARROWED FURTHER (now). 0 critical, 2 high remaining (cursor
finalizers + bare-except in error drain) - both Phase 28 scope.
 2026-05-05 03:40:39 -06:00
Ryan Malloy
5c4a7a57f1 Phase 26: Pool rollback-on-release - CRITICAL data-correctness fix (2026.05.05) 
Fixes the dirty-pool-checkout bug surfaced by Margaret Hamilton's
system-wide audit (Critical #1).

The bug: ConnectionPool.release() returned connections with open
server-side transactions still active. Request A's uncommitted
INSERTs would be inherited by Request B reusing the same connection -
B's commit would land A's writes permanently; B's rollback would
silently lose them. Same shape as psycopg2's pre-2.5 dirty-pool bug.

The fix: pool.release() now rolls back any open transaction before
returning the connection to the idle list. The rollback runs OUTSIDE
the pool lock since it's a wire round-trip - the connection is
already off the idle list and counted in _total, so no other thread
can grab it during the rollback window. If the rollback itself fails
(dead socket, etc.), the connection is evicted rather than recycled.

Async path covered automatically: AsyncConnectionPool.release()
delegates to the sync pool's release via _to_thread.

Margaret Hamilton review pass surfaced two findings, both addressed:
* Silent rollback failure: added a WARNING log via logging.getLogger
  ("informix_db.pool") so evictions are debuggable. First logger in
  the project.
* Async cancellation race: the fix doesn't introduce the
  asyncio.wait_for race (Critical #2, deferred to Phase 27), but it
  adds a code path that can trigger it. Documented loudly in
  pool.release() docstring, aio.py module docstring, and USAGE.md
  async section. Recommendation: use read_timeout on the connection
  instead of asyncio.wait_for until Phase 27 lands.

Two new regression tests in tests/test_pool.py:
* test_uncommitted_writes_invisible_to_next_acquirer (the bug)
* test_committed_writes_survive_pool_checkout (no over-correction)

Verified the regression test catches the bug: stashed the fix, ran
the test - it fails with "B sees 1 rows - leaked across pool
checkout boundary" - confirming it tests the real failure mode.

Total tests: 72 unit + 226 integration + 28 benchmark = 326.

Deferred to Phase 27 per Hamilton audit:
* Critical #2 (concurrency / per-connection wire lock)
* High #3 (async cancellation routes to broken=True)
* High #4 (bare except in _raise_sq_err drain)
* High #5 (no cursor finalizers - server-side resource leaks)
 2026-05-05 03:22:18 -06:00
Ryan Malloy
e9aed6ce59 Phase 25: Branch reorder + invariant tripwires (2026.05.04.10) 
Third-pass optimization on parse_tuple_payload's hot loop. Previous
phases removed redundant work; this one removes correct-but-wasteful
work: the if/elif chain checked branches in implementation order, not
frequency order. Fixed-width types (INT, FLOAT, DATE, BIGINT - the most
common columns in real queries) sat at the bottom, paying ~7 frozenset
misses per column.

Changes (src/informix_db/_resultset.py):
* Added _FIXED_WIDTH_TYPES = frozenset(FIXED_WIDTHS.keys()) at module
  load.
* New fast-path branch at the TOP of parse_tuple_payload's loop body
  that handles every _FIXED_WIDTH_TYPES column inline: one frozenset
  check, one dict lookup, one decode, continue. Skips every other
  branch.
* Cleaned up the bottom fall-through; it now genuinely only catches
  unknown types.

Performance vs Phase 24 baseline:
* parse_tuple_5cols_iso8859: 1659 ns -> 1400 ns (-16%)
* parse_tuple_5cols_utf8:    1649 ns -> 1341 ns (-19%)

Cumulative vs Phase 21 baseline (before any optimization):
* parse_tuple_5cols: 2796 ns -> 1400 ns (-50%) - HALF the time
* decode_int:        230 ns  -> 139 ns  (-40%)

Margaret Hamilton review surfaced one HIGH finding addressed before
tagging:
* H: The fast-path optimization assumes every FIXED_WIDTHS key is
  decodable WITHOUT qualifier inspection (encoded_length etc.). True
  today, but a future contributor adding a fixed-width type that
  needs qualifier bits (like DATETIME does) would silently get wrong
  decode behavior - Lauren-Bug class failure.

  Fix: added INVARIANT comment to FIXED_WIDTHS in converters.py AND
  added tests/test_resultset_invariants.py with three CI tripwire
  tests:
  - _FIXED_WIDTH_TYPES is disjoint from every other dispatch branch
  - Every FIXED_WIDTHS key has a DECODERS entry
  - DECODERS keys stay < 0x100 (Phase 24 collision-free guarantee)

  The tests carry instructions: if one fires, don't update the test
  to match - either restore the property or refactor the optimization.
  Comments rot when nobody reads them; tests fail loudly.

baseline.json refreshed; 72 unit + 224 integration + 28 bench = 324
tests; ruff clean.
 2026-05-04 23:34:05 -06:00
Ryan Malloy
dfa60ea501 Phase 24: Decoder dispatch split + struct precompilation (2026.05.04.9) 
Second pass of hot-path optimization on parse_tuple_payload. Two changes
to converters.py:

1. Split decode() into public + internal. Added _decode_base(base_tc,
   raw, encoding) that takes an already-base-typed code and skips the
   redundant base_type() call. Public decode() is now a one-line
   wrapper. parse_tuple_payload's 4 call sites swapped to use
   _decode_base directly. _fastpath.py's external decode() caller is
   unaffected.

2. Pre-compiled struct.Struct unpackers. The fixed-width integer/float
   decoders (_decode_smallint, _decode_int, _decode_bigint,
   _decode_smfloat, _decode_float, _decode_date) switched from per-call
   struct.unpack(fmt, raw) to module-level bound methods like
   _UNPACK_INT = struct.Struct("!i").unpack. Format-string parsed once
   at module load. Measured 37% faster than per-call struct.unpack on
   CPython 3.13 micro.

Performance vs Phase 23 baseline:
* decode_int: 173 ns -> 139 ns (-20%)
* decode_bigint: 188 ns -> 150 ns (-20%)
* parse_tuple_5cols: 2047 ns -> 1592 ns (-22%)
* 1k-row SELECT: 1255 us -> 989 us (-21%)

Cumulative vs original Phase 21 baseline:
* decode_int: 230 ns -> 139 ns (-40%)
* parse_tuple_5cols: 2796 ns -> 1592 ns (-43%)
* 1k-row SELECT: 1477 us -> 989 us (-33%)

Real-world fetch ceiling: 358K rows/sec -> ~620K rows/sec.

Margaret Hamilton review surfaced one HIGH-severity finding addressed
before tagging:
* H: The no-collision guarantee that makes _decode_base safe is
  structural but undocumented (all DECODERS keys are ≤ 0xFF, all flag
  bits are ≥ 0x100, so flagged inputs cannot coincidentally match).
  Added load-bearing INVARIANT comment at DECODERS dict explaining
  the constraint and what to do if violated. Cross-referenced from
  _decode_base's docstring for bidirectional traceability.

baseline.json refreshed; all 224 integration tests pass; ruff clean.
 2026-05-04 19:31:21 -06:00
Ryan Malloy
f3e589c5bf Phase 23: Hot-path optimization for parse_tuple_payload (2026.05.04.8) 
Per-row decode is hit on every row of every SELECT. The original code
had three forms of waste in the inner loop:

1. Redundant base_type() call. ColumnInfo.type_code is already
   base-typed by parse_describe at construction; calling base_type()
   again per column per row was pure waste. Single largest savings.
2. IntFlag->int conversions inline (~10x per iteration). Lifted to
   module-level _TC_X constants.
3. Lazy imports inside the loop body (_decode_datetime, _decode_interval,
   BlobLocator, ClobLocator, RowValue, CollectionValue). Moved to top.

Plus three precomputed frozensets (_LENGTH_PREFIXED_SHORT_TYPES,
_COMPOSITE_UDT_TYPES, _NUMERIC_TYPES) replace inline tuple-membership
checks. _COLLECTION_KIND_MAP is now MappingProxyType (actually frozen).

Performance:
* parse_tuple_5cols: 2796 ns -> 2030 ns (-27%)
* select_bench_table_all (1k rows): 1477 us -> 1198 us (-19%)
* Codec micro-bench, cold connect, executemany: unchanged

Real-world fetch ceiling on a single connection: 350K rows/sec ->
490K rows/sec.

Margaret Hamilton review surfaced four cleanup items, all addressed
before tagging:
* H1: cursor._dereference_blob_columns had the same redundant
  base_type() call - stripped for consistency.
* M1: documented the load-bearing invariant at parse_describe (the
  single producer site) so future contributors have a grep target.
* M2: _COLLECTION_KIND_MAP wrapped in MappingProxyType.
* L1: stale line-number comment fixed to point at the INVARIANT
  comment instead.

baseline.json refreshed; all 224 integration tests pass; ruff clean.
 2026-05-04 17:52:20 -06:00
Ryan Malloy
0e0dfcba26 Phase 22: User-facing documentation refresh (2026.05.04.7) 
The docs/USAGE.md predated Phases 17-21, so anyone landing on PyPI was
missing scrollable cursors, locale/Unicode, the autocommit cliff
finding, and the type-mapping reference.

Added sections to docs/USAGE.md:
* Locale and Unicode - client_locale, Connection.encoding, CLIENT_LOCALE
  vs DB_LOCALE, when characters can't fit the codec
* Type mapping reference - full SQL <-> Python type table, NULL
  sentinels subsection, IntervalYM
* Performance tips - 53x autocommit-cliff fix, 100x executemany win,
  72x pool win, with the actual benchmark numbers from Phase 21.1
* Scrollable cursors - fetch_* API, in-memory vs server-side trade-off,
  edge cases (past-end semantics, negative indexing, rownumber)
* Timeouts and keepalive subsection - production starting points
* Environment dictionary subsection - env={} parameter
* Known limitations - explicit table of what doesn't work (named
  params, complex UDT bind, GSSAPI, XA) with workarounds; "things
  that might surprise you" notes

README.md - added Documentation section linking to docs/USAGE.md
and tests/benchmarks/README.md.

Doc corrections caught during review:
* cursor.rownumber is 0-indexed (impl has always been correct; only
  the original docstring wording was loose)
* fetch_* methods work on BOTH scrollable=True and default cursors;
  the in-memory path supports them too

USAGE.md grew from 345 lines to 633.
 2026-05-04 17:33:37 -06:00
Ryan Malloy
495128c679 Phase 21.1: executemany perf - it was the autocommit cliff (2026.05.04.6) 
Investigation of the Phase 21 baseline finding that executemany(N) cost
scaled linearly per-row (1.74 ms x N) regardless of batch size.

Root cause: every autocommit=True INSERT forces a server-side
transaction-log flush. Not a wire-protocol bug.

Numbers:
* executemany(1000) autocommit=True: 1.72 s (1.72 ms/row)
* executemany(1000) in single txn:    32 ms (32 us/row)

53x speedup from changing the transaction boundary, not the driver.
Pure protocol overhead is ~32 us/row -> ~31K rows/sec sustained
throughput on a single connection. Comparable to pg8000.

Added test_executemany_1000_rows_in_txn benchmark to make this
visible. Updated README headline numbers and added a "Performance
gotchas" section explaining when autocommit=False matters.

Decision: don't pipeline. The remaining 32 us is already excellent;
the autocommit gotcha is the real user-facing footgun. Docs > code.
If someone reports needing >31K rows/sec single-connection, that
becomes Phase 22.
 2026-05-04 17:26:16 -06:00
Ryan Malloy
90ce035a00 Phase 21: Performance benchmarks (2026.05.04.5) 
Adds tests/benchmarks/ with pytest-benchmark coverage of the hot codec
paths and end-to-end SELECT/INSERT/pool/async round-trips. Establishes
a committed baseline.json so PRs can be regression-checked at review
via --benchmark-compare.

* test_codec_perf.py (16): decode/encode_param/parse_tuple_payload
  micro-benchmarks - run without container, suitable for pre-merge CI.
* test_select_perf.py (4): SELECT round-trips - 1-row latency floor,
  10-row, 1k-row full fetch, parameterized.
* test_insert_perf.py (3): single-row INSERT, executemany 100 / 1000.
* test_pool_perf.py (3): cold connect, pool acquire/release, pool
  acquire + query + release.
* test_async_perf.py (2): async round-trip overhead, 10x concurrent.
* baseline.json: committed snapshot, 28 measurements.
* benchmark pytest marker, gated off by default.
* Makefile: bench / bench-codec / bench-save targets;
  test-integration excludes benchmarks for speed.

Headline numbers (dev container loopback):
* decode(int): 181 ns
* parse_tuple 5 cols: 2.87 µs/row
* SELECT 1 round-trip: 177 µs
* Pool acquire+query+release: 295 µs
* Cold connect: 11.2 ms (72x slower than pool)

UTF-8 decode carries no measurable cost vs iso-8859-1 - confirms
Phase 20 didn't regress anything.

Total: 69 unit + 211 integration + 28 benchmark = 308 tests.
 2026-05-04 17:21:12 -06:00
Ryan Malloy
bea1a1cd0c Phase 20: UTF-8/multibyte locale support (2026.05.04.4) 
Thread CLIENT_LOCALE through to user-data string codecs. Driver previously
hardcoded iso-8859-1 for all string conversions, which broke any locale
outside Western European code points.

* Connection.encoding property derived from client_locale via
  _python_encoding_from_locale (en_US.utf8 -> utf-8, en_US.8859-1 ->
  iso-8859-1, etc.)
* encode_param / decode / parse_tuple_payload accept an encoding
  parameter; cursor and fast-path call sites forward conn.encoding
* Smart-LOB CLOB encode/decode and TEXT decode honor connection encoding
* DataError raised for non-representable chars; cursor releases the
  prepared statement before propagating so connection state stays clean

Boundary discipline: protocol-level strings (cursor names, function
signatures, SQ_FILE fnames, error near-tokens, SQL text) stay
iso-8859-1 (always ASCII, never user-controlled).

9 new integration tests in tests/test_unicode.py covering ASCII
round-trip, Latin-1 high-bit, full byte range, locale-mapping,
encoding property, UTF-8 negotiation, multibyte (skipped without
IFX_UTF8_DATABASE), DataError on non-representable, CLOB round-trip.

Total: 69 unit + 212 integration = 281 tests.
 2026-05-04 17:13:19 -06:00
Ryan Malloy
9703279bc8 Phase 19: resilience tests via fault injection (v2026.05.04.3) 
Fills the highest-priority gap from the test-adequacy audit:
connection-failure recovery. 12 new integration tests using a
thread-based TCP proxy (ControlledProxy) that can be kill()'d at
any moment to simulate network drops or server crashes via TCP RST
(SO_LINGER=0).

Coverage:
* Network drop mid-SELECT — OperationalError, not hang
* Network drop after describe, before fetch
* Network drop during fetch (already-materialized rows still
  readable; fresh execute fails)
* Local socket forced-close (kernel-level disconnect simulation)
* I/O error marks connection unusable post-failure
* Pool evicts connection that died mid-`with` block (size drops)
* Pool revives after all idle connections died (health check on
  acquire mints fresh)
* Async cancellation via asyncio.wait_for — pool stays usable
* Cursor reusable after SQL error
* Connection survives cursor close after error
* Sustained pool load (50 acquire/release cycles, no leak)
* read_timeout fires on a hung connection within bounds

Catches the failure classes that bite production users:
* Hangs (waiting forever on dead socket)
* Silent corruption (EOF treated as valid tuple)
* Double-fault (cleanup raises after primary error)
* Pool poisoning (broken connection returned to pool)
* Stale cursor reuse across error boundaries

Helper:
* tests/_proxy.py — ControlledProxy: thread-based TCP forwarder
  with kill() for fault injection. Two-thread pump model. SO_LINGER=0
  for RST-on-close (mimics router drop).

Total: 69 unit + 203 integration = 272 tests.

Remaining gaps from the audit (UTF-8 multibyte locale, server-version
matrix, performance benchmarks) are real but lower-severity. Phase 19
addressed the one most likely to bite production deployments.
 2026-05-04 16:57:06 -06:00
Ryan Malloy
a42dc5c5de Phase 18: server-side scrollable cursors via SQ_SFETCH (v2026.05.04.2) 
Opt-in via conn.cursor(scrollable=True). Opens the cursor with
SQ_SCROLL (24) before SQ_OPEN (6), keeps it open server-side, and
sends SQ_SFETCH (23) per scroll call instead of materializing the
result set up-front.

User-facing API is identical to Phase 17's in-memory scroll
(fetch_first/last/prior/absolute/relative, scroll, rownumber).
Only the internal mechanism differs:

  | feature           | default          | scrollable=True
  |-------------------|------------------|------------------
  | memory            | all rows         | one row at a time
  | round-trips/fetch | 0 (after NFETCH) | 1 per call
  | cursor lifetime   | closed after exec| open until close()
  | best for          | sequential iter  | random access on
                                         | huge result sets

Wire format (verified against JDBC ScrollProbe capture):
* SQ_SFETCH: [short SQ_ID=4][int 23][short scrolltype]
  [int target][int bufSize=4096][short SQ_EOT]
  scrolltype: 1=NEXT, 4=LAST, 6=ABSOLUTE
* SQ_SCROLL (24): emitted between CURNAME and SQ_OPEN
* SQ_TUPID (25): response tag with 1-indexed row position;
  authoritative source for client-side position tracking

Position tracking uses the server's SQ_TUPID rather than client-
computed indexes. Total row count discovered lazily via SFETCH(LAST)
when negative absolute indexing requires it; cached in
_scroll_total_rows.

Trap on the way: initial SFETCH used SHORT for bufSize → server
hung silently. Same SHORT-vs-INT diagnostic pattern as Phase 4.x's
CURNAME+NFETCH. Captured JDBC trace, byte-diffed against ours,
found the mismatch (bufSize is INT in modern Informix per
isXPSVER8_40 / is2GBFetchBufferSupported).

Tests: 14 integration tests in test_scroll_cursor_server.py
covering lifecycle, sequential fetch, fetch_first/last/prior/
absolute/relative, negative indexing, scroll, empty result sets,
past-end, and random-access on a 100-row result set.

Total: 69 unit + 191 integration = 260 tests.
 2026-05-04 16:41:25 -06:00
Ryan Malloy
461c62c8d3 Phase 17: scroll cursor API (v2026.05.04.1) 
Adds scroll/random-access methods on Cursor:
* scroll(value, mode='relative'|'absolute') — PEP 249 compatible
* fetch_first() / fetch_last() — jump to result-set ends
* fetch_prior() — step backward (SQL-standard: from past-end yields
  the last row, matching JDBC ResultSet.previous() semantics)
* fetch_absolute(n) — 0-indexed jump; negative n indexes from end
* fetch_relative(n) — n-step from current position
* rownumber property — current 0-indexed position

Implementation: replaced _row_iter (single-pass iterator) with
_row_index (random-access index) on the cursor. The result set
is already materialized in _rows during execute(); scroll just
repositions the index. No new wire protocol needed.

For server-side scroll over genuinely huge result sets, SQ_SFETCH
(tag 23) would be needed — JDBC has executeScrollFetch (line 3908)
but we only need it if someone hits the in-memory materialization
ceiling. Phase 18 if so.

Out-of-range scroll raises IndexError per PEP 249. Invalid mode
strings raise ProgrammingError. fetchall() now correctly returns
only the rows from the current position to end (not all rows).

14 new integration tests in test_scroll_cursor.py covering:
* fetchone advancing rownumber sequentially
* fetch_first reset
* fetch_last
* fetch_prior including the past-end-to-last-row semantics
* fetch_absolute with positive and negative indexes
* fetch_relative
* PEP 249 scroll(value, mode='relative'/'absolute')
* IndexError on out-of-range
* ProgrammingError on bad mode
* Empty-result-set edge cases
* fetchall after partial iteration

Total: 69 unit + 177 integration = 246 tests.
 2026-05-04 15:51:24 -06:00
110 changed files with 25904 additions and 396 deletions
Show Stats Expand all files Collapse all files

6
.gitignore vendored
View File

@ -58,3 +58,9 @@ build/*.jar
# Java reference client build outputs # Java reference client build outputs
*.class *.class
tests/benchmarks/.results/
# IfxPy comparison: keep Dockerfile, bench script, README;
# exclude the downloaded ODBC driver tarball and local venv.
tests/benchmarks/compare/venv-py311/
tests/benchmarks/compare/onedb/
tests/benchmarks/compare/onedb.tar

1416
CHANGELOG.md
View File

File diff suppressed because it is too large Load Diff

54
Makefile
View File

@ -32,15 +32,30 @@ format: ## Auto-format with ruff
test: ## Run unit tests (no Docker required) test: ## Run unit tests (no Docker required)
uv run pytest uv run pytest
test-integration: ## Run integration tests (needs Informix container; see `make ifx-up`) test-integration: ## Run integration tests (needs Informix container; see `make ifx-up`). Excludes benchmarks; use `make bench` for those.
uv run pytest -m integration uv run pytest -m "integration and not benchmark"
test-all: ## Run unit + integration tests test-all: ## Run unit + integration tests (no benchmarks; use `make bench` for those)
uv run pytest -m "" uv run pytest -m "not benchmark"
test-pdu: ## Run only the JDBC-vs-Python PDU regression test test-pdu: ## Run only the JDBC-vs-Python PDU regression test
uv run pytest tests/test_pdu_match.py -v uv run pytest tests/test_pdu_match.py -v
bench: ## Run all benchmarks (needs container for end-to-end; codec works standalone)
uv run pytest tests/benchmarks/ -m benchmark --benchmark-only \
--benchmark-columns=median,iqr,mean,stddev,ops,rounds \
--benchmark-sort=mean
bench-codec: ## Run codec micro-benchmarks only (no container required)
uv run pytest tests/benchmarks/test_codec_perf.py -m benchmark --benchmark-only \
--benchmark-columns=median,iqr,mean,stddev,ops,rounds \
--benchmark-sort=mean
bench-save: ## Save current bench run under .results/ (manual: copy to baseline.json)
uv run pytest tests/benchmarks/ -m benchmark --benchmark-only \
--benchmark-storage=tests/benchmarks/.results \
--benchmark-save=run
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# Informix dev container # Informix dev container
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
@ -64,6 +79,37 @@ ifx-status: ## Check container health and listener readiness
@docker ps --filter name=$(IFX_CONTAINER) --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' @docker ps --filter name=$(IFX_CONTAINER) --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
@nc -zv 127.0.0.1 9088 2>&1 | head -1 @nc -zv 127.0.0.1 9088 2>&1 | head -1
ifx-spaces: ## Create blobspace1 + sbspace1 in the Informix 15 container (needed for LOB tests)
tests/setup-spaces.sh $(IFX_CONTAINER)
# ----------------------------------------------------------------------------
# Server-compatibility matrix: Informix 12.10 / 14.10 alongside 15
# ----------------------------------------------------------------------------
# The legacy containers run on 9089 / 9090 so they coexist with the primary
# 15 container on 9088. The integration suite targets whichever server
# IFX_PORT names, so the matrix is just the same suite three times.
ifx-legacy-up: ## Start Informix 12.10 (9089) and 14.10 (9090) alongside 15
docker compose -f tests/docker-compose.legacy.yml up -d
@echo " 12.10 -> 127.0.0.1:9089"
@echo " 14.10 -> 127.0.0.1:9090"
@echo " Engine init takes ~1-3 min on first start; then: make ifx-legacy-setup"
ifx-legacy-setup: ## Create blobspace1 + sbspace1 in both legacy containers
tests/setup-spaces.sh informix-db-test-1210
tests/setup-spaces.sh informix-db-test-1410
ifx-legacy-down: ## Stop and remove the legacy containers
docker compose -f tests/docker-compose.legacy.yml down
test-matrix: ## Run the integration suite against 15 (9088), 14.10 (9090), and 12.10 (9089)
@echo "=== Informix 15 (9088) ==="
@IFX_PORT=9088 uv run pytest -m "integration and not benchmark" -q --no-header
@echo "=== Informix 14.10 (9090) ==="
@IFX_PORT=9090 uv run pytest -m "integration and not benchmark" -q --no-header
@echo "=== Informix 12.10 (9089) ==="
@IFX_PORT=9089 uv run pytest -m "integration and not benchmark" -q --no-header
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# Phase 0 spike: re-capture wire traffic against the dev container # Phase 0 spike: re-capture wire traffic against the dev container
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------

166
README.md
View File

@ -1,13 +1,34 @@
# informix-db # informix-driver
Pure-Python driver for IBM Informix IDS, speaking the SQLI wire protocol over raw sockets. **No IBM Client SDK. No JVM. No native libraries.** PEP 249 compliant; sync + async APIs; built-in connection pool; TLS support. Pure-Python driver for IBM Informix IDS, speaking the SQLI wire protocol over raw sockets. **No IBM Client SDK. No JVM. No native libraries.** PEP 249 compliant; sync + async APIs; built-in connection pool; TLS support.
**Docs:** [informix-driver.warehack.ing](https://informix-driver.warehack.ing) · **Source:** [git.supported.systems/warehack.ing/informix-db](https://git.supported.systems/warehack.ing/informix-db) · **PyPI:** [informix-driver](https://pypi.org/project/informix-driver/)
To our knowledge this is the **first pure-socket Informix driver in any language** — every other Informix driver (`IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix`) wraps either IBM's CSDK or the JDBC JAR. To our knowledge this is the **first pure-socket Informix driver in any language** — every other Informix driver (`IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix`) wraps either IBM's CSDK or the JDBC JAR.
```bash ```bash
pip install informix-db pip install informix-driver
``` ```
Imports as `informix_db` (the distribution name is `informix-driver` because the legacy `informixdb` package on PyPI from 2008 reserves close-by names — same separation Pillow uses with `import PIL`). Requires Python ≥ 3.10.
## Status
**Production ready.** Every finding from a system-wide failure-mode audit (data correctness, wire safety, resource leaks, concurrency, async cancellation) has been addressed:
| Severity | Finding | Status |
|---|---|---|
| Critical | Pool returns connections with open transactions | Fixed (Phase 26) |
| Critical | Unsynchronized wire path → PDU interleaving | Fixed (Phase 27) — per-connection wire lock |
| High | Async cancellation leaks running workers onto recycled connections | Fixed (Phase 27) |
| High | `_raise_sq_err` bare-except masks wire desync | Fixed (Phase 28) |
| High | Cursor finalizers — server-side resources leak on mid-fetch raise | Fixed (Phase 28+29) |
| Medium | 5 hardening items | Fixed (Phase 28+30) |
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
**Test coverage:** 400+ tests across unit / integration / benchmark suites. The integration suite passes 414/414 against **each** of Informix 12.10, 14.10, and 15 — `make test-matrix` runs all three.
## Quick start ## Quick start
```python ```python
@ -81,10 +102,11 @@ Informix uses dedicated TLS-enabled listener ports (configured server-side in `s
| SQL type | Python type | | SQL type | Python type |
|---|---| |---|---|
| `SMALLINT` / `INT` / `BIGINT` / `SERIAL` | `int` | | `SMALLINT` / `INT` / `SERIAL` / `BIGINT` / `BIGSERIAL` | `int` |
| `INT8` / `SERIAL8` | `int` (legacy 64-bit — a different wire format from `BIGINT`, not an alias) |
| `FLOAT` / `SMALLFLOAT` | `float` | | `FLOAT` / `SMALLFLOAT` | `float` |
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | | `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` |
| `CHAR` / `VARCHAR` / `NCHAR` / `NVCHAR` / `LVARCHAR` | `str` | | `CHAR` / `NCHAR` (fixed width) · `VARCHAR` / `NVCHAR` / `LVARCHAR` (variable) | `str` |
| `BOOLEAN` | `bool` | | `BOOLEAN` | `bool` |
| `DATE` | `datetime.date` | | `DATE` | `datetime.date` |
| `DATETIME YEAR TO ...` | `datetime.datetime` / `datetime.time` / `datetime.date` | | `DATETIME YEAR TO ...` | `datetime.datetime` / `datetime.time` / `datetime.date` |
@ -112,7 +134,7 @@ cur.write_blob_column(
) )
``` ```
Both work end-to-end in pure Python via the `lotofile` / `filetoblob` server functions intercepted at the `SQ_FILE` (98) wire-protocol level — no thread of native machinery. See [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) §10–11 for the architecture pivot that made this possible. Both work end-to-end in pure Python via the `lotofile` / `filetoblob` server functions intercepted at the `SQ_FILE` (98) wire-protocol level — no native machinery anywhere in the thread of execution. See [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) §10–11 for the architecture pivot that made this possible.
## Direct stored-procedure invocation (fast-path) ## Direct stored-procedure invocation (fast-path)
@ -128,50 +150,138 @@ The fast-path RPC (`SQ_FPROUTINE` / `SQ_EXFPROUTINE`) bypasses PREPARE → EXECU
## Server compatibility ## Server compatibility
Tested against IBM Informix Dynamic Server **15.0.1.0.3DE** (the official `icr.io/informix/informix-developer-database` Docker image). The wire protocol is stable across modern Informix versions; should work against 12.10+ unmodified. All three tested against the official IBM developer-edition Docker images, full integration suite, same commit:
For features that need server-side configuration (smart-LOBs, logged transactions), see [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md): | Server | Image | Integration suite |
|---|---|---|
| **15.0.1.0.3DE** | `icr.io/informix/informix-developer-database` | **414 / 414** |
| **14.10.FC7W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
| **12.10.FC12W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
Reproduce the whole matrix:
```bash
make ifx-up # Informix 15 on 9088
make ifx-legacy-up # 12.10 on 9089, 14.10 on 9090
make ifx-legacy-setup # blobspace1 + sbspace1 (smart-LOB tests need these)
make test-matrix # the suite against all three
```
The suite targets whichever server `IFX_PORT` names, so a single version is just `IFX_PORT=9089 uv run pytest -m "integration and not benchmark"`.
Beyond the suite passing, a 28-type round-trip (every scalar type we support, including the fiddly ones — `INT8`, `NCHAR`, `BOOLEAN`, `DATETIME YEAR TO FRACTION(5)`, `INTERVAL`, `DECIMAL`, `MONEY`, `LVARCHAR`) produces **byte-identical wire output** on all three versions: same type codes, same encoded lengths, same values.
Earlier releases of this README claimed the protocol was "stable across modern versions" and should "work against 12.10+ unmodified." That claim turned out to be correct, but it was written before anyone had run it against 12.10, and a user lost debugging time to a version-compatibility problem that didn't exist. It's measured now. Apologies for the earlier guess.
### Capability negotiation
SQLI settles some wire framing through a capability exchange (`SQ_PROTOCOLS`) rather than fixing it by version. This driver decodes that exchange and checks it against the framing it emits:
```python
conn = informix_db.connect(...)
conn.server_version # release, e.g. '…Version 12.10.FC6' (one cached query)
conn.server_version_internal # raw login string; 12.10 announces itself as 9.56
conn.server_capabilities.four_byte_offset # True
conn.server_capabilities.violated_assumptions() # [] — hardcoded framing matches
```
All three servers above negotiate an identical 64-bit capability set, and `violated_assumptions()` is empty on each. If it ever returns something, the driver logs a warning naming the specific bit at connect time — a framing mismatch otherwise shows up as silently corrupted rows, which is a miserable thing to debug.
The framing itself is still hardcoded rather than driven by those bits. That's a deliberate limit: it's correct everywhere we've measured, and rewriting working parse paths to be conditional without a server that needs it would trade certainty for risk. If you hit corrupted rows on a server outside that table, check `violated_assumptions()` first, then please open an issue with your server version, the capability mask, and the `cursor.description` of the offending query.
For features that need server-side configuration (smart-LOBs, logged transactions), see [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md):
- Phase 7 — logged-DB transactions - Phase 7 — logged-DB transactions
- Phase 8 — BYTE/TEXT (needs blobspace) - Phase 8 — BYTE/TEXT (needs blobspace)
- Phase 10/11 — BLOB/CLOB (needs sbspace + `SBSPACENAME` config + level-0 archive) - Phase 10/11 — BLOB/CLOB (needs sbspace + `SBSPACENAME` config + level-0 archive)
## Performance
Single-connection benchmarks against the dev container on loopback:
| Operation | Mean | Throughput |
|---|---:|---:|
| `decode(int)` per cell | 139 ns | 7.2M ops/sec |
| `parse_tuple_payload` per row (5 cols) | 1.4 µs | 715K rows/sec |
| `SELECT 1` round-trip | ~140 µs | ~7K queries/sec |
| 1000-row SELECT | ~1.0 ms | ~990K rows/sec sustained |
| `executemany(1000)` in transaction | 32 ms | **~31,000 rows/sec** |
| Pool acquire + query + release | 295 µs | ~3.4K queries/sec |
| Cold connect (login handshake) | 11 ms | ~90 connections/sec |
**Performance gotcha**: `executemany(...)` under `autocommit=True` is **53× slower** than the same call inside a single transaction (server flushes the transaction log per row). For bulk loads, `autocommit=False` (default) + `conn.commit()` at the end. See [`docs/USAGE.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/USAGE.md) for the full performance tips section.
### Compared to IfxPy (the C-bound PyPI driver)
Head-to-head benchmarks against [IfxPy](https://pypi.org/project/IfxPy/) on identical workloads, same Informix server, matched conditions. Using **median + IQR over 10+ rounds** to resist outlier-round noise:
| Benchmark | IfxPy 3.0.5 (C-bound) | `informix-driver` (pure Python) | Result |
|---|---:|---:|---:|
| Single-row SELECT round-trip | 118 µs | 114 µs | comparable |
| ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster |
| Cold connect (login handshake) | 11.0 ms | 10.5 ms | comparable |
| **`executemany(1k)` in transaction** | 23.5 ms | 23.2 ms | tied |
| **`executemany(10k)` in transaction** | 259 ms | **161 ms** | **`informix-driver` 1.6× faster** |
| **`executemany(100k)` in transaction** | 2376 ms | **1487 ms** | **`informix-driver` 1.6× faster** |
| `SELECT` 1k rows | 1.2 ms | 2.7 ms | IfxPy 2.3× faster |
| `SELECT` 10k rows | 11.3 ms | 25.8 ms | IfxPy 2.3× faster |
| `SELECT` 100k rows | 112 ms | 271 ms | IfxPy 2.4× faster |
**The honest summary:**
- **Bulk-insert workloads: `informix-driver` wins 1.6× at scale.** The pipelined `executemany` (Phase 33) sends all N BIND+EXECUTE PDUs before draining responses, eliminating per-row RTT. IfxPy still pays one round-trip per `IfxPy.execute(stmt, tuple)` call.
- **Large-fetch workloads: IfxPy wins 2.3× at scale.** Their C-level `fetch_tuple` decoder is genuinely faster than our Python `parse_tuple_payload` (~1.1 µs/row vs ~2.7 µs/row). At 100k rows, that 1.6 µs/row gap accumulates into a 160 ms wall-clock difference.
- **Small queries: comparable.** Both spend ~120 µs waiting for the server; the per-call codec cost is small relative to the round-trip.
**When to prefer `informix-driver`:**
- ETL pipelines, log shipping, bulk writes (1.6× faster at scale)
- Containerized / minimal-dependency environments (50 KB wheel vs IfxPy's 92 MB OneDB tarball + libcrypt.so.1 dependency hell)
- Modern Python (works on 3.10–3.14; IfxPy is broken on Python 3.12+)
- Async / FastAPI workloads (we have native async; IfxPy doesn't)
**When IfxPy may be faster:**
- Analytical reporting queries pulling 10k+ rows in a single SELECT
- Workloads where the per-row decode cost dominates (wide rows, tight read loops)
These results are reproducible from `tests/benchmarks/compare/` — the Dockerfile, bench script, and README walk through every step.
Full methodology, IQR caveats, install gauntlet, and reproduction in [`tests/benchmarks/compare/README.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare/README.md).
A note on IfxPy's install gauntlet: getting it to run on a modern system requires Python ≤ 3.11, setuptools <58, permissive CFLAGS, manual download of a 92 MB ODBC tarball, four `LD_LIBRARY_PATH` directories, and `libcrypt.so.1` (deprecated 2018, missing on Arch / Fedora 35+ / RHEL 9). `informix-driver`'s install: `pip install informix-driver`.
## Standards & guarantees ## Standards & guarantees
* **PEP 249** (DB-API 2.0): `connect()`, `Connection`, `Cursor`, `description`, `rowcount`, exception hierarchy * **PEP 249** (DB-API 2.0): `connect()`, `Connection`, `Cursor`, `description`, `rowcount`, exception hierarchy
* **`paramstyle = "numeric"`** (Informix's native ESQL/C convention; `?` and `:1` both work) * **`paramstyle = "numeric"`** (Informix's native ESQL/C convention; `?` and `:1` both work)
* **Threadsafety = 1**: threads may share the module but not connections; the pool gives per-thread connection access * **Threadsafety = 1**: threads may share the module but not connections; the pool gives per-thread connection access. Phase 27 added a per-connection wire lock that makes accidental sharing safe (interleaved PDUs serialize correctly), but PEP 249 advice still holds — give each thread its own connection.
* **CalVer versioning**: `YYYY.MM.DD` releases. PEP 440 post-releases (`.1`, `.2`) for same-day fixes. * **CalVer versioning**: `YYYY.MM.DD` releases. PEP 440 post-releases (`.1`, `.2`) for same-day fixes.
## Development ## Development
```bash The full test + lint workflow is in the [Makefile](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/Makefile). Quick summary:
# Set up the dev environment
uv sync --dev
# Run the test suite (unit-only by default; no Docker needed)
uv run pytest # 69 unit tests
uv run pytest -m integration # 163 integration tests (needs Docker)
# Lint
uv run ruff check src/ tests/
```
The integration suite expects an Informix Developer Edition container on `localhost:9088`:
```bash ```bash
docker compose -f tests/docker-compose.yml up -d make test # 77 unit tests (no Docker)
make ifx-up && make test-integration # 231 integration tests
make bench # benchmark suite
make lint # ruff
``` ```
For the smart-LOB tests specifically, the dev container needs additional one-time setup (blobspace + sbspace + level-0 archive). See [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) §10 for the exact `onspaces` / `onmode` / `ontape` commands. For the smart-LOB tests specifically, the dev container needs additional one-time setup (blobspace + sbspace + level-0 archive). See [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) §10 for the `onspaces` / `onmode` / `ontape` commands.
## Documentation
- [**`docs/USAGE.md`**](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/USAGE.md) — practical recipes: connections, parameter binding, type mapping, transactions, performance tips, scrollable cursors, BLOBs, async, TLS, locale/Unicode, error handling, known limitations
- [`tests/benchmarks/README.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/README.md) — performance baselines, headline numbers, how to run regressions
- `CHANGELOG.md` — phase-by-phase release notes
## Project history & design rationale ## Project history & design rationale
This driver was built incrementally over 16 phases, each with a focused scope and decision log. The full reasoning trail lives in: This driver was built incrementally across 30 phases, each with a focused scope and decision log. The reasoning trail lives in:
- [`docs/PROTOCOL_NOTES.md`](docs/PROTOCOL_NOTES.md) — byte-level SQLI wire-format reference - [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) — byte-level SQLI wire-format reference
- [`docs/JDBC_NOTES.md`](docs/JDBC_NOTES.md) — index into the decompiled IBM JDBC driver, used as a clean-room reference - [`docs/JDBC_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/JDBC_NOTES.md) — index into the decompiled IBM JDBC driver, used as a clean-room reference
- [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) — phase-by-phase architectural decisions, with the *why* preserved - [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) — phase-by-phase architectural decisions, with the *why* preserved
- [`docs/CAPTURES/`](docs/CAPTURES/) — annotated socat hex-dump captures - [`docs/CAPTURES/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/CAPTURES/) — annotated socat hex-dump captures
Notable architectural pivots documented in the decision log: Notable architectural pivots documented in the decision log:
- **Phase 10/11** (smart-LOB read/write): used `lotofile`/`filetoblob` SQL functions + `SQ_FILE` protocol intercept instead of the heavier `SQ_FPROUTINE` + `SQ_LODATA` stack — ~3x smaller than originally projected - **Phase 10/11** (smart-LOB read/write): used `lotofile`/`filetoblob` SQL functions + `SQ_FILE` protocol intercept instead of the heavier `SQ_FPROUTINE` + `SQ_LODATA` stack — ~3x smaller than originally projected

12
docs-site/.dockerignore Normal file
View File

@ -0,0 +1,12 @@
node_modules
dist
.astro
.env
.env.local
.git
*.log
README.md
.dockerignore
Dockerfile
docker-compose.yml
Makefile

4
docs-site/.env.example Normal file
View File

@ -0,0 +1,4 @@
COMPOSE_PROJECT=informix-db-docs
DOMAIN=informix-driver.warehack.ing
DEV_DOMAIN=informix-driver.l.warehack.ing
MODE=prod

21
docs-site/.gitignore vendored Normal file
View File

@ -0,0 +1,21 @@
# build output
dist/
# generated types
.astro/
# dependencies
node_modules/
# logs
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
# environment variables
.env
.env.production
# macOS-specific files
.DS_Store

20
docs-site/Caddyfile Normal file
View File

@ -0,0 +1,20 @@
:80 {
root * /srv
encode zstd gzip
file_server
@assets path *.css *.js *.woff2 *.woff *.svg *.png *.webp *.jpg *.jpeg *.ico
header @assets Cache-Control "public, max-age=31536000, immutable"
@html path *.html
header @html Cache-Control "public, max-age=300, must-revalidate"
header @html X-Content-Type-Options "nosniff"
header @html X-Frame-Options "SAMEORIGIN"
header @html Referrer-Policy "strict-origin-when-cross-origin"
handle_errors {
@404 expression `{err.status_code} == 404`
rewrite @404 /404.html
file_server
}
}

30
docs-site/Dockerfile Normal file
View File

@ -0,0 +1,30 @@
ARG NODE_VERSION=22-alpine
ARG CADDY_VERSION=2.10-alpine
FROM node:${NODE_VERSION} AS deps
WORKDIR /app
COPY package.json package-lock.json* ./
RUN --mount=type=cache,target=/root/.npm \
npm ci --no-audit --no-fund 2>/dev/null || npm install --no-audit --no-fund
FROM node:${NODE_VERSION} AS builder
WORKDIR /app
ENV ASTRO_TELEMETRY_DISABLED=1 NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build
FROM caddy:${CADDY_VERSION} AS prod
COPY --from=builder /app/dist /srv
COPY Caddyfile /etc/caddy/Caddyfile
EXPOSE 80
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD wget -q --spider http://127.0.0.1/ || exit 1
FROM node:${NODE_VERSION} AS dev
WORKDIR /app
ENV ASTRO_TELEMETRY_DISABLED=1 NODE_ENV=development
COPY --from=deps /app/node_modules ./node_modules
COPY . .
EXPOSE 4321
CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0"]

33
docs-site/Makefile Normal file
View File

@ -0,0 +1,33 @@
.PHONY: prod dev down logs build clean install local
include .env
export
prod:
docker compose --profile prod up -d --build
dev:
docker compose --profile dev up -d --build
down:
docker compose --profile prod --profile dev down
logs:
docker compose logs -f --tail=200
build:
npm run build
clean:
rm -rf dist node_modules .astro
install:
npm install
local:
npm run dev -- --host 127.0.0.1
status:
@echo "Domain: $(DOMAIN)"
@echo "Project: $(COMPOSE_PROJECT)"
@docker compose ps

49
docs-site/README.md Normal file
View File

@ -0,0 +1,49 @@
# Starlight Starter Kit: Basics
[![Built with Starlight](https://astro.badg.es/v2/built-with-starlight/tiny.svg)](https://starlight.astro.build)
```
npm create astro@latest -- --template starlight
```
> 🧑‍🚀 **Seasoned astronaut?** Delete this file. Have fun!
## 🚀 Project Structure
Inside of your Astro + Starlight project, you'll see the following folders and files:
```
.
├── public/
├── src/
│ ├── assets/
│ ├── content/
│ │ └── docs/
│ └── content.config.ts
├── astro.config.mjs
├── package.json
└── tsconfig.json
```
Starlight looks for `.md` or `.mdx` files in the `src/content/docs/` directory. Each file is exposed as a route based on its file name.
Images can be added to `src/assets/` and embedded in Markdown with a relative link.
Static assets, like favicons, can be placed in the `public/` directory.
## 🧞 Commands
All commands are run from the root of the project, from a terminal:
| Command | Action |
| :------------------------ | :----------------------------------------------- |
| `npm install` | Installs dependencies |
| `npm run dev` | Starts local dev server at `localhost:4321` |
| `npm run build` | Build your production site to `./dist/` |
| `npm run preview` | Preview your build locally, before deploying |
| `npm run astro ...` | Run CLI commands like `astro add`, `astro check` |
| `npm run astro -- --help` | Get help using the Astro CLI |
## 👀 Want to learn more?
Check out [Starlight’s docs](https://starlight.astro.build/), read [the Astro documentation](https://docs.astro.build), or jump into the [Astro Discord server](https://astro.build/chat).

119
docs-site/astro.config.mjs Normal file
View File

@ -0,0 +1,119 @@
// @ts-check
import { defineConfig } from 'astro/config';
import starlight from '@astrojs/starlight';
const DEV_DOMAIN = process.env.DEV_DOMAIN ?? 'informix-driver.l.warehack.ing';
// https://astro.build/config
export default defineConfig({
site: 'https://informix-driver.warehack.ing',
server: { host: '0.0.0.0', port: 4321 },
telemetry: false,
devToolbar: { enabled: false },
vite: {
server: {
host: '0.0.0.0',
hmr: {
host: DEV_DOMAIN,
protocol: 'wss',
clientPort: 443,
},
allowedHosts: [DEV_DOMAIN, '.warehack.ing', 'localhost', '127.0.0.1'],
},
},
integrations: [
starlight({
title: 'informix-driver',
description: 'Pure-Python driver for IBM Informix IDS. No CSDK, no JVM, no native libraries.',
logo: { src: './src/assets/logo.svg', replacesTitle: false },
favicon: '/favicon.svg',
tableOfContents: { minHeadingLevel: 2, maxHeadingLevel: 4 },
lastUpdated: true,
pagination: true,
editLink: {
baseUrl: 'https://git.supported.systems/warehack.ing/informix-db/_edit/branch/main/docs-site/',
},
social: [
{ icon: 'seti:git', label: 'Source (Gitea)', href: 'https://git.supported.systems/warehack.ing/informix-db' },
{ icon: 'seti:python', label: 'PyPI', href: 'https://pypi.org/project/informix-driver/' },
],
customCss: ['./src/styles/theme.css', './src/styles/components.css'],
components: {
Hero: './src/components/Hero.astro',
Footer: './src/components/Footer.astro',
},
expressiveCode: {
themes: ['github-dark', 'github-light'],
styleOverrides: {
borderRadius: '6px',
codeFontFamily: "'IBM Plex Mono', ui-monospace, SFMono-Regular, Menlo, monospace",
},
},
head: [
{
tag: 'link',
attrs: { rel: 'preconnect', href: 'https://rsms.me' },
},
{
tag: 'link',
attrs: {
rel: 'stylesheet',
href: 'https://rsms.me/inter/inter.css',
},
},
{
tag: 'meta',
attrs: { name: 'theme-color', content: '#0e0d0c' },
},
{
tag: 'meta',
attrs: { property: 'og:type', content: 'website' },
},
],
sidebar: [
{
label: 'Start here',
items: [
{ label: 'WTF did you build this for?', slug: 'start/wtf' },
{ label: 'Install & first query', slug: 'start/quickstart' },
{ label: 'Compared to IfxPy', slug: 'start/vs-ifxpy' },
],
},
{
label: 'How-to guides',
items: [
{ label: 'Connect with TLS', slug: 'how-to/tls' },
{ label: 'Use the connection pool', slug: 'how-to/pool' },
{ label: 'Async with FastAPI', slug: 'how-to/async-fastapi' },
{ label: 'Bulk inserts (executemany)', slug: 'how-to/executemany' },
{ label: 'Optimize bulk SELECT', slug: 'how-to/buffered-reader' },
{ label: 'BLOB / CLOB read & write', slug: 'how-to/smart-lobs' },
{ label: 'Migrate from IfxPy', slug: 'how-to/migrate-from-ifxpy' },
{ label: 'Run the dev container', slug: 'how-to/dev-container' },
],
},
{
label: 'Reference',
items: [
{ label: 'API surface', slug: 'reference/api' },
{ label: 'SQL ↔ Python types', slug: 'reference/types' },
{ label: 'Configuration & env flags', slug: 'reference/config' },
{ label: 'Exceptions & error codes', slug: 'reference/exceptions' },
{ label: 'Performance baselines', slug: 'reference/benchmarks' },
],
},
{
label: 'Explanation',
items: [
{ label: 'The SQLI wire protocol', slug: 'explain/sqli-protocol' },
{ label: 'Architecture overview', slug: 'explain/architecture' },
{ label: 'The buffered reader (Phase 39)', slug: 'explain/buffered-reader' },
{ label: 'Async strategy', slug: 'explain/async-strategy' },
{ label: 'Pure-Python tradeoffs', slug: 'explain/pure-python' },
{ label: 'The phase log', slug: 'explain/phase-log' },
],
},
],
}),
],
});

46
docs-site/docker-compose.yml Normal file
View File

@ -0,0 +1,46 @@
services:
docs:
profiles: ["prod"]
build:
context: .
target: prod
container_name: ${COMPOSE_PROJECT}-prod
restart: unless-stopped
networks:
- caddy
labels:
caddy: ${DOMAIN}
caddy.reverse_proxy: "{{upstreams 80}}"
docs-dev:
profiles: ["dev"]
build:
context: .
target: dev
container_name: ${COMPOSE_PROJECT}-dev
restart: unless-stopped
volumes:
- ./src:/app/src:cached
- ./public:/app/public:cached
- ./astro.config.mjs:/app/astro.config.mjs:cached
- ./tsconfig.json:/app/tsconfig.json:cached
- ./package.json:/app/package.json:cached
environment:
DEV_DOMAIN: ${DEV_DOMAIN}
networks:
- caddy
labels:
caddy: ${DEV_DOMAIN}
caddy.reverse_proxy: "{{upstreams 4321}}"
caddy.reverse_proxy.flush_interval: "-1"
caddy.reverse_proxy.transport: "http"
caddy.reverse_proxy.transport.read_timeout: "0"
caddy.reverse_proxy.transport.write_timeout: "0"
caddy.reverse_proxy.transport.keepalive: "5m"
caddy.reverse_proxy.transport.keepalive_idle_conns: "10"
caddy.reverse_proxy.stream_timeout: "24h"
caddy.reverse_proxy.stream_close_delay: "5s"
networks:
caddy:
external: true

6269
docs-site/package-lock.json generated Normal file
View File

File diff suppressed because it is too large Load Diff

17
docs-site/package.json Normal file
View File

@ -0,0 +1,17 @@
{
"name": "docs-site",
"type": "module",
"version": "0.0.1",
"scripts": {
"dev": "astro dev",
"start": "astro dev",
"build": "astro build",
"preview": "astro preview",
"astro": "astro"
},
"dependencies": {
"@astrojs/starlight": "^0.39.1",
"astro": "^6.2.2",
"sharp": "^0.34.5"
}
}

6
docs-site/public/favicon.svg Normal file
View File

@ -0,0 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
<rect width="32" height="32" rx="6" fill="#0e0d0c"/>
<rect x="8" y="11" width="6" height="6" rx="1" fill="#f5a524"/>
<rect x="8" y="19" width="16" height="2" rx="1" fill="#f5a524"/>
<rect x="8" y="23" width="11" height="2" rx="1" fill="#f5a524" opacity="0.55"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 338 B

66
docs-site/public/supported-systems-logo.svg Normal file
View File

@ -0,0 +1,66 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 75" height="100%" width="100%">
<!-- Gradient Definitions -->
<defs>
<linearGradient id="gradient1" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" stop-color="#60a5fa"></stop>
<stop offset="50%" stop-color="#3b82f6"></stop>
<stop offset="100%" stop-color="#2563eb"></stop>
</linearGradient>
<linearGradient id="gradient2" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" stop-color="#93c5fd"></stop>
<stop offset="50%" stop-color="#60a5fa"></stop>
<stop offset="100%" stop-color="#3b82f6"></stop>
</linearGradient>
<linearGradient id="flowGradient" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" stop-color="#2563eb"></stop>
<stop offset="50%" stop-color="#60a5fa"></stop>
<stop offset="100%" stop-color="#2563eb"></stop>
</linearGradient>
<pattern id="circuitPattern" patternUnits="userSpaceOnUse" width="12" height="45" patternTransform="scale(1)">
<rect width="12" height="45" fill="url(#gradient1)"></rect>
<path d="M2,5 h8 M2,5 v5 M10,5 v10 M5,15 h5 M5,15 v10 M3,25 h7 M7,25 v10 M3,35 h4" stroke="#dbeafe" stroke-width="0.5" fill="none" opacity="0.7"></path>
<circle cx="2" cy="5" r="1" fill="#dbeafe" opacity="0.7"></circle>
<circle cx="10" cy="5" r="1" fill="#dbeafe" opacity="0.7"></circle>
<circle cx="5" cy="15" r="1" fill="#dbeafe" opacity="0.7"></circle>
<circle cx="3" cy="25" r="1" fill="#dbeafe" opacity="0.7"></circle>
<circle cx="7" cy="35" r="1" fill="#dbeafe" opacity="0.7"></circle>
</pattern>
<pattern id="binaryPattern" patternUnits="userSpaceOnUse" width="12" height="35" patternTransform="scale(1)">
<rect width="12" height="35" fill="#2563eb"></rect>
<text x="3" y="8" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">10</text>
<text x="3" y="14" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">01</text>
<text x="3" y="20" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">11</text>
<text x="3" y="26" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">00</text>
<text x="3" y="32" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">10</text>
</pattern>
<pattern id="punchCardPattern" patternUnits="userSpaceOnUse" width="12" height="45" patternTransform="scale(1)">
<rect width="12" height="45" fill="#3b82f6"></rect>
<path d="M0,5 h12 M0,10 h12 M0,15 h12 M0,20 h12 M0,25 h12 M0,30 h12 M0,35 h12 M0,40 h12" stroke="#93c5fd" stroke-width="0.2" fill="none"></path>
<circle cx="3" cy="7" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="9" cy="7" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="6" cy="12" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="3" cy="17" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="9" cy="22" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="6" cy="27" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="3" cy="32" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="9" cy="37" r="1" fill="#1e3a8a" opacity="0.9"></circle>
</pattern>
</defs>
<!-- Flow lines behind bars -->
<g opacity="0.3">
<path d="M6,50 C20,40 40,55 48,35 C56,50 75,30 90,55" stroke="url(#flowGradient)" stroke-width="1" fill="none"></path>
<path d="M6,60 C30,50 50,40 70,55 C80,45 90,60 90,60" stroke="url(#flowGradient)" stroke-width="1" fill="none"></path>
</g>
<!-- Bar chart graphic - the "towers" -->
<g>
<rect x="0" y="45" width="12" height="25" rx="1" ry="1" fill="url(#binaryPattern)"></rect>
<rect x="14" y="35" width="12" height="35" rx="1" ry="1" fill="#2563eb"></rect>
<rect x="28" y="25" width="12" height="45" rx="1" ry="1" fill="url(#circuitPattern)"></rect>
<rect x="42" y="20" width="12" height="50" rx="1" ry="1" fill="url(#gradient2)"></rect>
<rect x="56" y="25" width="12" height="45" rx="1" ry="1" fill="url(#punchCardPattern)"></rect>
<rect x="70" y="35" width="12" height="35" rx="1" ry="1" fill="url(#circuitPattern)"></rect>
<rect x="84" y="45" width="12" height="25" rx="1" ry="1" fill="#2563eb"></rect>
<!-- Connecting glow -->
<path d="M12,55 L14,55 M26,45 L28,45 M40,40 L42,40 M54,40 L56,40 M82,55 L84,55" stroke="#bfdbfe" stroke-width="0.8" stroke-opacity="0.6"></path>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 4.3 KiB

7
docs-site/src/assets/logo.svg Normal file
View File

@ -0,0 +1,7 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 28 24" fill="none" role="img" aria-label="informix-driver">
<title>informix-driver</title>
<rect x="0" y="4" width="5" height="16" rx="1" fill="#f5a524"/>
<rect x="8" y="4" width="3" height="16" rx="1" fill="#f5a524" fill-opacity="0.78"/>
<rect x="14" y="4" width="3" height="16" rx="1" fill="#f5a524" fill-opacity="0.52"/>
<rect x="20" y="4" width="3" height="16" rx="1" fill="#f5a524" fill-opacity="0.28"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 481 B

43
docs-site/src/components/Footer.astro Normal file
View File

@ -0,0 +1,43 @@
---
// Footer override that preserves Starlight's default chrome
// (prev/next pagination, last-updated, edit link) and appends
// a Supported Systems "joint" badge below.
import Default from '@astrojs/starlight/components/Footer.astro';
---
<Default><slot /></Default>
<aside class="ifx-ss-badge" aria-label="Supported Systems">
<a class="ifx-ss-badge__link" href="https://supported.systems" rel="noopener">
<img
class="ifx-ss-badge__logo"
src="/supported-systems-logo.svg"
alt=""
width="60"
height="45"
loading="lazy"
/>
<div class="ifx-ss-badge__copy">
<h3 class="ifx-ss-badge__heading">A Supported Systems Joint</h3>
<p class="ifx-ss-badge__body">
<code>informix-driver</code> is built and maintained by
<span class="ifx-ss-badge__name">Supported Systems</span> &mdash; a
boutique software studio focused on thoughtful, user-first technology.
We take databases personally.
</p>
<span class="ifx-ss-badge__cta">
Visit supported.systems
<svg viewBox="0 0 16 16" width="14" height="14" aria-hidden="true">
<path
d="M4 8h7M8 5l3 3-3 3"
fill="none"
stroke="currentColor"
stroke-width="1.5"
stroke-linecap="round"
stroke-linejoin="round"
/>
</svg>
</span>
</div>
</a>
</aside>

79
docs-site/src/components/Hero.astro Normal file
View File

@ -0,0 +1,79 @@
---
// Custom hero for the homepage. Renders the wire-dump easter egg
// alongside the headline + CTA stack. Uses real captured SQLI bytes
// from docs/CAPTURES/01-connect-only.socat.log (truncated and curated).
---
<section class="ifx-hero">
<div class="ifx-hero__copy">
<span class="ifx-hero__eyebrow">Pure Python · No CSDK · No JVM · No libcrypt.so.1</span>
<h1 class="ifx-hero__title">
Talk to Informix without
<strong>linking against IBM's 92 MB tarball.</strong>
</h1>
<p class="ifx-hero__lede">
Every other Informix driver wraps IBM's C SDK or the JDBC JAR. We weren't into that.
So we read the protocol and wrote it ourselves: PEP 249, sync and async, pooled, TLS.
Within 10% of IBM's own C driver on bulk fetches, <strong>1.6× faster</strong> on
bulk inserts. No compile step. No <code>LD_LIBRARY_PATH</code> ritual. No
<code>libcrypt.so.1</code> from 2018.
</p>
<div class="ifx-hero__cta">
<a class="primary" href="/start/quickstart/">Get started →</a>
<a class="secondary" href="/start/vs-ifxpy/">Compared to IfxPy</a>
<a class="secondary" href="https://git.supported.systems/warehack.ing/informix-db">Source</a>
</div>
<div class="ifx-hero__install">pip install informix-driver</div>
</div>
<div class="ifx-hero__visual">
<div class="ifx-wiredump" aria-hidden="true">
<div class="ifx-wiredump__scroll" id="ifx-wire">
<span class="ifx-wiredump__line"><span class="ifx-wiredump__direction ifx-wiredump__direction--out">{'>'} OUT</span> 01 c3 01 3c 00 00 00 64 00 65 00 00 00 3d 00 06</span>
<span class="ifx-wiredump__line"> 49 45 45 45 4d 00 00 6c 73 71 6c 65 78 65 63 00 <span class="ifx-wiredump__byte--ascii">IEEEM..lsqlexec.</span></span>
<span class="ifx-wiredump__line"> 00 00 00 00 00 00 06 39 2e 32 38 30 00 00 0c 52 <span class="ifx-wiredump__byte--ascii">.......9.280...R</span></span>
<span class="ifx-wiredump__line"> 44 53 23 52 30 30 30 30 30 30 00 00 05 73 71 6c <span class="ifx-wiredump__byte--ascii">DS#R000000...sql</span></span>
<span class="ifx-wiredump__line"> 69 00 00 00 01 3c 00 00 00 00 00 00 00 00 00 01 <span class="ifx-wiredump__byte--ascii">i....{'<'}.........</span></span>
<span class="ifx-wiredump__line"> 00 09 69 6e 66 6f 72 6d 69 78 00 00 07 69 6e 34 <span class="ifx-wiredump__byte--ascii">..informix...in4</span></span>
<span class="ifx-wiredump__line"></span>
<span class="ifx-wiredump__line"><span class="ifx-wiredump__direction ifx-wiredump__direction--in">{'<'} IN </span> 01 14 02 3c 10 00 00 64 00 65 00 00 00 3d 00 06</span>
<span class="ifx-wiredump__line"> 49 45 45 45 49 00 00 6c 73 72 76 69 6e 66 78 00 <span class="ifx-wiredump__byte--ascii">IEEEI..lsrvinfx.</span></span>
<span class="ifx-wiredump__line"> 00 00 00 00 00 00 2f 49 42 4d 20 49 6e 66 6f 72 <span class="ifx-wiredump__byte--ascii">....../IBM Infor</span></span>
<span class="ifx-wiredump__line"> 6d 69 78 20 44 79 6e 61 6d 69 63 20 53 65 72 76 <span class="ifx-wiredump__byte--ascii">mix Dynamic Serv</span></span>
<span class="ifx-wiredump__line"> 65 72 20 56 65 72 73 69 6f 6e 20 31 35 2e 30 2e <span class="ifx-wiredump__byte--ascii">er Version 15.0.</span></span>
<span class="ifx-wiredump__line"></span>
<span class="ifx-wiredump__line"><span class="ifx-wiredump__direction ifx-wiredump__direction--out">{'>'} OUT</span> 00 02 00 00 00 00 00 49 73 65 6c 65 63 74 20 46 <span class="ifx-wiredump__byte--ascii">.......Iselect F</span></span>
<span class="ifx-wiredump__line"> 49 52 53 54 20 31 20 73 69 74 65 20 66 72 6f 6d <span class="ifx-wiredump__byte--ascii">IRST 1 site from</span></span>
<span class="ifx-wiredump__line"> 20 69 6e 66 6f 72 6d 69 78 2e 73 79 73 74 61 62 <span class="ifx-wiredump__byte--ascii"> informix.systab</span></span>
<span class="ifx-wiredump__line"> 6c 65 73 20 77 68 65 72 65 20 74 61 62 6e 61 6d <span class="ifx-wiredump__byte--ascii">les where tabnam</span></span>
<span class="ifx-wiredump__line"> 65 20 3d 20 27 20 47 4c 5f 43 4f 4c 4c 41 54 45 <span class="ifx-wiredump__byte--ascii">e = ' GL_COLLATE</span></span>
</div>
</div>
<div class="ifx-wiredump__caption">
live capture · 01-connect-only.socat · sqli/9088
</div>
</div>
</section>
<script>
// Reveal lines progressively with reduced-motion respect.
const reduce = window.matchMedia('(prefers-reduced-motion: reduce)').matches;
const root = document.getElementById('ifx-wire');
if (root) {
const lines = Array.from(root.querySelectorAll('.ifx-wiredump__line'));
if (reduce) {
lines.forEach((l) => l.classList.add('is-visible'));
} else {
let i = 0;
const reveal = () => {
if (i < lines.length) {
lines[i].classList.add('is-visible');
i++;
setTimeout(reveal, 110 + Math.random() * 60);
}
};
// Defer start until after first paint so it feels like a fresh capture.
requestAnimationFrame(() => setTimeout(reveal, 380));
}
}
</script>

7
docs-site/src/content.config.ts Normal file
View File

@ -0,0 +1,7 @@
import { defineCollection } from 'astro:content';
import { docsLoader } from '@astrojs/starlight/loaders';
import { docsSchema } from '@astrojs/starlight/schema';
export const collections = {
docs: defineCollection({ loader: docsLoader(), schema: docsSchema() }),
};

81
docs-site/src/content/docs/explain/architecture.md Normal file
View File

@ -0,0 +1,81 @@
---
title: Architecture overview
description: How the layers stack, from socket through framing, codec, resultset, cursor, connection, and pool.
sidebar:
order: 2
---
The driver is six layers, each with a single responsibility, each testable in isolation.
```text
┌──────────────────────────────────────────────────┐
│ Connection / Pool │ ← public API
├──────────────────────────────────────────────────┤
│ Cursor │ ← PEP 249 surface
├──────────────────────────────────────────────────┤
│ ResultSet │ ← row iteration, prefetch
├──────────────────────────────────────────────────┤
│ Codec / Per-column │ ← decode SQL types → Python
│ readers │
├──────────────────────────────────────────────────┤
│ Protocol / PDU framing │ ← SQLI PDUs over the wire
├──────────────────────────────────────────────────┤
│ IfxSocket (buffered) │ ← raw bytes, recv() management
└──────────────────────────────────────────────────┘
```
## IfxSocket
The lowest layer. Wraps `socket.socket` with a connection-scoped read buffer (Phase 39). One `recv(64K)` per ~64 KB of incoming data; parsers read into the buffer via `struct.unpack_from(buf, offset)` rather than slicing copies.
Everything above this layer is `bytes` and `bytearray` arithmetic, with no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
See [The buffered reader →](/explain/buffered-reader/) for why the buffer lives here and not on the parser.
## Protocol / PDU framing
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean*, only how to read and write the byte shapes.
The PDU types and their fields were reverse-engineered from three sources:
1. The decompiled IBM JDBC driver (`com.informix.jdbc.IfxConnection` and the `IfxProtocol` class hierarchy).
2. Annotated `socat` captures of real client/server exchanges (`docs/CAPTURES/`).
3. Differential testing against IfxPy on identical data.
## Codec / Per-column readers
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types. See [SQL ↔ Python types](/reference/types/) for the full table.
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]`, with no per-column dispatch overhead.
Phase 38 went further with `exec()`-based codegen: for the hottest tables, the driver generates a flat decoder function with all readers inlined and dispatch decisions baked in. The generated function is the equivalent of unrolling the per-column dispatch into straight-line code.
## ResultSet
`_resultset.py`. Holds the cursor's column descriptors, the pre-baked decoder list, and the in-flight prefetch state. Manages `fetch_one`, `fetch_many`, `fetch_all` semantics.
Most cursor calls land here: when you do `cur.fetchone()`, the cursor delegates to its ResultSet which reads the next `SQ_TUPLE` PDU, runs it through the per-column decoders, and returns a tuple.
## Cursor
`cursors.py`. The PEP 249 cursor surface. `execute()`, `executemany()`, `fetchone()`, `fetchmany()`, `fetchall()`, `description`, `rowcount`, scrollable cursor support.
This is where parameter binding lives: a SQL statement with `?` placeholders gets prepared via `SQ_PREPARE`, the driver introspects the parameter shape via `SQ_DESCRIBE`, and bound values get encoded according to the parameter types.
## Connection / Pool
`connections.py` and `pool.py`. The connection owns the IfxSocket, manages transaction state, and is the parent of all cursors. The pool is a wrapper around N connections with the usual acquire/release/timeout semantics.
`aio.py` mirrors all of the above with `async def` versions, implemented via thread-pool wrapping (see [Async strategy →](/explain/async-strategy/)).
## Why this layering
Each boundary is testable in isolation:
- `IfxSocket` tests can use a `socket.socketpair()` and assert on byte streams.
- Protocol tests parse known-good captures from `docs/CAPTURES/` and verify the typed PDUs come out correctly.
- Codec tests pass synthetic byte payloads to per-column readers and assert the Python output.
- ResultSet tests can use a fake protocol that emits canned PDU sequences.
- Cursor tests use a fake ResultSet.
When a regression appears, the layered structure narrows the search: a wire-format test failing means it's the protocol layer; a row-tuple test failing with a corrupt-bytes input means the codec; a `fetchall` test failing means the ResultSet's iteration logic. The 300+ test suite leans heavily on this isolation.

61
docs-site/src/content/docs/explain/async-strategy.mdx Normal file
View File

@ -0,0 +1,61 @@
---
title: Async strategy
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async, and what that costs.
sidebar:
order: 4
---
import { Aside } from '@astrojs/starlight/components';
`informix-driver`'s async API (`from informix_db import aio`) is implemented by wrapping the sync core in a thread pool. Every `await cur.execute(...)` schedules the underlying sync `execute()` on the pool's executor.
This is a deliberate architectural choice from Phase 16. Here's the reasoning.
## What we considered
Three options for adding async support to a sync database driver:
1. **Full async I/O refactor.** Rewrite the protocol layer on top of `asyncio.Protocol` or `asyncio.StreamReader`. The codec, framing, and connection state all become coroutines. ~2000 lines of code, full test rewrite.
2. **Thread-pool wrapping.** Keep the sync core. Wrap each public method with `loop.run_in_executor()`. ~250 lines of code, sync tests still apply, no protocol-layer changes.
3. **Dual implementations.** Maintain two parallel code paths, one sync and one async. Most code duplicated. Worst of both worlds.
We picked option 2.
## Why option 2 was the right call
For typical database workloads, meaning request-scoped connections that mostly wait on I/O, the practical difference between option 1 and option 2 is small:
- **Latency**: option 1 has a slight edge (no thread context switch), but the difference is dwarfed by the actual database round-trip (~80 µs LAN, ~ms WAN). For a single query, option 2 adds ~5–10 µs of executor overhead.
- **Throughput under concurrency**: option 1 wins when you have N coroutines on M physical cores with M < N. The thread pool needs to context-switch between threads; the async loop just runs the next coroutine. For 10–100 concurrent FastAPI requests on a 4-core box, this difference is small.
- **Code complexity**: option 1 is dramatically harder to write and test. The protocol layer becomes asynchronous everywhere; cancellation paths multiply; the shape of "what does a partial PDU read look like" becomes a state machine instead of a `while not done: read_more()`.
For a driver that needs to be production-ready in finite engineering time, option 2 was the right call.
## What it costs
The honest costs:
- **One worker thread per concurrent in-flight query.** With 100 concurrent queries, you have 100 threads. This is fine for I/O-bound work (Python releases the GIL during socket reads) but doesn't scale beyond a few hundred concurrent queries on a single process.
- **Thread-pool sizing matters.** The default executor size (5 × CPU count) is fine for most workloads. For high-concurrency workloads, you may want a larger executor.
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe, in that cancelled workers cannot leak onto recycled pool connections, but the underlying syscall does still complete.
## What it doesn't cost
- **Cancellation safety.** This was the original concern. Phase 27's per-connection wire lock + worker reaping makes async cancellation cancellation-safe in the same sense `asyncpg` is.
- **FastAPI integration.** The `aio.Pool` is a drop-in replacement for any "async database pool" pattern. `Depends(get_conn)` works exactly as you'd expect.
- **Async generator support.** `async for row in cur` works. The fetch is per-row chunked through the executor; the iteration shape is async-native.
## When option 1 might still be worth it
The two scenarios where a full async I/O implementation would matter:
1. **Very high concurrency on a single process** (1000+ in-flight queries). Thread context-switching cost becomes measurable. We haven't hit this in practice.
2. **Sub-millisecond query latencies on a unloaded server.** The 5–10 µs executor overhead is a meaningful percentage. For typical Informix workloads where round-trip is ~80 µs+, it isn't.
If either becomes a real production concern, the layered architecture lets us swap in a fully-async lower half without changing the upper half. The cursor / connection / pool API doesn't care how the bytes get to and from the server. That's the option-2 win we explicitly preserved.
<Aside type="note">
This is a Phase 16 decision. The pivot from "rewrite as async-native" to "wrap the sync core" is documented in [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md). Three years from now, if it turns out we should have gone with option 1, we have a clear path.
</Aside>

147
docs-site/src/content/docs/explain/buffered-reader.mdx Normal file
View File

@ -0,0 +1,147 @@
---
title: The buffered reader
description: How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1× IfxPy by moving the recv() buffer one level down the object graph.
sidebar:
order: 3
---
import { Aside } from '@astrojs/starlight/components';
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39, a connection-scoped buffered reader, closed it from 2.4× to ~1.05–1.15× in about thirty minutes of code plus ten minutes of architectural debugging.
This page is about both the technical change and the failure mode that hid the win for two phases.
## The lever we couldn't see
After Phase 38, profiling a 100,000-row fetch showed:
| Category | Self time | % of wall clock |
|---|---:|---:|
| I/O machinery | 555 ms | 66% |
| Codec | 205 ms | 24% |
| Other | ~80 ms | 10% |
The headline "I/O dominated" was true. The interesting half is the breakdown of that 555 ms:
- Actual `recv()` syscalls: ~153 ms
- Python wrapper overhead: ~400 ms
That ~400 ms was our own buffer abstraction: a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
The kernel was doing maybe 25–30 ms of work. The other 130 ms of the gap-vs-IfxPy was friction we had introduced ourselves.
## The architecture pattern
Both `asyncpg` (in `buffer.pyx`) and `psycopg3` (in `pq.PGconn`) put a single growing read buffer on the protocol/connection object. The parser indexes into it via `struct.unpack_from(buf, offset)` rather than slicing copies. Refills happen via one large `recv(64K)` rather than many small `recv()`s for individual fields.
Phase 39 ports that pattern to `informix-driver`. The state machine:
```text
┌───────────────────────────────┐
│ IfxSocket │
│ ─ socket: socket.socket │
│ ─ buf: bytearray (growable) │
│ ─ offset: int (read cursor) │
│ │
│ recv(64K) when buf exhausted │
└───────────────────────────────┘
▲
│ reads via read_exact(n)
│
┌───────────────────────────────┐
│ SocketReader (per-PDU) │
│ ─ short-lived view │
│ ─ no buffer of its own │
└───────────────────────────────┘
```
The reader is a parser-view. The buffer outlives the reader. When the parser asks for `read_short()`, the reader calls `socket.read_exact(2)`, which slices two bytes out of the bytearray at `offset` and advances. If the bytearray runs out, `socket.recv(64K)` refills it.
Result: **one `recv()` per ~64 KB of incoming data**, not per field.
## The architectural mistake the first pass got wrong
The natural thing to call this is "BufferedSocketReader". The natural thing to do is put the bytearray on the reader. That's what I did first.
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken`, meaning the process was blocked in `recv()` waiting for bytes that weren't coming.
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes, 200 for response 1 and 400 for response 2, it consumed bytes belonging to response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket`, the connection-scoped wrapper. Readers are short-lived parser-views, and the buffer outlives them.
```python
# WRONG (first pass): buffer scoped to reader
class BufferedSocketReader:
def __init__(self, sock):
self.sock = sock
self.buf = bytearray() # ← dies with the reader
self.offset = 0
# RIGHT (Phase 39): buffer scoped to connection
class IfxSocket:
def __init__(self, sock):
self.sock = sock
self._read_buf = bytearray() # ← outlives all readers
self._read_offset = 0
def read_exact(self, n):
if self._read_offset + n > len(self._read_buf):
self._refill()
...
```
asyncpg and psycopg3 both place the buffer on the protocol/connection object. The architectural template was sitting in front of me before I started; I built the wrong shape anyway because "buffered reader" implies the buffer is *on* the reader.
It is not. The reader is a view. The buffer is state.
## The numbers
A/B-measured against the same Docker container, warmed cache, only the env flag differing:
| Workload | Phase 38 | Phase 39 | Δ |
|---|---:|---:|---:|
| `select_scaling_1000` | 2.901 ms | 1.716 ms | **−41%** |
| `select_scaling_10000` | 24.317 ms | 16.084 ms | **−34%** |
| `select_scaling_100000` | 250.363 ms | 168.982 ms | **−32%** |
Re-running the IfxPy comparison after Phase 39:
| Workload | IfxPy 2.0.7 (C) | informix-driver Phase 39 | Ratio |
|---|---:|---:|---:|
| `select_scaling_1000` | 1.637 ms | 1.716 ms | **1.05×** |
| `select_scaling_10000` | 15.07 ms | 16.08 ms | **1.07×** |
| `select_scaling_100000` | 147.4 ms | 169.0 ms | **1.15×** |
The 2.4× steady-state gap that existed before Phase 37 is now within 5–15% of the C driver, and the lower bound may already be within IfxPy's own measurement noise (its IQR on the 100k workload is 21%; ours is 0.2%).
## What the feature flag does
The buffered reader ships **enabled by default** in version 2026.05.05.12. To opt out (debugging, regressing a workload, A/B-measuring your own code):
```bash
IFX_BUFFERED_READER=0 python my_app.py
```
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime, so close and reopen the pool to flip behavior.
<Aside type="note">
The flag exists to make A/B measurement easy. There's no expected reason to disable it in production. If you hit a workload where the buffered reader is slower, that's a bug and we'd like to know.
</Aside>
## What we learned
The general pattern: **what's visible gets optimization attention; what's invisible gets written off as irreducible**.
The codec is visible: there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads (`read_short`, `read_int`, `read_exact(payload_size)`) are one-liners. What could be slow about that?
It was carrying ~30% of total wall time. Two phases of changelogs implicitly blamed "the protocol" for the remaining gap. The actual culprit was a few lines of `bytes.join` in a wrapper from Phase 1 that nobody had revisited.
The lesson is small and easy to state: a profile turns vibes into an attack surface. Write the closing paragraph after you've measured, not before.
## Read more
- **[Architecture overview →](/explain/architecture/)**: where the buffered reader sits in the layer stack.
- **[Phase log →](/explain/phase-log/)**: the full progression from Phase 1 through Phase 39+.
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)**: Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.

150
docs-site/src/content/docs/explain/phase-log.md Normal file
View File

@ -0,0 +1,150 @@
---
title: The phase log
description: Phase-by-phase narrative of how informix-driver got built, with notable architectural decisions called out.
sidebar:
order: 6
---
The driver was built across 39+ phases, each with a focused scope and a decision log. This page is a high-level index; the gory details (with rationale, alternatives considered, and rollback notes) live in [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md).
## Foundation (Phases 1–10)
| Phase | Title | Outcome |
|---|---|---|
| 1 | Socket + minimal SQ_INFO | First handshake against the dev container |
| 2–4 | Login, DBOPEN, error decoding | Can connect and select a database |
| 5 | Statement execution | `SELECT 1` works |
| 6 | Parameter binding | `?`-placeholders, basic types |
| 7 | Logged-DB transactions | Discovered Informix needs explicit `SQ_BEGIN` per tx in non-ANSI mode |
| 8 | BYTE / TEXT (legacy in-row blobs) | Needs blobspace |
| 9 | Scrollable cursors | `SQ_SCROLL` PDU, position semantics |
| 10/11 | Smart-LOB read & write | **Architectural pivot** to `SQ_FILE` intercept; ~3× smaller than projected |
## Hardening (Phases 12–20)
| Phase | Title | Outcome |
|---|---|---|
| 12 | Type system overhaul | Per-column readers (predecessor to Phase 37) |
| 13 | DECIMAL / MONEY exact precision | `decimal.Decimal` round-trip |
| 14 | DATETIME range typing | Returns `date` / `datetime` / `time` per field range |
| 15 | INTERVAL types | Custom `IntervalYM`, `timedelta` for D-to-F |
| 16 | Async API | **Pivot** to thread-pool wrapping (~250 lines) instead of full async refactor (~2000 lines) |
| 17 | Connection pool (sync) | min/max sizing, acquire timeout, max idle |
| 18 | Connection pool (async) | Mirror of sync API on `aio.Pool` |
| 19 | TLS support | Bring-your-own-context, `tls=True` for dev |
| 20 | Locale / Unicode | `client_locale`, full mapping in [`Connection.encoding`](/reference/types/) |
## Production review (Phases 21–30)
| Phase | Title | Outcome |
|---|---|---|
| 21 | Type-checking pass | `py.typed`, full mypy/pyright coverage |
| 22 | Error code mapping | SQLCODE → exception per [reference](/reference/exceptions/) |
| 23 | Health checks | Pool validates idle connections before return |
| 24 | Statement caching | Per-connection prepared-statement cache |
| 25 | Fast-path call (`SQ_FPROUTINE`) | Direct UDF/SPL invocation, bypassing PREPARE |
| 26 | **CRITICAL** | Pool returned connections with open transactions (fixed) |
| 27 | **CRITICAL** | Per-connection wire lock + async cancellation safety |
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync (fixed) |
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise (fixed) |
| 30 | Hardening pass | 5 medium-severity audit findings, all closed |
After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver is production-ready.
## Performance (Phases 31–39)
| Phase | Title | Result |
|---|---|---|
| 31 | Statement cache LRU tuning | Better hit rate on repeated queries |
| 32 | Cursor lifecycle optimization | Fewer round-trips on small queries |
| 33 | **Pipelined `executemany`** | **1.6× faster than IfxPy on bulk inserts** |
| 34 | LRU caches for type lookup | Removed dispatch overhead on hot paths |
| 35 | Memory profile pass | Identified 100k-row baseline |
| 36 | `IfxPy` comparison harness | Established the 2.4× bulk-fetch gap |
| 37 | **Per-column reader strategy** | −10% on bulk SELECT, ratio → 2.10× |
| 38 | **`exec()`-based row-decoder codegen** | Further −12%, ratio → 2.04× |
| 39 | **Connection-scoped buffered reader** | **−32% on bulk SELECT, ratio → 1.05–1.15×** |
The Phase 37–39 trajectory is documented in detail at [The buffered reader →](/explain/buffered-reader/), including the architectural mistake the first pass got wrong.
## Field reports and systematic review (2026.08–2026.09)
Phase 30's audit found nothing left. Then real schemas arrived, and the audit's clean bill of health turned out to be a statement about the questions it had asked rather than about the driver.
A field report from a user running Informix 12 surfaced three type-framing bugs in a single afternoon: `INT8` / `SERIAL8` never decoded at all, `NCHAR` losing its first character, and `BOOLEAN` corrupting every column after it. Fuzzing the type matrix found more, and each one had the same shape as the last.
`2026.09.02` went back over the driver looking for that shape rather than for new symptoms, and found eleven bugs.
| Area | What was wrong |
|---|---|
| Row framing | Nothing ever checked that a row consumed its own payload. Fourteen framing bugs had reached users, every one detectable for free |
| Smart LOBs | Read as a flat 72-byte field when they occupy 149, shifting every column after a non-final `BLOB` |
| Composites | A NULL collection skipped its length field, the same bug as NULL `LVARCHAR` twelve lines away |
| Transactions | `rollback()` after a SQL `BEGIN WORK` sent nothing and reported success |
| Classification | Comments, CTEs and parenthesized selects were run as DML and failed with `-260` |
| Scrollable cursors | A second statement on the connection got `-285` and destroyed the cursor too |
| Statement release | Two of eight exits leaked a failed statement, bricking the connection |
| Cursor finalizers | Cleanup could land inside another statement, and the lock probe was blind to its own thread |
| Async | The whole layer shared the process-wide thread pool, so cancellations starved it |
| Socket | Two readers on one stream agreed only by accident of how the server replies |
The pattern behind almost all of them: a hazard that was understood and guarded at the site where it was first observed, rather than at the abstraction that owned it. The guard then never travelled to its siblings. Four separate hand-written copies of the same UDT envelope, three of them wrong. Six hand-rolled copies of the same statement-release cleanup, two missing entirely.
Two of the fixes came from asking the server instead of guessing. `statement_type` and `statement_id` had both been parsed out of every DESCRIBE response into a metadata dict that nothing read, while the code that needed them inferred the answer from the first word of the SQL and got it wrong five ways.
The cheapest fix was also the most valuable. Asserting that a row decoder lands exactly on the end of its payload costs one integer comparison, and it would have caught all fourteen framing bugs at the byte where each happened rather than three releases later in somebody's result set.
Test count went from 241 to 457 across three server versions over this stretch.
### 2026.09.03
Two more, one of each kind.
The `:N` placeholder rewriter was a bare regular expression, so it
rewrote the inside of string literals: `'http://host:8080/x'` was stored
as `'http://host?/x'`. Same family as the statement-classification bug
above, and the last place the driver inferred meaning from SQL text
instead of handling it. Replaced with a scanner using Informix's own
lexical rules, measured rather than assumed. One of those rules would
have been got wrong from habit: a backslash escapes nothing, so `'a\'b'`
is an unterminated string and draws `-282`. A scanner written to
Postgres reflexes desyncs there and corrupts everything after it.
`row_factory=Row` adds column-name and attribute access, on request from
a user coming from `pyodbc`. Opt-in, because it costs about 9% on bulk
fetch and that is not a trade to make for someone who never reads a
column by name.
Testing the second one found the pattern again, this time in code
written an hour earlier. The names a column could shadow were
hand-listed as `count` and `index`, the two `tuple` methods; then `keys`,
`_asdict` and `_fields` were added to the same class and the list was
never revisited. A column called `keys` returned a bound method, in
silence. The fix was to compute the set from the class rather than
lengthen the list, and to parametrize the test over that computed set so
it grows with the class.
The interesting part is where testing found it. A differential test
across twenty columns covering every awkward type, populated and fully
NULL, found nothing. The bug was in six lines of name bookkeeping that
looked obviously correct, and had been correct when written. Anything
that must stay in sync with a class by hand eventually will not.
## Notable architectural pivots
The decision log calls out four moments where the obvious choice would have been wrong:
1. **Phase 10/11**: abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
2. **Phase 16**: thread-pool async instead of a full async refactor. ~88% less code, same FastAPI surface.
3. **Phase 27**: adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
4. **Phase 39**: buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
## What's next
The roadmap (loose, not committed):
- **Phase 40+ (codec)**: Numpy-backed bulk decode for homogeneous columns. ~5× speedup target on analytical workloads.
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python", so it would sit behind a build flag.
- **Phase 5x (API)**: Native `callproc` with named parameters, IBM-specific scrollable cursor extensions for full IfxPy parity.
The phase log is updated as work lands. The repo's [`CHANGELOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md) is the source of truth for shipped changes.

63
docs-site/src/content/docs/explain/pure-python.md Normal file
View File

@ -0,0 +1,63 @@
---
title: Pure-Python tradeoffs
description: What pure-Python costs, what it pays for, and where the actual performance ceiling sits.
sidebar:
order: 5
---
The premise of this driver is *zero native code in the call stack*. No CSDK, no JDBC, no C extension we maintain ourselves. Just `socket`, `struct`, `decimal`, and the standard library.
This page is about what that costs and what it pays for.
## What pure-Python costs
The honest accounting:
| Cost | Magnitude | Mitigation |
|---|---|---|
| Per-row decode overhead | ~2.0 µs/row vs IfxPy's ~1.1 µs/row | Phases 37–38 codec inlining brought us from 4 µs to 2 µs. |
| Per-PDU parser overhead | ~5–10 µs vs C's ~1 µs | Phase 39 buffered reader removed the worst of it (the read-side wrapper cost). |
| GIL contention on multi-threaded decode | Threads serialize through codec hot loops | Pool gives one connection per thread; codec releases GIL during I/O. |
| Memory per connection | ~50–500 KB (Phase 39 buffer) | Pool keeps it bounded; freed on connection close. |
The order-of-magnitude intuition: pure-Python is ~2× slower than C-bound for **codec-bound workloads** (large analytical fetches), and **competitive or faster** for I/O-bound workloads (transactional, bulk-insert, FastAPI request-response).
## What pure-Python pays for
The benefits are mostly deployment, not performance:
- **50 KB wheel**: installable in a slim Docker image without a build toolchain.
- **No `libcrypt.so.1`**: works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
- **Python 3.10–3.14**: no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
- **Type annotations everywhere**: `py.typed` flag, full coverage in mypy / pyright.
- **Auditable codepaths**: every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
- **Async without `run_in_executor`**: native `async def` API, FastAPI-compatible.
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1`: those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
## Where the ceiling sits
For codec-bound workloads, the ceiling we've hit is around 2 µs/row for tabular data. The breakdown:
- `struct.unpack_from(format, buf, offset)` per field: ~80 ns
- `bytes` → `str` decoding (varchar): ~150 ns
- Tuple construction: ~100 ns
- Cursor / ResultSet bookkeeping: ~50 ns
Five fields × ~250 ns/field + ~250 ns overhead = ~1.5 µs. We're at ~2.0 µs which means ~30% overhead remains. That's the gap between "we've inlined everything that's reasonable" and "the C version still wins."
Strategies for closing further:
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim, because there'd be a build step.
- **Bytecode optimization via `exec()`-codegen** (the Phase 38 approach). Marginal further wins; we've already extracted most of what's available.
- **Numpy-backed bulk decode** for homogeneous columns. Promising for analytical workloads. ~5× speedup possible for `SELECT col FROM huge_table` over the current per-row approach. Probably Phase 41+.
For I/O-bound workloads we're already at the ceiling. The buffered reader closed the I/O gap; further wins are at the kernel level (e.g. `recvmsg` for vectored reads), which is moot since the kernel already isn't the bottleneck.
## The honest summary
Pure-Python costs us ~5–15% on bulk-fetch workloads and zero (or favorable) on everything else. The deployment, async, and modern-Python wins are large and don't depend on workload.
If the codec gap matters for your case, meaning analytical reporting against a wide table or pulling millions of rows in a single SELECT, IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
The driver chose the goal, *first pure-socket Informix driver in any language*, over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.

84
docs-site/src/content/docs/explain/sqli-protocol.mdx Normal file
View File

@ -0,0 +1,84 @@
---
title: The SQLI wire protocol
description: A short tour of Informix's SQLI protocol, covering PDU framing, the handshake, statement execution, and fetch.
sidebar:
order: 1
---
import { Aside } from '@astrojs/starlight/components';
SQLI is Informix's wire protocol, the same one IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
This page is a short tour. The byte-level reference (with hex annotations) lives in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) in the repo.
## PDU framing
Every PDU starts with a 1-byte tag and (mostly) ends with a 2-byte EOT marker. Common tags:
| Tag | Hex | Direction | Purpose |
|---|---|---|---|
| `SQ_INFO` | `0x01` | →S | Initial capability/identity exchange |
| `SQ_VERSION` | `0x14` | S→ | Server version response |
| `SQ_PASSWD` | `0x18` | →S | Authentication |
| `SQ_DBOPEN` | `0x24` | →S | Open database |
| `SQ_PREPARE` | `0x02` | →S | Prepare statement |
| `SQ_DESCRIBE` | `0x08` | →S | Describe column structure |
| `SQ_OPEN` | `0x06` | →S | Open cursor |
| `SQ_FETCH` | `0x04` | →S | Fetch rows |
| `SQ_TUPLE` | `0x09` | S→ | One row of data |
| `SQ_ID` | `0x37` | S→ | SQLCODE / status code |
| `SQ_FILE` | `0x62` | both | Smart-LOB transfer |
| `SQ_EOT` | `0x0c` | both | End-of-transmission |
The trailing `00 0c` (length=0, tag=0x0c) marks the end of every multi-PDU response.
## A connect, in bytes
Annotated output from `docs/CAPTURES/01-connect-only.socat.log`:
```text
> OUT 01 c3 01 3c 00 00 00 64 00 65 00 00 00 3d 00 06 ; SQ_INFO
49 45 45 45 4d 00 00 6c 73 71 6c 65 78 65 63 00 ; "IEEEM..lsqlexec"
00 00 00 00 00 00 06 39 2e 32 38 30 00 00 0c 52 ; ".......9.280...R"
44 53 23 52 30 30 30 30 30 30 00 00 05 73 71 6c ; "DS#R000000...sql"
69 00 00 00 01 3c 00 00 00 00 00 00 00 00 00 01 ; "i....<.........."
...
< IN 01 14 02 3c 10 00 00 64 00 65 00 00 00 3d 00 06 ; SQ_VERSION
49 45 45 45 49 00 00 6c 73 72 76 69 6e 66 78 00 ; "IEEEI..lsrvinfx."
00 00 00 00 00 00 2f 49 42 4d 20 49 6e 66 6f 72 ; "....../IBM Infor"
6d 69 78 20 44 79 6e 61 6d 69 63 20 53 65 72 76 ; "mix Dynamic Serv"
65 72 20 56 65 72 73 69 6f 6e 20 31 35 2e 30 2e ; "er Version 15.0."
```
The payload of `SQ_INFO` is a sequence of length-prefixed strings: byte-order marker (`IEEEM` = big-endian), client app name (`sqlexec`), client version (`9.280`), build ID, protocol token (`sqli`), feature flags. The server's `SQ_VERSION` response mirrors this with the server's own identification.
## Statement execution
The full lifecycle for `SELECT id FROM users WHERE id = ?`:
```text
→ SQ_PREPARE "SELECT id FROM users WHERE id = ?"
← SQ_ID (statement ID, parameter shape, ...)
→ SQ_DESCRIBE
← SQ_DESC (column descriptors: "id" SMALLINT)
→ SQ_OPEN (parameter values: 42)
← SQ_ID (cursor ID)
→ SQ_FETCH
← SQ_TUPLE (id=42)
← SQ_TUPLE (or SQ_DONE)
→ SQ_CLOSE (release cursor)
← SQ_ID
```
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts. See [Bulk inserts](/how-to/executemany/).
## Smart-LOB transfer
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side, with no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
This was the architectural pivot in [Phase 10/11](/explain/phase-log/) that made smart-LOBs work end-to-end in pure Python. Reading and writing GB-sized BLOBs goes through the same socket as any other query.
<Aside type="note">
The protocol has many more PDU types than this page covers, mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
</Aside>

93
docs-site/src/content/docs/how-to/async-fastapi.mdx Normal file
View File

@ -0,0 +1,93 @@
---
title: Async with FastAPI
description: Wire the async pool into a FastAPI app with proper lifecycle management.
sidebar:
order: 3
---
import { Aside } from '@astrojs/starlight/components';
`informix-driver` has a native async API. Use it from FastAPI by creating the pool in the app's lifespan and dependency-injecting connections per request.
## App skeleton
```python
from contextlib import asynccontextmanager
from fastapi import FastAPI, Depends, HTTPException
from informix_db import aio
@asynccontextmanager
async def lifespan(app: FastAPI):
app.state.pool = await aio.create_pool(
host="db.example.com",
user="informix", password="...",
database="mydb", server="informix",
min_size=2, max_size=20,
)
yield
await app.state.pool.close()
app = FastAPI(lifespan=lifespan)
async def get_conn():
async with app.state.pool.connection() as conn:
yield conn
@app.get("/users/{user_id}")
async def get_user(user_id: int, conn = Depends(get_conn)):
cur = await conn.cursor()
await cur.execute(
"SELECT id, name, email FROM users WHERE id = ?",
(user_id,),
)
row = await cur.fetchone()
if row is None:
raise HTTPException(404, "user not found")
return {"id": row[0], "name": row[1], "email": row[2]}
```
## Why this shape
- **Lifespan-scoped pool**: the pool lives for the lifetime of the app, login handshake amortized across all requests.
- **Per-request connection via `Depends`**: each request gets its own connection from the pool. The async generator pattern (`yield conn`) means the connection returns to the pool when the request finishes, including on exception.
- **No `run_in_executor`**: every call is `await`able natively. No event-loop blocking, no thread-pool tuning.
## Cancellation
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe: the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean, with transactions rolled back. You don't need to wrap anything in `try/finally`.
<Aside type="note">
This is a Phase 27 invariant: async cancellation cannot leak running workers onto recycled connections. The earlier behavior was a `High` audit finding; the fix is a CI tripwire test that's been green every commit since.
</Aside>
## Connection-level transactions
For request-scoped transactions, commit on the way out and roll back on any exception:
```python
@app.post("/orders")
async def create_order(order: OrderIn, conn = Depends(get_conn)):
cur = await conn.cursor()
try:
await cur.execute(
"INSERT INTO orders VALUES (?, ?, ?)",
(order.id, order.customer_id, order.total),
)
await cur.execute(
"UPDATE inventory SET qty = qty - ? WHERE sku = ?",
(order.qty, order.sku),
)
await conn.commit()
except Exception:
await conn.rollback()
raise
return {"ok": True}
```
`HTTPException` is an `Exception`, so raising one inside the block rolls back before FastAPI turns it into a response.
<Aside type="note">
There is no `conn.transaction()` context manager on either the sync or async connection, though an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
</Aside>

48
docs-site/src/content/docs/how-to/buffered-reader.md Normal file
View File

@ -0,0 +1,48 @@
---
title: Optimize bulk SELECT
description: How the buffered reader works in practice, when it's on, when it isn't, and how to A/B-measure your workload.
sidebar:
order: 5
---
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything, since the bulk-fetch gap against IfxPy is now ~5–15% rather than ~140%.
For the architectural rationale, see [The buffered reader →](/explain/buffered-reader/).
## Disabling the buffered reader
```bash
IFX_BUFFERED_READER=0 python my_app.py
```
The flag is read once at connection construction. To flip behavior on existing connections, close and reopen the pool.
There's no expected reason to disable it in production. The flag exists so you can A/B-measure your own workload and so we can debug regressions if they appear.
## A/B-measuring your workload
```bash
# Baseline: no buffered reader
IFX_BUFFERED_READER=0 python -m mybench
# With buffered reader
IFX_BUFFERED_READER=1 python -m mybench
```
For typical bulk-SELECT workloads expect a 30–40% wall-time reduction. For workloads dominated by single-row queries the impact is small (small queries are RTT-bound, not framing-bound).
## When the speedup is largest
Workloads where every column read makes ~4–5 small `recv()` calls, meaning tabular data, narrow rows, and large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
| Workload shape | Speedup |
|---|---:|
| Wide row, single fetch (1 row × 100 cols) | minimal |
| Narrow row, large fetch (100k rows × 5 cols) | 30–40% |
| `executemany` response drain (1k inserts) | 25–30% |
## Memory profile
The buffer is per-connection, sized to grow up to the largest single PDU it sees. Typical steady-state: 64 KB to a few hundred KB per connection. The buffer is freed when the connection closes; for long-lived pool connections it's amortized.
If you're running 10,000 connections at idle, the buffer cost is ~1–2 GB across the fleet. For typical pool sizes (10–50 connections) it's ~1–10 MB total.

93
docs-site/src/content/docs/how-to/dev-container.mdx Normal file
View File

@ -0,0 +1,93 @@
---
title: Run the dev container
description: IBM Informix Developer Edition in Docker, covering first-time setup, sbspace for smart-LOBs, and common troubleshooting.
sidebar:
order: 8
---
import { Aside } from '@astrojs/starlight/components';
The IBM Informix Developer Edition Docker image is the recommended dev / integration-test target. It's the same image our CI runs against.
## First-time setup
```bash
docker run -d --name informix-dev \
-e LICENSE=accept \
-p 9088:9088 \
-p 9089:9089 \
--privileged \
icr.io/informix/informix-developer-database:15.0.1.0.3DE
```
- **`9088`**: clear-text SQLI listener
- **`9089`**: TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
- **`--privileged`**: required for the dev image's shared-memory tuning
The image takes ~90 seconds to initialize. Watch for `oninit running`:
```bash
docker logs -f informix-dev
```
## Default credentials
| Field | Value |
|---|---|
| `user` | `informix` |
| `password` | `in4mix` |
| `database` | `sysmaster` (always exists) |
| `server` | `informix` |
For application use create your own database:
```bash
docker exec -it informix-dev bash -c '
echo "CREATE DATABASE app_db WITH LOG;" | dbaccess sysmaster
'
```
## Setup for smart-LOB tests
Smart-LOBs (BLOB / CLOB) require additional one-time setup:
```bash
# Inside the container
docker exec -it informix-dev su - informix -c '
onspaces -c -S sbspace1 -p $INFORMIXDIR/sbspace1 -o 0 -s 100000
# Edit $ONCONFIG to set SBSPACENAME sbspace1, then:
onmode -ky
oninit -y
# Take a level-0 archive so the sbspace is usable
ontape -s -L 0
'
```
After that, BLOBs and CLOBs work end-to-end. See [`docs/DECISION_LOG.md` §10](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) for the gory details.
## Running the integration tests
```bash
make ifx-up # starts the container if not already running
make test-integration # runs the 231 integration tests
```
Or directly:
```bash
pytest -m integration
```
## Troubleshooting
**"Connection refused" on 9088**: the image is still initializing. Wait for `oninit running` in the logs.
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing, since it always exists.
**`-908` (system error / shared memory)**: the container needs `--privileged`. Restart with that flag.
<Aside type="tip">
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart, and the database survives container shutdown.
</Aside>

107
docs-site/src/content/docs/how-to/executemany.mdx Normal file
View File

@ -0,0 +1,107 @@
---
title: Bulk inserts (executemany)
description: How to bulk-load with executemany, including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
sidebar:
order: 4
---
import { Aside } from '@astrojs/starlight/components';
`executemany()` is the right tool for bulk inserts and updates. With `informix-driver`'s pipelined implementation it's **1.6× faster than IfxPy** at 10k+ rows.
## The basic shape
```python
rows = [(1, "alice"), (2, "bob"), (3, "carol"), ...] # 10_000 tuples
with conn: # opens a transaction
cur = conn.cursor()
cur.executemany(
"INSERT INTO users (id, name) VALUES (?, ?)",
rows,
) # commits on normal exit
```
That inserts 10,000 rows in ~161 ms against a loopback Informix container.
## The 53× gotcha
<Aside type="caution">
**`executemany(...)` under `autocommit=True` is 53× slower than the same call inside an explicit transaction.**
The server flushes the transaction log to disk per row in autocommit mode. With 10,000 single-row autocommit inserts that's 10,000 log flushes. Inside one transaction it's one flush at commit.
</Aside>
```python
# SLOW: 8.5 seconds for 10k rows
conn = informix_db.connect(..., autocommit=True)
cur = conn.cursor()
cur.executemany("INSERT ...", rows)
# FAST: 161 ms for 10k rows
conn = informix_db.connect(..., autocommit=False) # default
with conn:
cur = conn.cursor()
cur.executemany("INSERT ...", rows)
```
The default is `autocommit=False`, so this only catches you if you've explicitly opted into autocommit.
## Why it's faster than IfxPy
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row, so for 10,000 rows on an 80 µs RTT it adds up to 800 ms of just waiting for ACKs.
Phase 33 changed our `executemany` to **pipeline** the BIND+EXECUTE PDUs:
1. Send all N BIND PDUs back-to-back without reading responses
2. Send all N EXECUTE PDUs back-to-back
3. Drain N response sets at the end
The kernel buffers the outbound bytes; the server processes the BIND/EXECUTE pipeline as fast as it can; we read all responses at the end. One RTT for the whole batch instead of N.
## Chunking large batches
For very large batches (millions of rows), break into chunks to bound memory:
```python
def chunks(it, n):
buf = []
for x in it:
buf.append(x)
if len(buf) >= n:
yield buf
buf = []
if buf:
yield buf
with conn:
cur = conn.cursor()
for batch in chunks(huge_iterator, 10_000):
cur.executemany("INSERT INTO logs ...", batch)
# one transaction, many batched executemany calls, one commit at the end
```
10,000 rows per chunk is a reasonable default; the per-chunk Python memory cost is `~ N × bytes_per_row`. For 10k tuples of 5 small fields that's a few MB.
## Returning generated keys
Informix's `SERIAL` columns are server-assigned, and `executemany` doesn't return per-row IDs. To get an ID back, insert one row at a time and ask the server for the value it assigned:
```python
cur.execute("INSERT INTO orders (customer_id, total) VALUES (?, ?)", (7, 42.0))
cur.execute("SELECT DBINFO('sqlca.sqlerrd1') FROM systables WHERE tabid = 1")
new_id = cur.fetchone()[0]
```
`sqlca.sqlerrd1` is where Informix records the serial value from the most recent INSERT on this connection, so read it before running any other statement.
There is no `cursor.lastrowid`, though an earlier version of this page said there was.
For batch inserts that need the IDs, the idiomatic pattern is:
```sql
INSERT INTO orders (id, customer_id, total)
SELECT MAX(id) + ROWNUM, ?, ? FROM orders
```
…or pre-generate IDs from a sequence in your app code. Or accept `IfxPy`'s same limitation: `IfxPy.executemany` doesn't return per-row generated keys either.

98
docs-site/src/content/docs/how-to/migrate-from-ifxpy.md Normal file
View File

@ -0,0 +1,98 @@
---
title: Migrate from IfxPy
description: API differences between IfxPy and informix-driver, what's the same, what's not, and how to migrate incrementally.
sidebar:
order: 7
---
If you have working IfxPy code, migration is mostly mechanical. Both drivers are PEP 249 with similar shapes; the differences are in connection construction, a few cursor extensions, and async support.
## Connection construction
```python
# IfxPy
import IfxPy
conn_str = (
"DATABASE=mydb;HOSTNAME=db.example.com;PORT=9088;"
"PROTOCOL=onsoctcp;UID=informix;PWD=...;SERVICE=informix"
)
conn = IfxPy.connect(conn_str, "", "")
# informix-driver
import informix_db
conn = informix_db.connect(
host="db.example.com", port=9088,
user="informix", password="...",
database="mydb", server="informix",
)
```
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported, deliberately: they're a security and parsing footgun.
## The DB-API surface is the same
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, and `close()` all behave per PEP 249.
The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `DatabaseError`, `DataError`, `OperationalError`, `IntegrityError`, `InternalError`, `ProgrammingError`, `NotSupportedError`.
## What IfxPy has that we don't (yet)
- **Named-parameter `callproc`**. We have `fast_path_call` for direct UDF/SPL invocation but the API shape differs.
- **IBM-specific scrollable cursor extensions**. We support PEP 249 scrollable cursors (`scroll(value, mode)`) but not IfxPy's `last`/`prior`/`relative` shortcuts.
- **`cursor.set_chunk_size`**. We tune fetch behavior via the buffered reader; no per-cursor knob.
## What we have that IfxPy doesn't
- **Async API** (`from informix_db import aio`)
- **Connection pool** (`informix_db.create_pool` / `aio.create_pool`)
- **Type-safe annotations**: `informix-driver` ships with `py.typed`
- **Python 3.12+ support**
- **Pipelined `executemany`**: 1.6× faster than IfxPy's per-row implementation
- **Rows addressable by name**: `row["col"]` and `row.col`, not just `row[0]`
## Reading rows by name
IfxPy gives you positional access and nothing else, which is why most
IfxPy codebases grow a helper like this:
```python
cols = [c[0].lower() for c in cur.description]
row_dict = dict(zip(cols, cur.fetchone()))
```
That gets you a dict and still no attribute access. Ask for `Row`
instead and all three work at once:
```python
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur.execute("SELECT config_key, config_value FROM settings")
row = cur.fetchone()
row[0], row["config_key"], row.config_key
```
Set on the connection, so it applies to every cursor from it. The
`.lower()` in the hand-written version is already a no-op, incidentally:
Informix folds unquoted identifiers to lower case.
It is opt-in because it costs about 9% on bulk fetch. See
[the API reference](/reference/api/#rows) for the numbers and the
edge cases.
## Migrating incrementally
If your codebase has hundreds of IfxPy call sites, you can do a partial migration:
1. Replace connection construction with `informix-driver` at the application boundary.
2. Where you used `IfxPy.fetch_assoc` (returns dict), wrap our cursor with a small adapter:
```python
def fetch_assoc(cur):
row = cur.fetchone()
if row is None:
return False
return dict(zip([c[0] for c in cur.description], row))
```
3. For `IfxPy.exec_immediate`, use `cur.execute(sql)` with no params.
4. For `IfxPy.bind_param`, use the params arg of `execute()`: `cur.execute(sql, (a, b, c))`.
Most application code can be migrated with `sed`-level transformations.

71
docs-site/src/content/docs/how-to/pool.mdx Normal file
View File

@ -0,0 +1,71 @@
---
title: Use the connection pool
description: Sync and async connection pools, covering sizing, timeouts, lifecycle, and threading.
sidebar:
order: 2
---
import { Aside } from '@astrojs/starlight/components';
The connection pool amortizes the ~11 ms login handshake across many queries and gives you a thread-safe / task-safe acquire-release API. Use it any time the same process makes more than a handful of queries.
## Sync pool
```python
import informix_db
pool = informix_db.create_pool(
host="db.example.com", port=9088,
user="informix", password="...",
database="mydb", server="informix",
min_size=2, # warm up at least 2 connections at create time
max_size=10, # hard cap; acquires beyond this block
acquire_timeout=5.0, # raise PoolTimeout if no connection in 5s
max_idle=600.0, # close connections idle longer than 10 min
)
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT id, name FROM users WHERE id = ?", (42,))
print(cur.fetchone())
pool.close()
```
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically, so you never see a dirty connection from `pool.connection()`.
## Async pool
```python
import asyncio
from informix_db import aio
async def main():
pool = await aio.create_pool(
host="db.example.com",
user="informix", password="...",
database="mydb",
min_size=2, max_size=10,
)
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
print(await cur.fetchone())
await pool.close()
asyncio.run(main())
```
Same semantics, `async`-aware. Cancellation is safe here too: a cancelled task does not leak an in-flight worker onto a recycled connection.
## Sizing
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes", usually 8–16 for typical web/API services.
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`**. The server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
## Threading
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally, since `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
Phase 27 added a per-connection wire lock that makes accidental sharing safe (interleaved PDUs serialize correctly), but you should still give each thread its own connection. The lock is a backstop, not a license.

83
docs-site/src/content/docs/how-to/smart-lobs.mdx Normal file
View File

@ -0,0 +1,83 @@
---
title: BLOB / CLOB read & write
description: Reading and writing smart-LOB columns end-to-end in pure Python.
sidebar:
order: 6
---
import { Aside } from '@astrojs/starlight/components';
`informix-driver` reads and writes smart-LOB columns (BLOB and CLOB) end-to-end without any native machinery. The implementation uses Informix's `lotofile` and `filetoblob` SQL functions, intercepted at the `SQ_FILE` (98) wire-protocol level.
## Reading a BLOB
```python
data: bytes = cur.read_blob_column(
"SELECT data FROM photos WHERE id = ?",
(42,),
)
```
`read_blob_column` returns the raw bytes. For very large BLOBs (multi-GB), see the streaming variant below.
## Writing a BLOB
```python
cur.write_blob_column(
"INSERT INTO photos VALUES (?, BLOB_PLACEHOLDER)",
blob_data=jpeg_bytes,
params=(42,),
)
```
The `BLOB_PLACEHOLDER` token in the SQL marks where the BLOB data goes. Other parameters are bound positionally to `params=`.
## Reading a CLOB
CLOBs use the same methods as BLOBs, and there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
```python
raw: bytes = cur.read_blob_column(
"SELECT body FROM articles WHERE id = ?",
(42,),
)
text: str = raw.decode("iso-8859-1") # or your DB_LOCALE's codec
```
Returning `bytes` rather than `str` is deliberate: the driver transports the LOB over the `SQ_FILE` channel and never sees the column's declared character set, so guessing an encoding here would be exactly the kind of silent assumption that produces mojibake in one deployment and works fine in another.
## Writing a CLOB
Pass `clob=True` so the write routes through `filetoclob` rather than `filetoblob`, and encode the text yourself:
```python
cur.write_blob_column(
"INSERT INTO articles VALUES (?, BLOB_PLACEHOLDER)",
"long article text… café résumé".encode("iso-8859-1"),
(42,),
clob=True,
)
```
The placeholder token is `BLOB_PLACEHOLDER` for both BLOB and CLOB writes.
## Server-side prerequisites
<Aside type="caution">
Smart-LOBs require server-side configuration that the IBM Developer Edition Docker image doesn't ship with by default:
- An **`sbspace`** must be created (`onspaces -c -S sbspace1 -p ...`)
- `SBSPACENAME` must be set in `$ONCONFIG`
- A **level-0 archive** must be taken (`ontape -s -L 0`) before BLOBs can be created
- The database must be created **with logging** (`CREATE DATABASE foo WITH LOG`)
Full setup commands are in [`docs/DECISION_LOG.md` §10](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md).
</Aside>
## How it works (briefly)
The `lotofile` server function returns a smart-LOB descriptor as a regular result column when called via `SELECT`. The driver intercepts the `SQ_FILE` (PDU 98) response that contains the LOB bytes and reassembles them client-side.
Writing reverses the flow: `filetoblob` is invoked via a placeholder pattern in the SQL, the driver sends the bytes via `SQ_FILE` PDUs, and the server stores them in the sbspace.
The architectural pivot from the heavier `SQ_FPROUTINE` + `SQ_LODATA` stack to this lighter `SQ_FILE` intercept is documented in [Phase 10/11 of the decision log](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md). The result is roughly 3× smaller than originally projected.

55
docs-site/src/content/docs/how-to/tls.mdx Normal file
View File

@ -0,0 +1,55 @@
---
title: Connect with TLS
description: TLS-listener configuration, bring-your-own SSL context, and dev-mode self-signed handling.
sidebar:
order: 1
---
import { Aside } from '@astrojs/starlight/components';
Informix uses **dedicated TLS-enabled listener ports** (configured server-side in `sqlhosts`) rather than STARTTLS upgrade. Point `port` at the TLS listener (typically `9089`) when `tls` is enabled.
## Production: bring your own SSL context
```python
import ssl
import informix_db
ctx = ssl.create_default_context(cafile="/path/to/ca.pem")
# Optional: client cert auth
# ctx.load_cert_chain(certfile="/path/to/client.pem", keyfile="/path/to/client.key")
conn = informix_db.connect(
host="db.example.com",
port=9089,
user="informix",
password="...",
database="mydb",
server="informix",
tls=ctx,
)
```
Bring-your-own context is the recommended production pattern, giving you full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
## Dev / self-signed: tls=True
```python
informix_db.connect(host="127.0.0.1", port=9089, ..., tls=True)
```
`tls=True` is a convenience for development. It builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
## Server-side configuration
The Informix server needs a TLS listener entry in `sqlhosts`:
```
informix_tls onsoctcp myhost 9089
```
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089`, with no configuration needed.
<Aside type="tip">
If your connection hangs at the handshake, you've probably pointed at the non-TLS port (`9088` instead of `9089`). The non-TLS listener will accept the TCP connection but won't speak TLS, so the handshake stalls.
</Aside>

92
docs-site/src/content/docs/index.mdx Normal file
View File

@ -0,0 +1,92 @@
---
title: informix-driver
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket, with no CSDK, no JVM, and no native libraries.
template: splash
hero:
tagline: ''
editUrl: false
lastUpdated: false
next: false
prev: false
---
import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
<div class="ifx-features">
<div class="ifx-feature">
<Icon name="rocket" class="ifx-feature__icon" />
<h3>1.6× faster bulk inserts than IfxPy</h3>
<p>Pipelined <code>executemany</code> sends every BIND+EXECUTE PDU before draining responses. IBM's C driver still pays one round-trip per row. We figured we could do better.</p>
</div>
<div class="ifx-feature">
<Icon name="seti:python" class="ifx-feature__icon" />
<h3>~10% behind IfxPy on bulk fetches</h3>
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics, for now.</p>
</div>
<div class="ifx-feature">
<Icon name="puzzle" class="ifx-feature__icon" />
<h3>50 KB wheel. Zero native deps.</h3>
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.10–3.14, including the versions IfxPy doesn't.</p>
</div>
<div class="ifx-feature">
<Icon name="sun" class="ifx-feature__icon" />
<h3>Async, like a modern driver should be</h3>
<p>FastAPI, aiohttp, asyncio. Pool, connections, cursors all have <code>async def</code> versions. IfxPy has none of this.</p>
</div>
<div class="ifx-feature">
<Icon name="approve-check" class="ifx-feature__icon" />
<h3>PEP 249, no surprises</h3>
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy, plus threadsafe sharing through a per-connection wire lock.</p>
</div>
<div class="ifx-feature">
<Icon name="document" class="ifx-feature__icon" />
<h3>Clean-room reverse engineering</h3>
<p>Decompiled IBM JDBC. Annotated <code>socat</code> captures. Differential testing against IfxPy on every codec path. Every architectural decision lives in <a href="/explain/phase-log/">the phase log</a>. Receipts.</p>
</div>
</div>
## A query, end to end
```python
import informix_db
with informix_db.connect(
host="db.example.com", port=9088,
user="informix", password="...",
database="mydb", server="informix",
) as conn:
cur = conn.cursor()
cur.execute("SELECT id, name FROM users WHERE id = ?", (42,))
user_id, name = cur.fetchone()
```
That's it. No `IBM_DB_HOME`. No DSN file. No `libcrypt.so.1`.
## WTF did you build this for?
The existing tools were not my style.
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That means all of them: `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9). It needs four `LD_LIBRARY_PATH` entries. It needs `setuptools < 58`. And IfxPy itself is broken on Python 3.12+. For containerized deployments, ETL pipelines, FastAPI services, or anywhere a build toolchain on the runtime is friction, this driver is the alternative that didn't previously exist. Now it does.
## Read next
<CardGrid>
<Card title="Install & first query" icon="rocket">
Up and running with the IBM Informix Developer Edition Docker image in five minutes.
[Get started →](/start/quickstart/)
</Card>
<Card title="Compared to IfxPy" icon="random">
Head-to-head benchmarks, install gauntlet, and when each driver wins.
[Read →](/start/vs-ifxpy/)
</Card>
<Card title="The buffered reader" icon="information">
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1×, and the architectural mistake the first pass got wrong.
[Read →](/explain/buffered-reader/)
</Card>
<Card title="Architecture" icon="puzzle">
SQLI on the wire. Sockets, framing, codec, resultsets. How the layers stack.
[Read →](/explain/architecture/)
</Card>
</CardGrid>

214
docs-site/src/content/docs/reference/api.md Normal file
View File

@ -0,0 +1,214 @@
---
title: API surface
description: Module-level functions, Connection / Cursor / Pool APIs, async equivalents.
sidebar:
order: 1
---
The top-level surface of `informix_db` and `informix_db.aio`. For full PEP 249 details, see the standard's [DB-API 2.0 spec](https://peps.python.org/pep-0249/).
## Module-level
```python
import informix_db
informix_db.connect(...) -> Connection
informix_db.create_pool(...) -> Pool
informix_db.apilevel # "2.0"
informix_db.threadsafety # 1
informix_db.paramstyle # "numeric"
```
```python
from informix_db import aio
await aio.connect(...) -> aio.Connection
await aio.create_pool(...) -> aio.Pool
```
## connect()
```python
informix_db.connect(
*,
host: str,
port: int = 9088,
user: str,
password: str,
database: str | None,
server: str, # DBSERVERNAME (not hostname)
autocommit: bool = False,
connect_timeout: float | None = None,
read_timeout: float | None = None,
keepalive: bool = False,
client_locale: str = "en_US.8859-1",
env: dict[str, str] | None = None,
tls: bool | ssl.SSLContext = False,
) -> Connection
```
## Connection
| Method / property | Description |
|---|---|
| `cursor(scrollable=False)` | Returns a new `Cursor`. Only one scrollable cursor may be open per connection; see below. |
| `commit()` | Commits the current transaction. |
| `rollback()` | Rolls back the current transaction. |
| `close()` | Closes the connection. Idempotent. |
| `fast_path_call(routine, *args)` | Direct UDF/SPL invocation, bypassing PREPARE/EXECUTE/FETCH. |
| `encoding` | Resolved Python codec for `client_locale`. |
| `closed` | `True` after `close()`. |
| `server_version` | Server release, e.g. `'…Version 12.10.FC6'`. Costs one query on first access, then cached. |
| `server_version_internal` | Raw login-response version string. Free. See note below. |
| `server_capabilities` | Negotiated `ServerCapabilities`, or `None` if the reply couldn't be decoded. |
`Connection` is also a context manager (`with informix_db.connect(...) as conn:`), which closes on exit.
There is no `conn.transaction()` helper and no `conn.autocommit` attribute. Earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
```python
conn = informix_db.connect(..., autocommit=False)
try:
cur = conn.cursor()
cur.execute("UPDATE accounts SET balance = balance - ? WHERE id = ?", (100, 1))
cur.execute("UPDATE accounts SET balance = balance + ? WHERE id = ?", (100, 2))
conn.commit()
except Exception:
conn.rollback()
raise
```
:::caution[One statement per session]
Informix gives a session a single statement slot. A non-scrollable cursor never trips over this, because it materializes its rows and releases the statement before `execute()` returns. A **scrollable** cursor holds the slot open on purpose, so while one is open, any other statement on that same connection raises `ProgrammingError`:
```python
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT id, body FROM big_table ORDER BY id")
scroll.fetch_first()
other = conn.cursor()
other.execute("SELECT COUNT(*) FROM audit") # ProgrammingError
```
Close the scrollable cursor first, or use a second connection for the other query. Re-running `execute()` on the scrollable cursor itself is fine; it closes its own server-side cursor first.
Before `2026.09.02` this was not refused. The server answered the second statement with `-285` and destroyed the scrollable cursor as well, whose next fetch then returned `-267`, "the transaction has been rolled back". Two errors, neither naming the cause.
:::
:::note[Transaction control written as SQL]
`commit()` and `rollback()` are the supported way to end a transaction, but `cursor.execute("BEGIN WORK")` and its `COMMIT` / `ROLLBACK` counterparts work too, and the connection tracks them from `2026.09.02` onward.
That matters because it used not to. Under `autocommit=True`, a SQL `BEGIN WORK` opened a real transaction the connection never learned about, and since `rollback()` is guarded by that state, **it returned successfully having sent nothing**. The rows it was asked to discard survived, and a pooled connection went back into circulation still holding the transaction and its locks.
:::
:::note[Two version numbers, and why]
The login response carries Informix's *internal* protocol version, not the release you installed: 12.10 announces itself as `9.56`, 14.10 as `9.59`, and 15 as `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers, which is why all three speak an identical SQLI dialect.
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')`. That is one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
:::
## Rows
Fetch methods return plain tuples by default. Pass `row_factory` to get rows that also answer to a column name and an attribute, the shape `pyodbc` and `mssql-python` provide:
```python
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur = conn.cursor()
cur.execute("SELECT tabid, tabname FROM systables")
row = cur.fetchone()
row[0] # 1
row["tabname"] # 'systables'
row.tabname # 'systables'
```
Set on the connection, so it applies to every cursor from it. A cursor can override it with `cur.row_factory = informix_db.Row`. Pools and the async API forward it unchanged.
`Row` subclasses `tuple`, so `row == (1, "systables")` is still true and anything treating rows as sequences keeps working. Slices return plain tuples, since a slice has no column map. `row.keys()` gives the column names and `row._asdict()` gives a dict.
:::note[Why it is opt-in]
On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` becomes a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript.
Defaulting it on would move the [1.05-1.15x ratio against IfxPy](/start/vs-ifxpy/) to roughly 1.15-1.25x. That is a good trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is yours to make.
:::
Four behaviours worth knowing:
- Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`. The `.lower()` people add by hand when building this themselves is already a no-op.
- Expression columns get server-generated names like `(count(*))`, which cannot be Python attributes. Those are reachable by subscript only.
- Duplicate column names resolve to the first occurrence, matching `pyodbc`. Positional access still reaches both.
- **A column beats a method of the same name.** `tuple` defines `count` and `index`; `Row` adds `keys`, `_asdict` and `_fields`. A column with any of those names wins, because otherwise it would hand back a bound method instead of a value. The methods stay reachable as `tuple.count(row, x)` and `Row.keys(row)`.
## Cursor
| Method / property | Description |
|---|---|
| `execute(sql, params=())` | Prepare + execute. Returns the cursor. |
| `executemany(sql, seq_of_params)` | Pipelined batch execute. |
| `fetchone()` | One row tuple, or `None`. |
| `fetchmany(size=arraysize)` | List of row tuples. |
| `fetchall()` | All remaining rows. |
| `scroll(value, mode="relative")` | Scrollable cursor positioning. Needs `cursor(scrollable=True)`. |
| `fetch_first()` / `fetch_last()` | Jump to the first / last row. Scrollable cursors only. |
| `fetch_prior()` / `fetch_relative(n)` / `fetch_absolute(n)` | Relative and absolute positioning. Scrollable cursors only. |
| `read_blob_column(sql, params)` | Read a BLOB **or** CLOB column → `bytes`. |
| `write_blob_column(sql, data, params, clob=False)` | Write a BLOB column; pass `clob=True` for CLOB. |
| `close()` | Closes the cursor + releases server resources. |
| `closed` | `True` after `close()`. |
| `description` | Sequence of column descriptors per PEP 249. |
| `rowcount` | Affected row count for DML; `-1` for SELECT. |
| `rownumber` | Current 0-indexed position, or `None` before the first row. |
| `arraysize` | Default `fetchmany()` size. |
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute. Earlier versions of this page listed all three, and none have existed.
CLOBs go through the BLOB methods. `read_blob_column` returns `bytes` either way, so decode it yourself with the column's encoding:
```python
cur.write_blob_column(
"INSERT INTO docs VALUES (?, BLOB_PLACEHOLDER)",
"café résumé".encode("iso-8859-1"),
(1,),
clob=True,
)
raw = cur.read_blob_column("SELECT txt FROM docs WHERE id = ?", (1,))
text = raw.decode("iso-8859-1")
```
For a server-assigned `SERIAL` after an INSERT, ask the server. Informix exposes it through `DBINFO`:
```python
cur.execute("INSERT INTO people (name) VALUES (?)", ("ada",))
cur.execute("SELECT DBINFO('sqlca.sqlerrd1') FROM systables WHERE tabid = 1")
new_id = cur.fetchone()[0]
```
## Pool
| Method | Description |
|---|---|
| `connection()` | Context manager yielding a connection from the pool. |
| `close()` | Closes all idle connections; waits for in-use to return. |
| `closed` | `True` after `close()`. |
| `size` | Current pool size. |
| `idle_count` | Connections currently idle in the pool. |
`aio.Pool` is identical except its `connection()` is an async context manager and `close()` is `async`.
## Exceptions
```
Error
├── Warning
└── DatabaseError
├── InterfaceError
├── DataError
├── OperationalError
│ └── PoolTimeout
├── IntegrityError
├── InternalError
├── ProgrammingError
└── NotSupportedError
```
See [Exceptions & error codes](/reference/exceptions/) for the SQLCODE → exception mapping.

76
docs-site/src/content/docs/reference/benchmarks.mdx Normal file
View File

@ -0,0 +1,76 @@
---
title: Performance baselines
description: Single-connection benchmark results for codec, framing, end-to-end queries, and IfxPy.
sidebar:
order: 5
---
import { Aside } from '@astrojs/starlight/components';
These are the steady-state numbers for the current release (`2026.05.05.12`), measured against the IBM Informix Developer Edition Docker container on loopback.
<Aside type="note">
All numbers are **median over 10+ rounds** (pytest-benchmark `--rounds 10`). Reproduce with `make bench` from the repo root.
</Aside>
## Codec micro-benchmarks
| Operation | Mean | Throughput |
|---|---:|---:|
| `decode(int)` per cell | 139 ns | 7.2M ops/sec |
| `decode(varchar)` per cell | 280 ns | 3.6M ops/sec |
| `decode(decimal)` per cell | 410 ns | 2.4M ops/sec |
| `parse_tuple_payload` per row (5 cols) | 1.4 µs | 715K rows/sec |
## End-to-end
| Operation | Mean | Throughput |
|---|---:|---:|
| `SELECT 1` round-trip | ~140 µs | ~7K queries/sec |
| 1000-row SELECT | ~1.0 ms | ~990K rows/sec sustained |
| `executemany(1000)` in transaction | 32 ms | ~31,000 rows/sec |
| Pool acquire + query + release | 295 µs | ~3.4K queries/sec |
| Cold connect (login handshake) | 11 ms | ~90 connections/sec |
## vs IfxPy 3.0.5
| Benchmark | IfxPy | informix-driver | Result |
|---|---:|---:|---:|
| Single-row SELECT round-trip | 118 µs | 114 µs | comparable |
| ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster |
| Cold connect | 11.0 ms | 10.5 ms | comparable |
| `executemany(1k)` | 23.5 ms | 23.2 ms | tied |
| `executemany(10k)` | 259 ms | **161 ms** | **informix-driver 1.6× faster** |
| `executemany(100k)` | 2376 ms | **1487 ms** | **informix-driver 1.6× faster** |
| `SELECT 1k` | 1.34 ms | 1.72 ms | IfxPy 1.28× |
| `SELECT 10k` | 11.7 ms | 16.1 ms | IfxPy 1.07× |
| `SELECT 100k` | 116 ms | 169 ms | IfxPy 1.15× |
For the methodology, IQR caveats, and reproduction instructions, see [Compared to IfxPy](/start/vs-ifxpy/).
## Phase progression on bulk SELECT
| Phase | 100k-row SELECT | Ratio vs IfxPy |
|---|---:|---:|
| Phase 36 | 280 ms | 2.40× slower |
| Phase 37 (per-column readers) | 250 ms | 2.10× slower |
| Phase 38 (codegen-inlined decoders) | 237 ms | 2.04× slower |
| **Phase 39 (connection-scoped buffered reader)** | **169 ms** | **1.15× slower** |
The Phase 39 jump is documented in [The buffered reader](/explain/buffered-reader/).
## Reproducing
```bash
git clone https://git.supported.systems/warehack.ing/informix-db
cd informix-db
make ifx-up # starts the dev container
make bench # runs all benchmarks
make compare # head-to-head vs IfxPy (handles IfxPy's install gauntlet)
```
For just the bulk-fetch progression:
```bash
pytest -m benchmark tests/benchmarks/test_select_scaling.py
```

67
docs-site/src/content/docs/reference/config.md Normal file
View File

@ -0,0 +1,67 @@
---
title: Configuration & env flags
description: Runtime environment variables, connection arguments, pool tunables.
sidebar:
order: 3
---
## Environment variables
| Variable | Default | Effect |
|---|---|---|
| `IFX_BUFFERED_READER` | `1` | Enable connection-scoped read buffer (Phase 39). Set to `0` to disable. |
| `IFX_DEBUG_WIRE` | unset | When set to a truthy value, log wire-level PDU framing to stderr. Verbose; for debugging only. |
| `IFX_PROTOCOL_TRACE` | unset | When set to a path, write annotated wire captures to the file. |
| `IFX_DISABLE_PIPELINE` | unset | Disables pipelined `executemany` (Phase 33). Use only to A/B-measure. |
| `INFORMIXSERVER` | *(none)* | Read by `connect()` if `server=` is not provided. |
Environment variables are read at connection construction. Changing them at runtime doesn't affect existing connections.
## Connection arguments
Full keyword list for `informix_db.connect()`:
| Arg | Type | Default | Notes |
|---|---|---|---|
| `host` | `str` | required | TCP host. |
| `port` | `int` | `9088` | TCP port. `9089` is the typical TLS listener. |
| `user` | `str` | required | |
| `password` | `str` | required | |
| `database` | `str \| None` | required | `None` logs in without selecting a DB. |
| `server` | `str` | required | DBSERVERNAME, not hostname. |
| `autocommit` | `bool` | `False` | |
| `connect_timeout` | `float \| None` | `None` | TCP+login timeout. |
| `read_timeout` | `float \| None` | `None` | Per-read timeout. |
| `keepalive` | `bool` | `False` | `SO_KEEPALIVE`. |
| `client_locale` | `str` | `"en_US.8859-1"` | See [SQL ↔ Python types](/reference/types/) for codec mapping. |
| `env` | `dict[str, str] \| None` | `None` | Server-side session env. |
| `tls` | `bool \| ssl.SSLContext` | `False` | See [Connect with TLS](/how-to/tls/). |
## Pool tunables
| Arg | Default | Effect |
|---|---|---|
| `min_size` | `1` | Connections to pre-warm at create time. |
| `max_size` | `10` | Hard cap. |
| `acquire_timeout` | `30.0` | Raise `PoolTimeout` after this many seconds. |
| `max_idle` | `600.0` | Close connections idle longer than this (seconds). |
| `health_check` | `True` | Validate idle connections before returning from the pool. |
All other connection arguments are forwarded to each pool-created connection.
## Server-side session env
The `env={}` parameter of `connect()` sets server session variables sent in the login PDU:
```python
informix_db.connect(
...,
env={
"OPT_GOAL": "-1", # optimize for first-row return
"OPTOFC": "1", # auto-free cursors at fetch-close
"IFX_AUTOFREE": "1",
},
)
```
`CLIENT_LOCALE` is set automatically from `client_locale=`, so don't put it in `env=`.

68
docs-site/src/content/docs/reference/exceptions.md Normal file
View File

@ -0,0 +1,68 @@
---
title: Exceptions & error codes
description: PEP 249 exception hierarchy, Informix SQLCODE → Python exception mapping.
sidebar:
order: 4
---
The exception hierarchy is per PEP 249. All exceptions live in `informix_db` and inherit from `informix_db.Error`.
## Hierarchy
```
Error
├── Warning
└── DatabaseError
├── InterfaceError
├── DataError
├── OperationalError
│ └── PoolTimeout
├── IntegrityError
├── InternalError
├── ProgrammingError
└── NotSupportedError
```
## When each is raised
| Exception | Typical cause |
|---|---|
| `InterfaceError` | Misuse of the driver API itself (e.g. fetch on a closed cursor). |
| `DataError` | Type conversion failures, codec errors. |
| `OperationalError` | Network errors, timeouts, server unavailable, login failures. |
| `IntegrityError` | Constraint violations (PK, FK, unique, NOT NULL). |
| `InternalError` | Driver-internal invariant violation (file a bug). |
| `ProgrammingError` | Bad SQL syntax, missing tables, parameter binding errors. |
| `NotSupportedError` | Driver doesn't support the requested operation. |
| `PoolTimeout` | Pool acquire exceeded `acquire_timeout`. |
## SQLCODE mapping
Informix returns SQLCODE values; the driver maps them to the appropriate exception. A few common ones:
| SQLCODE | Meaning | Exception |
|---|---|---|
| `-201` | Syntax error | `ProgrammingError` |
| `-206` | Table not found | `ProgrammingError` |
| `-239` / `-268` / `-691` / `-703` | Unique / FK / NOT NULL violation | `IntegrityError` |
| `-329` | Database not found | `OperationalError` |
| `-908` | Connection terminated by server | `OperationalError` |
| `-1820` | Codeset conversion failure | `DataError` |
| `-908` / network errors | Server unavailable | `OperationalError` |
The full mapping lives in `src/informix_db/_errcodes.py`.
## Inspecting errors
Every exception carries the original SQLCODE and message:
```python
try:
cur.execute("INSERT INTO users VALUES (?, ?)", (1, "alice"))
except informix_db.IntegrityError as e:
print(e.sqlcode) # -239 (unique violation)
print(e.isam_code) # secondary error code
print(str(e)) # human-readable
```
`sqlcode` is the primary error; `isam_code` is the underlying ISAM error (when applicable). For multi-statement transactions, use `e.statement` to see which statement failed.

121
docs-site/src/content/docs/reference/types.md Normal file
View File

@ -0,0 +1,121 @@
---
title: SQL ↔ Python types
description: Mapping table between Informix SQL types and Python types, with notes on edge cases.
sidebar:
order: 2
---
| SQL type | Python type | Notes |
|---|---|---|
| `SMALLINT` / `INT` / `SERIAL` | `int` | Arbitrary precision on the Python side. |
| `BIGINT` / `BIGSERIAL` | `int` | 8-byte two's complement on the wire. |
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer, a different wire format from `BIGINT` rather than an alias. See below. |
| `FLOAT` / `SMALLFLOAT` | `float` | IEEE 754 double / single. |
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | Exact precision preserved. |
| `CHAR` / `NCHAR` | `str` | Fixed width, space-padded; trailing spaces stripped on decode. |
| `VARCHAR` / `NVCHAR` / `LVARCHAR` | `str` | Variable length. Decoded using `client_locale`. |
| `BOOLEAN` | `bool` | |
| `DATE` | `datetime.date` | |
| `DATETIME YEAR TO …` | `datetime.datetime` / `datetime.time` / `datetime.date` | The Python type depends on the field range. |
| `INTERVAL DAY TO FRACTION` | `datetime.timedelta` | |
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type, because `datetime.timedelta` can't represent year-month intervals. |
| `BYTE` / `TEXT` (legacy in-row blobs) | `bytes` / `str` | |
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators, 72 bytes. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string, **not** decomposed into fields. See below. |
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema, **not** iterable. See below. |
| `NULL` | `None` | |
## INT8 and SERIAL8 are not BIGINT
Informix has two unrelated 64-bit integer types and they share nothing on the wire:
| | Wire format |
|---|---|
| `BIGINT` / `BIGSERIAL` (type 52 / 53) | 8 bytes, two's complement, big-endian |
| `INT8` / `SERIAL8` (type 17 / 18) | 10 bytes, sign-magnitude, halves stored high-last |
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `−n` with *identical* magnitude bytes and the sign in a leading word. Decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
`INT8`/`SERIAL8` are common in schemas predating Informix 11.50, which is when `BIGINT` arrived.
:::caution[Upgrade if you use these types]
`2026.08.27` fixed three decoding bugs: `INT8`/`SERIAL8` weren't decoded at all and returned raw `bytes`; `NCHAR` lost its first character; and `BOOLEAN` corrupted every column after it.
`2026.08.31` fixed two more in `LVARCHAR` framing: a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
`2026.09.02` fixed two more, both found by a new check that verifies each row consumed exactly its own payload. Smart LOBs were read as a flat 72-byte field when they actually occupy 149 bytes populated and 5 when NULL, so a `BLOB` or `CLOB` anywhere but the **last** column shifted every column after it. A NULL `SET` / `MULTISET` / `LIST` / `ROW` skipped its length field and did the same. That release also made a mismatch raise where it happens, naming the column and the byte delta, instead of handing back plausible-looking wrong values.
`2026.09.03` fixed one on the **write** path rather than the read path, which makes it the only entry here that affects data already stored. The `:N` placeholder rewriter was a regular expression and could not see a string literal, so it rewrote the inside of one: `'http://host:8080/x'` was stored as `'http://host?/x'`. Any `HH:MM` time, URL with a port, or `key:value` string was affected, whichever placeholder style you used. If you have written such values through this driver, they are worth checking.
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe. If you select a `BLOB` or `CLOB` in any position other than last, or a nullable collection column, nothing before `2026.09.02` is safe. Upgrade.
:::
## LVARCHAR wire framing
Worth knowing if you're reading captures. Every Informix server we've tested describes `LVARCHAR` columns as UDTVAR (type 40, `extended_name='lvarchar'`), and the value arrives wrapped in a UDT envelope:
```
[1-byte null indicator][4-byte length][content]
```
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte, since both carry a length of zero.
## DATETIME field ranges
Informix's `DATETIME YEAR TO X` is field-range typed. The Python type returned depends on which fields are present:
| Range | Python type |
|---|---|
| `YEAR TO YEAR` through `YEAR TO DAY` | `datetime.date` |
| `YEAR TO HOUR` through `YEAR TO FRACTION(5)` | `datetime.datetime` |
| `HOUR TO HOUR` through `HOUR TO FRACTION(5)` | `datetime.time` |
## Decimal precision
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly, with no float intermediate.
For binding `Decimal` values into INSERT/UPDATE, the driver uses the column's declared scale. Pass `Decimal` for exact values; `float` works but may cause rounding at the column scale.
## NULL
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL, because `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
## Type extensions
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count**, mirroring the server's own representation, with `years` and `remainder_months` available as derived properties:
```python
from informix_db import IntervalYM
ym = IntervalYM(27) # 2 years, 3 months
ym.years # 2
ym.remainder_months # 3
str(ym) # '2-03'
cur.execute("INSERT INTO contracts (term) VALUES (?)", (ym,))
cur.execute("SELECT term FROM contracts WHERE id = ?", (1,))
result = cur.fetchone()[0] # IntervalYM(months=27)
```
Negative intervals are supported; the sign lives on `months` and propagates to both derived properties.
## ROW and collection columns
`informix_db.RowValue` and `informix_db.CollectionValue` are returned for `ROW`, `SET`, `MULTISET`, and `LIST` columns. Both are **opaque wrappers, not decomposed values**:
```python
row_val.raw # bytes: the server's textual representation, e.g. b"ROW('Alice',30)"
row_val.schema # str: the column's declared schema
coll_val.raw # bytes, e.g. b'LIST{10,20,30}'
coll_val.kind # 'set' | 'multiset' | 'list' | 'collection'
coll_val.element_schema # str
```
They are not iterable and do not expose fields by name. Fully parsing a composite type means reimplementing JDBC's `IfxComplexInput`, which we haven't done.
If you need individual fields today, project them in SQL. The server does the decomposition for you and you get ordinary typed columns back:
```python
cur.execute("SELECT person.name, person.age FROM staff") # str, int
```

166
docs-site/src/content/docs/start/quickstart.mdx Normal file
View File

@ -0,0 +1,166 @@
---
title: Install & first query
description: Five minutes from pip install to a real SELECT against a Dockerised Informix server.
sidebar:
order: 2
---
import { Tabs, TabItem, Steps, Aside } from '@astrojs/starlight/components';
This page gets you from a clean Python environment to a working query against a real Informix server in five minutes. We'll use the official IBM Informix Developer Edition Docker image so you don't need an Informix license.
## Prerequisites
- Python 3.10 or newer
- Docker, for the dev server (skip if you already have an Informix instance)
- 4 GB of free RAM for the dev container
## 1. Install the driver
<Tabs>
<TabItem label="uv (recommended)">
```bash
uv add informix-driver
```
</TabItem>
<TabItem label="pip">
```bash
pip install informix-driver
```
</TabItem>
<TabItem label="poetry">
```bash
poetry add informix-driver
```
</TabItem>
</Tabs>
That's the entire dependency. No system packages, no `LD_LIBRARY_PATH`, no `libcrypt.so.1`.
## 2. Start the dev container
```bash
docker run -d --name informix-dev \
-e LICENSE=accept \
-p 9088:9088 \
-p 9089:9089 \
--privileged \
icr.io/informix/informix-developer-database:15.0.1.0.3DE
```
The image takes ~90 seconds to initialize. Watch the logs until you see `oninit running`:
```bash
docker logs -f informix-dev
```
<Aside type="tip">
The `--privileged` flag is required by the dev image, which tries to manage shared memory limits. For production servers this isn't a thing.
</Aside>
## 3. Run your first query
Save this as `hello.py`:
```python
import informix_db
with informix_db.connect(
host="127.0.0.1",
port=9088,
user="informix",
password="in4mix",
database="sysmaster",
server="informix",
) as conn:
cur = conn.cursor()
cur.execute(
"SELECT FIRST 5 dbsname, tabname "
"FROM systables WHERE tabid > 99"
)
for row in cur.fetchall():
print(row)
```
Then:
```bash
python hello.py
```
You should see five rows from Informix's system catalog. If you do, congratulations: you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
## What just happened
When `informix_db.connect()` returned, the driver had:
1. Opened a TCP socket to `127.0.0.1:9088`
2. Sent an `SQ_INFO` PDU containing client capabilities (locale, byte order, app name)
3. Received the server's identification (`IBM Informix Dynamic Server Version 15.0.1.0.3`)
4. Negotiated authentication via `SQ_PASSWD`
5. Opened the `sysmaster` database via `SQ_DBOPEN`
6. Returned a `Connection` object ready for queries
`cur.execute()` sent an `SQ_PREPARE` PDU with your SQL, parsed the response into a column descriptor, sent `SQ_DESCRIBE`, then `SQ_OPEN` to start the cursor. `cur.fetchall()` issued `SQ_FETCH` PDUs and decoded each `SQ_TUPLE` payload via per-column readers.
If you want to see this happen byte-by-byte, [the architecture page](/explain/architecture/) walks through the wire protocol with annotated captures.
## 4. Try parameter binding
```python
with informix_db.connect(host="127.0.0.1", port=9088, user="informix",
password="in4mix", database="sysmaster",
server="informix") as conn:
cur = conn.cursor()
cur.execute(
"SELECT tabname FROM systables WHERE tabid = ?",
(1,),
)
print(cur.fetchone())
```
`?` and `:1` both work. Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
Rows come back as plain tuples. If you would rather read them by column
name, pass `row_factory=informix_db.Row` to `connect()` and you get
`row["name"]` and `row.name` alongside `row[0]`. See
[the API reference](/reference/api/#rows).
## 5. Use the connection pool
For real applications, prefer the pool:
```python
import informix_db
pool = informix_db.create_pool(
host="127.0.0.1", port=9088,
user="informix", password="in4mix",
database="sysmaster", server="informix",
min_size=2, max_size=10,
acquire_timeout=5.0,
)
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
print(cur.fetchone())
pool.close()
```
The pool is thread-safe and has a per-connection wire lock, so accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
## What's next
<Steps>
1. **Going async?** [Async with FastAPI →](/how-to/async-fastapi/) walks through a real FastAPI app with the async pool.
2. **Connecting to production?** [Connect with TLS →](/how-to/tls/) covers TLS-listener configuration and bring-your-own-context patterns.
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/), including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
4. **Migrating from IfxPy?** [Migrate from IfxPy →](/how-to/migrate-from-ifxpy/) covers the API differences and the things IfxPy does that we don't (yet).
</Steps>

149
docs-site/src/content/docs/start/vs-ifxpy.mdx Normal file
View File

@ -0,0 +1,149 @@
---
title: Compared to IfxPy
description: Head-to-head benchmarks against IBM's C-bound Python driver. Where each one wins and why.
sidebar:
order: 3
---
import { Aside } from '@astrojs/starlight/components';
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver, a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Informix Developer Edition Docker container on the same host. Methodology and reproduction steps live in [`tests/benchmarks/compare/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare) in the repo.
## Headline numbers
| Benchmark | IfxPy 3.0.5 (C) | informix-driver (pure Python) | Result |
|---|---:|---:|---:|
| Single-row SELECT round-trip | 118 µs | 114 µs | comparable |
| ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster |
| Cold connect (login handshake) | 11.0 ms | 10.5 ms | comparable |
| `executemany(1k)` in transaction | 23.5 ms | 23.2 ms | tied |
| **`executemany(10k)` in transaction** | 259 ms | **161 ms** | **informix-driver 1.6× faster** |
| **`executemany(100k)` in transaction** | 2376 ms | **1487 ms** | **informix-driver 1.6× faster** |
| `SELECT 1k` rows | 1.34 ms | 1.72 ms | IfxPy 1.28× faster |
| `SELECT 10k` rows | 11.7 ms | 16.1 ms | IfxPy 1.07× faster |
| `SELECT 100k` rows | 116 ms | 169 ms | IfxPy 1.15× faster |
<Aside type="note">
Phase 39's connection-scoped buffered reader closed the bulk-fetch gap from a steady ~2.4× to ~1.05–1.15×. The story of how that landed is in [the buffered reader page](/explain/buffered-reader/).
</Aside>
## When informix-driver wins
### Bulk inserts at scale
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms**, so `informix-driver` is 1.6× faster.
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row, and N RTTs at ~80 µs each adds up to the 100 ms gap.
```python
# Both drivers
cur.executemany(
"INSERT INTO orders VALUES (?, ?, ?)",
rows, # list of 10_000 tuples
)
# informix-driver: 161 ms (10k PDUs sent, then 10k responses drained)
# IfxPy: 259 ms (10k round-trips, each blocking on response)
```
### Containerized deployment
`informix-driver` ships as a 50 KB pure-Python wheel with **zero runtime dependencies**. Your Dockerfile is:
```dockerfile
FROM python:3.13-slim
RUN pip install informix-driver
```
IfxPy's deployment surface is dramatically larger:
- 92 MB IBM OneDB Client tarball
- `setuptools < 58` build pin
- `LD_LIBRARY_PATH` configuration for four directories
- `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9)
- C compiler in the build image
For slim images, multi-stage builds, FaaS deployments, or anywhere build-toolchain-on-the-runtime is friction, `informix-driver` is the only reasonable option.
### Modern Python
IfxPy works on Python ≤ 3.11 currently. The C extension breaks on 3.12+ (PyConfig changes, removed `_PyImport_AcquireLock`, etc.).
`informix-driver` works unmodified on **3.10, 3.11, 3.12, 3.13, and 3.14**. We've kept a CI matrix on every minor version since 3.10 from the start.
### Async
`informix-driver` ships an async API:
```python
from informix_db import aio
async def main():
pool = await aio.create_pool(...)
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT ...")
rows = await cur.fetchall()
```
IfxPy has no async support, so every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
## When IfxPy wins
### Large analytical fetches
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 5–15% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock, which is meaningful but not disqualifying.
The gap is closing phase by phase:
| Phase | Bulk-fetch ratio vs IfxPy |
|---|---|
| Phase 36 | 2.40× slower |
| Phase 37 (per-column readers) | 2.10× slower |
| Phase 38 (codegen-inlined decoders) | 2.04× slower |
| **Phase 39 (connection-scoped buffered reader)** | **1.15× slower** |
If you're running analytical reports that pull millions of rows in a single SELECT and the per-row decode overhead is a measurable cost, IfxPy may be marginally faster today. For most application workloads it isn't.
### Workloads built around CSDK extensions
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API. See [the migration guide](/how-to/migrate-from-ifxpy/).
## Methodology
Benchmarks are pytest-benchmark fixtures in `tests/benchmarks/compare/` against the official `icr.io/informix/informix-developer-database:15.0.1.0.3DE` image, running on the same loopback as the Python process.
Reported numbers are **median over 10+ rounds**, with IQR included. Why median over mean: the first round of any run includes JIT warmup, page-cache miss, and a TCP slow-start round-trip. The mean is contaminated by these one-shot costs in a way that misrepresents steady-state behavior. Median + IQR is what we report.
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise. A fair reading is "5–15% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
To reproduce:
```bash
git clone https://git.supported.systems/warehack.ing/informix-db
cd informix-db/tests/benchmarks/compare
make ifx-up
make compare
```
The `Makefile` handles the IfxPy install gauntlet (Python ≤ 3.11 environment, `setuptools < 58`, `libcrypt.so.1` symlink, OneDB CSDK download, the four `LD_LIBRARY_PATH` exports) so you don't have to learn it manually.
## Summary
Use `informix-driver` when:
- You're writing new code in Python ≥ 3.10
- Your workload is bulk-insert / ETL / log-shipping
- You want async / FastAPI integration
- You're deploying in containers or to Python environments where build toolchains are friction
- Your platform doesn't have `libcrypt.so.1`
Use IfxPy when:
- You have an existing IfxPy codebase
- You're running large analytical SELECTs and the 5–15% decode-side gap matters
- You're constrained to Python ≤ 3.11 anyway
For everything else, the cost-benefit favors `pip install informix-driver`.

87
docs-site/src/content/docs/start/wtf.md Normal file
View File

@ -0,0 +1,87 @@
---
title: WTF did you build this for?
description: A pure-Python Informix driver, why it didn't exist before, and what it's good for.
sidebar:
order: 1
label: WTF did you build this for?
---
The existing tools were not my style.
Every Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That covers `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
## The problem with IBM's C SDK
The IBM Informix Client SDK (CSDK), now packaged as part of OneDB Client, is a 92 MB tarball with a non-trivial install gauntlet:
- Python ≤ 3.11 (IfxPy is broken on 3.12+)
- `setuptools < 58` (legacy build system)
- Permissive `CFLAGS` for the C extension build
- Manual download of the 92 MB ODBC tarball
- Four `LD_LIBRARY_PATH` directories
- `libcrypt.so.1`, deprecated in 2018 and missing on Arch, Fedora 35+, RHEL 9
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db`; the distribution name dodges PyPI's 2008-vintage `informixdb` package, while the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
## What it does
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly. It is the same protocol IBM's JDBC driver uses, and the same one the CSDK speaks under the hood. No native code is in the thread of execution.
The wire protocol was reverse-engineered through three sources:
1. **Decompiled IBM JDBC driver** (`com.informix.jdbc.IfxConnection` and friends), used as a clean-room reference for PDU shapes and protocol semantics.
2. **Annotated `socat` captures** of real client/server traffic against the IBM Informix Developer Edition Docker image.
3. **Differential testing** against `IfxPy`, so every codec path is checked against the C driver's behavior on the same data.
The result is a PEP 249 compliant driver with a sync API, an async API (FastAPI / asyncio compatible), a connection pool, TLS support, smart-LOB read/write, scrollable cursors, fast-path stored procedure invocation, and bulk-insert / bulk-fetch performance within ~10–60% of the C driver depending on workload.
## What it's good for
The places where `informix-driver` is unambiguously the right choice:
- **ETL and bulk-load pipelines.** Pipelined `executemany` (Phase 33) is 1.6× faster than IfxPy at scale because every BIND+EXECUTE PDU goes out before any responses are drained. IfxPy still pays one round-trip per `IfxPy.execute(stmt, tuple)` call.
- **Container deployments.** The 50 KB wheel and absent native deps mean a slim base image works. No multi-stage build to compile the CSDK.
- **Modern Python.** Works on 3.10 through 3.14 unmodified. IfxPy hasn't shipped 3.12 wheels.
- **Async / FastAPI.** Native async support via thread-pool wrapping. IfxPy is fully synchronous; using it from FastAPI requires `run_in_executor` boilerplate and gives up the connection pool's natural async semantics.
- **Anywhere `libcrypt.so.1` is missing.** Modern Linux distributions ship `libcrypt.so.2`. IfxPy refuses to load without `libcrypt.so.1`. We don't link against either.
## What IfxPy is still better at
Honesty matters here:
- **Large analytical fetches.** IfxPy's C-level `fetch_tuple` decoder is faster than our Python `parse_tuple_payload` (~1.1 µs/row vs ~2.0 µs/row after Phase 39). For workloads pulling 10k+ rows in a single SELECT where the per-row decode cost dominates, IfxPy is currently 5–15% faster. The gap is shrinking phase by phase.
- **Workloads built around the CSDK.** If your existing code already uses IfxPy idioms (`IfxPyDbi.connect_pooled`, IBM's specific cursor extensions), the migration to `informix-driver` is straightforward but not zero-cost.
The honest summary table from the [comparison page](/start/vs-ifxpy/):
| Workload | Winner | Margin |
|---|---|---|
| Bulk insert (`executemany` 10k–100k rows) | `informix-driver` | 1.6× faster |
| Bulk SELECT (10k–100k rows) | IfxPy | 1.05–1.15× faster |
| Single-row queries | tied | within noise |
| Cold connect | tied | within noise |
| Containerized deployment | `informix-driver` | no contest |
| Python 3.12+ | `informix-driver` | only option |
## Production-ready
Every finding from a system-wide failure-mode audit (data correctness, wire safety, resource leaks, concurrency, async cancellation) has been addressed:
- Pool no longer returns connections with open transactions
- Per-connection wire lock prevents PDU interleaving from accidental sharing
- Async cancellation cannot leak running workers onto recycled connections
- `_raise_sq_err` no longer masks wire desync via bare-except
- Cursor finalizers release server-side resources on mid-fetch raise
- 5 medium-severity hardening items resolved
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
470+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 472/472 on 15.0.1.0.3DE and 14.10.FC7W1DE, and 471/472 on 12.10.FC12W1DE (the single skip is a common table expression, which 12.10 predates). `make test-matrix` runs the lot.
That matrix exists because it turned out to be needed. A user reported corrupted result sets on Informix 12; the cause was three framing bugs that affected every version including the one we tested against, and they'd survived because no fixture used the affected types. Testing one server and inferring the rest is how that happens.
## Read next
- **[Install & first query →](/start/quickstart/)**: five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
- **[Compared to IfxPy →](/start/vs-ifxpy/)**: full head-to-head benchmarks, methodology, and reproduction.
- **[Architecture →](/explain/architecture/)**: how the layers stack, from socket through framing, codec, resultset, and cursor.

428
docs-site/src/styles/components.css Normal file
View File

@ -0,0 +1,428 @@
/* Component-scoped styles for the custom Hero and homepage modules */
.ifx-hero {
display: grid;
grid-template-columns: minmax(0, 1fr) minmax(0, 1.1fr);
gap: clamp(1.5rem, 4vw, 4rem);
align-items: center;
padding: clamp(2rem, 5vw, 4.5rem) 0 clamp(2rem, 4vw, 3.5rem);
border-bottom: 1px solid var(--sl-color-hairline-light);
}
@media (max-width: 800px) {
.ifx-hero {
grid-template-columns: 1fr;
}
}
.ifx-hero__eyebrow {
display: inline-flex;
align-items: center;
gap: 0.5rem;
font-family: var(--sl-font-mono);
font-size: 0.75rem;
letter-spacing: 0.18em;
text-transform: uppercase;
color: var(--sl-color-text-accent);
margin-bottom: 0.75rem;
}
.ifx-hero__eyebrow::before {
content: '';
width: 8px;
height: 8px;
background: var(--ifx-amber);
border-radius: 1px;
box-shadow: 0 0 12px var(--ifx-amber);
animation: ifx-pulse 2.4s ease-in-out infinite;
}
@keyframes ifx-pulse {
0%, 100% { opacity: 0.4; }
50% { opacity: 1; }
}
.ifx-hero__title {
font-size: clamp(2rem, 5vw, 3.25rem);
line-height: 1.05;
letter-spacing: -0.02em;
font-weight: 700;
margin: 0 0 1rem;
color: var(--sl-color-white);
}
.ifx-hero__title strong {
background: linear-gradient(180deg, var(--ifx-amber-bright) 0%, var(--ifx-amber) 100%);
-webkit-background-clip: text;
background-clip: text;
color: transparent;
font-weight: inherit;
}
.ifx-hero__lede {
font-size: clamp(1rem, 1.5vw, 1.15rem);
line-height: 1.55;
color: var(--sl-color-gray-2);
max-width: 36rem;
margin: 0 0 1.75rem;
}
.ifx-hero__cta {
display: flex;
flex-wrap: wrap;
gap: 0.75rem;
}
.ifx-hero__cta a {
display: inline-flex;
align-items: center;
gap: 0.4rem;
padding: 0.6rem 1.1rem;
border-radius: 6px;
font-weight: 500;
text-decoration: none !important;
transition: transform 0.08s ease, box-shadow 0.15s ease;
}
.ifx-hero__cta a:hover {
transform: translateY(-1px);
}
.ifx-hero__cta .primary {
background: var(--ifx-amber);
color: var(--ifx-charcoal-0);
}
.ifx-hero__cta .primary:hover {
box-shadow: 0 4px 16px rgba(245, 165, 36, 0.35);
}
.ifx-hero__cta .secondary {
border: 1px solid var(--sl-color-hairline-shade);
color: var(--sl-color-text);
background: transparent;
}
.ifx-hero__cta .secondary:hover {
border-color: var(--sl-color-accent);
}
.ifx-hero__install {
font-family: var(--sl-font-mono);
font-size: 0.95rem;
margin-top: 1.5rem;
padding: 0.75rem 1rem;
background: var(--sl-color-bg-inline-code);
border-left: 2px solid var(--ifx-amber);
border-radius: 0 4px 4px 0;
color: var(--sl-color-text);
user-select: all;
}
.ifx-hero__install::before {
content: '$ ';
color: var(--sl-color-text-accent);
user-select: none;
}
/* Wire-dump easter egg: types out real captured handshake bytes */
.ifx-wiredump {
font-family: var(--sl-font-mono);
font-size: 0.78rem;
line-height: 1.55;
background: var(--ifx-charcoal-0);
color: var(--ifx-amber);
border: 1px solid var(--sl-color-hairline);
border-radius: 8px;
padding: 1.25rem 1rem;
height: clamp(280px, 35vw, 360px);
/* x: auto allows hex content to scroll horizontally inside the dump
when the column is too narrow (tablet portrait, narrow desktop with
2-col hero); y: hidden keeps the dump's vertical bounds fixed so
unrevealed animation lines stay clipped below the fold */
overflow-x: auto;
overflow-y: hidden;
position: relative;
box-shadow: inset 0 0 60px rgba(245, 165, 36, 0.04);
}
:root[data-theme='light'] .ifx-wiredump {
background: #1a1612;
}
.ifx-wiredump__scroll {
white-space: pre;
height: 100%;
overflow: hidden;
}
.ifx-wiredump__line {
display: block;
opacity: 0;
white-space: pre;
}
.ifx-wiredump__line.is-visible {
opacity: 1;
}
.ifx-wiredump__byte--ascii {
color: #ffd884;
}
.ifx-wiredump__caret {
display: inline-block;
width: 8px;
height: 1em;
background: var(--ifx-amber);
vertical-align: text-bottom;
margin-left: 2px;
animation: ifx-blink 1.1s steps(2) infinite;
}
@keyframes ifx-blink {
0%, 50% { opacity: 1; }
51%, 100% { opacity: 0; }
}
.ifx-wiredump__direction {
color: var(--sl-color-gray-3);
margin-right: 0.5rem;
}
.ifx-wiredump__direction--out {
color: var(--ifx-amber);
}
.ifx-wiredump__direction--in {
color: #6dd2a4;
}
.ifx-wiredump__caption {
font-family: var(--sl-font-mono);
font-size: 0.7rem;
letter-spacing: 0.1em;
text-transform: uppercase;
color: var(--sl-color-gray-3);
margin-top: 0.6rem;
text-align: right;
}
/* Homepage feature grid */
.ifx-features {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
gap: 1.25rem;
margin: 3rem 0;
}
.ifx-feature {
padding: 1.5rem;
border: 1px solid var(--sl-color-hairline-light);
border-radius: 8px;
background: var(--sl-color-bg-nav);
transition: border-color 0.15s ease, transform 0.1s ease;
}
.ifx-feature:hover {
border-color: var(--sl-color-accent);
}
.ifx-feature__icon {
width: 28px;
height: 28px;
margin-bottom: 0.75rem;
color: var(--sl-color-text-accent);
}
.ifx-feature h3 {
font-size: 1.05rem;
margin: 0 0 0.5rem;
letter-spacing: -0.01em;
}
.ifx-feature p {
font-size: 0.92rem;
line-height: 1.5;
margin: 0;
color: var(--sl-color-gray-2);
}
/* Supported Systems "joint" badge, appears below every page's footer */
.ifx-ss-badge {
margin-top: 3.5rem;
padding: 0;
border-top: 1px solid var(--sl-color-hairline-light);
}
.ifx-ss-badge__link {
display: grid;
grid-template-columns: auto minmax(0, 1fr);
gap: 1.25rem;
align-items: center;
padding: 1.5rem 0 0.5rem;
text-decoration: none !important;
color: var(--sl-color-text);
transition: color 0.12s ease;
}
.ifx-ss-badge__link:hover {
color: var(--sl-color-text-accent);
}
.ifx-ss-badge__link:hover .ifx-ss-badge__cta {
gap: 0.5rem;
}
.ifx-ss-badge__logo {
width: 60px;
height: 45px;
flex-shrink: 0;
/* Slight lift so the punch-card pattern reads */
filter: drop-shadow(0 1px 2px rgba(37, 99, 235, 0.12));
}
.ifx-ss-badge__copy {
min-width: 0;
}
.ifx-ss-badge__heading {
font-family: var(--sl-font-mono);
font-size: 0.78rem;
font-weight: 600;
letter-spacing: 0.16em;
text-transform: uppercase;
color: var(--sl-color-text-accent);
margin: 0 0 0.4rem;
border: none !important;
padding: 0 !important;
}
.ifx-ss-badge__body {
font-size: 0.92rem;
line-height: 1.55;
color: var(--sl-color-gray-2);
margin: 0 0 0.5rem;
}
.ifx-ss-badge__name {
color: #3b82f6;
font-weight: 600;
}
:root[data-theme='light'] .ifx-ss-badge__name {
color: #2563eb;
}
.ifx-ss-badge__cta {
display: inline-flex;
align-items: center;
gap: 0.35rem;
font-size: 0.85rem;
font-weight: 500;
color: var(--sl-color-text-accent);
transition: gap 0.15s ease;
}
@media (max-width: 540px) {
.ifx-ss-badge__link {
grid-template-columns: 1fr;
gap: 0.85rem;
text-align: left;
}
.ifx-ss-badge__logo {
width: 50px;
height: 38px;
}
}
/* ============================================================
* Mobile (≤640px): tighten the hero, shrink the wire-dump,
* hide the ASCII column, prevent runaway horizontal overflow
* ============================================================ */
@media (max-width: 640px) {
/* Defensive guard against any descendant forcing horizontal scroll:
the wire-dump's white-space: pre content was overflowing the
hero column and propagating up to the page. overflow-x: hidden
on .ifx-hero contains it without affecting page-level scroll. */
.ifx-hero,
.ifx-hero__copy,
.ifx-hero__visual {
min-width: 0; /* allow flex/grid item to shrink below content min-width */
}
.ifx-hero {
padding: 1.25rem 0 1.75rem;
gap: 1.25rem;
}
.ifx-hero__eyebrow {
flex-wrap: wrap;
row-gap: 0.25rem;
font-size: 0.7rem;
letter-spacing: 0.12em;
}
.ifx-hero__title {
font-size: clamp(1.75rem, 7.5vw, 2.5rem);
line-height: 1.1;
}
.ifx-hero__lede {
font-size: 0.95rem;
line-height: 1.55;
}
.ifx-hero__cta {
gap: 0.5rem;
}
.ifx-hero__cta a {
padding: 0.55rem 0.9rem;
font-size: 0.92rem;
}
.ifx-hero__install {
font-size: 0.85rem;
word-break: break-all; /* survive any future longer install command */
}
/* The wire-dump is the biggest mobile risk: pre-formatted hex lines
are ~50 chars wide. Two things together make it readable on
narrow viewports: smaller font and ASCII column hidden. The
overflow-x: auto fallback already lives on the base rule. */
.ifx-wiredump {
font-size: 0.62rem;
height: clamp(220px, 60vw, 280px);
padding: 0.85rem 0.65rem;
}
.ifx-wiredump__byte--ascii {
display: none;
}
.ifx-wiredump__caption {
font-size: 0.62rem;
text-align: left;
}
.ifx-features {
gap: 0.9rem;
margin: 2rem 0;
}
.ifx-feature {
padding: 1.1rem;
}
.ifx-feature h3 {
font-size: 1rem;
}
}
/* Performance-bar visual for the comparison table */
.ifx-perfbar {
position: relative;
height: 6px;
background: var(--sl-color-bg-inline-code);
border-radius: 3px;
overflow: hidden;
margin-top: 0.25rem;
}
.ifx-perfbar__fill {
position: absolute;
inset: 0 auto 0 0;
background: linear-gradient(90deg, var(--ifx-amber) 0%, var(--ifx-amber-bright) 100%);
border-radius: 3px;
}

137
docs-site/src/styles/theme.css Normal file
View File

@ -0,0 +1,137 @@
/*
* informix-driver docs theme
* - Charcoal base (no purple gradients, ever)
* - Amber accent, a CRT-monitor nod distinct from sibling sites' cyan
* - Inter for body, IBM Plex Mono for technical bytes
*/
:root {
--ifx-amber: #f5a524;
--ifx-amber-bright: #ffb84d;
--ifx-amber-dim: #b87a18;
--ifx-charcoal-0: #0e0d0c;
--ifx-charcoal-1: #161412;
--ifx-charcoal-2: #1f1c1a;
--ifx-charcoal-3: #2a2622;
--ifx-paper: #faf7f2;
--ifx-paper-2: #f3eee5;
--ifx-ink: #1a1612;
--ifx-ink-soft: #4a443c;
--ifx-rule: rgba(245, 165, 36, 0.18);
--sl-font: 'Inter', system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
--sl-font-mono: 'IBM Plex Mono', ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
}
:root[data-theme='dark'] {
--sl-color-accent-low: #3a2810;
--sl-color-accent: var(--ifx-amber);
--sl-color-accent-high: var(--ifx-amber-bright);
--sl-color-white: #fbf7ee;
--sl-color-gray-1: #ece6da;
--sl-color-gray-2: #c8c1b1;
--sl-color-gray-3: #948b78;
--sl-color-gray-4: #5a5246;
--sl-color-gray-5: #2e2823;
--sl-color-gray-6: var(--ifx-charcoal-2);
--sl-color-black: var(--ifx-charcoal-0);
--sl-color-bg: var(--ifx-charcoal-0);
--sl-color-bg-nav: var(--ifx-charcoal-1);
--sl-color-bg-sidebar: var(--ifx-charcoal-1);
--sl-color-bg-inline-code: var(--ifx-charcoal-2);
--sl-color-bg-accent: var(--ifx-amber);
--sl-color-text: #ece6da;
--sl-color-text-accent: var(--ifx-amber-bright);
--sl-color-text-invert: var(--ifx-charcoal-0);
--sl-color-hairline: rgba(245, 165, 36, 0.14);
--sl-color-hairline-light: rgba(245, 165, 36, 0.08);
--sl-color-hairline-shade: rgba(245, 165, 36, 0.22);
}
:root[data-theme='light'] {
--sl-color-accent-low: #fbe8c8;
--sl-color-accent: var(--ifx-amber-dim);
--sl-color-accent-high: #7a4f0a;
--sl-color-white: #1a1612;
--sl-color-gray-1: #2e2823;
--sl-color-gray-2: #4a443c;
--sl-color-gray-3: #6e665a;
--sl-color-gray-4: #948b78;
--sl-color-gray-5: #d8d1c1;
--sl-color-gray-6: #ebe5d6;
--sl-color-gray-7: #f3eee5;
--sl-color-black: var(--ifx-paper);
--sl-color-bg: var(--ifx-paper);
--sl-color-bg-nav: var(--ifx-paper-2);
--sl-color-bg-sidebar: var(--ifx-paper-2);
--sl-color-bg-inline-code: #ece6d4;
--sl-color-bg-accent: var(--ifx-amber-dim);
--sl-color-text: var(--ifx-ink);
--sl-color-text-accent: var(--ifx-amber-dim);
--sl-color-text-invert: var(--ifx-paper);
--sl-color-hairline: rgba(120, 80, 12, 0.18);
--sl-color-hairline-light: rgba(120, 80, 12, 0.10);
--sl-color-hairline-shade: rgba(120, 80, 12, 0.28);
}
/* Tighten heading rhythm; Starlight defaults are a touch loose for technical docs */
.sl-markdown-content h2 {
margin-top: 2.5rem;
border-top: 1px solid var(--sl-color-hairline-light);
padding-top: 1.5rem;
letter-spacing: -0.01em;
}
.sl-markdown-content h3 {
margin-top: 1.75rem;
letter-spacing: -0.005em;
}
/* Inline code: monospace + amber tint, no jarring background block */
.sl-markdown-content :not(pre) > code {
font-feature-settings: 'ss02', 'cv02';
border: 1px solid var(--sl-color-hairline-shade);
font-size: 0.9em;
padding: 0.1em 0.35em;
border-radius: 4px;
}
/* Tables: dense, technical, amber column rules, for type-mapping & benchmark tables */
.sl-markdown-content table {
border-collapse: collapse;
font-variant-numeric: tabular-nums;
font-size: 0.92rem;
}
.sl-markdown-content thead th {
border-bottom: 2px solid var(--sl-color-accent);
text-align: left;
font-weight: 600;
letter-spacing: 0.02em;
text-transform: uppercase;
font-size: 0.78em;
color: var(--sl-color-text-accent);
}
.sl-markdown-content tbody td {
border-bottom: 1px solid var(--sl-color-hairline-light);
padding: 0.5rem 0.75rem;
}
/* Anchor links: underline, no rainbow */
.sl-markdown-content a:not(.sl-anchor-link) {
text-decoration: underline;
text-decoration-color: var(--sl-color-accent);
text-decoration-thickness: 1px;
text-underline-offset: 3px;
transition: text-decoration-thickness 0.1s ease;
}
.sl-markdown-content a:not(.sl-anchor-link):hover {
text-decoration-thickness: 2px;
}

5
docs-site/tsconfig.json Normal file
View File

@ -0,0 +1,5 @@
{
"extends": "astro/tsconfigs/strict",
"include": [".astro/types.d.ts", "**/*"],
"exclude": ["dist"]
}

307
docs/USAGE.md
View File

@ -26,6 +26,95 @@ conn = informix_db.connect(
`database` may be `None` to log in without selecting a database; the server still completes a successful login. Useful for cross-database queries that fully qualify table names. `database` may be `None` to log in without selecting a database; the server still completes a successful login. Useful for cross-database queries that fully qualify table names.
### Timeouts and keepalive
| Parameter | Purpose | Default |
|---|---|---|
| `connect_timeout` | Time-bound for the TCP connect + login handshake. `None` uses the OS default (typically minutes). | `None` |
| `read_timeout` | Per-read timeout on subsequent socket reads. Fires `OperationalError` on a hung server. | `None` |
| `keepalive` | Set `SO_KEEPALIVE` on the socket. Useful for long-lived idle connections behind aggressive NAT/firewalls. | `False` |
A reasonable production starting point: `connect_timeout=10.0, read_timeout=30.0, keepalive=True`. The connect timeout protects startup; the read timeout protects against a frozen server; keepalive protects against silent idle disconnection.
### Environment dictionary
The `env={}` parameter sets server-side session variables sent in the login PDU. Useful for things like `OPTOFC` (optimize-on-fetch-close), `IFX_AUTOFREE`, `OPT_GOAL`, or any other runtime knob the server reads from the session env block.
```python
informix_db.connect(
...,
env={
"OPT_GOAL": "-1", # optimize for first-row return
"OPTOFC": "1", # auto-free cursors at fetch-close
"IFX_AUTOFREE": "1",
},
)
```
`CLIENT_LOCALE` is set automatically from the `client_locale=` parameter — don't put it in `env=`.
## Locale and Unicode
The connection's `client_locale` controls how Python `str` values are encoded to bytes (and back) for CHAR / VARCHAR / NCHAR / NVCHAR / LVARCHAR / CLOB columns. The default `"en_US.8859-1"` is safe for ASCII + Western European text. **For multibyte text (CJK, Cyrillic, Arabic, emoji), set `client_locale="en_US.utf8"` AND make sure the database's `DB_LOCALE` is also UTF-8.**
```python
conn = informix_db.connect(..., client_locale="en_US.utf8")
print(conn.encoding) # "utf-8"
cur = conn.cursor()
cur.execute("INSERT INTO docs (body) VALUES (?)", ("你好世界",))
```
The `Connection.encoding` property reports the resolved Python codec name. Common mappings:
| Locale | Python codec |
|---|---|
| `en_US.8859-1` (default) | `iso-8859-1` |
| `en_US.utf8` / `en_US.UTF-8` | `utf-8` |
| `en_US.8859-15` | `iso-8859-15` |
| Anything without a codeset suffix, or unknown | falls back to `iso-8859-1` |
### CLIENT_LOCALE vs DB_LOCALE
* **`CLIENT_LOCALE`** is what *your code* uses to encode / decode string parameters and column values. Set per-connection.
* **`DB_LOCALE`** is what *the database* uses to store string columns. Set at `CREATE DATABASE` time, immutable afterwards.
If they match, no transcoding happens. If they differ, the server transcodes between them at the storage boundary — and any character in your data that doesn't exist in `DB_LOCALE`'s codeset is either replaced with `?` (lossy) or rejected with sqlcode `-1820` (depends on the server version). The IBM Developer Edition Docker image's default `testdb` is created with `DB_LOCALE=en_US.8859-1`; storing `"你好"` there will fail server-side regardless of `CLIENT_LOCALE`.
To create a UTF-8 database for full multibyte support:
```bash
# Inside the container, before CREATE DATABASE:
export DB_LOCALE=en_US.utf8
export CLIENT_LOCALE=en_US.utf8
```
```sql
CREATE DATABASE my_utf8db WITH LOG IN rootdbs;
```
### When characters can't fit the codec
Passing a `str` containing characters that can't be encoded under `client_locale` raises `informix_db.DataError` cleanly — the connection survives:
```python
conn = informix_db.connect(..., client_locale="en_US.8859-1")
cur = conn.cursor()
try:
cur.execute("INSERT INTO t VALUES (?)", ("你好",))
except informix_db.DataError as e:
print(e)
# cannot encode parameter under client_locale codec 'iso-8859-1':
# ordinal not in range(256) at position 0-2.
# Connect with a wider locale (e.g., 'en_US.utf8') if your data
# contains characters outside this codec.
# Connection is still good
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
```
Protocol-level strings (cursor names, function signatures, error "near tokens", SQL keywords) are always ASCII and stay `iso-8859-1` regardless of `client_locale`.
## Cursor lifecycle ## Cursor lifecycle
```python ```python
@ -71,7 +160,66 @@ cur.execute(
) )
``` ```
Type mapping: `int`, `float`, `str`, `bool`, `None`, `datetime.date`, `datetime.datetime`, `datetime.timedelta`, `decimal.Decimal`, `informix_db.IntervalYM`, `bytes` (BYTE/TEXT params). Supported parameter types: `int`, `float`, `str`, `bool`, `None`, `datetime.date`, `datetime.datetime`, `datetime.timedelta`, `decimal.Decimal`, `informix_db.IntervalYM`, `bytes` (BYTE/TEXT params).
## Type mapping reference
What you put in vs. what comes out:
| SQL type | Param accepts | Result returns |
|---|---|---|
| `SMALLINT` (16-bit) | `int` (range -32,767..32,767) | `int` |
| `INT` / `INTEGER` (32-bit) | `int` (range -2³¹+1..2³¹-1) | `int` |
| `BIGINT` (64-bit) | `int` | `int` |
| `SERIAL` / `BIGSERIAL` | `int` (omit for auto-assign) | `int` |
| `SMALLFLOAT` / `REAL` | `float` | `float` |
| `FLOAT` / `DOUBLE PRECISION` | `float` | `float` |
| `DECIMAL(p,s)` / `NUMERIC` | `decimal.Decimal` | `decimal.Decimal` |
| `MONEY(p,s)` | `decimal.Decimal` | `decimal.Decimal` |
| `CHAR(N)` | `str` (right-trimmed of trailing spaces) | `str` |
| `VARCHAR(N)` / `NVARCHAR(N)` | `str` | `str` |
| `NCHAR(N)` | `str` | `str` |
| `LVARCHAR(N)` | `str` | `str` |
| `BOOLEAN` | `bool` | `bool` |
| `DATE` | `datetime.date` | `datetime.date` |
| `DATETIME YEAR TO DAY` | `datetime.datetime` | `datetime.date` |
| `DATETIME YEAR TO SECOND` (etc.) | `datetime.datetime` | `datetime.datetime` |
| `DATETIME HOUR TO SECOND` | not yet | `datetime.time` |
| `INTERVAL DAY TO FRACTION(5)` | `datetime.timedelta` | `datetime.timedelta` |
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | `informix_db.IntervalYM` |
| `BYTE` (legacy in-row blob) | `bytes` | `bytes` |
| `TEXT` (legacy in-row clob) | `bytes` (or `str`, encoded under `conn.encoding`) | `str` |
| `BLOB` (smart-LOB) | use `cursor.write_blob_column` with `BLOB_PLACEHOLDER` | `informix_db.BlobLocator` (use `cursor.read_blob_column` for bytes) |
| `CLOB` (smart-LOB) | use `cursor.write_blob_column(..., clob=True)` | `informix_db.ClobLocator` |
| `ROW(...)` | not yet | `informix_db.RowValue` (raw payload + schema) |
| `SET(...)` / `MULTISET(...)` / `LIST(...)` | not yet | `informix_db.CollectionValue` |
| `NULL` (any type) | `None` | `None` |
### NULL sentinels
Informix encodes NULL inline rather than as a separate flag for fixed-width types:
* `INT`: `0x80000000` (`INT_MIN`)
* `SMALLINT`: `0x8000` (`SHORT_MIN`)
* `BIGINT`: `0x8000000000000000` (`LONG_MIN`)
* `REAL` / `FLOAT`: all-`0xff` bytes
* `DATE`: `0x80000000` (Day_MIN)
If your data legitimately contains these values, you'll see them surface as `None` on the Python side. (Real-world usage rarely hits this — `INT_MIN` as a valid value is uncommon — but it's documented behavior, not a bug.)
### `IntervalYM`
Year-month intervals can't collapse into `datetime.timedelta` because months have variable length. Provided as a small dataclass:
```python
from informix_db import IntervalYM
iv = IntervalYM(months=18)
print(iv.years, iv.remainder_months) # 1 6
print(str(iv)) # "1-06"
cur.execute("INSERT INTO leases (term) VALUES (?)", (iv,))
```
## Transactions ## Transactions
@ -115,6 +263,123 @@ cur.executemany(
conn.commit() conn.commit()
``` ```
## Performance tips
Three patterns dominate real-world performance. They're all about **batching the right thing**:
### 1. Wrap bulk INSERTs in a transaction (53× speedup)
Under `autocommit=True`, **every INSERT forces a server-side transaction-log flush**. Under `autocommit=False`, the flush happens once at COMMIT.
| Pattern | 1000 rows | Per row | Throughput |
|---|---|---|---|
| `executemany` autocommit=True | 1.72 s | 1.72 ms | ~580 rows/sec |
| `executemany` in single txn | 32 ms | **32 µs** | **~31,000 rows/sec** |
```python
# Slow — every row commits independently
conn = informix_db.connect(..., autocommit=True)
conn.cursor().executemany("INSERT ...", rows)
# Fast — one log flush at the end
conn = informix_db.connect(..., autocommit=False) # default
cur = conn.cursor()
cur.executemany("INSERT ...", rows)
conn.commit()
```
This is the single biggest win for any bulk-load workload.
### 2. Use `executemany`, not a loop of `execute` (≈100× speedup)
`executemany` PREPAREs once and BIND+EXECUTEs per row. A naive loop PREPAREs and RELEASEs per row — paying the server-side parse cost N times.
```python
# Slow: 1.88 ms per row, dominated by PREPARE/RELEASE overhead
for row in rows:
cur.execute("INSERT INTO t VALUES (?, ?, ?)", row)
# Fast: shares the prepared statement across all rows
cur.executemany("INSERT INTO t VALUES (?, ?, ?)", rows)
```
### 3. Use a connection pool (72× speedup over cold connect)
Cold connect takes ~11 ms (TCP + login handshake). Pool acquire takes ~150 µs. If your application opens a fresh connection per request, fix that first.
```python
# In a long-lived process (FastAPI, Django, worker), open the pool once
pool = informix_db.create_pool(host="...", min_size=2, max_size=10)
# Per request:
with pool.connection() as conn:
cur = conn.cursor()
cur.execute(...)
```
### Other tips
* **Cursor reuse is fine across queries** — but each `execute()` resets `description`, `rowcount`, and the materialized result set. If you need the prior query's data, capture it before re-executing.
* **`fetchall()` materializes the whole result set in memory.** For large queries, iterate (`for row in cur:`) or use `fetchmany(N)`. Internally the cursor still buffers a server-fetch worth of rows at a time.
* **The `fast_path_call` API is dramatically cheaper than equivalent SQL** for repeated UDF invocations — routine handles are cached per-connection, so the second call onwards skips the `SQ_GETROUTINE` round-trip.
For raw numbers (codec speed, round-trip latencies, full bench results), see `tests/benchmarks/README.md`.
## Scrollable cursors
A regular cursor walks rows forward only via `fetchone` / `fetchmany` / iteration. The **`fetch_*` family** lets you move backwards, jump to absolute positions, fetch the last row directly, and revisit rows already seen.
```python
cur = conn.cursor()
cur.execute("SELECT id, name FROM users ORDER BY id")
# Standard methods still work
first = cur.fetchone() # row 0
second = cur.fetchone() # row 1
# Plus the scroll surface
last = cur.fetch_last() # last row
prev = cur.fetch_prior() # one back from current
specific = cur.fetch_absolute(50) # row 50 (0-indexed)
relative = cur.fetch_relative(-3) # 3 rows back from current
back_to_start = cur.fetch_first() # row 0
# PEP 249 scroll()
cur.scroll(5, mode="relative") # forward 5 from current
cur.scroll(0, mode="absolute") # to row 0
# Where am I?
print(cur.rownumber) # 0-indexed; None at before-first / after-last
```
### Two modes: in-memory vs server-side
The default cursor materializes the full result set into Python memory on `execute`, then `fetch_*` methods operate on the buffer. Random access is essentially free, but memory grows with row count.
Pass `scrollable=True` to `cursor()` to get a **server-side** scroll cursor:
```python
cur = conn.cursor(scrollable=True)
cur.execute("SELECT id, name FROM big_table")
last_row = cur.fetch_last() # one round-trip, no buffer
row_500 = cur.fetch_absolute(500) # one round-trip
```
Server-side mode keeps the cursor open on the server and issues a `SQ_SFETCH` round-trip per scroll operation. Constant client memory, network round-trip per move. Use it when your result set is large enough that materializing it would be wasteful.
| Mode | When to use |
|---|---|
| `cursor()` (default) | Result fits comfortably in memory (~thousands of rows). All `fetch_*` methods are local; fastest random access. |
| `cursor(scrollable=True)` | Large result sets where memory matters. Each scroll operation is a round-trip; cursor stays open server-side until `close()`. |
Server-side scroll cursors require non-autocommit mode (the server needs an open transaction to keep the cursor alive across fetches).
### Edge cases
* `fetch_prior()` from past-end returns the **last** row (SQL standard semantics — the first prior from "after-last" is the last actual row, not the second-to-last).
* `fetch_absolute(0)` is the first row; `fetch_absolute(-1)` is the last row (Python-style negative indexing).
* `cursor.rownumber` is 0-indexed; returns `None` when positioned before-first or after-last, or when no result set exists.
## Smart-LOBs (BLOB / CLOB) ## Smart-LOBs (BLOB / CLOB)
### Read ### Read
@ -229,6 +494,23 @@ await pool.close()
The async API mirrors the sync API one-to-one. Each blocking I/O call is offloaded to a worker thread via `asyncio.to_thread` — the event loop never blocks; concurrent queries across an `asyncio.gather` actually run in parallel up to `max_size`. The async API mirrors the sync API one-to-one. Each blocking I/O call is offloaded to a worker thread via `asyncio.to_thread` — the event loop never blocks; concurrent queries across an `asyncio.gather` actually run in parallel up to `max_size`.
### Cancellation and timeouts
Both styles are safe under Phase 27:
```python
# Connection-level — socket-layer timeout, raises OperationalError
conn = await aio.connect(..., read_timeout=30.0)
# Awaitable-level — works because the pool evicts on CancelledError
# and the per-connection wire lock prevents interleaved I/O
await asyncio.wait_for(cur.execute(big_query), timeout=30.0)
```
How it works: every wire op acquires the connection's `_wire_lock` (a re-entrant lock). When an awaitable is cancelled, the underlying `to_thread` worker may still be running — but the pool's `release()` waits up to 5 seconds for the lock. If the worker finishes in time, normal release proceeds (with a transaction rollback if needed). If it doesn't, the connection is evicted instead of recycled. The pool never returns a connection that two threads are touching.
Pick whichever timeout style fits your code; you don't need to choose for safety reasons.
## TLS ## TLS
```python ```python
@ -319,6 +601,29 @@ CREATE DATABASE mydb WITH LOG;
These steps are detailed in the [DECISION_LOG](DECISION_LOG.md) §6.f and §10. These steps are detailed in the [DECISION_LOG](DECISION_LOG.md) §6.f and §10.
## Known limitations
Things that don't work yet (and the workaround when one exists):
| Limitation | Workaround |
|---|---|
| **Named parameters** (`paramstyle="named"` or `dict` parameters) | Use positional `?` / `:1` / `:2`. PEP 249 declares one paramstyle per module. |
| **Binding `ROW(...)` / `SET / MULTISET / LIST`** as a parameter | Decode side surfaces these as `RowValue` / `CollectionValue`. For *writes*, use SQL projections to build them server-side. |
| **GSSAPI / Kerberos / LDAP auth** | Username/password (plain or password obfuscation) only. |
| **Distributed transactions (XA)** | Out of scope for the current driver. |
| **Bulk-load via COPY** | Use `executemany` inside a transaction (≈31K rows/sec — see Performance tips). |
| **`executemany` on SELECT** | Loop `execute(select_sql, params)` — `executemany` is DML-only by design. |
| **Listener failover / sqlhosts groups** | Connect to a specific host:port. Implement failover at the application layer or behind a load balancer. |
| **DATETIME `HOUR TO FRACTION` as a parameter** | Use `DATETIME YEAR TO SECOND` (full datetime). Read side handles all qualifier ranges. |
| **`BlobLocator` / `ClobLocator` as a parameter** | The `read_blob_column` / `write_blob_column` cursor methods cover the BLOB / CLOB I/O cases. Direct locator-as-param will follow when there's a real use case. |
| **UDT-typed parameters / returns in `fast_path_call`** | Scalar params and returns only (INT / SMALLINT / BIGINT / FLOAT / REAL / CHAR / VARCHAR). Complex UDT bind needs the IfxComplexInput protocol layer (~700 lines, deferred). |
Things that work but might surprise you:
* **`autocommit=True` is opt-in.** PEP 249's default is `autocommit=False`, and that's our default too. Many users coming from `IfxPy` (which defaults to autocommit-on) will find this different — and dramatically faster for bulk loads (see Performance tips).
* **`commit()` / `rollback()` on an unlogged DB are silent no-ops.** The server returns sqlcode `-201` to `SQ_BEGIN`; the connection caches that and skips the round-trip on subsequent calls. Same client code works against logged and unlogged databases.
* **`SERIAL` / `BIGSERIAL` columns omitted from INSERT** auto-assign on the server. The auto-assigned value isn't currently exposed via `cursor.lastrowid` (PEP 249 optional surface) — round-trip via `SELECT DBINFO('sqlca.sqlerrd1') FROM systables WHERE tabid=1` if you need it.
## Migration from `IfxPy` / legacy `informixdb` ## Migration from `IfxPy` / legacy `informixdb`
The PEP 249 surface is identical — most code Just Works after switching the import: The PEP 249 surface is identical — most code Just Works after switching the import:

30
pyproject.toml
View File

@ -1,6 +1,6 @@
[project] [project]
name = "informix-db" name = "informix-driver"
version = "2026.05.04" version = "2026.09.03"
description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries." description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries."
readme = "README.md" readme = "README.md"
license = { text = "MIT" } license = { text = "MIT" }
@ -8,7 +8,7 @@ authors = [{ name = "Ryan Malloy", email = "ryan@supported.systems" }]
requires-python = ">=3.10" requires-python = ">=3.10"
keywords = ["informix", "database", "sqli", "db-api", "pep-249", "asyncio", "async"] keywords = ["informix", "database", "sqli", "db-api", "pep-249", "asyncio", "async"]
classifiers = [ classifiers = [
"Development Status :: 4 - Beta", "Development Status :: 5 - Production/Stable",
"Framework :: AsyncIO", "Framework :: AsyncIO",
"Intended Audience :: Developers", "Intended Audience :: Developers",
"License :: OSI Approved :: MIT License", "License :: OSI Approved :: MIT License",
@ -27,9 +27,11 @@ classifiers = [
dependencies = [] dependencies = []
[project.urls] [project.urls]
Homepage = "https://github.com/rsp2k/informix-db" Homepage = "https://informix-driver.warehack.ing"
Documentation = "https://github.com/rsp2k/informix-db/tree/main/docs" Documentation = "https://informix-driver.warehack.ing"
Issues = "https://github.com/rsp2k/informix-db/issues" Source = "https://git.supported.systems/warehack.ing/informix-db"
Issues = "https://git.supported.systems/warehack.ing/informix-db/issues"
Changelog = "https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md"
[project.optional-dependencies] [project.optional-dependencies]
dev = [ dev = [
@ -49,13 +51,15 @@ packages = ["src/informix_db"]
# (the wheel doesn't ship these by default, but the sdist would). # (the wheel doesn't ship these by default, but the sdist would).
# See ~/.claude/rules/python.md for the full pre-publish PII audit playbook. # See ~/.claude/rules/python.md for the full pre-publish PII audit playbook.
exclude = [ exclude = [
"CLAUDE.md", # operator-private context "CLAUDE.md", # operator-private context
".env", ".env.local", ".env.*", ".env", ".env.local", ".env.*",
".mcp.json", # may contain local filesystem paths ".mcp.json", # may contain local filesystem paths
"build/", # decompiled JDBC, downloaded JARs "build/", # decompiled JDBC, downloaded JARs
"audits/", "audits/",
"docs/CAPTURES/", # spike artifacts; tests can re-capture against the dev container "docs/**", # protocol notes / decision log / captures — go to GitHub for the depth
"tests/reference/", # Java reference client — spike infra "docs-site/**", # Astro/Starlight site sources — published at informix-driver.warehack.ing
"tests/reference/**", # Java reference client — spike infra
"tests/benchmarks/.results/**", # pytest-benchmark cache (gitignored, but still ships unless excluded)
".pytest_cache/", ".ruff_cache/", ".mypy_cache/", ".pytest_cache/", ".ruff_cache/", ".mypy_cache/",
"dist/", "*.egg-info/", "dist/", "*.egg-info/",
] ]
@ -93,13 +97,15 @@ addopts = [
"-ra", # short summary for non-passing "-ra", # short summary for non-passing
"--strict-markers", "--strict-markers",
"--strict-config", "--strict-config",
"-m", "not integration", # default: unit-only. Override with: pytest -m integration "-m", "not integration and not benchmark", # default: unit-only. Override with: pytest -m integration / -m benchmark
] ]
markers = [ markers = [
"integration: requires a running Informix container (docker compose up); skipped by default", "integration: requires a running Informix container (docker compose up); skipped by default",
"benchmark: pytest-benchmark performance test; skipped by default. Run with `make bench`.",
] ]
[dependency-groups] [dependency-groups]
dev = [ dev = [
"pytest-asyncio>=1.3.0", "pytest-asyncio>=1.3.0",
"pytest-benchmark>=5.2.3",
] ]

15
src/informix_db/__init__.py
View File

@ -23,6 +23,7 @@ from __future__ import annotations
import ssl import ssl
from importlib.metadata import PackageNotFoundError, version from importlib.metadata import PackageNotFoundError, version
from ._capabilities import ServerCapabilities
from .connections import Connection from .connections import Connection
from .converters import ( from .converters import (
BlobLocator, BlobLocator,
@ -49,6 +50,7 @@ from .pool import (
PoolTimeoutError, PoolTimeoutError,
create_pool, create_pool,
) )
from .rows import Row
# PEP 249 module-level globals # PEP 249 module-level globals
apilevel = "2.0" apilevel = "2.0"
@ -56,9 +58,14 @@ threadsafety = 1 # threads may share the module but not connections
paramstyle = "numeric" # locked in DECISION_LOG.md — matches Informix ESQL/C paramstyle = "numeric" # locked in DECISION_LOG.md — matches Informix ESQL/C
try: try:
__version__ = version("informix-db") # NOTE: the *distribution* is ``informix-driver``; the *module* is
# ``informix_db``. They differ, so this string can't be derived from
# ``__name__``. This looked up the old ``informix-db`` distribution
# until 2026.08.27, which meant __version__ silently reported
# "0.0.0+local" for anyone who installed the renamed package.
__version__ = version("informix-driver")
except PackageNotFoundError: except PackageNotFoundError:
# Editable install or running uninstalled; fall back to a sentinel. # Running from a source checkout without an install.
__version__ = "0.0.0+local" __version__ = "0.0.0+local"
__all__ = [ __all__ = [
@ -79,7 +86,9 @@ __all__ = [
"PoolClosedError", "PoolClosedError",
"PoolTimeoutError", "PoolTimeoutError",
"ProgrammingError", "ProgrammingError",
"Row",
"RowValue", "RowValue",
"ServerCapabilities",
"Warning", "Warning",
"__version__", "__version__",
"apilevel", "apilevel",
@ -104,6 +113,7 @@ def connect(
client_locale: str = "en_US.8859-1", client_locale: str = "en_US.8859-1",
env: dict[str, str] | None = None, env: dict[str, str] | None = None,
autocommit: bool = False, autocommit: bool = False,
row_factory: object | None = None,
tls: bool | ssl.SSLContext = False, tls: bool | ssl.SSLContext = False,
tls_server_hostname: str | None = None, tls_server_hostname: str | None = None,
) -> Connection: ) -> Connection:
@ -146,4 +156,5 @@ def connect(
client_locale=client_locale, client_locale=client_locale,
env=env, env=env,
autocommit=autocommit, autocommit=autocommit,
row_factory=row_factory,
) )

269
src/informix_db/_capabilities.py Normal file
View File

@ -0,0 +1,269 @@
"""Server capability decoding — the SQ_PROTOCOLS feature bitmap.
SQLI does not have a single "protocol version". It has a 64-bit feature
bitmap that client and server negotiate at connect time, and several bits
in it change **wire framing**, not just feature availability. Get one of
those wrong and the row decoder desyncs.
We already perform the negotiation (``Connection._init_session`` sends
``SQ_PROTOCOLS`` with the same 8-byte client offer the IBM JDBC driver
uses). Until now we discarded the server's reply and hardcoded the modern
framing. This module decodes the reply so the assumptions can at least be
*checked* instead of merely believed.
Two independent sources feed the bitmap, mirroring
``IfxSqliConnect.getServerVer`` / ``enhancedProtocolMechanism``:
1. ``Cap_1`` from the login (SLTYPE_CONACC) response. This is the client's
own protocol level echoed back — 316 means "speaks the enhanced
protocol". It is **not** a server version number, which is why JDBC
tests ``== 316`` rather than ``>=``.
2. The 8-byte mask from the SQ_PROTOCOLS reply.
When ``Cap_1`` is non-zero, JDBC unconditionally pre-sets bits
{0, 2, 3, 4, 49, 51} *before* applying the mask, and a Java ``BitSet.set``
only ever sets — never clears. So those six bits are true on any modern
server regardless of what the mask says. ``isUSVER`` (bit 2) is one of
them, which is why it never varies in practice and is a dead end when
chasing version-specific behaviour.
Bit numbering is MSB-first within each byte: byte 0 bit 0x80 is bit 0,
byte 0 bit 0x01 is bit 7, byte 1 bit 0x80 is bit 8, and so on.
A note on length. The client offer is 8 bytes, but every server we have
tested replies with **nine**. JDBC's ``enhancedProtocolMechanism``
dispatches on ``switch (i) case 0..7`` and silently discards anything
past byte 7, so its ``BitSet(64)`` never sees the ninth byte. We decode
it — bits 64-71 — because it turns out to be the only part of the mask
that differs between releases::
Informix 15.0.1.0.3 bdbe9ffe7fb7ffef ff
Informix 14.10.FC7W1 bdbe9ffe7fb7ffef f8
Informix 12.10.FC12W1DE bdbe9ffe7fb7ffef f0
^^^^^^^^^^^^^^^^ identical
The first 64 bits being byte-identical across those three releases is
why they speak an indistinguishable SQLI dialect. Bits 64+ have no known
meaning here; they are surfaced for diagnostics, not branched on.
"""
from __future__ import annotations
from dataclasses import dataclass, field
# The client's capability offer. Byte-for-byte the value IBM's JDBC driver
# sends (``IfxSqliConnect.clientProtocols``), replayed verbatim — the bits
# are a fixed constant there too, never computed.
CLIENT_PROTOCOLS_MASK = bytes.fromhex("fffc7ffc3c8caa97")
# ``Cap_1`` value meaning "enhanced protocol negotiation applies". JDBC
# compares with equality, not >=, because the server echoes the client's
# own declared level rather than reporting its own.
ENHANCED_PROTOCOL_CAP = 316
# Bits JDBC pre-sets for any non-zero Cap_1, before the mask is applied.
_PRESET_BITS = frozenset({0, 2, 3, 4, 49, 51})
# Named bits. Only the ones we can justify from the decompiled JDBC are
# listed; the mask has 64 and most are irrelevant to us.
BIT_USVER = 2 # isUSVER — always true on modern servers
BIT_LONG_ID_A = 10 # isLongID is bit 10 OR bit 18
BIT_LONG_ID_B = 18
BIT_VARCHAR_VAR_LEN = 13 # gated further by bit 40 + client ifxPADVARCHAR
BIT_DESCRIBE_INPUT = 34
BIT_LVARCHAR_GT_2K = 37
BIT_PAD_VARCHAR_GATE = 40
BIT_FP_DESCRIBE = 45
BIT_AUTO_GENERATED_KEYS = 48
BIT_CUR_SESS_INFO = 49
BIT_FOUR_BYTE_OFFSET = 50 # describe: string-table + field-index widths
BIT_GLS = 51
BIT_NAMED_PARAMETERS = 52
BIT_BIGINT = 54
BIT_SAVEPOINT = 56
BIT_SQ_BATCH = 61
BIT_REMOVE_64K_LIMIT = 62 # 4-byte vs 2-byte length prefixes
BIT_TWO_GB_FETCH_BUFFER = 63
def _decode_bits(mask: bytes) -> set[int]:
"""Expand a big-endian, MSB-first bit mask into a set of bit numbers."""
bits: set[int] = set()
for byte_index, value in enumerate(mask):
for offset in range(8):
if value & (0x80 >> offset):
bits.add(byte_index * 8 + offset)
return bits
@dataclass(frozen=True)
class ServerCapabilities:
"""Decoded SQ_PROTOCOLS feature bitmap plus login-response metadata.
``bits`` is the authoritative set; the named properties are readability
sugar over it. ``raw_mask`` is kept so a bug report can include the
exact bytes without needing a packet capture.
"""
bits: frozenset[int] = field(default_factory=frozenset)
raw_mask: bytes = b""
cap_1: int = 0
cap_2: int = 0
cap_3: int = 0
server_version: str = ""
serial_number: str = ""
applid_name: str = ""
@classmethod
def from_wire(
cls,
mask: bytes,
*,
cap_1: int = 0,
cap_2: int = 0,
cap_3: int = 0,
server_version: str = "",
serial_number: str = "",
applid_name: str = "",
) -> ServerCapabilities:
bits = _decode_bits(mask)
if cap_1 != 0:
# Mirrors JDBC: pre-set bits are OR-ed in and never cleared.
bits |= _PRESET_BITS
return cls(
bits=frozenset(bits),
raw_mask=bytes(mask),
cap_1=cap_1,
cap_2=cap_2,
cap_3=cap_3,
server_version=server_version,
serial_number=serial_number,
applid_name=applid_name,
)
def has(self, bit: int) -> bool:
return bit in self.bits
# -- named capabilities ------------------------------------------------
@property
def usver(self) -> bool:
return self.has(BIT_USVER)
@property
def long_id(self) -> bool:
return self.has(BIT_LONG_ID_A) or self.has(BIT_LONG_ID_B)
@property
def varchar_var_len(self) -> bool:
"""VARCHAR carries a 1-byte length prefix rather than being padded
to its full declared width.
JDBC additionally consults the client-side ``ifxPADVARCHAR``
setting; we never enable padding, so the client half is constant.
"""
return self.has(BIT_VARCHAR_VAR_LEN)
@property
def four_byte_offset(self) -> bool:
"""Describe records use 4-byte string-table size and field indices
rather than 2-byte."""
return self.has(BIT_FOUR_BYTE_OFFSET)
@property
def bigint(self) -> bool:
return self.has(BIT_BIGINT)
@property
def lvarchar_gt_2k(self) -> bool:
return self.has(BIT_LVARCHAR_GT_2K)
@property
def remove_64k_limit(self) -> bool:
"""4-byte length prefixes in the fast-path/SQ_FILE paths."""
return self.has(BIT_REMOVE_64K_LIMIT)
@property
def two_gb_fetch_buffer(self) -> bool:
return self.has(BIT_TWO_GB_FETCH_BUFFER)
@property
def gls(self) -> bool:
return self.has(BIT_GLS)
@property
def describe_input(self) -> bool:
return self.has(BIT_DESCRIBE_INPUT)
@property
def named_parameters(self) -> bool:
return self.has(BIT_NAMED_PARAMETERS)
@property
def savepoint(self) -> bool:
return self.has(BIT_SAVEPOINT)
@property
def enhanced_protocol(self) -> bool:
return self.cap_1 == ENHANCED_PROTOCOL_CAP
# -- assumption checking ----------------------------------------------
def violated_assumptions(self) -> list[str]:
"""Framing assumptions this driver hardcodes that the server
contradicts.
Each entry names a place where we emit or parse a fixed wire shape
that is actually capability-gated. An empty list means every
hardcoded choice matches what the server negotiated.
This is a diagnostic, not a guarantee: it only covers bits we know
to be framing-relevant from the decompiled JDBC. It exists so that
a server we have never tested produces a loud, specific warning
instead of silently corrupted rows.
"""
problems: list[str] = []
if not self.four_byte_offset:
problems.append(
"server did not negotiate 4-byte describe offsets "
f"(bit {BIT_FOUR_BYTE_OFFSET}), but parse_describe reads "
"4-byte string-table size and field indices; column "
"metadata will be misparsed"
)
if not self.varchar_var_len:
problems.append(
f"server did not negotiate variable-length VARCHAR "
f"(bit {BIT_VARCHAR_VAR_LEN}); VARCHAR values are padded to "
"the declared column width on the wire, but the row decoder "
"expects a 1-byte length prefix"
)
if not self.remove_64k_limit:
problems.append(
f"server did not negotiate the 64K limit removal "
f"(bit {BIT_REMOVE_64K_LIMIT}); fast-path routine signatures "
"are emitted with a 4-byte length prefix that the server "
"expects to be 2-byte"
)
return problems
def __repr__(self) -> str:
named = [
name
for name, value in (
("usver", self.usver),
("4byte_offset", self.four_byte_offset),
("varchar_var_len", self.varchar_var_len),
("bigint", self.bigint),
("long_id", self.long_id),
("lvarchar>2k", self.lvarchar_gt_2k),
("remove_64k", self.remove_64k_limit),
("gls", self.gls),
)
if value
]
return (
f"ServerCapabilities(cap_1={self.cap_1}, "
f"mask={self.raw_mask.hex()}, "
f"version={self.server_version!r}, "
f"flags={'|'.join(named) or 'none'})"
)

18
src/informix_db/_fastpath.py
View File

@ -39,9 +39,18 @@ def build_get_routine_pdu(signature: str) -> bytes:
``[short SQ_GETROUTINE=101][byte isRoutineById=0][int sigLen] ``[short SQ_GETROUTINE=101][byte isRoutineById=0][int sigLen]
[sig bytes][pad if odd][short fparamFlag=0][short SQ_EOT=12]`` [sig bytes][pad if odd][short fparamFlag=0][short SQ_EOT=12]``
JDBC's ``getJavaToIfxCharBytes`` uses 4-byte length prefix on JDBC's ``getJavaToIfxCharBytes`` uses a 4-byte length prefix when
modern servers (``isRemove64KLimitSupported``). We always emit the ``isRemove64KLimitSupported()`` (capability bit 62) is set, and a
4-byte form — works against 12.10+ unequivocally. 2-byte prefix otherwise. We always emit the 4-byte form.
Verified correct on 15.0.1.0.3DE and 12.10.FC12W1DE (2026-05-08) —
the fast-path RPC tests pass on both. Not yet checked on 14.10.
An earlier version of this docstring asserted 12.10+ compatibility
"unequivocally" before anyone had run it against 12.10. The claim
happened to hold, but it was a guess. The bit is negotiated via
``SQ_PROTOCOLS``, which this driver does not yet send, so this
remains an assumption on any server we haven't measured.
""" """
sig_bytes = signature.encode("iso-8859-1") sig_bytes = signature.encode("iso-8859-1")
sig_len = len(sig_bytes) sig_len = len(sig_bytes)
@ -77,6 +86,7 @@ def build_exfp_routine_pdu(
db_name: str, db_name: str,
handle: int, handle: int,
params: tuple, params: tuple,
encoding: str = "iso-8859-1",
) -> bytes: ) -> bytes:
"""Build a ``SQ_EXFPROUTINE`` request PDU. """Build a ``SQ_EXFPROUTINE`` request PDU.
@ -106,7 +116,7 @@ def build_exfp_routine_pdu(
if value is None: if value is None:
out.extend(struct.pack("!hhh", 0, -1, 0)) out.extend(struct.pack("!hhh", 0, -1, 0))
continue continue
ifx_type, prec, raw = encode_param(value) ifx_type, prec, raw = encode_param(value, encoding=encoding)
out.extend(struct.pack("!hhh", ifx_type, 0, prec)) out.extend(struct.pack("!hhh", ifx_type, 0, prec))
out.extend(raw) out.extend(raw)
if len(raw) & 1: if len(raw) & 1:

10
src/informix_db/_messages.py
View File

@ -35,6 +35,16 @@ class MessageType(IntEnum):
SQ_RELEASE = 11 SQ_RELEASE = 11
SQ_NDESCRIBE = 22 # numerical describe — request column metadata after a PREPARE/COMMAND SQ_NDESCRIBE = 22 # numerical describe — request column metadata after a PREPARE/COMMAND
SQ_WANTDONE = 49 # request a SQ_DONE completion notification SQ_WANTDONE = 49 # request a SQ_DONE completion notification
# Phase 18: server-side scrollable cursor.
SQ_SFETCH = 23 # scroll-fetch: ``[short SFETCH][short scrolltype]
# [int target][short bufSize]``. scrolltype values
# per JDBC IfxSqli.getaRow: 1=NEXT, 4=LAST, 6=ABSOLUTE.
SQ_SCROLL = 24 # cursor-open modifier — emitted *before* SQ_OPEN
# to mark the cursor as scrollable. Server keeps
# the result set materialized for random access.
SQ_TUPID = 25 # server response tag carrying the row's 1-indexed
# position. Body: ``[int tupleId]``. Sent before
# SQ_TUPLE in scrollable-cursor responses.
# --- Per-PDU framing --- # --- Per-PDU framing ---
SQ_EOT = 12 # end-of-transmission / flush marker; ends every PDU SQ_EOT = 12 # end-of-transmission / flush marker; ends every PDU

136
src/informix_db/_protocol.py
View File

@ -29,6 +29,29 @@ class ProtocolError(Exception):
"""Raised when wire bytes can't be parsed (truncated stream, bad framing).""" """Raised when wire bytes can't be parsed (truncated stream, bad framing)."""
# Every way a wire read/write can fail, as one tuple.
#
# This exists because the tuple was written out by hand in six places and
# drifted: four of them caught ``(ProtocolError, OSError)`` — which
# ``IfxSocket`` never raises. It converts *every* socket failure, including
# clean EOF, into ``OperationalError``, and that is a ``DatabaseError``, not
# an ``OSError``. So those four handlers could not catch the thing they
# existed to catch. The worst of them sat in a ``weakref.finalize`` callback,
# where the escaping exception is printed to stderr and swallowed, leaving a
# desynchronised connection to be returned to the pool marked healthy.
#
# Import this rather than re-typing the members. A tuple in one place cannot
# drift out of sync with itself.
def _wire_error_types() -> tuple[type[BaseException], ...]:
# Imported lazily: exceptions.py must stay free of protocol imports.
from .exceptions import InterfaceError, OperationalError
return (ProtocolError, OSError, OperationalError, InterfaceError)
WIRE_ERRORS: tuple[type[BaseException], ...] = _wire_error_types()
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Writer # Writer
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -209,6 +232,119 @@ class IfxStreamReader:
return raw.decode(encoding) return raw.decode(encoding)
class BufferedSocketReader(IfxStreamReader):
"""Phase 39: thin parser-view over the ``IfxSocket``'s persistent
read-ahead buffer.
The buffer itself (``_recv_buf`` + ``_recv_pos``) lives on the
``IfxSocket`` so it survives across reader instances. This is
critical for the pipelined-executemany path (Phase 33), which
streams N responses back-to-back — a per-reader buffer would lose
pre-fetched bytes when one reader is destroyed and the next is
created.
Why this is faster than ``_SocketReader``:
* **One ``recv()`` per ~64 KB** instead of one per field. A 100k-row
SELECT calls ``read_exact`` ~450k times today; with this reader,
``recv`` fires ~hundreds of times instead.
* **No ``bytes.join``.** The bytearray IS the buffer; reads are
slices, not chunk-joins.
* **``struct.unpack_from(buf, offset)``** for fixed-width ints — reads
directly from the bytearray at offset, avoiding the intermediate
slice the legacy reader created.
Each method reads through ``self._sock._recv_buf`` directly. The
``IfxSocket`` exposes ``fill_recv_buf(n)`` to ensure ``n`` bytes
are available from the current cursor position; this method
handles ``recv()``, compaction, and EOF detection.
``read_exact`` returns a fresh ``bytes`` (not a memoryview) —
copying is cheap and decouples consumer lifetime from buffer
compaction, removing a class of use-after-compact bugs at
near-zero cost.
"""
__slots__ = ("_sock",)
def __init__(self, sock):
# Skip super().__init__ to avoid the unused BytesIO allocation
# — we override every method that touches ``self._source``.
self._sock = sock
def read_exact(self, n: int) -> bytes:
if n <= 0:
return b""
sock = self._sock
sock.fill_recv_buf(n)
buf = sock._recv_buf
pos = sock._recv_pos
end = pos + n
out = bytes(buf[pos:end])
sock._recv_pos = end
return out
def read_byte(self) -> int:
sock = self._sock
sock.fill_recv_buf(1)
b = sock._recv_buf[sock._recv_pos]
sock._recv_pos += 1
return b
def skip(self, n: int) -> None:
# The base reader's skip delegates to read_exact, which returns
# b"" for n <= 0. This one advances the cursor arithmetically, so
# without the guard a negative n *rewinds* it — silently
# re-decoding bytes already consumed as if they were the next
# field. Same divergence as read_exact, which does guard.
if n <= 0:
return
sock = self._sock
sock.fill_recv_buf(n)
sock._recv_pos += n
def read_padded(self, n: int) -> bytes:
out = self.read_exact(n)
if n & 1:
self.skip(1)
return out
def read_short(self) -> int:
sock = self._sock
sock.fill_recv_buf(2)
v = _FMT_SHORT.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 2
return v
def read_int(self) -> int:
sock = self._sock
sock.fill_recv_buf(4)
v = _FMT_INT.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 4
return v
def read_long_bigint(self) -> int:
sock = self._sock
sock.fill_recv_buf(8)
v = _FMT_LONG.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 8
return v
def read_real(self) -> float:
sock = self._sock
sock.fill_recv_buf(4)
v = _FMT_FLOAT.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 4
return v
def read_double(self) -> float:
sock = self._sock
sock.fill_recv_buf(8)
v = _FMT_DOUBLE.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 8
return v
def make_pdu_writer() -> tuple[IfxStreamWriter, BytesIO]: def make_pdu_writer() -> tuple[IfxStreamWriter, BytesIO]:
"""Convenience: create a writer backed by a fresh in-memory buffer. """Convenience: create a writer backed by a fresh in-memory buffer.

942
src/informix_db/_resultset.py
View File

File diff suppressed because it is too large Load Diff

127
src/informix_db/_socket.py
View File

@ -17,11 +17,35 @@ the rest of the protocol layer.
from __future__ import annotations from __future__ import annotations
import contextlib import contextlib
import os
import socket import socket
import ssl import ssl
from ._protocol import ProtocolError
from .exceptions import InterfaceError, OperationalError from .exceptions import InterfaceError, OperationalError
def _max_read_bytes() -> int:
"""Ceiling for a single length-prefixed read, from IFX_MAX_READ_BYTES.
256 MiB by default: comfortably above any row, string table, or blob
chunk a real query produces, and far below the values a desynced
stream invents. A genuinely larger single value is possible (a very
large TEXT column read in one go), which is why the knob exists.
"""
raw = os.environ.get("IFX_MAX_READ_BYTES")
if raw:
try:
value = int(raw)
except ValueError:
value = 0
if value > 0:
return value
return 256 * 1024 * 1024
MAX_READ_BYTES = _max_read_bytes()
# A ``tls`` parameter to ``IfxSocket`` accepts: # A ``tls`` parameter to ``IfxSocket`` accepts:
# False (default) — plain TCP # False (default) — plain TCP
# True — TLS with verification disabled (dev / self-signed) # True — TLS with verification disabled (dev / self-signed)
@ -42,7 +66,16 @@ def _make_default_dev_context() -> ssl.SSLContext:
class IfxSocket: class IfxSocket:
"""Owns a connected TCP socket and provides exact-N read/write.""" """Owns a connected TCP socket and provides exact-N read/write.
Phase 39 also owns the read-ahead buffer for ``BufferedSocketReader``.
The buffer (``_recv_buf`` + ``_recv_pos``) is connection-scoped, not
reader-scoped — readers are short-lived per-PDU views, but pipelined
responses (e.g., Phase 33's pipelined ``executemany``) can stream
multiple responses across reader boundaries. Persisting the buffer
here means we never throw away pre-fetched bytes when one reader
is destroyed and the next one is created.
"""
def __init__( def __init__(
self, self,
@ -58,6 +91,11 @@ class IfxSocket:
self._port = port self._port = port
self._read_timeout = read_timeout self._read_timeout = read_timeout
self._sock: socket.socket | None = None self._sock: socket.socket | None = None
# Phase 39 read-ahead buffer. Empty until a BufferedSocketReader
# touches us; subsequent readers share this buffer state.
self._recv_buf: bytearray = bytearray()
self._recv_pos: int = 0
self._recv_size: int = 65536
try: try:
sock = socket.create_connection((host, port), timeout=connect_timeout) sock = socket.create_connection((host, port), timeout=connect_timeout)
@ -110,10 +148,42 @@ class IfxSocket:
raise OperationalError(f"write failed: {e}") from e raise OperationalError(f"write failed: {e}") from e
def read_exact(self, n: int) -> bytes: def read_exact(self, n: int) -> bytes:
"""Read exactly ``n`` bytes or raise on EOF / timeout.""" """Read exactly ``n`` bytes or raise on EOF / timeout.
Consumes the read-ahead buffer before touching the socket. This
matters because two readers share one stream: ``BufferedSocketReader``
fills ``_recv_buf`` (over-reading by design, up to ``_recv_size``),
while ``Connection._drain_to_eot``, ``_raise_sq_err`` and the
login path call this method directly. Recv'ing here while bytes
sat unconsumed in the buffer would skip them, and skipped bytes
in a length-framed protocol don't announce themselves — the next
read lands mid-field and every subsequent one is wrong.
No workload triggers it today: the server sends one response per
request, so recv returns exactly that response and the buffered
reader consumes all of it before control returns here. That is a
property of the traffic, not of the code. Pipelined executemany
already puts multiple responses in flight, and the buffer is
connection-scoped precisely so read-ahead can cross response
boundaries. Making the two paths agree by construction costs one
branch on a cold path.
"""
if self._sock is None: if self._sock is None:
raise InterfaceError("socket is closed") raise InterfaceError("socket is closed")
if n <= 0:
return b""
wanted = n
chunks: list[bytes] = [] chunks: list[bytes] = []
buffered = len(self._recv_buf) - self._recv_pos
if buffered > 0:
take = min(buffered, n)
chunks.append(
bytes(self._recv_buf[self._recv_pos : self._recv_pos + take])
)
self._recv_pos += take
n -= take
if n == 0:
return chunks[0]
remaining = n remaining = n
while remaining > 0: while remaining > 0:
try: try:
@ -124,12 +194,63 @@ class IfxSocket:
if not chunk: if not chunk:
self._force_close() self._force_close()
raise OperationalError( raise OperationalError(
f"server closed connection mid-read (wanted {n} bytes, got {n - remaining})" f"server closed connection mid-read "
f"(wanted {wanted} bytes, got {wanted - remaining})"
) )
chunks.append(chunk) chunks.append(chunk)
remaining -= len(chunk) remaining -= len(chunk)
return b"".join(chunks) return b"".join(chunks)
def fill_recv_buf(self, need: int) -> None:
"""Phase 39: ensure the read-ahead buffer holds ``need`` bytes
forward of ``_recv_pos``, recv'ing from the socket as needed.
Compaction: when the read cursor has advanced past
``_recv_size`` bytes, slice off the consumed prefix in place.
That bounds memory at roughly ``2 * recv_size`` while still
amortizing recv calls over many fields.
"""
if self._sock is None:
raise InterfaceError("socket is closed")
if need > MAX_READ_BYTES:
# ``need`` is almost always a length field straight off the
# wire. Trusting it means a corrupt or desynced stream turns
# into an allocation of whatever that field happened to say —
# a garbage 0x7FFFFFFF reads as a 2 GB request, and the loop
# below sits in recv until the read timeout while the buffer
# grows. Refusing names the number instead, which is the
# diagnostic: a length that absurd means framing is already
# lost upstream, not that the row is genuinely that big.
raise ProtocolError(
f"refusing to read {need} bytes in one field (limit "
f"{MAX_READ_BYTES}). Either the wire has desynced, or a "
f"single value really is this large — raise the limit "
f"with the IFX_MAX_READ_BYTES environment variable."
)
avail = len(self._recv_buf) - self._recv_pos
if avail >= need:
return
if self._recv_pos > self._recv_size:
del self._recv_buf[: self._recv_pos]
self._recv_pos = 0
avail = len(self._recv_buf)
target = max(need, self._recv_size)
sock = self._sock
while avail < need:
try:
chunk = sock.recv(target - avail)
except OSError as e:
self._force_close()
raise OperationalError(f"read failed: {e}") from e
if not chunk:
self._force_close()
raise OperationalError(
f"server closed connection mid-read "
f"(wanted {need} bytes, got {avail})"
)
self._recv_buf.extend(chunk)
avail += len(chunk)
def close(self) -> None: def close(self) -> None:
"""Close the socket. Idempotent and never raises.""" """Close the socket. Idempotent and never raises."""
if self._sock is None: if self._sock is None:

233
src/informix_db/aio.py
View File

@ -15,6 +15,29 @@ event loop yields during each ``await``; a worker thread does the
actual socket I/O. Only differs for thousands-of-concurrent-connections actual socket I/O. Only differs for thousands-of-concurrent-connections
workloads, which need native-async (Phase 17 if anyone asks). workloads, which need native-async (Phase 17 if anyone asks).
.. note::
**Cancellation handling (Phase 27).** ``asyncio.to_thread`` does
not interrupt the underlying worker thread when the awaitable is
cancelled — the thread keeps running until the sync call returns
naturally. The driver handles this in two ways:
1. Every wire operation acquires the connection's ``_wire_lock``
(an ``RLock``). Two threads — including a still-running worker
and the pool's release path — cannot interleave bytes on the
socket; the second blocks until the first releases.
2. The async pool's ``connection()`` context manager evicts the
connection (``broken=True``) on ``CancelledError`` /
``TimeoutError``, so a partially-cancelled query never returns
to the idle list. ``pool.release()`` waits up to 5 seconds for
the wire lock; if the worker is still busy past that, the
connection is evicted instead of recycled.
Net effect: ``asyncio.wait_for`` around ``aio`` DB calls is safe.
The connection is either successfully released (worker finished
in time) or evicted (worker exceeded the timeout); never
returned to the pool in a poisoned state.
Usage:: Usage::
import asyncio import asyncio
@ -39,7 +62,10 @@ from __future__ import annotations
import asyncio import asyncio
import contextlib import contextlib
import functools import functools
import threading
import weakref
from collections.abc import AsyncIterator, Awaitable, Callable from collections.abc import AsyncIterator, Awaitable, Callable
from concurrent.futures import ThreadPoolExecutor
from typing import Any, TypeVar from typing import Any, TypeVar
from . import connect as _sync_connect from . import connect as _sync_connect
@ -56,6 +82,61 @@ def _to_thread(fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
return asyncio.to_thread(fn, *args, **kwargs) return asyncio.to_thread(fn, *args, **kwargs)
def _run_on(
executor: ThreadPoolExecutor,
fn: Callable[..., T],
*args: Any,
**kwargs: Any,
) -> Awaitable[T]:
"""Await ``fn`` on a specific executor rather than the loop's default."""
loop = asyncio.get_running_loop()
return loop.run_in_executor(executor, functools.partial(fn, *args, **kwargs))
def _connection_executor(conn: _SyncConnection) -> ThreadPoolExecutor:
"""The dedicated worker thread for one connection, created on demand.
``asyncio.to_thread`` runs on the event loop's default executor, which
the whole process shares and which is sized from the CPU count —
``min(32, cpu_count + 4)``, so six threads on a two-CPU container.
Two consequences, both silent.
A cancelled await does not stop its worker. ``asyncio.to_thread``
cannot interrupt the thread, so it keeps running the wire call until
the read timeout expires. Cancellation is ordinary in a web app — a
client disconnect cancels the request task — so a handful of them
pins every thread in the shared pool and unrelated ``to_thread`` work
anywhere else in the process stops dead. Measured: six cancelled
calls against a six-worker default executor starve an unrelated
``to_thread`` indefinitely.
And pool concurrency was capped by the same number without saying so.
A pool with ``max_size=20`` on a two-CPU box ran six queries at once.
One thread per connection is the right size rather than a compromise:
the sync connection serializes every wire operation on its own lock,
so a second thread could do nothing but wait for the first. It also
rules out a deadlock that a shared pool-sized executor invites — with
N threads and N connections, N tasks blocked in ``acquire`` occupy
every thread while the connection they are waiting for is held by a
task that now needs a thread of its own to finish and release it.
The executor lives on the sync connection so it survives being
returned to the pool and handed out again. The finalizer is the
backstop for a connection dropped without ``close()``: a
``ThreadPoolExecutor`` that is never shut down leaves its worker
parked on the work queue for the life of the process.
"""
executor = getattr(conn, "_async_executor", None)
if executor is None:
executor = ThreadPoolExecutor(
max_workers=1, thread_name_prefix="informix-conn"
)
conn._async_executor = executor
weakref.finalize(conn, executor.shutdown, wait=False)
return executor
class AsyncCursor: class AsyncCursor:
"""Async wrapper over a sync :class:`Cursor`. Each I/O call awaits """Async wrapper over a sync :class:`Cursor`. Each I/O call awaits
a thread-offloaded version of the sync operation. a thread-offloaded version of the sync operation.
@ -65,10 +146,14 @@ class AsyncCursor:
paying the thread-hop cost. paying the thread-hop cost.
""" """
__slots__ = ("_cur",) __slots__ = ("_cur", "_run")
def __init__(self, cur: _SyncCursor): def __init__(self, cur: _SyncCursor, run: Callable[..., Awaitable[Any]]):
self._cur = cur self._cur = cur
# The owning connection's runner, so cursor I/O lands on that
# connection's dedicated thread rather than the shared default
# executor. See _connection_executor.
self._run = run
# -- Pass-through synchronous attributes (no I/O) --------------------- # -- Pass-through synchronous attributes (no I/O) ---------------------
@ -97,32 +182,32 @@ class AsyncCursor:
async def execute( async def execute(
self, operation: str, parameters: Any = None self, operation: str, parameters: Any = None
) -> None: ) -> None:
await _to_thread(self._cur.execute, operation, parameters) await self._run(self._cur.execute, operation, parameters)
async def executemany( async def executemany(
self, operation: str, seq_of_parameters: Any self, operation: str, seq_of_parameters: Any
) -> None: ) -> None:
await _to_thread( await self._run(
self._cur.executemany, operation, list(seq_of_parameters) self._cur.executemany, operation, list(seq_of_parameters)
) )
async def fetchone(self) -> tuple | None: async def fetchone(self) -> tuple | None:
return await _to_thread(self._cur.fetchone) return await self._run(self._cur.fetchone)
async def fetchmany(self, size: int | None = None) -> list[tuple]: async def fetchmany(self, size: int | None = None) -> list[tuple]:
return await _to_thread(self._cur.fetchmany, size) return await self._run(self._cur.fetchmany, size)
async def fetchall(self) -> list[tuple]: async def fetchall(self) -> list[tuple]:
return await _to_thread(self._cur.fetchall) return await self._run(self._cur.fetchall)
async def close(self) -> None: async def close(self) -> None:
await _to_thread(self._cur.close) await self._run(self._cur.close)
# Phase 10/11 BLOB helpers (preserve the sync API surface) # Phase 10/11 BLOB helpers (preserve the sync API surface)
async def read_blob_column( async def read_blob_column(
self, sql: str, params: tuple = () self, sql: str, params: tuple = ()
) -> bytes | None: ) -> bytes | None:
return await _to_thread(self._cur.read_blob_column, sql, params) return await self._run(self._cur.read_blob_column, sql, params)
async def write_blob_column( async def write_blob_column(
self, self,
@ -132,7 +217,7 @@ class AsyncCursor:
*, *,
clob: bool = False, clob: bool = False,
) -> None: ) -> None:
await _to_thread( await self._run(
functools.partial( functools.partial(
self._cur.write_blob_column, self._cur.write_blob_column,
sql, blob_data, params, clob=clob, sql, blob_data, params, clob=clob,
@ -154,14 +239,23 @@ class AsyncCursor:
class AsyncConnection: class AsyncConnection:
"""Async wrapper over a sync :class:`Connection`.""" """Async wrapper over a sync :class:`Connection`."""
__slots__ = ("_conn",) __slots__ = ("_conn", "_executor")
def __init__(self, conn: _SyncConnection): def __init__(self, conn: _SyncConnection):
self._conn = conn self._conn = conn
self._executor = _connection_executor(conn)
def _run(self, fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
"""Run a blocking connection call on this connection's own thread."""
return _run_on(self._executor, fn, *args, **kwargs)
@classmethod @classmethod
async def connect(cls, *args: Any, **kwargs: Any) -> AsyncConnection: async def connect(cls, *args: Any, **kwargs: Any) -> AsyncConnection:
"""Open a connection. Same parameters as :func:`informix_db.connect`.""" """Open a connection. Same parameters as :func:`informix_db.connect`.
The connect itself still goes to the default executor -- there is
no connection yet to own a thread, and it is one bounded call.
"""
sync_conn = await _to_thread( sync_conn = await _to_thread(
functools.partial(_sync_connect, *args, **kwargs) functools.partial(_sync_connect, *args, **kwargs)
) )
@ -172,22 +266,46 @@ class AsyncConnection:
return self._conn.closed return self._conn.closed
async def cursor(self) -> AsyncCursor: async def cursor(self) -> AsyncCursor:
sync_cur = await _to_thread(self._conn.cursor) sync_cur = await self._run(self._conn.cursor)
return AsyncCursor(sync_cur) return AsyncCursor(sync_cur, self._run)
async def commit(self) -> None: async def commit(self) -> None:
await _to_thread(self._conn.commit) await self._run(self._conn.commit)
async def rollback(self) -> None: async def rollback(self) -> None:
await _to_thread(self._conn.rollback) await self._run(self._conn.rollback)
async def close(self) -> None: async def close(self) -> None:
await _to_thread(self._conn.close) """Close the connection and stop the thread that served it.
One thread per connection is only affordable if the thread goes
away with the connection. ``ThreadPoolExecutor`` workers park on
the work queue rather than exiting when idle, so an executor that
is never shut down leaks its thread for the life of the process —
the ``weakref.finalize`` in ``_connection_executor`` is a backstop
for connections dropped without ``close()``, not a substitute for
closing here.
``wait=False`` because we are on the event loop: the worker has
just finished the close and needs no waiting, and blocking the
loop to confirm that would be the one thing this module exists to
avoid.
"""
try:
await self._run(self._conn.close)
finally:
self._executor.shutdown(wait=False)
# Drop the reference too. A shut-down executor rejects new
# work, so leaving it attached would turn any later wrap of
# this sync connection into a RuntimeError rather than a
# fresh thread.
with contextlib.suppress(AttributeError):
del self._conn._async_executor
async def fast_path_call( async def fast_path_call(
self, signature: str, *params: object self, signature: str, *params: object
) -> list[object]: ) -> list[object]:
return await _to_thread(self._conn.fast_path_call, signature, *params) return await self._run(self._conn.fast_path_call, signature, *params)
# Async context-manager support # Async context-manager support
async def __aenter__(self) -> AsyncConnection: async def __aenter__(self) -> AsyncConnection:
@ -226,25 +344,96 @@ class AsyncConnectionPool:
return self._pool.idle_count return self._pool.idle_count
async def acquire(self, timeout: float | None = None) -> AsyncConnection: async def acquire(self, timeout: float | None = None) -> AsyncConnection:
sync_conn = await _to_thread(self._pool.acquire, timeout) """Acquire a connection, surviving cancellation of the waiter.
``asyncio.to_thread`` cannot interrupt its worker. If the task
awaiting an acquire is cancelled while the worker is still
blocked waiting for a free connection, the worker eventually
succeeds and hands back a connection **that nobody owns** — it
is checked out of the pool and never returned. Repeat that and
the pool starves to death.
This is not hypothetical for anyone serving HTTP: a client
disconnecting cancels the request task, and under load those
cancellations land precisely while waiting for a connection. The
pool then dies one slot at a time, and only under load.
So the inner future is shielded — cancelling the caller must not
orphan a result we still need to see — and if the caller does go
away, a callback returns whatever the worker produced to the
pool. ``add_done_callback`` fires immediately when the future has
already resolved, so the "worker finished a moment before the
cancellation" race is covered by the same code path.
"""
fut = asyncio.ensure_future(_to_thread(self._pool.acquire, timeout))
try:
sync_conn = await asyncio.shield(fut)
except asyncio.CancelledError:
fut.add_done_callback(self._return_orphan)
raise
return AsyncConnection(sync_conn) return AsyncConnection(sync_conn)
def _return_orphan(self, fut: asyncio.Future) -> None:
"""Give back a connection whose acquirer was cancelled.
Runs on the event loop, so it must not block: ``release`` takes
the connection's wire lock and can wait. Hand off to a short-lived
daemon thread rather than the running loop — this path also has to
work while the loop is shutting down, which is exactly when
``create_task`` is unavailable.
"""
if fut.cancelled() or fut.exception() is not None:
return
conn = fut.result()
def _release() -> None:
# broken=False: the connection was never handed to anyone, so
# its wire is untouched and it is safe to reuse. Evicting here
# would trade a leak for needless reconnect churn.
with contextlib.suppress(Exception):
self._pool.release(conn, broken=False)
threading.Thread(target=_release, daemon=True).start()
async def release( async def release(
self, conn: AsyncConnection, *, broken: bool = False self, conn: AsyncConnection, *, broken: bool = False
) -> None: ) -> None:
await _to_thread( # On the connection's own thread, not the default executor. That
functools.partial(self._pool.release, conn._conn, broken=broken) # thread is idle by definition — the caller is done with the
# connection — and it keeps the release off a shared pool that
# tasks blocked in ``acquire`` may have filled. Release has to
# win that race: it is what frees the connection they are
# waiting for.
await _run_on(
conn._executor,
functools.partial(self._pool.release, conn._conn, broken=broken),
) )
@contextlib.asynccontextmanager @contextlib.asynccontextmanager
async def connection( async def connection(
self, timeout: float | None = None self, timeout: float | None = None
) -> AsyncIterator[AsyncConnection]: ) -> AsyncIterator[AsyncConnection]:
"""Async context-manager wrapper around acquire/release.""" """Async context-manager wrapper around acquire/release.
Phase 27: cancellation and timeouts route through ``broken=True``.
``asyncio.to_thread`` does not interrupt the underlying worker
when the awaitable is cancelled — the worker keeps running on
the socket. If the connection were returned to the pool while
the worker is still mid-write, the next acquirer would inherit
a desynchronized wire. Evicting on cancellation prevents that
(combined with the wire lock the pool's ``release()`` acquires
with a timeout — see ``pool.release``).
"""
conn = await self.acquire(timeout=timeout) conn = await self.acquire(timeout=timeout)
broken = False broken = False
try: try:
yield conn yield conn
except (asyncio.CancelledError, asyncio.TimeoutError):
# Cancellation or wait_for timeout. The to_thread worker
# may still be running; we cannot trust the connection's
# wire state. Evict.
broken = True
raise
except Exception as e: except Exception as e:
# Mirror sync pool's eviction policy: connection-related # Mirror sync pool's eviction policy: connection-related
# errors evict, application errors retain. # errors evict, application errors retain.

812
src/informix_db/connections.py
View File

@ -11,15 +11,19 @@ reference in ``docs/CAPTURES/01-connect-only.socat.log``.
from __future__ import annotations from __future__ import annotations
import contextlib
import logging
import os import os
import socket as socket_mod import socket as socket_mod
import ssl import ssl
import struct import struct
import threading import threading
import weakref
from io import BytesIO from io import BytesIO
from pathlib import Path from pathlib import Path
from . import _auth from . import _auth
from ._capabilities import CLIENT_PROTOCOLS_MASK, ServerCapabilities
from ._messages import ( from ._messages import (
APPL_ID, APPL_ID,
APPL_TYPE, APPL_TYPE,
@ -35,10 +39,16 @@ from ._messages import (
SLHeader, SLHeader,
StmtOptions, StmtOptions,
) )
from ._protocol import IfxStreamReader, IfxStreamWriter, ProtocolError, make_pdu_writer from ._protocol import (
WIRE_ERRORS,
IfxStreamReader,
IfxStreamWriter,
ProtocolError,
make_pdu_writer,
)
from ._socket import IfxSocket from ._socket import IfxSocket
from .cursors import Cursor from .cursors import Cursor
from .exceptions import InterfaceError, OperationalError from .exceptions import InterfaceError, OperationalError, ProgrammingError
# Default capability bits the JDBC reference sends. Validated against # Default capability bits the JDBC reference sends. Validated against
# 01-connect-only.socat.log via the PDU diff in tests/test_pdu_match.py: # 01-connect-only.socat.log via the PDU diff in tests/test_pdu_match.py:
@ -53,6 +63,136 @@ _DEFAULT_CAP_1 = 0x0000013C
_DEFAULT_CAP_2 = 0 _DEFAULT_CAP_2 = 0
_DEFAULT_CAP_3 = 0 _DEFAULT_CAP_3 = 0
# Phase 20: client_locale → Python encoding name. Used by user-data
# string codecs (CHAR/VARCHAR/LVARCHAR/CLOB/TEXT). Protocol-level
# strings (cursor names, signatures, error tokens) stay iso-8859-1.
_LOCALE_ENCODING_MAP = {
"8859-1": "iso-8859-1",
"819": "iso-8859-1",
"8859-15": "iso-8859-15",
"923": "iso-8859-15",
"utf8": "utf-8",
"utf-8": "utf-8",
"utf16": "utf-16",
"ucs2": "utf-16",
}
_log = logging.getLogger(__name__)
def _extract_server_error_text(payload: bytes) -> str | None:
"""Pull the longest printable run out of an opaque rejection payload.
Phase 30: connection-rejection bodies have a version-dependent
structured layout, but in practice the server's human-readable
error string is embedded somewhere as a length-prefixed or
nul-terminated ASCII run. Without doing the full structured
decode (deferred), we can still surface the diagnostic text by
finding the longest printable-ASCII run of length ≥ 8 and ≤ 256.
Returns ``None`` if no qualifying run exists.
"""
if not payload:
return None
longest = b""
current = bytearray()
for byte in payload:
# Printable ASCII range — 0x20 (space) through 0x7E (~)
if 0x20 <= byte <= 0x7E:
current.append(byte)
else:
if len(current) > len(longest):
longest = bytes(current)
current = bytearray()
if len(current) > len(longest):
longest = bytes(current)
if 8 <= len(longest) <= 256:
return longest.decode("ascii", errors="replace").strip()
return None
def _python_encoding_from_locale(locale: str) -> str:
"""Map an Informix CLIENT_LOCALE string to the matching Python codec.
The CLIENT_LOCALE format is ``<lang>_<region>.<codeset>`` — we
only care about the codeset suffix. Unknown / no-suffix locales
fall back to ``iso-8859-1`` (the Informix default).
"""
if "." not in locale:
return "iso-8859-1"
suffix = locale.split(".", 1)[1].lower()
return _LOCALE_ENCODING_MAP.get(suffix, "iso-8859-1")
def _decode_conacc(rest: bytes) -> dict | None:
"""Decode the SLTYPE_CONACC server-metadata block.
Mirrors ``com.informix.asf.Connection.DecodeAscBinary``. ``rest`` is
the login response with its 2-byte total-length prefix already
stripped, i.e. starting at the SLType byte.
Layout, verified byte-for-byte against Informix 12.10, 14.10 and 15::
byte SLType (2 = CONACC)
3 skip
short 100 marker
short 101 marker
4 skip
short len; skip len (codeset id, e.g. "IEEEI")
short 108 marker — JDBC hard-fails if this isn't 108
12 skip
short len; bytes server version string
short len; bytes serial number
short len; bytes applid name
int Cap_1 client protocol level, echoed (316 = enhanced)
int Cap_2
int Cap_3
Returns ``None`` if the block doesn't match that shape. Callers must
treat that as "no capability info", never as a connection failure —
every server we support connects fine without any of this being
decodable, and a future server variant should degrade, not break.
"""
try:
offset = 1 + 3 # SLType + 3 skipped bytes
marker_a, marker_b = struct.unpack_from("!hh", rest, offset)
offset += 4
if marker_a != 100 or marker_b != 101:
return None
offset += 4
(codeset_len,) = struct.unpack_from("!h", rest, offset)
offset += 2 + codeset_len
(marker_c,) = struct.unpack_from("!h", rest, offset)
offset += 2
if marker_c != 108:
return None
offset += 12
strings: list[str] = []
for _ in range(3):
(length,) = struct.unpack_from("!h", rest, offset)
offset += 2
if length < 0:
return None
strings.append(
rest[offset : offset + length].rstrip(b"\x00").decode(
"iso-8859-1", "replace"
)
)
offset += length
cap_1, cap_2, cap_3 = struct.unpack_from("!iii", rest, offset)
except (struct.error, IndexError, UnicodeDecodeError):
return None
return {
"server_version": strings[0],
"serial_number": strings[1],
"applid_name": strings[2],
"cap_1": cap_1,
"cap_2": cap_2,
"cap_3": cap_3,
}
# Default environment variables sent in the login PDU (SQ_ASCENV section). # Default environment variables sent in the login PDU (SQ_ASCENV section).
# These match what the JDBC driver sends for a vanilla en_US.8859-1 # These match what the JDBC driver sends for a vanilla en_US.8859-1
# connection. Anything missing makes the server fall back to defaults. # connection. Anything missing makes the server fall back to defaults.
@ -66,6 +206,62 @@ _DEFAULT_ENV: dict[str, str] = {
} }
class _WireLock:
"""A reentrant lock that can say whether *this* thread already holds it.
``threading.RLock`` cannot answer that question in public API, and
the difference is not academic. The cursor finalizer runs at GC time
on whatever thread happened to allocate — including a thread that is
at that moment mid-statement holding this lock. It probes with
``acquire(blocking=False)`` intending to mean "is anyone using the
wire?", but an RLock grants a reentrant acquire to its own owner, so
the probe returns True and the finalizer sends CLOSE/RELEASE into the
middle of the statement it interrupted. The victim gets ``-208``.
Reachability is not exotic. Refcounting frees a dropped cursor
immediately, before the next statement, which is why this went
unnoticed — but a cursor caught in a reference cycle waits for a
collection instead, and cycles are ordinary in Python. Any traceback
that holds a cursor makes one.
``held_by_current_thread`` reads ``_depth``/``_owner`` without the
lock, which is safe: both are mutated only under it, and the only
values another thread can leave behind are a depth of zero or an
owner that isn't us. Either way the answer is False, correctly.
"""
__slots__ = ("_depth", "_lock", "_owner")
def __init__(self) -> None:
self._lock = threading.RLock()
self._owner: int | None = None
self._depth = 0
def acquire(self, blocking: bool = True, timeout: float = -1) -> bool:
acquired = self._lock.acquire(blocking, timeout)
if acquired:
self._owner = threading.get_ident()
self._depth += 1
return acquired
def release(self) -> None:
self._depth -= 1
if self._depth == 0:
self._owner = None
self._lock.release()
def __enter__(self) -> _WireLock:
self.acquire()
return self
def __exit__(self, *exc_info: object) -> None:
self.release()
@property
def held_by_current_thread(self) -> bool:
return self._depth > 0 and self._owner == threading.get_ident()
class Connection: class Connection:
"""A SQLI session. Owns one TCP socket and the post-login state. """A SQLI session. Owns one TCP socket and the post-login state.
@ -87,6 +283,7 @@ class Connection:
client_locale: str = "en_US.8859-1", client_locale: str = "en_US.8859-1",
env: dict[str, str] | None = None, env: dict[str, str] | None = None,
autocommit: bool = False, # honored from Phase 3 onward autocommit: bool = False, # honored from Phase 3 onward
row_factory: object | None = None,
tls: bool | ssl.SSLContext = False, tls: bool | ssl.SSLContext = False,
tls_server_hostname: str | None = None, tls_server_hostname: str | None = None,
): ):
@ -96,15 +293,55 @@ class Connection:
self._database = database self._database = database
self._server = server self._server = server
self._client_locale = client_locale self._client_locale = client_locale
self._encoding = _python_encoding_from_locale(client_locale)
self._autocommit = autocommit self._autocommit = autocommit
self._closed = False self._closed = False
self._lock = threading.Lock() self._lock = threading.Lock()
# Phase 27: per-connection wire lock. Held for the duration of
# every send-PDU + drain-response round-trip. Two threads on
# one connection (or async cancellation leaving a worker still
# mid-operation) can no longer interleave bytes on the socket —
# the second thread blocks until the first releases.
#
# RLock (not Lock) because the Pool's release() path acquires
# this lock with a timeout, then calls ``conn.rollback()`` —
# which itself acquires the lock. Same thread, two acquires.
# Reentrance must be cheap and correct.
self._wire_lock = _WireLock()
# Phase 29: deferred-cleanup queue for cursor finalizers that
# couldn't acquire the wire lock at GC time. Each entry is a
# PDU's worth of bytes (typically a CLOSE or RELEASE) that
# gets sent + drained at the start of the next ``_send_pdu``
# on this connection. Closes the unbounded-leak gap on
# long-lived pooled connections — a connection that survives
# many cancellation events would otherwise accumulate orphaned
# server-side cursors until the per-session limit is hit.
#
# Two locks because they protect different concerns:
# ``_wire_lock`` is the send/recv atomicity lock (held for the
# whole round-trip); ``_cleanup_lock`` is a tiny critical
# section guarding only the list mutation (held microseconds).
# Lock-acquire order: never grab ``_cleanup_lock`` while
# holding ``_wire_lock`` recursively — see ``_drain_pending_cleanup``
# which copies-and-clears under ``_cleanup_lock`` then iterates
# under ``_wire_lock``.
self._pending_cleanup: list[bytes] = []
self._cleanup_lock = threading.Lock()
# Weak ref to the scrollable cursor currently holding the
# session's statement slot, or None. Weak so that abandoning a
# scrollable cursor still lets its finalizer run — a strong ref
# here would keep the very object alive whose GC we depend on.
self._open_scroll_cursor: weakref.ref | None = None
# Logged-DB transaction state: True iff there's an open server-side # Logged-DB transaction state: True iff there's an open server-side
# transaction (SQ_BEGIN sent, not yet committed/rolled-back). The # transaction (SQ_BEGIN sent, not yet committed/rolled-back). The
# cursor uses this to decide whether to send an implicit SQ_BEGIN # cursor uses this to decide whether to send an implicit SQ_BEGIN
# before the next DML in non-autocommit mode. We default to "no # before the next DML in non-autocommit mode. We default to "no
# open txn" — the first DML will trigger SQ_BEGIN. # open txn" — the first DML will trigger SQ_BEGIN.
self._in_transaction = False self._in_transaction = False
# Default row type for cursors from this connection. None means
# plain tuples, which is the zero-cost default; see
# informix_db.rows for the opt-in named-access type.
self.row_factory = row_factory
# Tri-state: True after first successful SQ_BEGIN, False after # Tri-state: True after first successful SQ_BEGIN, False after
# an unlogged-DB rejection (-201). None until we've tried. # an unlogged-DB rejection (-201). None until we've tried.
# Used to avoid repeatedly probing on unlogged DBs. # Used to avoid repeatedly probing on unlogged DBs.
@ -114,6 +351,16 @@ class Connection:
# SQ_GETROUTINE; subsequent calls skip that round-trip. # SQ_GETROUTINE; subsequent calls skip that round-trip.
self._fp_handle_cache: dict[str, tuple[str, int]] = {} self._fp_handle_cache: dict[str, tuple[str, int]] = {}
# Server metadata from the login response and the SQ_PROTOCOLS
# negotiation. Both are populated during connect; both degrade to
# None rather than failing the connection.
self._conacc: dict | None = None
self._server_protocols: bytes | None = None
self._capabilities: ServerCapabilities | None = None
# Lazily filled by the server_version property. None = not asked
# yet; "" = asked and failed (don't retry on every access).
self._server_version_full: str | None = None
# Build the env-var dict sent in the login PDU. # Build the env-var dict sent in the login PDU.
self._env = dict(_DEFAULT_ENV) self._env = dict(_DEFAULT_ENV)
self._env["CLIENT_LOCALE"] = client_locale self._env["CLIENT_LOCALE"] = client_locale
@ -154,18 +401,189 @@ class Connection:
def closed(self) -> bool: def closed(self) -> bool:
return self._closed return self._closed
def cursor(self) -> Cursor: @property
"""Return a new Cursor for executing SQL on this connection.""" def encoding(self) -> str:
if self._closed: """Python codec name for user-data strings (CHAR/VARCHAR/CLOB/TEXT).
raise InterfaceError("connection is closed")
return Cursor(self)
def _send_pdu(self, pdu: bytes) -> None: Derived from ``client_locale`` at connect time. Defaults to
"""Send an assembled PDU. Used by Cursor.""" ``"iso-8859-1"`` for the Informix default locale; ``"utf-8"``
when ``client_locale="en_US.utf8"`` (or similar).
"""
return self._encoding
def cursor(self, *, scrollable: bool = False) -> Cursor:
"""Return a new Cursor for executing SQL on this connection.
``scrollable=True`` opens a server-side scrollable cursor that
doesn't materialize all rows up-front. Each scroll method
(``fetch_first``/``fetch_last``/``fetch_absolute``/etc.) sends
``SQ_SFETCH`` to the server per call. Use this for huge result
sets where in-memory materialization (the default) would be
wasteful.
``scrollable=False`` (default): the cursor materializes the
whole result set on ``execute()`` and scroll methods do
index manipulation locally. Faster for moderate-sized result
sets.
"""
if self._closed: if self._closed:
raise InterfaceError("connection is closed") raise InterfaceError("connection is closed")
return Cursor(self, scrollable=scrollable)
def _send_pdu(self, pdu: bytes, *, statement_boundary: bool = False) -> None:
"""Send an assembled PDU. Used by Cursor.
Caller must hold ``_wire_lock`` (every actual call site does).
``statement_boundary=True`` additionally drains the deferred
cleanup queue first. **Only pass it when this PDU is the first
of a new statement**, meaning no statement is currently open
server-side.
The queue previously drained before *every* PDU, which was wrong
in a way that took a repro to see. ``SQ_CLOSE`` and
``SQ_RELEASE`` carry no statement identifier — they act on the
server's *current* statement. A finalizer enqueues precisely
because it lost the race for the wire lock, which means another
thread is mid-statement; that thread's next ``_send_pdu`` was
then its own ``CURNAME``/``NFETCH``, and the drain released the
statement out from under it. The caller saw a nonsense error on
valid SQL: ``-208`` when injected before the first fetch,
``-267`` "transaction has been rolled back" between fetch
batches. Both point nowhere near the actual cause, and the
window is exactly the window in which enqueueing happens.
Draining only at a boundary costs nothing: cleanup that misses
one statement is picked up by the next.
"""
if self._closed:
raise InterfaceError("connection is closed")
if statement_boundary and self._pending_cleanup:
self._drain_pending_cleanup()
self._sock.write_all(pdu) self._sock.write_all(pdu)
def _check_scroll_cursor_conflict(self, requester: object) -> None:
"""Refuse to start a statement while a scrollable cursor is open.
A server-side scrollable cursor occupies the session's statement
slot, and SQLI gives us no way to address around it: ``SQ_CLOSE``,
``SQ_RELEASE`` and ``SQ_SFETCH`` all act on the session's current
statement.
The server does not refuse politely. Starting another statement
returns ``-285``, and the scrollable cursor is collateral damage
— its next fetch comes back ``-267`` "the transaction has been
rolled back, all locks released". Two unattributable failures
from code that looks entirely ordinary: iterate a large result
set with a scrollable cursor, run a lookup query partway through.
Multiplexing is presumably expressible — JDBC prefixes every
statement-scoped PDU with the statement id, which the server does
assign distinctly (0 and 1 for two concurrent cursors). But the
id alone is not sufficient: addressing the second cursor's
``SQ_SFETCH`` to its own id returns ``-259`` "cursor not open".
Until that is understood, refusing is the honest behaviour. It
costs the caller a second connection and it never corrupts.
"""
ref = self._open_scroll_cursor
if ref is None:
return
other = ref()
if (
other is None
or other is requester
or not getattr(other, "_server_cursor_open", False)
):
self._open_scroll_cursor = None
return
raise ProgrammingError(
"a scrollable cursor is open on this connection; Informix "
"allows only one statement per session, so running another "
"statement here would fail with -285 and destroy the "
"scrollable cursor as well. Close the scrollable cursor "
"first, or use a separate connection for the other statement."
)
def _enqueue_cleanup(self, pdus: list[bytes]) -> None:
"""Append cleanup PDUs to the deferred queue.
Phase 29: called by the cursor finalizer when it can't acquire
the wire lock (cross-thread GC). Instead of leaking the
server-side resource, the finalizer hands the cleanup bytes
here; the next normal operation drains them.
Tiny critical section — holds ``_cleanup_lock`` only for the
list ``extend``. Safe to call from any thread, including a
finalizer on a thread that doesn't own the wire lock.
"""
with self._cleanup_lock:
self._pending_cleanup.extend(pdus)
def _drain_pending_cleanup(self) -> None:
"""Send + drain queued cleanup PDUs. Caller MUST hold ``_wire_lock``.
Pops the entire pending list under ``_cleanup_lock`` (small
critical section), then iterates under ``_wire_lock`` (which
the caller already holds) to send each PDU and drain its
SQ_EOT response.
On wire desync mid-drain (e.g., the server has gone away),
force-closes the connection — same doctrine as
:meth:`_raise_sq_err`. The remaining queued entries are
discarded; the server-side resources they would have released
are freed when the session ends anyway.
"""
with self._cleanup_lock:
if not self._pending_cleanup:
return
pending = self._pending_cleanup
self._pending_cleanup = []
for pdu in pending:
try:
self._sock.write_all(pdu)
self._drain_to_eot()
except Exception as exc:
if getattr(exc, "sqlcode", None) is not None:
# The *server* rejected the cleanup — a stale entry
# for a cursor it no longer has. Queued cleanup goes
# stale routinely: the finalizer enqueues, then the
# cursor gets closed properly before the drain runs.
#
# This is not a wire problem. ``_raise_sq_err``
# self-drains the trailing SQ_EOT, so the wire is
# still aligned and the remaining entries are still
# worth sending.
#
# It must not escape, and it must not be treated as
# fatal. Both were wrong before: a stale CLOSE draws
# ``-267``, which is an OperationalError, which is in
# WIRE_ERRORS — so a stale queue entry force-closed a
# perfectly healthy connection. And this runs at the
# start of somebody else's statement, so letting it
# out would fail their good SQL with an error about a
# cursor they never opened.
_log.debug(
"deferred cleanup rejected by server (stale entry), "
"continuing: %r",
exc,
)
continue
if not isinstance(exc, WIRE_ERRORS):
_log.warning(
"unexpected error draining deferred cleanup: %r", exc
)
# No sqlcode means the failure came from the wire, not
# the server: the socket died, or framing desynced and we
# can no longer say where a response ends. Force-close.
# Subsequent ``_send_pdu`` raises InterfaceError. The
# server-side resources the remaining entries would have
# freed are released when the session ends anyway.
self._closed = True
with contextlib.suppress(Exception):
self._sock.close()
return
def commit(self) -> None: def commit(self) -> None:
"""Commit the current transaction (SQ_CMMTWORK). """Commit the current transaction (SQ_CMMTWORK).
@ -183,9 +601,12 @@ class Connection:
# only set the flag after a successful BEGIN, so this branch # only set the flag after a successful BEGIN, so this branch
# also covers "no DML happened since last commit/rollback". # also covers "no DML happened since last commit/rollback".
return return
self._sock.write_all(struct.pack("!hh", MessageType.SQ_CMMTWORK, MessageType.SQ_EOT)) with self._wire_lock:
self._drain_to_eot() self._sock.write_all(
self._in_transaction = False struct.pack("!hh", MessageType.SQ_CMMTWORK, MessageType.SQ_EOT)
)
self._drain_to_eot()
self._in_transaction = False
def rollback(self) -> None: def rollback(self) -> None:
"""Roll back the current transaction (SQ_RBWORK). """Roll back the current transaction (SQ_RBWORK).
@ -203,11 +624,12 @@ class Connection:
# The savepoint short is REQUIRED — sending SQ_RBWORK alone hangs # The savepoint short is REQUIRED — sending SQ_RBWORK alone hangs
# the server (it's waiting for the next 2 bytes). SQ_CMMTWORK, # the server (it's waiting for the next 2 bytes). SQ_CMMTWORK,
# by contrast, takes no payload — confirmed in IfxSqli.sendCommit. # by contrast, takes no payload — confirmed in IfxSqli.sendCommit.
self._sock.write_all( with self._wire_lock:
struct.pack("!hhh", MessageType.SQ_RBWORK, 0, MessageType.SQ_EOT) self._sock.write_all(
) struct.pack("!hhh", MessageType.SQ_RBWORK, 0, MessageType.SQ_EOT)
self._drain_to_eot() )
self._in_transaction = False self._drain_to_eot()
self._in_transaction = False
def fast_path_call( def fast_path_call(
self, signature: str, *params: object self, signature: str, *params: object
@ -256,59 +678,62 @@ class Connection:
if self._closed: if self._closed:
raise InterfaceError("connection is closed") raise InterfaceError("connection is closed")
cached = self._fp_handle_cache.get(signature) with self._wire_lock:
if cached is None: cached = self._fp_handle_cache.get(signature)
# Resolve via SQ_GETROUTINE if cached is None:
self._sock.write_all(build_get_routine_pdu(signature)) # Resolve via SQ_GETROUTINE
self._sock.write_all(build_get_routine_pdu(signature))
reader = _SocketReader(self._sock)
tag = reader.read_short()
if tag == MessageType.SQ_ERR:
self._raise_sq_err()
if tag != MessageType.SQ_GETROUTINE:
raise OperationalError(
f"fast-path GETROUTINE: unexpected tag 0x{tag:04x}"
)
db_name, handle = parse_get_routine_response(reader)
tail = reader.read_short()
if tail != MessageType.SQ_EOT:
raise OperationalError(
f"GETROUTINE response: missing SQ_EOT (got 0x{tail:04x})"
)
self._fp_handle_cache[signature] = (db_name, handle)
else:
db_name, handle = cached
# Now execute via SQ_EXFPROUTINE
self._sock.write_all(
build_exfp_routine_pdu(
db_name, handle, params, encoding=self._encoding
)
)
reader = _SocketReader(self._sock) reader = _SocketReader(self._sock)
tag = reader.read_short() tag = reader.read_short()
if tag == MessageType.SQ_ERR: if tag == MessageType.SQ_ERR:
self._raise_sq_err() self._raise_sq_err()
if tag != MessageType.SQ_GETROUTINE: if tag != MessageType.SQ_FPROUTINE:
raise OperationalError( raise OperationalError(
f"fast-path GETROUTINE: unexpected tag 0x{tag:04x}" f"fast-path EXFPROUTINE: unexpected response tag 0x{tag:04x}"
) )
db_name, handle = parse_get_routine_response(reader) results = parse_fp_routine_response(reader)
tail = reader.read_short() # Drain any trailing tags until SQ_EOT (server may send
if tail != MessageType.SQ_EOT: # SQ_DONE/SQ_COST/SQ_XACTSTAT before SQ_EOT, same as SQL paths)
raise OperationalError( while True:
f"GETROUTINE response: missing SQ_EOT (got 0x{tail:04x})" tag = reader.read_short()
) if tag == MessageType.SQ_EOT:
self._fp_handle_cache[signature] = (db_name, handle) break
else: elif tag == MessageType.SQ_DONE:
db_name, handle = cached reader.read_exact(2 + 4 + 4 + 4) # warn + rows + rowid + serial
elif tag == 55: # SQ_COST
# Now execute via SQ_EXFPROUTINE reader.read_int()
self._sock.write_all( reader.read_int()
build_exfp_routine_pdu(db_name, handle, params) elif tag == MessageType.SQ_XACTSTAT:
) reader.read_exact(2 + 2 + 2)
reader = _SocketReader(self._sock) else:
tag = reader.read_short() raise OperationalError(
if tag == MessageType.SQ_ERR: f"fast-path response: unexpected tag 0x{tag:04x}"
self._raise_sq_err() )
if tag != MessageType.SQ_FPROUTINE: return results
raise OperationalError(
f"fast-path EXFPROUTINE: unexpected response tag 0x{tag:04x}"
)
results = parse_fp_routine_response(reader)
# Drain any trailing tags until SQ_EOT (server may send
# SQ_DONE/SQ_COST/SQ_XACTSTAT before SQ_EOT, same as SQL paths)
while True:
tag = reader.read_short()
if tag == MessageType.SQ_EOT:
break
elif tag == MessageType.SQ_DONE:
reader.read_exact(2 + 4 + 4 + 4) # warn + rows + rowid + serial
elif tag == 55: # SQ_COST
reader.read_int()
reader.read_int()
elif tag == MessageType.SQ_XACTSTAT:
reader.read_exact(2 + 2 + 2)
else:
raise OperationalError(
f"fast-path response: unexpected tag 0x{tag:04x}"
)
return results
def _ensure_transaction(self) -> None: def _ensure_transaction(self) -> None:
"""Open a server-side transaction if one isn't already open. """Open a server-side transaction if one isn't already open.
@ -321,7 +746,22 @@ class Connection:
Idempotent: subsequent calls are no-ops while the transaction Idempotent: subsequent calls are no-ops while the transaction
is open or while we've cached "this DB doesn't support BEGIN". is open or while we've cached "this DB doesn't support BEGIN".
**Precondition (Phase 27):** caller MUST hold ``self._wire_lock``.
Every actual call site is inside a cursor method that has
already acquired the lock; this method does its own wire I/O
but doesn't re-acquire to avoid redundant work.
""" """
# Defensive guard: fail loudly in development if a future caller
# forgets to lock. ``RLock._is_owned()`` is a CPython-private
# method but stable across versions; cheap (~50ns) and only
# checks the current thread. If it ever changes shape, drop
# this assert — the doc still names the precondition.
assert self._wire_lock.held_by_current_thread, (
"_ensure_transaction called without _wire_lock held; "
"the cursor method that called it must wrap its body in "
"`with self._conn._wire_lock:`"
)
if self._autocommit or self._in_transaction or self._closed: if self._autocommit or self._in_transaction or self._closed:
return return
if self._supports_begin_work is False: if self._supports_begin_work is False:
@ -339,13 +779,29 @@ class Connection:
raise raise
def close(self) -> None: def close(self) -> None:
"""Send SQ_EXIT and tear down the socket. Idempotent.""" """Send SQ_EXIT and tear down the socket. Idempotent.
Phase 27: tries to acquire the wire lock with a short timeout
before sending SQ_EXIT. If another thread is mid-operation,
``SQ_EXIT`` would interleave bytes with their PDU; better to
skip the polite-exit and just close the socket. The in-flight
thread observes EOF on its next read.
"""
with self._lock: with self._lock:
if self._closed: if self._closed:
return return
self._closed = True self._closed = True
try: try:
self._send_exit() # Short timeout — close() shouldn't block long. If the
# wire is busy, skip the polite SQ_EXIT and force-close
# the socket; the in-flight thread will get an OSError
# on its next read, which surfaces cleanly to the caller.
got_lock = self._wire_lock.acquire(timeout=0.5)
if got_lock:
try:
self._send_exit()
finally:
self._wire_lock.release()
finally: finally:
self._sock.close() self._sock.close()
@ -377,9 +833,9 @@ class Connection:
# The 8-byte protocols mask is the JDBC reference value from # The 8-byte protocols mask is the JDBC reference value from
# docs/CAPTURES/02-select-1.socat.log; we replay it verbatim # docs/CAPTURES/02-select-1.socat.log; we replay it verbatim
# since the bits are opaque (server-recognized features). # since the bits are opaque (server-recognized features).
protocols_mask = bytes.fromhex("fffc7ffc3c8caa97") self._send_protocols(CLIENT_PROTOCOLS_MASK)
self._send_protocols(protocols_mask) self._drain_to_eot() # captures the reply into self._server_protocols
self._drain_to_eot() self._build_capabilities()
# Step 2: SQ_INFO with INFO_ENV subtype + session env vars. # Step 2: SQ_INFO with INFO_ENV subtype + session env vars.
# The actual on-wire format (from JDBC's sendEnv at IfxSqli.java # The actual on-wire format (from JDBC's sendEnv at IfxSqli.java
@ -414,6 +870,54 @@ class Connection:
self._send_dbopen(self._database) self._send_dbopen(self._database)
self._drain_to_eot() self._drain_to_eot()
def _build_capabilities(self) -> None:
"""Assemble ``ServerCapabilities`` from the login response and the
SQ_PROTOCOLS reply, and warn about contradicted assumptions.
This is observation only — nothing in the parse paths branches on
these bits yet. The value is diagnostic: this driver hardcodes
several wire-framing choices that SQLI actually negotiates, and
those choices are correct on every server we have measured
(Informix 12.10, 14.10, 15). On some server we have not measured
they might not be, and the failure mode for a framing mismatch is
silently corrupted rows, which is about the worst way for a
database driver to fail. A log warning naming the specific bit
turns that into something diagnosable from a bug report.
Deliberately never raises: a server that declines to negotiate is
not necessarily broken, and refusing the connection over a
diagnostic would be a worse outcome than proceeding.
"""
if self._server_protocols is None:
return
conacc = self._conacc or {}
try:
caps = ServerCapabilities.from_wire(
self._server_protocols,
cap_1=conacc.get("cap_1", 0),
cap_2=conacc.get("cap_2", 0),
cap_3=conacc.get("cap_3", 0),
server_version=conacc.get("server_version", ""),
serial_number=conacc.get("serial_number", ""),
applid_name=conacc.get("applid_name", ""),
)
except Exception:
_log.debug("could not decode server capabilities", exc_info=True)
return
self._capabilities = caps
problems = caps.violated_assumptions()
if problems:
_log.warning(
"Informix server %r negotiated capabilities that contradict "
"this driver's hardcoded wire framing (mask=%s). Result rows "
"may be decoded incorrectly. Please report this with the "
"mask and your server version. Details: %s",
caps.server_version or "<unknown version>",
caps.raw_mask.hex(),
"; ".join(problems),
)
def _send_protocols(self, protocols: bytes) -> None: def _send_protocols(self, protocols: bytes) -> None:
"""Emit a SQ_PROTOCOLS PDU per ``IfxSqli.sendProtocols``. """Emit a SQ_PROTOCOLS PDU per ``IfxSqli.sendProtocols``.
@ -456,9 +960,13 @@ class Connection:
elif tag == MessageType.SQ_PROTOCOLS: elif tag == MessageType.SQ_PROTOCOLS:
# ``[short payloadLen][bytes payload][byte 0 if odd-len pad]`` # ``[short payloadLen][bytes payload][byte 0 if odd-len pad]``
# Then the loop continues and consumes the next tag (usually SQ_EOT). # Then the loop continues and consumes the next tag (usually SQ_EOT).
# The payload is the negotiated 64-bit feature bitmap; stash
# it for _capabilities decoding. Reading it was always
# necessary for stream alignment — we just used to throw it
# away instead of looking at it.
payload_len = struct.unpack("!h", self._sock.read_exact(2))[0] payload_len = struct.unpack("!h", self._sock.read_exact(2))[0]
if payload_len > 0: if payload_len > 0:
self._sock.read_exact(payload_len) self._server_protocols = self._sock.read_exact(payload_len)
if payload_len & 1: if payload_len & 1:
self._sock.read_exact(1) # writePadded's even-alignment pad self._sock.read_exact(1) # writePadded's even-alignment pad
elif tag == MessageType.SQ_DONE: elif tag == MessageType.SQ_DONE:
@ -493,6 +1001,9 @@ class Connection:
isamcode = struct.unpack("!h", self._sock.read_exact(2))[0] isamcode = struct.unpack("!h", self._sock.read_exact(2))[0]
offset = struct.unpack("!i", self._sock.read_exact(4))[0] offset = struct.unpack("!i", self._sock.read_exact(4))[0]
near_token = "" near_token = ""
# Phase 28: specific catches — truncated near_token is recoverable
# (proceed with empty token); a programming error here would
# otherwise be silently masked.
try: try:
name_len = struct.unpack("!h", self._sock.read_exact(2))[0] name_len = struct.unpack("!h", self._sock.read_exact(2))[0]
if name_len > 0: if name_len > 0:
@ -500,15 +1011,24 @@ class Connection:
if name_len & 1: if name_len & 1:
self._sock.read_exact(1) self._sock.read_exact(1)
near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace") near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace")
except Exception: except WIRE_ERRORS:
pass pass
# Phase 28: drain failure means wire desync — force-close so
# subsequent operations don't inherit the broken state.
# ``OperationalError`` is in the catch tuple because
# ``_drain_to_eot`` itself raises it for unknown tags during
# the drain (e.g., a SQ_ERR mid-drain after the initial error
# we already started decoding). Same desync taxonomy as
# ProtocolError/OSError: the wire is unrecoverable.
try: try:
while True: while True:
next_tag = struct.unpack("!h", self._sock.read_exact(2))[0] next_tag = struct.unpack("!h", self._sock.read_exact(2))[0]
if next_tag == MessageType.SQ_EOT: if next_tag == MessageType.SQ_EOT:
break break
except OperationalError: except WIRE_ERRORS:
pass self._closed = True
with contextlib.suppress(Exception):
self._sock.close()
text = _errcodes.text_for(sqlcode) text = _errcodes.text_for(sqlcode)
exc_class = _errcodes.exception_for(sqlcode) exc_class = _errcodes.exception_for(sqlcode)
@ -658,6 +1178,72 @@ class Connection:
# -- response parsing ------------------------------------------------- # -- response parsing -------------------------------------------------
@property
def server_capabilities(self):
"""The negotiated ``SQ_PROTOCOLS`` feature bitmap, or ``None``.
``None`` means the negotiation reply couldn't be decoded — the
connection still works, we just have no capability information.
See ``informix_db._capabilities.ServerCapabilities``.
"""
return self._capabilities
@property
def server_version_internal(self) -> str:
"""The version string carried in the login response.
This is Informix's *internal* protocol version, not the release
you installed: 12.10 reports ``9.56``, 14.10 reports ``9.59``,
and 15 reports ``15.0.1.0.3``. At the protocol level the two
older releases really are 9.x servers, which is why all three
speak an identical SQLI dialect.
Free — it arrives with the login response. For the release
number, use :attr:`server_version`.
"""
return self._conacc.get("server_version", "") if self._conacc else ""
@property
def server_version(self) -> str:
"""The server's release version, e.g.
``'IBM Informix Dynamic Server Version 12.10.FC6'``.
**Performs one query on first access**, then caches the result
for the life of the connection. The login response only carries
the internal protocol version (see
:attr:`server_version_internal`), which reads as a wrong answer
— a 12.10 server announces itself as 9.56 — so the release
number has to be asked for with ``DBINFO('version','full')``.
Falls back to the internal string if the query fails, which it
can when no database is open. Never raises: a version lookup
should not be able to break a working connection.
"""
if self._server_version_full is None:
self._server_version_full = self._query_server_version()
return self._server_version_full or self.server_version_internal
def _query_server_version(self) -> str:
"""One-shot ``DBINFO`` lookup for the release version.
Deliberately not done during connect: it costs a round-trip that
most callers never need, and connect latency is on the hot path
for pooled workloads.
"""
try:
cur = self.cursor()
try:
cur.execute(
"SELECT FIRST 1 DBINFO('version','full') FROM systables"
)
row = cur.fetchone()
finally:
cur.close()
except Exception:
_log.debug("could not query server version", exc_info=True)
return ""
return str(row[0]).strip() if row and row[0] else ""
def _parse_login_response(self) -> None: def _parse_login_response(self) -> None:
"""Read and parse the server's login response. """Read and parse the server's login response.
@ -691,34 +1277,53 @@ class Connection:
) )
elif sl_type != SLHeader.SLTYPE_CONACC: elif sl_type != SLHeader.SLTYPE_CONACC:
raise ProtocolError(f"unknown SLType in login response: {sl_type}") raise ProtocolError(f"unknown SLType in login response: {sl_type}")
# SLTYPE_CONACC — connection accepted. We don't (yet) decode the
# full server-side metadata. Phase 1 just needs to know "we got in". # SLTYPE_CONACC — connection accepted. Decode the server metadata
# block for the capability fields. Best-effort: a decode failure
# here must not fail an otherwise-good connection, since every
# server we've tested connects fine without any of this.
self._conacc = _decode_conacc(rest)
def _raise_from_rejection(self, reader: IfxStreamReader) -> None: def _raise_from_rejection(self, reader: IfxStreamReader) -> None:
"""Best-effort decode of the connection-rejection error block. """Best-effort decode of the connection-rejection error block.
Per PROTOCOL_NOTES.md §3c-d. We try to extract the SQLCODE and Per PROTOCOL_NOTES.md §3c-d. The full structured decode of the
message, but if the layout drifts we raise a generic rejection block (SQLCODE, isamcode, message) is deferred — the
OperationalError with whatever bytes we read. layout has version-dependent quirks. Phase 30: at minimum,
capture the server's human-readable error string from anywhere
in the rejection payload and include it in the exception
message. Wrong-password and wrong-database produced identical
generic errors before this; now they surface their server-side
text, which IBM's listener varies by reason.
""" """
# Drain whatever bytes remain — the rejection payload may be
# truncated, structured, or wrap inconsistencies across server
# versions. Capture defensively.
payload = bytearray()
try: try:
# Skip the SQ_ASSOC + SQ_ASCBINARY markers and the int 61 magic while True:
reader.skip(2 + 2 + 4) chunk = reader.read_exact(64)
# Then there's a length-prefixed block we skip payload.extend(chunk)
sub_length = reader.read_short() except Exception:
reader.skip(sub_length) # EOF on the reader is the expected termination; partial
# Then SQ_ASCBPARMS marker # last reads land in payload via successful prior iterations.
marker = reader.read_short() pass
if marker != LoginMarker.SQ_ASCBPARMS:
raise OperationalError("server rejected the connection (no decodable error block)") diagnostic = _extract_server_error_text(bytes(payload))
# Skip 12 bytes of fixed-position metadata, then the version if diagnostic:
# string, serial, applid, capabilities — we don't need any of raise OperationalError(
# that on the failure path, so we just bail out with a clear f"server rejected the connection: {diagnostic}"
# message. Phase 5 expands this to actually find the SQ_ASCINITRESP )
# block and pull svcError/osError/Warnings/errMsg. # Couldn't find a printable error string. Include a short hex
raise OperationalError("server rejected the connection") # preview for forensic logging — the user can paste it into a
except ProtocolError as e: # bug report if the server's rejection isn't self-describing.
raise OperationalError(f"server rejected the connection: {e}") from e if payload:
preview = bytes(payload[:64]).hex()
raise OperationalError(
f"server rejected the connection "
f"(rejection payload starts: {preview}...)"
)
raise OperationalError("server rejected the connection")
# -- disconnection ---------------------------------------------------- # -- disconnection ----------------------------------------------------
@ -726,8 +1331,14 @@ class Connection:
"""Send the bare ``[short SQ_EXIT=56]`` disconnect message. """Send the bare ``[short SQ_EXIT=56]`` disconnect message.
Per PROTOCOL_NOTES.md §8. Server echoes back ``SQ_EXIT`` or Per PROTOCOL_NOTES.md §8. Server echoes back ``SQ_EXIT`` or
``SQ_EOT``; we read and discard. Errors are swallowed because ``SQ_EOT``; we read and discard. **All errors are swallowed
we're already tearing down. because we're already tearing down** — the caller (``close``)
runs ``self._sock.close()`` in its finally regardless. Phase 30:
broadened the catch from a specific tuple to bare ``Exception``.
Any unexpected error here (struct.error from a malformed ack
byte, a future protocol-parse logic bug, anything) must not
escape ``close()`` and leave the FD half-closed. Best-effort
is best-effort.
""" """
try: try:
self._sock.write_all(struct.pack("!h", MessageType.SQ_EXIT)) self._sock.write_all(struct.pack("!h", MessageType.SQ_EXIT))
@ -743,6 +1354,9 @@ class Connection:
continue continue
# Unknown ack; bail out — we're closing anyway. # Unknown ack; bail out — we're closing anyway.
return return
except (OperationalError, InterfaceError, OSError, ProtocolError): except Exception:
# Already closing; nothing to do but suppress. # Already closing; suppress everything. The actual socket
# FD is freed by ``Connection.close()``'s finally block via
# ``self._sock.close()`` (which is itself idempotent and
# never-raising — see ``_socket.IfxSocket.close``).
return return

273
src/informix_db/converters.py
View File

@ -166,40 +166,67 @@ _REAL_NULL = b"\xff\xff\xff\xff"
_DOUBLE_NULL = b"\xff\xff\xff\xff\xff\xff\xff\xff" _DOUBLE_NULL = b"\xff\xff\xff\xff\xff\xff\xff\xff"
_DATE_NULL = 0x80000000 _DATE_NULL = 0x80000000
# Pre-compiled struct unpackers — bound methods bound at module load.
# 37% faster than ``struct.unpack(fmt, raw)`` because the format string
# is parsed once at compile time, not per call. Used by the fixed-width
# decoders below; saves ~9 ns/call x the row's int/float column count.
_UNPACK_SHORT = struct.Struct("!h").unpack
_UNPACK_INT = struct.Struct("!i").unpack
_UNPACK_LONG = struct.Struct("!q").unpack
_UNPACK_FLOAT = struct.Struct("!f").unpack
_UNPACK_DOUBLE = struct.Struct("!d").unpack
def _decode_smallint(raw: bytes) -> int | None: def _decode_smallint(raw: bytes) -> int | None:
val = struct.unpack("!h", raw)[0] val = _UNPACK_SHORT(raw)[0]
return None if val == -0x8000 else val return None if val == -0x8000 else val
def _decode_int(raw: bytes) -> int | None: def _decode_int(raw: bytes) -> int | None:
val = struct.unpack("!i", raw)[0] val = _UNPACK_INT(raw)[0]
return None if val == -0x80000000 else val return None if val == -0x80000000 else val
def _decode_bigint(raw: bytes) -> int | None: def _decode_bigint(raw: bytes) -> int | None:
val = struct.unpack("!q", raw)[0] val = _UNPACK_LONG(raw)[0]
return None if val == -0x8000000000000000 else val return None if val == -0x8000000000000000 else val
def _decode_smfloat(raw: bytes) -> float | None: def _decode_smfloat(raw: bytes) -> float | None:
if raw == _REAL_NULL: if raw == _REAL_NULL:
return None return None
return struct.unpack("!f", raw)[0] return _UNPACK_FLOAT(raw)[0]
def _decode_float(raw: bytes) -> float | None: def _decode_float(raw: bytes) -> float | None:
if raw == _DOUBLE_NULL: if raw == _DOUBLE_NULL:
return None return None
return struct.unpack("!d", raw)[0] return _UNPACK_DOUBLE(raw)[0]
def _decode_char(raw: bytes) -> str: def _decode_char(raw: bytes, encoding: str = "iso-8859-1") -> str | None:
"""Strip trailing spaces (CHAR is space-padded to declared length).""" """Decode CHAR / NCHAR: fixed width, space-padded to the declared length.
return raw.rstrip(b" \x00").decode("iso-8859-1")
A leading ``0x00`` is Informix's NULL marker for these types. Without
that check a NULL CHAR came back as ``''``, indistinguishable from a
genuinely empty one — and ``WHERE c IS NULL`` disagreeing with what
the driver hands you is a nasty thing to debug. Wire evidence for
``CHAR(6)``::
'ab' -> 61 62 20 20 20 20
'' -> 20 20 20 20 20 20 all spaces
NULL -> 00 20 20 20 20 20 leading nul
The two are distinguishable, so we distinguish them. A real value
cannot begin with a nul: character data has no use for one, and the
server reserves it precisely as this marker.
"""
if raw[:1] == b"\x00":
return None
return raw.rstrip(b" \x00").decode(encoding)
def _decode_varchar(raw: bytes) -> str | None: def _decode_varchar(raw: bytes, encoding: str = "iso-8859-1") -> str | None:
"""VARCHAR — variable-length string. NULL is the special sentinel ``\\x00`` """VARCHAR — variable-length string. NULL is the special sentinel ``\\x00``
(single nul byte). The row decoder peels off the length prefix and passes (single nul byte). The row decoder peels off the length prefix and passes
the content here. Note: VARCHAR cannot contain embedded nuls anyway, so the content here. Note: VARCHAR cannot contain embedded nuls anyway, so
@ -207,7 +234,7 @@ def _decode_varchar(raw: bytes) -> str | None:
""" """
if raw == b"\x00": if raw == b"\x00":
return None return None
return raw.rstrip(b"\x00").decode("iso-8859-1") return raw.rstrip(b"\x00").decode(encoding)
def _decode_bool(raw: bytes) -> bool: def _decode_bool(raw: bytes) -> bool:
@ -217,9 +244,44 @@ def _decode_bool(raw: bytes) -> bool:
return raw[0] in (ord("t"), ord("T"), 1) return raw[0] in (ord("t"), ord("T"), 1)
def _decode_int8(raw: bytes) -> int | None:
"""INT8 / SERIAL8 — the *legacy* 64-bit integer, 10 bytes on the wire.
NOT the same as BIGINT (52) / BIGSERIAL (53), which are plain 8-byte
big-endian. The INT8 layout mirrors ``ifx_int8_t`` and splits the
magnitude across two 32-bit halves in the opposite order you'd guess:
bytes 0-1 sign word: 0 = NULL, 1 = positive, -1 (0xFFFF) = negative
bytes 2-5 LOW 32 bits (unsigned, big-endian)
bytes 6-9 HIGH 32 bits (unsigned, big-endian)
Verified against Informix 12.10.FC12W1DE and 15.0.1.0.3DE, which emit
byte-identical encodings::
123456789012 -> 00 01 | be 99 1a 14 | 00 00 00 1c
-123456789012 -> ff ff | be 99 1a 14 | 00 00 00 1c
42 -> 00 01 | 00 00 00 2a | 00 00 00 00
NULL -> 00 00 | 00 00 00 00 | 00 00 00 00
Note the magnitude bytes are identical for +n and -n — the sign lives
entirely in the leading word, so this is sign-magnitude, not two's
complement. Decoding it as a signed 64-bit integer gives the wrong
answer for every negative value.
"""
if len(raw) < 10:
raise ValueError(f"INT8 payload too short: {len(raw)} bytes, need 10")
sign = _UNPACK_SHORT(raw[0:2])[0]
if sign == 0:
return None
low = int.from_bytes(raw[2:6], "big", signed=False)
high = int.from_bytes(raw[6:10], "big", signed=False)
value = (high << 32) | low
return -value if sign < 0 else value
def _decode_date(raw: bytes) -> datetime.date | None: def _decode_date(raw: bytes) -> datetime.date | None:
"""4-byte big-endian signed int = day count from 1899-12-31. NULL = 0x80000000.""" """4-byte big-endian signed int = day count from 1899-12-31. NULL = 0x80000000."""
days = struct.unpack("!i", raw)[0] days = _UNPACK_INT(raw)[0]
if days == -0x80000000: if days == -0x80000000:
return None return None
return _INFORMIX_DATE_EPOCH + datetime.timedelta(days=days) return _INFORMIX_DATE_EPOCH + datetime.timedelta(days=days)
@ -499,6 +561,21 @@ def _decode_decimal(raw: bytes) -> decimal.Decimal | None:
# slice column values out of an SQ_TUPLE payload for fixed-width types. # slice column values out of an SQ_TUPLE payload for fixed-width types.
# Variable-width types (CHAR, VARCHAR, DECIMAL, etc.) are length-prefixed # Variable-width types (CHAR, VARCHAR, DECIMAL, etc.) are length-prefixed
# on the wire and don't appear in this table. # on the wire and don't appear in this table.
#
# INVARIANT — every key here MUST be decodable by
# ``_decode_base(tc, raw, encoding)`` with NO per-column qualifier
# inspection (no ``col.encoded_length`` lookup, no extended_id check,
# no extended_name check). This is **load-bearing for correctness**:
# ``_resultset.parse_tuple_payload`` dispatches all ``FIXED_WIDTHS``
# types through a single fast-path branch that does not pass
# ``col.encoded_length`` to the decoder. If a new fixed-width type
# needs qualifier bits (the way DATETIME and INTERVAL do — both
# absent from this table for exactly that reason), give it its own
# explicit branch in ``parse_tuple_payload`` instead of adding it
# here. A test in ``tests/test_resultset_invariants.py`` enforces the
# disjointness of this set against every other dispatch branch's
# type set; another test enforces that every key here has a decoder
# in DECODERS.
FIXED_WIDTHS: dict[int, int] = { FIXED_WIDTHS: dict[int, int] = {
IfxType.SMALLINT: 2, IfxType.SMALLINT: 2,
IfxType.INT: 4, IfxType.INT: 4,
@ -509,17 +586,38 @@ FIXED_WIDTHS: dict[int, int] = {
IfxType.BIGSERIAL: 8, IfxType.BIGSERIAL: 8,
IfxType.DATE: 4, IfxType.DATE: 4,
IfxType.BOOL: 1, IfxType.BOOL: 1,
# INT8/SERIAL8 are fixed-width at 10 bytes — NOT 8. See _decode_int8.
# Omitting these was a silent-corruption bug: they fell through to the
# unknown-type path, which surfaces ``encoded_length`` raw bytes.
# ``encoded_length`` happens to be 10 for INT8, so the stream stayed
# aligned and the only symptom was a bytes object where an int belonged.
IfxType.INT8: 10,
IfxType.SERIAL8: 10,
} }
# Phase 2 MVP decoders. Phase 6+ adds DATETIME, INTERVAL, DECIMAL, # Phase 2 MVP decoders. Phase 6+ adds DATETIME, INTERVAL, DECIMAL,
# MONEY, LVARCHAR, BYTE/TEXT, BLOB/CLOB, ROW, COLLECTION. # MONEY, LVARCHAR, BYTE/TEXT, BLOB/CLOB, ROW, COLLECTION.
#
# INVARIANT — KEYS MUST REMAIN ≤ 0xFF (255). This is **load-bearing for
# correctness**, not just a convention. ``_decode_base`` (below) skips
# the ``base_type`` flag strip for performance; its safety relies on
# the fact that any *flagged* type code (NOTNULLABLE=0x100,
# DISTINCT=0x800, etc., per ``_types.py``) is ≥ 256 and therefore
# cannot collide with a DECODERS key in [0, 255]. If you add a decoder
# for a type code ≥ 256 (e.g., CLOB=0x65 itself is fine, but anything
# that uses bits ≥ 0x100 in its identifier), the collision-free
# guarantee weakens and ``_decode_base`` could silently dispatch to
# the wrong decoder when handed a flagged input. Either keep keys ≤
# 0xFF, or restore the ``base_type()`` call inside ``_decode_base``.
DECODERS: dict[int, DecoderFn] = { DECODERS: dict[int, DecoderFn] = {
IfxType.SMALLINT: _decode_smallint, IfxType.SMALLINT: _decode_smallint,
IfxType.INT: _decode_int, IfxType.INT: _decode_int,
IfxType.SERIAL: _decode_int, IfxType.SERIAL: _decode_int,
IfxType.BIGINT: _decode_bigint, IfxType.BIGINT: _decode_bigint,
IfxType.BIGSERIAL: _decode_bigint, IfxType.BIGSERIAL: _decode_bigint,
IfxType.INT8: _decode_int8,
IfxType.SERIAL8: _decode_int8,
IfxType.SMFLOAT: _decode_smfloat, IfxType.SMFLOAT: _decode_smfloat,
IfxType.FLOAT: _decode_float, IfxType.FLOAT: _decode_float,
IfxType.CHAR: _decode_char, IfxType.CHAR: _decode_char,
@ -534,23 +632,71 @@ DECODERS: dict[int, DecoderFn] = {
} }
def decode(type_code: int, raw: bytes) -> object: _STRING_DECODER_TYPES = frozenset({
"""Decode ``raw`` bytes for the given IDS type code into a Python value. int(IfxType.CHAR),
int(IfxType.VARCHAR),
int(IfxType.NCHAR),
int(IfxType.NVCHAR),
int(IfxType.LVARCHAR),
})
The high-bit flags (NOTNULLABLE etc.) are stripped before lookup.
Raises ``KeyError`` for unsupported types — Phase 6+ adds the rest. def _decode_base(base_tc: int, raw: bytes, encoding: str = "iso-8859-1") -> object:
"""Internal fast-path dispatch given an *already-base-typed* type code.
INVARIANT: ``base_tc`` MUST be base-typed (high-bit flags stripped).
Caller's responsibility — this function does NOT call ``base_type()``.
The producer-side counterpart of this invariant lives in
:func:`informix_db._resultset.parse_describe` — see the INVARIANT
comment at the ``ColumnInfo`` construction site. The contract is
bidirectional: producer must base-type before storing in
``ColumnInfo.type_code``; consumer (here) trusts that contract.
Used by ``parse_tuple_payload`` to skip the redundant base-type
strip when iterating column-by-column over a row payload. The
public ``decode()`` function below wraps this and strips flags
for callers who don't know whether they have a raw or base-typed
input.
Same dispatch logic as ``decode()`` — no behavior delta when
invariant holds. If a flagged type code reaches here, the
``DECODERS.get`` lookup will miss and ``NotImplementedError``
fires with a misleading message — the failure mode is loud,
not silent. The no-collision guarantee depends on DECODERS keys
staying ≤ 0xFF; see the INVARIANT comment at ``DECODERS`` below.
""" """
base = base_type(type_code) decoder = DECODERS.get(base_tc)
decoder = DECODERS.get(base)
if decoder is None: if decoder is None:
raise NotImplementedError( raise NotImplementedError(
f"decoder for IDS type code {base} not yet implemented " f"decoder for IDS type code {base_tc} not yet implemented "
f"(Phase 2 MVP supports: SMALLINT, INT, BIGINT, REAL, FLOAT, " f"(Phase 2 MVP supports: SMALLINT, INT, BIGINT, REAL, FLOAT, "
f"CHAR, VARCHAR, BOOL, DATE)" f"CHAR, VARCHAR, BOOL, DATE)"
) )
if base_tc in _STRING_DECODER_TYPES:
return decoder(raw, encoding)
return decoder(raw) return decoder(raw)
def decode(type_code: int, raw: bytes, encoding: str = "iso-8859-1") -> object:
"""Decode ``raw`` bytes for the given IDS type code into a Python value.
The high-bit flags (NOTNULLABLE etc.) are stripped before lookup.
Raises ``NotImplementedError`` for unsupported types — Phase 6+
adds the rest.
``encoding`` is honored for string types (CHAR/VARCHAR/NCHAR/NVCHAR/
LVARCHAR) and ignored otherwise — only those decoders touch user
text. Pass the connection's ``encoding`` (derived from CLIENT_LOCALE)
so multibyte locales round-trip correctly.
Public API — accepts type codes with high-bit flags. Internal hot-path
callers that know their type code is already base-typed should call
:func:`_decode_base` directly to skip the redundant flag strip.
"""
return _decode_base(base_type(type_code), raw, encoding)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Encoders for parameter binding (Phase 4) # Encoders for parameter binding (Phase 4)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -579,14 +725,32 @@ def _encode_bigint(value: int) -> EncodedParam:
return (52, 0x1300, value.to_bytes(8, "big", signed=True)) return (52, 0x1300, value.to_bytes(8, "big", signed=True))
def _encode_str(value: str) -> EncodedParam: def _encode_str(value: str, encoding: str = "iso-8859-1") -> EncodedParam:
"""Encode a Python str as Informix CHAR (type=0, length-prefixed). """Encode a Python str as Informix CHAR (type=0, length-prefixed).
JDBC sends Java strings as CHAR (type=0) on the wire — the server JDBC sends Java strings as CHAR (type=0) on the wire — the server
handles conversion to the actual column type (CHAR/VARCHAR/NVARCHAR). handles conversion to the actual column type (CHAR/VARCHAR/NVARCHAR).
Format: ``[short length][bytes]`` (writePadded adds even-byte pad). Format: ``[short length][bytes]`` (writePadded adds even-byte pad).
``encoding`` honors the connection's ``CLIENT_LOCALE``: pass
``"utf-8"`` for ``en_US.utf8`` connections so multi-byte chars
round-trip rather than crashing on UnicodeEncodeError.
A character outside the configured codec raises :class:`DataError`
rather than letting Python's :class:`UnicodeEncodeError` bubble up —
this matches PEP 249's category for "value can't fit the column"
and lets clean exception-handling work (``except informix_db.Error``).
""" """
encoded = value.encode("iso-8859-1") from .exceptions import DataError
try:
encoded = value.encode(encoding)
except UnicodeEncodeError as exc:
raise DataError(
f"cannot encode parameter under client_locale codec "
f"{encoding!r}: {exc.reason} at position {exc.start}-{exc.end}. "
f"Connect with a wider locale (e.g., 'en_US.utf8') if your "
f"data contains characters outside this codec."
) from exc
raw = len(encoded).to_bytes(2, "big") + encoded raw = len(encoded).to_bytes(2, "big") + encoded
return (0, 0, raw) return (0, 0, raw)
@ -597,8 +761,28 @@ def _encode_float(value: float) -> EncodedParam:
def _encode_bool(value: bool) -> EncodedParam: def _encode_bool(value: bool) -> EncodedParam:
"""Encode a Python bool as Informix BOOLEAN (type=45, 1 byte).""" """Encode a Python bool by binding the literal ``'t'`` / ``'f'``.
return (45, 0, b"\x01" if value else b"\x00")
Informix BOOLEAN has no bindable binary form we could find. This used
to send ``(45, 0, b"\\x01")``, and type 45 is not something the server
accepts as a bind type — it simply stopped responding, so any
``execute`` with a bool parameter **hung until the read timeout**, or
forever if none was set. Type 41 (which is how BOOLEAN is *described*
in results) hangs the same way; the descriptor type and the bind type
are not interchangeable.
Sending ``'t'`` / ``'f'`` as CHAR and letting the server cast on
assignment is what works, and it mirrors Informix's own literal
syntax (``INSERT ... VALUES ('t')``). It also matches how
:func:`_encode_str` already leans on server-side conversion rather
than trying to match the destination column's exact type.
Note the asymmetry with the read path, which is correct and stays
as it is: BOOLEAN comes *back* as UDTFIXED (41) wrapped in a UDT
envelope whose payload byte is ``0x74``/``0x66`` — ASCII ``t``/``f``,
the same characters we send here.
"""
return _encode_str("t" if value else "f")
def _encode_date(value: datetime.date) -> EncodedParam: def _encode_date(value: datetime.date) -> EncodedParam:
@ -614,9 +798,22 @@ def _encode_date(value: datetime.date) -> EncodedParam:
def _encode_datetime(value: datetime.datetime) -> EncodedParam: def _encode_datetime(value: datetime.datetime) -> EncodedParam:
"""Encode a Python ``datetime.datetime`` as Informix DATETIME (type=10). """Encode a Python ``datetime.datetime`` as Informix DATETIME (type=10).
Emit YEAR TO SECOND form — covers the common case of stored Emits YEAR TO SECOND when ``microsecond`` is zero, and YEAR TO
timestamps without microseconds. (Phase 6.x can add YEAR TO FRACTION(5) when it isn't.
FRACTION(N) variants if microseconds are needed.)
The conditional matters. Emitting YEAR TO SECOND unconditionally
silently discarded sub-second precision on every bind: a
``DATETIME YEAR TO FRACTION(5)`` column handed
``datetime(..., microsecond=120000)`` stored ``.00000``, with no
error. Only widening when there is a fraction to carry keeps the
long-exercised YEAR TO SECOND path byte-identical for the common
case, and Informix converts between qualifiers on assignment, so a
FRACTION(5) bind into a narrower column truncates server-side
rather than failing.
FRACTION(5) is the widest Informix supports and holds 10 µs
resolution; Python's microsecond field is finer, so the last digit
is dropped. Same trade-off ``_encode_timedelta`` already makes.
Format (per ``Decimal.javaToIfx`` line 457): Format (per ``Decimal.javaToIfx`` line 457):
byte[0..1] = short total length of data following (= digit_count/2 + 1) byte[0..1] = short total length of data following (= digit_count/2 + 1)
@ -640,10 +837,20 @@ def _encode_datetime(value: datetime.datetime) -> EncodedParam:
(value.second, 2), (value.second, 2),
] ]
digit_str = "".join(f"{v:0{w}d}" for v, w in fields) # 14 digits digit_str = "".join(f"{v:0{w}d}" for v, w in fields) # 14 digits
if value.microsecond:
# 6 fraction digits = exactly 3 more BCD pairs, so the digit
# string stays even and the exponent byte is unchanged (the
# integer part is still 7 base-100 pairs). FRACTION(5) carries
# 5 significant digits; the 6th is padding the wire format
# requires. Qualifier: digit_count=19, start=YEAR(0), end=
# FRACTION(5)=15 — matching what the decoder reads back.
digit_str += f"{value.microsecond:06d}" # -> 20 digits
prec = (19 << 8) | (0 << 4) | 15
else:
prec = (14 << 8) | (0 << 4) | 10
digit_bytes = bytes(int(digit_str[i : i + 2]) for i in range(0, len(digit_str), 2)) digit_bytes = bytes(int(digit_str[i : i + 2]) for i in range(0, len(digit_str), 2))
inner = bytes([0xC7]) + digit_bytes # 8 bytes (1 exp + 7 BCD pairs) inner = bytes([0xC7]) + digit_bytes # 1 exp byte + 7 or 10 BCD pairs
raw = len(inner).to_bytes(2, "big") + inner # +2 byte length prefix = 10 bytes raw = len(inner).to_bytes(2, "big") + inner # +2 byte length prefix
prec = (14 << 8) | (0 << 4) | 10
return (10, prec, raw) return (10, prec, raw)
@ -883,11 +1090,17 @@ def _encode_decimal(value: decimal.Decimal) -> EncodedParam:
return (5, prec_short, raw) return (5, prec_short, raw)
def encode_param(value: object) -> EncodedParam: def encode_param(
value: object, encoding: str = "iso-8859-1"
) -> EncodedParam:
"""Pick an encoder based on the Python value's type. """Pick an encoder based on the Python value's type.
Returns ``(ifx_type, precision_short, raw_bytes)`` for the parameter. Returns ``(ifx_type, precision_short, raw_bytes)`` for the parameter.
Returns ``(0, 0, b"")`` and the caller must use indicator=-1 for None. Returns ``(0, 0, b"")`` and the caller must use indicator=-1 for None.
``encoding``: Python codec name for ``str`` values. Should match
the connection's ``CLIENT_LOCALE``. Caller (typically the cursor)
forwards ``conn.encoding``.
""" """
if value is None: if value is None:
return (0, 0, b"") return (0, 0, b"")
@ -901,7 +1114,7 @@ def encode_param(value: object) -> EncodedParam:
if isinstance(value, float): if isinstance(value, float):
return _encode_float(value) return _encode_float(value)
if isinstance(value, str): if isinstance(value, str):
return _encode_str(value) return _encode_str(value, encoding=encoding)
# NB: datetime.datetime is a subclass of datetime.date — must check # NB: datetime.datetime is a subclass of datetime.date — must check
# datetime BEFORE date. # datetime BEFORE date.
if isinstance(value, datetime.datetime): if isinstance(value, datetime.datetime):

1251
src/informix_db/cursors.py
View File

File diff suppressed because it is too large Load Diff

124
src/informix_db/pool.py
View File

@ -40,6 +40,7 @@ Design notes:
from __future__ import annotations from __future__ import annotations
import contextlib import contextlib
import logging
import threading import threading
import time import time
from collections.abc import Iterator from collections.abc import Iterator
@ -51,6 +52,20 @@ from .exceptions import (
OperationalError, OperationalError,
) )
# Module-level logger. By default Python loggers without a configured
# handler emit nothing — applications that want pool diagnostics can
# wire up ``logging.getLogger("informix_db.pool")`` to their handler.
_log = logging.getLogger(__name__)
# Phase 27: how long ``release()`` will wait to acquire the connection's
# wire lock before evicting. The wire lock is only contended when
# another thread is mid-operation on the same connection — typically
# because an awaitable was cancelled but its underlying ``to_thread``
# worker is still running. 5 seconds is generous for any normal query
# to finish and short enough that a hung worker doesn't block the pool
# indefinitely.
_RELEASE_WIRE_LOCK_TIMEOUT = 5.0
class PoolClosedError(InterfaceError): class PoolClosedError(InterfaceError):
"""Pool was closed before/during acquire.""" """Pool was closed before/during acquire."""
@ -138,23 +153,31 @@ class ConnectionPool:
self._total -= 1 self._total -= 1
self._safe_close(conn) self._safe_close(conn)
continue continue
# Grow if we have room # Grow if we have room. We release the pool lock during
# the actual connect (slow I/O — login handshake is
# ~10ms on loopback, can be hundreds of ms on real
# networks) so other threads can grow / acquire idles
# in parallel. The two explicit re-acquires below replace
# an older try/finally that called the CPython-private
# ``_lock._is_owned()`` — fragile across versions.
if self._total < self._max_size: if self._total < self._max_size:
self._total += 1 self._total += 1
# Release the lock during connect (slow op) so
# other threads can also grow / acquire idles.
self._lock.release() self._lock.release()
try: try:
try: conn = self._make_connection()
conn = self._make_connection() except Exception:
except Exception: # Re-acquire to roll back the slot reservation
self._lock.acquire() # and notify waiters that capacity is again
self._total -= 1 # available. The outer ``with`` block expects
self._lock.notify() # to own the lock at exit; re-acquiring keeps
raise # that invariant.
finally: self._lock.acquire()
if not self._lock._is_owned(): self._total -= 1
self._lock.acquire() self._lock.notify()
raise
# Success path — re-acquire so the outer ``with``
# block has the lock to release on exit.
self._lock.acquire()
return conn return conn
# At max — wait for a free connection # At max — wait for a free connection
remaining = None remaining = None
@ -173,9 +196,82 @@ class ConnectionPool:
Pass ``broken=True`` to evict it (e.g., after a connection- Pass ``broken=True`` to evict it (e.g., after a connection-
related exception). Broken connections are closed and the related exception). Broken connections are closed and the
slot is freed for a new connection. slot is freed for a new connection.
**Session reset (Phase 26)**: any uncommitted transaction is
rolled back before the connection rejoins the idle list.
Without this, request A's uncommitted writes would be visible
to (and could be inadvertently committed by) request B who
reuses the same connection. If the rollback itself fails
(dead socket, etc.), the connection is evicted instead of
recycled — half-state is never returned to the pool. Failures
are logged at WARNING level via ``logging.getLogger(
"informix_db.pool")``.
**Concurrency (Phase 27)**: the rollback acquires the
connection's ``_wire_lock`` with a ~5s timeout before sending.
If another thread is mid-operation on the connection (e.g.,
a still-running worker after ``asyncio.wait_for`` cancelled
the awaitable), the release path either waits for them to
finish (if quick) or evicts the connection (if they exceed
the timeout). Either way, no two threads ever interleave
bytes on the socket.
""" """
if broken or self._closed or conn.closed:
with self._lock:
self._total -= 1
self._safe_close(conn)
self._lock.notify()
return
# Rollback any open transaction outside the pool lock —
# ``Connection.rollback`` does a wire round-trip and we don't
# want to block other pool operations during that. The
# connection isn't yet in ``_idle``, and ``_total`` already
# counts it as "owned by us", so no other thread can grab it
# while we're working.
#
# Phase 27: acquire the connection's wire lock with a timeout
# before rolling back. If another thread holds it (typically a
# cancelled-async worker that's still running on the socket),
# we evict instead of risking interleaved I/O. The connection
# is unsafe until that worker finishes; the next caller would
# rather get a fresh connection than a poisoned one.
if conn._in_transaction:
if not conn._wire_lock.acquire(
timeout=_RELEASE_WIRE_LOCK_TIMEOUT
):
_log.warning(
"wire lock held %ss on release; evicting connection "
"(another thread is still mid-operation — likely a "
"cancelled async query whose worker hasn't finished)",
_RELEASE_WIRE_LOCK_TIMEOUT,
)
with self._lock:
self._total -= 1
self._safe_close(conn)
self._lock.notify()
return
try:
conn.rollback()
except Exception as exc:
# Rollback failed — the wire is in an unknown state.
# Evict the connection rather than recycling unsafe
# state to the next acquirer. Log so the eviction is
# debuggable rather than silent.
_log.warning(
"rollback-on-release failed; evicting connection: %r",
exc,
)
with self._lock:
self._total -= 1
self._safe_close(conn)
self._lock.notify()
return
finally:
conn._wire_lock.release()
with self._lock: with self._lock:
if broken or self._closed or conn.closed: if self._closed:
# Pool was closed while we were rolling back. Don't
# add to ``_idle`` — close the connection instead.
self._total -= 1 self._total -= 1
self._safe_close(conn) self._safe_close(conn)
self._lock.notify() self._lock.notify()

0
src/informix_db/py.typed Normal file
View File

180
src/informix_db/rows.py Normal file
View File

@ -0,0 +1,180 @@
"""Rows that can be read by position, by column name, or by attribute.
PEP 249 only requires a sequence, and a sequence is what the driver
returns by default. That is fine for ``SELECT a, b`` and steadily worse
as the projection grows: ``row[11]`` tells a reader nothing, and stays
correct only until somebody adds a column in the middle.
Opting in with ``row_factory=Row`` gives the shape ``pyodbc`` and
``mssql-python`` provide, all three at once::
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur.execute("SELECT tabid, tabname FROM systables")
row = cur.fetchone()
row[0], row["tabname"], row.tabname
It is opt-in rather than the default because it is not free, and the
driver's whole argument is that pure Python can stay within noise of the
C driver on bulk fetch. Measured on a 20,000-row five-column fetch:
**37.2 ms with tuples, 40.8 ms with Row**, so about 9%. Defaulting it on
would move the published 1.05-1.15x ratio against IfxPy to roughly
1.15-1.25x, which is not a trade to make on everybody's behalf.
Where the 9% goes, against a plain tuple: about 30 ns per row to
construct, about 39 ns per ``row[0]`` because supporting ``row["name"]``
means ``__getitem__`` is a Python method rather than C-level tuple
indexing, and about 13 ns per unpack since CPython's fast path for
``a, b = row`` applies to exact tuples and not to subclasses.
Worth it for readability in application code. Not worth paying in a bulk
export that never looks at a column by name, which is exactly why it is
a choice.
``Row`` subclasses ``tuple``, so ``row == (1, "x")`` is still true and
existing code keeps working unchanged.
**A column always beats a method of the same name.** ``tuple`` defines
``count`` and ``index``; this class adds ``keys``, ``_asdict`` and
``_fields``. A column named any of those would otherwise resolve to the
method and hand back a bound method instead of a value, which is exactly
the shape of the framing bugs this driver spent a fortnight removing: a
plausible-looking wrong answer, in silence. So every such name gets a
descriptor and the column wins, and the reserved set is *computed* from
the class rather than hand-listed, because hand-listing it is what
missed ``keys``, ``_asdict`` and ``_fields`` the first time round. The
methods stay reachable through the class: ``tuple.count(row, x)``,
``Row.keys(row)``.
The machinery itself is name-mangled (``__fields`` / ``__map``) so that
shadowing ``_fields`` cannot break ``repr`` or ``_asdict``.
Column names come from ``cursor.description``. Informix folds unquoted
identifiers to lower case, so ``SELECT Config_Key`` is reachable as
``row.config_key``. Expressions get server-generated names that are not
Python identifiers, like ``(count(*))``; those are reachable by
subscript but not as attributes. Duplicate names resolve to the first
occurrence, matching ``pyodbc``. Dunder names cannot be shadowed and
stay subscript-only, which no real schema should notice.
"""
from __future__ import annotations
import operator
from functools import lru_cache
from typing import ClassVar
__all__ = ["Row", "make_row_class"]
class Row(tuple):
"""A result row addressable by position, name, or attribute.
Used as a ``row_factory``. The concrete class handed to each result
set is a subclass carrying that query's column names, built by
:func:`make_row_class`.
"""
__slots__ = ()
# Name-mangled to ``_Row__fields`` / ``_Row__map`` so that a column
# called "_fields" can be shadowed without breaking the machinery
# that reads it. Set on the per-result-set subclass.
__fields: ClassVar[tuple[str, ...]] = ()
__map: ClassVar[dict[str, int]] = {}
def __getitem__(self, key):
# ``key.__class__ is str`` rather than isinstance: this runs on
# every subscript.
if key.__class__ is str:
try:
return tuple.__getitem__(self, self.__map[key])
except KeyError:
raise KeyError(
f"no column named {key!r}; this row has "
f"{list(self.__fields)}"
) from None
return tuple.__getitem__(self, key)
def __getattr__(self, name):
# Only reached when normal attribute lookup has already failed,
# so this costs nothing for names that do not collide.
try:
return tuple.__getitem__(self, self.__map[name])
except KeyError:
raise AttributeError(
f"no column named {name!r}; this row has "
f"{list(self.__fields)}"
) from None
@property
def _fields(self) -> tuple[str, ...]:
"""Column names, in select order. Mirrors ``namedtuple._fields``."""
return self.__fields
def keys(self) -> tuple[str, ...]:
"""Column names, in select order."""
return self.__fields
def _asdict(self) -> dict:
"""A plain ``dict`` of the row.
On duplicate column names the last occurrence wins here, while
subscript access gives the first. A dict cannot represent both,
and quietly dropping a duplicate is better than raising on a
query that is otherwise fine.
"""
return dict(zip(self.__fields, self, strict=True))
def __repr__(self) -> str:
fields = self.__fields
if len(fields) != len(self):
return tuple.__repr__(self)
body = ", ".join(
f"{name}={value!r}"
for name, value in zip(fields, self, strict=True)
)
return f"Row({body})"
def __reduce__(self):
# The per-result-set class is created at runtime and cannot be
# pickled by reference, so rebuild it from the field names.
return (_rebuild_row, (self.__fields, tuple(self)))
# Every non-dunder attribute a Row already answers to. A column with one
# of these names gets a descriptor so the column wins. Computed, not
# hand-listed: the hand-listed version covered ``count`` and ``index``
# and silently missed ``keys``, ``_asdict`` and ``_fields``.
_RESERVED = frozenset(
name for name in dir(Row) if not name.startswith("__")
) - {"_Row__fields", "_Row__map"}
def _rebuild_row(fields: tuple[str, ...], values: tuple):
return make_row_class(fields)(values)
@lru_cache(maxsize=256)
def make_row_class(fields: tuple[str, ...]) -> type[Row]:
"""Build (and cache) the row class for one column-name shape.
Cached because a class per ``execute()`` would put a ``type()`` call
on the path of every small query, and applications run the same
handful of statement shapes over and over. Keyed on the names alone,
so two queries selecting the same columns share a class.
"""
namespace: dict = {
"__slots__": (),
"_Row__fields": fields,
# First occurrence wins on duplicates, matching pyodbc. Building
# the map in reverse and letting earlier entries overwrite later
# ones is the shortest way to say that.
"_Row__map": {
name: i for i, name in reversed(list(enumerate(fields)))
},
}
for i, name in enumerate(fields):
if name in _RESERVED:
# A column of this name would otherwise resolve to a method.
namespace[name] = property(operator.itemgetter(i))
return type("Row", (Row,), namespace)

117
tests/_proxy.py Normal file
View File

@ -0,0 +1,117 @@
"""Controlled TCP proxy for fault-injection testing.
Spins up a one-shot proxy in a thread that forwards bytes between the
test client and the real Informix server. The test can call
:meth:`ControlledProxy.kill` at any moment to simulate a network drop
or server crash mid-conversation.
Usage::
proxy = ControlledProxy("127.0.0.1", 9088)
proxy.start()
conn = informix_db.connect(host="127.0.0.1", port=proxy.port, ...)
cur = conn.cursor()
cur.execute(...)
proxy.kill() # simulated network drop
cur.fetchone() # should raise OperationalError
proxy.close()
"""
from __future__ import annotations
import contextlib
import socket
import threading
class ControlledProxy:
"""A TCP forwarder we can kill at will.
Listens on an ephemeral port on 127.0.0.1, forwards bytes to/from
the upstream Informix server. Forwarding runs in two daemon threads
(one per direction). ``kill()`` closes both sockets, simulating a
network drop. Idempotent.
"""
def __init__(self, upstream_host: str, upstream_port: int):
self.upstream_host = upstream_host
self.upstream_port = upstream_port
self._listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._listener.bind(("127.0.0.1", 0))
self._listener.listen(1)
self.port = self._listener.getsockname()[1]
self._client: socket.socket | None = None
self._upstream: socket.socket | None = None
self._threads: list[threading.Thread] = []
self._killed = False
def start(self) -> None:
"""Begin accepting on a daemon thread (returns immediately)."""
def accept_and_forward() -> None:
try:
client, _ = self._listener.accept()
upstream = socket.create_connection(
(self.upstream_host, self.upstream_port), timeout=5.0
)
self._client = client
self._upstream = upstream
t1 = threading.Thread(
target=self._pump, args=(client, upstream), daemon=True
)
t2 = threading.Thread(
target=self._pump, args=(upstream, client), daemon=True
)
t1.start()
t2.start()
self._threads.extend([t1, t2])
except Exception:
pass # caller's connect will fail visibly
accept_thread = threading.Thread(target=accept_and_forward, daemon=True)
accept_thread.start()
def _pump(self, src: socket.socket, dst: socket.socket) -> None:
try:
while not self._killed:
data = src.recv(8192)
if not data:
break
dst.sendall(data)
except OSError:
pass
def kill(self) -> None:
"""Sever the connection. Mimics network failure / server crash.
Closes both sockets *brutally* (SO_LINGER=0 for RST instead of FIN)
so the client sees a connection-aborted error, not a clean EOF.
"""
self._killed = True
for sock in (self._client, self._upstream):
if sock is not None:
with contextlib.suppress(OSError):
# Force RST instead of FIN: SO_LINGER=0
import struct
sock.setsockopt(
socket.SOL_SOCKET, socket.SO_LINGER,
struct.pack("ii", 1, 0),
)
with contextlib.suppress(OSError):
sock.close()
self._client = None
self._upstream = None
def close(self) -> None:
"""Final cleanup — closes everything including the listener."""
self.kill()
with contextlib.suppress(OSError):
self._listener.close()
def __enter__(self) -> ControlledProxy:
self.start()
return self
def __exit__(self, *_exc: object) -> None:
self.close()

97
tests/benchmarks/README.md Normal file
View File

@ -0,0 +1,97 @@
# Benchmarks (Phase 21)
Performance baselines for `informix-db`. Two layers:
1. **Codec micro-benchmarks** (`test_codec_perf.py`) — pure CPU, no
server. These set the *ceiling* for what end-to-end can achieve.
Run with `make bench-codec`. Suitable for CI's pre-merge job.
2. **End-to-end benchmarks** — exercise the full
PREPARE → BIND → EXECUTE → FETCH → CLOSE → RELEASE round-trip.
Need an Informix container (`make ifx-up`). Run with `make bench`.
## Headline numbers (baseline 2026-05-04, x86_64 Linux, dev container on loopback)
| Operation | Mean | Ops/sec |
|-|-:|-:|
| `decode(int)` (per cell) | 181 ns | 5.5M |
| `parse_tuple_payload(5 cols)` (per row) | 2.87 µs | 350K |
| `encode_param(int)` (per param) | 103 ns | 9.7M |
| `SELECT 1` round-trip | 177 µs | 5,650 |
| Pool acquire + tiny query + release | 295 µs | 3,400 |
| **Cold connect + close** (login handshake) | **11.2 ms** | **89** |
| 1000-row SELECT * | 1.56 ms | 640 |
| INSERT (single, prepared) | 1.88 ms | 530 |
| `executemany(100)` autocommit=True | 181 ms | ~550 rows/sec |
| `executemany(1000)` autocommit=True | 1.72 s | ~580 rows/sec |
| **`executemany(1000)` in single transaction** | **32 ms** | **~31,000 rows/sec** |
### What these tell you
- **Pool gives 72× speedup** over cold connect. If your app opens a
connection per request, fix that first.
- **Wrap bulk INSERTs in a transaction.** That's a **53× speedup** over
the autocommit-True default. With autocommit on, each row forces the
server to flush its transaction log; in transaction mode the flush
happens once at COMMIT. Per-row cost drops from 1.72 ms (storage-bound)
to 32 µs (pure protocol). PEP 249's default `autocommit=False` was
designed for this — we just default to `False`.
- **Codec is not the bottleneck.** Per-row decode (2.9 µs) is 1000× faster
than wire round-trip (177 µs for `SELECT 1`). Network and server-side
cost dominate.
- **UTF-8 carries no measurable cost.** `decode_varchar_utf8` runs at
216 ns vs `decode_varchar_short` at 170 ns — the 27% delta is the
multibyte string walk inherent in UTF-8 decoding, not Phase 20 overhead.
### Performance gotchas
- **`autocommit=True` + `executemany` is the slowest reasonable pattern.**
Use it only when each row genuinely needs to land independently. For
bulk loads, default `autocommit=False` and call `conn.commit()` at the
end of the batch.
- **Single `INSERT` in a tight loop is 1.88 ms each** — strictly worse
than `executemany` (which saves PREPARE/RELEASE overhead). If you find
yourself looping over `cur.execute("INSERT...")` hundreds of times,
switch to `executemany`.
- **Cold connect is 11 ms.** The login handshake is *expensive* compared
to anything you'll do with the connection. Pool everything in
long-lived processes.
## Regression policy
`baseline.json` is committed and represents the dev-container baseline.
Compare a current run against it with:
```bash
uv run pytest tests/benchmarks/ -m benchmark --benchmark-only \
--benchmark-compare=tests/benchmarks/baseline.json \
--benchmark-compare-fail=mean:25%
```
A 25% mean-regression fails the run. Adjust the threshold per CI noise
profile. CI's loopback-network-on-shared-runner is noisier than dev
container on a quiet box — start permissive and tighten as you collect
runs.
## Updating the baseline
When you intentionally change performance (an optimization, or accept
a regression for correctness), refresh:
```bash
make bench-save # writes .results/0001_run.json
cp tests/benchmarks/.results/Linux-CPython-*/0001_run.json tests/benchmarks/baseline.json
git add tests/benchmarks/baseline.json
```
Document the change in CHANGELOG so reviewers know why the floor moved.
## Files
- `test_codec_perf.py` — codec dispatch (decode, encode_param, parse_tuple_payload)
- `test_select_perf.py` — SELECT round-trips, single + multi-row
- `test_insert_perf.py` — INSERT single + executemany throughput
- `test_pool_perf.py` — cold connect vs pool acquire/release
- `test_async_perf.py` — async-path latency + concurrent throughput
- `conftest.py` — long-lived `bench_conn` and 1k-row `bench_table` fixtures
- `baseline.json` — committed baseline for regression comparison
- `.results/` — gitignored; per-run output from `make bench-save`

21
tests/benchmarks/__init__.py Normal file
View File

@ -0,0 +1,21 @@
"""Phase 21 — performance benchmarks for informix-db.
These tests are gated behind the ``benchmark`` marker and excluded from
the default ``pytest`` run. To run:
make bench # all benchmarks
uv run pytest -m benchmark tests/benchmarks/test_codec_perf.py
Codec micro-benchmarks (``test_codec_perf.py``) run without a server
and are fast enough for tight inner-loop iteration. End-to-end
benchmarks (SELECT/INSERT/pool) require an Informix container.
Output goes to ``.benchmarks/`` (gitignored). Persistent baseline at
``tests/benchmarks/baseline.json`` is updated manually with::
uv run pytest -m benchmark --benchmark-only \
--benchmark-save=baseline --benchmark-storage=tests/benchmarks/
Then copy ``.benchmarks/Linux-CPython-X.Y/000N_baseline.json`` to
``tests/benchmarks/baseline.json``.
"""

1256
tests/benchmarks/baseline.json Normal file
View File

File diff suppressed because it is too large Load Diff

42
tests/benchmarks/compare/Dockerfile.ifxpy Normal file
View File

@ -0,0 +1,42 @@
# IfxPy benchmark container — Ubuntu 20.04 base for libcrypt.so.1 compat.
#
# Runs side-by-side with the host's `informix-db` benchmarks against the
# same Informix dev container at host.docker.internal:9088. Both drivers
# hit the same server over loopback equivalent (Docker's host-gateway
# DNS), making the comparison apples-to-apples on the wire layer.
#
# Build:
# docker build -f tests/benchmarks/compare/Dockerfile.ifxpy \
# -t ifxpy-bench tests/benchmarks/compare/
#
# Run:
# docker run --rm --network=host ifxpy-bench
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
python3.9 python3-pip python3.9-dev \
build-essential \
libcrypt1 libcrypt-dev \
curl ca-certificates tar \
&& rm -rf /var/lib/apt/lists/*
# IfxPy needs setuptools <58 because its setup.py uses use_2to3
RUN python3.9 -m pip install --upgrade "pip<24" "setuptools<58" wheel
# Permissive CFLAGS bypass GCC's modern strict-pointer-types check.
ENV CFLAGS="-Wno-incompatible-pointer-types -Wno-error"
RUN python3.9 -m pip install IfxPy
# Pull OneDB ODBC drivers (92MB) — IfxPy's setup.py downloaded headers
# but not the runtime libs.
RUN mkdir -p /opt/onedb && cd /opt/onedb && \
curl -sSL https://hcl-onedb.github.io/odbc/OneDB-Linux64-ODBC-Driver.tar | tar xf -
ENV INFORMIXDIR=/opt/onedb/onedb-odbc-driver
ENV LD_LIBRARY_PATH=$INFORMIXDIR/lib:$INFORMIXDIR/lib/cli:$INFORMIXDIR/lib/esql:$INFORMIXDIR/lib/client:$INFORMIXDIR/gls/dll
# Sanity check: import + smoke connect.
COPY ifxpy_bench.py /opt/ifxpy_bench.py
WORKDIR /opt
CMD ["python3.9", "/opt/ifxpy_bench.py"]

120
tests/benchmarks/compare/README.md Normal file
View File

@ -0,0 +1,120 @@
# `informix-db` vs IfxPy comparison benchmark
Head-to-head benchmarks against [IfxPy](https://pypi.org/project/IfxPy/), the IBM-published C-bound Informix driver, on identical workloads against the same Informix Developer Edition Docker container.
## TL;DR
Using **median + IQR over 10+ rounds** (mean was unreliable on the slow benchmarks — see "Statistical robustness" below). Phase 36 added scaling benchmarks at 1k / 10k / 100k rows so the comparison shape is clearer:
| Benchmark | IfxPy 3.0.5 | informix-db | Result |
|---|---:|---:|---:|
| `select_one_row` | 118 µs | 114 µs | comparable |
| `select_systables_first_10` | 130 µs | 159 µs | IfxPy 22% faster |
| `cold_connect_disconnect` | 11.0 ms | 10.5 ms | comparable |
| **`executemany(1k)` in txn** | 23.5 ms | 23.2 ms | tied |
| **`executemany(10k)` in txn** | 259 ms | **161 ms** | **`informix-db` 1.6× faster** |
| **`executemany(100k)` in txn** | 2376 ms | **1487 ms** | **`informix-db` 1.6× faster** |
| `SELECT 1k rows` | 1.2 ms | 2.7 ms | IfxPy 2.3× faster |
| `SELECT 10k rows` | 11.3 ms | 25.8 ms | IfxPy 2.3× faster |
| `SELECT 100k rows` | 112 ms | 271 ms | IfxPy 2.4× faster |
**Two clear stories:**
**1. Bulk insert: `informix-db` wins 1.6× at scale.** The pipelined `executemany` (Phase 33) sends all N BIND+EXECUTE PDUs to the wire before draining responses, eliminating per-row RTT. IfxPy still pays one synchronous round-trip per `IfxPy.execute(stmt, tuple)` call — that's ~24 µs/row regardless of N. We pay ~15 µs/row at scale (the prepare/release overhead amortizes better at larger N).
**2. Large fetch: IfxPy wins 2.3-2.4× at scale.** Their C-level `fetch_tuple` decoder runs at ~1.1 µs/row; our pure-Python `parse_tuple_payload` runs at ~2.7 µs/row. At 100k rows, the 1.6 µs/row gap accumulates into a 160 ms wall-clock difference. **This is the C-vs-Python codec cost showing up at scale, where it actually matters.**
For everyday-application workloads (single SELECT in a request, INSERT a handful of rows, transactional UPDATE), the two drivers are within 5-25% of each other. For the workloads where the gap widens, the direction depends on what you're doing — bulk-write favors us, bulk-read favors IfxPy.
**The wire-alignment assumption** that makes pipelined `executemany` safe — that Informix sends exactly N responses for N pipelined PDUs even when one row fails — is verified by `tests/test_executemany_pipeline.py` (constraint violation at row 0/100, 99/100, 500/1000).
## Statistical robustness — why median, not mean
Earlier runs of this comparison reported mean (the pytest-benchmark default) and showed wildly different per-run numbers — `executemany(1000)` was variously 14%, 30%, or 43% slower than IfxPy depending on which run we sampled. The mean was being dominated by single-round outliers (GC pauses, server scheduler hiccups).
Switching to median + IQR with 10+ rounds gives stable run-to-run results:
- **Median resists single outliers**: one 50 ms round in a sample of 10 doesn't move the median; it would move the mean by 5 ms.
- **IQR (Q3 – Q1) is the noise estimator**: directly comparable across drivers. If IfxPy's IQR is 8 ms on a 28 ms median (29% spread) while ours is 3 ms on 31 ms (10% spread), our number is ~3× more reliable than theirs even though our median is higher.
- **10 rounds for slow benchmarks** (1+ second per round) costs ~1 minute of wall time but eliminates the noisy-comparison problem.
Both `tests/benchmarks/test_*_perf.py` (host-side, pytest-benchmark) and `ifxpy_bench.py` (container-side, hand-rolled `time.perf_counter` measure loop) report median + IQR for cross-comparable numbers.
## What this means
Conventional wisdom says C beats Python at I/O drivers. Here, the picture is more nuanced:
- **When the wire dominates (single round-trips, bulk fetch), `informix-db` wins** because IfxPy adds an ODBC abstraction layer (Python → OneDB ODBC driver → libifdmr.so → wire) where we go direct (Python → wire).
- **When per-row marshaling dominates (executemany, wider tuple construction), IfxPy wins** because its C-level `execute(stmt, tuple)` is faster than our Python BIND-PDU build.
- **When the wire handshake dominates (cold connect), they tie** because both drivers wait ~11 ms for the server's login response.
The takeaway is that pure-Python doesn't mean "performance compromise" — it means **different overhead distribution**. For most application workloads (web requests doing a handful of small queries), the wire round-trip is what matters, and the abstraction-layer overhead IfxPy carries means `informix-db` is typically the same speed or faster.
## Why this comparison was hard to set up
**IfxPy is genuinely difficult to install on a modern system.** Capturing the install gauntlet for the record:
| Step | Detail |
|---|---|
| 1. Pin Python 3.11 | Python 3.13 fails: IfxPy's `setup.py` uses `use_2to3`, removed from setuptools 58 (October 2021). |
| 2. Pin setuptools <58 | Same root cause. |
| 3. CFLAGS hack | GCC 11+ (default since 2021) escalates the C extension's pointer-type warnings to errors. Need `CFLAGS="-Wno-incompatible-pointer-types -Wno-error"` to demote them. |
| 4. Download OneDB ODBC drivers | A 92 MB tarball from `hcl-onedb.github.io/odbc/`. The `pip install` only fetches headers — the runtime libs are a separate, undocumented download. |
| 5. Set INFORMIXDIR + LD_LIBRARY_PATH | Across four directories (`lib/`, `lib/cli/`, `lib/esql/`, `gls/dll/`). |
| 6. Install `libcrypt.so.1` | The OneDB drivers link against the libcrypt-1 ABI (deprecated in 2018, replaced by libcrypt.so.2). Modern Arch / Fedora 35+ / RHEL 9 ship only libcrypt.so.2; you need a compatibility shim (Ubuntu 20.04 still has it; modern distros need `libxcrypt-compat` or similar). |
| 7. Build runtime container | We use `Dockerfile.ifxpy` here because Ubuntu 20.04 is the most recent base distro that still ships `libcrypt.so.1` natively. |
By contrast, `informix-db`'s install is `pip install informix-db`. No external downloads, no system packages, no LD_LIBRARY_PATH, no Docker required.
## Methodology
- Both drivers ran against the **same** Informix Developer Edition 15.0.1.0.3DE Docker container (`informix-db-test` from `tests/docker-compose.yml`).
- The host runs Arch Linux on x86_64; the IfxPy container runs Ubuntu 20.04 on x86_64. Both reach the server through the loopback path (host's `127.0.0.1:9088` for `informix-db`; `--network=host` for the IfxPy container).
- Each benchmark runs 100/20/3 rounds depending on per-iteration cost; we report the mean. Stddev is small (under 5%) for all reported numbers — within-run jitter doesn't affect the qualitative result.
- Workloads are matched semantically: same SQL, same row counts, same fetch patterns. Where they differ (IfxPy's `IfxPy.fetch_tuple` vs. our `cursor.fetchall`), we use whichever idiom exhausts the cursor in each driver.
## Reproduce
From the project root:
```bash
# 1. Start the dev Informix container
make ifx-up
# 2. Seed the 1k-row test table on the host (using informix-db)
uv run python -c "
import informix_db, contextlib
conn = informix_db.connect(host='127.0.0.1', port=9088,
user='informix', password='in4mix',
database='sysmaster', server='informix', autocommit=True)
cur = conn.cursor()
with contextlib.suppress(Exception): cur.execute('DROP TABLE p21_bench')
cur.execute('CREATE TABLE p21_bench (id INT, name VARCHAR(64), counter INT, value FLOAT, created DATE)')
cur.executemany('INSERT INTO p21_bench VALUES (?, ?, ?, ?, ?)',
[(i, f'row_{i:04d}', i*7, float(i)*1.5, None) for i in range(1000)])
conn.close()
"
# 3. Build + run the IfxPy benchmark container
docker build -f tests/benchmarks/compare/Dockerfile.ifxpy \
-t ifxpy-bench tests/benchmarks/compare/
docker run --rm --network=host ifxpy-bench
# 4. Run informix-db benchmarks for the matched comparison
uv run pytest tests/benchmarks/test_select_perf.py \
tests/benchmarks/test_pool_perf.py \
tests/benchmarks/test_insert_perf.py \
-m benchmark --benchmark-only --benchmark-warmup=on
```
## Files
- `Dockerfile.ifxpy` — Ubuntu 20.04 container with Python 3.9, IfxPy, and OneDB drivers installed
- `ifxpy_bench.py` — IfxPy benchmark workloads (mirrors `tests/benchmarks/test_*_perf.py`)
- This README
## Caveats
- IfxPy 3.0.5 is the latest PyPI version (from October 2020). It's the most actively-maintained C-bound option but hasn't shipped a release in ~5 years.
- Numbers will vary by host, distro, kernel, network stack — re-run on your own hardware before drawing strong conclusions.
- The 1k-row INSERT benchmark uses different APIs (IfxPy's `prepare`+`execute` loop vs our `executemany`); the comparison is by total wall-clock time for the equivalent workload, not by per-call overhead.

317
tests/benchmarks/compare/ifxpy_bench.py Normal file
View File

@ -0,0 +1,317 @@
"""IfxPy comparison benchmark.
Runs the same workloads as ``tests/benchmarks/test_*_perf.py`` against
the same dev-container Informix instance, but using IfxPy (the C-bound
PyPI driver) instead of ``informix-db``. Numbers go straight to stdout;
the host parses them and produces a side-by-side table.
Workloads:
* ``select_one_row`` — single-row SELECT round-trip latency
* ``select_systables_first_10`` — small server-side query
* ``select_bench_table_all`` — 1k-row sustained fetch
* ``executemany_1000_rows_in_txn`` — bulk INSERT throughput
* ``cold_connect_disconnect`` — login handshake cost
Each workload runs N times; we report mean and stddev.
"""
from __future__ import annotations
import statistics
import sys
import time
from collections.abc import Callable
import IfxPy
# Connect string — mirrors the conftest.py defaults the host uses.
CONN_STR = (
"SERVER=informix;"
"DATABASE=sysmaster;"
"HOST=127.0.0.1;"
"SERVICE=9088;"
"UID=informix;"
"PWD=in4mix;"
"PROTOCOL=onsoctcp"
)
ROUNDS_FAST = 100 # for sub-millisecond ops
ROUNDS_MED = 20 # for 1-100ms ops
ROUNDS_SLOW = 10 # for >1s ops; bumped from 3 in Tier 1 — the smaller
# sample produced unreliable means (cold-connect's stddev was 4.98 ms
# across 3 rounds; with 10 rounds the median is stable run-to-run).
def measure(name: str, rounds: int, body: Callable[[], None]) -> dict:
"""Run ``body`` ``rounds`` times; return median + IQR in seconds.
Median is more robust than mean against single-round outliers (GC
pauses, server scheduler hiccups). IQR (interquartile range) is
a noise estimator that also resists outliers — much better than
stddev when one bad round can dominate.
"""
timings: list[float] = []
for _ in range(rounds):
t0 = time.perf_counter()
body()
t1 = time.perf_counter()
timings.append(t1 - t0)
timings.sort()
median_s = timings[len(timings) // 2]
q1 = timings[len(timings) // 4]
q3 = timings[(3 * len(timings)) // 4]
return {
"name": name,
"rounds": rounds,
"median_s": median_s,
"iqr_s": q3 - q1,
"min_s": timings[0],
"max_s": timings[-1],
"mean_s": statistics.mean(timings), # kept for cross-checking
"stddev_s": statistics.stdev(timings) if len(timings) > 1 else 0.0,
}
def bench_select_one_row(conn) -> dict:
def run() -> None:
stmt = IfxPy.exec_immediate(
conn, "SELECT 1 FROM systables WHERE tabid = 1"
)
IfxPy.fetch_tuple(stmt)
IfxPy.free_stmt(stmt)
return measure("select_one_row", ROUNDS_FAST, run)
def bench_select_systables_first_10(conn) -> dict:
def run() -> None:
stmt = IfxPy.exec_immediate(
conn,
"SELECT FIRST 10 tabname, owner, tabid, ncols FROM systables",
)
while IfxPy.fetch_tuple(stmt):
pass
IfxPy.free_stmt(stmt)
return measure("select_systables_first_10", ROUNDS_FAST, run)
def bench_select_bench_table_all(conn) -> dict:
"""Requires p21_bench table to exist (created by host-side fixture)."""
# Probe whether the table exists; if not, skip
try:
stmt = IfxPy.exec_immediate(conn, "SELECT COUNT(*) FROM p21_bench")
row = IfxPy.fetch_tuple(stmt)
IfxPy.free_stmt(stmt)
if not row or row[0] == 0:
return {"name": "select_bench_table_all", "skipped": "p21_bench empty"}
except Exception as e:
return {"name": "select_bench_table_all", "skipped": f"p21_bench missing: {e}"}
def run() -> None:
stmt = IfxPy.exec_immediate(conn, "SELECT * FROM p21_bench")
while IfxPy.fetch_tuple(stmt):
pass
IfxPy.free_stmt(stmt)
return measure("select_bench_table_all", ROUNDS_MED, run)
def bench_executemany_1000_rows_in_txn() -> dict:
"""Open a connection on testdb, autocommit OFF, executemany 1000."""
try:
conn = IfxPy.connect(
CONN_STR.replace("DATABASE=sysmaster", "DATABASE=testdb"), "", ""
)
except Exception as e:
return {"name": "executemany_1000_rows_in_txn", "skipped": f"testdb: {e}"}
IfxPy.autocommit(conn, IfxPy.SQL_AUTOCOMMIT_OFF)
table = "p21_ifxpy_bench"
try:
try:
IfxPy.exec_immediate(conn, f"DROP TABLE {table}")
IfxPy.commit(conn)
except Exception:
pass
IfxPy.exec_immediate(
conn, f"CREATE TABLE {table} (id INT, name VARCHAR(64), value FLOAT)"
)
IfxPy.commit(conn)
counter = [0]
def run() -> None:
counter[0] += 1
base = counter[0] * 1000
stmt = IfxPy.prepare(
conn, f"INSERT INTO {table} VALUES (?, ?, ?)"
)
for i in range(1000):
IfxPy.execute(stmt, (base + i, f"row_{base + i}", float(base + i)))
IfxPy.free_stmt(stmt)
IfxPy.commit(conn)
result = measure("executemany_1000_rows_in_txn", ROUNDS_SLOW, run)
return result
finally:
try:
IfxPy.exec_immediate(conn, f"DROP TABLE {table}")
IfxPy.commit(conn)
except Exception:
pass
IfxPy.close(conn)
def bench_cold_connect_disconnect() -> dict:
def run() -> None:
conn = IfxPy.connect(CONN_STR, "", "")
IfxPy.close(conn)
return measure("cold_connect_disconnect", ROUNDS_SLOW, run)
# ----------------------------------------------------------------------------
# Phase 36 — scaling benchmarks (matched to test_scaling_perf.py)
# ----------------------------------------------------------------------------
def bench_executemany_scaling(n_rows: int) -> dict:
"""N-row insert in a single transaction. IfxPy doesn't pipeline —
each ``IfxPy.execute(stmt, params)`` is a synchronous round-trip
to the server. So per-row cost is roughly constant in N."""
rounds_for = {1_000: 10, 10_000: 5, 100_000: 3}
name = f"executemany_scaling_{n_rows}"
try:
conn = IfxPy.connect(
CONN_STR.replace("DATABASE=sysmaster", "DATABASE=testdb"), "", ""
)
except Exception as e:
return {"name": name, "skipped": f"testdb: {e}"}
IfxPy.autocommit(conn, IfxPy.SQL_AUTOCOMMIT_OFF)
table = f"p36_em_{n_rows}"
try:
try:
IfxPy.exec_immediate(conn, f"DROP TABLE {table}")
IfxPy.commit(conn)
except Exception:
pass
IfxPy.exec_immediate(
conn, f"CREATE TABLE {table} (id INT, name VARCHAR(64), value FLOAT)"
)
IfxPy.commit(conn)
counter = [0]
def run() -> None:
counter[0] += 1
base = counter[0] * n_rows
stmt = IfxPy.prepare(
conn, f"INSERT INTO {table} VALUES (?, ?, ?)"
)
for i in range(n_rows):
IfxPy.execute(stmt, (base + i, f"row_{base + i}", float(base + i)))
IfxPy.free_stmt(stmt)
IfxPy.commit(conn)
return measure(name, rounds_for[n_rows], run)
finally:
try:
IfxPy.exec_immediate(conn, f"DROP TABLE {table}")
IfxPy.commit(conn)
except Exception:
pass
IfxPy.close(conn)
def bench_select_scaling(n_rows: int) -> dict:
"""SELECT FIRST N from the pre-populated 100k-row p34_select table.
Tests IfxPy's per-row fetch cost at scale; should be roughly linear
in N like ours."""
rounds_for = {1_000: 10, 10_000: 5, 100_000: 3}
name = f"select_scaling_{n_rows}"
try:
conn = IfxPy.connect(
CONN_STR.replace("DATABASE=sysmaster", "DATABASE=testdb"), "", ""
)
except Exception as e:
return {"name": name, "skipped": f"testdb: {e}"}
try:
# Probe: does p34_select exist?
try:
stmt = IfxPy.exec_immediate(conn, "SELECT COUNT(*) FROM p34_select")
row = IfxPy.fetch_tuple(stmt)
IfxPy.free_stmt(stmt)
available = int(row[0])
if available < n_rows:
return {"name": name, "skipped": (
f"p34_select has only {available} rows; "
"run informix-db scaling benchmarks first to seed "
"the table"
)}
except Exception as e:
return {"name": name, "skipped": f"p34_select missing: {e}"}
def run() -> None:
stmt = IfxPy.exec_immediate(
conn, f"SELECT FIRST {n_rows} * FROM p34_select"
)
count = 0
while IfxPy.fetch_tuple(stmt):
count += 1
IfxPy.free_stmt(stmt)
if count != n_rows:
raise RuntimeError(
f"expected {n_rows} rows, got {count}"
)
return measure(name, rounds_for[n_rows], run)
finally:
IfxPy.close(conn)
def main() -> None:
print("# IfxPy benchmark results", file=sys.stderr)
print(f"# IfxPy version: {IfxPy.__version__ if hasattr(IfxPy, '__version__') else 'unknown'}", file=sys.stderr)
# Persistent connection for the read-mostly benchmarks
conn = IfxPy.connect(CONN_STR, "", "")
results = []
results.append(bench_select_one_row(conn))
results.append(bench_select_systables_first_10(conn))
results.append(bench_select_bench_table_all(conn))
IfxPy.close(conn)
results.append(bench_executemany_1000_rows_in_txn())
results.append(bench_cold_connect_disconnect())
# Phase 36 — scaling comparison. Skip 100k cases when --short is
# passed (e.g., for fast smoke runs); otherwise run all sizes.
short = "--short" in sys.argv
sizes = [1_000, 10_000] if short else [1_000, 10_000, 100_000]
for n in sizes:
results.append(bench_executemany_scaling(n))
for n in sizes:
results.append(bench_select_scaling(n))
# Emit machine-parseable lines on stdout. Reporting median (not
# mean) and IQR (not stddev) so a single outlier round can't
# dominate the comparison numbers — mirrors pytest-benchmark's
# ``--benchmark-columns=median,iqr`` reporting on the host side.
for r in results:
if r.get("skipped"):
print(f"SKIP {r['name']}: {r['skipped']}")
else:
print(
f"RESULT {r['name']} median={r['median_s']:.6f}s "
f"iqr={r['iqr_s']:.6f}s min={r['min_s']:.6f}s "
f"max={r['max_s']:.6f}s mean={r['mean_s']:.6f}s "
f"stddev={r['stddev_s']:.6f}s rounds={r['rounds']}"
)
if __name__ == "__main__":
main()

80
tests/benchmarks/conftest.py Normal file
View File

@ -0,0 +1,80 @@
"""Benchmark fixtures — long-lived connections + populated test tables.
The end-to-end benchmark suite needs:
* A persistent connection (creating one per benchmark inflates the cost
by the login handshake, ~5-15ms — distorts micro-second measurements).
* A pre-populated test table so SELECT/UPDATE benchmarks have rows to
iterate.
Both fixtures are session-scoped so the table is created exactly once
even when the same benchmark is iterated over many rounds.
"""
from __future__ import annotations
import contextlib
from collections.abc import Iterator
import pytest
import informix_db
from tests.conftest import ConnParams
BENCH_TABLE_ROWS = 1000 # rows in the populated benchmark table
@pytest.fixture(scope="session")
def bench_conn(conn_params: ConnParams) -> Iterator[informix_db.Connection]:
"""One long-lived autocommit connection for the entire bench session."""
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
)
try:
yield conn
finally:
conn.close()
@pytest.fixture(scope="session")
def bench_table(bench_conn: informix_db.Connection) -> Iterator[str]:
"""Create + populate a 1k-row table for SELECT/UPDATE benchmarks.
Yields the table name. The table is dropped at session teardown.
Schema covers the common type mix: INT id, VARCHAR name,
INT (counter), FLOAT (value), DATE (created).
"""
table = "p21_bench"
cur = bench_conn.cursor()
with contextlib.suppress(informix_db.Error):
cur.execute(f"DROP TABLE {table}")
cur.execute(
f"CREATE TABLE {table} ("
" id INT, name VARCHAR(64), counter INT,"
" value FLOAT, created DATE)"
)
# Populate via executemany so setup is fast.
rows = [
(
i,
f"row_{i:04d}",
i * 7,
float(i) * 1.5,
None, # DATE NULL — keeps fixture small
)
for i in range(BENCH_TABLE_ROWS)
]
cur.executemany(
f"INSERT INTO {table} VALUES (?, ?, ?, ?, ?)",
rows,
)
try:
yield table
finally:
with contextlib.suppress(informix_db.Error):
cur.execute(f"DROP TABLE {table}")

108
tests/benchmarks/test_async_perf.py Normal file
View File

@ -0,0 +1,108 @@
"""Async-path benchmarks.
The async layer is a thin ``_to_thread`` shim over the sync codec, so
the per-call delta vs sync is the event-loop hop cost (~tens of µs).
The win is **concurrency**: running 10 SELECTs through a pool with
``asyncio.gather`` returns in roughly the same wall-clock time as 1.
These benchmarks measure both:
* ``test_async_select_one_row`` — single-call overhead delta vs sync
* ``test_async_concurrent_10_selects`` — concurrent throughput
"""
from __future__ import annotations
import asyncio
import pytest
from informix_db import aio
from tests.conftest import ConnParams
pytestmark = [pytest.mark.benchmark, pytest.mark.integration]
@pytest.fixture
def event_loop():
"""A fresh event loop per benchmark — pytest-asyncio compat shim."""
loop = asyncio.new_event_loop()
yield loop
loop.close()
def test_async_select_one_row(
benchmark, conn_params: ConnParams
) -> None:
"""Single async round-trip — measure thread-hop overhead."""
loop = asyncio.new_event_loop()
async def setup() -> aio.AsyncConnection:
return await aio.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
)
conn = loop.run_until_complete(setup())
async def one_query() -> object:
cur = await conn.cursor()
await cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
row = await cur.fetchone()
await cur.close()
return row
def run() -> object:
return loop.run_until_complete(one_query())
try:
benchmark(run)
finally:
loop.run_until_complete(conn.close())
loop.close()
def test_async_concurrent_10_selects(
benchmark, conn_params: ConnParams
) -> None:
"""10 concurrent SELECTs through a pool — sub-linear vs serial."""
loop = asyncio.new_event_loop()
async def setup() -> aio.AsyncConnectionPool:
return await aio.create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
min_size=2,
max_size=10,
)
pool = loop.run_until_complete(setup())
async def one_through_pool() -> object:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
row = await cur.fetchone()
await cur.close()
return row
async def ten_concurrent() -> list:
return await asyncio.gather(*(one_through_pool() for _ in range(10)))
def run() -> list:
return loop.run_until_complete(ten_concurrent())
try:
benchmark(run)
finally:
loop.run_until_complete(pool.close())
loop.close()

199
tests/benchmarks/test_codec_perf.py Normal file
View File

@ -0,0 +1,199 @@
"""Codec micro-benchmarks — no server required.
These measure the tight inner loops the driver hits on every row:
``decode()`` per cell, ``parse_tuple_payload()`` per row,
``encode_param()`` per parameter. A 1M-row fetch hits ``decode()``
5-10M times; a 1% slowdown there is *visible*.
The fixtures synthesize realistic byte payloads — no need for the
Docker container. This makes the benchmarks usable in CI's pre-merge
job (which doesn't run integration tests).
"""
from __future__ import annotations
import datetime
import struct
from io import BytesIO
import pytest
from informix_db._protocol import IfxStreamReader
from informix_db._resultset import ColumnInfo, parse_tuple_payload
from informix_db._types import IfxType
from informix_db.converters import decode, encode_param
pytestmark = pytest.mark.benchmark
# ---------------------------------------------------------------------------
# decode() — per-value dispatch
# ---------------------------------------------------------------------------
def test_decode_int(benchmark) -> None:
"""Hot path: per-cell INT decode. ~5M calls/sec is the kind of speed
a 1M-row fetch with 5 INT columns needs."""
raw = struct.pack("!i", 42)
benchmark(decode, int(IfxType.INT), raw)
def test_decode_smallint(benchmark) -> None:
raw = struct.pack("!h", 100)
benchmark(decode, int(IfxType.SMALLINT), raw)
def test_decode_bigint(benchmark) -> None:
raw = struct.pack("!q", 1234567890123)
benchmark(decode, int(IfxType.BIGINT), raw)
def test_decode_float(benchmark) -> None:
raw = struct.pack("!d", 3.14159)
benchmark(decode, int(IfxType.FLOAT), raw)
def test_decode_date(benchmark) -> None:
raw = struct.pack("!i", 45678)
benchmark(decode, int(IfxType.DATE), raw)
def test_decode_varchar_short(benchmark) -> None:
"""20-byte ASCII string — typical name column."""
raw = b"hello world example "
benchmark(decode, int(IfxType.VARCHAR), raw)
def test_decode_varchar_long(benchmark) -> None:
"""255-byte VARCHAR — max non-LVARCHAR length."""
raw = b"x" * 255
benchmark(decode, int(IfxType.VARCHAR), raw)
def test_decode_varchar_utf8(benchmark) -> None:
"""Multi-byte UTF-8 decode — exercise Phase 20 path."""
raw = "café résumé naïve Zürich".encode()
benchmark(decode, int(IfxType.VARCHAR), raw, "utf-8")
# ---------------------------------------------------------------------------
# encode_param() — parameter-binding hot path
# ---------------------------------------------------------------------------
def test_encode_int(benchmark) -> None:
benchmark(encode_param, 42)
def test_encode_str_ascii(benchmark) -> None:
benchmark(encode_param, "hello world example", "iso-8859-1")
def test_encode_str_utf8(benchmark) -> None:
benchmark(encode_param, "café résumé naïve", "utf-8")
def test_encode_float(benchmark) -> None:
benchmark(encode_param, 3.14159)
def test_encode_date(benchmark) -> None:
benchmark(encode_param, datetime.date(2026, 5, 4))
def test_encode_datetime(benchmark) -> None:
benchmark(encode_param, datetime.datetime(2026, 5, 4, 12, 30, 45))
# ---------------------------------------------------------------------------
# parse_tuple_payload() — per-row decode
# ---------------------------------------------------------------------------
def _build_systables_row_payload() -> bytes:
"""Synthesize the SQ_TUPLE bytes a typical systables row produces.
Layout: [short warn=0][int size][payload][optional pad]
Payload has columns: tabname VARCHAR(128), owner VARCHAR(32),
tabid INT, partnum INT, ncols INT.
"""
payload = bytearray()
# tabname VARCHAR: [byte len][bytes] — single-byte length prefix per
# the discovered tuple format
name = b"systables"
payload.append(len(name))
payload.extend(name)
# owner VARCHAR
owner = b"informix"
payload.append(len(owner))
payload.extend(owner)
# tabid INT
payload.extend(struct.pack("!i", 1))
# partnum INT
payload.extend(struct.pack("!i", 1048578))
# ncols INT
payload.extend(struct.pack("!i", 32))
out = bytearray()
out.extend(struct.pack("!h", 0)) # warn
out.extend(struct.pack("!i", len(payload)))
out.extend(payload)
if len(payload) & 1:
out.append(0) # even-byte pad
return bytes(out)
_SYSTABLES_COLUMNS = [
ColumnInfo(
name="tabname",
type_code=int(IfxType.VARCHAR),
raw_type_code=int(IfxType.VARCHAR),
encoded_length=128,
),
ColumnInfo(
name="owner",
type_code=int(IfxType.VARCHAR),
raw_type_code=int(IfxType.VARCHAR),
encoded_length=32,
),
ColumnInfo(
name="tabid",
type_code=int(IfxType.INT),
raw_type_code=int(IfxType.INT),
encoded_length=4,
),
ColumnInfo(
name="partnum",
type_code=int(IfxType.INT),
raw_type_code=int(IfxType.INT),
encoded_length=4,
),
ColumnInfo(
name="ncols",
type_code=int(IfxType.INT),
raw_type_code=int(IfxType.INT),
encoded_length=4,
),
]
def test_parse_tuple_5cols_iso8859(benchmark) -> None:
"""Decode a 5-column row (2 VARCHAR + 3 INT) — typical `systables` shape."""
payload = _build_systables_row_payload()
def run() -> tuple:
reader = IfxStreamReader(BytesIO(payload))
return parse_tuple_payload(reader, _SYSTABLES_COLUMNS)
benchmark(run)
def test_parse_tuple_5cols_utf8(benchmark) -> None:
"""Same shape, UTF-8 codec path — verify Phase 20 isn't a bottleneck."""
payload = _build_systables_row_payload()
def run() -> tuple:
reader = IfxStreamReader(BytesIO(payload))
return parse_tuple_payload(reader, _SYSTABLES_COLUMNS, encoding="utf-8")
benchmark(run)

170
tests/benchmarks/test_insert_perf.py Normal file
View File

@ -0,0 +1,170 @@
"""End-to-end INSERT benchmarks — single-row, executemany, and the gap.
The single-row vs. executemany delta is the ``executemany`` win — we
PREPARE+RELEASE once and BIND+EXECUTE per row, vs PREPARE+RELEASE per
row. On any decent network this is 10-50x.
The autocommit-True vs. autocommit-False delta is the **transaction-flush
cost** — every autocommit INSERT forces the server to flush its
transaction log per row, drowning out everything else. The benchmark
splits these so we can see protocol overhead independently.
"""
from __future__ import annotations
import contextlib
from collections.abc import Iterator
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = [pytest.mark.benchmark, pytest.mark.integration]
@pytest.fixture(scope="module")
def txn_conn(conn_params: ConnParams) -> Iterator[informix_db.Connection]:
"""A separate connection with autocommit=False so we can wrap an
executemany call in a single explicit transaction. Uses ``testdb``
(the logged user DB) — autocommit-off is meaningless on unlogged DBs.
"""
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database="testdb",
server=conn_params.server,
autocommit=False,
)
try:
yield conn
finally:
conn.close()
def _setup_temp_table(conn: informix_db.Connection, name: str) -> None:
cur = conn.cursor()
with contextlib.suppress(informix_db.Error):
cur.execute(f"DROP TABLE {name}")
cur.execute(
f"CREATE TABLE {name} (id INT, name VARCHAR(64), value FLOAT)"
)
def _drop_temp_table(conn: informix_db.Connection, name: str) -> None:
cur = conn.cursor()
with contextlib.suppress(informix_db.Error):
cur.execute(f"DROP TABLE {name}")
def test_insert_single_row(benchmark, bench_conn: informix_db.Connection) -> None:
"""Single INSERT per call — full PREPARE+BIND+EXECUTE+RELEASE cycle."""
table = "p21_ins_single"
_setup_temp_table(bench_conn, table)
counter = [0]
def run() -> None:
counter[0] += 1
cur = bench_conn.cursor()
cur.execute(
f"INSERT INTO {table} VALUES (?, ?, ?)",
(counter[0], f"name_{counter[0]}", float(counter[0])),
)
cur.close()
try:
benchmark(run)
finally:
_drop_temp_table(bench_conn, table)
def test_executemany_100_rows(
benchmark, bench_conn: informix_db.Connection
) -> None:
"""100 INSERTs via executemany — one PREPARE, 100 BIND+EXECUTEs, one RELEASE."""
table = "p21_ins_emany_100"
_setup_temp_table(bench_conn, table)
counter = [0]
def run() -> None:
counter[0] += 1
base = counter[0] * 100
rows = [
(base + i, f"row_{base + i}", float(base + i)) for i in range(100)
]
cur = bench_conn.cursor()
cur.executemany(
f"INSERT INTO {table} VALUES (?, ?, ?)",
rows,
)
cur.close()
try:
benchmark(run)
finally:
_drop_temp_table(bench_conn, table)
def test_executemany_1000_rows(
benchmark, bench_conn: informix_db.Connection
) -> None:
"""1000 INSERTs via executemany under autocommit=True — every row
forces a transaction-log flush. Worst-case protocol *plus* server
storage cost."""
table = "p21_ins_emany_1000"
_setup_temp_table(bench_conn, table)
counter = [0]
def run() -> None:
counter[0] += 1
base = counter[0] * 1000
rows = [
(base + i, f"row_{base + i}", float(base + i)) for i in range(1000)
]
cur = bench_conn.cursor()
cur.executemany(
f"INSERT INTO {table} VALUES (?, ?, ?)",
rows,
)
cur.close()
try:
benchmark.pedantic(run, rounds=10, iterations=1)
finally:
_drop_temp_table(bench_conn, table)
def test_executemany_1000_rows_in_txn(
benchmark, txn_conn: informix_db.Connection
) -> None:
"""1000 INSERTs via executemany inside ONE transaction — single
log flush at COMMIT time. Isolates the protocol cost from the
autocommit-flush cost. The delta vs the autocommit variant is the
server-side log-flush penalty (un-fixable from the client side)."""
table = "p21_ins_emany_txn"
_setup_temp_table(txn_conn, table)
txn_conn.commit() # Land the CREATE TABLE before timing
counter = [0]
def run() -> None:
counter[0] += 1
base = counter[0] * 1000
rows = [
(base + i, f"row_{base + i}", float(base + i)) for i in range(1000)
]
cur = txn_conn.cursor()
cur.executemany(
f"INSERT INTO {table} VALUES (?, ?, ?)",
rows,
)
cur.close()
txn_conn.commit()
try:
benchmark.pedantic(run, rounds=10, iterations=1)
finally:
with contextlib.suppress(informix_db.Error):
_drop_temp_table(txn_conn, table)
txn_conn.commit()

218
tests/benchmarks/test_observability_perf.py Normal file
View File

@ -0,0 +1,218 @@
"""Tier 2 benchmarks — observability and concurrency.
The existing benchmark suite measures *single-call latency* under low
contention. This file adds three benchmark categories that verify
claims the driver makes elsewhere but doesn't currently prove:
1. **Memory-growth during streaming fetch** — USAGE.md claims iterator-
based fetch keeps memory flat. We verify by sampling RSS every 1000
rows during a 100k-row iteration; the slope must be near-zero.
2. **Latency percentiles** — most benchmarks report mean/median, but
for SLO-bound applications p95/p99/max matter more. We hammer
``SELECT 1`` with 1000 round-trips and report the full distribution.
3. **Concurrent pool throughput** — the pool is supposed to give
per-request fairness and aggregate throughput scaling. We verify
with N=2/4/8 worker threads each running 100 queries.
These benchmarks have a different shape than ``test_*_perf.py``: instead
of timing one operation per round and reporting median, they run the
entire workload once per round and assert the *shape* of the result.
The output goes to stdout for inspection rather than into the
pytest-benchmark JSON archive (which doesn't model multi-dimensional
results well).
"""
from __future__ import annotations
import gc
import statistics
import threading
import time
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = [pytest.mark.benchmark, pytest.mark.integration]
def test_streaming_fetch_memory_profile(
bench_conn: informix_db.Connection, bench_table: str
) -> None:
"""Document the memory profile of iterator-based fetch.
The current cursor materializes the full result set on
``execute()`` (Phase 17 in-memory model), so memory IS expected
to grow proportional to row count. This test:
1. Records the actual growth shape so it's visible in CI output.
2. Provides a regression baseline — if growth ever exceeds 100 MB
for a 1k-row table, something is leaking.
3. Is the future regression test for a streaming/server-cursor
mode that maintains constant memory.
"""
import resource
cur = bench_conn.cursor()
cur.execute(f"SELECT * FROM {bench_table}")
def rss_kb() -> int:
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
samples: list[tuple[int, int]] = []
rows_seen = 0
initial_rss = rss_kb()
samples.append((0, initial_rss))
for _ in cur:
rows_seen += 1
if rows_seen % 100 == 0:
samples.append((rows_seen, rss_kb()))
cur.close()
gc.collect()
final_rss = rss_kb()
print(f"\nstreaming_fetch memory profile ({rows_seen} rows from {bench_table}):")
for rows, rss in samples[::2]: # every other sample to keep output short
print(f" rows={rows:>5} rss={rss:>8} KB (Δ={rss - initial_rss:+} KB)")
print(f" final={final_rss} KB after gc.collect()")
# Regression wall: 100 MB growth for 1k rows would mean we're
# leaking ~100 KB/row. Realistic in-memory cost is ~500 bytes/row,
# so growth should be well under 1 MB.
growth_kb = final_rss - initial_rss
assert growth_kb < 100_000, (
f"streaming fetch grew RSS by {growth_kb} KB for {rows_seen} rows "
f"— cursor is leaking or holding references it shouldn't"
)
def test_select_1_latency_percentiles(
bench_conn: informix_db.Connection,
) -> None:
"""Run ``SELECT 1`` 1000 times; report p50/p90/p95/p99/max.
Mean alone is misleading for latency-sensitive applications —
the tail (p95/p99) is what actually breaks SLOs. A 200 us mean
with a 5 ms p99 means 1% of requests are 25x slower than typical.
No assertions: the test exists to surface the distribution shape
so a regression that worsens the tail without moving the mean
becomes visible to a human reviewer.
"""
timings: list[float] = []
# Warmup
for _ in range(20):
cur = bench_conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
cur.fetchone()
cur.close()
# Measure
for _ in range(1000):
t0 = time.perf_counter()
cur = bench_conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
cur.fetchone()
cur.close()
timings.append(time.perf_counter() - t0)
timings.sort()
def at(p: float) -> float:
return timings[int(p * len(timings))]
p50 = at(0.50)
p90 = at(0.90)
p95 = at(0.95)
p99 = at(0.99)
p_max = timings[-1]
print("\nSELECT 1 latency distribution (n=1000):")
print(f" p50 = {p50 * 1e6:>8.1f} µs")
print(f" p90 = {p90 * 1e6:>8.1f} µs")
print(f" p95 = {p95 * 1e6:>8.1f} µs")
print(f" p99 = {p99 * 1e6:>8.1f} µs")
print(f" max = {p_max * 1e6:>8.1f} µs")
print(f" mean = {statistics.mean(timings) * 1e6:>8.1f} µs")
print(f" ratio p99/p50 = {p99 / p50:.2f}x")
# Sanity check: p50 should be sub-millisecond on loopback. If
# this fails, something is wrong with the test environment, not
# the driver.
assert p50 < 0.001, f"p50 latency {p50 * 1e6:.0f} µs > 1 ms — env issue"
@pytest.mark.parametrize("n_threads", [2, 4, 8])
def test_concurrent_pool_throughput(
conn_params: ConnParams, n_threads: int
) -> None:
"""N worker threads each run M queries through a shared pool.
Reports aggregate queries/sec and per-thread mean latency. Verifies
the pool actually parallelizes work (aggregate throughput should
scale roughly linearly with N up to the wire's saturation point).
"""
QUERIES_PER_WORKER = 100
pool = informix_db.create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
min_size=n_threads,
max_size=n_threads,
)
try:
per_thread_timings: dict[int, list[float]] = {i: [] for i in range(n_threads)}
barrier = threading.Barrier(n_threads + 1) # +1 for main thread
def worker(tid: int) -> None:
barrier.wait() # synchronize start
for _ in range(QUERIES_PER_WORKER):
t0 = time.perf_counter()
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
cur.fetchone()
cur.close()
per_thread_timings[tid].append(time.perf_counter() - t0)
threads = [
threading.Thread(target=worker, args=(i,)) for i in range(n_threads)
]
for t in threads:
t.start()
barrier.wait() # release all workers simultaneously
wall_start = time.perf_counter()
for t in threads:
t.join(timeout=60.0)
assert not t.is_alive()
wall_total = time.perf_counter() - wall_start
total_queries = n_threads * QUERIES_PER_WORKER
agg_qps = total_queries / wall_total
all_timings = [t for ts in per_thread_timings.values() for t in ts]
all_timings.sort()
median_per_call = all_timings[len(all_timings) // 2]
# Per-thread fairness check: each thread's count should equal
# QUERIES_PER_WORKER (all completed)
for tid, ts in per_thread_timings.items():
assert len(ts) == QUERIES_PER_WORKER, (
f"thread {tid} only completed {len(ts)}/{QUERIES_PER_WORKER}"
)
print(f"\nconcurrent pool throughput (N={n_threads} threads):")
print(f" total queries = {total_queries}")
print(f" wall time = {wall_total * 1000:.1f} ms")
print(f" aggregate QPS = {agg_qps:.1f}")
print(f" median per-call = {median_per_call * 1e6:.1f} µs")
print(f" per-thread fairness: all {n_threads} completed all "
f"{QUERIES_PER_WORKER} queries")
finally:
pool.close()

85
tests/benchmarks/test_pool_perf.py Normal file
View File

@ -0,0 +1,85 @@
"""Connection-pool benchmarks — measure the cost of pool acquire/release
vs. fresh connect.
The win on the pool side is *avoiding the login handshake*. Cold connect
to Informix is ~5-15ms (server-side auth + protocol negotiation). Pool
acquire is ~50-200µs (validation only). The benchmark makes that delta
visible.
"""
from __future__ import annotations
import pytest
import informix_db
from informix_db.pool import ConnectionPool, create_pool
from tests.conftest import ConnParams
pytestmark = [pytest.mark.benchmark, pytest.mark.integration]
@pytest.fixture(scope="module")
def pool(conn_params: ConnParams):
"""Module-scoped pool kept warm across the bench file."""
p = create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
min_size=2,
max_size=10,
)
try:
yield p
finally:
p.close()
def test_cold_connect_disconnect(benchmark, conn_params: ConnParams) -> None:
"""Full login handshake + close per call — the worst case."""
def run() -> None:
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
)
conn.close()
# Cold-connect is slow (~10ms) and noisy run-to-run (server scheduling,
# network buffers). 15 rounds is enough to make the median stable
# without bloating the bench suite's runtime past ~3 minutes.
benchmark.pedantic(run, rounds=15, iterations=1)
def test_pool_acquire_release(benchmark, pool: ConnectionPool) -> None:
"""Pool acquire+release — the steady-state cost of a pooled query."""
def run() -> None:
with pool.connection() as _conn:
pass
benchmark(run)
def test_pool_acquire_query_release(
benchmark, pool: ConnectionPool
) -> None:
"""Realistic per-query cost: acquire, run a tiny query, release."""
def run() -> object:
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
row = cur.fetchone()
cur.close()
return row
benchmark(run)

461
tests/benchmarks/test_scaling_perf.py Normal file
View File

@ -0,0 +1,461 @@
"""Phase 34 — scaling benchmarks.
The existing benchmarks measure single-shape workloads (1k-row SELECT,
1k-row executemany). These add the scaling axes:
1. **executemany at 1k / 10k / 100k rows** in a transaction. Phase 33's
pipelining eliminates per-row RTT; this test confirms the speedup
scales linearly with N.
2. **SELECT at 1k / 10k / 100k rows**. Tests parse_tuple_payload
throughput at real-world scale. Could surface codec slowdown,
memory issues, or GC-pause amplification.
3. **Wide-row SELECT** (5 / 20 / 50 columns x 1k rows). More columns =
more decode calls per row. Different cost shape than row-count
scaling.
4. **Type-mix SELECT**: realistic application workload with INT +
VARCHAR + DECIMAL + DATE + FLOAT in one query. Tests the codec
dispatch under a representative mix of decoders.
Each benchmark is parametrized; pytest-benchmark groups them so we
get one row per scale point.
"""
from __future__ import annotations
import contextlib
import os as _os
from collections.abc import Iterator
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = [pytest.mark.benchmark, pytest.mark.integration]
# Module-level scaling sizes. The 1M row sizes are guarded by an
# environment flag (IFX_BENCH_1M=1) so the default `make bench` run
# stays under 5 minutes — 1M-row workloads add ~30s + the overhead
# of seeding a 1M-row table.
_BIG = _os.environ.get("IFX_BENCH_1M") == "1"
EXECUTEMANY_SIZES = [1_000, 10_000, 100_000]
SELECT_SIZES = [1_000, 10_000, 100_000]
if _BIG:
EXECUTEMANY_SIZES = [*EXECUTEMANY_SIZES, 1_000_000]
SELECT_SIZES = [*SELECT_SIZES, 1_000_000]
WIDTH_COLUMNS = [5, 20, 50, 100] # added 100-column case for codec stress
@pytest.fixture(scope="module")
def txn_conn(conn_params: ConnParams) -> Iterator[informix_db.Connection]:
"""Logged-DB connection with autocommit=False for in-transaction
bulk-insert benchmarks."""
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database="testdb",
server=conn_params.server,
autocommit=False,
)
try:
yield conn
finally:
conn.close()
# ----------------------------------------------------------------------------
# Bulk-insert scaling
# ----------------------------------------------------------------------------
@pytest.mark.parametrize("n_rows", EXECUTEMANY_SIZES)
def test_executemany_scaling(
benchmark, txn_conn: informix_db.Connection, n_rows: int
) -> None:
"""executemany(N) in a single transaction. Pipelined Phase 33 design
sends all N PDUs then drains all N responses — should scale roughly
linearly with N at very low per-row cost.
Round counts shrink as N grows so each scale point completes in
similar wall time:
1k rows → 10 rounds (~110 ms each = 1.1 s)
10k rows → 5 rounds (~1.1 s each = 5.5 s)
100k rows → 3 rounds (~11 s each = 33 s)
"""
rounds_for = {1_000: 10, 10_000: 5, 100_000: 3, 1_000_000: 2}
table = f"p34_em_{n_rows}"
cur = txn_conn.cursor()
with contextlib.suppress(informix_db.Error):
cur.execute(f"DROP TABLE {table}")
cur.execute(f"CREATE TABLE {table} (id INT, name VARCHAR(64), value FLOAT)")
txn_conn.commit()
counter = [0]
def run() -> None:
counter[0] += 1
base = counter[0] * n_rows
rows = [(base + i, f"row_{base + i}", float(base + i)) for i in range(n_rows)]
cur = txn_conn.cursor()
cur.executemany(f"INSERT INTO {table} VALUES (?, ?, ?)", rows)
cur.close()
txn_conn.commit()
try:
benchmark.pedantic(run, rounds=rounds_for[n_rows], iterations=1)
finally:
with contextlib.suppress(informix_db.Error):
cur = txn_conn.cursor()
cur.execute(f"DROP TABLE {table}")
txn_conn.commit()
# ----------------------------------------------------------------------------
# SELECT-scaling
# ----------------------------------------------------------------------------
@pytest.fixture(scope="module")
def scaling_select_table(conn_params: ConnParams) -> Iterator[str]:
"""Pre-populated 100k-row table for SELECT scaling. Built once per
module run; benchmarks select FIRST N rows.
Uses its OWN connection (not the shared txn_conn) so its
transaction state can't be polluted by other tests' executemany
work. Earlier attempts to share txn_conn produced silent
population failures (200 rows instead of 100k) likely from
cursor-state leakage across pipelined batches in the same
transaction.
"""
table = "p34_select"
setup_conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database="testdb",
server=conn_params.server,
autocommit=False,
)
cur = setup_conn.cursor()
with contextlib.suppress(informix_db.Error):
cur.execute(f"DROP TABLE {table}")
setup_conn.commit()
cur.execute(
f"CREATE TABLE {table} ("
f" id INT, name VARCHAR(64), counter INT,"
f" value FLOAT, label VARCHAR(32))"
)
setup_conn.commit()
# Population size scales with whether the 1M tests are enabled.
target = 1_000_000 if _BIG else 100_000
chunk = 10_000
for base in range(0, target, chunk):
rows = [
(base + i, f"name_{base + i:06d}", (base + i) * 7,
float(base + i) * 1.5, f"L{(base + i) % 100:02d}")
for i in range(chunk)
]
cur.executemany(
f"INSERT INTO {table} VALUES (?, ?, ?, ?, ?)", rows
)
setup_conn.commit()
# Verify population — fail loud if the multi-chunk insert dropped rows.
cur.execute(f"SELECT COUNT(*) FROM {table}")
(count,) = cur.fetchone()
assert count == target, (
f"fixture failed: {table} has {count} rows, expected {target}"
)
try:
yield table
finally:
with contextlib.suppress(informix_db.Error):
cur = setup_conn.cursor()
cur.execute(f"DROP TABLE {table}")
setup_conn.commit()
setup_conn.close()
@pytest.fixture(scope="module")
def select_read_conn(
conn_params: ConnParams,
) -> Iterator[informix_db.Connection]:
"""Dedicated read connection for SELECT scaling tests.
Sharing ``txn_conn`` across read and write tests caused a
transaction-isolation bug: ``txn_conn`` would have an open
read-snapshot from before the fixture's writes committed,
so SELECTs through it only saw 200 rows instead of 100k.
A separate read-side connection that's never been in a
transaction sees the committed state correctly.
"""
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database="testdb",
server=conn_params.server,
autocommit=True, # read-only — no transaction state to worry about
)
try:
yield conn
finally:
conn.close()
@pytest.mark.parametrize("n_rows", SELECT_SIZES)
def test_select_scaling(
benchmark,
select_read_conn: informix_db.Connection,
scaling_select_table: str,
n_rows: int,
) -> None:
"""SELECT FIRST N from a pre-populated 100k-row table. Tests
parse_tuple_payload throughput at production scale.
Per-row cost should stay roughly constant across N — if the per-row
median grows with N, something's wrong (memory pressure, GC,
codec degradation).
"""
rounds_for = {1_000: 10, 10_000: 5, 100_000: 3, 1_000_000: 2}
cur = select_read_conn.cursor()
cur.execute(f"SELECT COUNT(*) FROM {scaling_select_table}")
(count,) = cur.fetchone()
cur.close()
assert count >= n_rows, (
f"{scaling_select_table} has only {count} rows; "
f"can't benchmark SELECT FIRST {n_rows}"
)
def run() -> int:
cur = select_read_conn.cursor()
cur.execute(f"SELECT FIRST {n_rows} * FROM {scaling_select_table}")
rows = cur.fetchall()
cur.close()
assert len(rows) == n_rows, (
f"SELECT FIRST {n_rows} returned {len(rows)} rows"
)
return len(rows)
benchmark.pedantic(run, rounds=rounds_for[n_rows], iterations=1)
# ----------------------------------------------------------------------------
# Wide-row scaling
# ----------------------------------------------------------------------------
@pytest.mark.parametrize("n_cols", WIDTH_COLUMNS)
def test_wide_row_select(
benchmark, txn_conn: informix_db.Connection, n_cols: int
) -> None:
"""SELECT 1000 rows of width N columns. Tests the codec dispatch
under different per-row column-count loads.
parse_tuple_payload runs its dispatch loop N x 1000 times; doubling
the column count should roughly double the per-row decode cost.
"""
table = f"p34_wide_{n_cols}"
cur = txn_conn.cursor()
with contextlib.suppress(informix_db.Error):
cur.execute(f"DROP TABLE {table}")
# Mix of types: id (int), col0..N-2 (int)
col_defs = ", ".join([f"c{i} INT" for i in range(n_cols)])
cur.execute(f"CREATE TABLE {table} ({col_defs})")
txn_conn.commit()
rows = [tuple(j * 7 + i for j in range(n_cols)) for i in range(1000)]
placeholders = ", ".join(["?"] * n_cols)
cur.executemany(
f"INSERT INTO {table} VALUES ({placeholders})", rows
)
txn_conn.commit()
def run() -> int:
cur = txn_conn.cursor()
cur.execute(f"SELECT * FROM {table}")
rows = cur.fetchall()
cur.close()
return len(rows)
try:
benchmark.pedantic(run, rounds=10, iterations=1)
finally:
with contextlib.suppress(informix_db.Error):
cur = txn_conn.cursor()
cur.execute(f"DROP TABLE {table}")
txn_conn.commit()
# ----------------------------------------------------------------------------
# Type-mix workload — realistic application shape
# ----------------------------------------------------------------------------
@pytest.fixture(scope="module")
def type_mix_table(txn_conn: informix_db.Connection) -> Iterator[str]:
"""1000 rows mixing INT + VARCHAR + DECIMAL + DATE + FLOAT —
representative of a typical business-data row shape."""
import datetime
import decimal
table = "p34_typemix"
cur = txn_conn.cursor()
with contextlib.suppress(informix_db.Error):
cur.execute(f"DROP TABLE {table}")
cur.execute(
f"CREATE TABLE {table} ("
f" id INT, name VARCHAR(64),"
f" amount DECIMAL(12,2), event_date DATE, ratio FLOAT,"
f" tag SMALLINT)"
)
txn_conn.commit()
base_date = datetime.date(2024, 1, 1)
rows = [
(
i,
f"event_{i:05d}",
decimal.Decimal(f"{i * 1.5:.2f}"),
base_date + datetime.timedelta(days=i % 365),
float(i) * 0.001,
i % 100,
)
for i in range(1000)
]
cur.executemany(
f"INSERT INTO {table} VALUES (?, ?, ?, ?, ?, ?)", rows
)
txn_conn.commit()
try:
yield table
finally:
with contextlib.suppress(informix_db.Error):
cur = txn_conn.cursor()
cur.execute(f"DROP TABLE {table}")
txn_conn.commit()
def test_select_type_mix_1000_rows(
benchmark,
txn_conn: informix_db.Connection,
type_mix_table: str,
) -> None:
"""1000-row SELECT with INT/VARCHAR/DECIMAL/DATE/FLOAT/SMALLINT
columns — exercises 6 different decoders per row.
Compared to test_select_bench_table_all (which is mostly INT +
VARCHAR), this exercises the full decoder dispatch including the
DECIMAL BCD parser and DATE epoch math.
"""
def run() -> int:
cur = txn_conn.cursor()
cur.execute(f"SELECT * FROM {type_mix_table}")
rows = cur.fetchall()
cur.close()
return len(rows)
benchmark.pedantic(run, rounds=10, iterations=1)
# ----------------------------------------------------------------------------
# Memory profile at 100k rows
# ----------------------------------------------------------------------------
def test_streaming_fetch_100k_memory_profile(
select_read_conn: informix_db.Connection,
scaling_select_table: str,
) -> None:
"""Sample RSS during a 100k-row iteration. Verifies the cursor's
memory footprint scales reasonably with row count.
Current cursor materializes the full result set on execute() (Phase
17 in-memory model), so RSS WILL grow proportional to row count.
The test documents the actual growth shape and provides a
regression baseline — if growth ever exceeds 500 MB for a 100k-row
fetch, something is leaking heavily.
Future server-cursor mode would maintain constant memory; this
test would then confirm flatness.
"""
import gc
import resource
def rss_kb() -> int:
# Use /proc/self/status VmRSS for *current* RSS, not peak.
# ``ru_maxrss`` is monotonic peak — a 68 MB peak from earlier
# in the test session masks any fluctuation from this fetch.
try:
from pathlib import Path
with Path("/proc/self/status").open() as f:
for line in f:
if line.startswith("VmRSS:"):
return int(line.split()[1])
except OSError:
pass
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
gc.collect()
pre_execute_rss = rss_kb()
cur = select_read_conn.cursor()
cur.execute(f"SELECT FIRST 100000 * FROM {scaling_select_table}")
post_execute_rss = rss_kb() # rows materialized into self._rows here
samples: list[tuple[int, int]] = []
rows_seen = 0
samples.append((0, post_execute_rss))
for _ in cur:
rows_seen += 1
if rows_seen % 10_000 == 0:
samples.append((rows_seen, rss_kb()))
cur.close()
gc.collect()
final_rss = rss_kb()
materialization_growth = post_execute_rss - pre_execute_rss
iteration_growth = final_rss - post_execute_rss
print("\nstreaming fetch 100k memory profile:")
print(f" pre-execute RSS: {pre_execute_rss:>9} KB")
print(f" post-execute RSS: {post_execute_rss:>9} KB "
f"(Δ {materialization_growth:+} KB — materialization cost)")
for rows, rss in samples[1:]:
print(f" rows={rows:>6} rss={rss:>9} KB "
f"(Δ from post-execute: {rss - post_execute_rss:+} KB)")
print(f" final={final_rss} KB after cur.close() + gc.collect()")
print(" --")
print(f" rows iterated: {rows_seen}")
print(f" materialization: ~{materialization_growth * 1024 // 100_000} "
f"bytes/row (100k rows of 5 cols)")
print(f" iteration-side allocation: {iteration_growth} KB total "
f"(should be ~0 — iteration doesn't allocate)")
total_growth_kb = final_rss - pre_execute_rss
# 500 MB ceiling for 100k rows = ~5 KB/row max. Real cost is ~50-100
# bytes/row (5 cols x tuple+strings+ints) so this is plenty of
# headroom for the regression check.
assert total_growth_kb < 500_000, (
f"100k-row fetch grew RSS by {total_growth_kb} KB — cursor is leaking"
)
assert rows_seen == 100_000, (
f"expected 100000 rows iterated, got {rows_seen}"
)
# Iteration-side allocation should be near-zero — fetchall() / for
# loop just walks the already-materialized self._rows list. Allow
# 5 MB slack for opportunistic allocator behavior.
assert iteration_growth < 5_000, (
f"iteration over already-fetched rows grew RSS by "
f"{iteration_growth} KB — unexpected per-row allocation"
)

81
tests/benchmarks/test_select_perf.py Normal file
View File

@ -0,0 +1,81 @@
"""End-to-end SELECT benchmarks.
Measure the full PREPARE → EXECUTE → FETCH → CLOSE → RELEASE round-trip
for representative query shapes. The codec micro-benchmarks set the
*ceiling* (best-case CPU); these tell you how much of that ceiling
the wire protocol + server response time eats.
Layered comparison:
- ``select_one_row`` — protocol-overhead floor (single tiny round-trip)
- ``select_systables_first`` — small server-side query (~10 rows)
- ``select_bench_table_all`` — full 1k-row table fetch (sustained throughput)
"""
from __future__ import annotations
import pytest
import informix_db
pytestmark = [pytest.mark.benchmark, pytest.mark.integration]
def test_select_one_row(benchmark, bench_conn: informix_db.Connection) -> None:
"""Single-row round-trip — protocol-overhead floor."""
def run() -> object:
cur = bench_conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
row = cur.fetchone()
cur.close()
return row
benchmark(run)
def test_select_systables_first_10(benchmark, bench_conn: informix_db.Connection) -> None:
"""Small server-side query — describes 4 columns, returns ~10 rows."""
def run() -> list:
cur = bench_conn.cursor()
cur.execute(
"SELECT FIRST 10 tabname, owner, tabid, ncols FROM systables"
)
rows = cur.fetchall()
cur.close()
return rows
benchmark(run)
def test_select_bench_table_all(
benchmark, bench_conn: informix_db.Connection, bench_table: str
) -> None:
"""1000-row sustained fetch — covers the typical reporting query."""
def run() -> list:
cur = bench_conn.cursor()
cur.execute(f"SELECT * FROM {bench_table}")
rows = cur.fetchall()
cur.close()
return rows
benchmark(run)
def test_select_with_param(
benchmark, bench_conn: informix_db.Connection, bench_table: str
) -> None:
"""Parameterized SELECT — exercises the BIND path."""
def run() -> list:
cur = bench_conn.cursor()
cur.execute(
f"SELECT id, name FROM {bench_table} WHERE counter > ?",
(5000,),
)
rows = cur.fetchall()
cur.close()
return rows
benchmark(run)

48
tests/docker-compose.legacy.yml Normal file
View File

@ -0,0 +1,48 @@
# Older Informix versions, for the server-compatibility matrix in the README.
#
# The primary dev container (tests/docker-compose.yml) runs Informix 15 on
# 9088. These two run alongside it on 9089 / 9090 so all three can be tested
# without tearing anything down.
#
# These images live on Docker Hub under ibmcom/ rather than icr.io/informix/ —
# IBM stopped publishing the older developer editions to the newer registry.
# They're tagged rather than digest-pinned because the ibmcom repo is frozen
# (no new pushes since the icr.io move), so the tags are already immutable in
# practice.
#
# make ifx-legacy-up start both
# make ifx-legacy-setup create blobspace1 + sbspace1 (needed for LOB tests)
# make test-matrix run the integration suite against all three
# make ifx-legacy-down stop and remove
services:
ifx1210:
container_name: informix-db-test-1210
image: ibmcom/informix-developer-database:12.10.FC12W1DE
privileged: true
environment:
LICENSE: accept
SIZE: small
ports:
- "9089:9088"
healthcheck:
test: ["CMD-SHELL", "exec 3<>/dev/tcp/127.0.0.1/9088 && exec 3<&- && exec 3>&-"]
interval: 5s
timeout: 3s
retries: 60
start_period: 30s
ifx1410:
container_name: informix-db-test-1410
image: ibmcom/informix-developer-database:14.10.FC7W1DE
privileged: true
environment:
LICENSE: accept
SIZE: small
ports:
- "9090:9088"
healthcheck:
test: ["CMD-SHELL", "exec 3<>/dev/tcp/127.0.0.1/9088 && exec 3<&- && exec 3>&-"]
interval: 5s
timeout: 3s
retries: 60
start_period: 30s

76
tests/setup-spaces.sh Executable file
View File

@ -0,0 +1,76 @@
#!/usr/bin/env bash
# Create blobspace1 + sbspace1 in an Informix dev container.
#
# The developer-edition images ship with only rootdbs. BYTE/TEXT tests need a
# blobspace; BLOB/CLOB (smart-LOB) tests need an sbspace plus SBSPACENAME set.
# Without them, ~21 integration tests fail with errors that look like driver
# bugs but are pure server configuration.
#
# Usage: tests/setup-spaces.sh <container-name>
#
# Idempotent: re-running against a container that already has the spaces prints
# the server's "already exists" complaint and exits 0.
#
# Version quirks this handles, learned the hard way:
# * INFORMIXDIR differs — 12.10/14.10 use /opt/ibm/informix, 15 nests a
# versioned subdirectory (/opt/ibm/informix/v15.0.1.0.3).
# * ONCONFIG is named `onconfig` on some images and `onconfig.informix` on
# others; we probe for the file rather than guessing.
# * `bash -lc` wipes INFORMIXDIR from the environment on these images, which
# makes every utility fail with "Unable to read $INFORMIXDIR (/usr/informix)".
# Use `bash -c` and export explicitly.
# * 12.10 DE ships no `ontape`. The level-0 archive turns out to be
# unnecessary for the tests, so we attempt it and shrug if it's missing.
set -euo pipefail
CONTAINER="${1:?usage: setup-spaces.sh <container-name>}"
docker exec -u informix "$CONTAINER" bash -c '
set -u
# Locate INFORMIXDIR: either /opt/ibm/informix or a versioned subdir under it.
for cand in /opt/ibm/informix /opt/ibm/informix/v*; do
if [ -x "$cand/bin/onspaces" ]; then
export INFORMIXDIR="$cand"
break
fi
done
if [ -z "${INFORMIXDIR:-}" ]; then
echo "could not locate INFORMIXDIR (no bin/onspaces found)" >&2
exit 1
fi
export PATH="$INFORMIXDIR/bin:$PATH"
export INFORMIXSERVER=informix
export INFORMIXSQLHOSTS="$INFORMIXDIR/etc/sqlhosts"
# ONCONFIG name varies across images.
for cfg in onconfig onconfig.informix; do
if [ -f "$INFORMIXDIR/etc/$cfg" ]; then
export ONCONFIG="$cfg"
break
fi
done
: "${ONCONFIG:?no onconfig found in $INFORMIXDIR/etc}"
echo "INFORMIXDIR=$INFORMIXDIR ONCONFIG=$ONCONFIG"
SPACES=/opt/ibm/data/spaces
mkdir -p "$SPACES"
for f in blobspace1 sbspace1; do
[ -e "$SPACES/$f" ] || : > "$SPACES/$f"
chmod 660 "$SPACES/$f"
done
onspaces -c -b blobspace1 -g 1 -p "$SPACES/blobspace1" -o 0 -s 50000 2>&1 | grep -vE "^\s*$" || true
onspaces -c -S sbspace1 -p "$SPACES/sbspace1" -o 0 -s 50000 -Df "AVG_LO_SIZE=100" 2>&1 | grep -vE "^\s*$" || true
onmode -wm SBSPACENAME=sbspace1 2>&1 | tail -1 || true
# Level-0 archive. Informix warns it is required after adding a space; in
# practice the tests pass without it, and 12.10 DE has no ontape at all.
onmode -wm TAPEDEV=/dev/null >/dev/null 2>&1 || true
if command -v ontape >/dev/null 2>&1; then
ontape -s -L 0 2>&1 | tail -1 || true
else
echo "ontape not present on this image; skipping level-0 archive"
fi
'

265
tests/test_async_threads.py Normal file
View File

@ -0,0 +1,265 @@
"""The async layer used a thread pool it shared with the whole process.
``asyncio.to_thread`` runs on the event loop's default executor. That
executor belongs to the process, not to us, and it is sized from the CPU
count — ``min(32, cpu_count + 4)``, which is six threads on a two-CPU
container. Every blocking call in ``informix_db.aio`` went through it.
Two consequences, both silent.
**Cancelled calls hold their threads.** ``asyncio.to_thread`` cannot
interrupt a worker, so a cancelled await leaves the thread running the
wire call until the read timeout expires. Cancellation is ordinary in a
web app — a client disconnect cancels the request task — so a handful of
them pins every thread in the shared pool. Unrelated ``to_thread`` work
anywhere else in the process then stops dead, and so does the driver.
**Pool concurrency was capped by an unrelated number.** A pool with
``max_size=20`` on a two-CPU box ran six queries at a time, and nothing
said so.
Each connection now owns one thread. That is the right size rather than
a compromise: the sync connection serializes every wire operation on its
own lock, so a second thread could do nothing but wait for the first. It
also avoids a deadlock that a shared pool-sized executor invites — with
N threads and N connections, N tasks blocked in ``acquire`` occupy every
thread while the connection they wait for is held by a task that now
needs a thread to finish and release it.
"""
from __future__ import annotations
import asyncio
import threading
import time
from concurrent.futures import ThreadPoolExecutor
import pytest
from informix_db import aio
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _kw(conn_params: ConnParams) -> dict:
return {
"host": conn_params.host,
"port": conn_params.port,
"user": conn_params.user,
"password": conn_params.password,
"database": conn_params.database,
"server": conn_params.server,
"connect_timeout": 10.0,
"read_timeout": 20.0,
"autocommit": True,
}
@pytest.mark.asyncio
async def test_queries_run_with_the_default_executor_fully_held(
conn_params: ConnParams,
) -> None:
"""The starvation case, with real queries. Four unrelated jobs occupy
every thread of a four-worker default executor; the driver must not
be waiting behind them."""
loop = asyncio.get_running_loop()
previous = loop._default_executor
loop.set_default_executor(ThreadPoolExecutor(max_workers=4))
conns = [await aio.connect(**_kw(conn_params)) for _ in range(4)]
cursors = [await c.cursor() for c in conns]
gate = threading.Event()
hogs = [
asyncio.create_task(asyncio.to_thread(gate.wait, 30)) for _ in range(4)
]
await asyncio.sleep(0.3)
async def query(cursor, i: int):
await cursor.execute(f"SELECT FIRST 1 tabid + {i} FROM systables")
return await cursor.fetchone()
try:
rows = await asyncio.wait_for(
asyncio.gather(*(query(c, i) for i, c in enumerate(cursors))),
timeout=10.0,
)
assert len(rows) == 4
assert all(r is not None for r in rows)
finally:
gate.set()
await asyncio.gather(*hogs, return_exceptions=True)
for c in conns:
await c.close()
# set_default_executor rejects None, which is what the loop
# starts with before anything has used to_thread.
loop.set_default_executor(previous or ThreadPoolExecutor())
@pytest.mark.asyncio
async def test_each_connection_gets_its_own_thread(
conn_params: ConnParams,
) -> None:
a = await aio.connect(**_kw(conn_params))
b = await aio.connect(**_kw(conn_params))
try:
assert a._executor is not b._executor
assert a._executor._max_workers == 1, (
"more than one thread per connection cannot help — the wire "
"lock serializes them anyway"
)
finally:
await a.close()
await b.close()
@pytest.mark.asyncio
async def test_cursor_runs_on_its_connections_thread(
conn_params: ConnParams,
) -> None:
"""A cursor must not fall back to the default executor, or half the
work goes back to being shared."""
conn = await aio.connect(**_kw(conn_params))
try:
cur = await conn.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
names: list[str] = []
await conn._run(lambda: names.append(threading.current_thread().name))
assert names[0].startswith("informix-conn")
assert cur._run.__self__ is conn
finally:
await conn.close()
@pytest.mark.asyncio
async def test_pool_reuses_one_thread_per_connection(
conn_params: ConnParams,
) -> None:
"""The executor lives on the sync connection, so a connection handed
out, returned, and handed out again keeps the same thread instead of
spawning one per acquire."""
pool = await aio.create_pool(**_kw(conn_params), min_size=1, max_size=2)
try:
seen = []
for _ in range(4):
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
assert await cur.fetchone() is not None
seen.append(id(conn._executor))
assert len(set(seen)) <= 2, (
f"expected at most one executor per pooled connection, saw "
f"{len(set(seen))}"
)
finally:
await pool.close()
@pytest.mark.asyncio
async def test_pool_concurrency_is_not_capped_by_the_default_executor(
conn_params: ConnParams,
) -> None:
"""Six concurrent pooled queries against a two-worker default
executor. Under the old arrangement at most two could run."""
loop = asyncio.get_running_loop()
previous = loop._default_executor
pool = await aio.create_pool(**_kw(conn_params), min_size=6, max_size=6)
loop.set_default_executor(ThreadPoolExecutor(max_workers=2))
try:
async def one(i: int):
async with pool.connection(timeout=15.0) as conn:
cur = await conn.cursor()
# Long enough that serialized execution would be obvious.
await cur.execute(
"SELECT FIRST 200 a.tabid FROM systables a, systables b"
)
return len(await cur.fetchall())
started = time.monotonic()
results = await asyncio.wait_for(
asyncio.gather(*(one(i) for i in range(6))), timeout=25.0
)
assert all(r > 0 for r in results)
assert len(results) == 6
elapsed = time.monotonic() - started
assert elapsed < 20.0, f"queries appear serialized ({elapsed:.1f}s)"
finally:
await pool.close()
# set_default_executor rejects None, which is what the loop
# starts with before anything has used to_thread.
loop.set_default_executor(previous or ThreadPoolExecutor())
@pytest.mark.asyncio
async def test_closing_a_connection_stops_its_thread(
conn_params: ConnParams,
) -> None:
"""One thread per connection is only affordable if the thread goes
away with the connection."""
before = threading.active_count()
conns = [await aio.connect(**_kw(conn_params)) for _ in range(5)]
for c in conns:
cur = await c.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
for c in conns:
await c.close()
del conns
# Threads exit asynchronously after shutdown; give them a moment.
for _ in range(50):
if threading.active_count() <= before + 1:
break
await asyncio.sleep(0.1)
assert threading.active_count() <= before + 1, (
f"connection threads outlived their connections "
f"({before} -> {threading.active_count()})"
)
@pytest.mark.asyncio
async def test_row_factory_reaches_the_async_paths(
conn_params: ConnParams,
) -> None:
"""The async layer wraps the sync cursor, so row_factory should flow
through untouched. Each of the four async fetch routes is a separate
call site, and async iteration goes through __anext__ rather than
__next__."""
import informix_db
conn = await aio.connect(row_factory=informix_db.Row, **_kw(conn_params))
try:
cur = await conn.cursor()
sql = "SELECT FIRST 2 tabid, tabname FROM systables ORDER BY tabid"
await cur.execute(sql)
one = await cur.fetchone()
assert one[0] == one["tabid"] == one.tabid
await cur.execute(sql)
assert [r.tabid for r in await cur.fetchall()] == [1, 2]
await cur.execute(sql)
assert all(r.tabname for r in await cur.fetchmany(2))
await cur.execute(sql)
assert [r.tabid async for r in cur] == [1, 2]
finally:
await conn.close()
@pytest.mark.asyncio
async def test_async_pool_forwards_the_row_factory(
conn_params: ConnParams,
) -> None:
import informix_db
pool = await aio.create_pool(
row_factory=informix_db.Row, min_size=1, max_size=2, **_kw(conn_params)
)
try:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
assert (await cur.fetchone()).tabid == 1
finally:
await pool.close()

258
tests/test_batch_scroll_lob.py Normal file
View File

@ -0,0 +1,258 @@
"""Regression tests for pipelined executemany, scrollable cursors, and LOBs.
Found here by fuzzing: **a client-side encoding failure inside an
``executemany`` batch leaked the prepared statement.** The PDUs are built
after the PREPARE, so a value the connection's codec cannot represent
raises there — before anything is drained — and the exception escaped
without the RELEASE. The leaked statement then collided with the next
PREPARE and every later call on that connection failed with an error
pointing at the *previous* SQL.
``_execute_dml_with_params`` already guarded exactly this case for the
single-row path. The pipelined path was simply missed, which is the
recurring shape of these bugs: a hazard understood in one place and not
carried to its sibling.
The rest of this file is the coverage that proved the neighbouring paths
sound — batch failures at every position, scroll boundaries, LOB sizes —
kept so they stay that way. Each failure case ends in a health check,
because the interesting damage is never in the statement that failed.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=15.0,
read_timeout=30.0,
**kw,
)
def _assert_healthy(cur) -> None:
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None, "connection unusable"
# ---------------------------------------------------------------------------
# executemany — the encoding-failure leak, and the neighbours
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("n", [3, 50])
@pytest.mark.parametrize("position", ["first", "mid", "last"])
def test_encoding_failure_in_batch_does_not_leak(
conn_params: ConnParams, n: int, position: str
) -> None:
"""The bug. A value the codec can't encode raises after the PREPARE;
escaping without the RELEASE bricked the connection."""
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_enc (s VARCHAR(32))")
rows: list[tuple] = [(f"s{i}",) for i in range(n)]
rows[idx] = ("中文",) # not representable in iso-8859-1
with pytest.raises(Exception): # noqa: B017 — DataError or UnicodeError
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
# The connection must survive, repeatedly.
for _ in range(3):
_assert_healthy(cur)
with pytest.raises(Exception): # noqa: B017
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
_assert_healthy(cur)
@pytest.mark.parametrize("n", [2, 3, 10, 100])
@pytest.mark.parametrize("position", ["first", "mid", "last"])
def test_constraint_violation_in_batch_recovers(
conn_params: ConnParams, n: int, position: str
) -> None:
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_dup (k INT PRIMARY KEY)")
cur.execute("INSERT INTO t_em_dup VALUES (?)", (10_000,))
rows = [(i,) for i in range(n)]
rows[idx] = (10_000,)
with pytest.raises(informix_db.Error):
cur.executemany("INSERT INTO t_em_dup VALUES (?)", rows)
_assert_healthy(cur)
# Whatever the partial-batch semantics, COUNT(*) and a full fetch
# must agree — a disagreement means the row decoder and the server
# have different ideas about what is in the table.
cur.execute("SELECT COUNT(*) FROM t_em_dup")
(counted,) = cur.fetchone()
cur.execute("SELECT k FROM t_em_dup")
assert counted == len(cur.fetchall())
@pytest.mark.parametrize("n", [1, 2, 3, 10, 100, 1000])
def test_executemany_inserts_exactly_n_rows(
conn_params: ConnParams, n: int
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_ok (k INT, v VARCHAR(16))")
cur.executemany(
"INSERT INTO t_em_ok VALUES (?, ?)",
[(i, f"v{i}") for i in range(n)],
)
cur.execute("SELECT COUNT(*) FROM t_em_ok")
assert cur.fetchone() == (n,)
def test_executemany_empty_and_single(conn_params: ConnParams) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_edge (k INT)")
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [])
_assert_healthy(cur)
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [(1,)])
cur.execute("SELECT COUNT(*) FROM t_em_edge")
assert cur.fetchone() == (1,)
# ---------------------------------------------------------------------------
# Scrollable cursors — boundaries get their own round-trip each
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("n", [0, 1, 2, 5, 50, 300])
def test_scroll_boundaries(conn_params: ConnParams, n: int) -> None:
with _connect(conn_params, autocommit=True) as conn:
setup = conn.cursor()
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll")
setup.execute("CREATE TABLE t_scroll (k INT)")
try:
if n:
setup.executemany(
"INSERT INTO t_scroll VALUES (?)", [(i,) for i in range(n)]
)
cur = conn.cursor(scrollable=True)
try:
cur.execute("SELECT k FROM t_scroll ORDER BY k")
assert cur.fetch_first() == ((0,) if n else None)
assert cur.fetch_last() == ((n - 1,) if n else None)
if n >= 3:
assert cur.fetch_absolute(2) == (2,)
assert cur.fetch_prior() == (1,)
assert cur.fetch_relative(2) == (3,)
# Off both ends must be None — not a crash, not a wrap.
assert cur.fetch_absolute(n + 50) is None
cur.fetch_first()
assert cur.fetch_prior() is None
if n:
cur.fetch_first()
seen = [(0,)]
while (row := cur.fetchone()) is not None:
seen.append(row)
assert seen == [(i,) for i in range(n)]
finally:
cur.close()
_assert_healthy(setup)
finally:
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll")
def test_abandoned_scroll_cursors_do_not_leak(
conn_params: ConnParams,
) -> None:
"""Scroll cursors stay open server-side, so abandoning one leaks
unless the finalizer runs."""
with _connect(conn_params, autocommit=True) as conn:
setup = conn.cursor()
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll_ab")
setup.execute("CREATE TABLE t_scroll_ab (k INT)")
try:
setup.executemany(
"INSERT INTO t_scroll_ab VALUES (?)", [(i,) for i in range(20)]
)
for i in range(20):
c = conn.cursor(scrollable=True)
c.execute("SELECT k FROM t_scroll_ab ORDER BY k")
c.fetch_first()
if i % 2:
c.close()
else:
del c # rely on the finalizer
_assert_healthy(setup)
setup.execute("SELECT COUNT(*) FROM t_scroll_ab")
assert setup.fetchone() == (20,)
finally:
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll_ab")
# ---------------------------------------------------------------------------
# Smart LOBs
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
"size", [0, 1, 255, 256, 1023, 1024, 4095, 4096, 65535, 65536]
)
def test_blob_round_trip_sizes(conn_params: ConnParams, size: int) -> None:
"""Sizes straddle the 4096-byte SQ_FILE chunk and the 64K mark."""
payload = bytes((i * 7 + size) % 256 for i in range(size))
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_blob_rt")
try:
cur.execute("CREATE TABLE t_blob_rt (k INT, b BLOB)")
except informix_db.Error:
pytest.skip("no sbspace configured; see make ifx-spaces")
try:
cur.write_blob_column(
"INSERT INTO t_blob_rt VALUES (?, BLOB_PLACEHOLDER)",
payload, (1,),
)
got = cur.read_blob_column(
"SELECT b FROM t_blob_rt WHERE k = ?", (1,)
)
if size == 0:
assert got in (b"", None)
else:
assert got == payload
_assert_healthy(cur)
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_blob_rt")
def test_failed_blob_read_recovers(conn_params: ConnParams) -> None:
"""The SQ_FILE path involves a server-side temp file; a failure part
way through must not strand the connection."""
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
for _ in range(5):
with contextlib.suppress(Exception):
cur.read_blob_column("SELECT b FROM t_no_such_blob_tbl", ())
_assert_healthy(cur)

144
tests/test_capabilities.py Normal file
View File

@ -0,0 +1,144 @@
"""Integration tests for live SQ_PROTOCOLS negotiation.
These run against whatever server ``IFX_PORT`` points at, so `make
test-matrix` exercises them on 12.10, 14.10, and 15 in turn.
The important one is ``test_no_violated_assumptions``: this driver
hardcodes several wire-framing choices that SQLI actually negotiates, and
a mismatch corrupts rows silently. That test turns "we assume this" into
"we check this on every supported server".
"""
from __future__ import annotations
import pytest
import informix_db
from informix_db._capabilities import ENHANCED_PROTOCOL_CAP, ServerCapabilities
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=10.0,
)
def test_capabilities_are_decoded(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
caps = conn.server_capabilities
assert isinstance(caps, ServerCapabilities)
assert caps.raw_mask, "server sent an empty protocols reply"
assert caps.bits
def test_no_violated_assumptions(conn_params: ConnParams) -> None:
"""Every wire-framing shape we hardcode is one the server negotiated.
If this fails, rows are being decoded against the wrong framing and
the fix is to branch on the capability rather than assume it.
"""
with _connect(conn_params) as conn:
caps = conn.server_capabilities
assert caps is not None
assert caps.violated_assumptions() == []
def test_enhanced_protocol_negotiated(conn_params: ConnParams) -> None:
"""Cap_1 is the client's declared protocol level echoed back. Every
supported server accepts 316."""
with _connect(conn_params) as conn:
caps = conn.server_capabilities
assert caps is not None
assert caps.cap_1 == ENHANCED_PROTOCOL_CAP
assert caps.enhanced_protocol
def test_framing_capabilities_present(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
caps = conn.server_capabilities
assert caps is not None
assert caps.four_byte_offset
assert caps.varchar_var_len
assert caps.remove_64k_limit
assert caps.usver
def test_server_version_is_exposed(conn_params: ConnParams) -> None:
"""Assert the shape rather than a value so this holds across the matrix."""
with _connect(conn_params) as conn:
version = conn.server_version
assert "Informix" in version
assert "Version" in version
def test_server_version_reports_release_not_protocol_version(
conn_params: ConnParams,
) -> None:
"""A field report flagged ``server_version`` as looking like a
client-SDK version: the login response announces 12.10 servers as 9.56
and 14.10 as 9.59. ``server_version`` now answers with the release; the
raw login string moved to ``server_version_internal``."""
with _connect(conn_params) as conn:
release = conn.server_version
internal = conn.server_version_internal
assert "Informix" in internal
# On 12.10/14.10 these genuinely differ; on 15 they agree.
if "9.5" in internal:
assert release != internal
assert "9.5" not in release, (
f"server_version still reports the protocol version: {release!r}"
)
def test_server_version_is_cached(conn_params: ConnParams) -> None:
"""It costs a round-trip, so repeated access must not repeat it."""
with _connect(conn_params) as conn:
assert conn.server_version == conn.server_version
assert conn._server_version_full is not None
def test_server_version_degrades_without_a_database(
conn_params: ConnParams,
) -> None:
"""The DBINFO lookup needs an open database. With none, the property
must fall back rather than raise — a version lookup should never be
able to break a working connection."""
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=None,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=10.0,
)
try:
assert isinstance(conn.server_version, str)
finally:
conn.close()
def test_capabilities_survive_multiple_connections(
conn_params: ConnParams,
) -> None:
"""Negotiation happens per-connection; two connections to the same
server must agree."""
with _connect(conn_params) as first, _connect(conn_params) as second:
assert first.server_capabilities is not None
assert second.server_capabilities is not None
assert (
first.server_capabilities.raw_mask
== second.server_capabilities.raw_mask
)
assert first.server_capabilities.bits == second.server_capabilities.bits

200
tests/test_capabilities_unit.py Normal file
View File

@ -0,0 +1,200 @@
"""Unit tests for SQ_PROTOCOLS capability decoding — no server required.
The masks below were captured from live servers on 2026-08-27. Their most
interesting property is that the first eight bytes are identical across
Informix 12.10, 14.10, and 15 — which is why those three releases speak an
indistinguishable SQLI dialect.
"""
from __future__ import annotations
import dataclasses
import pytest
from informix_db._capabilities import (
BIT_FOUR_BYTE_OFFSET,
BIT_REMOVE_64K_LIMIT,
BIT_USVER,
BIT_VARCHAR_VAR_LEN,
CLIENT_PROTOCOLS_MASK,
ENHANCED_PROTOCOL_CAP,
ServerCapabilities,
_decode_bits,
)
MASK_15 = bytes.fromhex("bdbe9ffe7fb7ffefff")
MASK_1410 = bytes.fromhex("bdbe9ffe7fb7ffeff8")
MASK_1210 = bytes.fromhex("bdbe9ffe7fb7ffeff0")
ALL_MASKS = [
pytest.param(MASK_15, id="15.0.1.0.3"),
pytest.param(MASK_1410, id="14.10.FC7W1"),
pytest.param(MASK_1210, id="12.10.FC12W1DE"),
]
# --------------------------------------------------------------------------
# Bit expansion
# --------------------------------------------------------------------------
def test_bit_numbering_is_msb_first() -> None:
assert _decode_bits(b"\x80") == {0}
assert _decode_bits(b"\x01") == {7}
assert _decode_bits(b"\x00\x80") == {8}
assert _decode_bits(b"\x00\x01") == {15}
def test_bit_expansion_of_first_captured_byte() -> None:
"""0xBD is 1011 1101 -> bits 0, 2, 3, 4, 5, 7."""
assert _decode_bits(b"\xbd") == {0, 2, 3, 4, 5, 7}
def test_empty_mask_yields_no_bits() -> None:
assert _decode_bits(b"") == set()
# --------------------------------------------------------------------------
# The captured masks
# --------------------------------------------------------------------------
def test_first_eight_bytes_identical_across_versions() -> None:
"""The finding this whole module exists to record: 12.10, 14.10, and 15
negotiate exactly the same 64-bit capability set. Only the 9th byte —
which IBM's own JDBC driver discards — differs."""
assert MASK_15[:8] == MASK_1410[:8] == MASK_1210[:8]
assert len({MASK_15[8], MASK_1410[8], MASK_1210[8]}) == 3
@pytest.mark.parametrize("mask", ALL_MASKS)
def test_framing_bits_set_on_every_tested_server(mask: bytes) -> None:
"""Every wire-framing choice this driver hardcodes is one the server
actually negotiated. If this ever fails, the row decoder is wrong."""
caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP)
assert caps.four_byte_offset, "describe offsets would be misparsed"
assert caps.varchar_var_len, "VARCHAR framing would be misparsed"
assert caps.remove_64k_limit, "fast-path length prefix would be wrong"
assert caps.bigint
assert caps.long_id
assert caps.lvarchar_gt_2k
assert caps.gls
@pytest.mark.parametrize("mask", ALL_MASKS)
def test_no_violated_assumptions_on_tested_servers(mask: bytes) -> None:
caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP)
assert caps.violated_assumptions() == []
@pytest.mark.parametrize("mask", ALL_MASKS)
def test_ninth_byte_is_decoded(mask: bytes) -> None:
"""JDBC drops bits 64+; we keep them because they're the only part
that varies between releases."""
caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP)
assert any(b >= 64 for b in caps.bits)
# --------------------------------------------------------------------------
# Pre-set bits — the reason isUSVER is a dead end
# --------------------------------------------------------------------------
def test_preset_bits_applied_when_cap1_nonzero() -> None:
"""JDBC pre-sets {0,2,3,4,49,51} for any non-zero Cap_1 and BitSet.set
never clears, so these are true on any modern server regardless of the
mask. That's why isUSVER never varies and can't explain version-specific
behaviour."""
caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=ENHANCED_PROTOCOL_CAP)
assert {0, 2, 3, 4, 49, 51}.issubset(caps.bits)
assert caps.usver
def test_preset_bits_not_applied_when_cap1_zero() -> None:
caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=0)
assert caps.bits == frozenset()
assert not caps.usver
def test_preset_cannot_clear_a_bit_the_mask_set() -> None:
caps = ServerCapabilities.from_wire(MASK_15, cap_1=ENHANCED_PROTOCOL_CAP)
assert caps.has(BIT_USVER)
assert caps.has(BIT_FOUR_BYTE_OFFSET)
# --------------------------------------------------------------------------
# Assumption violations — the diagnostic path
# --------------------------------------------------------------------------
def test_missing_four_byte_offset_is_reported() -> None:
# Everything set except bit 50.
bits = bytearray(b"\xff" * 9)
bits[BIT_FOUR_BYTE_OFFSET // 8] &= ~(0x80 >> (BIT_FOUR_BYTE_OFFSET % 8)) & 0xFF
caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP)
problems = caps.violated_assumptions()
assert len(problems) == 1
assert "4-byte describe offsets" in problems[0]
def test_missing_varchar_var_len_is_reported() -> None:
bits = bytearray(b"\xff" * 9)
bits[BIT_VARCHAR_VAR_LEN // 8] &= ~(0x80 >> (BIT_VARCHAR_VAR_LEN % 8)) & 0xFF
caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP)
problems = caps.violated_assumptions()
assert len(problems) == 1
assert "variable-length VARCHAR" in problems[0]
def test_missing_remove_64k_is_reported() -> None:
bits = bytearray(b"\xff" * 9)
bits[BIT_REMOVE_64K_LIMIT // 8] &= ~(0x80 >> (BIT_REMOVE_64K_LIMIT % 8)) & 0xFF
caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP)
problems = caps.violated_assumptions()
assert len(problems) == 1
assert "64K limit" in problems[0]
def test_all_zero_mask_reports_every_assumption() -> None:
caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=ENHANCED_PROTOCOL_CAP)
assert len(caps.violated_assumptions()) == 3
# --------------------------------------------------------------------------
# Misc
# --------------------------------------------------------------------------
def test_client_offer_matches_jdbc_reference() -> None:
"""IfxSqliConnect.clientProtocols, verbatim. Changing this changes what
the server negotiates, so it is pinned."""
jdbc_reference = bytes([0xFF, 0xFC, 0x7F, 0xFC, 0x3C, 0x8C, 0xAA, 0x97])
assert jdbc_reference == CLIENT_PROTOCOLS_MASK
assert len(CLIENT_PROTOCOLS_MASK) == 8
def test_enhanced_protocol_requires_exact_cap1() -> None:
"""JDBC tests == 316, not >=, because the server echoes the client's own
declared level rather than reporting its own version."""
assert ServerCapabilities.from_wire(MASK_15, cap_1=316).enhanced_protocol
assert not ServerCapabilities.from_wire(MASK_15, cap_1=315).enhanced_protocol
assert not ServerCapabilities.from_wire(MASK_15, cap_1=317).enhanced_protocol
def test_repr_is_readable_and_includes_mask() -> None:
caps = ServerCapabilities.from_wire(
MASK_15, cap_1=316, server_version="IBM Informix Dynamic Server Version 15"
)
text = repr(caps)
assert "bdbe9ffe7fb7ffefff" in text
assert "usver" in text
assert "cap_1=316" in text
def test_capabilities_are_frozen() -> None:
"""Immutable so a connection's negotiated state can't be edited after
the fact and quietly disagree with what's on the wire."""
caps = ServerCapabilities.from_wire(MASK_15, cap_1=316)
with pytest.raises(dataclasses.FrozenInstanceError):
caps.cap_1 = 1 # type: ignore[misc]

151
tests/test_datetime_fraction.py Normal file
View File

@ -0,0 +1,151 @@
"""Regression tests for DATETIME sub-second precision on bind.
``_encode_datetime`` used to emit YEAR TO SECOND unconditionally, so
binding a ``datetime`` carrying microseconds into a
``DATETIME YEAR TO FRACTION(n)`` column stored zeros — silently, with no
error and no warning. Reads were always fine, which is what made it hard
to notice: the value only went missing on the way in.
It now emits YEAR TO FRACTION(5) when ``microsecond`` is non-zero and
keeps the original YEAR TO SECOND encoding otherwise.
FRACTION(5) is Informix's widest and resolves to 10 µs, so Python's
sixth microsecond digit is dropped. That's a real limit of the type, not
a driver choice, and the tests below pin the truncation so it can't drift
into something worse.
"""
from __future__ import annotations
import datetime
import pytest
import informix_db
from informix_db.converters import _encode_datetime
from tests.conftest import ConnParams
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=10.0,
)
# --------------------------------------------------------------------------
# Encoder unit tests — no server needed
# --------------------------------------------------------------------------
def test_encoder_uses_year_to_second_without_microseconds() -> None:
"""The long-exercised path must stay byte-identical."""
type_code, prec, raw = _encode_datetime(
datetime.datetime(2026, 8, 31, 12, 30, 15)
)
assert type_code == 10
assert prec == (14 << 8) | 10 # digit_count 14, YEAR..SECOND
assert raw == b"\x00\x08\xc7\x14\x1a\x08\x1f\x0c\x1e\x0f"
assert len(raw) == 10 # 2 len + 1 exp + 7 BCD pairs
def test_encoder_widens_to_fraction_when_microseconds_present() -> None:
type_code, prec, raw = _encode_datetime(
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
)
assert type_code == 10
assert prec == (19 << 8) | 15 # digit_count 19, YEAR..FRACTION(5)
assert len(raw) == 13 # 2 len + 1 exp + 10 BCD pairs
# Exponent byte is unchanged: the integer part is still 7 base-100
# pairs, the fraction just adds three more after the point.
assert raw[2] == 0xC7
# Trailing pairs carry 120000 as BCD 12/00/00.
assert raw[-3:] == b"\x0c\x00\x00"
def test_encoder_pads_fraction_to_six_digits() -> None:
"""One microsecond must not shift the BCD pairs."""
_, prec, raw = _encode_datetime(
datetime.datetime(2026, 8, 31, 12, 30, 15, 1)
)
assert prec == (19 << 8) | 15
assert len(raw) == 13
assert raw[-3:] == b"\x00\x00\x01" # 000001
# --------------------------------------------------------------------------
# Round-trip against a real server
# --------------------------------------------------------------------------
@pytest.mark.integration
@pytest.mark.parametrize(
("microsecond", "expected"),
[
(0, 0),
(120000, 120000),
(500000, 500000),
(1, 0), # below FRACTION(5) resolution
(999999, 999990), # truncated to 5 significant digits
],
)
def test_fraction_round_trip(
conn_params: ConnParams, microsecond: int, expected: int
) -> None:
value = datetime.datetime(2026, 8, 31, 12, 30, 15, microsecond)
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_dt_frac "
"(k INT, t DATETIME YEAR TO FRACTION(5))"
)
cur.execute("INSERT INTO t_dt_frac VALUES (?, ?)", (1, value))
cur.execute("SELECT t FROM t_dt_frac")
(got,) = cur.fetchone()
assert got == value.replace(microsecond=expected)
@pytest.mark.integration
def test_fraction_bind_into_year_to_second_column(
conn_params: ConnParams,
) -> None:
"""Widening the bind must not break narrower columns — Informix
converts between qualifiers on assignment, truncating server-side."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_dt_sec (k INT, t DATETIME YEAR TO SECOND)"
)
cur.execute(
"INSERT INTO t_dt_sec VALUES (?, ?)",
(1, datetime.datetime(2026, 8, 31, 12, 30, 15, 987654)),
)
cur.execute("SELECT t FROM t_dt_sec")
assert cur.fetchone() == (
datetime.datetime(2026, 8, 31, 12, 30, 15),
)
@pytest.mark.integration
def test_fraction_survives_alongside_lvarchar(
conn_params: ConnParams,
) -> None:
"""The reported schema pairs FRACTION(5) columns with LVARCHARs."""
ts = datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_dt_lv "
"(s LVARCHAR(512), t DATETIME YEAR TO FRACTION(5), n INT8 NOT NULL)"
)
cur.execute(
"INSERT INTO t_dt_lv VALUES (?, ?, ?)", ("PackageRoot", ts, 10)
)
cur.execute("SELECT s, t, n FROM t_dt_lv")
assert cur.fetchone() == ("PackageRoot", ts, 10)

214
tests/test_executemany_pipeline.py Normal file
View File

@ -0,0 +1,214 @@
"""Phase 33 integration tests — pipelined ``executemany`` correctness.
The pipelined executemany sends all N BIND+EXECUTE PDUs to the wire
before draining any response. Hamilton's review of Phase 33 flagged
C1: this assumes the server sends *exactly* N responses for N
pipelined PDUs even when one row fails. If the server cuts the
response stream short on first error, the drain loop would block
reading bytes that never arrive — the connection would deadlock on
the next read.
These tests verify the wire-alignment assumption holds:
1. Constraint violation at row 500 of 1000 — happy-failure case.
2. Wire-alignment recovery — connection is still usable after the
error (proving the RELEASE drain succeeded and we read all the
remaining error responses).
3. Subsequent operations on the same connection work — proves no
stray bytes on the wire.
"""
from __future__ import annotations
import contextlib
from collections.abc import Iterator
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
@pytest.fixture
def constraint_table(logged_db_params: ConnParams) -> Iterator[str]:
"""Table with a UNIQUE constraint on ``id`` so we can force a
constraint violation at a known row.
"""
table = "p33_constraint"
conn = informix_db.connect(
host=logged_db_params.host,
port=logged_db_params.port,
user=logged_db_params.user,
password=logged_db_params.password,
database=logged_db_params.database,
server=logged_db_params.server,
autocommit=True,
)
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute(f"DROP TABLE {table}")
cur.execute(
f"CREATE TABLE {table} (id INT NOT NULL PRIMARY KEY, name VARCHAR(64))"
)
conn.close()
try:
yield table
finally:
conn = informix_db.connect(
host=logged_db_params.host,
port=logged_db_params.port,
user=logged_db_params.user,
password=logged_db_params.password,
database=logged_db_params.database,
server=logged_db_params.server,
autocommit=True,
)
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute(f"DROP TABLE {table}")
conn.close()
def test_pipelined_executemany_mid_batch_constraint_violation(
logged_db_params: ConnParams, constraint_table: str
) -> None:
"""C1 (Hamilton): force a constraint violation at row 500 of 1000;
verify the pipeline drains cleanly and the connection is usable
afterward.
This is the test that validates Phase 33's wire-alignment
assumption. If Informix sends fewer than 1000 responses for 1000
pipelined PDUs after the row-500 failure, this test will hang on
the drain loop's read (eventually timing out, but the test will
fail loudly either way).
"""
conn = informix_db.connect(
host=logged_db_params.host,
port=logged_db_params.port,
user=logged_db_params.user,
password=logged_db_params.password,
database=logged_db_params.database,
server=logged_db_params.server,
autocommit=False,
read_timeout=30.0, # if the wire desyncs, fail loudly within 30s
)
try:
# Pre-seed row 500 so the executemany's row-500 INSERT will
# violate the UNIQUE constraint.
cur = conn.cursor()
cur.execute(
f"INSERT INTO {constraint_table} VALUES (?, ?)",
(500, "pre-existing"),
)
conn.commit()
# Now executemany 1000 rows; row 500 will collide
rows = [(i, f"row_{i}") for i in range(1000)]
with pytest.raises(informix_db.IntegrityError) as exc_info:
cur.executemany(
f"INSERT INTO {constraint_table} VALUES (?, ?)", rows
)
# The error message should identify which row failed in the batch
err_msg = str(exc_info.value)
assert "row 500" in err_msg or "500" in err_msg, (
f"error message should identify the failed row index: {err_msg}"
)
# Whatever the transaction state, rolling back is the correct
# response to a failed batch.
conn.rollback()
# The connection MUST be usable after the failed batch.
# If the wire is desynced, this query will block or fail
# with a ProtocolError. The test passing here proves the
# pipeline drained cleanly.
cur = conn.cursor()
cur.execute(f"SELECT COUNT(*) FROM {constraint_table}")
(count,) = cur.fetchone()
# After rollback, only the pre-seeded row 500 remains
assert count == 1, (
f"expected only the pre-seeded row to remain, got {count} "
"(transaction didn't roll back cleanly?)"
)
finally:
conn.close()
def test_pipelined_executemany_first_row_fails(
logged_db_params: ConnParams, constraint_table: str
) -> None:
"""Edge case: failure on the FIRST row of the pipeline. Tests that
the drain loop correctly handles "every response after this is an
error" without falling apart on the very first response."""
conn = informix_db.connect(
host=logged_db_params.host,
port=logged_db_params.port,
user=logged_db_params.user,
password=logged_db_params.password,
database=logged_db_params.database,
server=logged_db_params.server,
autocommit=False,
read_timeout=30.0,
)
try:
cur = conn.cursor()
cur.execute(
f"INSERT INTO {constraint_table} VALUES (?, ?)", (0, "seeded")
)
conn.commit()
rows = [(i, f"row_{i}") for i in range(100)]
with pytest.raises(informix_db.IntegrityError):
cur.executemany(
f"INSERT INTO {constraint_table} VALUES (?, ?)", rows
)
conn.rollback()
cur = conn.cursor()
cur.execute(f"SELECT COUNT(*) FROM {constraint_table}")
(count,) = cur.fetchone()
assert count == 1
finally:
conn.close()
def test_pipelined_executemany_last_row_fails(
logged_db_params: ConnParams, constraint_table: str
) -> None:
"""Edge case: failure on the LAST row of the pipeline. Tests that
we don't accidentally short-circuit the drain when we see the
"expected" rowcount before the actual error response arrives."""
conn = informix_db.connect(
host=logged_db_params.host,
port=logged_db_params.port,
user=logged_db_params.user,
password=logged_db_params.password,
database=logged_db_params.database,
server=logged_db_params.server,
autocommit=False,
read_timeout=30.0,
)
try:
cur = conn.cursor()
cur.execute(
f"INSERT INTO {constraint_table} VALUES (?, ?)",
(99, "seeded-last"),
)
conn.commit()
rows = [(i, f"row_{i}") for i in range(100)]
with pytest.raises(informix_db.IntegrityError):
cur.executemany(
f"INSERT INTO {constraint_table} VALUES (?, ?)", rows
)
conn.rollback()
cur = conn.cursor()
cur.execute(f"SELECT COUNT(*) FROM {constraint_table}")
(count,) = cur.fetchone()
assert count == 1
finally:
conn.close()

71
tests/test_int8_unit.py Normal file
View File

@ -0,0 +1,71 @@
"""Unit tests for the INT8 / SERIAL8 codec — no server required.
The byte vectors below were captured off the wire from both Informix
12.10.FC12W1DE and 15.0.1.0.3DE, which emit byte-identical encodings.
INT8 is sign-magnitude across two 32-bit halves, stored high-half-last:
bytes 0-1 sign word: 0 = NULL, 1 = positive, 0xFFFF = negative
bytes 2-5 LOW 32 bits, big-endian unsigned
bytes 6-9 HIGH 32 bits, big-endian unsigned
The trap: +n and -n have *identical* magnitude bytes. Anything that
treats this as a two's-complement integer is wrong for every negative
value while looking correct for every positive one.
"""
from __future__ import annotations
import pytest
from informix_db._types import IfxType
from informix_db.converters import FIXED_WIDTHS, _decode_int8
# (hex bytes, expected value) — captured from the wire.
WIRE_VECTORS = [
("0001be991a140000001c", 123456789012),
("ffffbe991a140000001c", -123456789012),
("00010000002a00000000", 42),
("ffff0000002a00000000", -42),
("00010000000000000000", 0),
("00000000000000000000", None), # sign word 0 -> NULL
("0001ffffffffffffffff", 2**64 - 1), # both halves saturated
("00010000000100000000", 1),
("00010000000000000001", 1 << 32), # high half only
]
@pytest.mark.parametrize(("hexbytes", "expected"), WIRE_VECTORS)
def test_decode_int8_wire_vectors(hexbytes: str, expected: int | None) -> None:
assert _decode_int8(bytes.fromhex(hexbytes)) == expected
def test_positive_and_negative_share_magnitude_bytes() -> None:
"""The whole reason a naive 8-byte read gets negatives wrong."""
pos = bytes.fromhex("0001be991a140000001c")
neg = bytes.fromhex("ffffbe991a140000001c")
assert pos[2:] == neg[2:], "magnitude bytes should be identical"
assert _decode_int8(pos) == -_decode_int8(neg)
def test_null_sign_word_beats_nonzero_magnitude() -> None:
"""Sign word 0 means NULL regardless of what the magnitude bytes hold."""
assert _decode_int8(bytes.fromhex("0000be991a140000001c")) is None
def test_short_payload_raises() -> None:
with pytest.raises(ValueError, match="too short"):
_decode_int8(bytes.fromhex("0001be991a14")) # 6 bytes, need 10
def test_int8_registered_as_ten_bytes() -> None:
"""A width of 8 here would desync every row containing an INT8."""
assert FIXED_WIDTHS[IfxType.INT8] == 10
assert FIXED_WIDTHS[IfxType.SERIAL8] == 10
def test_int8_width_differs_from_bigint() -> None:
"""INT8 (17) and BIGINT (52) are different types with different
widths — conflating them is the easy mistake."""
assert FIXED_WIDTHS[IfxType.INT8] != FIXED_WIDTHS[IfxType.BIGINT]
assert FIXED_WIDTHS[IfxType.BIGINT] == 8

249
tests/test_lvarchar_framing.py Normal file
View File

@ -0,0 +1,249 @@
"""Regression tests for LVARCHAR tuple framing, reported 2026-08-31.
Two independent framing errors, both of which shifted every column that
followed an LVARCHAR:
1. **A phantom pad byte.** We appended an even-byte pad when the value
length was odd. There is no pad — the next column begins immediately
after the last content byte.
2. **A missing length on NULL.** We returned as soon as the null
indicator said NULL, leaving the 4-byte length field unread. The
length is part of the envelope and is always present.
Both survived a 247-test suite because the only LVARCHAR fixture used
``'lv value'`` — 8 characters, even, and never NULL. Neither faulty
branch ever executed.
The tests below therefore vary length parity and nullness deliberately,
and always place a column *after* the LVARCHAR, because the damage lands
downstream: a trailing LVARCHAR can be mis-sized with no visible effect.
Wire evidence (Informix 12.10) for INT8 / LVARCHAR / INT8:
'odd' -> 00 | 00 00 00 03 | 6f 64 64 8 bytes
NULL -> 01 | 00 00 00 00 5 bytes
'' -> 00 | 00 00 00 00 5 bytes
NULL and empty string differ only in the indicator byte.
"""
from __future__ import annotations
import datetime
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=10.0,
)
# --------------------------------------------------------------------------
# Length parity — the phantom pad byte
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"text",
[
"", # 0 — even, empty
"a", # 1 — ODD
"ab", # 2
"abc", # 3 — ODD
"PackageRoot", # 11 — ODD, the reported value
"lv value", # 8 — the old fixture that hid the bug
"x" * 255, # 255 — ODD, spans a length byte boundary
"y" * 256, # 256
],
)
def test_lvarchar_length_parity_does_not_shift_next_column(
conn_params: ConnParams, text: str
) -> None:
"""A trailing sentinel column catches any over- or under-read."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_parity "
"(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL, c VARCHAR(8))"
)
cur.execute(
"INSERT INTO t_lv_parity VALUES (?, ?, ?, ?)",
(2001, text, 10, "tail"),
)
cur.execute("SELECT a, s, b, c FROM t_lv_parity")
assert cur.fetchone() == (2001, text, 10, "tail")
def test_odd_length_lvarchar_reproduces_the_report(
conn_params: ConnParams,
) -> None:
"""The exact failure shape: INT8 decoded as its own value shifted one
byte left (10 -> 2560), and the following string losing a character."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_report "
"(a INT8 NOT NULL, k LVARCHAR(512) NOT NULL,"
" b INT8 NOT NULL, v LVARCHAR(1024))"
)
cur.execute(
"INSERT INTO t_lv_report VALUES (?, ?, ?, ?)",
(2001, "PackageRoot", 10, "/content/package"),
)
cur.execute("SELECT a, k, b, v FROM t_lv_report")
row = cur.fetchone()
assert row == (2001, "PackageRoot", 10, "/content/package")
assert row[2] != 2560, "INT8 shifted one byte left"
assert row[3].startswith("/"), "leading character lost"
# --------------------------------------------------------------------------
# NULL — the missing length field
# --------------------------------------------------------------------------
def test_null_lvarchar_does_not_shift_next_column(
conn_params: ConnParams,
) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_null "
"(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL)"
)
cur.execute("INSERT INTO t_lv_null VALUES (?, ?, ?)", (3001, None, 20))
cur.execute("SELECT a, s, b FROM t_lv_null")
assert cur.fetchone() == (3001, None, 20)
def test_null_and_empty_lvarchar_are_distinguished(
conn_params: ConnParams,
) -> None:
"""They differ only in the indicator byte, so it's easy to conflate."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_ne (k INT, s LVARCHAR(64), tail INT)"
)
cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (1, None, 111))
cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (2, "", 222))
cur.execute("SELECT k, s, tail FROM t_lv_ne ORDER BY k")
assert cur.fetchall() == [(1, None, 111), (2, "", 222)]
def test_consecutive_null_lvarchars(conn_params: ConnParams) -> None:
"""Each NULL under-read by 4 bytes, so several in a row compound."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_many_null "
"(a INT8 NOT NULL, s1 LVARCHAR(256), s2 LVARCHAR(256),"
" s3 LVARCHAR(256), b INT8 NOT NULL)"
)
cur.execute(
"INSERT INTO t_lv_many_null VALUES (?, ?, ?, ?, ?)",
(4001, None, None, None, 40),
)
cur.execute("SELECT a, s1, s2, s3, b FROM t_lv_many_null")
assert cur.fetchone() == (4001, None, None, None, 40)
# --------------------------------------------------------------------------
# Wide, mixed shape — several LVARCHARs interleaved with other types
# --------------------------------------------------------------------------
_WIDE_COLS = [
"tag", "key_txt", "def_txt", "cur_txt", "note_txt",
"ver", "made_by", "ident", "made_on",
]
_WIDE_ROW = (
"Gadget",
"PackageRoot", # 11 — ODD
None, # NULL
"/content/package", # 16
"z", # 1 — ODD
77,
"maker",
3001,
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
)
def _make_wide(cur) -> None:
cur.execute(
"CREATE TEMP TABLE t_lv_wide ("
" tag VARCHAR(32) NOT NULL,"
" key_txt LVARCHAR(512) NOT NULL,"
" def_txt LVARCHAR(1024),"
" cur_txt LVARCHAR(1024),"
" note_txt LVARCHAR(1024),"
" ver INT8 DEFAULT 0 NOT NULL,"
" made_by VARCHAR(100),"
" ident INT8 NOT NULL,"
" made_on DATETIME YEAR TO FRACTION(5))"
)
cur.execute(
f"INSERT INTO t_lv_wide VALUES ({', '.join(['?'] * len(_WIDE_COLS))})",
_WIDE_ROW,
)
def test_wide_mixed_row(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
_make_wide(cur)
cur.execute(f"SELECT {', '.join(_WIDE_COLS)} FROM t_lv_wide")
assert cur.fetchone() == _WIDE_ROW
@pytest.mark.parametrize("shift", range(len(_WIDE_COLS)))
def test_wide_row_survives_column_reordering(
conn_params: ConnParams, shift: int
) -> None:
"""The reporter isolated this by reordering columns — values were
correct first in the list and wrong later on. Rotating the projection
exercises every position for every type."""
with _connect(conn_params) as conn:
cur = conn.cursor()
_make_wide(cur)
order = _WIDE_COLS[shift:] + _WIDE_COLS[:shift]
cur.execute(f"SELECT {', '.join(order)} FROM t_lv_wide")
expected = tuple(_WIDE_ROW[_WIDE_COLS.index(c)] for c in order)
assert cur.fetchone() == expected
def test_select_star_wide_row(conn_params: ConnParams) -> None:
"""``SELECT *`` raised IndexError once enough LVARCHARs accumulated."""
with _connect(conn_params) as conn:
cur = conn.cursor()
_make_wide(cur)
cur.execute("SELECT * FROM t_lv_wide")
row = cur.fetchone()
names = [d[0] for d in cur.description]
assert row == tuple(_WIDE_ROW[_WIDE_COLS.index(n)] for n in names)
def test_repeated_lvarchar_columns(conn_params: ConnParams) -> None:
"""Selecting the same LVARCHAR twice doubles any per-column drift."""
with _connect(conn_params) as conn:
cur = conn.cursor()
_make_wide(cur)
cur.execute(
"SELECT ident, key_txt, ident, key_txt, ver FROM t_lv_wide"
)
assert cur.fetchone() == (3001, "PackageRoot", 3001, "PackageRoot", 77)

76
tests/test_package_metadata.py Normal file
View File

@ -0,0 +1,76 @@
"""Guards on the package's own metadata.
The distribution is ``informix-driver``; the import module is
``informix_db``. That mismatch is a live trap: ``importlib.metadata``
needs the *distribution* name, so the string can't be derived from
``__name__``, and getting it wrong fails silently — ``__version__``
degrades to the ``0.0.0+local`` sentinel instead of raising.
That is exactly what shipped between the 2026-05-08 rename and
2026.08.27: ``__init__`` still asked for ``informix-db``, so anyone who
installed the renamed package saw ``0.0.0+local``. It went unnoticed
because a stale ``informix-db`` distribution lingered in the dev
environment and answered the query.
"""
from __future__ import annotations
from pathlib import Path
import pytest
import tomllib
import informix_db
PYPROJECT = Path(__file__).resolve().parent.parent / "pyproject.toml"
@pytest.fixture(scope="module")
def pyproject() -> dict:
if not PYPROJECT.is_file():
pytest.skip("pyproject.toml not present (installed-package test run)")
with PYPROJECT.open("rb") as handle:
return tomllib.load(handle)
def test_version_is_not_the_unresolved_sentinel() -> None:
"""If this fails, the distribution name in ``__init__`` no longer
matches ``[project].name`` and every user sees a bogus version."""
assert informix_db.__version__ != "0.0.0+local", (
"__version__ fell back to its sentinel — the distribution name "
"looked up in informix_db/__init__.py does not match the installed "
"package. Check [project].name in pyproject.toml."
)
def test_version_matches_pyproject(pyproject: dict) -> None:
declared = pyproject["project"]["version"]
# PEP 440 normalisation drops leading zeros: 2026.08.27 -> 2026.8.27.
normalised = ".".join(
str(int(part)) if part.isdigit() else part
for part in declared.split(".")
)
assert informix_db.__version__ in {declared, normalised}
def test_init_looks_up_the_declared_distribution_name(pyproject: dict) -> None:
"""Catch the rename trap directly: the name passed to
``importlib.metadata.version()`` must equal ``[project].name``."""
source = (
Path(informix_db.__file__).resolve().parent / "__init__.py"
).read_text(encoding="utf-8")
declared_name = pyproject["project"]["name"]
assert f'version("{declared_name}")' in source, (
f"informix_db/__init__.py should call version({declared_name!r}) to "
"match [project].name in pyproject.toml"
)
def test_version_is_importable_and_stringy() -> None:
assert isinstance(informix_db.__version__, str)
assert informix_db.__version__
def test_public_surface_is_exported() -> None:
for name in informix_db.__all__:
assert hasattr(informix_db, name), f"{name} in __all__ but not defined"

248
tests/test_placeholder_rewrite.py Normal file
View File

@ -0,0 +1,248 @@
"""Rewriting :N placeholders without rewriting the SQL around them.
We advertise ``paramstyle="numeric"`` to match Informix's ESQL/C
convention, and the wire protocol takes ``?``, so placeholders get
rewritten on the way out. That was ``re.sub(r":(\\d+)", "?", sql)``,
which cannot see a string literal and therefore rewrote the inside of
one::
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
stored as 'http://host?/x'
Any ``HH:MM`` time, any URL with a port, any aspect ratio, any
``key:value`` string. It wrote wrong data and said nothing.
Nothing about it was opt-in either. The rewrite runs whenever a
statement has parameters, whatever placeholder style the caller actually
used, so writing ``?`` everywhere and never touching numeric style did
not protect you. And it changed the placeholder *count* while
``num_qmarks`` was still computed from ``len(params)``, leaving driver
and server disagreeing about how many binds exist.
The lexical rules below are Informix's own, measured against 12.10,
14.10 and 15 rather than assumed from standard SQL. The one that matters
most is the backslash: ``'a\\'b'`` is an *unterminated string* to
Informix (``-282``), not an escaped quote. A scanner written to
Postgres habits would desync on it and corrupt everything after.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from informix_db.cursors import _rewrite_numeric_to_qmark as rewrite
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# Substitution happens where it should
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("sql", "expected"),
[
("SELECT * FROM t WHERE a = :1", "SELECT * FROM t WHERE a = ?"),
(
"SELECT * FROM t WHERE a = :1 AND b = :2",
"SELECT * FROM t WHERE a = ? AND b = ?",
),
("SELECT * FROM t WHERE a = :10", "SELECT * FROM t WHERE a = ?"),
# Already-? SQL is returned untouched, and cheaply.
("SELECT * FROM t WHERE a = ?", "SELECT * FROM t WHERE a = ?"),
("", ""),
],
)
def test_placeholders_are_rewritten(sql: str, expected: str) -> None:
assert rewrite(sql) == expected
# ---------------------------------------------------------------------------
# ...and nowhere else
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("label", "sql", "expected"),
[
(
"url-with-port",
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = :1",
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?",
),
(
"time-of-day",
"INSERT INTO log VALUES ('started at 09:15:30', :1)",
"INSERT INTO log VALUES ('started at 09:15:30', ?)",
),
(
"aspect-ratio",
"UPDATE t SET ratio = '16:9' WHERE id = :1",
"UPDATE t SET ratio = '16:9' WHERE id = ?",
),
(
"doubled-quote-escape",
"SELECT 'it''s 10:30' FROM t WHERE a = :1",
"SELECT 'it''s 10:30' FROM t WHERE a = ?",
),
(
"delimited-identifier",
'SELECT "col:1" FROM t WHERE a = :1',
'SELECT "col:1" FROM t WHERE a = ?',
),
(
"line-comment",
"-- ticket :99\nSELECT * FROM t WHERE a = :1",
"-- ticket :99\nSELECT * FROM t WHERE a = ?",
),
(
"block-comment",
"SELECT /* not :99 */ * FROM t WHERE a = :1",
"SELECT /* not :99 */ * FROM t WHERE a = ?",
),
(
"brace-comment",
"SELECT { not :99 } * FROM t WHERE a = :1",
"SELECT { not :99 } * FROM t WHERE a = ?",
),
(
"cast-operator",
"SELECT a::INT FROM t WHERE a = :1",
"SELECT a::INT FROM t WHERE a = ?",
),
(
"colon-not-a-placeholder",
"SELECT a FROM t WHERE b = ':x' AND c = :1",
"SELECT a FROM t WHERE b = ':x' AND c = ?",
),
(
"literal-after-placeholder",
"SELECT * FROM t WHERE a = :1 AND b = '10:30'",
"SELECT * FROM t WHERE a = ? AND b = '10:30'",
),
],
)
def test_quotes_and_comments_are_left_alone(
label: str, sql: str, expected: str
) -> None:
assert rewrite(sql) == expected, label
def test_backslash_does_not_escape_a_quote() -> None:
"""Informix answers -282 for ``'a\\'b'``: the backslash is an ordinary
character and the string is unterminated. A scanner that treated it
as an escape would think it was still inside the literal and stop
substituting, or worse, resume in the wrong place."""
sql = "SELECT 'a\\' FROM t WHERE x = :1"
# The quote closes at the character after the backslash, so :1 is
# outside the literal and gets substituted.
assert rewrite(sql) == "SELECT 'a\\' FROM t WHERE x = ?"
def test_unterminated_quote_substitutes_nothing_further() -> None:
"""Under-substituting leaves the server to reject SQL that was
already malformed. Guessing would corrupt a literal."""
assert rewrite("SELECT 'oops :1 FROM t") == "SELECT 'oops :1 FROM t"
def test_unterminated_block_comment_substitutes_nothing_further() -> None:
assert rewrite("SELECT /* oops :1 FROM t") == "SELECT /* oops :1 FROM t"
def test_block_comments_do_not_nest() -> None:
"""Measured: ``/* a /* b */ c */`` is a syntax error on all three
servers, so the first ``*/`` closes the comment. Treating them as
nesting would swallow live SQL."""
assert (
rewrite("SELECT /* a /* b */ :1 FROM t")
== "SELECT /* a /* b */ ? FROM t"
)
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
)
@pytest.mark.integration
def test_colon_literals_round_trip(conn_params: ConnParams) -> None:
"""The bug as a user meets it: the value stored is not the value
written. Both placeholder styles, because the rewrite ran for both."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rw (id INT, s VARCHAR(60))")
cur.execute(
"INSERT INTO t_rw VALUES (?, 'http://host:8080/path')", (1,)
)
cur.execute("INSERT INTO t_rw VALUES (:1, 'at 09:15:30')", (2,))
cur.execute("SELECT id, s FROM t_rw ORDER BY id")
assert cur.fetchall() == [
(1, "http://host:8080/path"),
(2, "at 09:15:30"),
]
@pytest.mark.integration
def test_colon_literals_round_trip_through_executemany(
conn_params: ConnParams,
) -> None:
"""executemany rewrites unconditionally, so it had the bug too."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rw2 (id INT, s VARCHAR(40))")
cur.executemany(
"INSERT INTO t_rw2 VALUES (?, '16:9')", [(1,), (2,)]
)
cur.execute("SELECT DISTINCT s FROM t_rw2")
assert cur.fetchall() == [("16:9",)]
@pytest.mark.integration
def test_comment_bearing_sql_still_binds(conn_params: ConnParams) -> None:
"""A leading comment reaches PREPARE now that classification asks the
server. Make sure the rewriter agrees and doesn't eat a placeholder
that follows one."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"/* report: daily */ SELECT FIRST 1 tabid FROM systables "
"WHERE tabid > :1",
(0,),
)
assert cur.fetchone() is not None
@pytest.mark.integration
def test_placeholder_count_matches_after_rewrite(
conn_params: ConnParams,
) -> None:
"""The old regex could add placeholders the driver never counted,
leaving num_qmarks (from len(params)) disagreeing with the SQL. A
literal containing three colon-digit sequences is the shape that
used to break it."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rw3 (id INT, s VARCHAR(40))")
with contextlib.suppress(Exception):
cur.execute("DELETE FROM t_rw3")
cur.execute(
"INSERT INTO t_rw3 VALUES (:1, 'a:1 b:2 c:3')", (7,)
)
cur.execute("SELECT id, s FROM t_rw3")
assert cur.fetchone() == (7, "a:1 b:2 c:3")

396
tests/test_pool.py
View File

@ -7,12 +7,16 @@ safety, and clean shutdown.
from __future__ import annotations from __future__ import annotations
import asyncio
import contextlib
import struct
import threading import threading
import time import time
import pytest import pytest
import informix_db import informix_db
from informix_db import aio
from tests.conftest import ConnParams from tests.conftest import ConnParams
pytestmark = pytest.mark.integration pytestmark = pytest.mark.integration
@ -287,3 +291,395 @@ def test_pool_thread_safe_concurrent_acquires(
assert pool.size <= 4 assert pool.size <= 4
finally: finally:
pool.close() pool.close()
# -------- Phase 26: pool rollback-on-release (CRITICAL data-correctness bug) --------
def test_uncommitted_writes_invisible_to_next_acquirer(
logged_db_params: ConnParams,
) -> None:
"""Critical regression test for the dirty-pool-checkout bug.
Pre-Phase-26 behavior:
Request A acquires → INSERTs (no commit) → releases. Server
transaction stays open. Request B acquires the SAME connection
(max_size=1 forces reuse) → its first SELECT sees A's
uncommitted row (because it's running inside A's transaction).
Worse: if B then commits, A's writes land permanently. If B
errors before commit, A's writes silently roll back.
This is the same shape as psycopg2's pre-2.5 dirty-pool bug.
Post-Phase-26: pool.release() rolls back any open transaction
before adding the connection to ``_idle``. A's uncommitted
writes are gone before B ever sees the connection.
"""
pool = informix_db.create_pool(
host=logged_db_params.host,
port=logged_db_params.port,
user=logged_db_params.user,
password=logged_db_params.password,
database=logged_db_params.database,
server=logged_db_params.server,
min_size=1,
max_size=1, # forces A and B to share the connection
)
table = "p26_dirty_pool"
try:
# Setup: fresh table, autocommit so the CREATE lands
with pool.connection() as setup:
cur = setup.cursor()
with contextlib.suppress(Exception):
cur.execute(f"DROP TABLE {table}")
cur.execute(f"CREATE TABLE {table} (id INT, label VARCHAR(64))")
setup.commit()
# Request A: insert without committing, then release
a_conn = pool.acquire()
try:
cur = a_conn.cursor()
cur.execute(
f"INSERT INTO {table} VALUES (?, ?)", (1, "A's dirty write")
)
# Confirm A sees its own write inside its own transaction
cur.execute(f"SELECT COUNT(*) FROM {table}")
assert cur.fetchone() == (1,), "A should see its own write pre-release"
assert a_conn._in_transaction, "A's connection should be in_transaction"
finally:
pool.release(a_conn) # NO commit — this is the critical case
# Request B: acquire the same connection (max_size=1 guarantees reuse).
# Note: we don't assert on ``_in_transaction`` after acquire — the
# pool's ``_is_alive`` health probe runs SELECT 1 which opens a
# fresh transaction under autocommit=False. The data-correctness
# check (the COUNT below) is the actual ground truth: if Phase 26
# didn't apply, A's uncommitted row would still be visible because
# B would be running INSIDE A's leftover transaction.
b_conn = pool.acquire()
try:
assert b_conn is a_conn, "max_size=1 must yield the same connection"
cur = b_conn.cursor()
cur.execute(f"SELECT COUNT(*) FROM {table}")
(count,) = cur.fetchone()
assert count == 0, (
f"B sees {count} rows — A's uncommitted writes leaked across "
"the pool checkout boundary. Phase 26 fix did not apply."
)
finally:
pool.release(b_conn)
# Cleanup
with pool.connection() as cleanup:
cur = cleanup.cursor()
with contextlib.suppress(Exception):
cur.execute(f"DROP TABLE {table}")
cleanup.commit()
finally:
pool.close()
def test_committed_writes_survive_pool_checkout(
logged_db_params: ConnParams,
) -> None:
"""Counterpart to the previous test: COMMITTED writes must persist.
This guards against the obvious over-correction — if Phase 26's
rollback also somehow nukes already-committed work (e.g., via a
second BEGIN+ROLLBACK round-trip), the bug fix would itself be
a data-loss bug. This test fails if rollback runs when it
shouldn't.
"""
pool = informix_db.create_pool(
host=logged_db_params.host,
port=logged_db_params.port,
user=logged_db_params.user,
password=logged_db_params.password,
database=logged_db_params.database,
server=logged_db_params.server,
min_size=1,
max_size=1,
)
table = "p26_committed"
try:
with pool.connection() as setup:
cur = setup.cursor()
with contextlib.suppress(Exception):
cur.execute(f"DROP TABLE {table}")
cur.execute(f"CREATE TABLE {table} (id INT)")
setup.commit()
# Request A: insert + commit + release
with pool.connection() as a_conn:
cur = a_conn.cursor()
cur.execute(f"INSERT INTO {table} VALUES (?)", (42,))
a_conn.commit()
assert not a_conn._in_transaction
# Request B: should see the committed row
with pool.connection() as b_conn:
cur = b_conn.cursor()
cur.execute(f"SELECT id FROM {table}")
assert cur.fetchone() == (42,), (
"Committed write disappeared — Phase 26 rollback ran when "
"it shouldn't have."
)
with pool.connection() as cleanup:
cur = cleanup.cursor()
with contextlib.suppress(Exception):
cur.execute(f"DROP TABLE {table}")
cleanup.commit()
finally:
pool.close()
# -------- Phase 27: wire-lock thread-safety + async cancellation eviction --------
def test_concurrent_threads_on_one_connection_dont_interleave_pdus(
conn_params: ConnParams,
) -> None:
"""Phase 27 wire-lock regression test.
Per PEP 249 Threadsafety=1, threads aren't supposed to share
connections — but the async layer effectively does this when a
cancelled task's worker keeps running. We verify the wire lock
serializes correctly: two threads doing concurrent SELECTs on
one Connection should produce correct results, not garbled wire
state.
Without the wire lock, the two threads' PDU bytes interleave on
the socket and at least one query produces wrong results, raises
``ProtocolError``, or hangs.
"""
import threading
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
)
try:
results: list[int] = []
errors: list[Exception] = []
results_lock = threading.Lock()
def worker(query_id: int) -> None:
try:
for _ in range(20):
cur = conn.cursor()
cur.execute(
"SELECT FIRST 1 tabid FROM systables WHERE tabid = ?",
(query_id,),
)
(val,) = cur.fetchone()
cur.close()
with results_lock:
results.append(val)
except Exception as exc:
with results_lock:
errors.append(exc)
# Two threads, each doing 20 queries with distinct expected results
t1 = threading.Thread(target=worker, args=(1,))
t2 = threading.Thread(target=worker, args=(2,))
t1.start()
t2.start()
t1.join(timeout=30.0)
t2.join(timeout=30.0)
assert not t1.is_alive(), "thread 1 hung — wire lock failed"
assert not t2.is_alive(), "thread 2 hung — wire lock failed"
assert errors == [], (
f"Threads errored out — likely PDU interleaving: {errors!r}"
)
# Each worker did 20 queries, so 40 results total. Each result
# should be the query_id its thread used.
assert results.count(1) == 20
assert results.count(2) == 20
finally:
conn.close()
async def test_async_wait_for_cancellation_evicts_connection(
conn_params: ConnParams,
) -> None:
"""Phase 27 async-cancellation regression test.
Before Phase 27, a cancelled awaitable left the connection in the
pool's idle list with a possibly-still-running worker writing to
its socket. Now: cancellation routes to ``broken=True``, and the
pool evicts the connection rather than recycling it.
"""
pool = await aio.create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
min_size=0,
max_size=2,
)
try:
# Force-grow to 1 connection so we have something to evict
async with pool.connection() as warmup_conn:
cur = await warmup_conn.cursor()
await cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
await cur.fetchone()
await cur.close()
size_before = pool.size
assert size_before == 1, f"expected 1 connection, got {size_before}"
# Trigger cancellation mid-query.
async def slow_query() -> None:
async with pool.connection() as conn:
cur = await conn.cursor()
# A query that will run for >100ms on the dev image:
# systables join itself a few times.
await cur.execute(
"SELECT COUNT(*) FROM systables a, systables b, "
"systables c WHERE a.tabid > 0"
)
await cur.fetchone()
await cur.close()
# Use pytest.raises (NOT contextlib.suppress) so the test fails
# if the timeout never fires — otherwise the test could pass on
# a fast CI runner where the query completes within 1ms,
# silently skipping the cancellation path it claims to test.
with pytest.raises(asyncio.TimeoutError):
await asyncio.wait_for(slow_query(), timeout=0.001)
# After cancellation, the connection must NOT have rejoined the
# pool's idle list. It should have been evicted (broken=True).
# Allow a moment for the release to complete.
await asyncio.sleep(0.5)
assert pool.size <= size_before, (
f"Connection wasn't evicted on cancellation; pool.size={pool.size} "
f"(expected ≤ {size_before}). The cancelled connection rejoined "
"the idle list — Phase 27 fix did not apply."
)
finally:
await pool.close()
# -------- Phase 29: deferred-cleanup queue --------
def test_enqueue_cleanup_drains_on_next_send_pdu(
conn_params: ConnParams,
) -> None:
"""Phase 29 regression test: cleanup PDUs queued by the finalizer
must actually drain on the next normal operation.
Simulates the scenario where a cursor finalizer couldn't acquire
the wire lock and enqueued its CLOSE+RELEASE PDUs. The next
``_send_pdu`` (any normal cursor operation) should drain the queue
*before* sending its own PDU — atomic completion under the wire
lock so the wire stays consistent.
We can't easily fire the finalizer in a controlled way, so we
inject the cleanup PDUs directly via ``_enqueue_cleanup`` and
confirm they're consumed.
"""
from informix_db._messages import MessageType
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
)
try:
# Inject something harmless into the pending queue. SQ_EOT
# alone is a valid no-op PDU on its own — server replies with
# nothing further (no actual statement to release), which the
# drain will handle.
# Actually, an empty drain only works with a real wire op;
# let's use an actual no-op: just SQ_EOT, which is what
# NULL-payload PDUs end with. But the simplest safe injection
# is a CLOSE for a non-existent statement — the server returns
# SQ_ERR which our drain handles.
# To keep this test deterministic without depending on server
# error semantics, just verify the queue mechanism itself:
# enqueue empty marker bytes, force drain, observe the queue
# is empty after.
assert conn._pending_cleanup == []
# Use a real benign cleanup: send a CLOSE PDU for whatever
# was the most recent cursor (none — server returns an error
# we drain). This exercises the queue path without requiring
# a leaked-cursor scenario to actually leak first.
# Simpler: enqueue an SQ_EOT "ping" that the drain will swallow.
eot_pdu = struct.pack("!h", MessageType.SQ_EOT)
with conn._wire_lock:
conn._enqueue_cleanup([eot_pdu])
assert conn._pending_cleanup == [eot_pdu]
# Trigger drain by calling _drain_pending_cleanup directly
# under the wire lock. Drain might force-close on
# unexpected server response; the queue should still be
# cleared regardless.
with contextlib.suppress(Exception):
conn._drain_pending_cleanup()
assert conn._pending_cleanup == [], (
"Phase 29 fix did not apply: pending cleanup PDU was "
"not consumed by _drain_pending_cleanup."
)
finally:
with contextlib.suppress(Exception):
conn.close()
def test_pending_cleanup_thread_safe_enqueue(
conn_params: ConnParams,
) -> None:
"""Multiple threads calling ``_enqueue_cleanup`` concurrently must
not corrupt the list (race on extend()).
The ``_cleanup_lock`` is a tiny critical section — this test
verifies it serializes correctly under heavy contention.
"""
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
)
try:
N_THREADS = 8
ENQUEUES_PER_THREAD = 50
marker = b"\x00\x0c" # SQ_EOT — harmless
def worker() -> None:
for _ in range(ENQUEUES_PER_THREAD):
conn._enqueue_cleanup([marker])
threads = [threading.Thread(target=worker) for _ in range(N_THREADS)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=10.0)
assert not t.is_alive()
# All enqueues should have landed; no race-loss
expected = N_THREADS * ENQUEUES_PER_THREAD
assert len(conn._pending_cleanup) == expected, (
f"Concurrent enqueue lost entries: got "
f"{len(conn._pending_cleanup)}, expected {expected}"
)
finally:
# Manually clear the queue so close() doesn't try to send
# nonsense PDUs we injected.
conn._pending_cleanup = []
conn.close()

64
tests/test_protocol.py
View File

@ -161,3 +161,67 @@ class TestFailureModes:
r = IfxStreamReader(BytesIO(b"\xff\xff")) r = IfxStreamReader(BytesIO(b"\xff\xff"))
with pytest.raises(ProtocolError, match="negative string length"): with pytest.raises(ProtocolError, match="negative string length"):
r.read_string_with_nul() r.read_string_with_nul()
# -------- Phase 30: server-error-text extraction (login rejection diagnostics) --------
def test_extract_server_error_text_finds_longest_printable_run() -> None:
"""Phase 30: ``_extract_server_error_text`` surfaces the human-readable
error string from an opaque rejection payload.
The function extracts the longest printable-ASCII run of length
≥ 8 and ≤ 256. Used to give login-rejection errors enough
diagnostic context to distinguish wrong-password from wrong-database
from version-mismatch — without doing the full structured decode
of the SLheader rejection block.
"""
from informix_db.connections import _extract_server_error_text
# Typical rejection payload: binary header + length-prefixed text
payload = (
b"\x00\x01\x00\x02\x00\x00\x00\x10"
b"incorrect password supplied"
b"\x00\x00\x00"
)
assert _extract_server_error_text(payload) == "incorrect password supplied"
def test_extract_server_error_text_picks_longest_run() -> None:
"""When multiple printable runs exist, return the longest."""
from informix_db.connections import _extract_server_error_text
payload = (
b"short\x00"
b"\x01\x02"
b"this is the longer error message we want\x00"
b"\x03"
b"medium length text\x00"
)
assert _extract_server_error_text(payload) == (
"this is the longer error message we want"
)
def test_extract_server_error_text_returns_none_when_too_short() -> None:
"""Runs under 8 chars don't qualify (avoids garbage matches)."""
from informix_db.connections import _extract_server_error_text
payload = b"\x00\x01abc\x02\x03def\x00ghij" # all runs < 8 chars
assert _extract_server_error_text(payload) is None
def test_extract_server_error_text_handles_empty_payload() -> None:
"""Empty input → None; doesn't crash."""
from informix_db.connections import _extract_server_error_text
assert _extract_server_error_text(b"") is None
def test_extract_server_error_text_caps_run_at_256() -> None:
"""Runs over 256 chars don't qualify (likely a binary block
misinterpreted as text)."""
from informix_db.connections import _extract_server_error_text
payload = b"a" * 300
assert _extract_server_error_text(payload) is None

459
tests/test_recovery_and_concurrency.py Normal file
View File

@ -0,0 +1,459 @@
"""Regression tests for error recovery, cursor lifecycle, and concurrency.
The type-matrix suite covers what a value looks like on the wire. This
covers everything around it: what happens after a statement fails, what
happens to abandoned cursors, and what happens when more than one thread
or task is involved.
Two bugs here were found by fuzzing, and both are the kind that never
show up in a single-threaded happy-path test:
* **A failed statement was never released.** Successful DML sent
PREPARE → EXECUTE → RELEASE; failing DML sent PREPARE → EXECUTE and
stopped. The leaked statement collided with the next PREPARE, and
every subsequent call on that connection returned a nonsense error
whose offset pointed back at the *failed* SQL. A duplicate-key
violation — about the most ordinary error an application can hit —
killed the connection outright.
* **Cancelling a pool acquire leaked the connection.** ``asyncio.to_thread``
cannot interrupt its worker, so a cancelled waiter left the worker to
finish and hand back a connection nobody owned. Under HTTP load, where
client disconnects cancel request tasks precisely while they wait for
a connection, the pool dies one slot at a time.
The tests that follow assert on *data* wherever they can, not merely the
absence of an exception: a worker reads back a value only it wrote, so a
crossed wire fails even when nothing raises.
"""
from __future__ import annotations
import asyncio
import contextlib
import threading
import pytest
import informix_db
from informix_db import aio
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
# Bounded: a leaked statement used to manifest as a hang.
read_timeout=25.0,
**kw,
)
def _pool_kw(conn_params: ConnParams) -> dict:
return {
"host": conn_params.host,
"port": conn_params.port,
"user": conn_params.user,
"password": conn_params.password,
"database": conn_params.database,
"server": conn_params.server,
"connect_timeout": 15.0,
"read_timeout": 30.0,
}
# ---------------------------------------------------------------------------
# Error recovery — a failed statement must not poison the connection
# ---------------------------------------------------------------------------
FAILING_SQL = [
pytest.param("SELECT FROM WHERE", id="syntax"),
pytest.param("SELECT * FROM no_such_table_xyz", id="no_such_table"),
pytest.param("SELECT no_such_col FROM systables", id="no_such_column"),
pytest.param("SELECT no_such_function_xyz(1) FROM systables", id="bad_func"),
pytest.param("SELECT 1/0 FROM systables", id="div_zero"),
pytest.param("SELECT 'notanumber'::INT FROM systables", id="bad_cast"),
]
@pytest.mark.parametrize("sql", FAILING_SQL)
def test_connection_survives_failed_statement(
conn_params: ConnParams, sql: str
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
for _ in range(4): # repeat: a leak accumulates
with pytest.raises(informix_db.Error):
cur.execute(sql)
cur.fetchall()
cur.execute("SELECT FIRST 1 tabid FROM systables ORDER BY tabid")
assert cur.fetchone() is not None
@pytest.mark.parametrize("autocommit", [True, False])
def test_duplicate_key_does_not_kill_the_connection(
conn_params: ConnParams, autocommit: bool
) -> None:
"""The bug that started this file. A unique-constraint violation left
the prepared statement allocated server-side; the next statement then
failed with a nonsense error (-255 "Not in transaction" in autocommit,
-285 otherwise) and the connection never recovered."""
with _connect(conn_params, autocommit=autocommit) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_dup (k INT PRIMARY KEY)")
cur.execute("INSERT INTO t_dup VALUES (1)")
if not autocommit:
conn.commit()
for i in range(5):
with pytest.raises(informix_db.IntegrityError):
cur.execute("INSERT INTO t_dup VALUES (1)")
if not autocommit:
conn.rollback()
cur.execute("SELECT k FROM t_dup")
assert cur.fetchall() == [(1,)], f"broken after {i + 1} violations"
def test_failed_dml_with_params_releases_statement(
conn_params: ConnParams,
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_nn (s VARCHAR(8) NOT NULL, k INT)")
for _ in range(4):
with pytest.raises(informix_db.Error):
cur.execute("INSERT INTO t_nn VALUES (?, ?)", (None, 1))
cur.execute("SELECT COUNT(*) FROM t_nn")
assert cur.fetchone() == (0,)
def test_new_cursor_works_after_error(conn_params: ConnParams) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
with pytest.raises(informix_db.Error):
cur.execute("SELECT * FROM no_such_table_xyz")
other = conn.cursor()
other.execute("SELECT FIRST 1 tabid FROM systables")
assert other.fetchone() is not None
other.close()
# ---------------------------------------------------------------------------
# Fetch batching — NFETCH is a byte budget, so batch edges move with width
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("n", [0, 1, 2, 3, 63, 64, 65, 199, 200, 201, 512, 1025])
def test_row_count_exact_across_fetch_styles(
conn_params: ConnParams, n: int
) -> None:
"""Every fetch style must agree, at counts that straddle plausible
server batch boundaries. A batching off-by-one shows up as a wrong
count, a duplicate, or a dropped row."""
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_batch "
"(k INT, a VARCHAR(32), b INT, c VARCHAR(32), d INT)"
)
if n:
cur.executemany(
"INSERT INTO t_batch VALUES (?, ?, ?, ?, ?)",
[(i, f"a{i}", i * 2, f"c{i}", i * 3) for i in range(n)],
)
want = [(i,) for i in range(n)]
sql = "SELECT k FROM t_batch ORDER BY k"
cur.execute(sql)
assert cur.fetchall() == want
cur.execute(sql)
got = []
while (row := cur.fetchone()) is not None:
got.append(row)
assert got == want
for size in (1, 7, 100):
cur.execute(sql)
got = []
while chunk := cur.fetchmany(size):
got.extend(chunk)
assert got == want, f"fetchmany({size}) disagreed at n={n}"
cur.execute(sql)
assert list(cur) == want
# ---------------------------------------------------------------------------
# Cursor lifecycle
# ---------------------------------------------------------------------------
def test_abandoned_partial_fetches_do_not_leak(
conn_params: ConnParams,
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_aband (k INT)")
cur.executemany(
"INSERT INTO t_aband VALUES (?)", [(i,) for i in range(200)]
)
for i in range(25):
c = conn.cursor()
c.execute("SELECT k FROM t_aband ORDER BY k")
c.fetchone()
if i % 2:
c.close()
else:
del c # rely on the finalizer
cur.execute("SELECT COUNT(*) FROM t_aband")
assert cur.fetchone() == (200,)
def test_re_execute_mid_fetch(conn_params: ConnParams) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_reex (k INT)")
cur.executemany(
"INSERT INTO t_reex VALUES (?)", [(i,) for i in range(100)]
)
for _ in range(10):
cur.execute("SELECT k FROM t_reex ORDER BY k")
cur.fetchone() # abandon mid-fetch
cur.execute("SELECT k FROM t_reex ORDER BY k")
assert len(cur.fetchall()) == 100
def test_interleaved_cursors_on_one_connection(
conn_params: ConnParams,
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_inter (k INT)")
cur.executemany(
"INSERT INTO t_inter VALUES (?)", [(i,) for i in range(50)]
)
a, b = conn.cursor(), conn.cursor()
a.execute("SELECT k FROM t_inter ORDER BY k")
b.execute("SELECT k FROM t_inter ORDER BY k DESC")
assert [a.fetchone() for _ in range(5)] == [(i,) for i in range(5)]
assert [b.fetchone() for _ in range(5)] == [
(i,) for i in range(49, 44, -1)
]
a.close()
b.close()
# ---------------------------------------------------------------------------
# Concurrency
# ---------------------------------------------------------------------------
def test_threads_sharing_one_connection_do_not_interleave(
conn_params: ConnParams,
) -> None:
"""One connection is one socket. Each thread reads back a value only
it supplied, so crossed wires fail even if nothing raises."""
failures: list[str] = []
lock = threading.Lock()
with _connect(conn_params, autocommit=True) as conn:
def worker(tid: int) -> None:
try:
for r in range(10):
cur = conn.cursor()
token = tid * 1000 + r
cur.execute(
"SELECT FIRST 1 ?::INT, ?::VARCHAR(16) FROM systables",
(token, f"t{tid}"),
)
row = cur.fetchone()
if row != (token, f"t{tid}"):
with lock:
failures.append(f"thread {tid}: got {row!r}")
return
cur.close()
except Exception as exc:
with lock:
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
for t in threads:
t.start()
for t in threads:
t.join()
assert not failures, failures
def test_pool_hands_out_clean_connections_under_load(
conn_params: ConnParams,
) -> None:
"""Borrowers that fail must still return a usable connection."""
failures: list[str] = []
lock = threading.Lock()
pool = informix_db.create_pool(
**_pool_kw(conn_params), min_size=2, max_size=4,
acquire_timeout=30.0, autocommit=True,
)
try:
def worker(tid: int) -> None:
try:
for r in range(8):
with pool.connection() as conn:
cur = conn.cursor()
if r % 3 == 0:
with contextlib.suppress(informix_db.Error):
cur.execute("SELECT * FROM no_such_tbl_xyz")
token = tid * 1000 + r
cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (token,)
)
got = cur.fetchone()
if got != (token,):
with lock:
failures.append(
f"thread {tid}: got {got!r}, want {token}"
)
return
cur.close()
except Exception as exc:
with lock:
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
for t in threads:
t.start()
for t in threads:
t.join()
finally:
with contextlib.suppress(Exception):
pool.close()
assert not failures, failures
def test_pool_does_not_leak_transactions_between_borrowers(
conn_params: ConnParams,
) -> None:
"""max_size=1 so both borrowers get the same underlying connection."""
pool = informix_db.create_pool(
**_pool_kw(conn_params), min_size=1, max_size=1,
acquire_timeout=30.0, autocommit=False,
)
try:
with pool.connection() as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_pool_txn")
cur.execute("CREATE TABLE t_pool_txn (k INT)")
conn.commit()
try:
for i in range(5):
with pool.connection() as conn: # dirties, never commits
conn.cursor().execute(
"INSERT INTO t_pool_txn VALUES (?)", (i,)
)
with pool.connection() as conn: # must not see it
cur = conn.cursor()
cur.execute("SELECT COUNT(*) FROM t_pool_txn")
assert cur.fetchone() == (0,), "transaction leaked"
conn.rollback()
finally:
with pool.connection() as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_pool_txn")
conn.commit()
finally:
with contextlib.suppress(Exception):
pool.close()
# ---------------------------------------------------------------------------
# Async
# ---------------------------------------------------------------------------
async def test_async_pool_survives_cancelled_acquires(
conn_params: ConnParams,
) -> None:
"""Cancelling a task blocked on acquire used to destroy that pool slot
permanently: ``asyncio.to_thread`` cannot interrupt its worker, so the
worker finished and handed back a connection nobody owned.
Two holders keep both connections checked out; three waiters block on
acquire and are cancelled there. Afterwards the pool must still serve.
"""
pool = await aio.create_pool(
**_pool_kw(conn_params), min_size=1, max_size=2, autocommit=True
)
try:
release = asyncio.Event()
async def holder(i: int) -> None:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (i,)
)
await cur.fetchone()
await release.wait()
async def waiter(i: int) -> tuple | None:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (i,)
)
return await cur.fetchone()
for cycle in range(2):
release.clear()
holders = [asyncio.create_task(holder(i)) for i in range(2)]
await asyncio.sleep(0.5) # both connections held
waiters = [asyncio.create_task(waiter(100 + i)) for i in range(3)]
await asyncio.sleep(0.5) # all blocked on acquire
for t in waiters:
t.cancel()
await asyncio.gather(*waiters, return_exceptions=True)
release.set()
await asyncio.gather(*holders, return_exceptions=True)
await asyncio.sleep(0.3) # orphan returns land
got = await asyncio.wait_for(waiter(200 + cycle), timeout=15)
assert got == (200 + cycle,), (
f"pool starved after cycle {cycle} — cancelled acquires leaked"
)
finally:
with contextlib.suppress(Exception):
await pool.close()
async def test_async_pool_concurrent_tasks_get_their_own_data(
conn_params: ConnParams,
) -> None:
pool = await aio.create_pool(
**_pool_kw(conn_params), min_size=2, max_size=4, autocommit=True
)
try:
async def worker(tid: int) -> None:
for r in range(6):
async with pool.connection() as conn:
cur = await conn.cursor()
token = tid * 1000 + r
await cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (token,)
)
assert await cur.fetchone() == (token,)
await asyncio.gather(*(worker(i) for i in range(6)))
finally:
with contextlib.suppress(Exception):
await pool.close()

405
tests/test_resilience.py Normal file
View File

@ -0,0 +1,405 @@
"""Phase 19 integration tests — connection resilience under fault injection.
Tests what happens when the network drops, the server crashes, or the
socket is forcibly torn down mid-conversation. Each test uses one of
two fault-injection mechanisms:
1. **Socket close from client side** — ``conn._sock._sock.close()``
simulates the OS forcibly closing the local end (e.g., kernel
socket-buffer overflow, signal handler).
2. **Controlled TCP proxy** (:class:`ControlledProxy` in ``_proxy.py``)
sits between the client and Informix; ``proxy.kill()`` severs the
connection with TCP RST, mimicking a router drop or server crash.
Both produce the same client-observable failure: the next I/O operation
raises ``OperationalError``. Verifying these paths catches several
classes of bugs:
- Hangs (waiting forever on a dead socket)
- Silent data corruption (treating EOF as a valid tuple)
- Double-fault (raising one error, then a different error on cleanup)
- Pool poisoning (returning a broken connection to the pool)
"""
from __future__ import annotations
import asyncio
import contextlib
import time
import pytest
import informix_db
from tests._proxy import ControlledProxy
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect_via_proxy(
proxy: ControlledProxy, params: ConnParams, **overrides
) -> informix_db.Connection:
kwargs = {
"host": "127.0.0.1",
"port": proxy.port,
"user": params.user,
"password": params.password,
"database": params.database,
"server": params.server,
"connect_timeout": 5.0,
"read_timeout": 5.0,
}
kwargs.update(overrides)
return informix_db.connect(**kwargs)
def _connect_direct(params: ConnParams, **overrides) -> informix_db.Connection:
kwargs = {
"host": params.host,
"port": params.port,
"user": params.user,
"password": params.password,
"database": params.database,
"server": params.server,
"connect_timeout": 5.0,
"read_timeout": 5.0,
}
kwargs.update(overrides)
return informix_db.connect(**kwargs)
# -------- Network-drop scenarios via ControlledProxy --------
def test_network_drop_mid_select_raises_operational_error(
conn_params: ConnParams,
) -> None:
"""Killing the proxy mid-query yields a clean ``OperationalError``."""
proxy = ControlledProxy(conn_params.host, conn_params.port)
proxy.start()
try:
conn = _connect_via_proxy(proxy, conn_params)
cur = conn.cursor()
# Drop the connection BEFORE issuing the query
proxy.kill()
# Next I/O must raise (not hang, not silently produce empty
# result set, not corrupt state)
with pytest.raises(informix_db.OperationalError):
cur.execute("SELECT FIRST 1 tabname FROM systables")
finally:
proxy.close()
def test_network_drop_after_describe_before_fetch(
conn_params: ConnParams,
) -> None:
"""Drop AFTER describe phase but before NFETCH — execute should raise."""
proxy = ControlledProxy(conn_params.host, conn_params.port)
proxy.start()
try:
conn = _connect_via_proxy(proxy, conn_params)
cur = conn.cursor()
# Establish the connection works first
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
assert cur.fetchone() == (1,)
# Now sever and verify the next query fails
proxy.kill()
with pytest.raises(informix_db.OperationalError):
cur.execute("SELECT 2 FROM systables WHERE tabid = 1")
finally:
proxy.close()
def test_network_drop_during_fetch_iteration(
conn_params: ConnParams,
) -> None:
"""Drop between fetches inside an open cursor.
For non-scrollable cursors (default), all rows are materialized
during ``execute()`` so subsequent ``fetchone`` calls don't do I/O —
they read from the local buffer. The drop is detected on the *next*
cursor lifecycle operation (close/release), but the in-memory rows
are still readable. We test that subsequent execute raises rather
than silently returning stale data.
"""
proxy = ControlledProxy(conn_params.host, conn_params.port)
proxy.start()
try:
conn = _connect_via_proxy(proxy, conn_params)
cur = conn.cursor()
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
# Materialized; we still have the rows
first = cur.fetchone()
assert first is not None
# Now sever the connection
proxy.kill()
# Continued reads from already-materialized buffer succeed
more = cur.fetchall()
assert len(more) == 4
# But a fresh execute over the dead socket fails
with pytest.raises(informix_db.OperationalError):
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
finally:
proxy.close()
# -------- Forcible local socket close --------
def test_local_socket_close_then_query(conn_params: ConnParams) -> None:
"""Forcibly close the underlying socket; next query raises cleanly."""
with _connect_direct(conn_params) as conn:
# Yank the rug
with contextlib.suppress(OSError):
conn._sock._sock.close()
cur = conn.cursor()
with pytest.raises(informix_db.OperationalError):
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
def test_io_error_marks_connection_unusable(conn_params: ConnParams) -> None:
"""After a transport failure, the connection's socket reports closed."""
conn = _connect_direct(conn_params)
try:
with contextlib.suppress(OSError):
conn._sock._sock.close()
cur = conn.cursor()
with contextlib.suppress(informix_db.Error):
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
# The IfxSocket's _force_close should have run
assert conn._sock.closed
finally:
with contextlib.suppress(Exception):
conn.close()
# -------- Pool eviction on connection failure --------
def test_pool_evicts_connection_after_proxy_kill(
conn_params: ConnParams,
) -> None:
"""A connection that died inside a pooled ``with`` block is NOT returned."""
proxy = ControlledProxy(conn_params.host, conn_params.port)
proxy.start()
try:
pool = informix_db.create_pool(
host="127.0.0.1",
port=proxy.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
min_size=0,
max_size=2,
)
try:
# Acquire one, kill the proxy mid-use
with (
pytest.raises(informix_db.OperationalError),
pool.connection() as conn,
):
cur = conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
cur.fetchone()
# Sever; next query inside the with-block will fail
proxy.kill()
cur.execute("SELECT 2 FROM systables WHERE tabid = 1")
# Pool should have evicted: zero connections owned now
assert pool.size == 0
finally:
pool.close()
finally:
proxy.close()
def test_pool_revives_after_all_idles_died(
conn_params: ConnParams,
) -> None:
"""If all idle connections are dead, acquire silently mints fresh ones."""
pool = informix_db.create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
min_size=2,
max_size=2,
)
try:
assert pool.idle_count == 2
# Forcibly kill both idle sockets
for c in pool._idle:
with contextlib.suppress(OSError):
c._sock._sock.close()
# The next acquire should detect dead connections via health
# check, drop them, and mint a fresh one.
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
assert cur.fetchone() == (1,)
finally:
pool.close()
# -------- Async cancellation --------
async def test_async_cancellation_during_execute(
conn_params: ConnParams,
) -> None:
"""Cancelling a coroutine mid-await leaves the pool in a sane state.
Uses ``asyncio.wait_for`` with an unrealistically short timeout so
the worker thread is still running ``cur.execute()`` when the
asyncio side gives up. The thread keeps going until I/O completes,
but the awaiting coroutine sees ``TimeoutError``. The connection
itself ends up in an ambiguous state — Phase 16's pool-eviction
policy kicks in: subsequent users get fresh connections.
"""
from informix_db import aio
pool = await aio.create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
min_size=0,
max_size=2,
acquire_timeout=2.0,
)
try:
# The cancellation behavior we want to verify: even if a query
# is interrupted, the pool stays healthy and subsequent queries
# work. We use a short timeout that may or may not fire (depends
# on local network speed); we assert the *post-condition*, not
# which path was taken.
async def worker() -> int | None:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute(
"SELECT FIRST 1 tabid FROM systables WHERE tabid = 1"
)
row = await cur.fetchone()
return row[0] if row else None
# Best-effort cancel attempt
with contextlib.suppress(asyncio.TimeoutError):
await asyncio.wait_for(worker(), timeout=0.001)
# Pool should still be usable for fresh queries
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
assert (await cur.fetchone()) == (1,)
finally:
await pool.close()
# -------- Cursor reuse after error --------
def test_cursor_can_be_reused_after_sql_error(
conn_params: ConnParams,
) -> None:
"""After a SQL-level error, the cursor remains usable for fresh queries."""
with _connect_direct(conn_params) as conn:
cur = conn.cursor()
with pytest.raises(informix_db.ProgrammingError):
cur.execute("SELECT * FROM no_such_table_zzz")
# Same cursor, fresh query — must work
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
assert cur.fetchone() == (1,)
def test_connection_survives_cursor_close_after_error(
conn_params: ConnParams,
) -> None:
"""Closing a cursor after an error doesn't poison the connection."""
with _connect_direct(conn_params) as conn:
cur = conn.cursor()
with pytest.raises(informix_db.ProgrammingError):
cur.execute("SELECT * FROM no_such_table_zzz")
cur.close()
# Brand-new cursor on the same connection
cur2 = conn.cursor()
cur2.execute("SELECT 1 FROM systables WHERE tabid = 1")
assert cur2.fetchone() == (1,)
# -------- Stress / timing --------
def test_pool_sustained_load_no_leaks(conn_params: ConnParams) -> None:
"""Open + close 50 connections via the pool; ``size`` doesn't grow unboundedly.
Catches the obvious leak class: each acquire/release minting a new
connection without recycling. Doesn't catch slow leaks (would need
tracemalloc for that), but is a sanity baseline.
"""
pool = informix_db.create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
min_size=0,
max_size=4,
)
try:
for _ in range(50):
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
cur.fetchone()
# Pool should have at most max_size connections owned
assert pool.size <= 4
finally:
pool.close()
def test_read_timeout_fires(conn_params: ConnParams) -> None:
"""A connection with ``read_timeout`` set raises on a hung server.
Set up via the proxy: connect, then kill the proxy *without* a TCP
RST so the read silently waits. The configured ``read_timeout``
should fire and produce a clear error rather than hanging forever.
"""
proxy = ControlledProxy(conn_params.host, conn_params.port)
proxy.start()
try:
conn = _connect_via_proxy(proxy, conn_params, read_timeout=1.0)
cur = conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
cur.fetchone()
# Soft-kill the upstream side WITHOUT triggering RST; reads will
# block forever (or until timeout). We do this by closing the
# listener, then severing only the upstream socket gracefully —
# the client-side socket sits there with no incoming data.
if proxy._upstream is not None:
with contextlib.suppress(OSError):
proxy._upstream.shutdown(2) # SHUT_RDWR
proxy._upstream.close()
# Mark the proxy as killed so its pump threads exit
proxy._killed = True
start = time.monotonic()
with pytest.raises(informix_db.OperationalError):
cur.execute("SELECT 2 FROM systables WHERE tabid = 1")
elapsed = time.monotonic() - start
# Should fire within ~2x the timeout, not hang forever
assert elapsed < 5.0
finally:
proxy.close()

98
tests/test_resultset_invariants.py Normal file
View File

@ -0,0 +1,98 @@
"""Phase 25 — invariant tripwires for parse_tuple_payload's fast-path dispatch.
These tests don't exercise behavior. They lock down the structural
invariants the optimized hot loop in :func:`informix_db._resultset.parse_tuple_payload`
relies on for correctness. Each test is a CI tripwire — if a future
contributor breaks an invariant, these fail at test time rather than
at a customer's wire-protocol mismatch six months later.
Lessons from Margaret Hamilton's review of Phases 23/24/25:
* The optimization is *correct* — but its correctness depends on
properties of unrelated tables (DECODERS keys, FIXED_WIDTHS keys,
IfxType flag bits) staying consistent.
* A comment at the table only helps if the next contributor reads it.
* A test fails loudly the moment the invariant is broken. Prefer that.
If one of these tests fires, **do not** simply update the test to
match the new state — that defeats the purpose. Instead read the
docstring on the failed test and the corresponding INVARIANT comment
in the source; either restore the property or refactor the
optimization to no longer depend on it.
"""
from __future__ import annotations
from informix_db._resultset import (
_COMPOSITE_UDT_TYPES,
_FIXED_WIDTH_TYPES,
_LENGTH_PREFIXED_SHORT_TYPES,
_NUMERIC_TYPES,
_TC_DATETIME,
_TC_INTERVAL,
_TC_LVARCHAR,
_TC_UDTFIXED,
_TC_UDTVAR,
)
from informix_db.converters import DECODERS, FIXED_WIDTHS
def test_fixed_width_types_disjoint_from_other_dispatch_sets() -> None:
"""parse_tuple_payload's fast path is silently wrong if the FIXED_WIDTHS
type set overlaps any other branch.
The optimization in ``parse_tuple_payload`` puts the FIXED_WIDTHS
branch FIRST. If a type is also in (e.g.) _NUMERIC_TYPES, the fast
path swallows it before the DECIMAL/MONEY-specific handler runs —
silently producing wrong values.
If this test fails, you've added a new entry somewhere that
overlaps. Either move it to FIXED_WIDTHS exclusively (and remove
its specialized branch) or remove it from FIXED_WIDTHS.
"""
other_branch_types = (
_LENGTH_PREFIXED_SHORT_TYPES
| _NUMERIC_TYPES
| _COMPOSITE_UDT_TYPES
| {_TC_LVARCHAR, _TC_DATETIME, _TC_INTERVAL, _TC_UDTFIXED, _TC_UDTVAR}
)
overlap = _FIXED_WIDTH_TYPES & other_branch_types
assert overlap == set(), (
f"FIXED_WIDTHS overlap with another parse_tuple_payload branch: {overlap}. "
f"See the INVARIANT comment on FIXED_WIDTHS in converters.py."
)
def test_every_fixed_width_type_has_a_decoder() -> None:
"""The fast path calls ``_decode_base(tc, raw, encoding)`` for every
FIXED_WIDTHS key. If a key has no entry in DECODERS, we'd raise
``NotImplementedError`` for that column — surprising the user.
If this test fails, you've added a key to FIXED_WIDTHS without
adding a corresponding decoder. Add the decoder, or remove the
key.
"""
missing = [tc for tc in FIXED_WIDTHS if tc not in DECODERS]
assert missing == [], (
f"FIXED_WIDTHS has keys without DECODERS entries: {missing}. "
f"Every fixed-width type must be decodable by _decode_base."
)
def test_decoders_keys_stay_below_0x100() -> None:
"""The Phase 24 optimization in ``_decode_base`` skips ``base_type()``
by relying on a structural guarantee: all DECODERS keys are ≤ 0xFF
and all flag bits in _types.py are ≥ 0x100, so a flagged type code
cannot coincidentally match a DECODERS key.
If this test fails, you've added a decoder for a type code with
bits ≥ 0x100. The collision-free guarantee weakens — re-introduce
``base_type()`` inside ``_decode_base`` (and remove the Phase 24
optimization), OR keep the new key but verify it cannot clash with
any flagged input.
"""
high_keys = [tc for tc in DECODERS if tc >= 0x100]
assert high_keys == [], (
f"DECODERS contains keys with bits >= 0x100: {high_keys}. "
f"See the INVARIANT comment on DECODERS in converters.py."
)

230
tests/test_row_reconciliation.py Normal file
View File

@ -0,0 +1,230 @@
"""End-of-row reconciliation, and the two bugs it found immediately.
Fourteen framing bugs reached users before this check existed. Every one
was the same defect — a column consuming the wrong number of bytes — and
not one raised at the point of the mistake. The wrong width produced a
plausible value and corrupted whatever came *next*, so the damage always
surfaced in a different column, row, or statement.
All fourteen were detectable for free. ``payload`` is a fully extracted
``bytes`` object of known length, so after decoding N columns the offset
must land exactly on the end. ``_row_not_consumed`` raises when it
doesn't, naming the column shape and the byte delta.
Turning the check on found two more bugs on its first run against the
existing suite:
* **Smart LOBs were read as a flat 72-byte field.** They use the UDT
envelope: 149 bytes populated (``[ind=0][len=144][144 hex chars]``),
5 bytes NULL. We consumed 72 and left 77 behind, so any LOB not in the
final column position corrupted everything after it. The 144 bytes are
the 72-byte locator *hex-encoded*, which means ``BlobLocator.raw`` had
never actually held a locator — only the first half of the hex text.
Nobody noticed because ``read_blob_column`` resolves LOBs through the
server-side ``lotofile`` function and never uses the locator.
* **NULL composite UDTs skipped their length field**, byte-for-byte the
LVARCHAR NULL bug in the branch twelve lines away.
Both were the same ``[indicator][int32 length][data]`` envelope, written
out by hand a third and fourth time. ``_read_udt_envelope`` is now the
single implementation.
"""
from __future__ import annotations
import contextlib
import io
import struct
import pytest
import informix_db
from informix_db._protocol import IfxStreamReader, ProtocolError
from informix_db._resultset import (
ColumnInfo,
_read_udt_envelope,
parse_tuple_payload,
)
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The check itself — unit level, no server
# ---------------------------------------------------------------------------
def _tuple_pdu(body: bytes) -> IfxStreamReader:
"""Wrap a payload in the SQ_TUPLE framing parse_tuple_payload expects."""
return IfxStreamReader(
io.BytesIO(struct.pack("!h", 0) + struct.pack("!i", len(body)) + body)
)
def test_under_read_is_detected() -> None:
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
with pytest.raises(ProtocolError, match="under-read by 6"):
parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5) + b"LEFTOV"), cols)
def test_exact_consumption_passes() -> None:
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
assert parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5)), cols) == (5,)
def test_error_names_the_column_shape() -> None:
""""row decode failed" is not actionable. The message has to say which
shape and by how much, so the next report arrives with the answer."""
cols = [
ColumnInfo(name="ident", type_code=2, raw_type_code=2, encoded_length=4),
ColumnInfo(name="qty", type_code=2, raw_type_code=2, encoded_length=4),
]
body = struct.pack("!i", 1) + struct.pack("!i", 2) + b"UNCONSUMED"
with pytest.raises(ProtocolError) as exc:
parse_tuple_payload(_tuple_pdu(body), cols)
msg = str(exc.value)
assert "ident" in msg and "qty" in msg, "message must name the columns"
assert "tc=2" in msg
assert "consumed 8 of 18" in msg, "message must give the byte counts"
assert "under-read by 10" in msg
# ---------------------------------------------------------------------------
# The shared UDT envelope
# ---------------------------------------------------------------------------
def test_envelope_reads_length_even_when_null() -> None:
"""The length belongs to the envelope, not the value. Returning early
on the indicator is the bug that hit LVARCHAR and then composites."""
payload = bytes([1]) + struct.pack("!i", 0) + b"NEXT"
offset, body = _read_udt_envelope(payload, 0)
assert body is None
assert offset == 5, "NULL envelope must still consume its length field"
assert payload[offset:] == b"NEXT"
def test_envelope_reads_body_when_present() -> None:
payload = bytes([0]) + struct.pack("!i", 3) + b"abc" + b"NEXT"
offset, body = _read_udt_envelope(payload, 0)
assert body == b"abc"
assert offset == 8
assert payload[offset:] == b"NEXT"
def test_envelope_rejects_negative_length() -> None:
"""A negative length off the wire would rewind the offset and silently
re-decode earlier bytes as a later column."""
payload = bytes([0]) + struct.pack("!i", -4) + b"abc"
with pytest.raises(ProtocolError, match="negative length"):
_read_udt_envelope(payload, 0)
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
pytestmark_integration = pytest.mark.integration
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host, port=conn_params.port, user=conn_params.user,
password=conn_params.password, database=conn_params.database,
server=conn_params.server, connect_timeout=15.0, read_timeout=30.0,
autocommit=True,
)
@pytest.mark.integration
def test_null_composite_does_not_shift_next_column(
conn_params: ConnParams,
) -> None:
"""A NULL SET made the following VARCHAR return '' instead of its value."""
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_nullset")
cur.execute(
"CREATE TABLE t_nullset (a INT, s SET(INT NOT NULL), b VARCHAR(10))"
)
try:
cur.execute("INSERT INTO t_nullset VALUES (7, SET{1,2}, 'abc')")
cur.execute("INSERT INTO t_nullset VALUES (8, NULL, 'xyz')")
cur.execute("SELECT a, s, b FROM t_nullset ORDER BY a")
rows = cur.fetchall()
assert rows[0][0] == 7 and rows[0][2] == "abc"
assert rows[1] == (8, None, "xyz"), (
"NULL composite shifted the following column"
)
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_nullset")
@pytest.mark.integration
@pytest.mark.parametrize("populated", [False, True])
def test_lob_in_non_final_position(
conn_params: ConnParams, populated: bool
) -> None:
"""A LOB was read as 72 bytes when it occupies 149 (or 5 when NULL), so
anything selected after it was garbage."""
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_lobmid")
try:
cur.execute(
"CREATE TABLE t_lobmid (a INT, b BLOB, c INT, d VARCHAR(10))"
)
except informix_db.Error:
pytest.skip("no sbspace configured; see make ifx-spaces")
try:
if populated:
cur.write_blob_column(
"INSERT INTO t_lobmid VALUES (?, BLOB_PLACEHOLDER, ?, ?)",
b"hello", (1, 222222, "tail"),
)
else:
cur.execute(
"INSERT INTO t_lobmid VALUES (1, NULL, 222222, 'tail')"
)
cur.execute("SELECT a, b, c, d FROM t_lobmid")
a, b, c, d = cur.fetchone()
assert (a, c, d) == (1, 222222, "tail"), (
"columns after the LOB were shifted"
)
if populated:
assert isinstance(b, informix_db.BlobLocator)
assert len(b.raw) == 72, (
"locator must be the decoded 72 bytes, not hex text"
)
else:
assert b is None
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_lobmid")
@pytest.mark.integration
def test_wide_mixed_row_reconciles(conn_params: ConnParams) -> None:
"""Belt and braces: a row using most of the tricky types must consume
its payload exactly. The check runs on every fetch, so merely getting
a row back proves reconciliation held."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_recon ("
" a INT8 NOT NULL, k LVARCHAR(512), n NCHAR(8), v VARCHAR(32),"
" d DECIMAL(16), b BOOLEAN, t DATETIME YEAR TO FRACTION(5),"
" tail INT)"
)
cur.execute(
"INSERT INTO t_recon VALUES (?, ?, ?, ?, ?, ?, CURRENT, ?)",
(1, "PackageRoot", "nch", "v", None, True, 424242),
)
cur.execute(
"SELECT a, k, n, v, d, b, t, tail FROM t_recon"
)
row = cur.fetchone()
assert row[0] == 1
assert row[-1] == 424242

435
tests/test_rows.py Normal file
View File

@ -0,0 +1,435 @@
"""Named row access, and the cost of it.
A field request: `row[11]` tells a reader nothing on a wide projection,
and `pyodbc` / `mssql-python` both hand back rows that answer to
position, column name, and attribute at once. `row_factory=Row` gives
the same three.
It is opt-in. Defaulting it on would tax the thing this driver is
measured against, since supporting `row["name"]` means `__getitem__`
becomes a Python method rather than C-level tuple indexing, which costs
roughly 39 ns on every subscript. Users who want readable column access
in application code should pay that; a bulk export that never looks at a
column by name should not.
`Row` subclasses `tuple`, so `row == (1, "x")` still holds and nothing
that already worked stops working. That constraint drove the design: the
existing suite compares fetched rows against plain tuples in hundreds of
places, and so does everybody's code.
"""
from __future__ import annotations
import contextlib
import pickle
import pytest
import informix_db
from informix_db.rows import _RESERVED, Row, make_row_class
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The type itself
# ---------------------------------------------------------------------------
def _row(fields, values):
return make_row_class(tuple(fields))(values)
def test_three_ways_to_reach_a_column() -> None:
row = _row(("tabid", "tabname"), (1, "systables"))
assert row[0] == 1
assert row["tabname"] == "systables"
assert row.tabname == "systables"
def test_still_equal_to_a_plain_tuple() -> None:
"""The compatibility constraint. Existing code and the existing test
suite compare fetched rows against tuples everywhere."""
row = _row(("a", "b"), (1, "x"))
plain = (1, "x")
assert row == plain
# Both directions: tuple.__eq__ on the left has to accept a subclass
# on the right, or `expected == fetched` assertions break.
assert plain == row
assert list(row) == [1, "x"]
assert len(row) == 2
a, b = row
assert (a, b) == (1, "x")
assert row in [(1, "x")]
def test_slice_degrades_to_a_plain_tuple() -> None:
"""A slice has no meaningful column mapping, so it should not pretend
to be a Row."""
row = _row(("a", "b", "c"), (1, 2, 3))
assert row[0:2] == (1, 2)
assert type(row[0:2]) is tuple
def test_negative_index_still_works() -> None:
assert _row(("a", "b"), (1, 2))[-1] == 2
def test_keys_and_asdict() -> None:
row = _row(("a", "b"), (1, "x"))
assert row.keys() == ("a", "b")
assert row._asdict() == {"a": 1, "b": "x"}
def test_repr_names_the_columns() -> None:
assert repr(_row(("a", "b"), (1, "x"))) == "Row(a=1, b='x')"
def test_pickles() -> None:
"""The per-shape class is built at runtime, so it cannot be pickled by
reference. Multiprocessing users would hit that immediately."""
row = _row(("a", "b"), (1, "x"))
restored = pickle.loads(pickle.dumps(row))
assert restored == (1, "x")
assert restored.b == "x"
def test_missing_column_says_what_is_there() -> None:
row = _row(("tabid", "tabname"), (1, "x"))
with pytest.raises(KeyError, match="tabid"):
_ = row["nope"]
with pytest.raises(AttributeError, match="tabid"):
_ = row.nope
def test_class_is_cached_per_shape() -> None:
"""A type() call per execute() would land on every small query."""
assert make_row_class(("a", "b")) is make_row_class(("a", "b"))
assert make_row_class(("a", "b")) is not make_row_class(("a", "c"))
def test_duplicate_names_resolve_to_the_first() -> None:
"""``SELECT tabid, tabid`` is legal and both columns are named
``tabid``. pyodbc gives the first; so do we."""
row = _row(("tabid", "tabid"), (1, 2))
assert row["tabid"] == 1
assert row.tabid == 1
assert row[1] == 2, "positional access must still reach the second"
@pytest.mark.parametrize("name", sorted(_RESERVED))
def test_a_column_always_beats_a_method_of_the_same_name(name: str) -> None:
"""Parametrized over the *computed* reserved set rather than a
hand-written list, because the hand-written list is what missed
``keys``, ``_asdict`` and ``_fields`` on the first attempt. Adding a
method to Row later cannot silently reopen the hole: this test grows
with it.
Without the guard these return a bound method, which is the exact
failure shape this driver spent a fortnight removing from its
decoders: a plausible-looking wrong answer, in silence."""
row = _row((name, "other"), (7, 9))
assert getattr(row, name) == 7
assert row[name] == 7
def test_shadowed_methods_stay_reachable_through_the_base_class() -> None:
row = _row(("count", "keys", "_asdict"), (1, 2, 3))
assert tuple.count(row, 1) == 1
assert Row.keys(row) == ("count", "keys", "_asdict")
assert Row._asdict(row) == {"count": 1, "keys": 2, "_asdict": 3}
def test_shadowing_fields_does_not_break_repr_or_asdict() -> None:
"""The machinery reads its own field list, so if a column named
``_fields`` shadowed it, repr and _asdict would report the column
value instead of the names. Mangled attributes keep them separate."""
row = _row(("_fields", "x"), ("not the names", 2))
assert row._fields == "not the names"
assert Row.keys(row) == ("_fields", "x")
assert repr(row) == "Row(_fields='not the names', x=2)"
def test_non_identifier_names_are_subscript_only() -> None:
"""Informix names expression columns things like ``(count(*))``,
which cannot be an attribute."""
row = _row(("(count(*))", "ok"), (5, 1))
assert row["(count(*))"] == 5
assert row.ok == 1
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
**kw,
)
@pytest.mark.integration
def test_default_is_still_a_plain_tuple(conn_params: ConnParams) -> None:
"""The opt-in has to be genuinely opt-in. Anyone who does not ask for
Row should not pay for it or see any behaviour change."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 1 tabid, tabname FROM systables")
row = cur.fetchone()
assert type(row) is tuple
with pytest.raises(TypeError):
_ = row["tabname"]
@pytest.mark.integration
def test_row_factory_on_the_connection(conn_params: ConnParams) -> None:
"""One line at connect time, then every cursor from it, which is what
'without any additional coding' has to mean in practice."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor()
cur.execute(
"SELECT FIRST 1 tabid, tabname FROM systables ORDER BY tabid"
)
row = cur.fetchone()
assert row[0] == row["tabid"] == row.tabid
assert row[1] == row["tabname"] == row.tabname
@pytest.mark.integration
def test_every_fetch_path_returns_rows(conn_params: ConnParams) -> None:
"""fetchone, fetchmany, fetchall and iteration each return rows by a
different route through the cursor."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
sql = "SELECT FIRST 4 tabid, tabname FROM systables ORDER BY tabid"
cur = conn.cursor()
cur.execute(sql)
assert cur.fetchone().tabname is not None
cur.execute(sql)
assert all(r.tabname is not None for r in cur.fetchmany(2))
cur.execute(sql)
assert all(r.tabname is not None for r in cur.fetchall())
cur.execute(sql)
assert all(r.tabname is not None for r in cur)
@pytest.mark.integration
def test_scrollable_cursor_returns_rows(conn_params: ConnParams) -> None:
"""Scrollable cursors return each row straight from the wire rather
than from the materialized list, so they are a separate path."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT tabid, tabname FROM systables ORDER BY tabid")
assert cur.fetch_first().tabname is not None
assert cur.fetch_absolute(1).tabname is not None
cur.close()
@pytest.mark.integration
def test_per_cursor_override(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
named = conn.cursor()
named.row_factory = informix_db.Row
plain = conn.cursor()
sql = "SELECT FIRST 1 tabid FROM systables"
named.execute(sql)
plain.execute(sql)
assert isinstance(named.fetchone(), Row)
assert type(plain.fetchone()) is tuple
@pytest.mark.integration
def test_informix_lowercases_so_the_obvious_name_works(
conn_params: ConnParams,
) -> None:
"""Informix folds unquoted identifiers, which is why the .lower() in
the workaround people write by hand is a no-op."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rows (Config_Key INT, Cnt INT)")
cur.execute("INSERT INTO t_rows VALUES (1, 2)")
cur.execute("SELECT Config_Key, Cnt FROM t_rows")
row = cur.fetchone()
assert row.config_key == 1
assert row.cnt == 2
@pytest.mark.integration
def test_pool_forwards_the_factory(conn_params: ConnParams) -> None:
pool = informix_db.create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
row_factory=informix_db.Row,
min_size=1,
max_size=2,
)
try:
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone().tabid is not None
finally:
pool.close()
def test_zero_column_row() -> None:
row = make_row_class(())(())
assert row == ()
assert row.keys() == ()
assert repr(row) == "Row()"
def test_wide_row() -> None:
"""500 columns: the name map is a dict, so this should be flat, but
it is the shape most likely to expose an off-by-one."""
fields = tuple(f"c{i}" for i in range(500))
row = make_row_class(fields)(tuple(range(500)))
assert row["c499"] == row.c499 == row[499] == 499
assert row["c0"] == row[0] == 0
def test_names_that_cannot_be_attributes_are_subscript_only() -> None:
row = _row(("col with space", "1leading_digit", ""), (1, 2, 3))
assert row["col with space"] == 1
assert row["1leading_digit"] == 2
assert row[""] == 3
def test_unicode_column_name() -> None:
row = _row(("café", "x"), (1, 2))
assert row["café"] == 1
assert row.café == 1
def test_dunder_named_column_is_subscript_only() -> None:
"""A dunder cannot be shadowed without breaking the object protocol,
so the column is reachable by subscript and the attribute keeps its
ordinary meaning. No real schema should notice."""
row = _row(("__class__", "ok"), (1, 2))
assert row["__class__"] == 1
assert row.__class__.__name__ == "Row"
def test_row_outlives_eviction_of_its_class_from_the_cache() -> None:
"""The class cache is bounded. A row already handed to the caller
holds its class alive, so eviction must not affect it."""
row = make_row_class(("z1", "z2"))((9, 8))
for i in range(300):
make_row_class((f"evict{i}", "b"))
assert (row.z1, row["z2"], row[0]) == (9, 8, 9)
def test_class_cache_is_bounded() -> None:
for i in range(400):
make_row_class((f"bounded{i}", "b"))
assert make_row_class.cache_info().currsize <= 256
# ---------------------------------------------------------------------------
# Row must not change a single value
# ---------------------------------------------------------------------------
_WIDE_DDL = """CREATE TABLE t_rowdiff (
c_int INT, c_small SMALLINT, c_big BIGINT, c_int8 INT8, c_serial SERIAL,
c_float FLOAT, c_smallfloat SMALLFLOAT, c_dec DECIMAL(16,4),
c_decu DECIMAL(16), c_money MONEY(12,2), c_char CHAR(10),
c_vchar VARCHAR(40), c_nchar NCHAR(8), c_lvar LVARCHAR(200), c_date DATE,
c_dt DATETIME YEAR TO FRACTION(5), c_ivl INTERVAL YEAR TO MONTH,
c_bool BOOLEAN, c_set SET(INT NOT NULL), c_tail INT)"""
_WIDE_COLS = (
"c_int,c_small,c_big,c_int8,c_serial,c_float,c_smallfloat,c_dec,c_decu,"
"c_money,c_char,c_vchar,c_nchar,c_lvar,c_date,c_dt,c_ivl,c_bool,c_set,"
"c_tail"
)
@pytest.mark.integration
def test_rows_are_value_identical_to_tuples(conn_params: ConnParams) -> None:
"""The strongest statement available: across every awkward type, with
a fully populated row and a fully NULL one, wrapping must change
nothing. If it does, Row is not a presentation layer, it is a bug."""
sql = f"SELECT {_WIDE_COLS} FROM t_rowdiff ORDER BY c_tail"
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_rowdiff")
cur.execute(_WIDE_DDL)
try:
cur.execute(
"INSERT INTO t_rowdiff VALUES (1,2,3,4,0,1.5,2.5,12.34,99,"
"9.99,'ch','vc','nc','lv',TODAY,CURRENT,"
"INTERVAL(1-2) YEAR TO MONTH,'t',SET{1,2},77)"
)
cur.execute(
"INSERT INTO t_rowdiff VALUES (NULL,NULL,NULL,NULL,0,NULL,"
"NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,"
"NULL,NULL,88)"
)
cur.execute(sql)
plain = cur.fetchall()
with _connect(conn_params, row_factory=informix_db.Row) as named_conn:
named = named_conn.cursor()
named.execute(sql)
rows = named.fetchall()
assert rows == plain, "wrapping changed a value"
names = _WIDE_COLS.split(",")
first = rows[0]
for i, name in enumerate(names):
assert first[i] == first[name] == getattr(first, name), name
assert rows[1]["c_int"] is None
assert rows[1].c_tail == 88
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_rowdiff")
@pytest.mark.integration
def test_threads_share_one_class_per_shape(conn_params: ConnParams) -> None:
"""The cache is process-wide and the pool hands connections to many
threads, so two threads running the same query must land on the same
class rather than racing to build competing ones."""
import threading
seen: list[type] = []
errors: list[Exception] = []
def worker() -> None:
try:
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor()
for _ in range(5):
cur.execute(
"SELECT FIRST 1 tabid, tabname FROM systables"
)
row = cur.fetchone()
assert row.tabname == row["tabname"] == row[1]
seen.append(type(row))
except Exception as exc:
errors.append(exc)
threads = [threading.Thread(target=worker) for _ in range(6)]
for t in threads:
t.start()
for t in threads:
t.join(60)
assert not errors, errors[:2]
assert len({id(c) for c in seen}) == 1, "same shape built more than once"

233
tests/test_scroll_cursor.py Normal file
View File

@ -0,0 +1,233 @@
"""Phase 17 integration tests — scroll cursor API.
The cursor materializes all rows on ``execute()`` (current behavior),
and Phase 17 adds index-based scroll methods on top of that:
``scroll()``, ``fetch_first()``, ``fetch_last()``, ``fetch_prior()``,
``fetch_absolute()``, ``fetch_relative()``, plus a ``rownumber`` property.
No new wire protocol — pure-Python feature on top of the existing
materialized result set.
"""
from __future__ import annotations
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=params.host,
port=params.port,
user=params.user,
password=params.password,
database=params.database,
server=params.server,
)
# -------- Basic scroll API --------
def test_fetchone_advances_index(conn_params: ConnParams) -> None:
"""Sequential ``fetchone`` advances ``rownumber`` 0, 1, 2, ..."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 4 tabname FROM systables ORDER BY tabid")
assert cur.rownumber is None # before-first
for expected_idx in range(4):
row = cur.fetchone()
assert row is not None
assert cur.rownumber == expected_idx
# After last row
assert cur.fetchone() is None
def test_fetch_first_resets_position(conn_params: ConnParams) -> None:
"""``fetch_first`` repositions to row 0 regardless of current position."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 5 tabname FROM systables ORDER BY tabid")
# Advance a few rows
cur.fetchone()
cur.fetchone()
cur.fetchone()
assert cur.rownumber == 2
# Reset
first = cur.fetch_first()
assert first is not None
assert cur.rownumber == 0
def test_fetch_last(conn_params: ConnParams) -> None:
"""``fetch_last`` jumps to the final row."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 5 tabname FROM systables ORDER BY tabid")
last = cur.fetch_last()
assert last is not None
assert cur.rownumber == 4
# No more rows after
assert cur.fetchone() is None
def test_fetch_prior(conn_params: ConnParams) -> None:
"""``fetch_prior`` moves backward one row.
Semantics match SQL-standard FETCH PRIOR: from "past-end" (after
fetchone returned None), the first fetch_prior returns the *last*
row. From row N, fetch_prior returns row N-1.
"""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 5 tabname FROM systables ORDER BY tabid")
rows = []
# Fetch all forward
while (row := cur.fetchone()) is not None:
rows.append(row)
# Now scroll back — 5 fetch_priors take us from past-end to row 0
for expected_idx in (4, 3, 2, 1, 0):
row = cur.fetch_prior()
assert row == rows[expected_idx]
assert cur.rownumber == expected_idx
# One more prior → before-first → None
assert cur.fetch_prior() is None
assert cur.rownumber is None
def test_fetch_absolute(conn_params: ConnParams) -> None:
"""``fetch_absolute(n)`` jumps to row ``n`` (0-indexed)."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"SELECT FIRST 5 tabid FROM systables ORDER BY tabid"
)
# Pre-collect all rows for comparison
all_rows = cur.fetchall()
cur.fetch_first() # rewind via the pre-fetch path
# Re-execute since fetchall consumed
cur.execute(
"SELECT FIRST 5 tabid FROM systables ORDER BY tabid"
)
# Jump around
assert cur.fetch_absolute(0) == all_rows[0]
assert cur.fetch_absolute(4) == all_rows[4]
assert cur.fetch_absolute(2) == all_rows[2]
assert cur.rownumber == 2
def test_fetch_absolute_negative(conn_params: ConnParams) -> None:
"""Negative absolute indexes count from the end (Python-style)."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
rows = cur.fetchall()
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
assert cur.fetch_absolute(-1) == rows[-1]
assert cur.rownumber == 4
assert cur.fetch_absolute(-2) == rows[-2]
def test_fetch_relative(conn_params: ConnParams) -> None:
"""``fetch_relative(n)`` moves ``n`` rows from current position."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
cur.fetchone() # at row 0
cur.fetchone() # at row 1
# Jump forward 2
cur.fetch_relative(2)
assert cur.rownumber == 3
# Jump back 3
cur.fetch_relative(-3)
assert cur.rownumber == 0
# -------- PEP 249 scroll() --------
def test_scroll_relative(conn_params: ConnParams) -> None:
"""``scroll(value)`` defaults to relative mode."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
cur.fetchone() # row 0
cur.scroll(2) # relative +2 → row 2
assert cur.rownumber == 2
cur.scroll(-1) # relative -1 → row 1
assert cur.rownumber == 1
def test_scroll_absolute(conn_params: ConnParams) -> None:
"""``scroll(value, mode='absolute')`` jumps to row N (1-indexed per PEP 249)."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
cur.scroll(3, mode="absolute") # to row 3 (1-indexed)
assert cur.rownumber == 2 # 0-indexed: position 2
# The next fetchone advances to position 3
cur.fetchone()
assert cur.rownumber == 3
def test_scroll_out_of_range_raises(conn_params: ConnParams) -> None:
"""``scroll`` past the result set raises IndexError per PEP 249."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 3 tabid FROM systables ORDER BY tabid")
with pytest.raises(IndexError):
cur.scroll(100)
with pytest.raises(IndexError):
cur.scroll(-5)
def test_scroll_invalid_mode_raises(conn_params: ConnParams) -> None:
"""Unknown scroll mode raises ProgrammingError."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 3 tabid FROM systables ORDER BY tabid")
with pytest.raises(informix_db.ProgrammingError, match="scroll mode"):
cur.scroll(1, mode="forward") # type: ignore[arg-type]
# -------- Edge cases --------
def test_scroll_on_empty_result_set(conn_params: ConnParams) -> None:
"""Scroll methods on empty result return None gracefully."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT tabid FROM systables WHERE tabid = -999")
assert cur.fetch_first() is None
assert cur.fetch_last() is None
assert cur.fetch_absolute(0) is None
assert cur.fetch_relative(1) is None
assert cur.fetch_prior() is None
def test_fetchall_after_partial_iteration(conn_params: ConnParams) -> None:
"""``fetchall()`` returns only remaining rows after partial iteration."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
cur.fetchone() # consume row 0
cur.fetchone() # consume row 1
remaining = cur.fetchall()
assert len(remaining) == 3 # rows 2, 3, 4
def test_fetchall_then_fetch_first_restarts(conn_params: ConnParams) -> None:
"""After ``fetchall``, ``fetch_first`` repositions back to row 0."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 3 tabid FROM systables ORDER BY tabid")
all_rows = cur.fetchall()
assert len(all_rows) == 3
# Cursor now at past-end. Rewind.
first = cur.fetch_first()
assert first == all_rows[0]

239
tests/test_scroll_cursor_server.py Normal file
View File

@ -0,0 +1,239 @@
"""Phase 18 integration tests — server-side scrollable cursor.
When ``conn.cursor(scrollable=True)`` is set, the cursor opens with
``SQ_SCROLL`` (tag 24) before ``SQ_OPEN``, doesn't materialize the
result set, and uses ``SQ_SFETCH`` (tag 23) for each fetch. The
server-side cursor stays open across scroll operations and is
closed by ``cursor.close()``.
The user-facing API surface (``fetch_first``, ``fetch_last``,
``fetch_prior``, ``fetch_absolute``, ``fetch_relative``, ``scroll``,
``rownumber``) is identical to the in-memory scroll mode (Phase 17).
The internal mechanism is what changes.
"""
from __future__ import annotations
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=params.host,
port=params.port,
user=params.user,
password=params.password,
database=params.database,
server=params.server,
autocommit=True,
)
# -------- Cursor lifecycle --------
def test_scrollable_cursor_opens_and_closes(conn_params: ConnParams) -> None:
"""A scrollable cursor reports its server-side state correctly."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
assert cur._scrollable is True
cur.execute("SELECT FIRST 3 tabid FROM systables ORDER BY tabid")
assert cur._server_cursor_open is True
cur.close()
assert cur._server_cursor_open is False
assert cur.closed is True
def test_scrollable_default_off(conn_params: ConnParams) -> None:
"""``conn.cursor()`` without args still produces a non-scrollable cursor."""
with _connect(conn_params) as conn:
cur = conn.cursor()
assert cur._scrollable is False
# -------- Forward sequential --------
def test_scrollable_sequential_fetchone(conn_params: ConnParams) -> None:
"""``fetchone`` advances through rows when scrollable=True."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
rows = []
while (row := cur.fetchone()) is not None:
rows.append(row[0])
assert rows == [1, 2, 3, 4, 5]
cur.close()
def test_scrollable_fetchall(conn_params: ConnParams) -> None:
"""``fetchall`` drains all rows from current position to end."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
rows = cur.fetchall()
assert [r[0] for r in rows] == [1, 2, 3, 4, 5]
cur.close()
# -------- Scroll API --------
def test_fetch_first_via_sfetch(conn_params: ConnParams) -> None:
"""``fetch_first`` sends SFETCH(ABSOLUTE, 1)."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
# Advance a few rows
cur.fetchone()
cur.fetchone()
# Reset
first = cur.fetch_first()
assert first == (1,)
assert cur.rownumber == 0
cur.close()
def test_fetch_last_caches_total_rows(conn_params: ConnParams) -> None:
"""``fetch_last`` populates ``_scroll_total_rows`` from the SFETCH(LAST) TUPID."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 7 tabid FROM systables ORDER BY tabid")
last = cur.fetch_last()
assert last is not None
assert cur._scroll_total_rows == 7
cur.close()
def test_fetch_prior_walks_backward(conn_params: ConnParams) -> None:
"""Sequential ``fetch_prior`` from the last row walks back to the first."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 4 tabid FROM systables ORDER BY tabid")
cur.fetch_last()
# last gave row 4; fetch_prior walks 3, 2, 1
assert cur.fetch_prior() == (3,)
assert cur.fetch_prior() == (2,)
assert cur.fetch_prior() == (1,)
assert cur.fetch_prior() is None
cur.close()
def test_fetch_absolute_random_access(conn_params: ConnParams) -> None:
"""``fetch_absolute(n)`` jumps to row ``n`` (0-indexed)."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 10 tabid FROM systables ORDER BY tabid")
# Random access in arbitrary order
assert cur.fetch_absolute(0) == (1,)
assert cur.fetch_absolute(9) == (10,)
assert cur.fetch_absolute(4) == (5,)
assert cur.fetch_absolute(2) == (3,)
assert cur.rownumber == 2
cur.close()
def test_fetch_absolute_negative(conn_params: ConnParams) -> None:
"""Negative absolute indexes count from the end (Python-style)."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 5 tabid FROM systables ORDER BY tabid")
# Without prior fetch_last, abs(-1) probes via SFETCH(LAST)
assert cur.fetch_absolute(-1) == (5,)
assert cur.fetch_absolute(-2) == (4,)
assert cur._scroll_total_rows == 5
cur.close()
def test_fetch_relative(conn_params: ConnParams) -> None:
"""``fetch_relative(n)`` moves ``n`` rows from the current position."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 8 tabid FROM systables ORDER BY tabid")
cur.fetch_first()
# Currently at row 0 (tabid=1); jump to position 4
assert cur.fetch_relative(4) == (5,)
# Jump back 3
assert cur.fetch_relative(-3) == (2,)
cur.close()
def test_scroll_relative_and_absolute(conn_params: ConnParams) -> None:
"""The PEP 249 ``scroll`` method works in both modes."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 6 tabid FROM systables ORDER BY tabid")
cur.fetchone() # row 0 (tabid=1)
cur.scroll(2, mode="relative") # to row 2
# rownumber tracks via TUPID; for scroll(no-fetch), our local
# _row_index moves but no SFETCH happens until next fetchone
assert cur.rownumber == 2
# Verify the position by fetching at the new position
cur.scroll(4, mode="absolute") # absolute index 4 (1-indexed → row 4 in API)
# absolute 4 in PEP 249 maps to _row_index = 3 (row at tabid=4)
assert cur.rownumber == 3
cur.close()
# -------- End-of-cursor / empty result set --------
def test_scrollable_empty_result_set(conn_params: ConnParams) -> None:
"""Scroll methods on empty result return None gracefully."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT tabid FROM systables WHERE tabid = -999")
assert cur.fetch_first() is None
assert cur.fetch_last() is None
assert cur.fetch_absolute(0) is None
assert cur.fetchone() is None
cur.close()
def test_scrollable_past_end_returns_none(conn_params: ConnParams) -> None:
"""Fetching past the end returns None rather than wrapping."""
with _connect(conn_params) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 3 tabid FROM systables ORDER BY tabid")
cur.fetch_last()
# We're at the last row; one more fetchone exceeds end
assert cur.fetchone() is None
cur.close()
# -------- Mixed: 1000-row scrollable workload --------
def test_scrollable_random_access(conn_params: ConnParams) -> None:
"""Random-access into a moderate-size result set without OOM.
Doesn't assume contiguous tabids (systables has gaps); instead,
cross-checks scrollable-cursor results against a non-scrollable
materialized fetch.
"""
with _connect(conn_params) as conn:
# Reference: pull the first 100 rows once, materialized
ref_cur = conn.cursor()
ref_cur.execute("SELECT FIRST 100 tabid FROM systables ORDER BY tabid")
reference = ref_cur.fetchall()
ref_cur.close()
assert len(reference) >= 50 # systables has plenty of rows
# Now hit the same query through a scrollable cursor and
# verify random-access matches the reference.
cur = conn.cursor(scrollable=True)
cur.execute("SELECT FIRST 100 tabid FROM systables ORDER BY tabid")
# Random sampling
for idx in (0, 1, 5, 25, len(reference) - 1):
assert cur.fetch_absolute(idx) == reference[idx]
# Walk backward from the middle
mid = len(reference) // 2
cur.fetch_absolute(mid)
for offset in range(1, 5):
assert cur.fetch_prior() == reference[mid - offset]
cur.close()

28
tests/test_smart_lob_write.py
View File

@ -247,6 +247,34 @@ def test_write_blob_column_requires_placeholder(
) )
def test_write_blob_column_rejects_multiple_placeholders(
logged_db_params: ConnParams, blob_table: str
) -> None:
"""Phase 28 regression: SQL containing BLOB_PLACEHOLDER twice is rejected.
Pre-Phase-28, ``str.replace`` silently substituted EVERY occurrence,
corrupting any SQL that legitimately contained the literal string
in (e.g.) a comment. Now we fail loudly so the user gets a clear
error rather than mysterious server-side syntax errors.
"""
with _connect(logged_db_params) as conn:
cur = conn.cursor()
with pytest.raises(
informix_db.ProgrammingError,
match=r"BLOB_PLACEHOLDER.*2 times",
):
cur.write_blob_column(
# The /* BLOB_PLACEHOLDER */ comment is the trap; in the
# old code this would have been substituted along with
# the real slot, producing a SQL syntax error from the
# server with no hint that the comment was the cause.
f"INSERT /* BLOB_PLACEHOLDER comment */ INTO {blob_table} "
f"VALUES (?, BLOB_PLACEHOLDER)",
b"data",
(1,),
)
def test_virtual_files_cleared_after_call( def test_virtual_files_cleared_after_call(
logged_db_params: ConnParams, blob_table: str logged_db_params: ConnParams, blob_table: str
) -> None: ) -> None:

Some files were not shown because too many files have changed in this diff Show More